Flattening nested pages in connector builder for API with rate limit

Summary

How to flatten nested pages in connector builder for an API with rate limit to retrieve content for each page


Question

I am using connector builder to make a connector for a service that has a rate limit. When we reach the rate limit we can get a response like this one. This API I am building for has “nested” pages. Is there a way I can tell AirByte to flatten out those pages so in my next stage can get the content for each page?



This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["connector-builder", "api", "rate-limit", "nested-pages", "flatten"]

Here is a sample of what the json response looks like pages array has pages where each page has a pages array, and so on:

    "status": "<status_code>",
    "body": {
        "object": "revision",
        "id": "<revision_id>",
        "parents": [
            "<parent_id>"
        ],
        "pages": [
            {
                "id": "<group_id>",
                "title": "<Group Title>",
                "kind": "<type_of_group>",
                "type": "<group_type>",
                "path": "<path_to_group>",
                "slug": "<group_slug>",
                "createdAt": "<creation_date_time>",
                "updatedAt": "<last_update_date_time>",
                "pages": [
                    {
                        "id": "<nested_page_id>",
                        "title": "<Nested Page Title>",
                        "kind": "<type_of_nested_page>",
                        "type": "<nested_document_type>",
                        "urls": {
                            "app": "<app_url_to_nested_page>"
                        },
                        "path": "<path_to_nested_page>",
                        "slug": "<nested_page_slug>",
                        "documentId": "<nested_document_id>",
                        "createdAt": "<nested_creation_date_time>",
                        "updatedAt": "<nested_last_update_date_time>",
                        "pages": [
                            {
                                "id": "<nested_page_id>",
                                "title": "<Nested Page Title>",
                                "kind": "<type_of_nested_page>",
                                "type": "<nested_document_type>",
                                "urls": {
                                    "app": "<app_url_to_nested_page>"
                                },
                                "path": "<path_to_nested_page>",
                                "slug": "<nested_page_slug>",
                                "documentId": "<nested_document_id>",
                                "createdAt": "<nested_creation_date_time>",
                                "updatedAt": "<nested_last_update_date_time>",
                                "pages": [],
                                "layout": {
                                    "cover": "<boolean>",
                                    "coverSize": "<size>",
                                    "title": "<boolean>",
                                    "description": "<boolean>",
                                    "tableOfContents": "<boolean>",
                                    "outline": "<boolean>",
                                    "pagination": "<boolean>"
                                }
                            }
                        ],
                        "layout": {
                            "cover": "<boolean>",
                            "coverSize": "<size>",
                            "title": "<boolean>",
                            "description": "<boolean>",
                            "tableOfContents": "<boolean>",
                            "outline": "<boolean>",
                            "pagination": "<boolean>"
                        }
                    }
                ],
                "layout": {
                    "cover": "<boolean>",
                    "coverSize": "<size>",
                    "title": "<boolean>",
                    "description": "<boolean>",
                    "tableOfContents": "<boolean>",
                    "outline": "<boolean>",
                    "pagination": "<boolean>"
                }
            }
        ],
        "files": [],
        "urls": {
            "app": "<app_url_to_revision>"
        },
        "createdAt": "<creation_date_time>",
        "type": "edits"
    },
    "headers": {
        "Date": "<current_date_time>",
        "Content-Type": "<content_type>"
    }
}```