Flattening nested pages in connector builder for API with rate limit

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

How to flatten nested pages in connector builder for an API with rate limit to retrieve content for each page


Question

I am using connector builder to make a connector for a service that has a rate limit. When we reach the rate limit we can get a response like this one. This API I am building for has “nested” pages. Is there a way I can tell AirByte to flatten out those pages so in my next stage can get the content for each page?


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["connector-builder", "api", "rate-limit", "nested-pages", "flatten"]
2

Here is a sample of what the json response looks like pages array has pages where each page has a pages array, and so on:

    "status": "<status_code>",
    "body": {
        "object": "revision",
        "id": "<revision_id>",
        "parents": [
            "<parent_id>"
        ],
        "pages": [
            {
                "id": "<group_id>",
                "title": "<Group Title>",
                "kind": "<type_of_group>",
                "type": "<group_type>",
                "path": "<path_to_group>",
                "slug": "<group_slug>",
                "createdAt": "<creation_date_time>",
                "updatedAt": "<last_update_date_time>",
                "pages": [
                    {
                        "id": "<nested_page_id>",
                        "title": "<Nested Page Title>",
                        "kind": "<type_of_nested_page>",
                        "type": "<nested_document_type>",
                        "urls": {
                            "app": "<app_url_to_nested_page>"
                        },
                        "path": "<path_to_nested_page>",
                        "slug": "<nested_page_slug>",
                        "documentId": "<nested_document_id>",
                        "createdAt": "<nested_creation_date_time>",
                        "updatedAt": "<nested_last_update_date_time>",
                        "pages": [
                            {
                                "id": "<nested_page_id>",
                                "title": "<Nested Page Title>",
                                "kind": "<type_of_nested_page>",
                                "type": "<nested_document_type>",
                                "urls": {
                                    "app": "<app_url_to_nested_page>"
                                },
                                "path": "<path_to_nested_page>",
                                "slug": "<nested_page_slug>",
                                "documentId": "<nested_document_id>",
                                "createdAt": "<nested_creation_date_time>",
                                "updatedAt": "<nested_last_update_date_time>",
                                "pages": [],
                                "layout": {
                                    "cover": "<boolean>",
                                    "coverSize": "<size>",
                                    "title": "<boolean>",
                                    "description": "<boolean>",
                                    "tableOfContents": "<boolean>",
                                    "outline": "<boolean>",
                                    "pagination": "<boolean>"
                                }
                            }
                        ],
                        "layout": {
                            "cover": "<boolean>",
                            "coverSize": "<size>",
                            "title": "<boolean>",
                            "description": "<boolean>",
                            "tableOfContents": "<boolean>",
                            "outline": "<boolean>",
                            "pagination": "<boolean>"
                        }
                    }
                ],
                "layout": {
                    "cover": "<boolean>",
                    "coverSize": "<size>",
                    "title": "<boolean>",
                    "description": "<boolean>",
                    "tableOfContents": "<boolean>",
                    "outline": "<boolean>",
                    "pagination": "<boolean>"
                }
            }
        ],
        "files": [],
        "urls": {
            "app": "<app_url_to_revision>"
        },
        "createdAt": "<creation_date_time>",
        "type": "edits"
    },
    "headers": {
        "Date": "<current_date_time>",
        "Content-Type": "<content_type>"
    }
}```
3

I’m confused :upside_down_face:
How does your request look? How does pagination work? And how does a response look when you’re not rate limited?

4

<@U069EMNRPA4> it is like this:
https://developer.gitbook.com/gitbook-api/reference/spaces/content

5

See <https://api.gitbook.com/v1/spaces/{spaceId}/content>

6

<@U069EMNRPA4> think you can give me a hint on how to do this?

7

Hey <@U069EMNRPA4> any chance to look at this?

8

Yep! I figured out one conversation yesterday, will look at this one today.

9

That request returns an object like:

Space

      :
   "pages": [
       {
                :
            "pages": [
                    ...
             ]
        }
   ]
}```
I want to have another requests request where the parent is pace and for every page (no matter what level it is at) to get the "page_content"

Something like this but currently it only gets page content at the first layer
10

Alright, thank you!
I want to help, but I’m slammed today. Don’t hesitate to remind me. I’ll try to take a look!

11

<@U069EMNRPA4> Here is your reminder you asked for

12

<@U069EMNRPA4> Is it possible do what I need to do for this?

13

posted to main channel for help:
https://airbytehq.slack.com/archives/C027KKE4BCZ/p1714494544428899

14

<@U069EMNRPA4> just checking in to see how it is going with looking into this