Modifying Recurly Marketplace Connector

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

User seeks guidance on modifying the Recurly marketplace connector to control the number of API requests due to account-based limitations.


Question

Hey Airbyte community! :octavia-wave: I’m hoping to modify the existing Recurly marketplace connector so that I can modify how many requests get sent at a time. Recurly has account-based API limitations, and we’re integrating with it elsewhere.

I’m new to Airbyte and this is my first attempt to modify a connector. Any advice is appreciated!


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

['recurly-connector', 'api-limitations', 'modify-connector']
2

Hey Erin and welcome to the project! We don’t have documentation explaining the steps to contribute to manifest-only connectors, which is the case of Recurly. I’ll work to have this in future weeks. By the steps are:

  1. create a source-recurly in your workspace from scratch
  2. copy the content from this https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-recurly/manifest.yaml|file
  3. run tests to streams you changed
  4. make changes and go to top right corner Publish -> Contribute to Airbyte
    This will generate the pull requests with your changes.
3

You should be able to bypass manual copying by forking recurly directly as a new custom connector:

4

<@U07VD0FANPP> this is a new feature that <@U047ANT3J84> worked on! Both Ella and I would love to hear your feedback on it.

5

Woo! Thanks, everyone, for hopping in to help me :smile: Ella and Natik, I was able to fork the Recurly code directly in builder :rocket: and there are two things I’d like to work on (and maybe contribute back). Since I’m new to Airbyte I might need some guidance.

6
  1. I need to use the begin_time parameter so that I don’t overload our systems when calling out to the Recurly API. However, when I enter a timestamp in the required YYYY-MM-DDTHH:mm:ss format, I receive a ValueError:
    ValueError: time data '2024-11-11T00:00:00' does not match format '%Y-%m-%dT%H:%M:%SZ'
    I know that the API creds are working fine, since I was able to sample a few records in Builder. Any idea how I could fix the timestamp?
7
  1. I’d love to limit the number of API calls that are dedicated to this connector, since Recurly’s rate limits apply to the account not just to the key. I’ve seen something in <tap-recurly - Meltano Hub tap >that would achieve this by limiting the percent of quota limit dedicated to that sync. Is anything similar on the roadmap (or been implemented elsewhere that I could learn from)?
8

Hi <@U047ANT3J84>! Thanks again for your help :slightly_smiling_face: When I enter 2024-11-11T00:00:00Z or any timestamp format other than 2024-11-11T00:00:00, the UI doesn’t allow me to proceed with testing the connection. The set up source button is still illuminated but when I click it, nothing happens.

9

I’m going to layer on the questions as I think of them :slightly_smiling_face: I will look into how the quota limiting is set up in the Meltano connector (it’s open sourced as well); but I could also achieve the same objective of not overloading our rate limits in other ways. Would any of these options be more common or reusable?
• Set a limit on the number of calls per minute (or other time period)
• Increase the wait time between calls

10

Is the subscription_changes https://recurly.com/developers/api/v2021-02-25/index.html#operation/get_subscription_change|endpoint a part of the connector at this time? I see it listed in as an <Recurly Connector | Airbyte Documentation schema>, but I can’t find it as an endpoint in the Builder YAML.

11

<@U07VD0FANPP> are you editing this in cloud? If so I would love a link to your builder project as I can’t seem to reproduce the date issue, but definitely looks like something is up there!

12

Have asked about the proactive rate limiting

13

I think you’re right that that stream isn’t available. I looked at the (very old) PR where that line in the docs was added, and it didn’t seem to actually add a subscription changes stream - just a better output schema for pending_changes on the subscription stream https://github.com/airbytehq/airbyte/pull/9866/files#diff-dca03786c3fa8032d71c1a32a226beeac3dc4cbc06f08e0916ebcc06e1f149ccR118

14

<@U047ANT3J84> yes I’m doing this in Cloud. This is my first connector, and I haven’t modified anything in Builder. I am also getting the same error in Sources when I try to connect to Recurly 1.2.0 directly and enter the start time from there. I enter my API key, and it connects fine. You were able to get it to connect successfully, with a declared start time?

I submitted a support ticket in the Cloud UI but it didn’t give a ticket number back :disappointed: and now has disappeared. I mentioned that I was working with you on this issue. I’d prefer to keep the links to my project inside the UI and support portals, out of an abundance of caution, so hopefully submitting the ticket is okay!

15

I believe I figured it out :smile: In the manifest.yaml https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-recurly/manifest.yaml|file, lines 19004-19021:

        type: string
        description: &gt;-
          ISO8601 timestamp from which the replication from Recurly API will
          start from.
        examples:
        - "2021-12-01T00:00:00"
        pattern: ^$|^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}$
        order: 1
      end_time:
        type: string
        description: &gt;-
          ISO8601 timestamp to which the replication from Recurly API will stop.
          Records after that date won't be imported.
        examples:
        - "2021-12-01T00:00:00"
        pattern: ^$|^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}$
        order: 2```
Changing to
```pattern: ^$|^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$```
in Builder did the trick. It now allows me to enter dates such as `2024-11-14T00:00:00Z` into the UI and test the connection. I’m still having some issues with the streams, but that’s one down! :tada:
16

What’s the best way to remove streams that are not applicable to me? Some of them are causing errors when I test the newly-created custom connector, so I’m going through and commenting them out in Builder, but that will take a while!

17

If they’re causing problems just when syncing, you can opt not to sync those streams in the connection. If it’s in the builder itself that you’re testing it and don’t want them anymore, you can delete the streams with the trash can icon on each stream:

18

(it’s red on hover, not by default)

19

Very cool! I think there’s a troublesome steam in there that was causing issues for me in Builder and in the connection test. Now that I’ve commented out a few streams, I can use the Builder UI (previously not working).

Is there a way that I can re-sync the Recurly code into my forked copy? I’d like to log these issues in GitHub as I uncover them, but I’ve already removed the offending stream. It will also be good if I ever need to update the connector from source.

20

Hey Erin and welcome to the project! We don’t have documentation explaining the steps to contribute to manifest-only connectors, which is the case of Recurly. I’ll work to have this in future weeks. By the steps are:

  1. create a source-recurly in your workspace from scratch
  2. copy the content from this https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-recurly/manifest.yaml|file
  3. run tests to streams you changed
  4. make changes and go to top right corner Publish -> Contribute to Airbyte
    This will generate the pull requests with your changes.