You should be able to bypass manual copying by forking recurly directly as a new custom connector:
<@U07VD0FANPP> this is a new feature that <@U047ANT3J84> worked on! Both Ella and I would love to hear your feedback on it.
Woo! Thanks, everyone, for hopping in to help me 
Ella and Natik, I was able to fork the Recurly code directly in builder 
and there are two things I’d like to work on (and maybe contribute back). Since I’m new to Airbyte I might need some guidance.
- I need to use the
begin_timeparameter so that I don’t overload our systems when calling out to the Recurly API. However, when I enter a timestamp in the requiredYYYY-MM-DDTHH:mm:ssformat, I receive a ValueError:
ValueError: time data '2024-11-11T00:00:00' does not match format '%Y-%m-%dT%H:%M:%SZ'
I know that the API creds are working fine, since I was able to sample a few records in Builder. Any idea how I could fix the timestamp?
- I’d love to limit the number of API calls that are dedicated to this connector, since Recurly’s rate limits apply to the account not just to the key. I’ve seen something in <tap-recurly - Meltano Hub tap >that would achieve this by limiting the percent of quota limit dedicated to that sync. Is anything similar on the roadmap (or been implemented elsewhere that I could learn from)?
Re your first question: I do see '%Y-%m-%dT%H:%M:%SZ' as the format in the YAML descriptor. I’m not sure why the UI is telling you differently (I assume you’re talking about the image below). Can you try 2024-11-11T00:00:00Z and let me know if it works?
Your second question is interesting - I don’t know of any implementations on our end that would have implemented that. Do you have any idea how it works under the hood (presumably by doing so you are setting some option on the request to tell the API to give you basically a stricter rate limit)?
Unfortunately behavior like that I think is unlikely to make it into builder-native since it’s particular and likely implemented differently by each connector that would implement it - it would probably be behavior implemented by a custom connector. But depending on how it’s set in the request, I could be wrong and maybe it’s easily hardcoded
Hi <@U047ANT3J84>! Thanks again for your help 
When I enter 2024-11-11T00:00:00Z or any timestamp format other than 2024-11-11T00:00:00, the UI doesn’t allow me to proceed with testing the connection. The set up source button is still illuminated but when I click it, nothing happens.
I’m going to layer on the questions as I think of them I will look into how the quota limiting is set up in the Meltano connector (it’s open sourced as well); but I could also achieve the same objective of not overloading our rate limits in other ways. Would any of these options be more common or reusable?
• Set a limit on the number of calls per minute (or other time period)
• Increase the wait time between calls
Is the subscription_changes https://recurly.com/developers/api/v2021-02-25/index.html#operation/get_subscription_change|endpoint a part of the connector at this time? I see it listed in as an <Recurly Connector | Airbyte Documentation schema>, but I can’t find it as an endpoint in the Builder YAML.
<@U07VD0FANPP> are you editing this in cloud? If so I would love a link to your builder project as I can’t seem to reproduce the date issue, but definitely looks like something is up there!
Have asked about the proactive rate limiting
I think you’re right that that stream isn’t available. I looked at the (very old) PR where that line in the docs was added, and it didn’t seem to actually add a subscription changes stream - just a better output schema for pending_changes on the subscription stream https://github.com/airbytehq/airbyte/pull/9866/files#diff-dca03786c3fa8032d71c1a32a226beeac3dc4cbc06f08e0916ebcc06e1f149ccR118
<@U047ANT3J84> yes I’m doing this in Cloud. This is my first connector, and I haven’t modified anything in Builder. I am also getting the same error in Sources when I try to connect to Recurly 1.2.0 directly and enter the start time from there. I enter my API key, and it connects fine. You were able to get it to connect successfully, with a declared start time?
I submitted a support ticket in the Cloud UI but it didn’t give a ticket number back 
and now has disappeared. I mentioned that I was working with you on this issue. I’d prefer to keep the links to my project inside the UI and support portals, out of an abundance of caution, so hopefully submitting the ticket is okay!
I believe I figured it out In the manifest.yaml https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-recurly/manifest.yaml|file, lines 19004-19021:
type: string
description: >-
ISO8601 timestamp from which the replication from Recurly API will
start from.
examples:
- "2021-12-01T00:00:00"
pattern: ^$|^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}$
order: 1
end_time:
type: string
description: >-
ISO8601 timestamp to which the replication from Recurly API will stop.
Records after that date won't be imported.
examples:
- "2021-12-01T00:00:00"
pattern: ^$|^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}$
order: 2```
Changing to
```pattern: ^$|^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$```
in Builder did the trick. It now allows me to enter dates such as `2024-11-14T00:00:00Z` into the UI and test the connection. I’m still having some issues with the streams, but that’s one down! :tada:What’s the best way to remove streams that are not applicable to me? Some of them are causing errors when I test the newly-created custom connector, so I’m going through and commenting them out in Builder, but that will take a while!
If they’re causing problems just when syncing, you can opt not to sync those streams in the connection. If it’s in the builder itself that you’re testing it and don’t want them anymore, you can delete the streams with the trash can icon on each stream:
(it’s red on hover, not by default)
Very cool! I think there’s a troublesome steam in there that was causing issues for me in Builder and in the connection test. Now that I’ve commented out a few streams, I can use the Builder UI (previously not working).
Is there a way that I can re-sync the Recurly code into my forked copy? I’d like to log these issues in GitHub as I uncover them, but I’ve already removed the offending stream. It will also be good if I ever need to update the connector from source.