Octavia Apply Disable automatic sync

Connector Questions & Issues
1
  • Is this your first time deploying Airbyte?: No /
  • OS Version / Instance: Ubuntu
  • Memory / Disk: 4GB
  • Deployment: Kubernetes
  • Airbyte Version: 0.40.6
  • Source name/version: Postgres
  • Destination name/version: 1.0.7
  • Step: During octavia apply
  • Description:

Sometimes when running octavia apply if there is a change the connection is automatically triggered and starts a sync.

Is it possible to disable this behavior? I would like to control when a sync is triggered

Thanks for your answer

2

Hello there! You are receiving this message because none of your fellow community members has stepped in to respond to your topic post. (If you are a community member and you are reading this response, feel free to jump in if you have the answer!) As a result, the Community Assistance Team has been made aware of this topic and will be investigating and responding as quickly as possible.
Some important considerations that will help your to get your issue solved faster:

  • It is best to use our topic creation template; if you haven’t yet, we recommend posting a followup with the requested information. With that information the team will be able to more quickly search for similar issues with connectors and the platform and troubleshoot more quickly your specific question or problem.
  • Make sure to upload the complete log file; a common investigation roadblock is that sometimes the error for the issue happens well before the problem is surfaced to the user, and so having the tail of the log is less useful than having the whole log to scan through.
  • Be as descriptive and specific as possible; when investigating it is extremely valuable to know what steps were taken to encounter the issue, what version of connector / platform / Java / Python / docker / k8s was used, etc. The more context supplied, the quicker the investigation can start on your topic and the faster we can drive towards an answer.
  • We in the Community Assistance Team are glad you’ve made yourself part of our community, and we’ll do our best to answer your questions and resolve the problems as quickly as possible. Expect to hear from a specific team member as soon as possible.

Thank you for your time and attention.
Best,
The Community Assistance Team

3

There are a couple ways to do this with Octavia. You can set the schedule with cron where it will wait to run until the time you configured, or you can set it to Manual where it will only run upon being manually triggered. These config options can be found in the connector’s generated configuration.yaml before you run the final octavia apply to add/update the connector in Airbyte.

For example, connectors/<connection_name>/configuration.yaml includes something like the below. By default it will schedule the sync to run once every hour:

schedule_type: basic # OPTIONAL | string | Allowed values: basic, cron, manual
schedule_data: # OPTIONAL | object
  basic_schedule:
    time_unit: hours # REQUIRED | string | Allowed values: minutes, hours, days, weeks, months
    units: 1 # REQUIRED | integer
# cron:
#   cron_time_zone: "UTC" # REQUIRED | string
#   cron_expression: "* */2 * * * ?" # REQUIRED | string

If you want manual syncs only, change schedule_type to manual and then remove or comment out the schedule_data block:

schedule_type: manual # OPTIONAL | string | Allowed values: basic, cron, manual
#schedule_data: # OPTIONAL | object
#  basic_schedule:
#    time_unit: hours # REQUIRED | string | Allowed values: minutes, hours, days, weeks, months
#    units: 1 # REQUIRED | integer
#  cron:
#    cron_time_zone: "UTC" # REQUIRED | string
#    cron_expression: "* */2 * * * ?" # REQUIRED | string

If you want to schedule your syncs with cron, change schedule_type to cron and add your cron expression. It doesn’t matter if you leave basic_schedule in the config since the schedule_type will override it. The below expression runs the sync every Tuesday at 16:14 Eastern Time:

schedule_type: cron # OPTIONAL | string | Allowed values: basic, cron, manual
schedule_data: # OPTIONAL | object
  basic_schedule:
    time_unit: hours # REQUIRED | string | Allowed values: minutes, hours, days, weeks, months
    units: 1 # REQUIRED | integer
  cron:
    cron_time_zone: "America/New_York" # REQUIRED | string
    cron_expression: "0 14 16 ? * TUE *" # REQUIRED | string

Once this is configured, you can finally run octavia apply to add or update the connector in Airbyte without it syncing immediately.

Although this isn’t documented in the readme, Octavia communicates with the Airbyte public API so you can find more info on the settings here: https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/connections/update