Manually defined cursor

A few versions ago (v0.33.xx), I used to update some .yaml config files by exporting my airbyte config through the UI, update some of the config files, then re-upload the whole folder to apply my fixes.

Now, after upgrading my airbyte version (v0.35.67) and from what I understood, the process has changed as we have to directly update the psql database. My question here are:

  • Is there already an existing guide explaining how to do so ?
  • It is my understanding that we have to update the cursor defined in the state table. Is it the only manipulation or do we have to update other tables ?

Today you need to update using the psql command. Please read about how to do it here: Update the cursor manually breaks my connection - #4 by boggdan

Great, thanks for the reply @marcosmarxm !

I still have a question regarding this subject: In many scenario, the reason I have to define my cursor manually is because my source has evolved → How can I update my connection manually ?

For instance, a new table is added in my source and I don’t want to reset my destination data or my connection. Then, what I used to do was taking my .yaml files, adding manually my new table with its cursor, then re-uploading the config to apply my changes and integrate my new table. And it was going the same way when I wanted to add new columns in my source tables.

I know that the schema change subject is in the Airbyte roadmap, but as of today, what would be the best approach to apply this kind of changes in the Airbyte database ? What table should I target ?

1 Like