Airbyte connector state being set incorrectly?

Platform, Deploy & Infra Issues
,
1

version: 0.39.32-alpha

I’ve built a connector that’s got an incremental sync stream that just doesn’t seem to be working when run through airbyte.

I’ve tested it locally with python main.py read ... and I’ve tested the Docker image locally as well with docker run --rm ... local/source-myconnector:1 read ... and they both work fine.

I’ve also tested locally passing in the --state argument to both commands to simulate an existing state and it works just fine, but when I run it through the Airbyte UI it seems to fail. The error message seems to imply that the state isn’t being set as a dict but as a string, which causes the code to fail.

Any ideas on what’s actually happening here?

logs-26.txt (71.5 KB)

edit 1: I noticed that the connection state reported by Airbyte doesn’t look like the state that I usually use when developing

image
image1450×749 37.7 KB

Where my sample_state.json uses: {"events": {"timestamp": "2022-07-25T05:20:31.775025Z"}}

2

I’ve figured it out, and I’ll post my solution here for posterity - I’ve completely forgotten that self.state needs to be set to a dict and not merely a string. Once I’d fixed the part of self.get_updated_state() that sets it, I had to reset the data to clear the “invalid” state shown by Airbyte, and from there it was smooth sailing.

As a matter of QoL perhaps it’d be best to allow each state to be editable via the web interface in case it’s ever set incorrectly? I’d hate to have to remove an entire dataset in prod over a minor mistake like that.