Pg source: fixing schema fetching

Bringing the discussion that I started on slack a couple days ago here, since I believe that this is a better channel to discuss the needed changes in order to try to fix the issue at hand (be it through my PR or other needed changes):

I just submitted a PR for a fix, I’d like to know if this is all accord to the guidelines, and also (in case it gets merged) if there’s a release window for this to go into production.

Another thing that I’m interested is the fetch size for JDBC sources, as mentioned in issue #11715 (sorry, I’m limited to just 2 links per post) and #4314. I understand that trying to manually increase the hard coded value is not a proper solution, like it was proposed here, however, maybe a simple solution would be to pass that value through an env var with a default of 1000 , this should be easy and fast to implement (I’m just not sure if it’d require changes in all connectors, or only in the base abstraction).
The proper solution would be the one mentioned in the first issue, but IMO it’d take quite a while since it’s a more complex approach, what do you people think?

Hey @igormp, we really appreciate your contribution, and our team will review it soon.

Maybe a simple solution would be to pass that value through an env var with a default of 1000

We’re not supporting passing custom environment variables to specific connectors. Connector’s container are dynamically started by the airbyte_worker and we are not offering a feature to set specific env var to the containers.

Our team will work on [EPIC] Avoid OOM errors in JDBC source connectors · Issue #11715 · airbytehq/airbyte · GitHub pretty soon, it’ll make the fetchSize dynamic and hopefully, it won’t be something users will need to care about.

For this kind of question feel also free to comment directly on the GitHub issue.

Thanks! I’m open to any suggestions :slight_smile:

Understandable.

I see that tuliren will handle that issue. Is there anything that I could help, be it with code or some tests? It is in my interest to have this feature asap, and I’d love to help however I can (as long as it’s according to the team’s guidelines/planning, of course).

@tuliren is indeed working on this epic. I’d suggest you to comment on this issue to share what you’re up to work on and sync with our team. I’m sure they will appreciate your wish to help!

1 Like