What exactly is modules.yml / module-settings.yml for?

I noticed that Airbyte seems to be generating these two files every time I run python main.py read .... and it caught me by surprise because it contains the full Authorization header that I’m using. It shouldn’t be too much of an issue as its an Oauth token that will expire in 1hr but it’s been committed to the repository already, and if this were any other type of token it’d be an issue.

What are these files even for and is there any way to turn this feature off?

edit: now that I’ve done some digging, this MAY be the result of the Zoho API that I’m working with, but I’m going to keep this up just to make sure anyways

I’m pretty sure I"ve got it, and in case anyone else wonders - its Airbyte’s Cache mechanism! Under the hood its using vcr.py to write http requests to .yml files that bear the name of the class that’s running (by default).

I simply removed use_cache = True from the stream in question and the files no longer appear.

Perhaps those files ought to be excluded via .gitignore by default (or maybe they already are and we just haven’t caught up?)

1 Like

Hi @cornjuliox,
Yes, you are right! Airbyte’s CDK leverages the VCR library to cache requests if use_cache = True. They can be gitignored. I guess they are not at the moment because connectors usually run in a docker container. Hence, by definition, these files are ephemeral.

I’ve been developing locally and run the connector by way of python main.py ..... Is building the Docker image first the preferred way to do it?

For quick iteration, it’s indeed more convenient to run python directly from your environment. But the final artifact for a connector is its docker image, so you’ll eventually have to build and run the connector locally to make sure it’s working.