Docker-free commands to sync a source?

[alafanechere]

Hi @arbiter,
Unfortunately there’s no single run down process in Python that can be done for this use case. Airbyte’s protocol leverage docker containerization and is not language specific, this is why we also have java source connectors.
The standard way of obtaining a catalog is running the discover command on a source connector: e.g.

docker run airbyte/my-source:dev discover --config path_to_secret.json

@arbiter

If I know that, then I can peek into the stage where a catalog.json is created and hack together a script that directly produces this catalog and simply produces the normalization models without having to depend on any source credentials.

As you can see from the command I shared above, the discover command requires secrets, hence access to data. This is because some connector have dynamically discovered schema (and not hard coded JSON schemas).

TLDR; the standard way of getting a source catalog is running the discover command on a source connector. To get it you need access to the source because some connectors dynamically generate schema. This is why normalization models are generated at sync time and not before the sync run.