Missing data tables when connecting Postgres to S3

Connector Questions & Issues
, , ,
1
  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: Ubuntu 18.04.4 LTS
  • Memory / Disk: 1 Tb
  • Deployment: Docker
  • Airbyte Version: v0.37.1
  • Source name/version: Postgres
  • Destination name/version: aws S3
  • Step: after sync
  • Description: First of all, I know that both postgres and s3 connectors are not the latest ones, but I’ve had this issue the first time I tried this connexion back when the connectors were at their most recent version at the time. I am trying to configure Airbyte to ingest data from a Postgres database to an S3. The connexions all pass, the only issue I have is Airbyte doesn’t seem to see all the tables. I thought it might be an issue of user or schema that do not have access to all tables, but using a Query tool to connect to the same database with the same user credentials and on the same schema, I can see all the tables that I’m suppose to.
    I thought it can be a connector bug; also I heard it can be an issue of table authors (even if airbyte has full admin access to the tables, it doesn’t see some tables that its user is not considered the author of), I am really unsure what the issue is. Anybody has some kind of insight as to what the problem might be / how can I solve it?
2

Please try with latest version of Postgres OR update your issue saying what version are you using. Without the connector version is impossible to help. Also how many tables aren’t displayed?

3

I’m on Airbyte Cloud, Postgres connector version 0.4.30, connecting to Google Cloud SQL running PostgreSQL 11.
My database has 39 tables, but Airbyte only see 34 tables even after I refresh the source schema.

4

I found my answer. It seems that the user for the Source is missing Postgres privileges to SELECT the table.

1 Like
5

I will once I get Airbyte running again. However, again, when I use another Query tool, I can see all the tables with the same user and same schema so it’s not a user or schema permissions.

I can only see 8 tables out of about 40. Will update the post!

6

Ernest I’d recommend to you query postgres and try to find what parameter the tables you can select has different from the tables you can’t select in Airbyte. Airbyte runs a query using JDBC lib and try to select all tables the user has permission to.

7

Hello, the issue was solved after increasing the size of the VM in which my docker was running.