Error Reading Streams from PostgreSQL

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

User encounters an error while attempting to read streams from PostgreSQL using Airbyte in a Docker environment.


Question

Hi, I am trying to read data from PostgreSQL but face an error when trying to read streams via read.

Docker Engine: 27.3.1
PyAirByte: 0.19.0
Python: 3.11
Miniconda virtual env```
Thanks

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1730813816279189) if you want
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
['postgresql', 'read-streams', 'docker-desktop', 'pyairbyte', 'python']
</sub>
2 
/home/tahir/miniconda3/envs/airbyte/lib/python3.11/site-packages/snowflake/sqlalchemy/base.py:1068: SAWarning: The GenericFunction 'flatten' is already registered and is going to be overridden.
  functions.register_function("flatten", flatten)
&gt;&gt;&gt; source = ab.get_source(
...     "source-postgres",
...     config={
...         "host": "host.docker.internal",
...         "port": 5432,
...         "schema_name": "public",
...         "username": "dwh_user",
...         "password": "dwh_pass",
...         "database": "dwh_db"
...     }
... )
Writing PyAirbyte logs to file: /tmp/airbyte/logs/2024-11-05/airbyte-log-JBY8FM6EC.log
Writing `source-postgres` logs to file: /tmp/airbyte/logs/source-postgres/source-postgres-log-JBY8FM6ET.log
&gt;&gt;&gt; source.check()
Connection check succeeded for `source-postgres`.
&gt;&gt;&gt; source.get_available_streams()
['sample_table', '_airbyte_destination_state', '_airbyte_state', '_airbyte_streams']
&gt;&gt;&gt; for record in source.get_records('sample_table'):
...     print(record)
... 
{'name': 'a', '_airbyte_raw_id': '0672a1ef-179d-76b6-8000-9daedd4d31a4', '_airbyte_extracted_at': datetime.datetime(2024, 11, 5, 13, 34, 41, 383000, tzinfo=&lt;UTC&gt;), '_airbyte_meta': {}}
&gt;&gt;&gt; source.select_streams(['sample_table'])
&gt;&gt;&gt; source.read()
ERROR: Something went wrong in the connector. See the logs for more details.
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/home/tahir/miniconda3/envs/airbyte/lib/python3.11/site-packages/airbyte/sources/base.py", line 660, in read
    result = self._read_to_cache(
             ^^^^^^^^^^^^^^^^^^^^
  File "/home/tahir/miniconda3/envs/airbyte/lib/python3.11/site-packages/airbyte/sources/base.py", line 744, in _read_to_cache
    cache._write_airbyte_message_stream(  # noqa: SLF001  # Non-public API
  File "/home/tahir/miniconda3/envs/airbyte/lib/python3.11/site-packages/airbyte/caches/base.py", line 306, in _write_airbyte_message_stream
    cache_processor.process_airbyte_messages(
  File "/home/tahir/miniconda3/envs/airbyte/lib/python3.11/site-packages/airbyte/shared/sql_processor.py", line 268, in process_airbyte_messages
    for message in messages:
  File "/home/tahir/miniconda3/envs/airbyte/lib/python3.11/site-packages/airbyte/sources/base.py", line 573, in _read_with_catalog
    yield from progress_tracker.tally_records_read(message_generator)
  File "/home/tahir/miniconda3/envs/airbyte/lib/python3.11/site-packages/airbyte/progress.py", line 261, in tally_records_read
    for count, message in enumerate(messages, start=1):
  File "/home/tahir/miniconda3/envs/airbyte/lib/python3.11/site-packages/airbyte/_connector_base.py", line 440, in _execute
    raise exc.AirbyteConnectorFailedError(
airbyte.exceptions.AirbyteConnectorFailedError: Connector failed. (AirbyteConnectorFailedError)
------------------------------------------------------------
AirbyteConnectorFailedError: Connector failed.
    Please review the log file for more information.
    Connector Name: 'source-postgres'
    Exit Code: 1
    Log file: /tmp/airbyte/logs/source-postgres/source-postgres-log-JBY8FM6ET.log```