Issue with pyAirbyte and ConnectorStateManager in Airflow

Slack Knowledge Base Q&A API, Terraform and Other Topics
, , , ,
1

Summary

User is experiencing a ConnectorStateManager issue while running the source-google-sheets connector in Airflow using pyAirbyte. The problem arises due to version mismatches and limitations in downgrading the airbyte-cdk version within the virtual environment.


Question

Hi Team,

I’m having issues with pyairbyte in airflow. Trying to run the source-google-sheets but keep getting the connection state manager (same as this PR: https://github.com/airbytehq/airbyte/issues/46283). Due to the circumstances I can’t run a docker container in my airflow task because airflow is running form a docker compose in EC2. When trying to run the following script I get the ConnectorStateManager issue mentioned in the above PR. I have tried to force an older version which can work locally on my test python files but in production airflow does not as it appears unable to downgrade the airbyte-cdk version in the venv that it is running. I have also tried forcing the pip version field but it seems it does nothing to downgrade the connector (error also included below with mismatch on target)

  File "/Users/teddykosciuszek/Documents/CodeRepos/data-infra-airflow/dags/airbyte/test.py", line 3, in <module>
    sheet = get_source(
            ^^^^^^^^^^^
  File "/opt/anaconda3/envs/airbyte_py311/lib/python3.11/site-packages/airbyte/sources/util.py", line 107, in get_source
    executor=get_connector_executor(
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/airbyte_py311/lib/python3.11/site-packages/airbyte/_executors/util.py", line 261, in get_connector_executor
    executor.ensure_installation()
  File "/opt/anaconda3/envs/airbyte_py311/lib/python3.11/site-packages/airbyte/_executors/python.py", line 275, in ensure_installation
    raise exc.AirbyteConnectorInstallationError(
airbyte.exceptions.AirbyteConnectorInstallationError: Connector's reported version does not match the target version. (AirbyteConnectorInstallationError)
------------------------------------------------------------
AirbyteConnectorInstallationError: Connector's reported version does not match the target version.
    Connector Name: 'source-google-sheets'
    Venv Name: '.venv-source-google-sheets'
    Target Version: '0.7.0'
    Original Installed Version: None
    Version After Reinstall: '0.7.4'```
```from airbyte import get_source

sheet = get_source(
    "source-google-sheets",
    config={
        "spreadsheet_id": "",
        "credentials": {
            "auth_type": "Service",
            "service_account_info": {{ var.google_sheets_gcp_sa_key}}
        },
        
    },
    version= "0.7.0",
    # docker_image= "airbyte/source-google-sheets:0.7.0",
    install_if_missing=True,
    # pip_url= "git+<https://github.com/pixisai/airbyte.git@SOURCE_GOOGLE_SHEET_CDK_VERSION>",
    streams= "Sheet1",
)

read_result = sheet.read()

for record in read_result["Sheet1"]:
    print(record)```

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1730458498153289) if you want
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
['pyairbyte', 'airflow', 'source-google-sheets', 'connectorstatemanager', 'version-mismatch', 'airbyte-cdk']
</sub>