Adding debug info to Airbyte extracts

Summary

Looking to include jobid, pod/environment details in Airbyte extracts to identify issues and redo specific extractions. Seeking information on which variable contains this detail.


Question

Hi All… am looking to add some debug info to our airbyte extracts. Looking to include jobid, pod/environment details etc which pulled the record from source. This will help us to identify if specific job id had any issue & cleanup specific jobid or pod & redo specific extraction again. As per documentation, {{ config }} doesn’t have this detail. Do you know which variable has it?

<#C027KKE4BCZ|help-connector-development> <@U04SE163WC8>



This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["debug-info", "airbyte-extracts", "jobid", "pod-details", "variable"]

Based on the information provided in the extracts, it seems that Airbyte does not directly provide a way to include job id or pod/environment details in the extracted data. However, you can use environment variables to pass certain information to your Airbyte jobs. For example, in Kubernetes deployment, you can use job-specific variables like SPEC_JOB_KUBE_NODE_SELECTORS, CHECK_JOB_KUBE_NODE_SELECTORS, DISCOVER_JOB_KUBE_NODE_SELECTORS, etc. to define one or more pod node selectors for the respective jobs. Similarly, you can use SPEC_JOB_KUBE_ANNOTATIONS, CHECK_JOB_KUBE_ANNOTATIONS, DISCOVER_JOB_KUBE_ANNOTATIONS, etc. to define one or more pod annotations for the respective jobs. These variables can be set in the Kubernetes configuration files (source). For debugging purposes, you can pass environment variables through the worker container which launches other containers. This can be done by setting the DEBUG_CONTAINER_IMAGE environment variable to the name of the container you’re targeting (source). However, these methods do not directly include the job id or pod/environment details in the extracted data. They are more for controlling and debugging the job execution environment. For more specific needs, you might need to modify the source code or create a custom connector. If you need further assistance, I recommend reaching out to the Airbyte community on their forum or GitHub for more detailed guidance.