Creating Source Stream without HTTP API

Summary

User is inquiring about creating a source stream without using an HTTP API.


Question

Can I create a source stream that doesn’t use an HTTP API?



This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["create-source-stream", "without-http-api", "connector"]

Yes, you can create a source stream that doesn’t use an HTTP API. While the Airbyte CDK provides the HTTPStream object to simplify the process of creating source streams for HTTP APIs, developers are welcome to implement their own objects if they are working with non-HTTP APIs. For more details on streams and sources, you can refer to the basic concepts section in the Airbyte documentation.

<@U04SE163WC8> Is there example code for a connector that does not use an HTTP API?

Yes, there are examples of connectors that do not use an HTTP API. These are simple Python connectors using the barebones Source abstraction. Here are a couple of examples: - Google Sheets - Mailchimp You can find these examples in the Airbyte documentation.

<@U04SE163WC8> How do I volume mount a directory into a docker image for an airbyte source?

A message from http://kapa.ai|kapa.ai

To volume mount a directory into a Docker image for an Airbyte source, you can use the -v or --volume option in your Docker run command. This option allows you to map a host directory to a directory in the Docker container. Here’s an example of how you can do this: bash docker run -it --rm --volume /host/directory:/container/directory image_name In this command, /host/directory is the directory on your host machine that you want to mount, and /container/directory is the path where you want to mount the directory in the Docker container. For Airbyte specifically, you might want to mount the airbyte_local directory, which is where Airbyte writes data for local file-based destinations. Here’s an example: bash docker run -it --rm -v /tmp/airbyte_local:/local image_name In this command, /tmp/airbyte_local is the airbyte_local directory on your host machine, and /local is the directory in the Docker container where Airbyte writes data for local file-based destinations. You can find more information about this in the Airbyte documentation. If you’re working with a locally stored file on Windows OS, you’ll need to open the .env file in your local Airbyte root folder and change the values for LOCAL_ROOT, LOCAL_DOCKER_MOUNT, and HACK_LOCAL_ROOT_PARENT to an existing absolute path on your machine. Colons in the path need to be replaced with a double forward slash, //. LOCAL_ROOT & LOCAL_DOCKER_MOUNT should be set to the same value, and HACK_LOCAL_ROOT_PARENT should be set to their parent directory. You can find more information about this in the Airbyte documentation.