Inquiring about the necessity of knowing the input stream before developing a custom Airbyte destination.
Should we know the input stream beforehand, before writing our own Airbyte DESTINATION? <@U04SE163WC8>
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Yes, it is important to understand the input stream before writing your own Airbyte destination. The input stream is a part of the Airbyte Protocol and it consists of AirbyteMessages. When creating a destination, the destination should read in the AirbyteMessages and write any that are of type AirbyteRecordMessage to the underlying data store. The destination should ignore fields or streams that are out of sync with the catalog and should always make its best effort to load what data is there that does match that catalog. Here is an extract from the Airbyte Protocol documentation that explains this in more detail: > “The destination should read in the AirbyteMessages and write any that are of type AirbyteRecordMessage to the underlying data store. The destination should ignore fields or streams that are out of sync with the catalog. The destination should always make its best effort to load what data is there that does match that catalog. e.g. if the User Stream has the fields first_name and last_name in the catalog, but the record has first_name and eye_color, the destination should persist first_name, even though last_name is missing. It should ignore eye_color as extraneous.” Therefore, understanding the input stream is crucial for correctly implementing a custom Airbyte destination.
<@U04SE163WC8> but how can we make our destination abstract so that any source can be worked with it?
A message from http://kapa.ai|kapa.ai
To make your destination abstract and compatible with any source, you can leverage the AbstractJdbcDestination superclass provided by Airbyte. This superclass makes it easy to create a destination for a database or data warehouse if it has a compatible JDBC driver. Here is an extract from the Airbyte documentation that explains this: > “The Postgres destination leverages the AbstractJdbcDestination superclass which makes it extremely easy to create a destination for a database or data warehouse if it has a compatible JDBC driver. If the destination you are implementing has a JDBC driver, be sure to check out AbstractJdbcDestination.” By using this superclass, you can ensure that your destination is abstract and can work with any source that is compatible with the JDBC driver. This allows you to focus on implementing the specific functionality of your destination, while the superclass handles the common functionality that is shared across all destinations. It’s also important to understand the Airbyte Catalog, which is a description of the data available in a source. It allows the source to declare what data it can send and allows the destination to choose what it wants to receive. For more information on the Airbyte Catalog, you can check out the Beginner’s Guide to the Airbyte Catalog.