Custom source connector dynamic response

Connector Development
1

Hi there,

I want to know how I can dynamically set the catalog schema cause I have a use case where my custom response can have fields changing (not static schema).

Examples (simple):
Responses:

{
   "language":"en",
   "Date of registration":"May. 02, 2022",
   "Last modified on":"May. 02, 2022",
   "ID":"15465456481",
   "Contact":{
      "Name":"ALbert",
      "Affiliation":"Center",
      "Address":"Texas 15487",
      "Telephone":"+51548798544",
      "E-mail":"a@a.com"
   },
   "Study":"Interventional",
   "Study Design":{
      "allocation":"single arm study",
      "masking":"open(masking not used)",
      "control":"uncontrolled control",
      "assignment":"single assignment",
      "purpose":"treatment purpose"
   },
   "Key inclusion & exclusion criteria":{
      "Inclusion Criteria":"1. Distinct number of brain metastases <= 4, One with surgical indication",
      "Age Maximum":"79age old under",
      "Gender":"Both"
   }
}

response

{
   "language":"en",
   "Date of registration":"May. 02, 2022",
   "Last modified on":"May. 02, 2022",
   "ID":"154658878979"
}

response

{
   "language":"en",
   "Date of registration":"May. 02, 2022",
   "Last modified on":"May. 02, 2022",
   "ID":"15465456481",
   "Contact":{
      "Name":"Cheil",
      "Affiliation":"Mall",
      "Address":"NY 658",
      "Telephone":"+515454844",
      "E-mail":"addd@a.com"
   },
   "Key inclusion & exclusion criteria":{
      "Inclusion Criteria":"1. Distinct number of brain metastases <= 4, One with surgical indication",
      "Exclusion Criteria":"Documented history of active other malignancy ",
      "Age Minimum":"20age old over",
      "Age Maximum":"79age old under",
      "Gender":"Both"
   }
}

Base on that, on the read I want to dynamically change the catalog to match the response and yield the Airbyte record. Does that makes sense?

2

Hi @theArg,
Dynamic schema change is not something supported by Airbyte at the moment.
Your connector must be able to know the schema of the records it will handle before running a replication.
You can implement your own discover method on your Source to infer a dynamic schema from a source, but this schema should not change from sync to sync.
You can checkout our Salesforce connector to check a custom implementation of discover .

3

Many thanks I will have a look!

5

Hey @theArg,
If you have a finite set of document types from which you can generate a static schema (in other words having the discover command on our connector always returning the same value with the same configuration) you should already be able to write a connector for your usecase.
If your schemas are evolving and you want to dynamically propagate these changes to your destination, this is indeed not yet supported by Airbyte but our team is actively working on a schema evolution feature.

6

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.