Custom source connector dynamic response

Hi there,

I want to know how I can dynamically set the catalog schema cause I have a use case where my custom response can have fields changing (not static schema).

Examples (simple):
Responses:

{
   "language":"en",
   "Date of registration":"May. 02, 2022",
   "Last modified on":"May. 02, 2022",
   "ID":"15465456481",
   "Contact":{
      "Name":"ALbert",
      "Affiliation":"Center",
      "Address":"Texas 15487",
      "Telephone":"+51548798544",
      "E-mail":"a@a.com"
   },
   "Study":"Interventional",
   "Study Design":{
      "allocation":"single arm study",
      "masking":"open(masking not used)",
      "control":"uncontrolled control",
      "assignment":"single assignment",
      "purpose":"treatment purpose"
   },
   "Key inclusion & exclusion criteria":{
      "Inclusion Criteria":"1. Distinct number of brain metastases <= 4, One with surgical indication",
      "Age Maximum":"79age old under",
      "Gender":"Both"
   }
}

response

{
   "language":"en",
   "Date of registration":"May. 02, 2022",
   "Last modified on":"May. 02, 2022",
   "ID":"154658878979"
}

response

{
   "language":"en",
   "Date of registration":"May. 02, 2022",
   "Last modified on":"May. 02, 2022",
   "ID":"15465456481",
   "Contact":{
      "Name":"Cheil",
      "Affiliation":"Mall",
      "Address":"NY 658",
      "Telephone":"+515454844",
      "E-mail":"addd@a.com"
   },
   "Key inclusion & exclusion criteria":{
      "Inclusion Criteria":"1. Distinct number of brain metastases <= 4, One with surgical indication",
      "Exclusion Criteria":"Documented history of active other malignancy ",
      "Age Minimum":"20age old over",
      "Age Maximum":"79age old under",
      "Gender":"Both"
   }
}

Base on that, on the read I want to dynamically change the catalog to match the response and yield the Airbyte record. Does that makes sense?

Hi @theArg,
Dynamic schema change is not something supported by Airbyte at the moment.
Your connector must be able to know the schema of the records it will handle before running a replication.
You can implement your own discover method on your Source to infer a dynamic schema from a source, but this schema should not change from sync to sync.
You can checkout our Salesforce connector to check a custom implementation of discover .

Many thanks I will have a look!

Hey @theArg,
If you have a finite set of document types from which you can generate a static schema (in other words having the discover command on our connector always returning the same value with the same configuration) you should already be able to write a connector for your usecase.
If your schemas are evolving and you want to dynamically propagate these changes to your destination, this is indeed not yet supported by Airbyte but our team is actively working on a schema evolution feature.