Failed to fetch streams from S3 Connector

  • OS Version / Instance: Ubuntu

  • Deployment: Kubernetes ?

  • Airbyte Version: 0.35.31-alpha

  • Source name/version: S3/0.1.10

  • Destination name/version: Redshift/0.3.28

  • Step: Discover Schema while making Connection

  • Description:
    We are using S3 as a source connector, source gets configured successfully but it’s failing on fetching the streams (discover schema). While debugging the issue, found the following ERROR in
    worker logs (Block Size is set to Default Value - 10000):

    2022-05-19 11:52:39 ERROR i.a.w.p.a.DefaultAirbyteStreamFactory(internalLog):95 - Detected mismatched datatype on column 'id', in file 'myfile.csv'. Should be 'integer', but found 'string'.

But if we set block size to 100 or 1000, it works perfectly fine.