Source File (sftp) - Only reading 43 of ~2000 records

Connector Questions & Issues
, ,
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Amazon Linux (Docker)
  • Memory / Disk:
  • Deployment: Docker
  • Airbyte Version: * 0.39.37-alpha
  • Source name/version: File 0.2.14
  • Destination name/version: Redshift 0.3.47
  • Step: Reading the records
  • Description: I just recently upgraded to 0.39.37 and i’m having a problem with one of my jobs. I have a csv file picked up from SFTP that has approximately 2000 rows in it. I have the sync mode set to Full refresh | Overwrite, but when it reads the file, it shows all the 2000 records in the logs, but then the logs say: Total records read: 43 (35 KB) I can see very clearly in the logs that there are over 2000 records read, but it only processes 43 of them. Can anyone help me understand why this might be? It’s been working really well for quite a while, so this is a new one for me.
2

Hey @krishagel, thanks for your post. Very sorry to hear that the connector broke after upgrading. At first glance, I’m not sure what the problem is either and need a bit more context. Firstly would it be impossible to post the logs? Secondly, what previous version of Airbyte were you upgrading from? Thanks in advance.

3

We were on 36.6 before.

I removed the 2024 lines of data out of the logs as I couldn’t share that data publicly.

In digging in, I found I have one other job that broke like this as well (normally ~12k records, only passing through ~1k). But I do have 15 or so jobs that are still running normally, some with over 1.2M records. The only difference I can find in the jobs are that the ones that are running fine don’t have quotes around the data in the CSV file, and these two that are failing do.

logs-8488.txt (31.2 KB)

4

Ok, just as a test, I reset the file source back to 2.10 and now it’s working again. It looks like it has something to do with the file source between 2.10 and 2.14

5

Hey @krishagel, happy to hear it’s working again although a bit disappointed to hear you had to downgrade.

Is the other broken job that you found also working again?

6

Yes, both jobs started fully functioning by rolling back.