Cannot load JSON files with S3 Source

alexandrecaze · June 23, 2022, 3:43pm

Hi there

Is this your first time deploying Airbyte?: Yes
OS Version / Instance: AWS EC2 instance
Memory / Disk: 30Gb
Deployment: Docker
Airbyte Version: 0.39.23-alpha
Source name/version: S3
Destination name/version: Redshift
Step: The issue is happening when creating the connection.
Description: I have JSON files in a S3 bucket that I want to sync to my Redshift cluster using Airbyte. I successfully created an S3 source in the Airbyte UI, although I was unable to select JSON as “File Format (The format of the files you’d like to replicate)”. I tried selecting CSV, parquet and Avro but (logically) my sync never happened.

Is is not possible yet to load JSON files from an S3 bucket ?

Thanks !
Alex

alexandrecaze · June 24, 2022, 7:25am

Closing this topic, there is an open issue on the subject here

github.com/airbytehq/airbyte

Support JSON format in S3 source

opened 02:17PM - 30 Jul 21 UTC

Phlair

type/enhancement area/connectors size/L lang/python connectors/sources-files connectors/source/s3 team/connectors-python

## Tell us about the problem you're trying to solve Currently S3 (via abstract …files source) doesn't support syncing JSON files. ## Describe the solution you’d like Build a JSONParser [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-s3/source_s3/source_files_abstract/fileformatparser.py) and add appropriate changes to [spec.py](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-s3/source_s3/source_files_abstract/spec.py), [unit tests](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-s3/unit_tests/test_fileformatparser.py) & documentation. If those links are broken we may have refactored where that code sits and I forgot to update this, comment @ me and I'll fix them! ## Additional context Adding this will support syncing JSON files in any new files storage source that builds from [abstract-files-source](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-s3/source_s3/source_files_abstract).