JSON schema validation failed (Redshift -> Snowflake)

Connector Questions & Issues
1

Is this your first time deploying Airbyte : No
Deployment : Docker
Airbyte Version : 0.35.3-alpha
Source name/version : Redshift
Destination name/version : Snowflake
Step : Schema validation
Description :
Hi all, I have successfully loaded data from Redshift to Snowflake already, but I am experiencing a JSON schema validation error when trying to load from another cluster. This does not seem to be a normalization issue, as it shows as completing successfully in the logs, which are attached. Thanks very much.

logs-161.txt (1.8 MB)

2

Hey @kev1021 can you try reloading the schema and try syncing again

4

Hi @harshith thanks for the response. I reloaded the schema and reset the data, but unfortunately, the issue is still ongoing.

5

Hey, can you share server and scheduler logs here?

6

Hi @harshith I am trying to download through the UI but nothing is happening when I click on the buttons. Is there a way that I can do so through the command line? Sorry if this is a silly question, but not seeing a way that other people have done so in the Slack chat, although a few people have reported the same issue with the logs files not downloading locally when the buttons are clicked.

7

@harshith looks the download button in the UI is not working can you create an issue in Github and tag the frontend team?

8

I met the same problem when I use snowflake as a source.

the spec uses OneOf for credentials, but somehow it forces me to use OAuth for auth_type when I pick username/password

errors: $.client_id: is missing but it is required, $.client_secret: is missing but it is required, $.auth_type: must be a constant value OAuth
9

Hi @yli can you check whether you have this line in your logs Invalid operation: permission denied for schema?

Our problem ended up being that the person who set up the source did not specify the schema / namespace, and we only had permissions to access one schema / namespace across the cluster, so we hit permissions issues that were resolved once we specified the schema. Even if you only have objects from the schema(s) that you have permissions for specified in your connection, it will still throw this error.

UPDATE: Just checked your logs and I don’t see that line but not specifying your source schema could possibly still be the issue here. Hopefully this helps.

10

@harshith please see my response to @yli above. In digging deeper into the logs, we realized that the problem was not actually the JSON schema validation failure, but that it was returning as such because there was an underlying permissions error that prevented the connection from accessing the object. Once we resolved by specifying the schema in the source, the connection successfully ran. This issue can be closed.

Perhaps a permission denied for schema issue could be flagged more explicitly as an error for these logs?

11

@kev1021 there’s no such log in the job. The connection reads the sample data from snowflake and loads to postgres (tpch_sf1.nation). I didn’t run dbt or other transformation. I checkecd the query history in snowflake, all queries are run successfully.

The dataset is very small, only 25 rows, 4 columns. And the error doesn’t block me to complete the sync. I just got the error of JSON schema validation failed, and the validation says the configuration on source-snowflake is wrong at credentials, though data is read correctly.

Edit: I put the logs in my original thread.

12

Thanks for confirming Kevin

13

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.