- Is this your first time deploying Airbyte: No
- OS Version / Instance: AWS EC2
- Memory / Disk: 8Gb/30 GB
- Deployment: Docker
- Airbyte Version: 0.35.62-alpha
- Source name/version: MySQL (0.5.6)
- Destination name/version: Clickhouse (0.1.4)
- Source name/version : MySQL 8.0.28
- Destination name/version: Clickhouse 22.1.3.7
- Step: Setting new connection, source / On sync
- Description: I created a new MySQL source and then a fresh connection from scratch to ClickHouse. When I use the sync mode of
Incremental | Deduped + History
I faced an error that its logs attached. The tables are created but they are empty in the Clickhouse. The exact same steps for MySQL 5.7.33 work fine in the first place. You said you support MySQL 8.0 in the documentation. Right?
logs-5851.txt (171.1 KB)
logs-5859.txt (170.7 KB)
Hi @arashlayeghi,
I don’t find any relevant error in both of your logs. The table are empty because the source is not outputting any message:
recordsEmitted=0,bytesEmitted=0,stateMessagesEmitted=0,recordsCommitted=
Are you using CDC?
Without CDC the normal behavior is to have the user pick the cursor field, you should not have a sourceDefined cursor field as you do in your screenshot.
Hi @alafanechere,
Thanks.
Yes, I’m using CDC.
As you can see even if there is no logs, but it says attempt failed:
Anything I can do regarding the cursor field?
As an experiment, I changed the Replication method to STANDARD and reset the source and then reset the data and synced again. It seems there is no difference.
Any special configuration in MySQL 8.0 is different?
This is the log and screenshot of the error:
logs-5861.txt (170.8 KB)
This is the Replication of my other connection to MySQL v5.7.33 which does work successfully.
Hey @arashlayeghi,
For the experiment, could you please try to use a different destination or a different destination stream prefix for you Clickhouse destination?
Hey @alafanechere,
We don’t have any other clickhouse destination. However, I changed the destination stream prefix to something else test_, but the result is the same as before.
logs-5867.txt (171.0 KB)
Now, I changed the Transformation tab to Raw data (JSON) and this is the new log if it helps.
logs-5869.txt (164.7 KB)
Hi again @alafanechere ,
I created a Postgres destination to experiment if the issue is related to the destination or source.
This is the Postgres destination:
And this is the connection:
And this is the log:
The same happened to this:
logs-5871.txt (170.0 KB)
So this is definitely regarding MySQL source version 8.0.
We need this feature in MySQL 8.0.
Would you please prioritize this issue or find a workaround?
Is there any group parameter that should be set for MySQL v8.0?
Why version 5.7.33 works fine and does this encounters errors?
Thanks for your wonderful work.
Hello @arashlayeghi,
Could you please try to sync with full refresh instead of incremental?
According to your logs I think I can deduce the following:
- The connection to the source MySQL succeeds
- The connector does not read any data from your trips table:
Total records read: 0 (0 bytes)
Maybe it’s something related with the cursor field. If you are not using CDC you should be able to pick a cursor field in the replication configuration. Do you mind trying to recreate a MySQL source without CDC, with a full refresh to postgres?
I’ve feedback from users that are able to use our connector with MySQL 8, and the error in the log is not explicit enough to identify a root cause.
Hi @alafanechere,
When you said the Total records read: 0 (0 bytes) I suspect maybe I didn’t enough permissions for the CDC to the user I created in MySQL 8.0 instance. I ran this statement and the problem is now resolved. I didn’t think I forgot to give enough permissions for that instance.
GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'airbyte'@'%';
Thanks a lot for you guidance and help. We can close this ticket.
1 Like
Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.
