OS Version / Instance: AWS EC2 instance type t3.medium
Memory / Disk: 8Gb/30 GB
Deployment: Docker
Airbyte Version: 0.36.1-alpha
Source name/version: MySQL (0.5.11)
Destination name/version: Clickhouse (0.1.5)
Source name/version: AWS Aurora MySQL Serverless v2 (Compatible with MySQL 8.0.23)
Destination name/version: Clickhouse 22.1.3.7
Step: The issue is happening during sync and source and connectors are created successfully.
Description: I created a source for our recently migrated AWS RDs MySQL to AWS Aurora MySQL Serverless v2 with the Replication method of CDC and it connected successfully. Then I created a connection to our ClickHouse destination as you can see in the screenshot below and then hit sync. I faced the error logs attached. Do you know what’s the cause?
It seems the main error is this:
2022-05-11 14:13:22 source > 2022-05-11 14:13:22 INFO i.d.c.m.SnapshotReader(execute):448 - snapshot continuing with database(s): [crm_trips]
2022-05-11 14:13:22 source > 2022-05-11 14:13:22 INFO i.d.c.m.SnapshotReader(readBinlogPosition):906 - Step 5: read binlog position of MySQL primary server
2022-05-11 14:13:22 source > 2022-05-11 14:13:22 INFO i.d.c.m.SnapshotReader(lambda$readBinlogPosition$18):918 - using binlog ‘mysql-bin-changelog.000005’ at position ‘1272169’ and gtid ‘’
2022-05-11 14:13:22 source > 2022-05-11 14:13:22 INFO i.d.c.m.SnapshotReader(execute):494 - Step 6: generating DROP and CREATE statements to reflect current database schemas:
2022-05-11 14:13:24 source > 2022-05-11 14:13:24 INFO i.d.c.m.SnapshotReader(execute):754 - Step 7: rolling back transaction after abort
2022-05-11 14:13:24 source > 2022-05-11 14:13:24 ERROR i.d.c.m.AbstractReader(failed):219 - Failed due to error: Aborting snapshot due to error when last running ‘SHOW CREATE TABLE mysql.host’: Can’t find file: ‘host’ (errno: 2 - No such file or directory)
2022-05-11 14:13:24 source > org.apache.kafka.connect.errors.ConnectException: Can’t find file: ‘host’ (errno: 2 - No such file or directory) Error code: 1017; SQLSTATE: HY000.
Hello Arash, you’re using Incremental+Dedup+history and the stream doesn’t have the cursor field defined, see log error:
2022-05-11 14:19:31 e[44msourcee[0m > Exception in thread "main" java.lang.IllegalStateException: No cursor field specified for stream attempting to do incremental.
If you standard method to sync instead of CDC it works?
logs-5889.txt (348.5 KB) @marcosmarxm Please note that the main log file for ncremental+Dedup+history is attached to this post and you mistakenly checked the other log file I attached for the Standard method instead of CDC. Please check this one.
I didn’t find a proper solution at the moment Arash. Would be possible to you share how did you configure your CDC with Aurora Mysql so our team can reproduce with enviroment and apply same logic with our tests?
As a workaround you can sync data without CDC at the moment.
@marcosmarxm I created a custom Parameter Group based on default.aurora-mysql8.0 parameter group and changed the value of log_bin_trust_function_creators to 1.
Also, I created a DB cluster parameter group from default.aurora-mysql8.0 and overridden these values:
Then modified the cluster and instance to use these DB Parameter Group and DB cluster Parameter Group and then reboot the instance to use these parameter groups.
Hi @marcosmarxm was the issue resolved? I’m trying to research if using Airbyte for Aurora Serverless v2 CDC is supported. However, my use case is Postgres, not MySQL.
Sorry Arash, I didn’t have time to investigate it. The Airbyte connectors is compatible with MySQL database, using with Aurora can’t be 100% compatible and you must debug any further issues.
Also, I wanted to ask if you’ve been successful in syncing with the “Full Refresh | Overwrite” option. I’m encountering problems with that method as well.
