Summary
The user is facing errors related to a ‘java.lang.NoSuchMethodError’ when running a MySQL to Iceberg(S3) process on Airbyte on AWS EC2. The error seems to be originating from the destination connector. The user has provided error logs and stack traces but has not been able to resolve the issue.
Question
Hello everyone
I’m comming here because i have a problem when mysql to iceberg(s3) process(airbyte on aws ec2).
source connection, destination connection was no problem.
when i started connection process, i could see following errors.
i read these, but no luck, i couldn’t solve. can you help me on this issue?
},
"processFromDest" : {
"elapsedTimeInNanos" : 10772635,
"executionCount" : 1,
"avgExecTimeInNanos" : 1.0772635E7
},
"writeToDest" : {
"elapsedTimeInNanos" : 0,
"executionCount" : 0,
"avgExecTimeInNanos" : "NaN"
},
"readFromDest" : {
"elapsedTimeInNanos" : 10638369635,
"executionCount" : 1873,
"avgExecTimeInNanos" : 5679855.651361452
}
}
}
2024-01-16 06:39:35 platform > failures: [ {
"failureOrigin" : "destination",
"failureType" : "system_error",
"internalMessage" : "java.lang.NoSuchMethodError: org.apache.logging.slf4j.Log4jLoggerFactory: method 'void <init>()' not found",
"externalMessage" : "Something went wrong in the connector. See the logs for more details.",
"metadata" : {
"attemptNumber" : 0,
"jobId" : 1,
"from_trace_message" : true,
"connector_command" : "write"
},
"stacktrace" : "java.lang.NoSuchMethodError: org.apache.logging.slf4j.Log4jLoggerFactory: method 'void <init>()' not found\n\tat org.slf4j.impl.StaticLoggerBinder.<init>(StaticLoggerBinder.java:53)\n\tat org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:41)\n\tat org.apache.spark.internal.Logging$.org$apache$spark$internal$Logging$$isLog4j2(Logging.scala:232)\n\tat org.apache.spark.internal.Logging.initializeLogging(Logging.scala:129)\n\tat org.apache.spark.internal.Logging.initializeLogIfNecessary(Logging.scala:115)\n\tat org.apache.spark.internal.Logging.initializeLogIfNecessary$(Logging.scala:109)\n\tat org.apache.spark.SparkContext.initializeLogIfNecessary(SparkContext.scala:84)\n\tat org.apache.spark.internal.Logging.initializeLogIfNecessary(Logging.scala:106)\n\tat org.apache.spark.internal.Logging.initializeLogIfNecessary$(Logging.scala:105)\n\tat org.apache.spark.SparkContext.initializeLogIfNecessary(SparkContext.scala:84)\n\tat org.apache.spark.internal.Logging.log(Logging.scala:53)\n\tat org.apache.spark.internal.Logging.log$(Logging.scala:51)\n\tat org.apache.spark.SparkContext.log(SparkContext.scala:84)\n\tat org.apache.spark.internal.Logging.logInfo(Logging.scala:61)\n\tat org.apache.spark.internal.Logging.logInfo$(Logging.scala:60)\n\tat org.apache.spark.SparkContext.logInfo(SparkContext.scala:84)\n\tat org.apache.spark.SparkContext.<init>(SparkContext.scala:195)\n\tat org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2714)\n\tat org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:953)\n\tat scala.Option.getOrElse(Option.scala:201)\n\tat org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:947)\n\tat io.airbyte.integrations.destination.iceberg.IcebergDestination.getConsumer(IcebergDestination.java:85)\n\tat io.airbyte.cdk.integrations.base.Destination.getSerializedMessageConsumer(Destination.java:54)\n\tat io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:186)\n\tat io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.java:125)\n\tat io.airbyte.integrations.destination.iceberg.IcebergDestination.main(IcebergDestination.java:42)\n",
"timestamp" : 1705387173918
}, {
"failureOrigin" : "destination",
"internalMessage" : "Destination process exited with non-zero exit code 1",
"externalMessage" : "Something went wrong within the destination connector",
"metadata" : {
"attemptNumber" : 0,
"jobId" : 1,
"connector_command" : "write"
},
"stacktrace" : "io.airbyte.workers.internal.exception.DestinationException: Destination process exited with non-zero exit code 1\n\tat io.airbyte.workers.general.BufferedReplicationWorker.readFromDestination(BufferedReplicationWorker.java:477)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsync$2(BufferedReplicationWorker.java:228)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n",
"timestamp" : 1705387174349
}, {
"failureOrigin" : "source",
"internalMessage" : "Source process read attempt failed",
"externalMessage" : "Something went wrong within the source connector",
"metadata" : {
"attemptNumber" : 0,
"jobId" : 1,
"connector_command" : "read"
},
"stacktrace" : "io.airbyte.workers.internal.exception.SourceException: Source process read attempt failed\n\tat io.airbyte.workers.general.BufferedReplicationWorker.readFromSource(BufferedReplicationWorker.java:376)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsyncWithHeartbeatCheck$3(BufferedReplicationWorker.java:235)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\nCaused by: java.lang.IllegalStateException: Source process is still alive, cannot retrieve exit value.\n\tat com.google.common.base.Preconditions.checkState(Preconditions.java:502)\n\tat io.airbyte.workers.internal.DefaultAirbyteSource.getExitValue(DefaultAirbyteSource.java:127)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.readFromSource(BufferedReplicationWorker.java:362)\n\t... 5 more\n",
"timestamp" : 1705387174824
}, {
"failureOrigin" : "replication",
"internalMessage" : "io.airbyte.workers.exception.WorkerException: Destination process exit with code 1. This warning is normal if the job was cancelled.",
"externalMessage" : "Something went wrong during replication",
"metadata" : {
"attemptNumber" : 0,
"jobId" : 1
},
"stacktrace" : "java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Destination process exit with code 1. This warning is normal if the job was cancelled.\n\tat io.airbyte.workers.general.BufferedReplicationWorker$CloseableWithTimeout.lambda$close$0(BufferedReplicationWorker.java:522)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsyncWithTimeout$5(BufferedReplicationWorker.java:256)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\nCaused by: io.airbyte.workers.exception.WorkerException: Destination process exit with code 1. This warning is normal if the job was cancelled.\n\tat io.airbyte.workers.internal.DefaultAirbyteDestination.close(DefaultAirbyteDestination.java:161)\n\tat io.airbyte.workers.general.BufferedReplicationWorker$CloseableWithTimeout.lambda$close$0(BufferedReplicationWorker.java:520)\n\t... 5 more\n",
"timestamp" : 1705387174826
} ]
2024-01-16 06:39:35 platform >
2024-01-16 06:39:35 platform > ----- END REPLICATION -----
2024-01-16 06:39:35 platform >
2024-01-16 06:39:35 platform > Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, totalCompleteFailureLimit=10, successivePartialFailureLimit=1000, totalPartialFailureLimit=10, successiveCompleteFailures=1, totalCompleteFailures=1, successivePartialFailures=0, totalPartialFailures=0)
Backoff before next attempt: 10 seconds```
<br>
---
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1705387599268449) if you want to access the original thread.
[Join the conversation on Slack](https://slack.airbyte.com)
<sub>
["mysql-to-iceberg-s3", "airbyte", "aws-ec2", "java-lang-nosuchmethoderror", "destination-connector", "error"]
</sub>