Summary
The user is facing a java.lang.NoSuchMethodError related to org.apache.logging.slf4j.Log4jLoggerFactory when trying to integrate destination-iceberg with Airbyte on self-hosted setup.
Question
Could really use some help getting data into destination-iceberg on self-hosted. I’ve tried every possible configuration, and still getting the same error.
Versions:
Airbyte: 0.50.53, destination-iceberg: 0.1.5
Error:
2024-02-26 14:54:53 destination > at org.slf4j.impl.StaticLoggerBinder.<init>(StaticLoggerBinder.java:53) ~[log4j-slf4j-impl-2.17.2.jar:2.17.2]
2024-02-26 14:54:53 destination > at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:41) ~[log4j-slf4j-impl-2.17.2.jar:2.17.2]
2024-02-26 14:54:53 destination > at org.apache.spark.internal.Logging$.org$apache$spark$internal$Logging$$isLog4j2(Logging.scala:232) ~[spark-core_2.13-3.3.2.jar:3.3.2]
2024-02-26 14:54:53 destination > at org.apache.spark.internal.Logging.initializeLogging(Logging.scala:129) ~[spark-core_2.13-3.3.2.jar:3.3.2]
2024-02-26 14:54:53 destination > at org.apache.spark.internal.Logging.initializeLogIfNecessary(Logging.scala:115) ~[spark-core_2.13-3.3.2.jar:3.3.2]
2024-02-26 14:54:53 destination > at org.apache.spark.internal.Logging.initializeLogIfNecessary$(Logging.scala:109) ~[spark-core_2.13-3.3.2.jar:3.3.2]
2024-02-26 14:54:53 destination > at org.apache.spark.SparkContext.initializeLogIfNecessary(SparkContext.scala:84) ~[spark-core_2.13-3.3.2.jar:3.3.2]
2024-02-26 14:54:53 destination > at org.apache.spark.internal.Logging.initializeLogIfNecessary(Logging.scala:106) ~[spark-core_2.13-3.3.2.jar:3.3.2]
2024-02-26 14:54:53 destination > at org.apache.spark.internal.Logging.initializeLogIfNecessary$(Logging.scala:105) ~[spark-core_2.13-3.3.2.jar:3.3.2]
2024-02-26 14:54:53 destination > at org.apache.spark.SparkContext.initializeLogIfNecessary(SparkContext.scala:84) ~[spark-core_2.13-3.3.2.jar:3.3.2]
2024-02-26 14:54:53 destination > at org.apache.spark.internal.Logging.log(Logging.scala:53) ~[spark-core_2.13-3.3.2.jar:3.3.2]
2024-02-26 14:54:53 destination > at org.apache.spark.internal.Logging.log$(Logging.scala:51) ~[spark-core_2.13-3.3.2.jar:3.3.2]
2024-02-26 14:54:53 destination > at org.apache.spark.SparkContext.log(SparkContext.scala:84) ~[spark-core_2.13-3.3.2.jar:3.3.2]
2024-02-26 14:54:53 destination > at org.apache.spark.internal.Logging.logInfo(Logging.scala:61) ~[spark-core_2.13-3.3.2.jar:3.3.2]
2024-02-26 14:54:53 destination > at org.apache.spark.internal.Logging.logInfo$(Logging.scala:60) ~[spark-core_2.13-3.3.2.jar:3.3.2]
2024-02-26 14:54:53 destination > at org.apache.spark.SparkContext.logInfo(SparkContext.scala:84) ~[spark-core_2.13-3.3.2.jar:3.3.2]
2024-02-26 14:54:53 destination > at org.apache.spark.SparkContext.<init>(SparkContext.scala:195) ~[spark-core_2.13-3.3.2.jar:3.3.2]
2024-02-26 14:54:53 destination > at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2714) ~[spark-core_2.13-3.3.2.jar:3.3.2]
2024-02-26 14:54:53 destination > at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:953) ~[spark-sql_2.13-3.3.2.jar:3.3.2]
2024-02-26 14:54:53 destination > at scala.Option.getOrElse(Option.scala:201) ~[scala-library-2.13.10.jar:?]
2024-02-26 14:54:53 destination > at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:947) ~[spark-sql_2.13-3.3.2.jar:3.3.2]
2024-02-26 14:54:53 destination > at io.airbyte.integrations.destination.iceberg.IcebergDestination.getConsumer(IcebergDestination.java:85) ~[io.airbyte.airbyte-integrations.connectors-destination-iceberg-0.50.41.jar:?]
2024-02-26 14:54:53 destination > at io.airbyte.cdk.integrations.base.Destination.getSerializedMessageConsumer(Destination.java:54) ~[airbyte-cdk-core-0.2.0.jar:?]
2024-02-26 14:54:53 destination > at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:186) ~[airbyte-cdk-core-0.2.0.jar:?]
2024-02-26 14:54:53 destination > at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.java:125) ~[airbyte-cdk-core-0.2.0.jar:?]
2024-02-26 14:54:53 destination > at io.airbyte.integrations.destination.iceberg.IcebergDestination.main(IcebergDestination.java:42) ~[io.airbyte.airbyte-integrations.connectors-destination-iceberg-0.50.41.jar:?]```
<br>
---
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1708959584039959) if you want to access the original thread.
[Join the conversation on Slack](https://slack.airbyte.com)
<sub>
["destination-iceberg", "self-hosted", "java.lang.NoSuchMethodError", "org.apache.logging.slf4j.Log4jLoggerFactory", "error"]
</sub>