- Is this your first time deploying Airbyte?: No
- OS Version / Instance: Ubuntu
- Deployment: k8s
- Airbyte Version: 0.35.0-alpha
- Destination name/version: destination-elasticsearch 0.1.2
- Step: creating the destination
- Description:
I’m not able to create an ElasticSearch destination with AWS OpenSearch. I’m simply passing the endpoint as https://<HOST>:443 and testing the destination, but the check throws failed to ping elasticsearch. The failed logs are empty so they’re not much help. I am able to reach the host from another pod running on the same k8s cluster, so I can rule out networking issues. The OpenSearch domain is running OpenSearch 1.2.
The documentation is inconsistent with the implementation, in that it says the connector takes a port parameter which defaults to 9002, when in reality there is no port parameter. Am I right in adding the port to the endpoint? Has anyone managed to connect Airbyte to an AWS OpenSearch domain?
PS I tried, but couldn’t create a destination-elasticsearch tag.
You must pass the port in the endpoint parameter. I opened a ticket to correct docs: https://github.com/airbytehq/airbyte/pull/13665
Can you share the logs Oliver? Are you trying use the https endpoint?
Thanks @marcosmarxm. So the endpoint I’m passing (https://<host>:<port>) is correct.
The logs in the UI are essentially empty, but I just realized there’s another log file in S3 for the same job and this one contains a bit more info:
2022-06-10 06:11:38 e[32mINFOe[m i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-06-10 06:11:38 e[32mINFOe[m i.a.i.d.e.ElasticsearchConnection(<init>):54 - creating ElasticsearchConnection: https://<REDACTED>:443
2022-06-10 06:11:38 e[32mINFOe[m i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-06-10 06:11:38 e[32mINFOe[m i.a.i.d.e.ElasticsearchConnection(checkConnection):109 - checking elasticsearch connection
2022-06-10 06:11:38 e[32mINFOe[m i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-06-10 06:11:38 e[1;31mERRORe[m i.a.i.d.e.ElasticsearchConnection(checkConnection):118 - unknown exception while pinging elasticsearch server
2022-06-10 06:11:38 e[32mINFOe[m i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - java.lang.NullPointerException: status
2022-06-10 06:11:38 e[32mINFOe[m i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - at java.util.Objects.requireNonNull(Objects.java:233) ~[?:?]
2022-06-10 06:11:38 e[32mINFOe[m i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - at co.elastic.clients.base.ElasticsearchError.<init>(ElasticsearchError.java:44) ~[elasticsearch-java-7.15.0.jar:?]
2022-06-10 06:11:38 e[32mINFOe[m i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - at co.elastic.clients.base.ElasticsearchError$Builder.build(ElasticsearchError.java:120) ~[elasticsearch-java-7.15.0.jar:?]
2022-06-10 06:11:38 e[32mINFOe[m i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - at co.elastic.clients.json.BuildFunctionDeserializer.deserialize(BuildFunctionDeserializer.java:43) ~[elasticsearch-java-7.15.0.jar:?]
2022-06-10 06:11:38 e[32mINFOe[m i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - at co.elastic.clients.json.JsonpDeserializer$LazyDeserializer.deserialize(JsonpDeserializer.java:205) ~[elasticsearch-java-7.15.0.jar:?]
2022-06-10 06:11:38 e[32mINFOe[m i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:102) ~[elasticsearch-java-7.15.0.jar:?]
2022-06-10 06:11:38 e[32mINFOe[m i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - at co.elastic.clients.base.RestClientTransport.getHighLevelResponse(RestClientTransport.java:210) ~[elasticsearch-java-7.15.0.jar:?]
2022-06-10 06:11:38 e[32mINFOe[m i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - at co.elastic.clients.base.RestClientTransport.performRequest(RestClientTransport.java:93) ~[elasticsearch-java-7.15.0.jar:?]
2022-06-10 06:11:38 e[32mINFOe[m i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - at co.elastic.clients.elasticsearch.ElasticsearchClient.info(ElasticsearchClient.java:866) ~[elasticsearch-java-7.15.0.jar:?]
2022-06-10 06:11:38 e[32mINFOe[m i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - at io.airbyte.integrations.destination.elasticsearch.ElasticsearchConnection.checkConnection(ElasticsearchConnection.java:111) [io.airbyte.airbyte-integrations.connectors-destination-elasticsearch-0.36.1-alpha.jar:?]
2022-06-10 06:11:38 e[32mINFOe[m i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - at io.airbyte.integrations.destination.elasticsearch.ElasticsearchDestination.check(ElasticsearchDestination.java:47) [io.airbyte.airbyte-integrations.connectors-destination-elasticsearch-0.36.1-alpha.jar:?]
2022-06-10 06:11:38 e[32mINFOe[m i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - at io.airbyte.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:138) [io.airbyte.airbyte-integrations.bases-base-java-0.36.1-alpha.jar:?]
2022-06-10 06:11:38 e[32mINFOe[m i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:105) [io.airbyte.airbyte-integrations.bases-base-java-0.36.1-alpha.jar:?]
2022-06-10 06:11:38 e[32mINFOe[m i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - at io.airbyte.integrations.destination.elasticsearch.ElasticsearchDestination.main(ElasticsearchDestination.java:30) [io.airbyte.airbyte-integrations.connectors-destination-elasticsearch-0.36.1-alpha.jar:?]
Which also isn’t super explicit, but at least shows that the correct endpoint is being used. It could be an authentication issue, I’ll look into it.
@marcosmarxm me again - I’m reading through AWS OpenSearch docs and they list recommended versions of ES clients to use to avoid compatibility issues. The version used by the connector is higher than the recommended version, so it could also be that the client itself is breaking the connection because it’s an OpenSearch cluster.
Before I jump into this rabbit hole, do you know of anyone who successfully connected Airbyte to an AWS OpenSearch domain? I don’t see myself downgrading the client or switching to an OpenSearch client since it’s written in Java : /
I didn’t see any other users reaching out because of OpenSearch. Oliver do you mind creating an issue in Github so team in the future can implement the connector for OpenSearch?
Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.