Using private CA certificate to validate elasticsearch endpoint

Connector Questions & Issues
1
  • Is this your first time deploying Airbyte: Yes
  • OS Version / Instance: EC2
  • Deployment: Docker
  • Step: Setting new destination
  • Description: We have a plain-jane install of the current release. when I configure an elasticsearch destination, it fails and the logs show that it’s because the elasticsearch certificate can’t be validated. we use a a private CA cert on our elasticsearch clusters. how do I install the CA cert into airbyte or tell airbyte not to validate the connection?

thanks,
Chris

2

Hello @cbeck today it’s not possible to connect using a Private CA Certificate. I opened the issue Destination ElasticSearch: allow usage of Private CA Certificate · Issue #11356 · airbytehq/airbyte · GitHub requesting this feature. Feel free to follow the issue to receive updates of the implementation.

3

You’ve probably already come up with your own solution, but just in case, I had this problem once Elasticsearch starting securing their connections by default (v8). I came up with a workaround by making a new “version” of the destination-elasticsearch connector by writing a Dockerfile that builds from the existing version, but adds the extra CA into the correct place for the JVM:

FROM airbyte/destination-elasticsearch:0.1.3
COPY ./http_ca.crt /usr/local/openjdk-17/lib/security/
COPY ./gd_bundle-g2.crt /usr/local/openjdk-17/lib/security/

RUN openssl x509 -outform der -in /usr/local/openjdk-17/lib/security/http_ca.crt -out /usr/local/openjdk-17/lib/security/es_ca.der
RUN keytool -import -alias es -keystore /usr/local/openjdk-17/lib/security/cacerts -file /usr/local/openjdk-17/lib/security/es_ca.der -noprompt -keypass changeit -storepass changeit
RUN keytool -import -alias godaddy -keystore /usr/local/openjdk-17/lib/security/cacerts -file /usr/local/openjdk-17/lib/security/gd_bundle-g2.crt -noprompt -keypass changeit -storep
ass changeit

Then I make the image:

docker build . -t airbyte/destination-elasticsearch:0.1.3-ca

Then from within the Airbyte UI, I configured the destination-elasticsearch connector to use the new version of “0.1.3-ca”.

Try your sync again and you should be good.

Hope that makes sense. Let me know if you have any other questions about it!