Using the Airbyte UI crashes Airbyte

  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Debian
  • Memory / Disk: 8Gb / 30Gb
  • Deployment: GCP Compute Engine n1-standard-2
  • Airbyte Version: 0.39.13-alpha
  • Step: Navigating the UI, looking at logs, adding a new source, updating a connection, etc.
  • Description: The terminal where I have sshed into GCP becomes unresponsive, the UI becomes unresponsive, metrics stop being produced on GCP.

This usually happens after memory usage starts to increase and I have seen java ExitOnOutOfMemoryError processes.
Here is a screenshot of memory usage. It increases and then metrics stop a little after 9:25. I started using the UI around 9:10 where the memory first starts increasing.

I am currently investigating whether this was due to the connector I am developing, which dynamically pulls schemas. This was initially an intensive process and the schemas were quite large.

@wjwatkinson let me know if you still have this issue while disabling your new connector.
It might help if you could share the memory consumption per docker container (using docker stats ). Your instance is not too small in terms of memory to run Airbyte but upsizing it to 16GB might help for resource-demanding discovery jobs.

Thanks I will l try that if I run into the issue again! I have not had it since I stopped using starmap in my new connector. It makes sense that that could have eaten a lot of memory.