- Is this your first time deploying Airbyte?: No
- OS Version / Instance: Debian
- Memory / Disk: 8Gb / 30Gb
- Deployment: GCP Compute Engine n1-standard-2
- Airbyte Version: 0.39.13-alpha
- Step: Navigating the UI, looking at logs, adding a new source, updating a connection, etc.
- Description: The terminal where I have sshed into GCP becomes unresponsive, the UI becomes unresponsive, metrics stop being produced on GCP.
This usually happens after memory usage starts to increase and I have seen java ExitOnOutOfMemoryError processes.
Here is a screenshot of memory usage. It increases and then metrics stop a little after 9:25. I started using the UI around 9:10 where the memory first starts increasing.
I am currently investigating whether this was due to the connector I am developing, which dynamically pulls schemas. This was initially an intensive process and the schemas were quite large.
@wjwatkinson let me know if you still have this issue while disabling your new connector.
It might help if you could share the memory consumption per docker container (using docker stats ). Your instance is not too small in terms of memory to run Airbyte but upsizing it to 16GB might help for resource-demanding discovery jobs.
Thanks I will l try that if I run into the issue again! I have not had it since I stopped using starmap in my new connector. It makes sense that that could have eaten a lot of memory.
Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.
