Need advice on memory profiling a failing Airbyte Connector

logs-4105.txt (138.5 KB)

  • Is this your first time deploying Airbyte: No / Yes
  • OS Version / Instance: Ubuntu 18.04, Mac OS, Windows, GCP , EC2 micro.a4
  • Memory / Disk: 16Gb / 1Tb SSD
  • Deployment: Docker / Kubernetes
  • Airbyte Version: 0.35.15-alpha
  • Source name/version: File 0.24
  • Destination name/version: Amazon S3
  • Step: Problem occurs on-sync.

I’ve got a connector here that failed to sync a few times and the logs report that the Java VM is out of memory. The VM that Airbyte is running on has 8 GB of RAM and that’s been enough for us since we started.

I’d like to rule out the possibility of poor coding on my part by figuring out exactly how much RAM the connector is using while syncing.

I’ve never profiled anything Python before, is there a “preferred” set of tools / methods that should be used when profiling Airbyte connectors?

@cornjuliox could you try to configure Dockprom to profile your Airbyte server? GitHub - stefanprodan/dockprom: Docker hosts and containers monitoring with Prometheus, Grafana, cAdvisor, NodeExporter and AlertManager

@marcosmarxm yes we can. it might take some time because ops is a little pre-occupied but i’ll get back to you.

Its worth noting that the last couple of syncs with this connector have succeeded without error so this may have been an anomaly.