Oracle large source schema refresh issue

Connector Questions & Issues
, ,
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Ubuntu, Windows or MacOS… Windows 10
  • Memory / Disk: 32GB/256GB
  • Deployment: Docker
  • Airbyte Version: * 0.39.41-alpha
  • Source name/version: [airbyte/source-oracle]
  • Destination name/version: Snowflake
  • Step: The issue is happening during sync, creating the connection or a new source?
  1. Created the source using Oracle-source connector
  2. Added the snowflake as the destination
  3. Trying to create the new connection
  • Description:
    After created the connection, its trying to pull the tables from APPS schema from Oracle source. Its one of the large schema in Oracle database.
    But its almost taking more than 2 hours and failing with error say(Failed retry again) and same issue.
    Server Log file almost 8 MB unable to attach here.
2

Hi @tagnev, I need a few more details to help you debug this:

  1. What system are you doing this on?
  2. Do you see any specific errors in the logs?

Sometimes there are timeout errors with large amounts of data, but it’s hard to say at this stage.

3

Thanks for your prompt response.

  1. What system are you doing this on?
    Initially i tried my local windows 11. There also same issue.
    Now, I am using Ubuntu
    Hardware: 64 GB Ram & 256 GB Hard Disk
  2. Do you see any specific errors in the logs?
    Dont have specific errors in the logs.
4

The system looks good. I know you said the logs are too big to attach, but I need more information - could you look if there are any specific exit codes or exceptions being raised, and perhaps attach the second half of the logs?

5

will re run again and share the logs

6

Thank you, I’ll keep an eye on this thread!

7

airbyte_server_log.txt (64.5 KB)
Attached the today server log from file from UI. Badly need your support on this please.

8
  1. Could you update Airbyte to the latest version? We are up to 0.40.0 now
  2. What are the versions for both connectors?
  3. How many tables do you have? I think this may be an open issue you’re running into that’s still being worked on:
    https://github.com/airbytehq/airbyte/issues/4564
  4. Here’s a guide on how you might work around that:
    https://discuss.airbyte.io/t/how-to-sync-thousands-of-table-using-airbyte/1858
9

PFB the response inline,

  1. Could you update Airbyte to the latest version? We are up to 0.40.0 now
    We are in 0.39.1. Sure will upgrade.

  2. What are the versions for both connectors?

    image
    image1390×174 13.2 KB

  3. How many tables do you have? I think this may be an open issue you’re running into that’s still being worked on:
    May be around 500+ tables
    🐛 Airbyte Core: Large schema fetching failure · Issue #4564 · airbytehq/airbyte · GitHub

  4. Here’s a guide on how you might work around that:
    How to sync thousands of table using Airbyte
    Thanks. Will check it out.

10

Here are the docs on how to upgrade Airbyte:
https://docs.airbyte.com/operator-guides/upgrading-airbyte/

This will upgrade unused connectors and will handle data migration for you. Let me know if you have further questions and if you get the chance to look at that guide!