Connecting appstore and bigquery

regitze · August 22, 2022, 12:40pm

Is this your first time deploying Airbyte?: Yes
OS Version / Instance: MacOS
Memory / Disk: you can use something like 16Gb
Deployment: docker
Airbyte Version: 3.7
Source name/version: appstore
Destination name/version: bigquery
Step: The issue is happening creating the connection
Description: I’m a trying to build a connection between appstore and bigquery. However, when I press sync schemas I don’t get anything back from airbyte, and thus can’t choose which tables I want to sync.

The app store connect key user that I have created have role admin, hence I am assuming that I have the right permissions.

I tried following this

github.com

miroapp/tap-appstore/blob/2c448137f6dead04148653ad721d5f23186bb0ee/tap_appstore/init.py#L122


      
              return schemas
          
          

          
def discover(api: Api):
              raw_schemas = load_schemas()
              streams = []
              for schema_name, schema in raw_schemas.items():
                  report_date = datetime.strptime(get_bookmark(schema_name), "%Y-%m-%dT%H:%M:%SZ").strftime("%Y-%m-%d")
                  filters = get_api_request_fields(report_date, schema_name)
          
          
        report = _attempt_download_report(api, filters)
                  if report:
                      # create and add catalog entry
                      catalog_entry = {
                          'stream': schema_name,
                          'tap_stream_id': schema_name,
                          'schema': schema,
                          'key_properties': []
                      }
                      streams.append(catalog_entry)

And I was able to download a report but I only got back empty data

{'streams': [{'stream': 'subscription_report',
   'tap_stream_id': 'subscription_report',
   'schema': {'type': ['null', 'object'],
    'properties': {'_line_id': {'type': ['null', 'integer']},
     '_time_extracted': {'type': ['null', 'string'], 'format': 'date-time'},
     '_api_report_date': {'type': ['null', 'string'], 'format': 'date'},
     'app_name': {'type': ['null', 'string']},
     'app_apple_id': {'type': ['null', 'string']},
     'subscription_name': {'type': ['null', 'string']},
     'subscription_apple_id': {'type': ['null', 'string']},
     'subscription_group_id': {'type': ['null', 'string']},
     'standard_subscription_duration': {'type': ['null', 'string']},
     'promotional_offer_name': {'type': ['null', 'string']},
     'promotional_offer_id': {'type': ['null', 'string']},
     'customer_price': {'type': ['null', 'string']},
     'customer_currency': {'type': ['null', 'string']},
     'developer_proceeds': {'type': ['null', 'string']},
     'proceeds_currency': {'type': ['null', 'string']},
     'preserved_pricing': {'type': ['null', 'string']},
     'proceeds_reason': {'type': ['null', 'string']},
     'client': {'type': ['null', 'string']},
     'device': {'type': ['null', 'string']},
     'state': {'type': ['null', 'string']},
     'country': {'type': ['null', 'string']},
     'active_standard_price_subscriptions': {'type': ['null', 'integer']},
     'active_free_trial_introductory_offer_subscriptions': {'type': ['null',
       'integer']},
     'active_pay_up_front_introductory_offer_subscriptions': {'type': ['null',
       'integer']},
     'active_pay_as_you_go_introductory_offer_subscriptions': {'type': ['null',
       'integer']},
     'free_trial_promotional_offer_subscriptions': {'type': ['null',
       'integer']},
     'pay_up_front_promotional_offer_subscriptions': {'type': ['null',
       'integer']},
     'pay_as_you_go_promotional_offer_subscriptions': {'type': ['null',
       'integer']},
     'marketing_opt_ins': {'type': ['null', 'integer']},
     'billing_retry': {'type': ['null', 'integer']},
     'grace_period': {'type': ['null', 'integer']}}},
   'key_properties': []},
  {'stream': 'subscriber_report',
   'tap_stream_id': 'subscriber_report',
   'schema': {'type': ['null', 'object'],
    'properties': {'_line_id': {'type': ['null', 'integer']},
     '_time_extracted': {'type': ['null', 'string'], 'format': 'date-time'},
     '_api_report_date': {'type': ['null', 'string'], 'format': 'date'},
     'event_date': {'type': ['null', 'string'], 'format': 'date'},
     'app_name': {'type': ['null', 'string']},
     'app_apple_id': {'type': ['null', 'string']},
     'subscription_name': {'type': ['null', 'string']},
     'subscription_apple_id': {'type': ['null', 'string']},
     'subscription_group_id': {'type': ['null', 'string']},
     'standard_subscription_duration': {'type': ['null', 'string']},
     'promotional_offer_name': {'type': ['null', 'string']},
     'promotional_offer_id': {'type': ['null', 'string']},
     'subscription_offer_type': {'type': ['null', 'string']},
     'subscription_offer_duration': {'type': ['null', 'string']},
     'marketing_opt_in_duration': {'type': ['null', 'string']},
     'customer_price': {'type': ['null', 'string']},
     'customer_currency': {'type': ['null', 'string']},
     'developer_proceeds': {'type': ['null', 'string']},
     'proceeds_currency': {'type': ['null', 'string']},
     'preserved_pricing': {'type': ['null', 'string']},
     'proceeds_reason': {'type': ['null', 'string']},
     'client': {'type': ['null', 'string']},
     'device': {'type': ['null', 'string']},
     'country': {'type': ['null', 'string']},
     'subscriber_id': {'type': ['null', 'string']},
     'subscriber_id_reset': {'type': ['null', 'string']},
     'refund': {'type': ['null', 'string']},
     'purchase_date': {'type': ['null', 'string'], 'format': 'date'},
     'units': {'type': ['null', 'integer']}}},
   'key_properties': []},
  {'stream': 'sales_report',
   'tap_stream_id': 'sales_report',
   'schema': {'type': ['null', 'object'],
    'properties': {'_line_id': {'type': ['null', 'integer']},
     '_time_extracted': {'type': ['null', 'string'], 'format': 'date-time'},
     '_api_report_date': {'type': ['null', 'string'], 'format': 'date'},
     'provider': {'type': ['null', 'string']},
     'provider_country': {'type': ['null', 'string']},
     'sku': {'type': ['null', 'string']},
     'developer': {'type': ['null', 'string']},
     'title': {'type': ['null', 'string']},
     'version': {'type': ['null', 'string']},
     'product_type_identifier': {'type': ['null', 'string']},
     'units': {'type': ['null', 'integer']},
     'developer_proceeds': {'type': ['null', 'string']},
     'begin_date': {'type': ['null', 'string'], 'format': 'date'},
     'end_date': {'type': ['null', 'string'], 'format': 'date'},
     'customer_currency': {'type': ['null', 'string']},
     'country_code': {'type': ['null', 'string']},
     'currency_of_proceeds': {'type': ['null', 'string']},
     'apple_identifier': {'type': ['null', 'string']},
     'customer_price': {'type': ['null', 'string']},
     'promo_code': {'type': ['null', 'string']},
     'parent_identifier': {'type': ['null', 'string']},
     'subscription': {'type': ['null', 'string']},
     'period': {'type': ['null', 'string']},
     'category': {'type': ['null', 'string']},
     'cmb': {'type': ['null', 'string']},
     'device': {'type': ['null', 'string']},
     'supported_platforms': {'type': ['null', 'string']},
     'proceeds_reason': {'type': ['null', 'string']},
     'preserved_pricing': {'type': ['null', 'string']},
     'client': {'type': ['null', 'string']},
     'order_type': {'type': ['null', 'string']}}},
   'key_properties': []},
  {'stream': 'subscription_event_report',
   'tap_stream_id': 'subscription_event_report',
   'schema': {'type': ['null', 'object'],
    'properties': {'_line_id': {'type': ['null', 'integer']},
     '_time_extracted': {'type': ['null', 'string'], 'format': 'date-time'},
     '_api_report_date': {'type': ['null', 'string'], 'format': 'date'},
     'event_date': {'type': ['null', 'string'], 'format': 'date'},
     'event': {'type': ['null', 'string']},
     'app_name': {'type': ['null', 'string']},
     'app_apple_id': {'type': ['null', 'string']},
     'subscription_name': {'type': ['null', 'string']},
     'subscription_apple_id': {'type': ['null', 'string']},
     'subscription_group_id': {'type': ['null', 'string']},
     'standard_subscription_duration': {'type': ['null', 'string']},
     'promotional_offer_name': {'type': ['null', 'string']},
     'promotional_offer_id': {'type': ['null', 'string']},
     'subscription_offer_type': {'type': ['null', 'string']},
     'subscription_offer_duration': {'type': ['null', 'string']},
     'marketing_opt_in': {'type': ['null', 'string']},
     'marketing_opt_in_duration': {'type': ['null', 'string']},
     'preserved_pricing': {'type': ['null', 'string']},
     'proceeds_reason': {'type': ['null', 'string']},
     'consecutive_paid_periods': {'type': ['null', 'integer']},
     'original_start_date': {'type': ['null', 'string'], 'format': 'date'},
     'client': {'type': ['null', 'string']},
     'device': {'type': ['null', 'string']},
     'state': {'type': ['null', 'string']},
     'country': {'type': ['null', 'string']},
     'previous_subscription_name': {'type': ['null', 'string']},
     'previous_subscription_apple_id': {'type': ['null', 'string']},
     'days_before_canceling': {'type': ['null', 'string']},
     'cancellation_reason': {'type': ['null', 'string']},
     'days_canceled': {'type': ['null', 'integer']},
     'quantity': {'type': ['null', 'integer']}}},
   'key_properties': []}]}

I have several apps in the app store so I’m wondering if that could be the reason.

regitze · August 23, 2022, 8:43am

turns out that I had set a start date too early before there was any data and that was why it failed. I think it would be nice to have an error message for this

harshith · August 23, 2022, 5:43pm

Got it. Feel free to create an issue in github and yeah also to create an PR is you find some time. Thanks a lot for the feedback.

Topic		Replies	Views
Worday-BigQuery Sync succeeded with errors Connector Questions & Issues connectors	7	501	August 11, 2022
Unable to update connection syncCatalog using the Airbyte Configuration API Connector Questions & Issues data-loading	6	881	July 14, 2022
Mongodb connector schema discovery issue Connector Questions & Issues connectors	2	385	November 14, 2022
Having problem setting up BigQuery Destinaton via Airbyte UI Connector Questions & Issues data-loading , connectors	3	1311	February 13, 2023
Missing data when connecting from Shopify to Big Query Connector Questions & Issues destination-bigquery	4	481	October 3, 2022

Connecting appstore and bigquery

Related topics