Summary
The user is looking to move an API ingestion process from a Python script to Airbyte for the Power BI REST API. They are inquiring about creating a custom connector in Airbyte, managing connections for access tokens, handling parameters like activityDate, and managing continuation token URLs. They also seek advice on whether to perform these tasks in a no code, low code, or language-specific manner.
Question
<@U04SE163WC8>
Hi dears
My manager ask me to move an one API ingestion from a python script to Airbyte (a connection , a stream)
May I show you the script logic :
- Get an access token from a first API (Thanks to an Azure AD Service Principal)
client_id = “AAA”
client_secret = “LLLL”
authority_url = “https://login.microsoftonline.com/kbubalo.com”
scope = [“https://analysis.windows.net/powerbi/api/.default”]
2.#Set Power BI REST API to get Activities for yesterdays
#Get yesterdays date and convert to string
activityDate = date.today() - timedelta(days=1)
activityDate = activityDate.strftime(“%Y-%m-%d”)
url = “https://api.powerbi.com/v1.0/myorg/admin/activityevents?startDateTime='” + activityDate + “T00:00:00’&endDateTime='” + activityDate + “T23:59:59’”
-
Get first call JSON results and manage a continuation token
result = api_call.json()[‘activityEventEntities’]
df1 = pd.DataFrame(result)
pd.concat([df, df1])
#Call Continuation URL as long as results get one back to get all activities through the day
while contUrl is not None:
api_call_cont = requests.get(url=contUrl, headers=header)
contUrl = api_call_cont.json()['continuationUri']
Et voila
Sorry , I 'm very new on Airbyte , “globally” do I need to perform theses tasks on Airbyte , please ?
Create a custom connector for https://api.powerbi.com/v1.0/myorg/admin/activityevents ?
Who can manage
==> connection : get an access token regarding AAD service principal + secret
==> One parameter activityDate we can get 1) via caching file LOOP (a list of dates)
or a formula/expression like : activityDate = date.today() - timedelta(days=1)
activityDate = activityDate.strftime(“%Y-%m-%d”)
Can an Airbyte connector accept kind of formula for a parameter (kind of expression to grab the context : like current date …)
==> Manage the continuation token URL
Any advise / experience / feedback to guide me please ? To do on No code or low code or language specific ?
Tanks a lot for any help
My best
Christophe
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["api-ingestion", "python-script", "airbyte", "power-bi-rest-api", "custom-connector", "access-token", "activitydate", "continuation-token", "no-code", "low-code", "language-specific"]
Sure, you can definitely use Airbyte to ingest data from an API. Airbyte provides a step-by-step guide on how to create an Airbyte source in Python to read data from an HTTP API. This guide uses the Exchange Rates API as an example, but the principles should apply to your API as well. You can find the guide here. If you want to orchestrate data ingestion and transformation pipelines, you can use Dagster with Airbyte. This allows you to ingest and transform data from multiple sources and transform it using SQL and Python-based transformations. You can find a tutorial on how to do this here. To set up Airbyte, you can run it to configure data replication jobs from applications to databases. You can install Airbyte with Docker Compose and access the UI locally. Here is the code to do that: bash $ git clone <https://github.com/airbytehq/airbyte.git> $ cd airbyte $ docker-compose up You can find more details on setting up Airbyte here. Remember, if you need to configure Airbyte connections dynamically depending on external factors such as changing API inputs, you can use the Dagster feature to configure Airbyte with Python code. You can find a tutorial on how to do this here. I hope this helps! Let me know if you have any other questions.
Hello Christophe, Based on the information provided in the extracts, it seems like you can indeed create a custom connector in Airbyte to handle your use case. Here’s a general outline of how you can approach this: 1. Create a custom connector: You can create your own connectors in Airbyte. The tutorial on building a connector to extract data from the Webflow API provides a detailed guide on how to create a custom Airbyte source connector written in Python. This should be generalizable to a large number of API data sources, including the Power BI API you’re working with. 2. Handle authentication: The tutorial covers topics including authentication, which you’ll need for getting an access token from the Azure AD Service Principal. In your Python script, you’re using client_id and client_secret for authentication. In Airbyte, you can handle this using the TokenAuthenticator or BasicHttpAuthenticator depending on your API’s requirements. A forum post in the extracts suggests that switching from TokenAuthenticator to BasicHttpAuthenticator resolved a user’s issue. 3. Manage parameters: In your script, you’re using a date parameter (activityDate). In Airbyte, you can handle this using dynamic stream generation. The tutorial also covers this topic. 4. Manage continuation token: Your script uses a continuation token to get all activities through the day. This seems like a good use-case for full refresh with stream slices in Airbyte. A support user in the forum post suggested using stream slices for a similar scenario. 5. Add the connector to the API/UI: Once you’ve built your connector, you can add it to the Airbyte UI. The documentation on building a Python source provides information on how to do this. As for your question about using a formula for a parameter, the extracts do not provide specific information on this. However, since Airbyte connectors can be written in any language, it’s likely that you can implement such functionality in your custom connector. Remember, when submitting an Airbyte connector, you need to ensure that the connector passes the CAT suite, the metadata.yaml file is filled out and valid, and you have created appropriate documentation for the connector. I hope this helps! If you have further questions, don’t hesitate to ask. Best, http://kapa.ai|kapa.ai