Using Airbyte to get data from websites with scraping

Q&A
1

I am new to Airbyte, our team is looking to use airbyte for different sources - ranging from http api (web scraped website) to websites containing datasets like kaggle etc. we are looking to create custom connectors for these sources. I am looking for some guide on how to get started with this resources.

I have implemented the custom connector for a sample api using below guide. Step 1: Creating the Source | Airbyte Documentation

I need to look at other ways of

  1. getting data from a website (scrapped into my destination) using custom connector.
  2. getting data from kaggle or equivalent data source using custom connector. please let me know how to achieve above tasks.
2

Hi @AditModi,
Our connector development kit is mainly targeted for building connectors for API sources.
You can however build a custom source connector that adheres to the Airbyte protocol interface.. So you are completely free to implement your scraping logic with the tool of your choice (e.g beautifulsoup in Python) as long as your application exposes the expected interface for source connectors.

For Kaggle specifically: they do have a semi public API so you might be able to use our CDK to create a source connector with the guide you shared: