Hello!
New to Airbyte. Testing it out as a solution to gathering inventory data across online clothing retailers.
I am following the tutorial found here for automating data scraping: https://airbyte.com/tutorials/data-scraping-with-airflow-and-beautiful-soup
I had no issues up until adding in the custom DAG to airflow, specifically Step 4. I was able to successfully set up the airbyte_linkedin_connection connection. I then went to add the python code to the dags/airbyte_airflow_dag.py file under the airflow directory. Firstly, the provided code doesn’t actually compile after replacing the variables so it required a little python formatting. Once I got the script working, I saved the file and restarted the webserver and scheduler as instructed. However, when I went to my DAGs on Airflow, I was unable to locate the one I set up. I can’t for the life of me find out why my webserver isn’t picking up on the new DAG. Any help would be greatly appreciated!
If you create an example dag like this one https://airflow.apache.org/docs/apache-airflow/stable/tutorial.html it shows in Airflow? The code is public in git so become easier to understand maybe what is missing.
Hi! Following that tutorial led me to my problem. It was an issue of location of the dag folder.
Basically, when using airflow, there was a discrepancy between my airflow.cfg file and my airflow CLI tool.
I was placing dags in the folder specified by the dags_folder config in the airflow.cfg file, but I needed to be placing the dags in the folder specified by the dags_folder config shown when running airflow config list.
Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.