Thank you so much! I'm thrilled to hear you enjoyed the content. Your positive feedback motivates me to keep creating and sharing. Stay tuned for more, and if you have any topics or suggestions you'd like to see covered, please let me know. Cheers! 😊
awwsome brooo , I want to ask a question: after I complete the project, do I still need to pay for Azure Synapse and Azure Data Lake Storage Gen2 services, and how much is the fee during this project?
Cloud fees are on a pay as you go basis, I cannot say how much it will cost you. In my case, I was using the free credit upon sign up and it cost less than $5.
Thank you for the video. When I run requests module from Python it works, but in Airflow this tasks runs forever and my Python process also takes up 95% of CPU. Any idea what could be the problem?
I've having a hard time figuring out the docker part. Where can I find the link for the docker-compose file. That will help me figure out as to what's exactly going on.
Here is how I wrote my get_wikipedia_data def. The html changed. def get_wikipedia_data(html): from bs4 import BeautifulSoup soup = BeautifulSoup(html, 'html.parser') table = soup.find_all("table", {"class": "wikitable"})[1] table_rows = table.find_all('tr') return table_rows Another thing: As stated by the instructor, we can get an error when building the transform pipeline. Here is what I used instead of Nominatim. You need to use import geopy. def get_lat_long(country, city): location = geocoder.arcgis(f'{city}, {country}') if location.ok: return location.latlng[0], location.latlng[1] return None
I have ignored the lat long functionality in my code as its blocking me and continued to read the converted json string in write function but i keep on getting this error " the JSON object must be str, bytes or bytearray, not NoneType" . This is how my JSON in XCom looks like {"rank":{"0":1,"1":2,"2":3,"3":4},"stadium":{"0":"Rungrado 1st of May Stadium","1":"Michigan Stadium","2":"Ohio Stadium","3":"Melbourne Cricket Ground","4":"Camp Nou"}} Ps: As a workaround I have written dataframe to csv directly , but kindly help me why i am unable to json.loads from Xcom in write method
This error usually means that you're not able to fetch data from the xcom. You should check the stack trace in the error reported to get the line throwing this error. You should also try double checking the line where you're fetching from xcom? Below is how you can pull from XCom: data = kwargs['ti'].xcom_pull(key='XCOM_KEY_HERE', task_ids='TASK_ID_HERE') data = json.loads(data)
I had the same issue. It turned out the the task_ids in the wikipedia_pipeline.py file didn't match the task_id in the wikipedia_flow.py file. Once these were updated to match, everything went through. @@harizizzudin
Thats cool man, your projects are amazing, i just have some confusion about the airflow you are using, is it a docker based airflow or native installed airflow on your machine with pip. I have a windows machine and i cant install it directly using pip install
Hi @aminemaasri2622, Thank you so much for the kind words about my projects; I really appreciate it! I'm using a Mac so I am able to use both native and docker based airflow. Unfortunately, Airflow is not supported for Windows systems, to use it on Windows, it is recommended you use Windows Linux Subsystem (WLS) with a technology like Docker. You can check airflow documentation for further information on this. Please let me know if you have further questions. Cheers, Yusuf.
Hi @CodeWithYu thanks a lot for an excellent content it's very helpful. I have an small issue with scheduler server, when all servers are running, the scheduler server stopped automatically , in airflow web server, I can see this message: "The scheduler does not appear to be running. The DAGs list may not update, and new tasks will not be scheduled." the logs for scheduler server show this: ERROR: Could not open requirements file: [Errno 2] No such file or directory: './requirements.txt' [notice] A new release of pip is available: 23.1.2 -> 24.0 [notice] To update, run: python -m pip install --upgrade pip Defaulting to user installation because normal site-packages is not writeable I check docker compose file and remove requirements.txt in all places and see the configuration for scheduler and I observe this: command: bash -c "pip install --no-cache-dir -r ./requirements.txt && airflow db upgrade && airflow scheduler" I try quit ./requirements.txt but doesn't work, any suggestion? any idea about it? thanks!
