***
My FREE guide on how to become a data engineer:
karolinasowinska.notion.site/...
***
I currently work as a data engineer and I'd like to tell you what data engineering is all about! It's commonly confused with machine learning, data science etc., when in fact data engineering is quite a distinct area of technology.
So, what do big data engineers do?
The job revolves around bringing new data to the company (ETL process if you like) and then maintaining the data flow!
Is data engineering a good career choice in 2020? Long story short - yes. As the amount of data we produce is growing rapidly, so is the demand for data professionals. However, my prediction is that many of the current data engineering tasks might get standardised, and eventually automated in the future. But hey, this is the XXI century! We'll have to learn new skills and adapt all the time!
-------
If you'd like to learn data engineering, I recommend following the 4 simple steps below to land you the first job interview:
In this long-awaited Airflow for Beginners video I'm showing you how to install Airflow from scratch, and how to schedule your first ETL job in Airflow!
We will run the Spotify ETL job that we created in my Data Engineering Course for Beginners. But if you have your own code to schedule in Airflow - feel free to use it instead!
In this Airflow tutorial I just wanted to get you started with Airflow, because I know that configuring Airflow can be non-trivial. I won't be going in-depth into all Airflow's features, so hopefully this tutorial will be as beginners friendly as it can get!
I'll show you how to fix some installation errors, how to set up your project folders, how to edit Airflow config file and where it sits, and how to write your first DAG file!
Get ready for a ride!
Oh, but grab some useful links first:
-github code repo: github.com/karolina-sowinska/...
-Airflow quick start guide: airflow.apache.org/docs/stabl...
-Spotify access token: developer.spotify.com/console...
Don't forget to subscribe if you're new here, more content is coming!
:)
---------
If you'd like to learn data engineering, I recommend following the 4 simple steps below to land you the first job interview:
1. Learn Python
I recommend following the Python for Everybody specialization course on Coursera, which is one of the most popular courses there:
imp.i384100.net/x9gVO3
2. Learn SQL
SQL is still the lingua franca of data. I recommend going with Learn SQL Basics for Data Science course, because it contains some chapters which are very releavant to data engineering in partcular, e.g. distributed computing with Spark
imp.i384100.net/QOMZ09
3. Learn Bash scripting/Linux
I wouldn't take a full course on it, but at least read a good article.
if you do prefer to take a course/guided project, I think this one is short and good:
www.coursera.org/projects/com...
4. Learn how to develop on the cloud, e.g. on AWS
There are a few good courses around there, but I think the Coursera one is the most comprehensive
imp.i384100.net/P0MJBM
Discloure: The above contain affiliate links, meaning when you click the links and make a purchase, I receive a commission!
-------
Find me on Instagram: @karo_sowinska
And if you want to make my day with a cup of coffee... :)
ko-fi.com/karolina_sowinska
3 авг 2024
