Should You Become A Data Engineer? Reasons Why People Are Becoming DEs

Подписаться 97 тыс.

Просмотров 31 тыс.

50% 1

Data engineering is a growing field.
Whether it be billion dollar start-ups or increasing jobs, everyone needs data engineers.
The question is, do you want to become a data engineer?
Do you like automating systems? Or building data analytics systems?
Do you like being on the cutting edge of technology?
Then maybe being a data engineer is right for you!
What Skills Do Data Engineers Need?
• What Skills Do Data En...
If you want to learn more about data engineering, check out Googles Data Engineering Course
* bit.ly/3c3woxu
If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here.
seattledataguy.substack.com/
If you like this video, then subscribe to the channel!
/ @seattledataguy
Or check out my blog
www.theseattledataguy.com/
Back to my story:
Early in my career I started as an analyst working in healthcare but I eventually found myself in the role of data engineer at a FAANG and it's really been a great learning experience.
Along the way I learned how to operate on an engineering team.
How to follow good engineering practices and just in general, develop maintainable systems.
Now that companies are starting to hire even more rapidly for data engineers, I wanted to provide my story for how I went from being a data analyst to data engineer.
From taking the initiative to develop products, to self learning. All of it was required to really challenge myself to be a DE.
Thanks for reading and good luck on your data engineering journey.
Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio
0:00 Intro
1:24 You Enjoy The Big Picture And Thinking Of Systems
3:09 You're A Recovering Data Scientist
4:21 You Like Creating Automated Systems
5:28 You Like Working With Cutting Edge Tech
6:21 You Like Data
About Me:
I have spent his career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consults on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data.
*I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you.

Опубликовано:

3 авг 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 122

@JohnGonsalves 3 года назад

I really love being a Data Engineer! The amount of technologies, frameworks e patterns that we learn is amazing! And another good point is that we have a lot of open positions on companies!

@SeattleDataGuy 3 года назад

Yes! I actually pointed this out on the tiktok i just made. But there are tons of openings, tons of consulting projects, and just tons of DE work. Even with tools like Fivetran trying to reduce the number of DEs, large companies will need to take on large migration projects to switch. So we have at least 5-10 years with this position being very popular (at the least)

@user-hw9fr4fr6q 2 года назад

You really solidified the reasons and goal for my major! Thank you!

@HarshKukreja052 3 года назад

I have started following your channel since last week.. and dude I must say "You are doing a great job" ❤️ I'm myself preparing for Data Scientist role but like you I've already started liking the DE aspect of the whole DS pipeline

@anthonymous7191 3 года назад

we need to give this guy a medal! Love the content

@SeattleDataGuy 3 года назад

You're too kind!!

@minthura24 3 года назад

Thanks for the video! Your videos are really great and helpful for people like me who are new in DE field.

@SeattleDataGuy 3 года назад

I am glad you enjoyed the video! How goes your journey into the DE field.

@mark.AI404 4 месяца назад

Great video! I'm currently deciding between being a DA, ML engineer, and Data engineer. I've finally gained the confidence to focus on Data engineering.

@MichaelWSchmidt 2 года назад

This video is LIFE CHANGING!! Thanks!

@SeattleDataGuy 2 года назад

This comment is life changing! Thanks for your support!!!

@brian.javier 3 года назад

Still a beginner Data Engineer, coming from Data Analytics, and all your videos have been great in developing a learning development plan. Thanks man!

@SeattleDataGuy 3 года назад

I am glad they are helping! Let me know if you have any questions.

@the_invisible__ 3 года назад

Thanks buddy....i am following all ur videos.. It helps me alot.. learning all technologies Will become DE one day.. :)Thank u again.

@SeattleDataGuy 3 года назад

Thank you for the comment! Do you have any particular technologies you enjoy?

@Chiefnice22 3 года назад

Great video! A video on walk-through examples/ a case study for data modeling would be helpful!

@SeattleDataGuy 3 года назад

Thank you! It's on the schedule. I think this weekend will be focused on data engineering projects and then next weekend I will aim to talk about data modeling!

@Chiefnice22 3 года назад

@@SeattleDataGuy Awesome!!

@mirandeli Год назад

maaaaan, the "recovering data scientist" bit of the video really resonated with me. I was having doubts about it, not sure if those preferences justified trying DE out. Thanks for the video! :) You are helping me a lot to understand more about data engineering!

@SeattleDataGuy Год назад

Glad the video resonated! I hope you find a role that you enjoy!

@danianceb 3 года назад

Thank you, this was a great video. I've been in data analytics for a little under a year but I am starting to think that DE is a better fit for me.

@SeattleDataGuy 3 года назад

That is awesome! What are you liking about it thus far?

@danianceb 3 года назад

@@SeattleDataGuy In my current role, I do enjoy using SQL and I have had the opportunity to see how other analysts use dbt and BigQuery to do ELT/ETL work. I find that I am more interested in how to prep data for analysts to use.

@chaitanyadeshmukh1912 3 года назад

I'm am currently working as a DE. still at the early stage. This will definitely help me. Thank you for the video. 💯

@SeattleDataGuy 3 года назад

I am glad this video helped! How are you enjoying your data engineering experience thus far?

@chaitanyadeshmukh1912 3 года назад

@@SeattleDataGuy Struggling right now, to be honest. Learning a lot. Bunch of new technologies. Currently working on to get my basics strong.

@mushroomvv 2 года назад

Great vid! I am a DE but I find it hard to illustrate how much impact we've made through automation, pipeline, etc. using the same toolset for similar purposes during performance calibration. I guess the criticality of downstream projects is one input, before/after manual labor cost comparison is another.

@SeattleDataGuy 2 года назад

Yeah...we are often just viewed as a cost sadly.

@elimgarak3597 3 года назад

I work as a BI Developer, and DEs seem to be the BI Devs of the Big Data world, so it is a really interesting and exciting topic for me. Ty for the very informative vids. Btw, perhaps a vid explaining the differences between both roles would be interesting.

@SeattleDataGuy 3 года назад

Thanks for comment! Yeah I actually put BI dev instead of project analyst when I was applying for data engineering jobs because the work that I was doing was more BI Dev. So often there is a lot of overlap for this skill set. That would be a great topic. There is also now the term analytics engineer...so many terms...

@danielhooverc 3 года назад

Thank you. You give me confidence of becoming a DE. I am studying hard now.

@SeattleDataGuy 3 года назад

Good luck! What's been your favorite subject so far?

@danielhooverc 3 года назад

@@SeattleDataGuy I am a new man in this area. There are tons of materials I am trying to learn. Hadoop,Kafka,Spark,Hive,Zookeeper,Airflow, AWS,Data Modeling,ETL etc. Sometimes , I feel overwhelmed and confused facing such tools. I was wondering what are your daily tasks as a data engineer? Thanks.

@SeattleDataGuy 3 года назад

@@danielhooverc I would say start with SQL, python, data modeling and data pipelines. Those are really the core skills. The other ones like spark, hive, hadoop, and so on are sometimes abstracted with tools and third parties so you don't have to spend too much time learning about the nitty gritty right away. There will always be companies that are using hadoop, spark and so forth. But like you said, its overwhelming. So start with the basics and eventually you can start looking into tools like hadoop and spark.

@danielhooverc 3 года назад

@@SeattleDataGuy Thank you for such a detailed answer! It is informative.

@SeattleDataGuy 3 года назад

@@danielhooverc I am glad it helped! good luck!!!

@JoesKidsChannel 3 года назад

Recovering data scientist - first of the lot and OG

@SeattleDataGuy 3 года назад

All the inside jokes!

@Happyflap 2 года назад

I am starting my data journey. I aspire to become a data engineer and currently I am a software engineer in test. I like how you explained the role of a data engineer in this video. I resonate with all the points that you have given and I am happy that I am choosing this for myself. Thanks again!

@sharad3877 Год назад

I am also working as a Software Engineer and trying to switch to data engineering. Can we connect on LinkedIN?

@reviewguy1766 3 года назад

Subbed! Do you have any advice or could you make a video on how to get into data engineering if you already graduated and working in an unrelated field but have a non-technical unrelated to technology?

@SeattleDataGuy 3 года назад

Thank you so much for the sub! I have a video about how I went from analyst to data engineer, but i guess an analyst is somewhat technical. What is your background in?

@datadrivendecisions0 3 года назад

Having been a data engineer for the past few years I can't see myself changing profession anytime soon! I became a data engineer because I love harnessing the power of huge datasets

@SeattleDataGuy 3 года назад

Yeah! It's a great career and under discussed for sure. Good luck in your data engineering career. Also, I will need to check out your youtube channel

@datadrivendecisions0 3 года назад

@@SeattleDataGuy Thank you, you too! Be my guest! I'm currently working on my crypto currency trading bot before going back to my football betting algorithm. Maybe we can talk about the differences between data engineering on either side of the pond sometime!

@brianl839 2 года назад

@@datadrivendecisions0 how's the pay?

@datadrivendecisions0 2 года назад

@@brianl839 very good rates

@thirumalaip6458 3 года назад

Please make a video regarding your favorite books :) I see some books next to you :)

@SeattleDataGuy 3 года назад

Great idea!! I do need to do that. Some of the books there are books I enjoy, others are just the ones I have in my room at the moment.

@thebushscientist4473 2 года назад

I am currently a DevOps engineer and I like it because of new tools I get to work with but it is a deal breaker for me as I have to support the applications and infra 24×7. I am thinking of switching to Data Engineer role. What are your thoughts on this. I kind of looking for a job role in which I work hard for office hours and go home to have a life. Is the DE role suitable for me?

@JenTingHuang 3 года назад

Thanks for the video! I have a question. I am an undergraduate student, and I want to become DE in the future, still, I am wondering which master's degree I should choose to improve my ability or is better for my career choice, data science, software engineering, or computer science? I think it seems like CS or SE's skills are more helpful for DE job.

@SeattleDataGuy 3 года назад

Glad you enjoyed the video. I think for data engineering it makes sense to get a degree in SE or CS instead of data science.

@uranio58 Год назад

Hi Ben! I would like to work remotely for USA as a Data Engineer. What job search sites do you recommend besides Linkedin? Thanks!

@uzferry5524 2 года назад

I'm on a SWE to DE path right now. Let's see if it's going to work out for me

@SeattleDataGuy 2 года назад

Good luck!!

@Sawako255 2 года назад

Hi, Ben. Thanks for posting such a useful video. I've recently started to ramp up my studies in Python and MySQL, but the sheer amount of possibilities for different career tracks is just so debilitating sometimes. Not to mention the amount of technologies to learn. I have a bachelor's degree in Anthropology so it's not exactly useful for finding work, so I've started taking courses on Udemy and Coursera to try to skill up. How would you recommend a newbie approach the tech job market? Do you think it's best to learn certain tech skills first or just pick a job posting and learn the required skills? I've thought about the second option but some of these adverts list so many things I feel like it would take me years to learn it all.

@SeattleDataGuy 2 года назад

Hmm, this is a harder situation since your degree has nothing to do with technical skills. Can you pick up the skills...yes! But the hard part comes from hiring managers and recruiters looking at your degree and lack of experience.. Here are some thoughts. 1. If you can find an affordable bootcamp...this could be helpful. Make sure they provide a lot of help getting you a job. I have seen people with english and non-tech degrees get jobs through this route 2. Start creating content and projects. You will have to work doubly hard since you have nothing to point to 3. See if you can find a start-up. They might pay you a little less, but maybe you can gain experience.

@Sawako255 2 года назад

@@SeattleDataGuy Thanks so much for taking the time to reply, and for your advice!

@elmurat Год назад

Hi Rilken! How is your search going on? In my opinion, it would be easier for you to enter into Data field through BI Analyst/Developer role. Which is mainly learning the BI tools like Tableau or Power BI and SQL. After you find a job as a BI developer, you will have opportunity to work with Data and then you can switch to Data Engineering role.

@Sawako255 Год назад

@@elmurat Hi Elmurat, thanks for the advice. I think you're right. I actually got a couple of replies to applications for python with barely anything to show in my portfolio, but I wasn't able to pass the technical assessment. I was surprised that I got any responses back so it was very encouraging. I'm doing as you said now, and I was able to get a more technical role in my current company.

@harshitsharma502 3 года назад

Hi First and first thanks a lot for your videos I am following you. I have one question I have experienced around 10 years and now I am preparing for FAANG companies should I focus on DS&A too? And if then how much? And from where should I prepare this ? Thanks in advance!

@SeattleDataGuy 3 года назад

What specific role are you interviewing for? Data engineering?

@asirbillah4987 2 года назад

Hello, nowadays I can't decide whether I should go for MSc in CS/DS abroad as I don't know whether it would really help me smoothly start my Career as a DE. I'm from a developing country, currently a front-end web dev. Just from financial aspect, should I switch to DE? Or should I just practise coding/ algorithms more for being a better SWE?

@SeattleDataGuy 2 года назад

I could see some value in honing your SWE skills. But what is the income difference of swe vs de in your country?

@avneetkaur7606 3 года назад

Hi, Thanks for this amazing informative video. I have a question: how can a fresher really figure out if the Data Engineering role is appropriate for him/her? Like at university we learn a wide variety of stuff but we don't really get exposure to any particular profession. I am a Physics and Mathematics student, so I definitely have a liking for Data and Mathematical stuff but how to exactly choose that one career, for example, even in the Data side there are roles of Data analyst, Data Scientist, Data Engineer which require knowledge of different tools and have different lifestyle altogether. Is experiencing these professions the only option? If yes, what skills to have to be open to all of these roles?

@SeattleDataGuy 3 года назад

Those are a lot of questions! I will do my best to answer what I can. As far as knowing which role is best for you. Life is long and you will likely have the chance to try a little bit of all these roles especially if you join a start-up. But early on I would try to play around with different roles. Personally I think it makes the most sense to compare data engineering vs data scientist, since you will be doing a lot of data analysis type work in both roles. These are probably the two core data positions and there are so many different types even here. Some people are more research scientists (PhDs usually) that do a lot with complex formulas where as other data scientists are a little more practical. For data engineers you might find yourself building data infrastructure and code, data pipelines, data viz, etc. Its a pretty broad role and every company will have different tools. Even in the DE world in terms of tools. There are so many types of tools. Low code, libraries, custom code, streaming, batch etls, cloud data warehouses, databases, distributed file systems... So personally, just try different roles and see which one you like best. Also I have a video called the realities of being a data engineer. That might help?

@avneetkaur7606 3 года назад

@@SeattleDataGuy Thank you for your efforts to answer my questions. It has certainly given me clarity in what to look for.

@SeattleDataGuy 3 года назад

@@avneetkaur7606 I hope it helps! Good luck!!

@Alex-es3fd 3 года назад

Greetings! Is it possible to become a Data Engineer with a Business undergrad and Data Analytics Masters degree? Is a DA degree frowned upon compared to a DS or CS degree? I am aware I’d have to work as a Data Analyst first and try to venture over.

@SeattleDataGuy 3 года назад

I had an information systems background which was rooted in a business background, was an analyst and moved into data engineer. So i think its fine. Are there major skill gaps you have?

@darint07 Год назад

I come from a software developer background and what I am finding is that data engineering definitely doesn't really have an entry point. You either have 5-6 years of experience via some back-end programming job or you're stuck. Am I wrong? Are there 'jr' or entry level data engineer jobs that I am just not seeing. Thanks for the content by the way. I am hooked.

@datapower6162 2 года назад

Hello, Thanks for these detailed videos. I am computer engineering graduate, who is interested in data engineering and I know python well. I want to get knowledge and start career in data engineering, so how much time will I need to start my first job as a DE, I am motivated and learning it every day

@SeattleDataGuy 2 года назад

Generally you will want some SQL and data warehousing knowledge to break into DE roles.

@hg270 2 года назад

Hey seattle guy, any good platforms to search for entry level data engineering jobs without experience? I am about to graduated from a bootcamp

@SeattleDataGuy 2 года назад

There isn't really a great platform. The best thing is 1. internships, 2. Referrals, 3. Apply to 100 jobs and pray, 4. put together an online presence(This could actually be 3, but it takes time to create this)

@adityaanand835 2 года назад

Hi sir, I am working as an ETL INFORMATICA DEVELOPER in an IT firm for the past 3yrs, hv worked in etl & sql developement, creating workflow, mapping, optimization etc, have little good exp and knowledge in building those stuffs so then i thought why not go for data engineering as these are the initial phase for it so i did few basic certifications in AZURE cloud data engg. Really got intrested in this domain, watching and experiencing this data engg world, the concepts ,sql queries and data behaviours , really want to build my career in this path. Sometimes i felt bored uninterested as there are lots of components , applications & theories to learn if we work in some cloud architecture like azure in ths case but I feel i can do something good and learn better into this. could u suggest am i doing good, going good? what more i need to learn and work on? i am also confused because my current organization doesnt hv that relevant projects too so there are quite less experience in this field for me i guess..? Kindly help waiting for ur reply?? :)

@SeattleDataGuy 2 года назад

I feel like you're on the right path. Have you tried interviewing? You might need some other big data concepts like hadoop and such. However that will depend on the company.

@adityaanand835 2 года назад

@@SeattleDataGuy Thanks for replying! I hv not tried interviewing for data engineer but for etl informatica developer YES! and also trying to learn datascence concepts currently.. will try to apply and see something would be possible i believe..

@olarius650 2 года назад

Does mature workers(over 40, with different IT background) have any realistic chances to enter the field of data engineering?

@SeattleDataGuy 2 года назад

Of course. Still 25 years of work ahead 😅

@Crazy8xxx 2 года назад

Take into account you will be expected to support production over holiday periods and weekends. Not every team or organisation you will work for will have a dedicated production support team. When you are starting out in DE there is a pecking order where you will be that developer supporting production issues and implementing bug fixes while your peers are involved in project work. That’s just how it works

@SeattleDataGuy 2 года назад

Yes this is true. i have done my fair share of holiday on call.

@bhargav8232 3 года назад

I also tried to pursue Data science career and invested a lot of time and energy. Only to realize there aren't many jobs and it's also not very interesting. Could you please share some resources where I could learn DE from basics?

@SeattleDataGuy 3 года назад

Yeah, I think this is a common path. Data science gets you into data, data engineering keeps you here! I have a video on data engineering courses online: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-lVj0RlSxTXk.html

@yashmoyal8543 3 года назад

Hello sir I wanted to know as I am fresher and in 3 rd(B.Tech) of my college can I also become data engineer as or it needs to be transitioned from software engineering for this domain Please help!!

@SeattleDataGuy 3 года назад

It's usually easier to start either as a software engineer or a very technical data analyst/bi developer. Data engineers tend to be very specialized. Usually a good software background + data pipelines and data warehouse. But I am seeing a lot more data engineering intern positions out there. So I think there will be a clearer path over the next few years. So you might be in luck! What is your background?

@yashmoyal8543 3 года назад

@@SeattleDataGuy I am computer science student. And in 3rd year of my college

@SeattleDataGuy 3 года назад

@@yashmoyal8543 Well you have the right background. I would say apply for both data engineer and software engineer jobs! See which sticks. You still have so much time to learn and switch jobs.

@yashmoyal8543 3 года назад

@@SeattleDataGuy Thank you so much sir!!

@SeattleDataGuy 3 года назад

@@yashmoyal8543 Glad to help! Good luck on your data engineering or software journey.

@wendellsmalls637 3 года назад

@Seattle Data Guy, How much statistical Analysis and Data visualization does a Data Engineer Have to do, as of right now I hate creating reports and doing visualizations.

@SeattleDataGuy 3 года назад

My experience is usually someone else does the statistical analysis or machine learning models and then sometimes we implement them. As for data viz. This varies, but I would say you should know at least one or two data viz tools and know some basics in terms of data viz best practices. I like this medium article in terms of data viz. uxplanet.org/10-rules-for-better-dashboard-design-ef68189d734c

@wendellsmalls637 3 года назад

@@SeattleDataGuy How much do you guys work with Excel ?? Right now as a data Analyst i am living in Excel and Sql for reports

@SeattleDataGuy 3 года назад

@@wendellsmalls637 I don't use Excel that often. Maybe for the occasional quick adhoc thing. But its more about SQL, data modeling, data pipelines, automation, programming, etc.

@wendellsmalls637 3 года назад

@@SeattleDataGuy From a data Analyst to Data Engineer, how long can it take to be competent in this career.

@SeattleDataGuy 3 года назад

@@wendellsmalls637 Good question. It does depend how much diligent practice and practical experience you get the opportunity to do. For example, I have had to interview data engineers with varying levels of experience and sometimes people with a lot of experience struggle because they don't write SQL or python often. They might use more drag and drop style tools. However, I would say 1-2 years to create a good foundation and 3-5 years to start being confident and starting to take on projects by yourself. from there you can either go the management route or the techlead route.

@reyesdf 3 года назад

Will you code as being a DE?

@SeattleDataGuy 3 года назад

Yes, coding is required for Data engineers

@bhalchandrakolekar8176 3 года назад

I love doing DE stuff on cloud everyday....

@SeattleDataGuy 3 года назад

I do too! What is your favorite data engineering tool(s)?

@bhalchandrakolekar8176 3 года назад

@@SeattleDataGuy Currently I am working on GCP, mainly with BigQuery and Cloud function. Loving these tools totally. Also, now I am planning to learn Snowflake. The demand for Snowflake has gone insane. Crazy shifts there.

@SeattleDataGuy 3 года назад

@@bhalchandrakolekar8176 Yeah I totally agree. GCP and Snowflake are both going crazy from a popularity stand-point and Snowflake is constantly pushing with their marketing.

@shivendusrivastava3608 2 года назад

Hey, i'm from india . How can i get remote data engineering job ?? Please guide me

@SeattleDataGuy 2 года назад

I don't know if i have a good answer. Where would you like to do remote work?

@shivendusrivastava3608 2 года назад

@@SeattleDataGuy at usa

@jifanlou7893 2 года назад

How can I be a self-learn data engineer?

@jifanlou7893 2 года назад

Do we have to be a data analyst/data scientist first then switch to data engineer?

@SeattleDataGuy 2 года назад

I have a video on a data engineering roadmap as well as a video about how I went from data analyst to data engineer. Perhaps that will help! ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-lGzh-QendJc.html

@jifanlou7893 2 года назад

@@SeattleDataGuy Thank you

@hg270 2 года назад

limited competition? are you sure?

@SeattleDataGuy 2 года назад

Yes and no. If you have experience, yes, if you have no experience, it doesn't matter the job, it will always be hard to break in.

@brianl839 2 года назад

@@SeattleDataGuy what are the most important skills to focus on? Python? SQL? What else?

@lilbean_eth 3 года назад

Who is Saylor?

@SeattleDataGuy 3 года назад

Saylor? is that something I said at some point?

@lilbean_eth 3 года назад

@@SeattleDataGuy I noticed the book by Saylor on your bookstand, theres a meme circulating around the phrase "Who is Saylor?"

@SeattleDataGuy 3 года назад

@@lilbean_eth Haha...I just read books I find interesting..I didn't even realize the michael saylor wrote this book. I did pull this book from the Facebook Library before the whole WFH thing...so i guess it makes a little sense