What Is Data Engineering - Why Is Data Engineering Important?

Подписаться 99 тыс.

Просмотров 72 тыс.

50% 1

What do data engineers do?
Do they build data pipelines?
Do they build data warehouses?
Sure, but why?
If you're team needs help setting up your data engineering processes, roadmap, or infrastructure , then set-up some time with me here - calendly.com/b...
If you enjoyed this video, check out some of my other top videos.
Top Courses To Become A Data Engineer
• Top Courses To Become ...
What Is The Modern Data Stack - Intro To Data Infrastructure Part 1
• What Is The Modern Dat...
If you would like to learn more about data engineering, then check out Googles GCP certificate
bit.ly/3NQVn7V
If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here.
seattledataguy...
Or check out my blog
www.theseattle...
And if you want to support the channel, then you can become a paid member of my newsletter
seattledataguy...
Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio
_____________________________________________________________
Subscribe: / @seattledataguy
_____________________________________________________________
About me:
I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consult on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data.
*I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you.

Опубликовано:

7 сен 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 56

@SeattleDataGuy 10 месяцев назад

If you like data engineering and you are looking to start you're own data engineering/analytics consulting company, then you should check out my new course here - courses.technicalfreelanceracademy.com/courses/starting-6-7-figure-consulting - and use the coupon code "deconsult" to get 50% off

@dAntony1 Год назад

Thanks for the quick rundown and the channel. I’m currently a mathematician who’s retraining in a Computer Science master’s program and have been looking to specialize in an area of CS. I think data/database systems/data engineering is the way forward for me. I like the idea of being able to provide utility for other people using computer systems.

@JackalBruit 6 месяцев назад

. . . in under 10[min] u have reminded me JUST HOW MUCH i still have / want to learn

@juan.o.p. Год назад

You've definitely improved the video quality, I don't know if it is just the light but it looks really professional 🤩

@SeattleDataGuy Год назад

Hahaha this...was a 2x thing. I have another video that looks like this. But I rented this studio. Really good deal though 40$ an hour.

@gavinkalaher7314 Год назад

Hey Ben, hope you're good! I'm still learning like crazy 6-7 days a week. Every day gets easier in regards to the knowledge and coding practice. Every day, enjoying it more annd feeling pumped from conquering the tough challenges.

@SeattleDataGuy Год назад

I am great! I hope you continue to stay pumped! You got this!!

@gavinkalaher7314 Год назад

@@SeattleDataGuy awesome, glad to hear it! Many thanks. I'll definitely make sure I continue to keep smashing it ;) 💪

@antoniusdaivap7759 Год назад

would you summarize a bit how you learn data engineering? I'm kinda lost

@gavinkalaher7314 Год назад

@antoniusdaivap7759 are you already a software engineer? If yes, learn data engineering. If not, learn data analytics....check out: Microsoft Certified: Power BI Data Analyst Associate. Data engineering is not entry level, it requires minimum 3-5 years experience. Data Analytics is entry level and is the first step towards entering the data industry.

@ks8411 Год назад

Every upload makes me confident with the topics im learning of DE. I have a sense of what im doing. Thanks Ben!

@SeattleDataGuy Год назад

Great to hear!

@josephk.9567 Год назад

leveling up the cinematography!

@SeattleDataGuy Год назад

sadly a 1x thing for now. I rented a great studio!

@sanjayplays5010 Год назад

Whoa, loving the new lighting and background! Also great content, as always. Just fyi, your videos have helped me progress so much in my DE career. Thanks so much Ben!

@herrunbekannt3822 Год назад

Good content. The part about having to deal with data curation as a Data Engineer cannot be understated. I never saw godd enough source data that did not require some kind of repair during the ingest phase

@fazerydevs Год назад

thank you Ben I usually hard to understand your content bcs the speaking pace is a lil bit too fast but seeing this video convince me, ur already corrected it. great content!

@twistedsno6734 9 дней назад

I’m naïve to this field, but I have a question. Couldn’t someone in the IT department take on most of what you’re talking about? And with a trained AI, could half of a data engineer’s job be automated? What’s stopping a company from shifting a data engineer’s role to the IT department, with some AI assistance?

@preservemind225 Год назад

Hey, I got your reference form Luke Barousse's video and after watching one, I keep watching your videos for more information about the DE, I am really curious and eagur to learn and because of that I changed my line form Computer engineering to Network security to Data Science and Now I am just looking and learning new stuff from your channel. currently working on Pysparks and if you can suggest me better resources for it. Keep it up, and appreciate your work. Thank you. I feel like writing a mail to you🤣🤣

@thegreatplanet-ir8lp Год назад

Data engineering or DevOps needs some advice for the career.

@alisonhenley2551 Год назад

Super great vid, thanks for the info

@SeattleDataGuy Год назад

Glad it was helpful!

@IDSEC_VaibhavKhanna 7 месяцев назад

Thanks man!

@SeattleDataGuy 7 месяцев назад

you're welcome!

@ChristianRRL Год назад

Thank you for the videos Ben. I’ve already started reading through The Fundamentals of Data Engineering and it’s a phenomenal book so far! One unrelated question though, I’m personally very interested in networking more in this field and I’m wondering if there are any professional or technical “data conferences” you might recommend?

@malharyadav3357 Год назад

what are the base fundamentals to expertise to become a good data engineer like tools and languages may vary and change but some base concepts haven't changed for years?

@SeattleDataGuy Год назад

Good question! Have you checked out this video on DE skills? ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-LgSHaOvNodA.html I tried to show how you should build a foundation of skills in it

@govardhandigumurthi592 Год назад

Could you please tell what are the certifications that will weigh on freshers resume to get hired as a data engineer? Or what are the things that hiring managers are looking for in a freshers profile to most likey get a call

@SeattleDataGuy Год назад

If you do want a certificate, i'd only recommend you get 1 and then focus more of your time on data engineering projects. In terms of which one, any of the 3 big cloud providers will likely be good.

@raviwelcome19 4 месяца назад

Guruji Pranam.

@JeffersonOsorio-gx1gw Год назад

Could you make a tutorial with a project with airflow??? NICE CONTENT! thank you!!!

@claykc Год назад

Can you explain a very basic example of an ETL moving from SQL Server to S3 to Redshift? Or a different combination of technologies. I just want to see where SQL fits in with which tool, same for python. Everyone talks about needing these languages but doesn't explain how to actually use them. I have experience using the languages but not sure how to build a production ETL with them.

@ady3m_Islam Год назад

It depends if you want real time data transformation and loading into redshift, from what I understand you can use bash shell script automation to read the data from SQL server then transform it with bash or python, then load them to redshift, you can use something like Apache airflow to set up this ETL job, hope this is helpful

@ady3m_Islam Год назад

You can use something like Crontab to set up automating the execution of your scripts on your desired time interval

@iheanachochigozirim7740 Год назад

yaay new studio💥💥

@SeattleDataGuy Год назад

Sadly this is not permanent

@dmitriyobidin6049 Месяц назад

How much time you spend on average doing k8s/docker/system administration?

@ajr1791ze Год назад

Should I learn Scala

@yusriyunus150 4 месяца назад

Ben, I really enjoy your videos, but the frequent quick pauses in your talks can be distracting and make it hard to follow your points. Speaking more smoothly to improve clarity would be nice. Keep up the good work!

@BelkisQuirogaE 5 месяцев назад

Is that the same as being a statistician?

@IamMQaisar 4 месяца назад

0:33 DAG is represented as a cyclic figure ???

@spence2294 2 месяца назад

Because it’s a cyclic graph

@mansoorahmadkhan3623 Год назад

Hey, I want to pursue my career in data science (Forcefully) as I will be moving to another country and there is less chance of getting a job in my current domain (game-dev) as only a few companies exist. I do have a programming exp. of 5 years. What is your general perspective/thoughts, can anyone with programming skills adjust himself as a data-engineer? Is it future proof?

@samuel-je1xt Год назад

Hi Ben, Thanks for Great Content! Quick question how to create something like dynamic gif or video something like that you showed at @2:13 ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-wyGAYa2UMXQ.html ?

@JackalBruit 6 месяцев назад

woah! ya know how u build an image of someone in ur head .. maybe it's a coworker ur S.O. talks about .. or a SubStack author nothing bad - but u did not match at ALL the image i had concocted 😅

@SeattleDataGuy 6 месяцев назад

haha..are you saying I don't look like what you expected? I honestly feel like I have a few different looks...this one is shaggy hair but shaved...

@leschymero9724 Год назад

Kinda preaching to the choir here

@SeattleDataGuy Год назад

🤣

@darshantawte7435 Год назад

For someone who has worked in DE in top companies like Facebook, you tend to confuse your audience a lot. Please either support the case or reject it completely 😂. I am an aspiring Data engineering but some of your videos really confused whether to go for it or not.

@SeattleDataGuy Год назад

could you give me an example when i have said not to go for data engineering? I am just not sure of when I have said not to become one. I have a video that titled "don't become a data engineer" but i believe even there, its more of a title and the actual content is still pretty supportive.

@darshantawte7435 Год назад

@@SeattleDataGuy Lol you took it seriously man I was kidding 😂. Love your videos by the way.

@SeattleDataGuy Год назад

@@darshantawte7435 OOOOh! Hahaha, I misunderstood. I would hate to know I confused anyone. 🤣

@darshantawte7435 Год назад

@@SeattleDataGuy just one question, i know Data engineering is not a job for typical freshers so what can we do regarding that. Also how are DE opportunities in US as I had done some research earlier but haven't found a lot of jobs in this field(Atleast not by DE as the job description). Do companies provide internship for DE ?