The ONE Thing Every Data Engineer MUST Know 

@jashradia Sir can u make a vedio of roadmap to become data engineer by defining the tools that are in hipe becuase there is too many tools available

@@white8254 I have a video on this: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-u2kBe9tbUsI.html

Yess. Glad it was helpful

By the way I'm using Apache Airflow and Redshift for my data engineering tasks.

That's good. Both of them have many tools to enable data lineage and Metadata!

Pleasure! 😁

Thank you, the best place to get started would be through documentation of some of these tools mentioned. I have added some document link and blogs etc in the description.

Thanks for watching 😄

Thank you! I did find veey interesting video on this with a demo that can help you get started: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-rO3BPqUtWrI.htmlsi=mO7czONV9_7tLiw4

@JashRadia thank you

Thank you 😁

Definitely one of the most secure domains of tech! All the best 😊

Yes. Just like any other SWE roles, data engineering roles also require and often have openings for freshers.

Yes, data engineering is also a part of software engineering. You will find a lot of overlap for sure. All the best!

Thank you @JashRadia so much for answering my question! Your explanation was really helpful. Keep up the great work!🥰

Hi, first of all, for any data role (ML or data eng etc) prerequisites are data modeling, data warehousing, sql and programming (python) once you have that, upskill in the area of your choice. Be it ML or distributed compute etc. This should be enough for you to land a job in this field.

Both viewpoints about the future of Data Engineering have their merits. On one hand, advancements in AI and ML are making some tasks in Data Engineering more automated. Tools are being developed to handle repetitive tasks like data cleaning and integration more efficiently. On the other hand, as AI and ML become more advanced, the need for high-quality data is growing. Data Engineering will remain crucial because ensuring data is accurate, consistent, and well-structured is essential for training effective AI models. So, while some routine tasks in Data Engineering might get automated, the overall importance of the role will likely increase. For anyone looking to switch into Data Engineering, it’s a field with a lot of potential and evolving opportunities.

@@JashRadia Hey man, thanks alot for taking time to give an depth answer. I really appreciate that. Just one more follow-up question: You mentioned that data engineering will remain crucial and that's wht I believe too. I'm keen to know which tools or aspects of DE you think will remain crucial in future too. Additionally, what new and upcoming opportunities or tasks are emerging that data engineers can look forward to? Like in GenAi, Ai or Machine Learning. Sorry for the late response😅

​@@shubhamguptaherehi shubham, how are you going to switch into this DE field? Are you going to join any courses? I'm alike you.

@@husler7424 I'm following the roadmap that Jash provided-it's the best one available on the internet. I'm also working on some projects tht will help me transition into a DE.

​@@JashRadia Hey Jash, could you please respond to my last question? I know you are very busy, but if you have a moment, I would really appreciate it🥺

Thank you!

Yes, data engineers also have freshers. They are ones who know more SQL along with normal programming

Checkout the book definitive guide to data modelling by kimball

@@JashRadia it's covers datawhare house right?

AI engineering is 50% same as data engineering. Rest depends on what field of ML you go to. I was in LLMs so apart from basic ML, I had to upskill in LLM application integrations, fine tuning and parameters tuning

​@@JashRadiaplz reply to my comment 🙏🙏

Hi Shlok, how are you planning to start DE? Are you joining any course?

@@husler7424 Not as of now. First I will probably go through company training and then talk to seniors to understand exactly what to learn, from companies point of view also. Then I might take up a course

But you can see Jash's Data Engineer Roadmap video, he provides a good resource there.

A good insection between MLE and App development is hard to find. Python and Go can be good options. And thanks 😊

@@JashRadia thanks a lot brother, it means a lot

brother why not c++ or the java

​@@JashRadiapls tell in a bit of detail. It's a request

@@hemadevi7106 c++ while is the fastest to execute, it isn't used in ML. Java is fairly complicated to learn compared to Python or Golang

See my roadmap on this channel

@@JashRadia there are many. Can you plz share the link?

@@husler7424 there is only one. Roadmap.

​@@JashRadia As there are not entry-level internship for data engineers so can you give suggestion about it. I have found every where they ask for experience level only.plz reply ❤🙏🙏

First thing is, make sure your resume is well suited for ATS screening for that job description. Then ask for a referral as much as possible. Hunting a job is a long game. Keep trying

@@JashRadia thanks😀

There are entry level internships. Just search for swe roles in data teams