If you guys want to learn more about data engineering, then sign up for my newsletter here seattledataguy.substack.com/ or join the discord here discord.gg/2yRJq7Eg3k
I have been watching your videos for a long time but it feels like you keep circling back to similar topics such as this....maybe show us actual data engineering; what's under the hood. Just a suggestion. A portion of your followers are people looking to break into DE and honestly when I started all I saw were creators defining a star schema but never showed me an actual one from BigQuery or Snowflake. I made one wrong because my only reference was the diagram creators kept showing haha but all love!! Your content was one of the few things that pushed me to switch roles in my company to pursue an engineering track
your story is exactly why I stopped pursuing data science. About 5 years ago there was a big push to start training people to move into data science. I went through a 16 week training program at my job and got to talk to experienced data scientists and they all talked about how many times you will fail as a data scientist. With that, I thought I'll probably never be able to deliver anything of any value to anyone. So, I gave up and I eventually my existing role sort of morphed into more of an analytics engineer. I'm def grateful I didn't keep pursuing data science because i'd probably be miserable
Wow, this describes exactly my journey over the years: the growing disappointment about the lack of groundwork - that I did not know was a thing in the first place. I wish I had realized sooner! I'm now in MLOps, but frankly want to acquire DE skills as it not only will it enable me to push initiatives forward but also give me more independence. And to be honest, a lot of MLOps is very much related to data quality and pipelines - so it feels like just another spin off of the data scientist. Who knows, might become a DE for good, if not a very well rounded ML/SWE engineer. And to start a company in the data space, one would definitely need DE knowledge in order to lead and push forward. Thanks for sharing your story! It was inspiring to hear it and I'm sure more people will realize they are not alone in the "data scientist disappointment".
I did the same. Started as a software engineer working on on data management systems to data scientist back to data engineering. My biggest problem with DS was that management always expect results and want to put you into the same agile structures with the software engineers. I found it almost impossible to correctly estimate the effort let alone the outcome of a DS project. And also tooling. I neither like R nor pandas.
Thanks you a lot for your youtube channel. As a newbie working in Big Data, i started too as a Data Analyst in BI. Your advices and expertise helped me choose Data Engineering rather than Data Science. I would probably have moved to a scientist position otherwise. The technical aspect you can do day-to-day is really something i am looking for and missed as a Data Analyst. Moreover, your presentation of the different softwares in the industry and guides on the basic knowledges helped me recently get a position as a Data Engineer. Thank you for all the good work.
This whole Data revolution started after 2010. The problem was this revolution started even before data infrastructure was set up in many companies. Just having a database doesn't make you ready for data science. Before 2010, we used to call it Data mining. Now companies are seeing the importance of Data Engineers and are setting up data infrastructure. Data Science will boom again once this is sorted out. Moreover for most companies sophisticated data analytics is more than enough to improve their business. They don't need Data Science algorithms. Our data needs are growing at a rapid pace both personal as well as commercial. Right now, data Engineering is far more important than data science.
The reason DS become DE’s is because businesses can’t and don’t want to afford both fields. Also, DS’s have to do some sort of data engineering as part of the model development. The issue I see with the DS field, is that we still work in the place where all the managers have zero clues as to what DS is. They love talking like they do, using all the cool words like machine learning and AI, when they are in meetings. But they have zero clues what those are. So the old fart managers hire young data scientists, who come in and don’t do any data science. And the managers eventually turn them into data analysts, who do tons of ETL work. Which is just data engineering lite. Which I bet is more why you became a data engineer. 😁
this is exactly where im at right now, the questioning probing with clients drains me out and i wanted a more focus task, because from where I am working right now, you do basically everything from gathering all the requirement up to the datasets and presenting them after. It kind of left me questioning if I really wanted to be in this industry in the first place. This made me realize that I do want to continue but towards data engineering and not analytics. Thank you!
lol you should have switch to Deep Learning Data Science, since that's where all the cool stuff happens. Data Eng is just data plumbing which will be automated as the industry matures.
I think I currently re-live your journey just 8 years later ;) i am a data scientist that is becoming a data engineer due to the same reasons you mentioned.
I have the same experience working as a data scientist in healthcare after 3 years. Stuck in research instead of actually building a product. My question is does this feeling arise due to a particular domain (life science) vs eg. Ecommerce or is it the Data scientist role in general? Would love to know your thoughts on this.
I feel like most people have No idea that you cant learn DS by watching RU-vid tutorials or projects, you have to be really smart, a lot of mathematics & statistics knowledge is required - boring!
After 4 years as data scientist, from last 8 months ventured into the field of data engineering to explore. But my experience with data science was great as well.
MLE is a Data Scientist or a Data Engineer that knows how to build a ML model ? Im about to start my first job as Data Analyst and want to put some future plans. If i want to be MLE. Should i focus to get a job in Data Engineering role then switch to Machine Learning Engineering?
I was working as a data scientist for almost 2 years and I am currently having the same experience as you mentioned in the video an endless loop of questions and research without building a permanent product/solution. I have just a silly question, I love coding, as a data engineer will a get the chance to write a lot of code?
Can you be more specific as to why you consider an endless loop of research questions undesirable so we can pinpoint the idiosyncrasies of your experience of data science work? I would guess that this sort of loop would be candy for a voraciously curious person. Perhaps I misunderstand. So correct me. If we focus on the endless research questions vs. more sense of permanence in problem solving and broader set of problems…. What sort of person would prefer data science? What sort of person would prefer data engineering? Great video. Your channel has helped me a lot.
Is that same for mle .i am thinking to get a job as data analyst and after some years of experience try for mle jobs is that this situation is applicable for mle ( because iam thinking that ds and mle role has similarities ) .
Love this channel! I have a question. I have been doing Helpdesk for the last 2 years. I just passed the DP-900 Azure Data Fundamentals also just graduated with my Software development degree. I know sql and python. What skills and roles should I focus on to be a good Data Engineer one day?
Hello Ben, I am relatively a new follower of your content both linkedin and youtube. Accidentally, I found out your content on Palantir Foundry and I was intrigued because 1. There is very less talk about it 2. And I have been working on Foundry for 2 years and have involved in end to end data life cycle from data ingestion to creating an operational application and analytics. Recently, working on their new streaming app. I am trying to come out of Foundry and work on so called "modern data stack". Any positive on Foundry in 2023? I still don't see much traction for Foundry. What do you think? Hoping you will reply 🙂
I have seen you doing same video again and again. Request you to talk about some of the best practices about data engineering . I thought your channel is different but all you doing recently is talking same stuff over and over again . I guess channels do get boring after a certain time
