@@SeattleDataGuy I'm currently working on SAP BI module which is more a operational role. I want to switch to other technologies and build BI products. How do I go about it?
I also started as an analyst and then moved to data engineering. The cool thing about that route is that you can practice data engineering within the scope of an analyst role. For example, you can take over parts (or all) of the data delivery process feeding your analyst work.
I've been an analyst for the last 2.5 years and I just signed the contract for my first Data Engineering gig! I'm so excited and I owe a lot to your videos when it comes to understanding the field a bit better. Thank you.
Thanks for your video! I have been working as an analyst back then, and now I'm in transition to data engineering role. Will be back once I landing my DE job.🤞
I’m actually looking to do the reverse (sort of): DE to DA hahhaha Although tbh, if you work with data you’ll end up being a bit of both regardless. I started looking more into Data Analysis cause it’s sometimes a pretty crucial step in the decision process of building a pipeline or developing some analytics process. At the end of the day, IT people gotta be wearing lots of hats to ultimately get the job done.
We really do. Hey titles are just that...titles. If you can write print("hello world") people expect you to be able to code a website, query a database and build an ml model.
Great video Ben! Also, thank you for your useful tips in the Facebook ds group about focusing on one area and specific skills that match SWE, DS & DE. Greetings from the Netherlands
@@ampersignia I put together this group. Really...at this point I am just sharing content. But feel free to start a discussion on becoming a data engineer or something!! facebook.com/groups/472227477321326
I've been an MI analyst for 3 years and have an interview for junior data engineer next week. Thanks so much for your insight, it's great to know how you've developed! You're a great inspiration!
I'm currently working on a data ingestion project on my data analyst job. It's been really challenging since my experience is only with python requests and APIs. Thanks for the motivation!
I am currently in your position as data analyst.. I feel like I wasn’t learning anything…not gaining much knowledge skills. I’m looking into company that’s has new system software. Health company would be good one and any technology companies. I’m glad that I came to watch your video!
I appreciate you covering this topic but it was mostly just how you personally made the transition instead of how someone else could reasonably make the transition. I’m trying to do this myself. Do you know of key words to look for in job posts that may favor someone who used to be an analyst?
Oh that's fair. Yes I guess I see that the title and the thumbnail don't exactly match. I should change it to "How I become a data engineer from analyst" or something. I think its less about job description words and more about what you have done. So if you have worked on a data warehouse, SQL, automated a process, or used some python/programming language in a project then you have work that you can relate to data engineering. That's kind of the point I was trying to make with my own story. It always helps to have the right projects on your resume. Does that help?
Thank you sharing this. You are covering a very unique topic early on which is genius. I’ve been reading up on Data Engineering a LOT the past few weeks, and I’m beginning to wonder if that suits me more. I am desperate to adopt software engineering principles on my way to one day becoming a Data Scientist, but I want to be fully functional. I want to be able to do Data Engineering, Database Design, Analysis, ML training, and MOST OF ALL, Deployment. As I said, I’m desperate to do more coding now. I teach myself ALL the time off the clock about advanced software engineering. I never get to code anymore at my job, I’m not even allowed to build much in SQL. It’s been very demoralizing, because I can do all of these things, so I’ve been asking myself if this is my next step. I just want to code again.
@@SeattleDataGuy I’ve been updating my resume and checking out some possible positions elsewhere! Actually, I’ve seen you’re a Data Engineer at Amazon, and I’ve found that exact position online for one of Amazon’s teams, which has a strong emphasis on software engineering and agile in the job description. That really excites me! There and maybe Microsoft, as well - I’d love to work for them. Have you learned more software engineering concepts on the job at Amazon? Or have you had to learn them before getting hired there? Also, do you ever do any analysis yourself, or is it strictly engineering? Thanks for responding to me!
Thanks for your videos on data engineering, they seem very grounded and more relatable than a lot of other testimonies out there which usually talk about already going into the DE field with a masters in computer science, a million portfolio projects and infinite related experience going into it. I recently quit my job and enrolled into a Data Analyst bootcamp to start a new career in it, and a quarter of the way in I learned more about DE on the side and decided that's what I want my new career to be in. After the DA bootcamp, I plan on taking Coursera's "IBM Data Engineering Professional Certificate" program. Is there anything else I can do to try and maximize my likelihood of landing a DE job afterwards besides listing projects on my resume and continuing to network with companies/people already in the field?
From there I would say try to get a job in a start-up or large company as either a data engineer or data analyst. Getting a DE job at a start-up might be easier. It does depend on your experience. Putting together a project would also be good as well as some blogs.
Been subscribing to your great contents. But I'm not even a working Data Analyst yet? Been working on a few beginner SQL and R projects and some MOOC classes and move forward from there. Working on Capstone for Google Data Analytics Certificate. Hopefully I get lucky someday. Thanks for sharing.
@SeattleDataGuy do you think you can go straight into data engineering without being an analyst? With some respectable SQL and DE certs + programming knowledge and a big portfolio project?
Hi Ben, I was once a Data Analyst and Business Analyst too. Last week I was just had my first day as a Data Engineer in a startup. I think I am now in a "cut my teeth" stage since I can now apply what I have learned in real world problem. I am kind of afraid that I will lose my desire to learn because of working. How can you maintain your curiosity and willingness to continuously learn?
That's always hard. I think one thing is to not always treat learning as a task, learning isn't always operational. Sometimes you have to force yourself to learn, but if you're not enjoying it, you're going to have a hard time remembering it. I think projects are often the best way to learn because you can apply your data engineering and programming skills while looking for new ones and having fun.
Thanks for the video, I have read a lot of your articles so excited that I found your channel through the Reddit post on this. I have a question on the website you built for your first automation project. What did you use to host the website at your company? Was it python code?
I am glad to hear that you have enjoyed my past articles! It was hosted on a local server. I was using spring framework back then. I have pretty much switched all my website programming to flask or django.
Hi, Ben! Thank you for you work for us!! It is such helpful to build the vision on the future steps. I am currently enrolled in Data Analytics program (college) and shifted this semester to Business Intelligence concentration because I think it would better for establishing needful skills for DE.. I will end in 2023 and I still be wondering, why colleges do not have the whole 2or 4 year program with specific concentration in data engineering, do you think it is because the data engineering is broad area itself, and there are a lot overlapping with other concentration? . My college advisor suggested Artificial Intelligence in Software Development program(they created it as a new one) as additional to my degrees, but it looks for me as more shift to Machine Learning that I am not interested in. Your point about needful skills, helped me mapping that I need to do, I wish you created a course/boot camp by yourself because it seems that when you work inside industry you understand more, about tendencies on job market and can be a create course lead!!
I’ve been learning DA, DS and then DE but still don’t get it. Planning to take a course on Coursera. Could you recommend me which one is better to find the job ASAP and easier “DA from Google” or “Database from Meta”??
Hi, i really like your story. i have a masters degree in epidemiology and just completed my web dev training, in the process of launching my first real world App (react nodejs mysql) for a local business. i'm really interested in data engineering i'm looking to start in a few months. how was your background in epidemiology usefull on your journey as a data engineer ?
Hi Seatle Data Guy! Great content and suggestions. Do you have resources where I can practice SQL and Spark for coding interviews? I am stuck at that stage.
I am working as an data analyst but I am looking for good opportunities to work with data in higher level rather than use only some sql/linux command to analysis. I am thinking of doing masters in data science which can help me to do research in future in data sector(bioinformatics or something like these type of significant field) . Should i do masters in data science or data analytics& business or some other field?? please suggest.
It really depends where you want to go. If you want to go down the data science route, you will likely need a masters or PhD(Not always, but it helps). If thats the case, it doesn't have to be a data science masters, but perhaps something like economics or computer science with a focus on data science. I am not a huge fan of masters in business analytics..because you can learn more working.
Hi, really love your videos and LinkedIn posts. Very valuable information 👏🏻👏🏻 I was an ETL developer (Informatica which is outdated I think now) and also worked in data operations and also as a data analyst. I really want to make the switch to Data Engineer. How do I prepare for DE role? How do I start training myself into big data? Could you share any structure or format of preparation for getting into data engineering job? Like what tech stack is required? Do you think it’d be difficult to make the switch to DE role? Thanks so much for everything you do! So helpful 😊
Sorry, just getting to some of these comments. I am glad you are enjoying the videos. In terms of structured. I have this video on data engineering roadmap.This will be great for you because you can see what you already have skill wise and only focus on what you need ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-SpaFPPByOhM.html
Essentially yes. We were using SQL server so you could create temp tables. Of course, that has it own trade-offs. You accidentally can blow up the tempdb if you don't drop them, if you need to run the same task at the same time you would have to create a dynamic table name to avoid overwriting one temp table vs another, etc. In worked in our case since we really only ran a few main pipelines. You could also use CTEs but even that can get messy.
No you're totally right, I over-engineered a much more complicated solution that still required manual intervention. I honestly think I just wanted to use spring framework. We actually did use cron jobs to load the file meta data as it was pushed into the SFTP. I am surprised we didn't just automatically load it. It just seemed like they preferred having people load it manually. I think my point there was, just try things. I think some people spend so much time looking for the right or perfect answer they never even attempt to solve a problem. I was wrong and we never used the site, but I got to do engineering work in the end. So that was a win.
@@SeattleDataGuy Ha, I’m in a similar situation right now. Just tried to shoehorn Airflow into a project that really only requires one manual run of a Python script each month. “Wasted” a lot of time, but it was worth exploring for curiosity’s sake, and now I may have found a new genuine use case solving a different problem.
Hello! I loved your video, really appreciate you sharing! I am at a crossroads in my life where I am pursuing my master's in either CS or Business Analytics. I want to eventually land a job as a data engineer! My background is a mechanical engineering major and minor in computer science. It would seem that CS master will offer a better foundation, but the BA program I've been accepted to is one of the best in the country, strong tech curriculum which teaches me Python, SQL, R, Tableau, ML, etc. I felt like leveraging the program for Data Analytics/Data Science jobs and working my way up as a Data Engineering is a viable option. I want to ask, what do you think is the most optimal choice in this situation? Should I still pursue a CS master? I don't have any work experience in this field, so I would love to hear your thoughts! Mainly, which would offer the smoothest transition, and if so, why? Thank you!!
I think a CS masters might just be better in terms of providing you a deeper understanding. If you're good at CS and have a good set of soft skills you can pick up analytical tools pretty quickly. Also, try to get an internship.
I don''t know about this specific degree, and I haven't heard of Western Governors University. I think as long as you get an internship while in college for a data engineering position, you will likely be set.
I really enjoy listen to you especially when you are talking about Data Engineering. My python programming is very very suck but I know SQL, and I know how to use Pandas to manipulate data to some extent and learning Numpy right now, am in college studying computing as a course. Please I need your advice on how to go about Data engineering because it require programming which I really really suck but I love working with data and also I love technical side of it?. Thanks.
@@SeattleDataGuy No, I dont know how to go about it. I dont know how to apply it and how they are putting the lines of codes together I mean the logic on how to link the codes, I have looked everywhere on RU-vid and also books nothing like that. Thanks.
I have very similar story, I have joined small company as Data Analyst/Scientist and I had plenty of proof of concepts ML projects, some paid off with commercial projects but nothing like data warehousing. So i switched to data engineering building extraction RPAs in Python, focused on building BI applications and ETL as well. Now I am thinking about switching job next year. I am a bit afraid that people will assume too much by my data scientist title. Can you give some info regarding QA requirements and knowledge in data engineering. (E.g. quality tests)
I wouldn't worry too much about data science vs data engineer. I am sure people will still consider you. Also, if you feel like you were doing more data engineering work, then you should consider using Data engineer as your job title. As far as QA goes. I don't have a course per se...I will have to think that through.
Just finished my google data analytics cert n I’m struggling to find employment but I already know eventually I want to upskill to a data engineer I’m wondering if I should start the google cloud engineer course or focus on honing my analyst skills for a few months to a year
@@SeattleDataGuy I’ve been applying to entry level data analyst positions, just completed a skills test for a Junior data scientist role and I did a video call interview for a data analyst role , still waiting on confirmations for them 2 positions was told it could take up to 3 weeks
@@jdcarter157 Don't rely completely on Google Data Analyst Certification. It's very good cert for Data Analyst though ngl. But, having knowledge on Advanced SQL and one of BI tool like Tableau or Power BI would land you a good entry level data analyst job from there you can upskill yourself to data engineer. To me getting my first job as Analyst is hard than switching to data engineer from data analyst.
I'm a data analyst who builds ETL pipelines, does transformations and works a little on deployment. The only forms of analysis I do is check if the pipeline is giving the correct results and so on. I'm learning tons tho! Great team, amazing engineers and pretty tough problems. Any tips on what else I can do to get the DE title? I am a recent graduate (August 2020) and I've been working for this start-up for 4 months after switching from a ETL support role.
First, there isn't a huge rush. So I wouldn't worry about trying to get the title too quickly. Just keep trying to take on more data engineering like work. Eventually you will likely get the title. If not, switch companies to where you can get the title.
I'm pursuing my bachelor's degree in supply chain,Transportation,and Logistics management. If I pursue my masters degree in data engineering, would this blend be good with my undergraduate degree.
In general, most jobs in the DE space do require you have some form of degree. They usually want comp sci, MIS, , Math, Stat, Physics, etc. However, I know plenty of people with degrees in bio, english and so on that got a job in data engineering. But thats because they usually had some experience in tech prior.
Hey @seattledataguy. Have you heard of MITxPRO Data Engineering bootcamp? Do you know anything about it? I can send you their brochure. Would be great if you can have a look and let us know if the curriculum is relevant. Thanks
