Hey, what is that "other" channel? I'm just about to turn into data fields. I've discovered you a couple weeks ago and found really interesting your videos. 😉
Super helpful! I was getting overwhelmed with the amount of tools I thought I had to learn. Now I'm just focusing on building my python and sql skills then going from there. Thanks as always for being a reliable source of data engineering related content!
Yeah! Focus on the basics. The rest comes, whether you want it too or not...I was literally walking someone through setting up a docker compose and docker file to spin up airflow, redis and postgres from memory...and I was like..why do I know this.
SQL, Python, Linux CLI, a text editor (I use sublime text all the time to make changes to files that I’m having formatting issues with if it’s a one-time thing), and then maybe node.js will get you really far.
lol, a of people working with data have this misconception that if business people write to them all the time they sometimes think that they like you. Unfortunately it's not the truth in most of the cases and those people just try to exploit and use you as a tool unfortunately (I was on both sides so I know it perfectly well). The easiest way to get over this and do the job that actually matters and is planned for you to do is bypass them to your manager and (even better) categorize it with manager and if something needs to be done quickly - do it without consulting him as it can have meaningful impact on the business side (which means money). But I think it comes with experience - unfortunately I'm seeing a lot of newcomers who are pushed so much from business people that out of nowhere they have 10 tasks to do and can't concentrate on the actual goals. They're trying hard to impress I assume and that's not necessary because your work will be evaluated by your manager, not some random guy from the company that you're helping out all the time.
If possible it would be great to hear more about your story going from BI to Data Engineering (I remember you mentioned this in one of your videos). I am in a similar place right now and would love to learn from your experience/steps you took. Otherwise, really great video, as always!
I end up having to use the Accounting Software as a source of truth to gauge precision, accuracy, and alignment. Believe it or not, sometimes Accountants get the data entry wrong (2020 instead of 2021 happens all the time). The human factor in the source of truth needs to always be considered.
Oh yes! Yeah I have seen issues like that. That's what anomaly detection is for...sometimes. Oh an accountant put in a -300k charge when they have never put in more than a -5k charge. Lets make sure thats right
There's so much to learn about data engineering; from programming skills, tooling knowledge, to interview prep. One could spend a wealth of time just on each.
@@SeattleDataGuy I'm a Storage guy currently working as a Service Account Manager handling tech and non tech tasks. I just have a wavering interest in Data Engineering. 😂 I use Excel to create performance reports, capacity reports and at times use readymade templates from within Tableau. I have done a bit of scripting and automation earlier.
Great video, biggest thing like you said is to focus on certain things and build upon that foundation, don’t get caught up on the newest technology because it’ll be an endless cycle.
Hey man I’m enjoying your channel, I think it’s quality content! Just a request.. would it be possible to get videos / series in the future in data engineering projects / walkthroughs, implementing various data engineering tools? I think you’d be great to make those type videos and it would be a huge value add! Thanks
@@SeattleDataGuy just saw your reply, appreciate you getting back to me! Can’t wait till your able to make it happen, thanks for making the content you do while balancing a life ! Take care!
Oh my man someone spoke the truth. Fellow mates , Yes don't accept every damm adhoc request 🙏 Also the crm thing happen with me recently , change in existing sfdc construct just made my sync pipeline useless.. & ideal data source doesn't exist🥲
We like to have two source of truths. One that just reads the data (no transformations) and other with transformations. Both reading independently from the logging pipeline. That way, we know there is no integrity loss due to any of the transformations that are performed on the data.
Yeah, I agree with this. Then you can usually run some QA and compare the two sources. In theory if you right a query that does similar work to your pipelines, but on the raw form of data..you should get the same output
Great tips! One other thing about data is how often the quality of the data is so bad. Many times I’ve been banging my head trying to get my query right, when it’s actually the data that is of bad quality.
"a self-service BI tool or a single source of truth as overused marketing terms"...was therapeutic to hear. It made me think it might be time we drop this unrealistic bar I find myself striving towards. What do you think the tech field is closer to achieving, the concepts of self-service BI tool/single data source of truth or having SQL as a standard competency for non-data professions (i.e. product, marketing, ops, management)?
Hey SeatleDataGuy, idk if U remeber, but we talked few moths ago about MS Azure certs. I did 3 of them so far, Az900 Dp900 and now Dp203. First two are entry level exams, general knowlege, however az900 is harder than you could inmagine for such a trivia. Dp900 was very easy, but still this is a fundamental exam, so kinda what you want tl expect from entry level. However, MS associate level exams can be a little bit of a pain: dp203 data engineer was a HARD exam, no walkover. A lot of data warehousing and processing questions, mkstly from azure synapse, and i think equaly emphase for Databricks related qustions. I think some experience from these two and free content from the internet should get You easily trough this exam.
Wow congrats on the certs! Also thanks for sharing what they were about. Its not only helpful to me but I am sure everyone else who will read this comment in the future!
@@SeattleDataGuy np! I am looking forward to do at least two more on Azure, at least one associate and one of expert level. I will share here how it was when ill get trough it 4 sure.
Generally, I try to figure out where the data is coming from, the logic that populates that table, if there are similar fields in other tables, duplicates, correct data types, and going forward, probably set some anomaly checks.
@@SeattleDataGuy checking data types and using trim for leading/trailing spaces, hidden characters, or zero-width characters should really be higher in the data audit methodology.
@Guy Thompson Hahahaha very true :). The hidden character one...always the worst. Whoops I was expecting a different kind of line break so I just inserted everything into one line.
I appreciate this kind of insight as someone who's more or less spearheading the burgeoning data culture at my work. It often feels like I have to learn all aspects of data at once, since we basically don't have governance or management, so I have to research and implement those principals myself. Do you have any recommended resources for simple frameworks that someone can use if they have to be a data manager, engineer, and analyst at the same time? Thanks for the video!
This will always be hard. The best place to start is to try to get buy-in from management. If you don't have stakeholder buy-in...its going to be an uphill battle. But after that you can check out DAMA-DMBOK: Data Management Body of Knowledge. This is kind of a broad view of data management. From there I would scale a strategy that works for you. It sounds like you have a small or mid sized company so you will need to pick your battles. What makes the most sense to focus on? What has the highest ROI? These are questions you will need to ask. Also probably limit the number of dashboards and insights you deploy. You want to avoid trying to support too much early on. Just focus on applying some basic concepts that you pick up from something like the DMBOK and once you have that set-up, continue to grow the whole data strategy.
@@SeattleDataGuy Thank you for the advice! I checked out DMBOK and ordered their Navigating the Labyrinth book for starters. I'll definitely focus more on choosing appropriate battles and being mindful of how much analysis I'm trying to support. Thanks!
Hi Ben! Thank you for your video! I'm from Brazil and looking forward to learn more about data engineering. Where can I learn more about using SFTP servers, as seen in the roadmap?
I get data engineer interview calls where they keep asking for spark, but i don't use spark in my current job(we use dataflow,cloud functions,Airflow, Bigquery, snowflake) for our ETLs. I was thinking should I start learning spark if that is something which is really needed for a Data engineer. What are your thoughts?
It depends, do you want to work for these companies? If so, then yeah. If not, then maybe just have a high level understanding. The truth is, in 10 years, the DE tools we work with will likely change...I mean it might even be 5 years or 2. So if you don't want to work with the companies, then you might find that in a few years that tool isn't the cool thing to use anymore.
Great video! Just as you mentioned at 8:24 I am at that stage currently. Got myself an opportunity to work as a Data Engineer recently which starts after my 4 months internship. Among projects, Data tools and Courses, what would you suggest me to do in the mean time to get ready for the overwhelming experience? 😬
Oh the is an amazing stage. Everything works and it takes like 2 days to put up some piece of code that takes 3 weeks in a company. I would say practice interviews...sadly interviews are different than work..but you have to interview to get work.
Non related Question:- I am curious to know how do you track/monitor 100's of data pipelines whether they are completed success/failed or meeting the sla or not ? I am looking for a solution inputs for this problem.
How are you currently managing this? Part of this can be managed by having at least a few DEs to manage hundreds of pipelines because at a certain point regardless the tool, you will need to break up some of the work. As far as tools. Airflow has a good UI that can be used to see failures and track complex DAGs. It doesn't have SLAs per say. But, it does have all that data stored so you can see runtimes and landing times and do some analysis from there. Plus I am sure there are a few plugins you can add in to do that automatically.
Hi SDG, questions on data modeling. I am a new data engineer and wondering where I can improve my data modeling skills? I know a lot of people mention Ralph Kimball's book, do we have anything else? Like where I can get database design from companies like uber/Facebook?
Maybe their engineering blogs? Even though I don't think they talk much about data modeling. Sidenote, in Facebook's interview they will likely ask questions about data modeling that can often best be solved with Kimball. So that's a great place to start. Also, Data Vaults are making a resurgence here and there. So maybe read up on that. hanshultgren.files.wordpress.com/2012/09/data-vault-modeling-guide.pdf
frati ma! Hahaha, my parents both moved from Romania to the US in the 1980s. Prior to the wall falling and the whole Christmas thing. My Romanian is terrible but I can understand a decent amount. You will also notice that the Funko pop in my background is Florian Munteanu i.e. Razor Fist.
@@SeattleDataGuy Haha, didn't expected that to be honest :D. I am really happy to know this because since I decided to try a career switch from SAP consulting to Data Engineering I found your channel and your advices and explanations are very useful. Now, knowing that you have Romanian roots, I will have an even bigger smile on my face when you will upload a new video!
There are masters in DE! I haven't seen it. I think most engineering skills don't need a masters degree. For us its more about experience. Unless you're coming from a non technical background and looking to break into tech
@@SeattleDataGuy it's northeastern uni in Boston, i hope you've heard the name 😂😅 I agree with your point but breaking into data engineering as a fresher is tough
hello sir, i am on my 2nd year in college, i wanna be a DE after grad, still learning python and sql, can u give me some tips or topics i should learn after learning the basics? cuz i dont know what to do or to study next after learning the basics. thankyou in advance :)
Hello! I have a video on a data engineering roadmap. This should help answer what you should study next. ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-SpaFPPByOhM.html
This is a good point. There are so many pitfalls. I always give the example to people of trying to develop a data warehouse for doordash and if you're too early in your knowledge of data warehouses you might develop an order fact table, some form of menu item dimension table and a restaurants dimension table...Now I ask the question which items from what restaurants aren't being ordered and you have no way of answering this question because the relationship is stored on the fact table.