Top AWS Services A Data Engineer Should Know 

Thanks for the wonderful feedback! I'm glad the way I explained it was helpful. Thanks for subscribing! More AWS related content to come.

@@DataEngUncomplicated thank you Eagerly waiting for more content 😁

Thanks, I try to strike a balance with overview videos and technical tutorials on aws. Fun fact, I think you are my 5,000 subscriber!

@@DataEngUncomplicated 👏🥳🎉🎊

Thank you for the kind words Renz! I'm glad it was helpful.

hey there, could you give a link to a resource for data engineering? im about to start a job in DE and im kinda intimidated with the various skills needed for the job. I already know Python and SQL (which is why i was hired, or so im told) but i know nothing about DE. im about to start this udemy course on Python, SQL, and Pyspark, but im afraid it might not be enough. any help would be appreciated, thanks!

@@francismagnusson378 Hi, how did you proceed? How's your job going on?

Glad it was helpful!

Thank you very much!

Thanks so much for your kind words Saidulu. I really appreciate it.

Thanks for the feedback and subscribing!

Thank you! I'm glad the video was help.

Hi Adrian, Thanks for the feedback and subscribing! Can you elaborate on your suggestion? Are you thinking of an actual hands on tutorial or overview use case type video?

You're welcome. I'm glad you found the video helpful

Thanks Shrey, much appreciated!

Thanks for the comment! Yes, new videos related to data engineering and AWS every week!

Thanks Nic!

Thanks, Best of luck with your interview!

You're welcome Chris, I'm glad it was helpful. Good luck on your AWS certification!

Thanks Sushila, I'm glad you found it helpful!

Thanks John, yea I was thinking at some point to update this for 2024 or 2025. You are right, there are also new services like lake house architecture related data formats such as iceberg, delta or hudi that are now supported.

Thanks Victor, much appreciated!

You're welcome. I'm glad it was helpful!

Thanks Andraya!

Thanks Obie! Much appreciated

You're welcome! Thanks for subscribing!

Thank you Oluwatobi!

Glad it was helpful!

Thanks Dan!

Thanks Tiisetso! I'm glad this video was helpful. Thank you for leaving me a comment

Thanks for your kind words!

Thanks so much for the kind words! I'm glad it was helpful. Good luck on the exam!

Haha. You're welcome!

Hi Nazz, thank you for your kind words! Yes I plan on making a playlist that has technical tutorials on implementing each component so if you subscribe to my channel, you will get notified when those videos are released! Unfortunately I don't offer any courses at this time, I'm just focusing on making RU-vid videos to help data engineers on AWS!

You're welcome Sergio, thanks for leaving a comment.

Thanks Senthil, I'm glad it was helpful.

Thanks sail! I'm glad it was what you were looking for. What were you searching for on RU-vid exactly?

@@DataEngUncomplicated i am familiar with hadoop environment..i wanted to know how to do all of it in aws..now i know! thanks

You're welcome!

Thanks Andrew!

Thanks for your kind words John!

AWS glue appears to have salesforce connectors so that would be an option. I'm sure you could do it in lambda functions as well if your data is small enough as well

Hi Sandeep, yes, this is high on my video list! Thanks for the suggestion!

Great point, I know this service is being used more recently

Sure, I can provide some suggestions on how to start learning AWS services: 1. Start with the AWS Free Tier: AWS offers a free tier for many of its services, which allows you to explore and experiment with them without incurring any charges. This is a great way to get started and familiarize yourself with the AWS platform. 2.Take online courses and tutorials: AWS provides a wealth of resources for learning, including online courses, tutorials, and documentation. You can start with the AWS Training and Certification website, which provides a range of free and paid courses on various AWS services. 3. Join AWS user groups and forums: Joining user groups and forums can be a great way to learn from other AWS users and get answers to your questions. AWS provides an official forum, as well as many user groups around the world. 4. Practice with real-world scenarios: Once you have a basic understanding of AWS services, try to apply what you have learned to real-world scenarios. This will help you understand how the services work together and how they can be used to solve real-world problems. 5. Get certified: AWS offers a range of certifications for different roles and levels of expertise. Getting certified can be a great way to demonstrate your skills and knowledge to potential employers.

Thanks Larry!

Hi, it all depends on your use case for your data and access patterns. For example although you have 10 TB of data, it doesn't mean you are querying all 10 TB in every query and rather only doing queries on subsets of your data.

Thanks! Glad the videos are helpful. I hope your interview went well!

Thanks Jamison! Yes! This is important but there isn't a dedicated data lineage or governance service released yet in aws...datazone was announced at reinvent which should fill this gap hopefully

Hi Gabriel, thank you! Yea this is a great point, this could have been a service added to the orchestration component of the diagram. It's a good option but I don't think it's "better "necessarily since you it's another server you have to pay for the server to keep running 24/7 vs step functions and glue orchestration are serverless and only pay per x # of invocations.

@@DataEngUncomplicated Thanks for the answer and the great insight there. I guess going serveless is always the best option. But execution logs of both from glue orchestration and step functions are accessible in cloud trail?

The logs for glue and step functions are actually accessible in cloud watch logs.

Hey Ben, good call out on DMS. DMS is a good service for data engineers to learn if their focus is on data migration. For database changes, I have used both aws glue or lambda functions depending on the size of the data and building the delta logic in python.

Thanks Dare, Unfortunately i don't yet

Glue catalog works on databases as well as data lakes so you can define your redshift datasets in AWS glue to keep track of them

Hi Draco, It sounds like using the glue catalog would be a good choice to register your data in as it handles drifting schema. You can use a crawler to automatically scan and identify the changes in the schema

Yea you got it!

Thanks Dan, you are right, I left out security. I would throw up KMS in the security section if individuals wanted to encrypt their data with kms keys

Hi Suleiman, sorry I'm not as familiar with Azure services. AWS is what I am currently focus on.

When you say alternatives, do you mean AWS native alternatives? AWS announced an auto load feature from s3 to redshift I guess assuming that the schemas are the same.

@@DataEngUncomplicated Yes. For example: lambda and EMR before curated layer and after from curated to redshift and Athena do you recommend any aws service to load the data?

Great question Dave! I would recommend using lambda functions to ingest the data in S3. Glue is for processing large amounts of data and has a bit of a start up time. You will probably want a lambda function pulling data from your API frequently so the data load size would probably be relatively low.

Thanks, thats direction I went. I have a meta lambda, a datasource lambda (1 for each data source) and a s3 upload lambda. Using step functions. This way the meta lambda gets the customer infromation required, spawns parallel datasource lambdas which all pass data to s3 upload lambdas. ​ My only concern is how to best structure it in s3 for Glue. End goal here is Athena / Quicksite for BI purposes. I looked at AppFlow for some of this but hated it since I couldn't get all the object at one time and had to build an object per flow. So if a single data source has a lot of objects thats a lot of flows which seems annoying.

@@DataEngUncomplicated Thanks. I went with step fucntions. A master function that gets customer meta data and spawns functions for different datasources that all end up calling a s3 upload function. Now my only concern is am I storing the data properly in s3 for glue to make use of. Something like # Format the file name based on the current date and data type file_name = f"{data_type}_{year}-{month}-{day}.json" # Update the S3 key (path) to use the 'year=YYYY/month=MM/day=DD' partitioning convention s3_key = f"{customer_id}/year={year}/month={month}/day={day}/{data_type}/{file_name}"

@@DaveThomson I hope you figured this structure out by now, but if you want to use athena, you need to have your datasets seperated into different objects (folders) in S3. I would add a partitioning strategy as well which will save you in query costs if you know how your data will be queried.

@@DataEngUncomplicated Thanks!

Yes, I have done projects using most of these services in the past.

Hello, these are good questions that lots of people starting with AWS might have! Yes, AWS is just a cloud platform. I would say you should still have the foundational data analytics skillset in order to he succesful. You definitely don't need to specialize in 200 services to get a job. I would focus on learning the services that are relevant for a particular role. Nobody knows every single AWS service there is just too many. For your question about is AWS certification enough to get a job, it all depends on the role, the employer and what they are looking for. I would say it can't hurt your chances of getting a job if you are looking for a role that involves AWS.

No sorry, unfortunately I don't have one.

Are you talking about using Amazon Managed Streaming for Apache Kafka?

@@DataEngUncomplicated yes!

Its not much data. A max of a couple hundred lines but I still want to be able to show an employer I can use different services

I think your asking how to ingest data from an api into aws? There are many ways to do this but for your purpose you can write a lambda function that uses the requests library to read data from the API and use the python library aws data wrangler to write the data to s3.

@@DataEngUncomplicated Thats exactly what I was asking, thanks. Can you make sure this sounds correct though? 1. AWS lambda to ingest data from API call and write that data to an s3 bucket 2. Read data from s3 using Python notebook file (that is using PySpark package) or read data from s3 using AWS EMR

@@rememberthename911g yup this works, you might want to define your data source in a glue catalog table so it will be more easily ingested into a glue job or pyspark job.

Hey David, I'm actually a full-time AWS D&A consultant for a company that is an AWS partner. Let me know if you want to chat.

@@DataEngUncomplicated I would like to chat. I too work full time for a partner.

@@DaveThomson Great, feel free to contact me through the email I have posted on my channel.

@@DataEngUncomplicated sent you an email.

Hi Naveen! yea it is confusing because there isn't really a specific data engineering certification. The Developer associate and the AWS Data Analytics Specialty are the best one. I would also go after the database specialty if you think you will be working a lot with databases

I have no context to what this means but I'm going to respond with we can process millions or billions of records in data engineer with AWS 😉

Hello I know there are a lot of concepts and technologies to learn! you can reach me at dataenguncomplicated@gmail.com