Chapter #9 - How to design data pipeline on gcp (Google Cloud Platform) ?

Подписаться 427 тыс.

Просмотров 100 тыс.

50% 1

Chapter #9 - Designing data pipeline solution on GCP
#datapipeline #googlecloud #gcp
What is Data Pipeline? - • Chapter #8 - Cloud IAM...
**Google Cloud Platform Beginner Series**
Google Cloud Platform Beginner Series (2020) - • Google Cloud Platform ...
**Networking Basics Playlist**
Networking and Infra Concepts - • Networking & Infra Con...
** Other Popular Playlists**
Crunching Data Series (2020) - • Learn - Data Engineeri...
Latest technology tutorial (2020) -
• What is a Data Vault ?...
Hi Friends, I am Anshul Tiwari, and welcome to our youtube channel IT k Funde.
More about this video -
In this video, we will learn something different, instead of discussing any specific product on google cloud platform, we will learn how to design a data pipeline solution on google cloud platform.
This will help us in 2 ways -
1. This will helps us improve our design thinking
2. We will get an overview of all google cloud products that can be used in developing a data pipeline.
We will start with understanding a typical on premise data pipeline and enterprise data warehouse (edw) design. We will then take this same solution and try to implement it on google cloud platform.
On-Prem solution design contains below products -
#Streaming data - Apache Kafka
#ETL - Informatica, IBM Datastage, SAP Data Services
#EDW - Teradata
#Dashbaords & reporting - Tableau and SAP Business Objects
GCP cloud design contains below products -
#Streaming data - Cloud PubSub
#ETL - Cloud Data Fusion
#Storage - Google Cloud Storage
#EDW - Google Big Query
#Dashboards - Looker, Google Data Studio
#ML - AutoML
#Metadata Management - Data Catalog
PLEASE WATCH ALL THE VIDEOS IN ANY PLAYLIST FOR BETTER UNDERSTANDING !!
Credits & Resources for further study -
GCP resources documentation for detailed study -
#Cloud PubSub - cloud.google.com/pubsub/docs/...
#Cloud Data Fusion - cloud.google.com/data-fusion/...
#Google Cloud Storage - cloud.google.com/storage/docs...
#Google Big Query- cloud.google.com/bigquery/doc...
#Data Studio - cloud.google.com/bi-engine/do...
#Looker - docs.looker.com/
#AutoML - cloud.google.com/automl/docs
#Data Catalog - cloud.google.com/data-catalog...
images - pixabay.com
**Social Links**
RU-vid - / itkfunde
Facebook - / itkfunde
Linkedin - / ansh9685
Twitter - / ansh9685
Instagram - / itkfunde
**About This Channel**
Friends ITkFUNDE channel wants to bring I.T. industry knowledge, information, career advice, and much more to every individual regardless of whether he or she belongs to I.T or not. This channel is for everyone interested in learning something new!

Опубликовано:

30 мар 2021

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 149

@sanjayroberts3453 2 года назад

Incredibly helpful -- interviewing for senior level data engineering positions and many have data pipeline design sections. This is perfect for that

@metaocloudstudio2221 2 года назад

As a Data Engineer who worked on Cloud solutions for 7 years, I have never seen such a correct and clean solution. I am interested to see the same solution on AWS as well.

@antonioflores4240 2 года назад

I'm trying to build a data lake for an undergrad cloud class and honestly I was feeling pretty hopeless but after watching your video I feel way more prepared to tackle this!!!! Thank you!!!

@ITkFunde 2 года назад

Thanks Antonio I am so happy it helped in some little way 😊

@ankursahu4731 Год назад

This is one of the great video to make me understand how Project architecture is there in GCP, I would request to please create a series to build end to end demo project in GCP utilizing all these services.

@AmeyChittar 3 года назад

Hi. So glad for posting this. I was going through the coursera course for this and I never understood the big picture. It's so clear now. Thank you so much.

@ITkFunde 3 года назад

Thanks Amey so happy it helped 🙏

@hellmutmatheus2626 2 года назад

I really learned a lot through your videos. Thank you very much, cheers from Brazil!

@chehz 3 года назад

Thank you very much for all your hard work and the nice lecture.

@jackleung2519 2 года назад

really enjoy watching your video, very clear and well-explained.

@ChoandCho 2 года назад

Thank you so much for the hard work you have taken for making these concepts very easy to understand.. Great work.. Keep posting such informative videos

@maayanyacobi8665 2 года назад

great lecture, well explained - Thank you

@rushabhparmar7519 2 года назад

Very informative and detailed! Thank you!

@hemanth1262 3 месяца назад

Very informative to the beginners of cloud but helped with on-prem solutions, thanks tons!!!

@naginafathima4562 2 года назад

Great explanation. Can you also do a video on how the traditional DW system is used in AWS and AZURE?

@sanjayg2686 Год назад

Sooooper Simple, you made sir, Thanks a lot

@ITkFunde Год назад

You are most welcome

@hareeshgunda8371 2 года назад

Great work. We expect some realtime project design on GCP.

@joannwatu7603 2 года назад

Thank you for sharing. Your explanation of the data lake structure for both on-prem and cloud solutions was very clear and easy to follow.

@ITkFunde 2 года назад

Thanks Joan ☺️🙏

@Poornima_life Год назад

It’s perfect explanation….you are genius man

@gauravjha7249 2 года назад

Very nice ......thanks for enlightning with so many information

@LeonidAndrianov Год назад

Thank you, mate! I like your optimism as well as the explanation.

@ITkFunde Год назад

Thanks Leonid🙏

@pankajmanik7789 2 года назад

This video is too good !! So many topics covered crisply in 1 video ... Thank you very much Sir :)

@ITkFunde 2 года назад

Thanks Pankaj ☺️🙏

@gauthamvijayan 2 года назад

Phenomenal Content.

@prasann26 2 года назад

Very nicely explained. All the best. thank you!

@prithvinoojibail3734 3 года назад

I am glad you have tutorials in English (since I don't speak or understand Hindi). Thanks for this video!

@ITkFunde 3 года назад

Thanks Prithvi

@anandakumarsanthinathan4740 2 года назад

Absolutely, absolutely wonderful and insightful presentation. Thank you very much. Just an observation. With BigQuery as your data warehouse, companies need not go for datamarts any more. BigQuery being a scalable solution, personnel from the various departments such as Sales, HR, Marketing, Finance can run their queries against a single database and still get results back without compromising on performance. Security can be implemented at the user/department level.

@ITkFunde 2 года назад

Thanks Ananda I cant agree more, BQ could be a one stop solution for vatious diff use cases

@victorganguly4230 2 года назад

Nice explaination. 👍 Want to have some more videos like this for real time data processing with Google cloud (for example IOT data,sensor data)

@SamvithDevi 3 года назад

This has been very helpful. Thanks a lot for the video, and your detailing.

@ITkFunde 3 года назад

Thanks a lot🙏☺️

@yshah9042 Год назад

Excellent.

@aneyewitness Год назад

Awesome video - thank you! 👍 Would love to see an equivalent video for AWS and Azure also.

@rohanasnani4768 3 месяца назад

Loved it, Thank you soo much for this!

@pankajchaudhari95 3 года назад

Thanks, Amazing video and very helpful

@MubeenEbrahim-gk5xh Год назад

Thank you. Very useful video.

@siamakfarjami2116 Год назад

Thank you very much. Great video.

@RizalIsmail 2 года назад

Really great video. So humble you are and you give good first impressions at 1:25

@ITkFunde 2 года назад

Thanks Rizal ☺️🙏

@thammayagupta Год назад

Awesome Explanation.....

@swarupdeshpande4759 5 месяцев назад

Thanks. it was good presentation. To the point.

@vanib6123 7 месяцев назад

Very Informative !! Thank you for the great efforts !!

@manuelbolivar9791 3 года назад

Thanks for posting and thanks for your hard working!! I'd like to see, whenever possible, similar videos for AWS and Azure cloud offerings. Using the same left side starting point. Cheers!

@ambar752 3 года назад

Very well explained, loved it !!

@ITkFunde 3 года назад

Thanks Ambar🙏🙏☺️☺️

@sanaayakurup5453 Год назад

brilliant brilliant explaination!keep it up!

@shameemferdous Год назад

Amazing !

@sreetenap 2 года назад

Thx a bunch! Excellent explanation.

@ITkFunde 2 года назад

Thanks Sreedhar☺️

@learner9670 Год назад

Too good learning material!

@bhalachandrapatil Год назад

excellent

@vardannegi 3 года назад

For data transformation and enrichment in GCP, Dataprep is the best tool

@ITkFunde 3 года назад

Yes Vardan Dataprep is good tool

@meeral8703 2 года назад

You are amazing!! Thanks for the clear video

@ITkFunde 2 года назад

Thanks Meera ☺️

@mainajnabee 2 года назад

Profound knowledge you are sharing with us. Thank you so much! One request: whenever you refer your other videos, can you please display them in those bubbles in a corner so that we can click on them and go to those videos? You know those small bubbles with link that pop up..

@lifeofpayal972 2 года назад

Thanks for the video..!

@ramkumarsundaravel8301 6 месяцев назад

Really good video and explained very well.

@informationsatellite5155 Год назад

great content..thank you so much...

@hailstorm7868 Год назад

3rd party -> gcs stage can also be done via arbitrary code inside a container on "cloud run jobs". I did that, and downstream what is shown in the video almost exactly on one of my freelance projects.

@syedaliuddin7046 3 года назад

Thank you for sharing the information really helpful

@ITkFunde 3 года назад

Thanks Syed

@siddeshsharma968 Год назад

Really Great video, been looking for such explaination

@felipeozoski 3 дня назад

Great stuff thank you

@JOHNSMITH-ve3rq 3 года назад

absolutely amazing video!!!!!!!!!!!!!!!!!!

@ITkFunde 3 года назад

Thanks John

@shubhamsinnarkar2923 3 года назад

AMAZING VIDEO!

@SeafireCH Год назад

Very nice video, great overview that is very helpful in navigating the jungle of google tools. Thanks a lot!

@ITkFunde Год назад

Thanks Pesche

@MoisesTrelles 2 года назад

Great explanation, very well presented

@ITkFunde 2 года назад

Thanks Moises ☺️🙏

@samirchandra2637 2 года назад

Very informative boss, keep going

@moushmidas2584 3 года назад

Thanku sir As i requested you really created a video❤️❤️🙏 so nice video...

@ITkFunde 3 года назад

Thanks Moushmi hope you learnt something new from it pls keep sharing ur suggestions

@59600muslim 2 года назад

Très intéressant merci

@dr.kumaraswamybattula8428 6 месяцев назад

Thank you ❤

@swatitraveltales 3 года назад

Reallly Helpful videos! An aadvanced video on Bigquery & one on Cloud data fusion too:). Thanks for sharing these videos

@ITkFunde 2 года назад

Thanks Swati sure will try ☺️

@vamsikri1234 2 года назад

Really good video and good explanation learned a lot. I just wanted to ask one thing ,the cloud model you showed is it conventional ETL flow or its more of ELT with subset of ETL , because here we are using both the flow or its some kind of new approach which we can use, if would be really good if you could share your thoughts on this

@yuvabhagvatkathakarshriyag9305 3 года назад

this is indeed a helpful video

@ITkFunde 3 года назад

Thanks a lot

@oguzhangunes 2 года назад

thank you !

@priteshraka6408 3 года назад

Thanks for this, I was waiting for one such video from long time.

@ITkFunde 3 года назад

Thanks Pritesh🙏

@khanbabababa6396 3 года назад

Okay

@emmanuelihetu9848 2 месяца назад

This is soo good

@Rise_Citizens 2 года назад

You might make my next job a bigger paycheck,nice video

@ITkFunde 2 года назад

Thanks buddy

@Pkmafffy Год назад

Great video! I really appreciate the effort you put into this video and it really helped me out. Thanks a lot1

@praveengarg6090 Год назад

Great

@CPat-yt6dm 10 месяцев назад

Thanks

@aamirsuleman9815 10 месяцев назад

Dataform is another amazing tool for ELT jobs

@pradeepdewani6267 3 года назад

This is really helpful ..

@ITkFunde 3 года назад

Thanks Pradeep

@TradeWithCodeOfficial 3 года назад

very helpful video.. one question.. do we need to create a VM instance if we go by this approach for GCP. Where do we trigger the gsutil command ? How do we go to the backend of fusion instance?

@capooti 2 года назад

Nice intro! well done

@ITkFunde 2 года назад

Thanks a lot Roland

@sanjayg2686 Год назад

Thanks a lot

@ITkFunde Год назад

Most welcome

@venkatanithish7757 2 года назад

Good one :) :D

@manpreetsohal11 2 года назад

Informative video. Why haven't you used cloud dataflow for ETL? Can cloud data fusion perform better ETL tasks than cloud dataflow?

@rdhundare 3 года назад

Gud explanation

@ITkFunde 3 года назад

Thanks Rahul

@user-kk4yp8vd4g 8 месяцев назад

In your architecture where is the pub sub messages are saving is in cloud storage and then etl pupeline pick and load in to big query?

@VinodKumar-wc6bu Год назад

It's really excellent with kind of explanation . Sir do you have gcp pde paid course as well,plz comment

@meeral8703 2 года назад

Question sir: on the Google solution, why do you have two separate Cloud Storage components?

@anandnerurkar8482 Год назад

good one. what is datamart, for datamart you have been using data studio,automl,looker? can u pls explain why??

@hemantfegde460 Год назад

Good explanation Sir, could you please make video on Streamsets pipeline?

@ArjunSingh-fm9sm Год назад

Hey Thanks for this video, I have 1 question. If we want to do performance test for this type on msging queue application, what should be our approach.?? and how we can do this.. plz

@ankitajain6817 3 года назад

Very very informative and useful video.Thanks for sharing !

@ITkFunde 3 года назад

Thanks Ankita 😊

@amitjaiswal781 Год назад

Please make a video real time projects in gcp

@kushagrak4903 3 года назад

Sir really amazing video ! I was wondering where can I learn basics of the software which you mentioned ( SAP , IBM ) ?

@ITkFunde 2 года назад

Thanks youtube has everything for free or else try udemy for reasonable courses

@pankajsangle2752 6 месяцев назад

om shree ganpatye namo:

@RakeshGupta23 2 года назад

Great explanation..one request .. please use mic 🎙️ as voice is Little low as compared to other educational videos.... great job..

@ITkFunde 2 года назад

Sure Rakesh thx for feedback and support 🙏

@sathishgowda4272 2 года назад

Very good explanation, please make some videos related to complete data engineer project flow with pyspark or SQL

@ITkFunde 2 года назад

Thanks Sathish ☺

@pawnyogi 2 года назад

@@ITkFunde hi , is this is roadmap for data engineer using google cloud ?

@zudt 8 месяцев назад

just G.R.E.A.T.

@sunilsahoo5199 Год назад

can you do an end to end project in google cloud using this services.Then it will good for us.A lot of thanks to you for giving this type of content.

@ITkFunde Год назад

thanks for suggestion

@rushipuneet1270 2 года назад

Is there any practical video creating a data pipeline?

@nilavasen8631 Год назад

Dear Anshul, how are you ? I have been working as Data Engineer in on-prem solution of our company for more than 10 years and now wish to move to Cloud Based solution, like GCP Data Engineer. Can you please do help me with the below queries I have :- 1. I am having total 15+ years of IT Experience. Will it be a good decision to switch to Cloud Data Engineer Roles now ? It seems to me bit late to start journey in this domain now. Please do correct me , if I am wrong. 2. In order to start / work with the GCP Data Engineering Role, can you kindly suggest me what are the topics / tools / GCP Services I need to learn ? I have basic ETL Concept and also working with PLSQL , Python stuff. Thanks in advance !!

@ITkFunde Год назад

Hi Nilava, 15+ years of exp is quite a lot, in todays age tech roles are defying age and exp boundaries but in indian IT your total exp and respective role does put an image on your career profile thus i would suggest you move towards cloud data architect sort of roles or solution architects. Also if you starting from a clean slate then you should also think of becoming a Data Lead or Manager and manage a team of Data engineers under you. Think big !

@nilavasen8631 Год назад

@@ITkFunde yes very true Anshul.. thats what I was thinking also. 🙂

@SancheeKaushik Год назад

what can be the most cost efficient way to ingest a few tables from snowflake to GCP bigQuery

@SancheeKaushik Год назад

also is it possible to do Export and then transform before saving to bigQuery instead to do ELT as explained in the video. if yes then will that be a good option which is cost efficient.

@krish_telugu 2 года назад

We are facing problems with almost all services that we have been using with GCP, very limited plugins and features serve to basic use cases. even connectivity between services is also troublesome. long way to go

@Kunal4980 2 года назад

Hi I see that your videos are getting older and platforms like GCP changes their platforms very frequently, could you please update your videos to sync up or upload delta videos to complement older ones up to the latest ones...