No video :(

Automating Data Loading from Google Cloud Storage to BigQuery using Cloud Function and Dataflow

Подписаться 7 тыс.

Просмотров 9 тыс.

50% 1

Looking to get in touch?
Drop me a line at vishal.bulbule@gmail.com, or schedule a meeting using the provided link topmate.io/vis... Automating Data Loading from Google Cloud Storage to BigQuery
In this video, I showcase the process of automating the transfer of data from Google Cloud Storage to BigQuery. The automation is achieved through the utilization of Cloud Functions, demonstrating step-by-step how this setup streamlines and orchestrates the data loading process seamlessly between these Google Cloud services.
Associate Cloud Engineer -Complete Free Course
• Associate Cloud Engine...
Google Cloud Data Engineer Certification Course
• Google Cloud Data Engi...
Google Cloud Platform(GCP) Tutorials
• Google Cloud Platform(...
Generative AI
• Generative AI
Getting Started with Duet AI
• Getting started with D...
Google Cloud Projects
• Google Cloud Projects
Python For GCP
• Python for GCP
Terraform Tutorials
• Terraform Associate C...
Linkedin
/ vishal-bulbule
Medium Blog
/ vishalbulbule
Github
Source Code
github.com/vis...
Email - vishal.bulbule@techtrapture.com
#gcs #googlecloud #cloudstorage #bigquery #datapipeline #automation #cloudfunction

Опубликовано:

5 сен 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 38

@abhaybulbule6557 8 месяцев назад

Proud to see that among of us ( software engineer) achieving their goals by dedication and commitment Great congratulations whatever you achieved in your life

@alanguev 5 месяцев назад

You're the best, brother. The information was really helpful, I appreciate it a lot. Greetings from Argentina, Buenos Aires.

@techtrapture 5 месяцев назад

Thank you brother ❤️🔥

@Ranjana_DE 5 месяцев назад

I have been looking for this content for a long time, thank god I found this, very useful

@amritapattnaik3345 7 месяцев назад

i loved all your videos.Keep posting 😇🙂🙃

@techtrapture 7 месяцев назад

Thanks

@noolusireesha205 19 дней назад

Sir, i have done the same process as you have menctioned in the video ....i'm getting error "java.lang.RuntimeException: Failed to serialize json to table row" . Could you please reply me with the solution.

@vignesh004 2 дня назад

even i'm getting the same error

@nitinhbk 4 месяца назад

Thank you. Really helpful session.

@faroozrimaaz7092 8 месяцев назад

Your videos are informative..keep going

@earthlydope 3 месяца назад

There's a catch here, we need to create BQ-Table-Schema and UDF.js file everytime before uploading a new flat file into the system.

@user-hx8hq4st3r Месяц назад

You could do away with dataflow here. A simple python job using load_table_from_uri with auto schema detect enabled from trigger function would do this work.

@techtrapture Месяц назад

Yes , single python would work definitely. This is to learn different services in GCP.

@zzzmd11 5 месяцев назад

Hi, Thanks for the great informative video. can you explain the flow if the data source is from a Rest API. Can we have a dataflow configured to extract from a Rest API to big query with dataflow without having cloud functions or Apache beam scripts involved? Thanks a lot in advance..

@hunterajones 5 месяцев назад

did the schema originally fail since the headers would not be an integer? With headers removed the original scheme would work right? Also, is there a way to automate header row removal? I need to daily auto add a CSV like this but it will always have a header row needing removed. Thanks for the video!!

@nitinhbk 4 месяца назад

Could you please let me know what was the COST shown in GCP for this activity?

@user-zj3yx8rk3u 6 месяцев назад

Very good Video. Where can I get more cloud function templates ??

@arerahul 7 месяцев назад

Insightful video. Just a question - Cant we write the data load job in cloud functions, rather than using dataflow ? Also how do we create a delete job - data is deleted whenever the file is deleted from GCS

@techtrapture 7 месяцев назад

Yes we can write everything in python and put in cloud function or composer. 2nd question - here you need to add something to identify which data is loaded by your file so your code can delete that data only.

@python_code08 2 месяца назад

Can we add this project in resume as a mini-project

@vinnakollurakesh8481 7 месяцев назад

Hi sir can you help me to pull the data from Kinaxis rapid response API to GCS, any related documentation or videos will be helpful, thanks

@mulshiwaters5312 3 месяца назад

This is what exactly I need however instead of Trigger I would like to use Schedular with certain time interval like Daily Weekly . How can I achieve this.. Cloud COmposer ?Workflow ? Schedular

@techtrapture 3 месяца назад

In scheduler you can use cronjob expression to mention date and time at which you need to trigger job

@mulshiwaters5312 2 месяца назад

@@techtrapture Thanks Appreciate your help on this !

@srikarfarmacy 2 месяца назад

thank you for the video, i have one doubt , if my csv file have header then do i need to have JSON code for schema?

@techtrapture 2 месяца назад

Yes, dataflow job ask for mandatory JSON file

@srikarfarmacy 2 месяца назад

@@techtrapture Thank you for your prompt response. Could you provide a solution for this issue? Every day, my bucket is automatically uploaded with data that contains headers organized by date.

@subhashs5275 3 месяца назад

Which location was template path in Python file?

@joshhicks2444 Месяц назад

Subscribing!

@ayush10_08 8 месяцев назад

Hello sir, I watched your lot of videos related to function , dataflow I have one question As a GCP Data engineer who is responsible for writing code for dataflow or for data fusion?

@techtrapture 8 месяцев назад

Data fusion is a code free ETL tool. But as a general data engineer is responsible for writing all code for the data pipeline.

@ayush10_08 8 месяцев назад

@@techtrapturemeans having a knowledge of only data related services is not important we have to learn coding ?

@techtrapture 8 месяцев назад

@@ayush10_08 yes for data engineer role we need coding