export REGION=
------------------------------------------------------------------------------------------------------------------------
curl -LO raw.githubusercontent.com/QUICK-GCP-LAB/2-Minutes-Labs-Solutions/main/ETL%20Processing%20on%20Google%20Cloud%20Using%20Dataflow%20and%20BigQuery%20Python/gsp290.sh
sudo chmod +x gsp290.sh
./gsp290.sh
------------------------------------------------------------------------------------------------------------------------
pip install apache-beam[gcp]==2.24.0
cd dataflow/
python dataflow_python_examples/data_ingestion.py \
--project=$PROJECT --region=$REGION \
--runner=DataflowRunner \
--staging_location=gs://$PROJECT/test \
--temp_location gs://$PROJECT/test \
--input gs://$PROJECT/data_files/head_usa_names.csv \
--save_main_session
python dataflow_python_examples/data_transformation.py \
--project=$PROJECT \
--region=$REGION \
--runner=DataflowRunner \
--staging_location=gs://$PROJECT/test \
--temp_location gs://$PROJECT/test \
--input gs://$PROJECT/data_files/head_usa_names.csv \
--save_main_session
sed -i "s/values = \[x.decode('utf8') for x in csv_row\]/values = \[x for x in csv_row\]/" ./dataflow_python_examples/data_enrichment.py
python dataflow_python_examples/data_enrichment.py \
--project=$PROJECT \
--region=$REGION \
--runner=DataflowRunner \
--staging_location=gs://$PROJECT/test \
--temp_location gs://$PROJECT/test \
--input gs://$PROJECT/data_files/head_usa_names.csv \
--save_main_session
python dataflow_python_examples/data_lake_to_mart.py \
--worker_disk_type="compute.googleapis.com/projects//zones//diskTypes/pd-ssd" \
--max_num_workers=4 \
--project=$PROJECT \
--runner=DataflowRunner \
--staging_location=gs://$PROJECT/test \
--temp_location gs://$PROJECT/test \
--save_main_session \
--region=$REGION
Join this channel to get access to perks:
/ @guptasir2
Keep Supporting Us :-
Email : guptadivyansh023@gmail.com
Social Media Handles
***********************
Follow us on:
Telegram
t.me/StudyTime8
Get the latest deal now for any online shopping
earnkaro.com/?...
Daily get gift code
t.me/giftscode2
tinyurl.com/4t...
#gsp290
#thearcadechallenge
#googlecloudarcade
#arcade2024
#googlecloud2024
#qwiklabs
#googlecloudarcade2024
Disclaimer- The video is made solely for educational purposes and is not created with an intent to harm, injure or defame any person, body of persons, association, company or anyone. Details in the video like Home address, Email addresses, Sign-in credentials, Phone numbers, Passport number, Bank account information are demo do not belong to a real human. Some contents are used for educational purposes under fair use. Under Section 107 of the Copyright Act 1976, allowance is made for "fair use" for purposes such as criticism, comment, news reporting, teaching, scholarship, and research. Fair use is a use permitted by copyright statute that might otherwise be infringing. Non-profit, educational or personal use tips the balance in favor of fair use. All credit for copyright material used in the video goes to the respected owner. This video is not intended to spread rumours, offend, or hurt the sentiments of any religion, communities or individuals, or to bring disrepute to any person (living or dead). The viewer should always do their diligence and anyone who wishes to apply the ideas contained in the video takes full responsibility for it. Also, it is done at their own risk and consequences. The material contained in this video cannot replace or substitute for the services of trained professionals in any field including, but not limited to, financial, medical, psychological or legal matters.
11 окт 2024
