Vertex AI Pipelines - The Easiest Way to Run ML Pipelines

Подписаться 1,9 тыс.

Просмотров 17 тыс.

50% 1

Google Vertex AI: The Easiest Way to Run ML Pipelines
📓 Notebook: colab.research.google.com/dri...
📖 Article: / google-vertex-ai-the-e...
If you enjoyed this video, please subscribe to the channel ❤️
🎉 Subscribe for Article and Video Updates!
/ subscribe
/ membership
You can find me here:
LinkedIn: / saschaheyer
Twitter: / heyersascha
If you or your company is looking for advice on the cloud or ML, check out the company I work for.
www.doit.com/
We offer consulting, workshops, and training at zero cost. Imagine an extension for your team without additional costs.
#vertexai #googlecloud #machinelearning #mlengineer #doit
▬ My current recording equipment ▬▬▬▬▬▬▬▬
► Camera for recording and streaming in 4K amzn.to/3QQzwiN
► Lens with nice background blur amzn.to/3dVDAjb
► Connect the camera to PC 4K amzn.to/3ciYyrE
► Light amzn.to/3Rb065M
► Most flexible way to mount your camera + mic amzn.to/3TedZC5
► Microphone (I love it) amzn.to/3QV3mmb
► Audio Interface amzn.to/3CBxj5M
Support my channel if you buy with those links on Amazon
▬ Timestamps ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
00:00 Introduction to Vertex AI Pipelines
00:41 Statement
00:55 Vertex AI and Kubeflow
02:00 Basic Pipeline
04:08 Basic Pipeline Code
09:05 Machine Types
09:57 Components
12:00 Share Components
12:26 Predefined Components
13:50 Parameter and Artifacts
14:50 Model Lineage
15:29 Parameterized Pipelines
15:53 Base Image
16:39 Additional Packages
17:03 Monitor Ressource Consumption
17:19 Conditions
17:42 Compare Runs
18:12 Graveyard
18:33 Permissions
19:29 Client Libraries
20:06 API and SDK
20:39 Pricing
20:56 Bye

Наука

Опубликовано:

16 июл 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 68

@razalminhas6349 Год назад

This video should be first in search results when searching for Vertex AI pipelines. Thanks for making it!

@jobiquirobi123 2 года назад

Thank you Sascha. Great video to start learning more about all the features in Vertex AI. Keep the good work!

@Pirake123 2 месяца назад

This is better than the GCP videos, amazing thankyou!!

@ml-engineer 2 месяца назад

Thank you Pirake

@fredericmolina6890 2 года назад

Awesome video ! Thanks

@ml-engineer 2 года назад

Thanks Fréderic

@zbynekba 6 месяцев назад

Sascha, you can significantly enhance the intelligibility of your presentation by improving the audio quality. The distracting sound reflections from your office walls make listening stressful. The easiest no-cost remedy is close-miking, such as using a headset microphone for recording. Alternatively, if you prefer speaking to a distant microphone during recording, you could consider some acoustic treatment for your office space.

@ml-engineer 5 месяцев назад

great feedback. currently testing different setups to improve it.

@miguelalba2106 Год назад

Great video! I really like your channel, everything is super clear

@ml-engineer Год назад

Thank you Miguel Any other ML related topics that you are interested in?

@miguelalba2106 Год назад

@@ml-engineert would be very nice a video explaining how to run components that were already contenarized and then run using dsl.ContainerSpec or how to do CI/CD for a vertex AI pipeline

@ml-engineer 9 месяцев назад

I wrote an article quite some time ago about CI/CD for Vertex AI Pipelines medium.com/google-cloud/how-to-implement-ci-cd-for-your-vertex-ai-pipeline-27963bead8bd

@ronaldboodram6466 Год назад

Excellent video

@eliegakuba 2 года назад

Thank you so much for the video. it is well explained and very helpful. I think one thing could be notably mentioned is that the introduction of artifacts as parameter was to make it easier working with gcsfuse as the artifacts path points to the mounted folder instead of the actual location in GCS. Also if possible can you make a video explaining improvement that kfp v2 brings compared to kfp v1? thanks.

@ml-engineer 2 года назад

True the switch to artifacts as a reference path helped to also introduce the concept of ML Metadata.

@souravthakur6222 Год назад

Thank you ! Please share more end to end ML projects using Vertex AI pipelines plz

@ml-engineer Год назад

Hi Google has a great list of examples on their GitHub repository github.com/GoogleCloudPlatform/vertex-ai-samples/tree/main/notebooks/official/pipelines Check it out almost all of them are end to end examples.

@ml-engineer Год назад

Just recently I released a new video including a end to end pipeline to create Recommendations.

@irfandogic9579 2 года назад

Hi Sascha! First of all thank you for the great explanation and source code. I am using Vertex AI and want to automate our ML process using Pipelines. I‘ve followed yout „basic pipelines“ code and it worked. My question is: I have seen everywhere that when installing kfp, aiplatform and pipeline-components it should be installed with -USER, but in your example is working without it (and in my vertex project also). Do I still need to install it with -USER or I can just use it without? Regards, Irfan

@ml-engineer 2 года назад

Hi Irfan --user you only need if you don't have root access to install the packages. When ever you get access errors try to add --user.

@o_o610 10 месяцев назад

Thank you so much for the video ! Do you know if Vertex AI Pipeline handle Pipeline versioning or historize the evolution of the pipeline ?

@ml-engineer 10 месяцев назад

I always recommend to put your pipeline code into git. This way you have the perfect pipeline version over time available. Is that what you meant with versioning?

@LucasGomide Год назад

Hey dude, I have one more question. In my context, the search must be filtered by UserID in order to avoid returning results from another user. What's the best approach to do that, creating an index for each user? By using MatchingEngine Filters?

@ml-engineer Год назад

Hi Lucas I guess you are referring to one of those videos? ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-inAY6M6UUkk.html ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-KMTApM5ajAw.html No need to build an index for each user that would be way to expensive and you would reach the number of allowed indexes probably pretty quickly. The best solution is the built in filtering that Matching Engine is providing. This features is meant for exactly those use case like yours. cloud.google.com/vertex-ai/docs/matching-engine/filtering Have a good day

@raharijaonazolalainayannic7851 2 года назад

Great video Sascha, Is it easy to manage autoscaling with VertexAI?

@ml-engineer 2 года назад

Vertex AI pipelines do not support autoscaling. But if you want to autoscale your deployed models for serving that is possible.

@ml-engineer 2 года назад

Thanks

@raharijaonazolalainayannic7851 2 года назад

When you write your pipeline on top of kubeflow cluster does it support the autoscaling?

@ml-engineer 2 года назад

No autoscaling for self managed KFP on GKE either. You define CPU and memory needed.

@gokulramasamy8361 Год назад

Thank you Sascha for the video. Is it possible to apply unit test for Vertex AI pipeline? If yes, Can you give me a suggestion, how to do?

@ml-engineer Год назад

Not as straightforward as it could be. It's best if you think of each component as a simple python function. This way you can abstract away some of the unnecessary parts that are not required to be tested. The part that should be tested is your python code for each of the components. You can create a component from a simple python function by using create_component_from_func

@Smart-ls6xi 8 месяцев назад

Hello, I have a question. If I am working with a team, is it one person who is supposed to have a vertexAI account that will be charged? Or will each user, though sharing the same project, be charged in their account?

@ml-engineer 7 месяцев назад

Hi One person need to register on Google Cloud and create a project. This person can invite additional people to the project. All user that are invited will share the same project and the project get charged. For each project you have a billing account that uses a credit card for payments.

@kadapa-rl6jg Год назад

Hi, I saw your medium post where you are reffering to cloud composer when you are using cloud run as your personal note. Can you please share clarification on why you are advising cloud composer for cloudrun jobs

@ml-engineer Год назад

Hi many companies use Cloud Composer for processing heavy workloads. From my experience this can lead to a lot of challenges. By design Composer is a Orchestration tool and only meant for orchestration. That's why I recommend to offload processing heavy workloads to Cloud Run or Cloud Dataflow. If you don't need a orchestration tool and simply want to run a few Cloud Run Jobs you don't need Composer. Let me know if that helps answer your question =)

@kadapa-rl6jg Год назад

@@ml-engineer can you please let me know if there are any documents or medium note or any blog on how to work with cloud composer.

@ml-engineer Год назад

@@kadapa-rl6jg I am a big fan of the Google documentation cloud.google.com/composer/docs/tutorials Neil also has a few very good articles around Cloud Composer medium.com/@kolban1

@kadapa-rl6jg Год назад

@@ml-engineer thanks for the information I shall go through it to understand this

@geeglu Год назад

Error importing aiplatform Tried following the vertex ai documentation and while running: from google_cloud_pipeline_components import aiplatform as gcc_aip I get an error: Import error: Cannot import name '_dynamic' from 'kfp.components' (/opt/conda/lib/python3.10/site_packages/kfp/components/init.py) Any suggestions to resolve this error?

@ml-engineer Год назад

Hi Geeglu aiplatform is not part of the google_cloud_pipeline_components package. For importing AI Platform you need to use pypi.org/project/google-cloud-aiplatform/

@TheRobertjoellewis Год назад

Great intro! Hmmm I get an error when I run the basic pipeline. "Internal error encountered. Please try again" - moving over to the docs.

@ml-engineer Год назад

Take the compiled pipeline JSON and upload it via UI to see if you get a different error there.

@TheRobertjoellewis Год назад

@@ml-engineer It was indeed a permissions error. I gave my account the right permissions and it worked :)

@ml-engineer Год назад

@@TheRobertjoellewis good glad it is working now

@kadapa-rl6jg Год назад

Can you also create a session for troubleshooting Vertex AI

@ml-engineer Год назад

Hi Sure any specific service? Usually everything is logged. Though batch predictions can get a bit more complicated to troubleshoot.

@mariannakovalova8849 Год назад

@@ml-engineer it would be great! Because I have an error "The DAG failed because some tasks failed. The failed tasks are: [concat]" for this tutorial and have no idea why and how to fix... And can't move on

@ml-engineer Год назад

@@mariannakovalova8849 Hi Marianna I just ran the notebook to ensure everything is working as expected. Could not reproduce the error you get for the basic pipeline. Head to the logs and check the detailed error information. If you like, post them here and I might see why it is failing.

@mariannakovalova8849 Год назад

@@ml-engineer com.google.cloud.ai.platform.common.errors.AiPlatformException: code=RESOURCE_EXHAUSTED, message=The following quota metrics exceed quota limits: aiplatform.googleapis.com/custom_model_training_cpus, cause=null; Failed to create custom job for the task. Task: Task name: concat, Task state: DRIVER_SUCCEEDED, Execution name: projects/2

@mariannakovalova8849 Год назад

@@ml-engineer but when I go by link the usage of the resources are 0 or some small percentage

@kanavdua4587 6 месяцев назад

Hi Sascha. I have been facing an error for the last 3 days. Please help me resolve it.

@ml-engineer 6 месяцев назад

Hi What kind of error?

@kanavdua4587 6 месяцев назад

I am not able to write it as a comment. I don't know why.

@kanavdua4587 6 месяцев назад

The DAG failed because some tasks failed. The failed tasks are: [concat].; Job (project_id = practice-training, job_id = 125471868915286016) is failed due to the above error.; Failed to handle the job: {project_number = 385236764312, job_id = 125471868915286016}

@ml-engineer 6 месяцев назад

@@kanavdua4587 you can check what happened in the logs for each step/ component in your pipeline.

@kanavdua4587 6 месяцев назад

@@ml-engineer Please can you guide me a little 🙏🏻🙏🏻. @component() def concat(a:str,b:str)->str: Logging.info(f"concatenating '{a}' and '{b}' results in '{a+b}' ") return a+b I am a beginner. I don't have any knowledge. Please help. return

@frederikbode9880 Год назад

so how's the stress? :D

@ml-engineer Год назад

Which stress? 🙂🙂

@Juliodonadello Год назад

0,96 ? overfitted xd

@ml-engineer Год назад

It was a very easy dataset 0.96 is indeed correct.

@ml-engineer Год назад

It's in the notebook you can run it yourself l. Breast cancer dataset. Scores around 95 upwards are achievable and the normal range for his dataset. You can get up to 98.

@Juliodonadello Год назад

@@ml-engineer 92 mine