DevOps Real-time Project #2 - How to Stream Logs to Azure Blob Storage

Подписаться 29 тыс.

Просмотров 4,5 тыс.

50% 1

In this video, I will show you how to stream your AKS(Kubernetes) logs to Azure Blob Storage using Vector( A free tool by DataDog).
What is Vector? 🤷‍♂️
Vector is a data pipeline solution that helps collect, transform, and route logs, metrics, and traces to any cloud-native service or third-party application.
📌Below steps were performed in this video:
- Login to your Azure portal and provision an AKS cluster through the Azure portal or Azure CLI.
- Once the AKS cluster is up and running, log in to the cluster using the Azure CLI, create a test namespace, and deploy a sample Nginx deployment.
- Create an Azure blob storage account and container called “akslogs” to store the logs.
- Prepare a Vector configuration file called “vector-config.yaml” by adding the storage account connection string and other relevant fields, allowing Vector to access the storage container and stream the logs.
📌By the end of this video, you will learn -:
◾ What is Vector ( By DataDog)?
◾ How to Provision Azure resources such as resource group, aks cluster, storage account, etc,.
◾ How to create vector configuration to stream logs.
◾ Different sections in the vector configuration file.
◾ How to get your first DevOps job as a fresher?
◾ Test the setup.
VECTOR CONFIGURATION YAML 👀
github.com/piyushsachdeva/Aks...
𝗖𝗢𝗠𝗣𝗟𝗘𝗧𝗘 𝗧𝗨𝗧𝗢𝗥𝗜𝗔𝗟 𝗦𝗘𝗥𝗜𝗘𝗦 👌 It’s 🅵🆁🅴🅴 ⛳
Namaste Google Cloud | GCP Tutorial for beginners, a complete playlist for Associate Cloud Engineer Certification Exam:
• Namaste Google Cloud, ...
DevOps Tutorial for beginners
• DevOps Tutorial for be...
AWS tutorial for beginners
• AWS (Amazon Web Servic...
Azure Data fundamentals DP 900 full course
• Microsoft Azure Data F...
Jenkins Tutorial for beginners
• Jenkins Tutorial For B...
DevOps and Cloud Podcasts:
• DevOps and Cloud Podcasts
Subscribe to our channel to get notified about the latest videos.
🔥 / @techtutorialswithpiyush
Connect with me 👋
🌎LINKEDIN: ►
/ piyush-sachdeva
🐦TWITTER: ►
/ techie_piyush
Join our learning communities 👋
/ 1015771332531944
/ 12561913
References 📚:
/ streaming-aks-kubernet...
vector.dev/docs/reference/vrl...
vector.dev/docs/reference/con...
vector.dev/docs/reference/con...
vector.dev/docs/reference/con...
#kubernetes #devops #aks #devopsproject

Наука

Опубликовано:

22 янв 2023

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 22

@phanindrachitta 3 месяца назад

Excellent

@TechTutorialswithPiyush 3 месяца назад

Thank you so much 😀

@dr.hemantchauhan2613 Год назад

Good one

@TechTutorialswithPiyush Год назад

Thank you 😊🙏

@MatthewKennedyUK 25 дней назад

Can you do another video setting up the vector agent and stateless aggregator separately?

@TechTutorialswithPiyush 24 дня назад

Will do if more people request it

@YouTubers-rj9xv Год назад

Bro super teacher

@TechTutorialswithPiyush Год назад

Thank you so much bro :) I am glad that you found it helpful.

@user-gs5xl9zl9o Год назад

Thank you for this tutorial, very clear. However, my connection string doesn't work, I don't know why :( ERROR vector::topology::builder: msg="Healthcheck failed." error=Invalid connection string specified component_kind="sink" component_type="azure_blob" component_id=azure_blob component_name=azure_blob

@TechTutorialswithPiyush Год назад

Hi Rachel, Thanks for the feedback! Regarding your query, Looks like you have used the incorrect connection string, make sure to use the connection string in your vector config.yaml file from Azure storage and not the access keys. Here's the blog as well that you can refer quickly and see if you missed anything medium.com/@piyush.sachdeva055/streaming-aks-kubernetes-logs-to-azure-blob-storage-using-vector-a-step-by-step-guide-e3044aa92d57

@AjayKumar-lm4yr 9 месяцев назад

How to store Grafana Loki logs in Azure Blob Storage

@TechTutorialswithPiyush 9 месяцев назад

I dont think Loki/Grafana is one of the supported source by Vector vector.dev/docs/reference/configuration/sources/

@user-jb9ml3xg6k 10 месяцев назад

but issue is that sir ..... after interval time 7 or 8 min new log is generating and it is not append to a exiting on same log file ....pls help on that issue

@TechTutorialswithPiyush 10 месяцев назад

Hello, As per the documentation, the default maximum file size is 102400 bytes before it is truncated and create a new file. You can customize it vector.dev/docs/reference/configuration/sources/file/#max_line_bytes

@user-jb9ml3xg6k 10 месяцев назад

@@TechTutorialswithPiyush correct sir but my blob log file size is 2kb only then why is it not append new log data on same blog log file ..

@user-jb9ml3xg6k 10 месяцев назад

sources: my_source_id: type: kubernetes_logs auto_partial_merge: true max_line_bytes: 600000000

@user-jb9ml3xg6k 10 месяцев назад

need help sir

@TechTutorialswithPiyush 10 месяцев назад

@@user-jb9ml3xg6k It might be the issue with the vector version you are using, can you try using a newer or latest version?