Monitoring, Logging, And Alerting In Kubernetes

Подписаться 73 тыс.

Просмотров 28 тыс.

50% 1

What is the best combination of tools for monitoring, logging, and alerting in Kubernetes?
#prometheus #grafana #loki #robusta
Consider joining the channel: / devopstoolkit
▬▬▬▬▬▬ 🔗 Additional Info 🔗 ▬▬▬▬▬▬
➡ Gist with the commands: gist.github.com/e593ff9fa1a34...
🔗 Prometheus: prometheus.io
🔗 Robusta: robusta.dev
🔗 Loki: grafana.com/oss/loki
🔗 Grafana: grafana.com/oss/grafana
🎬 Kubernetes Notifications, Troubleshooting, And Automation With Robusta: • Kubernetes Notificatio...
▬▬▬▬▬▬ 💰 Sponsoships 💰 ▬▬▬▬▬▬
If you are interested in sponsoring this channel, please use calendly.com/vfarcic/meet to book a timeslot that suits you, and we'll go over the details. Or feel free to contact me over Twitter or LinkedIn (see below).
▬▬▬▬▬▬ 👋 Contact me 👋 ▬▬▬▬▬▬
➡ Twitter: / vfarcic
➡ LinkedIn: / viktorfarcic
▬▬▬▬▬▬ 🚀 Courses, books, and podcasts 🚀 ▬▬▬▬▬▬
📚 Books and courses: www.devopstoolkitseries.com
🎤 Podcast: www.devopsparadox.com/
💬 Live streams: / devopsparadox
▬▬▬▬▬▬ ⏱ Timecodes ⏱ ▬▬▬▬▬▬
00:00 Introduction to monitoring, logging, and alerting
00:35 Metrics And Alerting With Prometheus
08:04 Notifications With Robusta
10:01 Logs Collection With Loki
13:45 Dashboards With Grafana

Наука

Опубликовано:

4 июл 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 58

@DevOpsToolkit 2 года назад

IMPORTANT: I made a mistake in the video by saying that AlertManager is querying Prometheus. That's incorrect. It's the other way around. Prometheus is evaluating the rules and sending alerts to AlertManager which, in turn, is forwarding them to final destinations like Slack, email, etc. What do you use for monitoring, logging, and alerting? What's your favorite stack?

@oftheriverinthenight 2 года назад

Prometheus, Loki, alert manager, grafana, slack

@sriveralopez 2 года назад

pin this comment!! great video, loving it

@nas1k 2 года назад

I think we should include Tracing here. It can be Jaeger, Temp or something else. And all those thing should be standardized by OpenTelemetry.

@DevOpsToolkit 2 года назад

You're right. I should have added tracing to that video. I'll work on making a follow up with tracing

@DevOpsToolkit Год назад

Here's Jaeger: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-FK0uh-7nDSg.html

@coocoobau 2 года назад

I highly recommend kube-prometheus-stack, all-in-one helm chart to deploy prometheus, grafana and alertmanager, each with its own operators. So instead of pre-defining things in values.yaml, you can use CRs to define targets, rules, alerts, dashboards, datasources, etc - in a Kubernetes way. For the logging part, I found banzaiclud's logging-operator to be very interesting, again a way to simplify the deployment of software for logging collection, aggregation and shipment (Loki being just one possible destination). It is also built around an operator and deploys instances of fluentd and fluentbit.

@daivol666 2 года назад

It would be interesting to see an example using opentelemetry to gather the observability data (avoding agents vendor lock-in) and use the otel pipelines to expose the data to different vendor solutions.

@DevOpsToolkit 2 года назад

Adding it to my TODO list... :)

@soubinan 2 года назад

Great video!!! Observability is so important and allow a lot off evolution not yet explored today I have the mnemonic word AMLET for alerting, monitoring, logging, eventing (context and others) and tracing I think also that grafana is the de facto place to have all data to observe even as a saas) thanks to tempo and loki we can add more meaning to metrics dashboards (and I have a small preference for sensu go over robusta to serve as a glue around all that) and leverage all that with a runbook system for auto remediation (stackstorm, awx, ansible platform, jenkins, rundeck....). The dream!

@luismorteo6112 Год назад

as ever you are rigth, i try loki and wooow! woks perfect with grafana thanks a lot genius!

@RaviSharma-vw7py 2 года назад

Thanks Viktor for your nice video & informative , really helpful

@andriespiitso9797 Месяц назад

I suggest you also add the alert example just like you do with querying. Otherwise great video enjoyed it 👍🏽

@azerbaijan50 Год назад

Nice explanation! Thank you very much.

@jemag 2 года назад

Would be interesting to have a deeper dive, things like Thanos, Tempo, Mimir, etc. Also, what do you think of using their jsonnet libraries to manage those? I found the community helm charts to be not that well maintained and jsonnet is actually pretty flexible for an enterprise setup

@DevOpsToolkit 2 года назад

I have Tanos on my TODO list. Adding tempo and Mimir as well... :)

@claytoncastro2734 2 года назад

I second that request for tempo. Great video as always viktor.

@fpvclub7256 2 года назад

Fantastic!

@jirityr 2 года назад

If you are serious about monitoring, you need to setup your own monitoring system even on managed kubernetes like EKS, GKE and AKS. I hope you will take the topic of monitoring further with introduction of Prometheus Operator, Grafana Cloud Agent (and GCA Operator), Grafana Operator and perhaps also Grafana Tempo. I would also love to see separate video about VictoriaMetrics that is much better than Prometheus itself.

@onemanops 2 года назад

Yes!

@richarmunicosamaniego8216 Год назад

MELT stack = Monitoring, Event (alerting +OnCall), Logging and Tracing

@oftheriverinthenight 2 года назад

In the latest versions grafana also shows the alert manager alerts and can be silenced from there too (bell icon)

@DevOpsToolkit 2 года назад

I was not aware of that. That's great news.

@MusheghDavtyan 2 года назад

very interesting

@vMILYAv 2 года назад

VictoriaMetrics (Operator)

@lhxperimental 2 года назад

This could not have come at a better time! Looking forward to part2 with tracing, open telemetry etc. and maybe also cover the maintenance aspects. Prometheus does automatic data purging which makes it maintenance free; how does loki compare with it. With logs the data volumes are going larger and much more workload dependent so one could easily overwhelm the system. Plus some organizations may need log archives to be kept for several years, how loki supports that use case would be interesting to see. My organization uses elastic search. Can loki be a replacement for elastic search today, or in future? The reason I would prefer loki over elastic is because I can co-relate logs with metrics, events and maybe even traces. In case of java/spring boot based apps, tracing can be very simple to achieve with auto-instrumentation. This would provide great visibility into the working of the application. I am myself exploring it this week.

@DevOpsToolkit 2 года назад

Adding those to my TODO list... :) In the meantime, you might want to join ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-Jqe7kkXZzCo.html and ask those questions there as well.

@lhxperimental 2 года назад

@@DevOpsToolkit Sure, added a reminder

@DevOpsToolkit Год назад

OpenTelemetry is finally finished and available at ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-oe5YYh9mhzw.html. Tracing is coming next.

@vn7057 Год назад

After few mo later Grafana stack extended make more flexible Grafana tempo + open telemetry for auto instrument + Grafana agent Grafana loki Prometheus Grafana Alertmanger Basically included Metrics , log, apm/tracing and alert Also Grafana able to adding silence by UI so we don’t need expose Prometheus alertmanger to make alert mute

@joebowbeer 2 года назад

I concur with others that Tracing is conspicuously absent, as is OpenTelemetry (OTEL), which is the emerging standard that ties all these CNCF pieces together with others such as Fluent Bit

@DevOpsToolkit 2 года назад

You're right. I got complaints in the past that the videos were too long so I started making shorter ones and that often results in things missing. I'll work on a second part of that video with tracing and open. telemetry

@DevOpsToolkit Год назад

I finally got around making a video about OpenTelemetry (I wanted to explore it separately first). It's available at ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-oe5YYh9mhzw.html. Tracing is coming next.

@dmsalomon 2 года назад

Great stuff, but the in my opinion the really tricky part is managing these things at scale. First of all there is the storage aspect, but also Prometheus seems to breakdown when the cluster gets too big. At that point you either need to use a federated setup or something else and it would be useful to hear your thoughts on that.

@DevOpsToolkit 2 года назад

You're right. When running at scale, a single Prometheus does not work (it cannot scale). I already have it on my TODO list to tackle that subject in one of the upcoming videos.

@dmsalomon 2 года назад

@@DevOpsToolkit amazing! Very much looking forward to that

@AhmedAyman-gs7oz 29 дней назад

Great video. Are there any other self-managed logging solutions other than ELK/EFK and Loki-Grafana?

@DevOpsToolkit 29 дней назад

There's fluentd and fluentbit for shipping logs.

@AhmedAyman-gs7oz 29 дней назад

@@DevOpsToolkit Thanks. I mean more on the storage side. Like if I want to move away from AWS open search to decrease cost. Are there any alternatives than Loki?

@DevOpsToolkit 29 дней назад

@AhmedAyman-gs7oz Loki is the only one I used besides elasticsearch (excluding managed solutions).

@ZachLanich Год назад

You're awesome

@DevOpsToolkit Год назад

Thanks a ton Zach

@SerhiiHromov Год назад

How did I miss this video, 2 days wasted. Thanks

@gaetanbloch7119 Год назад

Thanks!

@DevOpsToolkit Год назад

Thanks a ton!

@gaetanbloch7119 Год назад

@@DevOpsToolkit My pleasure! Can’t help but contributing a bit as I’m binge watching your videos. Plus, it was a pleasure collaborating with you on Geekle’s conferences. Keep up the great work!

@samehammar8062 2 года назад

Thanks for this great video , what about black-box exporter ?

@DevOpsToolkit 2 года назад

I should probably create a separate video about a selection of Prometheus exporters.

@samehammar8062 2 года назад

@@DevOpsToolkit this would be nice 😊. Thank you 🙏

@vn7057 2 года назад

Grafana is a great tools but somehow like Jenkins If the plug-in no longer maintain then you may get trouble it have chance may your Grafana unable to perform upgrade otherwise graph will dead But I do love the trend they go for Loki+Tempo+Display Prometheus Love to see if there have tools to generate dashboard and integrated 3 of them Do you have plan or already have video for Tempo?

@DevOpsToolkit 2 года назад

Tempo is on my todo list :)

@jgarfield 2 года назад

How do we configure SSO for Grafana login?

@DevOpsToolkit 2 года назад

As far as I know, there is no SSO in Grafana open-source version (not sure about Enterprise). I would probably try something like Teleport (you'll find a video in this channel).