No video :(

Kubernetes Logging | Amazon EKS logging using Fluentbit and Amazon OpenSearch Service | Dashboards

Подписаться 15 тыс.

Просмотров 10 тыс.

50% 1

===================================================================
1. SUBSCRIBE FOR MORE LEARNING :
/ @cloudquicklabs
===================================================================
2. CLOUD QUICK LABS - CHANNEL MEMBERSHIP FOR MORE BENEFITS :
/ @cloudquicklabs
===================================================================
3. BUY ME A COFFEE AS A TOKEN OF APPRECIATION :
www.buymeacoff...
===================================================================
This video shows how to configure Kubernetes logging i.e Amazon EKS logging using fluentbit and Amazon OpenSearch service and view the log in dashboard for analyzing, reporting etc.
It has design diagram level overview and flow explanation.
It has from scratch to end required configuration explanation to achieve the required set for the demo.
This shows command by command walk through for configurations management and finally demo of showing logs being forwarded to OpenSearch service. And visualizing it.
This video helps Kubernetes engineers - SME , Architects and Observability engineers etc.
The command file , .yaml and .json files used in this video can be found at repo link : github.com/Rek...
#aws #kubernetes #logging #opensearch #eks #cloudquicklabs

Опубликовано:

5 сен 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 33

@carlpei6570 11 месяцев назад

This is such really great tutorial !!! You have deep understanding of both AWS EKS and Opensearch Services. Hence your simple explanation totally makes sense about this integration. There a lot😊😊😊

@cloudquicklabs 11 месяцев назад

Thank you for watching my videos. Glad that it helped you.

@vrajbhatt8540 Месяц назад

Unable to fetch cronjobs logs using fluent bit

@cloudquicklabs Месяц назад

Thank you for watching my videos. Check out the path where this log is stored and then that should be configured log scrapers.

@systechadmin8368 Год назад

Nice❤

@cloudquicklabs Год назад

Thank you for watching my videos. Glad that it helped you.

@carlpei6570 11 месяцев назад

Subscribed

@cloudquicklabs 11 месяцев назад

Thank you, keep watching my videos.

@khoale8688 Год назад

Greate video!! How could I add more than one EKS clusters in same Opensearch. In my case, first I configured opensearch for staging cluster it was fine. I did the same thing for production cluster (difference VPC) and production works fine, then the staging has no permission and could not get log anymore. Thank you.

@cloudquicklabs Год назад

Thank you for watching my videos. Indeed this is great way to use the AWS resource, as you want point multiple eks cluster to a same OpenSearch cluster. It should support this until you have required access

@khoale8688 Год назад

@@cloudquicklabs Thank for you reply, I have found the solution for my case, just add the arn IAM for every cluster and that should get logs for multiple EKS cluster

@5nine838 Год назад

Hi, can you please tell me the why does the curl command you ran to link sa with internal users was required? Is it because there are internal users in the database?

@cloudquicklabs Год назад

Thank you for watching my videos. This is to provide EKS OIDC role permissions on OpenSearch (es) to push the logs from EKS to es.

@saad2966 Год назад

how to create index for K8 logs what command did you run in dev console

@cloudquicklabs Год назад

Thank you for watching my videos. I have shown how to creat log index using OpenSearch UI, you could follow the same. But how to do it with CLI if any I need to explore it.

@carlpei6570 11 месяцев назад

Liked

@cloudquicklabs 11 месяцев назад

Thank you so much.

@zenithpatel5926 Год назад

Thank you for making a nice educational video. How can I configure fluenbit to send eks control planes logs to OpenSearch?

@cloudquicklabs Год назад

Thank you for watching my videos. Glad that it helped you. Forwarding control plane logs to AWS OpenSearch can be configured at fluentbit specification file.

@hankukwak5550 Год назад

Thanks for Good video. if you possible, please show me this Github repo URL this lecture. thanks.

@cloudquicklabs Год назад

Thank you for watching videos. You can find GitHub repo at this video's description, please find from there.

@RenM908 Год назад

How did you come up with the fluent policy at 18:55? Where does the "Resource" come from and what makes up that file?

@cloudquicklabs Год назад

Thank you for watching my videos. With fluent-bit-Policy we are giving fluentbit access on Open Search service to push the logs. With Resource being mentioned in policy, we narrow down the permissions to a specific elastic domain name service where we are actually dumping the logs.

@hackerup Год назад

I followed however the k8s demon sets emitting errors as : [2022/11/29 07:42:41] [ warn] [engine] failed to flush chunk '1-1669707737.413578655.flb', retry in 106 seconds: task_id=2, input=tail.0 > output=es.0 [2022/11/29 07:42:41] [error] [output:es:es.0] HTTP status=400 URI=/_bulk, response: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}

@cloudquicklabs Год назад

Thank you for watching my videos. This looks to be syntactical issue. Could you please let me know in which step you are experiencing this issue

@work8597 Год назад

@@cloudquicklabs [2023/04/07 15:56:42] [error] [output:es:es.0] HTTP status=400 URI=/_bulk, response: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400} im getting the same, ive followed all the steps. deployed the fluent bit, checked the logs and this is what i get

@carlpei6570 11 месяцев назад

As '_type' parameter is removed from latest ES or Open search versions you need to set "Suppress_Type_Name" to On in fluentbit.yml in Output section

@pinkutharu8037 11 месяцев назад

Yes @@carlpei6570 that's true @work8597 you need to set [ suppress_type_name On ] in output section of fluentbit configuration file.

@ashokexp Год назад

Its not working on non eks k8s cluster

@cloudquicklabs Год назад

Thank you for watching my videos. Indeed it would not support for non-eks cluster if you follow same steps. There should be different ways for different cluster, may I know which cluster that you are trying

@ashokexp Год назад

Self managed k8s cluster I have modified the yamls of fb it's throws with 401 i have added the role with os domain full permission on it to make aws signed request still didn't working