Thanks for your video. I have spun the fluenbit pods in one of namespaces but getting this error. I have double checked var/log/containers folder have read permissions. I ran these pods and deployment files as root user. below is the error. [2024/07/01 12:32:44] [error] [input:tail:tail.0] read error, check permissions: /var/log/containers/*.log
Hi there, thank you for your video and the GitHub repo, currently im using these k8s filter. But i wonder how can we parse multi lines logs that our microservices generating in k8s? I check the official documents and try multiline parsers in input section or custom parsers in filter sections, but still i see some of my logs are spilited in one line! Our microservices usually are spring boot and they have stack traces in their console logs
Why is the config format different to Fluentd? It is even from the same developers, and on a first glance describing similar things. That makes it hard to port from FluentBit to Fluentd, when things get larger and you need a plugin that FluentD has but FluentBit has not.
Hi, I will try to build another episode on Fluentbit with one of the contributors to answer that question. Otherwise, if you are already using Fluentd , you can still utilize FluentBit and Fluentd together. Fluentbit has a Fluentd exporter.
That is a nice observation, for instance, fluentbit has an output plugin to loki as we watch here as well, but fluent d does not send logs to loki. So If I want to use loki as a logging backend system, do I have to choose just between fluentbit or promtail as a collectors? It seems fluentd cannot transport to loki ... How can I deal with this if I am collecting logs using fluentd in order to transform them in a unified format and then I want to visualize them in loki? I heard we can use fluentbit as a collector and fluentd as a transformer or aggregator, so they can work together, but then the order FluentBit --- FluentD --- Loki cannot be used since FluentD does not send logs to Loki. Is possible to use first FluentD --- FluentBit --- Loki? I would say that will not be possible because FluentD does not have output to FluentBit ...
Hi Bernardo, if you are already have a log stream pipeline with Fluentd , the you could use it and then use the output plugin Forward to send it to FluentBit ( with the loki output plugin installed) to forward your logs to Loki. The best is to convert your fluentd pipeline into a fluentbit.
I also detailed in your explanation here, with the filters plugins like nest or record_modifier we can re-format the logs we collect with fluentbit, so then if I want to use fluentd to get an uniform format, I can also pursue that by fluent bit filters, so in this case I wouldn't need fluentd and then I can send to loki directly from fluentbit. Am I right? But then, in which situations will be good to use fluentbit and fluentd together if I can enrich, parse and filter with both? Sometimes their functionalities overlap each other.
Fluentbit is designed for Kubernetes. so if you have your pipeline in fluentbit format then you won't need fluentd. the value of fluentd is the large number of plugins. So if fluentbit has a missing plugin , you can imagine to chain fluentbit and fluentd to take advantage of the plugins.
If jobs with titles is sending them logs to stdout , then you should have your logs . Otherwise you will have to mount a pvc to your job and to the fluentbit agent
I think that you should mention that other videos are a prerequisite to understanding the tutorial of this video. I felt like I wasted quite a bit of time only to realize that in the tutorial section you keep on mentioning what you did the last time.
Hi Rehan, I'm sorry. I'm also trying to promote the other videos..but each tutorial is covering the topic of the episode. To reduce the duration of the tutorial, i explain quickly the first step of the tutorial : create a c Cluster, deploy Prometheus,deploy the demo APP...and then spend more time on the main topic of the episode. Maybe i was a bit unclear while explaining the tutorial. Sorry to be unclear.
@@isitobservable No worries. I understand that. It's pretty hard to structure videos on these topics. It would be nice and maybe easier if you could put the links to those videos in the description so if someone needs to look them up, they can find them here directly.
Hi, so if your application is writing in a custom folder then you need to make sure that the folder is mounted to a volume of the node ( or pvc) . The objective is to make sure that another pod ( in our case fluentbit) would be able to access this filesystem. Then to collect this log you will need to mount this volume in fluentbit , and add a second tail input plugin to read the file from this folder.
Hi, The tail plugin is can read any files. but keep in mind the fluentbit agent is running on the node. it means that you will need to specify the right path to reach container file or volume .
