[ ElasticSearch 15 ] Elastic Stack | Running Filebeat in a container 

Many thanks for watching. Glad it helped. Cheers.

Hi Shishir, thanks for watching.

Hi Victor, Thanks for watching.

Hi Alexander, Thanks for watching.

You are welcome. Thanks for watching.

Thanks for watching.

Many thanks.

Hi, many thanks for your interest in my videos.

Hi, thanks for watching.

Hi Daryl, thanks for your interest in my videos. I haven't tried containerized logstash yet but could give it a try. Cheers.

Hi Nagesh, thanks for watching. It will collect logs from all the containers.

Hi Rathin, thanks for watching. I don't think there is a way to do that without installing something on the production server. Filebeat, as far as I know can only pull data from the instance it is installed on. You can have Fluentd installed on a separate server, but again you need to install a forwarding agent on the production server.

@@justmeandopensource Thanks for thr reply. But All filebeat needs is the log folder as it's input right which we provjde in the filebeat-docker.yml file. For eg. Default json driver docker logs gets stored in /var/logs/docker/containers inside which we have the container logs. Is my understanding right ?

Yes. If you want to collect logs from the containers on your production server, filebeat on this separate machine needs access to /var/lib/docker/containers and /var/run/docker.sock from theat production server.

As you said, it should be possible but I haven't tried that yet.

Not exactly as you requested. Thanks for watching.

Yes you can use filebeat. Infact I did a video on Grafana Loki and released it today. You can use the Loki stack with Filebeat for collecting logs. ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-UM8NiQLZ4K0.html But in this video, I didn't use filebeat but Promtail.

@@justmeandopensource Thank you so much for quick turnout...i will try

@@d4devops30 no worries

Hi Swaraj, thanks for watching. Filebeat's default configuration has that index name format. However you can configure it with any name you like. Cheers.

@@justmeandopensource Can you share some references for creating index through filebeat?

Hi Swaraj, thanks for watching. This is a requirement as per Elastic docs. If you want to know what this kernel parameter is, here is the explanation from kernel.org. www.kernel.org/doc/Documentation/sysctl/vm.txt max_map_count: This file contains the maximum number of memory map areas a process may have. Memory map areas are used as a side-effect of calling malloc, directly by mmap, mprotect, and madvise, and also when loading shared libraries. While most applications need less than a thousand maps, certain programs, particularly malloc debuggers, may consume lots of them, e.g., up to one or two maps per allocation. The default value is 65536.

Hi Maxime, thanks for watching. I never thought about that scenario. Sure there must be a way as containers restarting is an usual thing.

​@@justmeandopensource As I'm running filebeat stateless, I think the best solution would be to use the figerprint processor (www.elastic.co/guide/en/beats/filebeat/master/filebeat-deduplication.html ) and use fields like date, container_id and offset to create the fingerprint. Then a restart would overwrite values in ES (still a lot of processing though). I can't understand why this issue isn't describe in filebeat/docker autodiscovery documentation. Maybe I'm missing something :/ I think this functionnality was really thought for K8s because giving full read-only access ton all containers data (including mounted docker secrets) just to get the logs out... Well I can only say that Docker/Swarm has it's limitation and you need to make a lot of trade-off in order to get "simplicity" :D

@@Mr3maxmax3 page shows 404

The link has an extra ")" at the end. Try this www.elastic.co/guide/en/beats/filebeat/master/filebeat-deduplication.html

@@justmeandopensource You can use Logstash too because, Logstash create since db that stores pointer and sometimes timestamp to resolve duplication issues.

I get the same as you... I'm sure I run a filebeat index. But I can't see the the log in the UI

and i am using python to do that

You are right.. No logstack component inbetween. Thanks for watching. Cheers.

@@justmeandopensource You are my inspiration. Want to be expert in the field like you

@@ahsanraza4762 Well, I am not an expert. If you just know how to read the docs, you can do it :P

In docker youve got two listening logs, filebeat will listen the both of them (STDERR and STDOUT)

Hi, thanks for watching. Its just a combination of Zsh, oh-my-zsh, zsh-autosuggestions, zsh-syntax-highlighting, powerlevel10k.

Thanks I'll be sure to try that out!

@@anmolmajithia You are welcome.

Hi, you can find my terminal/shell customizations in this video. ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-PUWnCbr9cN8.html

@@justmeandopensource 🙏

Hi Vijay, How are you intending to use Logstash?

@@justmeandopensource I have a elk+kibana+logstash on same server and another one is application server. I need to monitor user login details,any modifications in "passwd" file and uptime for application server. my elasticsearch search server only access with localhost, can't access directly from clients. so my logs are send to Logstash only.

I see. Thats a typical setup. Clients (Beats) can send data directly to ElasticSearch, in which case, you need to make elasticsearch available on public interface. The need for logstash is to filter and transform incoming logs before storing it in the elastisearch engine. Are you client machines Windows?

@@justmeandopensource Thanks for your quick reply. My clients are linux machine. I am trying to setup ELK for prod environment. I can't make elasticsearch as public interface. As you know its create a security issues. Thats the reason, i would like to forward logs to Logstash.

@@justmeandopensource filebeat with logstash working fine on same client machine.it create indices automatically in kibana Dashboard and i can create index pattern as well...but auditbeat indices not create jn kibana

Hi Shaheer, thanks for watching.

hi there, did you find a solution for this issue? I am facing the same error

hi there, did you find a solution for this issue? I am facing the same error

@@ahmedfayez nope