Graylog: Your Comprehensive Guide to Getting Started Open Source Log Management 

My Graylog 5 Forum Post with commands lawrence.video/graylog5

Tom, thank you. This is what makes you stand out. You dont explain only how but also why. So many things now people write a guide only on how.

I have been a linux admin for about as long as linux admins have been a thing, but I have managed to avoid Docker for some reason. I saw that this was on docker and it was a project I wanted to try... my first instinct was "No, find the source" but I decided to give it a shot. Thanks for making this really easy.

Minor thing - I'd recommend adding an extra space to the beginning of the echo command at the early stage where you create the SHA256sum for the password - this stops the password being visible in that user's history. Minor thing but I've heard of history files being a juicy target like this.

I did this as an assignment a few months before I graduated. I did not set it up on my own server at the time. Thanks for making this video!

Must have paused and rewound the video about 100 times but got my pfsense logs flowing to a graylog testbed as per this video. Requested a login acct as "mark" on your Forum to post some further questions. Great video.

Great guide, thanks for the info. Tip for those who use proxmox as vm host. Put your CPU in Host mode as otherwise mongodb will not work.

This was great. Thank you Lawrence for taking the time to do this for us. I for one am adding this to the list of things to build this weekend.

Thanks for the updated tutorial! I set up graylog using docker compose a few months back, and followed the old video for the in-app setup part. It's great for folks to have an up-to-date version of the instructions. For anyone looking to set this up - inevitably some servers will disagree about the timezone, so even if you set your user and timezone correctly, its worth having a pipeline that can adjust a source between UTC and local time.

14:08 you can mark, that new user with own timezone will be see logs with corrected time and mark diff that admin see utc. This video is better then previous. Good job and I hope you create a video about extractors.

So Tom really just made the video to get his shirt fixed. I knew he didn't just make videos for education alone! 😂

Thanks. Using Grayling but your video showed some great ways to modify it. And love the glasses look!

Fantastic tutorial, Tom. I'd love to see how to bring in pfBlocker logs into Graylog.

I was literally about to go over your previous video for exactly this. Great video as always! Thanks Tom!

Haven't watched the video YET.... But love the "tutorial" image on the thumbnail. Nice touch!

Just what I needed! Thanks Tom for all you hard work.

Strange, last night I finally got around to starting work on testing a Graylog server. noticed your instructions were for V4, and decided I would do get the docker image working today.. now Look at this! what timing.

Thank you Tom , awesome video ... graylog looks very interesting as a centralized logging solution

I'd really like to use it but the fact that "log view" is behind the enterprise version paywall is just insane. will stay with Grafana Loki as logs are just so much easier to read there - such a shame for homelab users like me.

Very useful video, THX Lawrence .

super awesome video, I'm a huge fan of graylog.

great tutor, i see the workflow and what it can do.

Thanks for the recommendations! Was a bit finicky but got it running on my proxmox cluster and ingesting logs from the xigmanas box now! nice to have logs I can search instead of losing on reboot. Anyone else going thru the install make sure you set the CPU up to at least x86_64_v3 for the intruction set for mongodb. Took me a bit to find the error.

Great quick tutorial! However the part I am most struggling with is the connection between inputs, streams, indices and extractors. A comprehensive overview of the architectural model of Graylog would be much appreciated.

Thanks for the great tutorial! I would be interested in a discussion about Sentry - an open source tool for catching unhandled exceptions, collecting related context data and alerting the developers. It seems that some of this logic can be implemented with Graylog, and I was wondering whether it makes sense to use both systems, or if one would suffice.

But 1514 is unencrypted right?, I mean syslog data are being sent "naked"? It means that network connection should be trustfull. Like separate VLAN or something?

Ubuntu redirecting Apt Get commands to Snap Install is so dishonest. Want to promote snap, fine. Lying to the user should not be tolerated.

Maybe I missed it but I don't think you mentioned the pros/cons of installing via docker instead of a "normal" install. I'd also be interested in your opinion on graylog vs loki/grafana. Also you're using opensearch and I think elastic was my only option when I set this up and I'm not a fan of elastic -- would be interested in hearing why you chose opensearch. I have graylog running in a proxmox vm that I set up years ago. Struggled to get it setup and configured, I have some ongoing issues where some feeds have accurate times and others have their timestamps in a different timezone, but it feels like such a headache to configure as a hobbyist who doesn't work with it daily. I know there's a lot more I could be getting out of it, but right now it basically sits as a "well if something goes terribly wrong i can search graylog" and that's about the extent of the value I get from it. Thinking about switching to loki/grafana in the hopes the config is easier for someone who doesn't interact with it daily where currently any changes I want to make mean I'm going to spend hours researching the syntax or formatting for graylog. It's 100% lack of familiarity on my part combined with user error but the thought of having to make changes to graylog gives me a headache.

Really great video, thank you. Very clear, detailed and last but not least: usefull

7:30 How does one sign up for MailHop? Looks like their website is just a page stating there's no website. 😅

TOM IS BACK WITH THE GOODS!

Thank you for making this video. I know we all copy and paste at times for expediency. However, to recommend that users do this, in a video, may enforce dangerous behaviors. Should people just have common sense and read the commands before they paste them? Yes, of course. But, hey, that's what we have disclaimers for. "If you feel confident in my instructions, and you are running this in a development environment, you can go ahead and copy and paste these commands into your terminal." Obviously, if your hat is really, really dark, making people dumber is obviously a worthwhile goal.

Having multiple issues with docker compose erroring on the depends_on section of the YAML, first error is needs to be an array and then values need to be a string, any ideas ?

Great video! I would love to see a comparison of Graylog and ELK stack.

Hello Tom! I managed to setup this just like you. I use version 5.1. Is there a guide or is there a way you can help to setup the SSL certs so I can use a https?

One thing I cannot for the life of me figure out is how to use NFS to store the actual log data (opensearch). If you try and use docker-compose to store the data on an NFS volume, the container fails to launch as it seems the image is trying to run chown on the data storage directory, which I guess nfs doesn't allow.

Very cool video! I'll definitely be watching it! If I may make a request for another video: could you do one on Fluentbit/Fluentd? (I never know what to call it). It's always been such a headache for me to get back into the config logic once something decides to break again, but it's otherwise been working so perfectly for us! I'd love to see your take on it and see if I missed anything.

Finnnnnnalllly the notification I’ve been waiting for

I was getting some "depends_on" errors when trying to run the docker-compose I fixed it by using "docker compose" instead of "docker-compose" command

Well done - thank you!

I thought we should not install and use docker-compose anymore, but use the docker compose command in the newer versions of docker?

@16:35 - why does graylog don't have template extractors (plus make it auto detect) for standard stuff: windows.linux-pc/laptops/servers, mac, pfsense, etc...?

So an index is just a way to do high level categorizing/grouping of data sets/sources?

Saw in the latest docs that the virtual appliances is no longer available, neither able to find the OVA image. Not sure if its possible to install this in docker on a mac setup.

If we are using elastic search then what's the advantage with this tool? Why should we use it?

i'm trying to find a way to have it alert me when dhcp leases are given out for new MAC addresses on the network, i have this working via syslog-ng and a bash script but if i can do through the GUI in Greylog that would be great

Can anyone tell me which one is best for log managment between Graylog, Wazuh and ELK? Mainly for Windows servers and Mikrotik routers.

Does gray log provide functionality in addition to wazuh ? Or they are same.

Great job 🎉

Just set it up, super easy! now who knows where I can find a json for unifi extractors?!?

This was a great tutorial, Thank you. Do you know if its possible to have Graylog record information on each of the TCP sessions from PFsense firewall, for example, how many bytes sent/received for each TCP session, and if the TCP session ended with FIN or RST ?

I've more than 25 docker containers running on few different VMs, I'm no expert in docker but not really a newbie either But starting Graylog? I just can't do it The way they implemented the $USER is beyond my understanding Keep getting stuck at this error when Graylog is starting: ERROR org.graylog2.bootstrap.CmdLineTool - Couldn't load configuration: Properties file /usr/share/graylog/data/config/graylog.conf doesn't exist! (And yes it exist, and it is mapped correctly) I've tried to set user variables, tried to change directly the mounted directory ownership to 1100:1100 I've tried with other versions of docker-compose Tried also changing the owner to docker:docker Executed multiple times that "sudo usermod -aG docker $USER" Rebooted the server, tried other mounting points that are not in the /home directory Nothing works Sorry but the Graylog docker image is broken for me (and no I'm not using snap docker package even tho I'm running on Ubuntu Server) Thank you for the tutorial but sadly I might have to many skill issues to solve this

It’s odd I set this up and found that windows 11 default firewall blocks port 9000 so I thought it wasn’t working and then decided to try my phone and it was working except that some reason my password I placed was not working.

very helpful thank you

@Lawrencesystems Tom, Have you used the SIEM product (Graylog Security) before. I'm interested in a platform like that to help with cyber threats. Do have any other suggestions as far as an SIEM log platform?

Hey Tom, could you make a video about zabbix as a comparison. It has pre-defined templates and triggers for the most popular systems, linux, windows, firewalls, etc. Very powerful tool. I would love to see it on your channel. It comes containerized as well.

Is there a way to set this up over https? I want greylog itself to have the https cert.

This is very good!👏

I may have done something wrong because messages are only hitting the very last stream/indices I created. In other words, PFsense was the first one created, and messages were hitting it. The last one I created was for a Cisco switch, and now no PfSense messages, but lots of messages to the Cisco switch. Any thoughts on this? Thanks!

hi good evening, very good works...please a question?...how do yo do your prompt console??? many thanks in advance

Would like to see how the logs can be redirected to another mnt drive. The quick explanation did not work.

Great video.

Great Video, but why are you not dealing with IPv6? The world is 33% based IPv6 and growing. So anyone with a IPv6 extractor that work?

Thanks for the video =)

Why would I want to configure smtp logging for graylog?

Hey Tom, thanks for your amazing videos! Small request, would it be possible to raise the volume on your videos, I find it even with my speakers cranked to to max I still have a hard time hearing you. (If it's too loud for someone they can always reduce volume vs raising isn't always possible).

Thanks for the video for deploying graylog. It seems your demo server has 8 core 4GB memory. I know it is for demo purpose. But how can I calcurate the necessary hardware resource for certain system ?

great video

Wish you would do a install version of this on scale. It seems impossible to get it to work. Everyone and their mom is using yaml and scale doesn't.

Is there any way to get UniFi Firewall logs into Graylog?

Hi. Can you help mo sir how can I add router and switches in the graylog for me to monitor my network

Clean install of Ubuntu 22.04. Graylog container wont start. Stays in thee 'starting' status. I then instead install graylog natively with opensearch and mongo. Runs without issue. Something wrong with the compose file maybe?

maybe you can compare Graylog to other open source systems?

What the f, i gave my graylog vm the same static last octet as you did by coincidence, only difference is mine uses 49 as octet 3

Great video! What terminal are you using? Looks awesome

@Lawrencesystems: did you get a new t-shirt? :)

is any specific reason do you using opensearch instead elasticsearch ?

How weird. I was just trying to get Graylog running in Docker on my Synology this morning.

BUT How do I make a cluster system with redundancy purposes

Tom, is it recommended to use docker compose for production?

Did you get a new shirt though?

any reason why greylog instead of elastic?

"docker compose up" works for me (without "-" in "docker-compose").

Great video - I used your compose file and i get this "mongodb exited with code 132" every time I try to run docker-compose up. I can't find any errors - It runs on proxmox in a ubuntu 22.04 LTS VM. any ideas ?

How about a video with a sidecar and windows logs?

What happened at 16:14?

Does Lawrence System have a vpn tutorial for mobile clients (including android 12 / 13)?

what about loki?

Graylog here for: Syslog: pfsense, unifi, synology nas. Netflow: pfsense Works great 👍

Do a pipeline vidjayo

bro got that long ahh screen

This turns painful really quick if your processor doesn't support AVX.

Can I attach any dashboard to greylog?

First!

this was such a fucking mess for me. Once I got permissions all figured out, I found out that mongo 5.0+ required hardware that apperantly my box didn't have, and then I tried to figure out compatability between all three, and i just gave up, it's not worth it for something to needless for me...

The snap install was a lot better though. Just sayin'. Docker is equal to useless overhead and useless complexity.

I don't think it's open source

Seq is better…

Hi Tom, I added this to my existing docker (installed via apt) but the graylog container is not starting up. I'm getting this in the logs: com.mongodb.MongoSocketException: mongodb: Temporary failure in name resolution Caused by: java.net.UnknownHostException: mongodb: Temporary failure in name resolution 2023-05-13 15:13:58,222 INFO : org.graylog2.bootstrap.preflight.MongoDBPreflightCheck - MongoDB is not available. Retry #1 2023-05-13 15:14:00,222 INFO : org.mongodb.driver.cluster - Cluster description not yet available. Waiting for 30000 ms before timing out I've tried removing and re-deploying but no luck.

What about Grafana & Prometheus? What are the differences?