Open Source Logging: Getting Started with Graylog Tutorial

Просмотров 222 тыс.

% 3 176

Connecting With Us
---------------------------------------------------
+ Hire Us For A Project: lawrencesystems.com/hire-us/
+ Tom Twitter 🐦 TomLawrenceTech
+ Our Web Site www.lawrencesystems.com/
+ Our Forums forums.lawrencesystems.com/
+ Instagram lawrencesystems
+ Facebook Lawrencesystems/
+ GitHub github.com/lawrencesystems/
+ Discord discord.gg/ZwTz3Mh
Lawrence Systems Shirts and Swag
---------------------------------------------------
►👕 lawrence.video/swag
AFFILIATES & REFERRAL LINKS
---------------------------------------------------
Amazon Affiliate Store
🛒 www.amazon.com/shop/lawrencesystemspcpickup
UniFi Affiliate Link
🛒 store.ui.com?a_aid=LTS
All Of Our Affiliates that help us out and can get you discounts!
🛒 lawrencesystems.com/partners-we-love/
Gear we use on Kit
🛒 kit.co/lawrencesystems
Use OfferCode LTSERVICES to get 5% off your order at
🛒 lawrence.video/techsupplydirect
Digital Ocean Offer Code
🛒 m.do.co/c/85de8d181725
HostiFi UniFi Cloud Hosting Service
🛒 hostifi.net/?via=lawrencesystems
Protect you privacy with a VPN from Private Internet Access
🛒 www.privateinternetaccess.com/pages/buy-vpn/LRNSYS
Patreon
💰 www.patreon.com/lawrencesystems
⏱️ Timestamps ⏱️
0:00 Getting Started With Graylog
1:46 Open Source vs Enterprise
3:42 Installing Graylog
5:06 How Graylog Processes Data
8:18 Configuring Syslog Inputs
9:52 Graylog Extractors for pfsense
12:04 configure Graylog streams
17:28 Creating New Graylog Indices
18:12 Creating New Graylog Inputs
20:31 Creatting new Gralog Streams
26:06 Finding Data & Parsing Logs
30:11 Resource Usage
forums.lawrencesystems.com/t/using-graylog-for-log-management/8797

Наука

Опубликовано:

13 фев 2021

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 153

@LAWRENCESYSTEMS 3 года назад

Related Forum Post forums.lawrencesystems.com/t/open-source-logging-getting-started-with-graylog/8797

@Graylogvideos 3 года назад

Hey, we are doing some RU-vid clean up and just came across the video! What a great tutorial! Thanks for taking the time to make it :)

@LAWRENCESYSTEMS 3 года назад

Thanks!

@Miles-co5xm Год назад

@mjgritli 2 года назад

Thank you, I spent lots of time configuring ELK from scratch, but the work greylog has done is awesome, its simple and does the job well thank you for showing this

@adancalderon8915 3 года назад

This is Great Tom. I have been looking for this video on this topic. Thanks.

@earthling_parth 3 года назад

I keep referencing this video again and again. This is a great beginner level tutorial to basics of getting logs into Graylog, separating them into streams, and searching through logs with Graylog.

@LAWRENCESYSTEMS 3 года назад

Glad you enjoyed it!

@nlay42 2 года назад

Thank You Tom. I am looking at implementing Graylog in my home network and your video content was very helpful!

@bobbinatorrah67 3 года назад

I’ve just been thinking about how there must be something like this out there. Thank you! I’ll play with this!!

@loupalladino9578 3 года назад

I've been dragging my feet for about a year now on making a decision with respect to log aggregation from a handful of proxies I manage all over the world. I checked this video out and decided to give Graylog a try and I absolutely love it! The install is super, super easy and I had a Graylog instance running and ingesting data from several of my proxies within 2 hours. Now its just a matter of a tweaking queries and dashboards to let me see precisely what I need. Awesome video, as always, Tom - I for sure would have spun my wheels on the streams / indices / extractors /etc!

@JohnRRugge 2 года назад

could you share what queries you've used for your dashboards or any free resources available. Thanks

@earthling_parth 3 года назад

I was hung up on how to identify and separate out logs for a project/application once I send the logs from FluentD to Graylog. Your explanation on streams/indices/rules helped cleared out that confusion. Thank you so much

@LAWRENCESYSTEMS 3 года назад

Fantastic!

@blackranger07 2 года назад

Thanks! This video helped me to get graylog to start seeing incoming data.

@bamhm182 3 года назад

Thanks for the great video! I have been wanting to get into graylog for a while, this video finally got me to get off my butt. Still trying to figure it all out, but this was a great start. I was able to very easily set up the free enterprise license since it seems highly unlikely I will be ingesting over 5gb/day in my homelab.

@quebirt Год назад

Thanks again for another really good breakdown using real world and human understandable examples.

@FireBean8504 3 года назад

Thank You for showing this piece of software. I was working on setting up an ELK stack for just syslogs and is has been a few days utter failure and making me question my chosen profession and my proficiency at it. I have chosen to take a different route for logs because of the sheer admin cost. It's just two of use for 4500+ Customers and 100 Employee's.

@Mister_Morise 8 месяцев назад

So helpful! Great tuto! New sub in here. Greets from Uruguay.

@tommy11sixorozco9 3 года назад

We use this at my work. It’s dope.

@MicheeNzamununu 2 года назад

I set this up in 2016, we had 3 customers all sending logs to centralized Graylog server; it was fun!

@willblanton3120 3 года назад

I’ve played around a bit now and I’ve found you can really set a single “syslog” input for multiple servers. Then you create the index and streams. But when you create the stream “rule”, you can use the “gl2_remote_ip” field to only filter by certain syslog sources. So for pfSense, it would be the router. And for any unifi devices, it’s the IP of the device itself (AP, switch, etc). You can set the stream to be a so for device, or a group. This way you don’t have to have a separate input with a unique port number for EVERY remote server :)

@beerfarmer1828 2 года назад

Thanks, I was just wondering how to mitigate this problem. Your explanation was perfect.

@cobaltclaudia 3 года назад

This is awesome!! Great video!

@MrITGuy 3 года назад

Hi beb

@russellbaker4256 3 года назад

Ran a Graylog VM and couldn't figure out why it wouldn't ingest my ESXi, TrueNAS and NetScaler logs. I imagine it was the extractor, stream, index architecture that I didn't understand. Great job of addressing that upfront and not just going thru a procedural next, next, next configuration

@sheepd0gonwatch215 Год назад

I've been using Graylog at many of my customers for a few years now. Excellent product. I've been able to setup some really informative dashboards and alerts. It works well after you make a few tweaks. One thing I found is to make sure to adjust the heap size to get good performance. Other than that, it works great. We are ingesting Windows logs, NAS storage logs, WIFI AP logs, Firewall and Switch logs and VMware logs. The difficult part is narrowing down the scope of the data to the things you really need, but once you have it you can build dashboards that provide concise information. I have been using the grok patterns to categorize data from firewalls and it helps to build more informative dashboards and allow greater flexibility in presenting the data. Excellent tutorial.

@Sladeofdark 2 года назад

can't wait to use this !!

@brandonbrooks2845 2 года назад

Been wanting to move away from Splunk for a while, thanks for hitting the high points!!

@Feed9Will 3 года назад

Great & Useful vid

@mmobini1803 3 года назад

Thank you Tom.

@ZAndrsn 3 года назад

Thanks for this video Tom! I was just starting to work on this. I’d love to see a video that is specifically about getting Suricata logs into Graylog if that’s something you’re interested in!

@ZAndrsn 3 года назад

I am also hoping to export Suricata events to Grafana for visualization if that's something you're interested in exploring.

@mariorodrigoblock1023 2 года назад

Hi, I'm trying to send the suricata logs on pfsense to graylog and then show into grafana. but not luck yet. Only can show filterlog logs but not from suricata. if someone have this done I appreciate some kind of help. Thanks a lot

@nycksw 2 года назад

Great video.

@rodneyaltamera4057 3 года назад

Nice video. I'll try this on my network and see what it can do.

@ithelpers4625 2 года назад

very cool.. Thank you!

@stijnphilips 3 года назад

Awe-some, Tom!

@WhatAmIGonnaGet Год назад

Nice tutorial. Thank you for not spending the first 30 minutes explaining your life history, begging to be excused for not posting on RU-vid, a tour of your house, with 10 minutes of please like and subscribe

@SuperChelseaSW6 3 года назад

Make more vids bout graylog. Keep it up . We will help you out with the subs 👊🙏

@cryptolicious3738 3 года назад

very interesting! is there a log4j adapter? work? will there be a AI Threat analysis followup video?

@DJ-Manuel 3 года назад

Very good explaining video. I have one question: I have multiple servers / raspberry pis where i want to get the syslogs, however with 100 raspis, i dont like to create 100 different inputs with different ports. Can different hosts ude the same port and can the graylog distinct between the indiviual sources? In the end, all the data of those devices can land in the same location, with some filter to separate out some specific messages to be saved in another location, however i would need to know where the logentry came from, when using the same port. Is this possible?

@LAWRENCESYSTEMS 3 года назад

You could use one port and then parse the data by host name.

@SiBex_ovh Год назад

Good instroduction, even for GL v5 THANKS

@zparihar 3 года назад

Thanks for this! I'm also interested in how this compare to ELK and even Splunk.

@3ilson 3 года назад

You could compare for yourself... install Kibana and visualize the same data for comparison or install two instances (one Graylog and one ElasticStack) to evaluate the two. GrayLog is a bit more intuitive than setting up an ElasticStack instance and is a matter of preference. Here's a ElasticStack alternative to try and compare for yourself: github.com/pfelk/pfelk A video comparison would be great!

@MiguelCruzer 3 года назад

Graylog FTW Been using it since 2013. Pair it with fluentd and you are done.

@capricornnnn 3 года назад

Add your manual :)

@MajesticBlueFalcon 3 года назад

Ea! que tu haces por estos lares? jajaja Saludos mi pana

@whatevah666 2 года назад

I had to use RAW udp (in order to see pfsense logs) and not syslog for some readon for the logs to actually popup, it was a rough start.... I'm at pfsense 2.4.4-p3 and i can see you have other options in yours as BSDlogging in your fw, so updating it might be good/fix things for me ... :)

@shetuamin 3 года назад

How do i send log from different subnet? I created a pfsense vip for graylog server. I can ssh and ping graylog server by this ip but not able to open web gui and send log.

@limyangyi9100 Год назад

thanks, Thanos, glad you're getting into soft instead of...well...

@renanfeitosa101 3 года назад

Hey Lawnrence any recommendations on user authentication and navigation logs?

@LAWRENCESYSTEMS 3 года назад

Graylog

@renanfeitosa101 3 года назад

@@LAWRENCESYSTEMS thanks i'll try that

@PaulMansfield 3 года назад

How does this compare with splunk?

@ronaldocorrea8007 3 года назад

thanks

@berndeckenfels 3 года назад

Good Timing considering the ElasticSearch license melt down

@4polelowpass 3 года назад

ElasticSearch licence meltdown? What happened?

@berndeckenfels 3 года назад

It’s no longer open source and they blamed Amazon for it.

@3ilson 3 года назад

GrayLog utilizes ElasticSearch - what do you exactly mean?

@edgecrush3r Год назад

I see you also have Elastic Search running. would be interesting to hear what your pro's cons are vs using Greylog.

@LAWRENCESYSTEMS Год назад

Graylog is much easier to setup and maintain.

@MajesticBlueFalcon 3 года назад

Have you ever covered SIEMs? I would love to get pointed to the best ones out there, OSS if possible. Thanks!

@Yarisken12 3 года назад

@S K We use Qradar. It's a great product only very expensive.

@jonoinnz 3 года назад

Check out wazuh

@stephanietan4389 3 года назад

Hi, have you tried integrating syslog-ng to graylog ? Have tried it but the format was bad. If you have tried it, do you have any suggestions ?

@LAWRENCESYSTEMS 3 года назад

I have not, but their forums can be helpful for parsing formats.

@RossenAntonov 3 года назад

I subscribed to your channel recently and I am very glad now because of videos like this. Q: I have services sending emails. Can Graylog receive or check email? Also, can I set it up so that it alerts me if an email for a task that is scheduled was not received? Thank you!

@LAWRENCESYSTEMS 3 года назад

It can send an email based on parameters that you define. www.graylog.org/features/alerting

@RossenAntonov 3 года назад

@@LAWRENCESYSTEMS I have services and routers sending email messages when an event appears. I was wondering if Graylog can extract those messages via IMAP/POP3 and analyse them?

@MultiTechspec 3 года назад

wonder how Graylog compares with solutions such as Kafka

@aqkhana2002 2 года назад

Question to Master : how can i secure graylog to only receive secured message ? i do not want every one can use my graylog server to send message , please HELP !!!!

@dechmusic 2 года назад

Many-many-many-many thanks! Very good tutorial! Still curious, what is in "Memez" bookmark folder xD

@LAWRENCESYSTEMS 2 года назад

My meme stash

@farcryf 3 года назад

👍😁 since 2018

@SuperChelseaSW6 3 года назад

Make a demo bout geo iplocation dashboard. Thsnks!

@jamess1787 3 года назад

Last time I used elastics, I cried every 3 months having to manually rotate the DB.

@Gaming_qube 3 года назад

Interesting video, I decided to create my own laboratory! What about alerts? How do I make an alert for a specific log?

@LAWRENCESYSTEMS 3 года назад

docs.graylog.org/en/4.1/pages/alerts.html#alerts

@lauramaizels1967 3 года назад

I am listening to someone speak about graylog and said the words star 410 or start up 410 . something 410. Do anyone know what that is?

@LAWRENCESYSTEMS 3 года назад

¯\_(ツ)_/¯

@jimchisholm2433 2 года назад

Great video.. but on a default Ubuntu 21 Graylog install I've found that using port 514 results in "permission denied" as the lower ports are restricted to all but root users (which the Graylog server apparently is not running as) My pfsense will not send logs on any other port than 514, despite what may be entered in the System Logs settings. Sio I've configured Graylog's input to port 1514 and set the server input like this: iptables -t nat -A PREROUTING -p tcp --dport 514 -j REDIRECT --to 1514 iptables -t nat -A PREROUTING -p udp --dport 514 -j REDIRECT --to 1514 Oh goody. Now I'm flooded with logs in Graylog :/

@Mr.Leeroy 3 года назад

So it just accepts entries on specific port or is there any auth? How do I know that gathered data is legit?

@LAWRENCESYSTEMS 3 года назад

Syslog does not use auth, but some of the other support input types do.

@Mr.Leeroy 3 года назад

@@LAWRENCESYSTEMS Tell me that it is not plain text at least..

@LAWRENCESYSTEMS 3 года назад

@@Mr.Leeroy You should probably read up a bit more on how syslog works.

@Mr.Leeroy 3 года назад

@@LAWRENCESYSTEMS it has been on 2do list for far too long.

@3ilson 3 года назад

@Leeroy - Syslog was created in the 1980s and by default does not encrypt, transmitting everything in the clear. However, Syslog-NG is capable of transmitting over TLS via TCP. Additionally, Syslog-NG was developed to add additional security and filtering options. This same setup is feasible (replacement to syslog) and capable of leveraging TLS (e.g. encryption).

@JB-mn4ff Год назад

vAppliance no longer available/supported from Graylog: 4:00 "no time commitment loading a VM." ☹

@RuchiSharma-hv2xk Год назад

I am completely newbie in logging analysis and this domain, can you please tell me if Graylog is SAAS based solution in any way? I mean when we ingest log data for analysis do we need to ingest it in their SAAS platform?

@Zwiesel66 Год назад

As he already told us, it is not, because you can run it on your own server.

@pmsrodrigues 3 года назад

SYSLOG appears to be a no-brainer. But what about Windows server logs?

@LAWRENCESYSTEMS 3 года назад

Yes docs.graylog.org/en/4.0/pages/sending/windows.html

@SuperChelseaSW6 3 года назад

What version on graylog are you using?

@LAWRENCESYSTEMS 3 года назад

GRAYLOG 4.0

@MirkWoot 3 года назад

I don't know if UniFi would report switch MAC/port changes over syslog?. Try getting a Cisco switch to report MAC table 😅 to graylog, omg. I spent 2 last days trying to.. syslog, nope.. Then SNMP should be possible, but SNMP graylog plugin refuses to play nice, least with cisco MIBS... OK, i get it thru Telegraf then it has options for that. Switch sends snmp notifications to Telegraf, and Telegraf reports back to graylog (it has nice and simple output for that), but nope, at most i can see something changed, but not the MAC... so far that i can see. It's possible however to request the mac table over snmp, but needs some serious parsing to understand which port its on.. but/and then youd haft to do that every minute, instead of just getting a notification.

@jus4027 Год назад

t-shirt made me laugh.

@mrraspberry8263 Год назад

Ever made a comparison between ELK-Stack and Greylog?

@LAWRENCESYSTEMS Год назад

Nope, don't really plan to

@attilavidacs24 3 года назад

Does this integrate with Active Directory?

@BenTyger 3 года назад

Yes, via LDAP.

@djdawso 3 года назад

I might be able to help you with some regex stuff, so hit me up if you still have questions.

@bryan.bayesian Год назад

Graylog tweeted your "almost done my 2023 Graylog update, need some help with an issue" tweet. Docker seems to be the way these days. I saw that you were able to fix the issue you were having with the config file. Are you going to release a tutorial soon?

@LAWRENCESYSTEMS Год назад

Yeah, hoping to get it done by early next week.

@bryan.bayesian Год назад

@@LAWRENCESYSTEMS awesome, thank you!

@SB-qm5wg 3 года назад

Now if it had SNMP too.

@coldhardwick 3 года назад

How is it you always post a video for a solution right when I'm looking for a solution?

@loupalladino9578 3 года назад

Same here!

@frostfire462 3 года назад

Very neat product, I actually work with Splunk. This is super cool but doesn't have all the features that Splunk does. You should totally take a look at it, I know it is closed source but it is a damn good product.

@foobarturkey 3 года назад

Expensive as frig

@frostfire462 3 года назад

@@foobarturkey You are telling me lol. They have some trial license floating around that lets you do some stuff at home.

@neilbroomfield3080 2 года назад

Looks like they've changed the offering around enterprise and free 5GB. Looks like it might only be 2GB now.

@michaelperugini4199 2 года назад

not finding an ova for use

@ctownskier 3 года назад

Tom, regex101.com is my go to for testing out new expressions.

@LAWRENCESYSTEMS 3 года назад

Thanks I will check that out, I have been using regexr.com/

@louisdodge104 2 года назад

Unfortunately Graylog has removed the pre-built Virtual Machine Appliance downloads from the website.

@LAWRENCESYSTEMS 2 года назад

Yup. but they do have docker images

@Tntdruid 3 года назад

Does it need a lot of ram?

@emanuelmilani7976 3 года назад

Yes. Elastisearch is a big problem. One server is not enough even for relative small logs docs.graylog.org/en/4.0/pages/configuration/elasticsearch.html We strongly recommend to use a dedicated Elasticsearch cluster for your Graylog setup

@Tntdruid 3 года назад

@@emanuelmilani7976 ah Thanks

@sku2007 3 года назад

have it running in proxmox as CT container, with 4gb ram and data on zfs mirror with two 6-10 year old HDD. syslog from various vm, physical linux machines, raspberries and opnsense (but without every firewallevent). use only default index set, after about one year there are 6.8GiB of data. query works just fine, nothing to complain. so I would say, give it a try!

@Tntdruid 3 года назад

@@sku2007 will do, thanks

@harjeetsinghramgharia6493 3 года назад

Recently configured myself in production. Works perfectly. github.com/pfelk/pfelk

@mochammadawaludin2073 2 года назад

This all use free or enterprise...? because all view different with free mode

@nekooolay 2 года назад

Can this be run on windows machine?

@LAWRENCESYSTEMS 2 года назад

Don't think so

@ItsQuintFX 3 года назад

tried it, it's nice, but they need to resolve the timezone issue if that hasn't been solved already

@LAWRENCESYSTEMS 3 года назад

Not an issue that I had so maybe it was resolved.

@MiguelCruzer 3 года назад

What timezone issue?

@ItsQuintFX 3 года назад

@@MiguelCruzer I live in the Netherlands and my timezone is thus gmt+1 , at the time the docker image only supported utc so I had to modify the dockerfile and re-compile it.

@MiguelCruzer 3 года назад

@@ItsQuintFX That sounds like a config issue, I'm not sure if this is new or not but docs say any config option can be passed as an ENV_VAR preceded with the GRAYLOG_ name. docs.graylog.org/en/4.0/pages/installation/docker.html#configuration Hope this helpsl

@atraxotoxin3840 2 года назад

Can you do an video on putting graylog logs into grafana. That would be much appreciated.

@LAWRENCESYSTEMS 2 года назад

I don't use grafana so not likely.

@atraxotoxin3840 2 года назад

@@LAWRENCESYSTEMS thank you for your response. If you happen to change things up and start working with grafana. You can send graylog logs to grafana and have beautiful siem graphs.

@atraxotoxin3840 2 года назад

Then you can integrate zabbix with grafana and pretty much have a one stop shop.

@atraxotoxin3840 2 года назад

The whole idea here is A.I for cyber security. With the graphs and the alerting system. Have scripts programmed in python or an other language to react to it. That's my project I'm working on.