Using NETCONSOLE to debug Linux (and Proxmox) Kernel Panics

Подписаться 59 тыс.

Просмотров 3,4 тыс.

50% 1

In this video, I'm going to setup Netconsole, so you can capture kernel panics and logs on headless systems. I know some of you are doing wild things with graphics drivers and iGPU passthrough, so hopefully this helps you debug them when your other display outputs aren't working.
Blog with commands:
www.apalrd.net/posts/2024/pve...
Support me on Ko-Fi if you enjoy my content and find it useful:
ko-fi.com/apalrd
Feel free to chat about my upcoming projects on Discord!
/ discord
Timestamps:
00:00 - Introduction
00:45 - Overview
02:44 - Simple Setup
04:56 - Log on Boot
09:20 - Systemd Journal

Наука

Опубликовано:

17 июл 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 20

@LampJustin 12 дней назад

Btw the systemd journald also logs kernel messages, which is quite helpful when they get truncated by dmesg or when the system got rebooted as you show the logs from the last boot: journalctl -b -1 -ke

@blevenzon 12 дней назад

Thanks so much, you rock dude. Awesome content as always

@AntranigVartanian 12 дней назад

Somehow doing this on Solaris 10, around 20 years ago, was much simpler and debugging was easier as well. Good to see Linux catching up.

@autohmae 10 дней назад

This is how easy it has been since 2001, so 'catching up' is a big word, nothing changed. BTW Warning this might make you feel old: Solaris 10 was from 1992, 32 years ago.

@drunkbear889 12 дней назад

Another viewpoint: If using a P2P L3 link to the server (from a router, or L3 switch). I would not see a problem with leaving it on all the time. Some switches can can do inbound rules on the interface connected to the server for forwarding packets based on filters to only select destinations (think MT CRS devices). Such that even devices on the same LAN segment, will not receive the broadcast packets destined to ff:ff:ff:ff:ff:ff (switch chip ACL rules). (~port mirroring, but filtered for very specific traffic)

@apalrdsadventures 12 дней назад

It's also possible to not use the broadcast mac if you want to leave it on all the time, you then need to set both the unicast IP and unicast MAC since it doesn't do ARP / NDP. I also tried using multicast and that works fine on the Netconsole side, but receiving multicast is a lot harder than netcat, so I didn't go down that path. That may be easier to filter depending on your switch.

@someoneelse3876 12 дней назад

I got a lot of kernel panics when I switched from iscsi to nvme over fabrics/tcp, I gave up and went back to regular iscsi.

@PCMagikHomeLab 12 дней назад

Thumbs UP! :)

@ws_stelzi79 12 дней назад

well *someone* "had got" the need to make a video on an advanced Linux/kernel debugging! 😉😇

@NiHaoMike64 12 дней назад

Wouldn't serial console also be a good option if the machine in question has a serial port?

@apalrdsadventures 12 дней назад

Yes, and I have a video on that too. A lot of machines don't have them, though.

@autohmae 10 дней назад

@@apalrdsadventures every Linux VM on KVM/Qemu does of course... what would be a good serial console server software which has the option to log everything ? A quick look around I think I can probably just run a bunch of stty processes to capture the logs.

@apalrdsadventures 10 дней назад

For VMs of course, but when you are doing experimental hardware passthrough you tend to crash the host kernel

@autohmae 10 дней назад

@@apalrdsadventures ahh, that's sad, that shouldn't really be able to happen.

@apalrdsadventures 10 дней назад

Usually it's because the host kernel was already using that device and you took it away improperly to give to the VM. If you do that with the primary GPU, then you have no graphics device to debug from either.