In this post (and video) I’m going to setup Netconsole, so you can capture kernel panics and logs on headless systems. I know some of you are doing wild things with graphics drivers and passthrough, so hopefully this helps you debug them.

Enable Now⌗

This option enables the module immediately, so you can use it before you do dangerous things. You Simply rebooting clears the setting, so you won’t continue to spam your kernel messages on the local network.

modprobe netconsole netconsole=@/eth0,@<destination_IP_address>/

If you want the full syntax of the argument, you can find it here. I’m leaving a lot of things as defaults, so it will use the broadcast MAC and UDP port 6666.

And to disable it without rebooting:

rmmod netconsole

Enable On Boot⌗

This will take effect on the next boot. Similar arguments to above, except we need to add this to the kernel commandline. Also, since the interface won’t have this addresses early in boot the process, it’s easiest if we use IPv6 link-local addresses. The kernel will generate one from the MAC address. The other option if you want to use legacy IPv4 is to set a source IP address in the kernel command line, this does not have to be the same address you use in the system.

On Debian systems and some Proxmox systems, the file we need to edit is /etc/default/grub. On Proxox systems using UEFI boot but not secure boot, the file we need to edit is /etc/kernel/cmdline. If you don’t have an /etc/kernel/cmdline, then you need to edit /etc/default/grub. Anyway, add this to the command line:

netconsole=@/eth0,@<destination_IPv6_LL_address>/

Then we need to run either update-grub (if you used /etc/default/grub) or proxmox-boot-tool refresh (if you used /etc/kernel/cmdline, and possibly on any Proxmox systems, it doesn’t hurt)

When you reboot, you can check dmesg | grep netcon to see if it started or if there were errors. If your Ethernet devices goes by multiple names in the boot process, try to dmesg | grep eth0 and see what other names it goes by, and try those if it fails.

Receive Netconsole with Netcat⌗

Now, we need to setup Netcat to listen to that broadcast. It’s sending it to the broadcast MAC address (all F’s), but to a unicast IP address, so anyone on the network will see this traffic with Wireshark. But netcat is easier. The default netcat version with Debian doesn’t support IPv6, but you can apt install ncat for a better version, and it will install itself as the nc command.

#Call netcat to listen
#-l = listen
#-u = udp
#-p 6666 = the default port for netconsole 
nc -l -u -p 6666

This can run as long as you want to monitor the netconsole output.

Receive Netconsole with a simple systemd unit⌗

This is a super simple systemd service which will run netcat and save it to the systemd journal. Useful for cases when your crash is intermittent. You can run this on any machine on the network (of course you need to have netconsole set for this IP as well). Save this as /etc/systemd/system/netconsole-log.service:

[Service]
ExecStart=/usr/bin/nc -l -u -p 6666

[Install]
WantedBy=multi-user.target

Then start it:

systemctl daemon-reload
systemctl enable --now netconsole-log

And view the logs:

journalctl -xeu netconsole-log

Panic⌗

If you want to panic the system, then you can send the sysreq ‘c’ (as root):

echo c > /proc/sysrq-trigger

Here’s a list of all the sysreqs you can trigger for fun.