Debugging on Flatcar Container Linux

Collecting crash logs on Flatcar Container Linux

Mon, 01 Jan 0001 00:00:00 +0000

In the unfortunate case that an OS crashes, it’s often extremely helpful to gather information about the event. There are two popular tools used to accomplished this goal: kdump and pstore. Flatcar Container Linux relies on pstore, a persistent storage abstraction provided by the Linux kernel, to store logs in the event of a kernel panic. Since this mechanism is just an abstraction, it depends on hardware support to actually persist the data across reboots. If the hardware support is absent, the pstore will remain empty. On AMD64 machines, pstore is typically backed by the ACPI error record serialization table (ERST) or EFI variables.

Debugging tools on Flatcar Container Linux

Mon, 01 Jan 0001 00:00:00 +0000

You can use common debugging tools like tcpdump or strace with Toolbox. Using the filesystem of a specified Docker container Toolbox will launch a container with full system privileges including access to system PIDs, network interfaces and other global information. Inside of the toolbox, the machine’s filesystem is mounted to /media/root.

Quick debugging

By default, Toolbox uses the stock Fedora Docker container. To start using it, simply run:

/usr/bin/toolbox

NOTE: For Fedora, it’s recommended to use at least 2048 MB RAM to avoid the following dnf operation being killed by the OOM manager.

Reading the system log

Mon, 01 Jan 0001 00:00:00 +0000

journalctl is your interface into a single machine’s journal/logging. All service files insert data into the systemd journal. There are a few helpful commands to read the journal:

Read the entire journal

$ journalctl

-- Logs begin at Fri 2013-12-13 23:43:32 UTC, end at Sun 2013-12-22 12:28:45 UTC. --
Dec 22 00:10:21 localhost systemd-journal[33]: Runtime journal is using 184.0K (max 49.9M, leaving 74.8M of free 499.0M, current limit 49.9M).
Dec 22 00:10:21 localhost systemd-journal[33]: Runtime journal is using 188.0K (max 49.9M, leaving 74.8M of free 499.0M, current limit 49.9M).
Dec 22 00:10:21 localhost kernel: Initializing cgroup subsys cpuset
Dec 22 00:10:21 localhost kernel: Initializing cgroup subsys cpu
Dec 22 00:10:21 localhost kernel: Initializing cgroup subsys cpuacct
Dec 22 00:10:21 localhost kernel: Linux version 3.11.7+ (buildbot@10.10.10.10) (gcc version 4.6.3 (Gentoo Hardened 4.6.3 p1.13, pie-0.5.2)
...
1000s more lines

Read entries for a specific service

Read entries generated by a specific unit:

Performing manual Flatcar Container Linux rollbacks

Mon, 01 Jan 0001 00:00:00 +0000

In the event of an upgrade failure, Flatcar Container Linux will automatically boot with the version on the rollback partition. Immediately after an upgrade reboot, the active version of Flatcar Container Linux can be rolled back to the version installed on the rollback partition, or downgraded to the version current on any lower release channel. There is no method to downgrade to an arbitrary version number.

This section describes the automated upgrade process, performing a manual rollback, and forcing a channel downgrade.

Working with btrfs and common troubleshooting

Mon, 01 Jan 0001 00:00:00 +0000

btrfs is a copy-on-write filesystem with full support in the upstream Linux kernel and several desirable features. In the past, Flatcar Container Linux shipped with a btrfs root filesystem to support Docker filesystem requirements at the time. As of version 561.0.0, Flatcar Container Linux ships with ext4 as the default root filesystem by default while still supporting Docker. Btrfs is still supported and works with the latest Flatcar Container Linux releases and Docker, but we recommend using ext4.