Reading a Crash the Way You Read a Core Dump

A core dump looks like the least friendly file on your disk, and most of its reputation is undeserved. It's a photograph of a process at the moment it died. You don't need to read all of it. You need three facts, in order, and the first minute of reading goes to finding them.

The checklist

1. The signal. What killed the process? Segmentation fault, abort, bus error. This tells you the family of the bug before you look at any code.

2. The faulting address. If it's zero or very small, something dereferenced a null pointer. If it's huge and strange, something wrote garbage over a pointer. If it's near a valid address, think off-by-one.

3. The frames. Frame zero is where it died. Frame one is who called it. The cause is often one frame up from where it died, because that's where a bad value was handed down.

Debuggers print these facts from a core file. On the laptop I wrote this on, core files are switched off (ulimit -c prints 0) and the system core directory isn't writable, so I used a sanitizer instead. It prints the same three facts at the moment of the crash, which makes it a good place to practise.

A crash to read

Fifteen lines of C, with the bug a code review would catch and a tired person at 2 a.m. would not:

#include <stdio.h>
#include <stdlib.h>

struct job { int id; char *name; };

static int name_length(const struct job *j) {
    int n = 0;
    while (j->name[n] != '\0') n++;
    return n;
}

int main(void) {
    struct job j = { 7, NULL };
    printf("job %d\n", j.id);
    return name_length(&j);
}

Built plainly and run in a terminal, it prints job 7 and the shell reports exit status 139, which is 128 plus signal 11. That's the signal, already. (Redirect the output to a file and job 7 never appears: the line was still in the stdio buffer when the process died. That's a small lesson of its own about trusting logs from a crash.) Built with the sanitizer, the report gives the rest:

cc -g -O0 -fsanitize=address -fno-omit-frame-pointer -o crash_asan crash.c && ./crash_asan

ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000
The signal is caused by a READ memory access.
Hint: address points to the zero page.
    #0 0x... in name_length crash.c:8
    #1 0x... in main crash.c:15
SUMMARY: AddressSanitizer: SEGV crash.c:8 in name_length

I trimmed the ==pid== prefixes and the addresses, which differ on every run, plus a third frame inside the system loader and the register dump that follows.

Reading it

Signal: segmentation fault. Address: zero, and the access was a read, so something followed a null pointer. Frame zero is name_length, line 8, which reads j->name[n]. Frame one is main, line 15, the call.

Notice that frame zero is where it died and not where it went wrong. The bug is on line 13, where main stored NULL into name and handed the struct to a function that trusted it. The null was created one frame up and detonated one frame down. That's the pattern worth remembering, and it holds whether the report comes from a sanitizer, a debugger or a core file from production.

What to do next

Don't fix line 8. Ask who is allowed to call name_length with a null name, and make that impossible at the boundary. Either validate on entry, or change the type so a job can't exist without a name. Then keep the crash as a test, because a bug you found by reading a dump is exactly the one that will come back.