Cold Start

Reversing a 64-bit ELF binary to extract and brute-force a Frankenstein hash for a 24-bit activation seed.

Reverse EngineeringBDSecCTFMedium

1. Challenge Description

The system lost its activation seed during shutdown. Recover the correct value and restore the boot sequence. Flag Format: BDSEC{s0mething_he3r}

We're given a single file, cold_start.bdsec, which is actually a stripped ELF binary in disguise.

$ file cold_start.bdsec
cold_start.bdsec: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV),
dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2,
BuildID[sha1]=aca3bc58dd60d91bfc452c092d38718dd91d49f8,
for GNU/Linux 4.4.0, stripped

Key facts:

2. Static Analysis

Decompiling the single interesting function (FUN_001010d0, effectively main's logic) in Ghidra gives a wall of bit-twiddling code. Reading through it, the function breaks into four clear stages:

Stage 1 — Input validation

fgets(local_88, 0x40, stdin);
strcspn(...);                 // strip newline
if (strlen(local_88) == 6) {  // must be exactly 6 characters
    // every character must be a hex digit (isxdigit via __ctype_b_loc, mask 0x10)
    uVar9 = strtoul(local_88, &local_220, 0x10);
    if (*local_220 == '\0' && uVar9 < 0x1000000) {
        ...

So the "activation seed" must be exactly 6 hex digits, parsed as a value strictly less than 0x1000000 (16,777,216). This single constraint is the whole ballgame: the entire keyspace is only ~16.7 million values, which is trivially brute-forceable once the check function is fast to evaluate.

Stage 2 — A custom 96-round mixing/hash function

The seed (called uVar13/uVar9 in the decompilation) is churned through a loop that runs exactly 0x60 (96) times, filling an internal state array local_218[0..95]. Each round:

Stage 3 — Three verification checks

After the 96 rounds, three values are derived from specific indices of the state array and compared against hard-coded constants:

// 16-bit checksum, must equal 0x9c8c
(ushort)( (local_218[7]>>5) ^ (local_218[0x37]>>0xb) ^ local_218[0x5b] ^ uVar9 )

// two more 32-bit finalizer-mixed values
fin17 = local_218[0xb] ^ local_218[0x53] ^ rotl(local_218[0x2f], 9) ^ uVar17;
fin17 = finalize(fin17);           // must fold to 0x91e50c54

fin11 = rotl(local_218[0x44], 13) + local_218[0x17] + uVar15 + uVar11;
fin11 = finalize(fin11);           // must fold to 0xc2e4f8bd

Only if all three checks pass does it print System restored. and proceed to stage 4. Otherwise: Boot sequence rejected.

Stage 4 — Flag generation

If validation succeeds, a second loop runs 42 times (0x2a), producing one output byte per iteration by XOR-ing together:

The output bytes are the ASCII flag characters, printed one by one.

The important realization: the flag isn't hidden anywhere in the binary as a static string — it's computed from the seed and only exists after the seed is known, because local_218 (which stage 4 reads from) is derived entirely from the seed in stage 2.

3. Solving Strategy

Given:

The plan: faithfully reimplement stage 2 + stage 3 in C, brute-force every seed 0 → 0xFFFFFF, and run stage 4 on whichever seed passes.

This is much faster and more reliable than trying to invert the hash algebraically (it's non-linear and irreversible by design), and far faster than spawning the actual binary as a subprocess 16.7 million times.

3.1 Extracting the lookup tables

The decompiled code references two globals, DAT_00102140 (256×4-byte table used throughout the hash) and DAT_00102100 (a 42-byte table used only in flag generation). Ghidra's addresses for PIE binaries are offset by a fixed image base (0x100000), so the real virtual addresses are 0x2140 and 0x2100.

$ objdump -h cold_start.bdsec | grep rodata
13 .rodata   00000540  0000000000002000  0000000000002000  00002000  2**5

.rodata spans file offset 0x20000x2540, so both tables land cleanly inside it (and DAT_00102140's 256×4 = 1024 bytes end exactly at 0x2540, confirming the boundary). I extracted both tables directly from the file:

data = open('cold_start.bdsec', 'rb').read()
tab1 = data[0x2100:0x2140]   # 64 bytes (only first 42 used) — flag XOR table
tab2 = data[0x2140:0x2540]   # 256 x uint32 LE — hash lookup table

3.2 Reimplementing the algorithm — the operator-precedence trap

Ghidra's decompiler output is valid C, which means it's easy to copy-paste, but it also means C's operator precedence rules are load-bearing and easy to misread when skimming. Two lines were particularly nasty:

uVar15 = (uVar13 * 0x1bbcd880 | uVar13 * -0x61c8864f >> 0x19) ^ 0x1b873593;

At first glance this looks like the ^ 0x1b873593 might bind to just the second half of the | expression. It doesn't — the whole (A | B) is inside explicit parentheses, so the XOR applies to the entire OR'd result. My first implementation got this backwards (A | (B ^ C) instead of (A | B) ^ C), which silently produced a wrong hash for every seed and made the initial brute-force run find zero matches. Re-reading the parenthesization carefully and fixing it to:

uint32_t part1 = uVar13 * 0x1bbcd880u;
uint32_t part2 = (uVar13 * (uint32_t)(-0x61c8864f)) >> 0x19;
uVar15 = (part1 | part2) ^ 0x1b873593u;

...immediately fixed it. This is the single most important lesson from this challenge: when transcribing Ghidra output, trust the explicit parentheses over your intuition about "what the hash probably does."

The other subtlety was the second lookup index:

(uVar1 >> 8) + (uVar4 & 0xff) + (int)uVar14 & 0xff

+ binds tighter than &, so this is ((uVar1>>8) + (uVar4&0xff) + uVar14) & 0xff, not (uVar1>>8) + (uVar4 & (0xff & (int)uVar14)) or similar — and critically, the uVar4 referenced here is the old value from earlier in the same statement block, before it gets reassigned two lines later.

3.3 The full C reimplementation

#include <stdio.h>
#include <stdint.h>
#include <string.h>

static uint32_t table2[256];   // DAT_00102140
static uint8_t  table1[64];    // DAT_00102100

static inline uint32_t ROTL(uint32_t v, uint32_t s){
    s &= 31; return s ? (v << s | v >> (32 - s)) : v;
}

int compute(uint32_t uVar9, uint32_t local_218[96]){
    uint32_t uVar13 = uVar9;
    uint32_t uVar17 = uVar13 ^ 0xa3c59ac3u;
    uint32_t iVar20 = 0, uVar19 = 0, uVar18 = 0, uVar14;
    uint32_t uVar11 = (uVar13 << 5) ^ (((uint32_t)(uVar9 >> 3)) & 0x1fffffffu) ^ 0x811c9dc5u;

    uint32_t uVar15;
    {
        uint32_t part1 = uVar13 * 0x1bbcd880u;
        uint32_t part2 = (uVar13 * (uint32_t)(-0x61c8864f)) >> 0x19;
        uVar15 = (part1 | part2) ^ 0x1b873593u;      // <-- the tricky grouping
    }

    uint32_t uVar1, uVar4, uVar16;
    uint8_t  bVar3;

    for (uVar14 = 0; uVar14 != 0x60; uVar14++){
        bVar3 = (uint8_t)(uVar14 & 0x1f);
        uVar4 = uVar11 ^ uVar17 ^ uVar18 ^ ROTL(uVar15, bVar3);
        uVar1 = table2[uVar4 & 0xff];
        uVar17 = (uVar19 ^ uVar13) + uVar17 + uVar1;
        bVar3 = (uint8_t)(((uint8_t)(uVar1 >> 0x1b) + 1) & 0x1f);
        uVar17 = ROTL(uVar17, bVar3);

        uint32_t idx2 = ((uVar1 >> 8) + (uVar4 & 0xff) + (int32_t)uVar14) & 0xff;
        uVar4 = uVar11 + iVar20 + uVar17 + table2[idx2];
        uVar4 = (uVar4 >> 0x10 ^ uVar4) * 0x7feb352du;
        uVar4 = (uVar4 ^ uVar4 >> 0xf) * 0x846ca68bu;    // == -0x7b935975 (two's complement)

        uVar16 = uVar15 ^ uVar4;
        uVar15 = uVar16 ^ (uVar4 >> 0x10);

        bVar3 = (uint8_t)(((uVar14 % 13) + 1) & 0x1f);
        uVar18 += 0x45d9f3bu;
        uVar19 += 0x27d4eb2du;
        iVar20 += 0x9e3779b9u;                            // == -0x61c88647

        uVar11 = (ROTL(uVar15, bVar3) ^ uVar17) + uVar1 + uVar11 * 0x1000193u + uVar14;
        local_218[uVar14] = (uVar15 << 0xb | uVar16 >> 0x15) ^ uVar17 ^ uVar11;
    }

    uint32_t fin17 = local_218[0xb] ^ local_218[0x53] ^ ROTL(local_218[0x2f], 9) ^ uVar17;
    fin17 = (fin17 >> 0x10 ^ fin17) * 0x7feb352du;
    fin17 = (fin17 ^ fin17 >> 0xf) * 0x846ca68bu;

    uint32_t fin11 = ROTL(local_218[0x44], 0xd) + local_218[0x17] + uVar15 + uVar11;
    fin11 = (fin11 >> 0x10 ^ fin11) * 0x7feb352du;
    fin11 = (fin11 ^ fin11 >> 0xf) * 0x846ca68bu;

    uint16_t chk = (uint16_t)((uint16_t)(local_218[7] >> 5) ^ (uint16_t)(local_218[0x37] >> 0xb)
                             ^ (uint16_t)local_218[0x5b] ^ (uint16_t)uVar9);

    return chk == 0x9c8c
        && (fin17 ^ fin17 >> 0x10) == 0x91e50c54u
        && (fin11 ^ fin11 >> 0x10) == 0xc2e4f8bdu;
}

void gen_flag(uint32_t uVar13, uint32_t local_218[96], char *out){
    uint32_t uVar11 = 0x5a;
    uint64_t uVar9 = 0, uVar14 = 0xb, uVar21 = 7;
    int pos = 0;
    do {
        uint64_t uVar12 = uVar9 + 1;
        // classic "multiply-by-reciprocal" unsigned divide-by-3 trick, done by hand:
        uint64_t high  = ((__uint128_t)uVar9 * 0xaaaaaaaaaaaaaaabULL) >> 64;
        uint8_t  X     = (uint8_t)(high & 0xff) & 0xfe;
        int8_t   Y     = (int8_t)(uVar9 / 3);
        int8_t   diff  = (int8_t)uVar9 - (int8_t)(X + Y);
        uint32_t shift = ((uint32_t)((int32_t)diff * 8)) & 0x1f;

        uint32_t li14 = local_218[uVar14 % 0x60];
        uint32_t li21 = local_218[uVar21 % 0x60];

        uint8_t inner = (uint8_t)(((int8_t)uVar9 * (int8_t)'%') ^ table1[uVar9]
                                 ^ (uint8_t)li14 ^ (uint8_t)(li14 >> 8));

        uVar11 ^= (uint32_t)inner ^ (li21 >> 0x10) ^ (li21 >> 0x18) ^ (uVar13 >> shift);
        out[pos++] = (char)(uVar11 & 0xff);

        uVar9 = uVar12; uVar14 += 0x11; uVar21 += 0x1d;
    } while (uVar9 != 0x2a);
    out[pos] = 0;
}

int main(){
    FILE *f1 = fopen("tab1.bin","rb"); fread(table1,1,64,f1);  fclose(f1);
    FILE *f2 = fopen("tab2.bin","rb"); fread(table2,4,256,f2); fclose(f2);

    uint32_t local_218[96];
    for (uint32_t seed = 0; seed < 0x1000000; seed++){
        if (compute(seed, local_218)){
            char flag[64];
            gen_flag(seed, local_218, flag);
            printf("SEED FOUND: 0x%06x\nFLAG: %s\n", seed, flag);
        }
    }
    return 0;
}

Compiled with gcc -O2 and run against the two extracted .bin table files, this sweeps the entire 16.7M keyspace in well under a second.

4. Result

$ gcc -O2 -o crack crack.c
$ ./crack
SEED FOUND: 0x93c7a4
FLAG: BDSEC{th3_k3y_w4s_s0m3wh3r3_1n_16_m1ll10n}

Exactly one seed in the entire ~16.7 million-value space satisfies all three checks — as expected for a well-designed 32/16-bit hash comparison, collisions are astronomically unlikely.

5. Verification against the real binary

To be certain the reimplementation is correct (not just "a" solution that happens to satisfy the same three numeric checks by luck of my own reimplementation bugs), I fed the recovered seed straight into the original binary:

$ printf "93c7a4\n" | ./cold_start.bdsec

========================================
               COLD START
========================================

The system was powered down unexpectedly.

activation seed> System restored.
BDSEC{th3_k3y_w4s_s0m3wh3r3_1n_16_m1ll10n}

Confirmed — matches exactly.

6. Takeaways

Easy RE
The Bytecode Vault