aboutskillsprojectsexperiencewriteupscontactfunnies
~/writeups/layercake

Layer Cake — Docker Layer Forensics

Investigated Docker layer history in an OCI image archive to recover a database password deleted in a later build layer.

ForensicsDockerOCIMetaCTFMedium

We're handed a file called layer_cake.tar from a fictional company called Noirtech, who helpfully reassure us that even though they sometimes put secrets in their Docker images, they removed them this time. The challenge description is basically a spoiler if you've been doing this long enough — but let's walk through it properly.

The Core Vulnerability: Deleting a file in a later layer does not remove it from the Docker image history. It merely covers it with a whiteout marker. The original data remains intact in the earlier layer blob.

Reconnaissance

First, unpack the tar and see what we're dealing with:

terminal
$ mkdir layer_cake && cd layer_cake
$ tar -xf ../layer_cake.tar
$ ls
blobs/ index.json manifest.json oci-layout repositories

This is an OCI image archive — the standard format produced by docker save. The manifest.json is the entry point.

terminal
$ cat manifest.json | python3 -m json.tool

Key findings from the manifest:

Reading the Build History

The config blob contains the full Dockerfile history. This is where things get interesting:

terminal
$ python3 -c "import json; cfg = json.load(open('blobs/sha256/6c7d365f50bf...')); [print(f'[{i:02d}] {h.get(\"created_by\",\"\")[:100]}') for i, h in enumerate(cfg['history'])]"

The full layer map (condensed):

#
Layer Hash
Command
00
f44f286046d9
ADD alpine-minirootfs-3.18.12-x86_64.tar.gz /
04
beb52a9ce8ec
apk add bash curl
05
c5a09b1f5d09
adduser app
06
9b30f91b1f23
mkdir -p /opt/app/config /opt/app/logs ...
08–13
various
COPY requirements.txt, main.py, models.py, routes.py...
14
dc2cc8ec1fca
COPY production.env → /opt/app/config/production.env ← 🔑
15
6a24d71e3999
Write migrations log
16
33a1d06f9937
Write app.env (non-secret config)
17
a9c3e1df9b0d
rm /opt/app/config/production.env ← the "fix"
18
df439fd84bd4
Write healthcheck.sh
The Smoking Gun: The secret configuration file production.env was added in layer 14, and then later deleted in layer 17. Because the file was committed to the image filesystem history in layer 14, deleting it later does not remove it from layer 14's tarball.

Extracting the Secret Layer

To retrieve the file, we can extract the layer 14 tarball directly:

terminal
$ mkdir -p extracted/dc2cc8
$ tar -xf blobs/sha256/dc2cc8ec1fcad611b11b3ddfd22fd508d15a78bf823ba806222d053f5aaff370 -C extracted/dc2cc8
$ cat extracted/dc2cc8/opt/app/config/production.env
# Noirtech Production Configuration
# !! DO NOT COMMIT — contains production secrets !!

APP_ENV=production
APP_SECRET_KEY=b3f2a1c8d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9

# Primary database
DB_HOST=db-primary.noirtech.internal
DB_PORT=5432
DB_NAME=noirtech_prod
DB_USER=noirtech_app
DB_PASSWORD=VFdWMFlVTlVSbnRrTUdOck0zSmZjbTFmWkRBemMyNTBYM0l6ZDNKcGRETmZhR2x6ZERCeWVYMD0K

Confirming the Whiteout (Layer 17)

Just for completeness, let's look at the deletion layer (17):

terminal
$ mkdir -p extracted/a9c3e1
$ tar -xf blobs/sha256/a9c3e1df9b0d228f47d7575d254c0c4f2f7a1fe4a1b2e9dd54aafbb8da69fb52 -C extracted/a9c3e1
$ find extracted/a9c3e1 | grep wh
extracted/a9c3e1/opt/app/config/.wh.production.env

The zero-byte .wh.production.env file is a whiteout marker. The Docker union filesystem reads this and hides the file from the final running container, but does nothing to purge it from previous layers.

Decoding the Flag

The DB_PASSWORD value in the production configuration was double-base64 encoded:

terminal
$ echo "VFdWMFlVTlVSbnRrTUdOck0zSmZjbTFmWkRBemMyNTBYM0l6ZDNKcGRETmZhR2x6ZERCeWVYMD0K" | base64 -d
TWV0YUNURntkMGNrM3Jfcm1fZDAzc250X3Izd3JpdDNfaGlzdDByeX0=

$ echo "TWV0YUNURntkMGNrM3Jfcm1fZDAzc250X3Izd3JpdDNfaGlzdDByeX0=" | base64 -d
MetaCTF{d0ck3r_rm_d03snt_r3writ3_hist0ry}

Flag

Decrypted from production.env
MetaCTF{d0ck3r_rm_d03snt_r3writ3_hist0ry}

What Noirtech Should Have Done

To prevent secrets leakages, Noirtech should employ one of the following methods:

  1. BuildKit Secret Mounts: Mount secrets dynamically during build time without baking them into layers. 
    RUN --mount=type=secret,id=prod_env,target=/opt/app/config/production.env python3 setup.py configure
  2. Multi-stage Builds: Copy build outputs into a fresh image layer, leaving any intermediate credential layers discarded.
  3. Runtime Injection: Keep secret files out of images entirely, fetching them at runtime from dynamic environment variables or secret vaults.
← Name Game
DNS-Based Steganography →