ScreaMy7

← Blog

An AI agent in the fuzzing loop: re-finding CVE-2017-11729 in libming

Re-finding CVE-2017-11729, a heap over-read in libming's SWF decompiler, with an AI coding agent running the fuzzing. Here's where it tried to lie to me.

8 min read

#fuzzing
On this page
  1. Step 1 — Pick a target the agent can actually reach
  2. Step 2 — Let the agent fight the build
  3. Step 3 — The harness question (and where the agent lied)
  4. Step 4 — Fuzz it
  5. Step 5 — Hand triage to the agent (the safe hand-off)
  6. Step 6 — Make the agent prove the root cause
  7. Step 7 — Minimize, in-process, and the harness pays off
  8. Step 8 — Fix-verification
  9. Impact
  10. The pattern

I wanted to re-discover a known CVE the honest way — build the vulnerable code, fuzz it, and let the fuzzer find the crash on its own — and I wanted to run the whole thing with an AI coding agent (Claude Code) as the driver. Not “ask the model if the code is vulnerable,” but actually have it do the work: pick the target, fix the build, write a harness, triage crashes, pin the root cause.

The target was CVE-2017-11729: a heap-based buffer over-read in OpCode, reached from decompileINCR_DECR, in util/decompile.c of libming 0.4.8. libming is a small C library for reading and writing Flash (.swf) files; its swftophp tool decompiles the ActionScript bytecode inside a SWF back into script. Decompiling attacker-controlled bytecode in C is where these bugs live.

The short version of what I learned: an LLM agent is a fast pair of hands that you must never trust on a conclusion. It writes code and grinds through bookkeeping brilliantly, and it will also confidently hand you a root cause it never checked, a harness that reaches nothing, and a “we found a CVE!” on an unvalidated crash. Every step below is paired with the verification that keeps it honest.

Step 1 — Pick a target the agent can actually reach

The first thing I had the agent do was read, not run. Pointed at a target, an LLM is good at the question “where would a fuzzer trip, and how deep is it?” — which functions are on the default input path versus buried behind optional features and multi-condition state.

For swftophp the answer was encouraging: main() reads the file, readMovie() walks every tag, and every DoAction tag’s bytecode goes straight to decompileAction(), which dispatches to a family of decompileXXX helpers. The vulnerable code is on that mandatory path — parse, decompile, done — not behind a flag. That’s shallow, and it’s the kind of judgement the agent gets right because it’s reading structure, not asserting exploitability.

The leash: “this looks reachable” is a claim. It’s cheap to confirm later with coverage or just by getting a crash, so I treated it as a hypothesis, not a fact.

Step 2 — Let the agent fight the build

Building a 2018 C project with a 2025 toolchain is tedious, mechanical, and exactly what an agent should own. It worked through three breakages without hand-holding:

  • libming’s config.guess predates aarch64 → pass an explicit --build triplet.
  • Modern FreeType dropped the freetype-config script libming looks for → --disable-freetype (swftophp doesn’t need it).
  • The bison-based ActionScript compiler won’t build under modern bison — but swftophp is a decompiler and never needs it.

That last realization is the useful one, and the agent got there by reading the dependency graph: skip autotools and the compiler entirely, and build swftophp directly from just its reader/decompiler sources, instrumented with AFL++ and ASan:

AFL_USE_ASAN=1 afl-clang-fast -fcommon -g -O1 -DSWFPHP \
  -I. -Iutil -Isrc -Isrc/blocks \
  util/main.c util/outputscript.c util/action.c util/blocktypes.c \
  util/decompile.c util/parser.c util/read.c util/vasprintf.c \
  src/blocks/error.c \
  -lz -lm -o swftophp

The leash: none needed here — a build either links or it doesn’t. This is the pure-upside use of an agent: no conclusions to second-guess, just toil it does faster than you.

Step 3 — The harness question (and where the agent lied)

swftophp is a command-line tool, so the real campaign needed no harness at all — AFL fuzzes the binary directly, which is also the most faithful thing you can do. The agent was right to say so, and it’s worth repeating because people reflexively over-engineer a harness when the target already takes a file.

But the common case is a library with no CLI entry, and that’s where you have the agent write a libFuzzer harness. This is the step where it will quietly betray you. The first two harnesses it produced looked correct and were wrong, and I only caught them because of one rule:

A harness is guilty until it reproduces a crash you already understand.

  • v1 used fmemopen() to turn the fuzz buffer into a FILE*. Idiomatic and silently broken: readMovieHeader calls fstat(fileno(f)), a memory stream has no file descriptor, so it bailed in the header before decompiling anything. It ran clean and found nothing — which looks exactly like success.
  • v2 exposed libming’s static entry functions with a blunt -Dstatic= across the translation unit. That changed a static local’s lifetime and produced a stack-use-after-scope in the header’s printf path — a crash that does not exist in the real tool. A generated harness will happily manufacture its own bugs for you to chase.
  • v3 is the faithful one: un-static only the two entry functions, write each input to a real temp file (real fd), install a longjmp error handler because libming’s default SWF_error calls exit(), and reset the global read cursor between runs.
static jmp_buf g_jb;
static void onError(const char *msg, ...) { (void)msg; longjmp(g_jb, 1); }

int LLVMFuzzerInitialize(int *a, char ***v) {
    freopen("/dev/null", "w", stdout);     /* swftophp prints the decompiled script */
    setSWFErrorFunction(onError);          /* default handler calls exit() */
    return 0;
}

int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
    if (size < 8 || size > (1 << 20)) return 0;
    char path[] = "/dev/shm/ming_XXXXXX";  /* real fd: readMovieHeader fstat()s it */
    int fd = mkstemp(path);
    if (fd < 0) return 0;
    write(fd, data, size); close(fd);
    FILE *f = fopen(path, "rb");
    fileOffset = 0;                         /* reset the global read cursor */
    int compressed = 0;
    if (setjmp(g_jb) == 0)
        if (readMovieHeader(f, &compressed) == 0 && !compressed)
            readMovie(f);                   /* the REAL flow, not a reimplementation */
    fclose(f); unlink(path);
    return 0;
}

The leash: I didn’t trust v3 because it compiled; I trusted it because it reproduced a crash I already understood (more on that below), and only then let it fuzz freely. A harness the agent writes is a hypothesis about reachability until a known bug proves it.

Step 4 — Fuzz it

afl-fuzz -i seeds -o out -m none -t 2000 -- ./swftophp @@

Two hand-built minimal SWFs as seeds (a header plus a DoAction tag), and because the bug is on the default path, this is shallow: AFL++ saved its first crashes within seconds and piled up dozens shortly after. The agent set up the seeds and the invocation; the fuzzer did the finding. Nothing to second-guess here either — a crash file is a crash file.

Step 5 — Hand triage to the agent (the safe hand-off)

This is where an agent earns its keep without any trust risk, because everything it produces is independently checkable. Fifty-odd crash files, replayed under ASan, grouped by the top non-interceptor stack frame, each distinct stack matched against the published advisories by function + caller + line. Turning a pile of id:0000NN files into “three distinct bugs, and this one is the CVE” is tedious, deterministic, and verifiable — the ideal job to delegate.

The clean centerpiece replayed under ASan as:

AddressSanitizer: SEGV on unknown address
    #0 OpCode               util/decompile.c:868
    #1 decompileINCR_DECR   util/decompile.c:1440
    #2 decompileAction      util/decompile.c
    ...
    #5 outputSWF_DOACTION   util/outputscript.c:1548
    #6 readMovie            util/main.c:277
    #7 main                 util/main.c:350

OpCode called from decompileINCR_DECR at line 1440 — which is precisely how CVE-2017-11729 is described. The agent proposed the match; I confirmed it against the advisory. (That function+caller+line triple is also what let me trust it was the same bug, not a cousin — I never looked at the public PoC.)

Step 6 — Make the agent prove the root cause

Now the one claim that matters, and the one an LLM will most happily fake: why does it crash. The agent’s answer was correct, but I believed it only after reading the four lines it pointed at.

static inline int OpCode(SWF_ACTION *actions, int n, int maxn)
{
    if(!n || n >= maxn)
        return -999;
    return actions[n].SWF_ACTIONRECORD.ActionCode;
}

This is meant to be the safe accessor. But !n rejects index 0 and n >= maxn rejects the top — nothing rejects a negative n. And the caller, decompileINCR_DECR, decides pre- vs post-increment by probing the surrounding instructions:

... OpCode(actions, n-1, maxn) ...
    OpCode(actions, n-2, maxn) ...
    OpCode(actions, n-3, maxn) ...

When the increment/decrement action is near the start of the action array — index 1 or 2 — those go negative, pass the broken guard, and dereference actions[-1], actions[-2], actions[-3], reading off the front of the heap allocation. The entire bug is one missing comparison: the guard needed n < 0.

The leash — this is the important one. For a memory bug this verification is two minutes: show me the bounds check. The failure mode to expect from an agent is a fluent, plausible root cause it never actually checked. Read the lines yourself, every time.

Step 7 — Minimize, in-process, and the harness pays off

The agent drove afl-tmin down to a 16-byte trigger:

46 57 53 | 30 | 30 30 30 30 | 00 30 30 30 30 | 01 03 | 50
 F  W  S | ver|  file length |  header fields | taghdr |body
  • 46 57 53 — the FWS (uncompressed SWF) signature;
  • 30 — version byte (junk, accepted);
  • 30 30 30 30 — the declared file-length field, left as a bogus 0x30303030; the parser never validates it;
  • 00 30 30 30 30 — frame-rect / rate / frame-count header fields (junk, accepted);
  • 01 03 — a tag header. Little-endian 0x0301: type 0x0301 >> 6 = 12 (DoAction), body length 0x0301 & 0x3f = 1;
  • 50 — the single body byte: an increment/decrement opcode. It lands at a low index, so the backward probes go negative and OpCode reads before the array.

Feeding those 16 bytes to the v3 harness is what validated it — it reproduced the same bug in-process, and ASan reported it as a precise heap-buffer-overflow in OpCode ← decompileINCR_DECR (here at the line-1474 probe, which is the sibling CVE-2017-11730 — same missing-guard root cause as -11729 at line 1440). Then it found crashes from the two seeds in under a minute. Worth noting: in-process the harness gives a crisp heap-buffer-overflow, where the CLI run bottomed out in a raw SEGV. The harness is the better oracle — which is exactly why you write one for a library, and exactly why you make it reproduce a known crash before you trust a word it says.

Step 8 — Fix-verification

The last thing I had the agent do was build current master the same way and re-run the 16 bytes. It no longer crashes — but OpCode and decompileINCR_DECR are byte-for-byte unchanged there, so the decompiler accessor still has the incomplete guard; what differs upstream is earlier handling that keeps this input from reaching the vulnerable call. I had the agent report that distinction rather than round it up to “fixed,” because “my input happens not to reach it” is thin cover if you feed libming’s decompiler untrusted data.

Impact

This is an out-of-bounds read (CWE-125), reached by passing a crafted SWF to the swftophp command-line tool. The demonstrated impact is a crash — denial of service for anything that shells out to swftophp on untrusted files. It is not remote code execution, and an out-of-bounds read of a few words off the front of an allocation doesn’t get you there on its own. I’m keeping that line precise on purpose: overstated impact is the other thing an eager agent will hand you if you let it.

The pattern

Across every step the division of labour was the same. Let the agent write the code and do the bookkeeping — the build, the harness, the afl-tmin runs, the crash triage — and keep every conclusion on a short leash:

  • a harness has to reproduce a crash you already understand before you trust it;
  • a root cause has to point at the exact line;
  • “it’s a CVE” has to map to an advisory;
  • impact is whatever you actually demonstrated, and nothing more.

The agent is happy to skip all four. The whole skill of using one for this work is not skipping them for it.

← back to blog