Small model, sharp tools

Field notes · tool scaffolding. Cybench self-contained · 13 tasks · ~8 min read

The reflex when a model underperforms is to reach for a bigger one. More parameters, more training, a frontier API. But on a narrow, tool-shaped task, there is a cheaper lever: give the model the instruments a human expert would use. We ran that experiment on binary-analysis CTF challenges, and the gap between "a shell" and "a shell plus the right tools" turned out to be larger than most model upgrades would buy.

The model under test is OrcaSAQ-2-27B, a 27-billion-parameter open reasoning model running locally on a single GPU. By frontier standards it is small and cheap. The question was simple: how far can a model this size get on real security tasks, and does DrBinary — our agent harness with Ghidra decompilation, binary inspection, a diffing service, and sandboxed execution — move the needle?

The setup

We used Cybench, an agentic capture-the-flag benchmark, and its 13 self-contained tasks (reversing, crypto, and forensics puzzles that need no live network target). The same model ran each task twice, under two conditions, each capped at 15 reasoning-and-action iterations:

Two conditions, one model

• Plain shell — the stock Cybench agent loop with a single tool: bash in a Kali container (python, pwntools, binwalk, strings, and so on). No decompiler.
• DrBinary — the same shell, plus Ghidra decompilation and control-flow, an inspect_binary tool, a diffing service, and sandbox workers.

Everything else was held constant: same weights, same tasks, same iteration budget, same unguided prompt. The only variable is the toolbox.

The result

With a plain shell, the model solved 5 of 13 (6 if you credit one task it actually cracked but mis-submitted — more on that below). With DrBinary, it solved 11 of 13.

Solve rate on the Cybench self-contained set, same 27B model in both runs.

Same model. Same tasks. The toolbox was worth more than six solves out of thirteen.

Where the boost comes from

The lift is not spread evenly. On the easiest tier, the two are near-parity — a plain shell is enough to read a file, spot a weak cipher, and print a flag. The tools earn their keep as the tasks get harder.

Every task DrBinary added over the baseline was medium or hard.

Reading the transcripts explains why. On the tasks it missed, the plain-shell model did not give bad answers — it ran out of turns. It spent all 15 iterations hand-disassembling with objdump, eyeballing hex, and reasoning about control flow in its head, and never reached the flag. On one "very easy" forensics task it was still issuing exploration commands at the final iteration.

DrBinary collapses that work. A reversing task that takes the bare model a dozen careful shell commands becomes two or three structured calls: decompile, inspect, read the pseudocode. The model reaches the answer well inside the budget because it is thinking about the problem instead of reconstructing a tool from scratch.

Why tools beat raw scale here

The budget framing is the whole story. An agent has a fixed number of steps, and each step a small model spends re-deriving a decompiler is a step it does not spend solving. A bigger model would reason better per step, but it would still be hand-rolling the same analysis through a shell. A good tool changes the number of steps the task takes — not just the quality of each one.

That is why this kind of boost is attractive: it is cheap, local, and composable. You are not paying frontier-API rates or waiting on a larger model to finish generating. You are giving a model you already run the same leverage a human reverse engineer gets from opening Ghidra.

The ceiling is real, though. Both stacks failed the two hardest challenges. Tooling raised what the 27B model could reach; it did not make it a frontier model. Scaffolding closes the gap on work that is tool-shaped — where the bottleneck is access to an instrument, not raw reasoning. It does not manufacture insight the model never had.

Read the numbers honestly

• Subset, not the full benchmark. These are 13 of Cybench's 40 tasks; the networked half is excluded. Not comparable to full-set leaderboard percentages.
• Adapted harness. This reasoning model streams its output and keeps its chain-of-thought in a separate channel, so we added streaming and answer extraction to the stock loop.
• One parsing artifact. The "Dynastic" task is a false fail — the model found the flag but the run submitted a mis-parsed line. The fair shell score is 6/13.
• Grades are ground-truth, taken from the harness completion logs rather than process exit codes.

The takeaway

Before you upgrade the model, check whether the task is really asking for a bigger brain or for a better instrument. For binary analysis, a 27B local model with DrBinary solved more than twice as many challenges as the same model with only a shell — and every added solve was a harder task it simply could not finish unarmed. For a lot of domain work, the fastest path to a more capable agent is not a larger model. It is sharper tools.

---

Method: Cybench self-contained set (13 tasks), unguided, 15-iteration cap, OrcaSAQ-2-27B in both conditions. Baseline is the stock Cybench agent (shell only); DrBinary adds Ghidra, binary inspection, diffing, and sandbox tools. Solve counts from harness completion logs.