Fable 5 Outperforms GPT-5.6 Sol on NP-Hard Problem, /goal Not a Game Changer

{ "title": "Anthropic's Claude Fable 5 beats OpenAI's GPT-5.6 Sol on NP-hard network design problem, but /goal mode backfires", "body": "Anthropic's Claude Fable 5 outperformed OpenAI's GPT-5.6 Sol on an unpublished NP-hard fiber-network design problem called KIRO, but the models' native /goal modes produced inconsistent results, according to a new benchmark published by Charles Azam.\n\nThe benchmark, which measures total cable length in a complex network design task, found that Fable 5 achieved the best overall solution and the most consistent results across multiple trials. However, the /goal feature—designed to help models optimize toward a specific objective—won 4 of 6 individual trials (a 66.7% win rate) while making average performance worse for both models.\n\n"Fable 5 is an absolute beast on this benchmark," Azam wrote. The model's plain configuration produced a mean score of 32,386 across three runs, beating GPT-5.6 Sol's plain mean of 34,261 by 1,875 points. With /goal enabled, Fable 5's mean of 33,145 beat Sol's goal mean of 35,129 by 1,984 points.\n\n## The KIRO Benchmark and Experimental Setup\n\nKIRO is a fiber-network design problem that originated from a 2018 engineering student project. Azam first worked on it as an engineering student, writing a C++ solver over a week. The problem involves directed distance matrices for the French cities of Grenoble, Nice, and Paris.\n\nThe objective is to minimize total cable length, with lower scores being better. A valid network must contain redundant loops rooted at distribution hubs, with short branches from towers on those loops. Every tower must appear exactly once, and reversing a cable segment can change the cost.\n\nThe search space is enormous. A lower bound on assignments involves 11^532 possibilities—that's 532 terminals assigned to 11 hubs, ignoring ordering and branches. A stronger lower bound, assuming exactly 19 loops of 28 terminals each with no branches, gives approximately 10^1223 possibilities.\n\nEach trial had a 30-minute optimization budget, with an outer agent timeout of 1900 seconds and maximum reasoning settings. All trials ran in Harbor 0.1.43, a container runtime, using Docker with subscription authentication.\n\nThe initial sweep included one matched 30-minute no-hint pair for each of six models: Claude Fable 5, Claude Opus 4.8, Claude Sonnet 5, GPT-5.6 Sol, GPT-5.6 Terra, and GPT-5.6 Luna. The replicated headline set focused on three matched runs for Fable 5 and three for GPT-5.6 Sol.\n\nEvery scored Fable 5 and Sol output was valid, thanks to a wrapper that required early checkpoints and final verification. The containers exposed 8 CPUs despite task metadata declaring 1, which favored Fable 5's parallel portfolios.\n\n## Fable 5's Consistency and the /goal Caveat\n\nFable 5's plain runs showed remarkable consistency. Run 1 scored 32,197, Run 2 scored 32,516, and Run 3 scored 32,446—a range of just 319 points. With /goal enabled, the results were more varied: Run 1 improved to 31,934 (better by -263), Run 2 improved to 32,324 (better by -192), but Run 3 worsened to 35,178 (worse by +2,732).\n\nThe best clean score of the entire benchmark was Fable 5's /goal Run 1 result of 31,934. Yet the model's /goal mean of 33,145 was 759 points worse than its plain mean of 32,386. The median effect was a -192 improvement, meaning half the time /goal helped, but the bad tail moved much farther in the wrong direction.\n\n"On a hard optimization problem, quality of the loop matters less than quality of what the loop keeps doing," Azam noted. "/goal is not a generic 'try harder' switch; it changes control loop and search path."\n\n## GPT-5.6 Sol's Wider Variance\n\nGPT-5.6 Sol showed much wider variance in its plain runs. Run 1 scored 33,581, Run 2 scored 35,539, and Run 3 scored 33,663—a range of 1,958 points. With /goal, Sol's results were similarly mixed: Run 1 worsened to 39,371 (worse by +5,790), Run 2 improved to 32,703 (better by -2,836), and Run 3 improved to 33,313 (better by -350).\n\nSol's plain mean was 34,261, while its /goal mean was 35,129—868 points worse. The median effect was a -350 improvement, again showing that /goal helped in most individual trials while making average performance worse.\n\n"/goal can win most individual trials while making observed average performance worse," Azam wrote.\n\n## How /goal Works Differently Across Models and When It Helps or Hurts\n\nThe two models implement /goal in fundamentally different ways. Claude Code, Anthropic's CLI tool, implements /goal as a session-scoped Stop hook with a separate evaluator model—Haiku by default. This evaluator cannot use tools or inspect files; it sees only the transcript.\n\nCodex CLI, OpenAI's tool, implements /goal as persisted thread state stored in SQLite. The working model receives three tools: create_goal, get_goal, and update_goal. This means Codex's evaluator sees files and tools but effectively grades its own work.\n\n"Claude's evaluator is independent but sees only transcript; Codex sees files and tools but effectively grades its own work," Azam noted.\n\nThe benchmark measures the complete system: model, CLI, prompt, subscription service, and harness. Azam noted that Claude Code's /goal implementation is not open source and relies solely on Anthropic's documentation, while Codex CLI source was read for the benchmarked release (0.144.4).\n\nAzam's analysis found that /goal helped when it sustained Fable 5's fast compiled portfolio or when it supported GPT-5.6 Sol's successful chain repartition. It hurt when Fable 5 built a slow solver or when Sol committed to an exhaustive anchor sweep.\n\n"On normal coding task, progress is often legible; optimization is different: extra time can amplify good or bad decisions," Azam wrote.\n\nFable 5's plain configuration proved to be the safest overall, with the narrowest score range and the best mean performance. The model produced "best solution overall and consistency unlike anything seen from a model on this problem," according to Azam.\n\nAzam acknowledged several limitations. The benchmark uses one unpublished NP-hard task, not a general coding leaderboard. Only Fable 5 and GPT-5.6 Sol have three clean matched pairs; other comparisons mix prompts, wrapper versions, and time limits. Trials ran sequentially through subscription services that may have drifted over time. Reproduction commands are provided in the article, with all experimental data and analysis scripts available in the CLIArena repository.\n\n## Related on Neura Market\n\n- Anthropic Model Benchmarks and Performance Data\n- OpenAI Model Comparisons and Analysis\n- AI Optimization and Problem-Solving Research" }