Congrats to the Kaggle Benchmarking Challenge Winners!

A few weeks ago we teamed up with Kaggle to ask a deceptively simple question: how do you actually...

A few weeks ago we teamed up with Kaggle to ask a deceptively simple question: how do you actually know whether a model is any good at the work you do? The Kaggle Benchmarking Challenge asked you to stop reading someone else's leaderboard and build your own.

You did not hold back. Submissions measured whether models notice their "practice" hacking target is a real company's live system, whether they can reproduce the exact hash input for Spain's VeriFactu invoice ledger or just invent one, whether small local models hold their ground against fake authoritative notes buried in untrusted documents, and plenty more. Nearly every one of them started from something specific the author already knew was hard, which is exactly where the best evaluations come from.

Thank you to everyone who built, measured, and wrote. These benchmarks are a real contribution, and we hope you keep running them as new models ship.

Now, let's celebrate our winners!

🎉 Congratulations To…

{% card %}

Every Model Got Calgary Wrong

{% embed https://dev.to/jonathansolvesstuff/alberta-stopped-changing-its-clocks-in-june-19-of-19-frontier-models-still-put-calgary-on-standard-3b33 %}

@jonathansolvesstuff asked 19 of the newest models a simple question: if it's 9 a.m. in Calgary in November, what time is it in Toronto? Alberta stopped changing its clocks in June, so the answer is 10:00, and every single model said 11:00. Their answer key comes from the public time zone database rather than anything written by hand, so the test can be run again each time that database updates.

{% endcard %}

{% card %}

The Models Read the Changelog, Then Misapplied It

{% embed https://dev.to/nazar-boyko/the-models-read-the-release-notes-i-measured-the-runtime-gei %}

@nazar-boyko noticed that common advice about how much memory PHP arrays and Go goroutines use has quietly stopped being true. So they measured the real numbers themself across four versions of each language, then asked 12 models the same questions. The best models could quote the right release note but applied it to the wrong situation, and most reported high confidence even when they were wrong.

{% endcard %}

{% card %}

Running Is Not Knowing

{% embed https://dev.to/rustam335/ask-ten-llms-for-a-blender-50-script-then-run-it-in-blender-50-1pe3 %}

@rustam335 asked ten models to write Blender scripts for four specific versions of the software, then actually ran every script in those versions to see what worked. Plenty of models produced code that ran fine while stating nothing had changed between versions. Making each answer list the changes it relied on is what exposed that gap.

{% endcard %}

{% card %}

Would Today's AI Have Caught Ariane 5?

{% embed https://dev.to/kaichen_dev/would-todays-ai-have-caught-ariane-5-i-rebuilt-26-of-historys-costliest-bugs-to-find-out-5d8m %}

@kaichen_dev rebuilt 26 of the most expensive software failures in history as small code samples, then asked 13 models to review them as if the code hadn't shipped yet. The models flagged a problem almost every time, but frequently for the wrong reason, including spotting SQL injection in code that never touches a database. They also included disguised and already-fixed versions of each case, to catch models that merely recognize the famous bug and models that cry wolf on working code.

{% endcard %}

{% card %}

Spell It Wrong First, and a Third of the Answers Keep It

{% embed https://dev.to/danielbutnar/brasov-or-brasov-15-llms-spell-my-city-right-until-you-spell-it-wrong-first-75a %}

@danielbutnar tested 43 models on two Romanian letters that look almost identical to their Turkish counterparts, where typing the wrong one is a common mistake. The models spell them correctly when the input is clean, but copy the wrong letter back about a third of the time when the input has it wrong. It even spread into words the models chose themselves, and their one-sentence fix for Romanian promptly broke Turkish, where the other letter is correct.

{% endcard %}

Prizes

Each of our five winners will receive:

• $500 USD
• DEV++ Membership
• Exclusive winner badge

All participants with a valid submission will receive a completion badge on their DEV profile.

What's Next?

The Presage SmartSpectra SDK Challenge is live! Build an experience that responds to real-time human signals captured through a camera. $3,000 in prizes across ten winners.

{% embed https://dev.to/devteam/join-the-presage-smartspectra-sdk-challenge-3000-in-prizes-across-10-winners-1370 %}

Congratulations again to our winners, and thank you to Kaggle for partnering with us on this one 💙