How to Benchmark a Sudoku Solver Fairly

What it means to benchmark a Sudoku solver

A benchmark is a controlled measurement, not a lucky run. When you benchmark Sudoku solver code, the goal is to answer a precise question: how long does this solver take, on which puzzles, under which conditions? A single time on a single puzzle tells you almost nothing, because Sudoku difficulty varies wildly. Some grids fall to simple logic in microseconds, while a few pathological ones force millions of backtracking steps. If you time only one puzzle, you are measuring that puzzle, not your solver.

Choose a representative puzzle set

Fair benchmarking starts with fair inputs, so collect a mix rather than cherry-picking the fast ones.

• Easy grids solvable by naked and hidden singles alone, to measure baseline overhead.
• The 17-clue collection, the largest published set of minimal puzzles, which stresses search.
• Known hard grids nicknamed for their difficulty, which force deep backtracking.
• Randomly generated puzzles across difficulty bands, so the average reflects real usage.

Freeze the set and version it. If you change the puzzles later, old and new numbers are no longer comparable, and your benchmark quietly loses its meaning.

Warm up before you time

Modern runtimes mislead you on the first run. A just-in-time compiler for JavaScript, Java, or C-sharp needs several executions before it optimizes hot code, and caches start cold. If you time the very first solve, you are measuring compilation and cache misses instead of the algorithm. Run the full set once or twice untimed, then start recording. For garbage-collected languages, consider triggering a collection before the timed region so a stray pause does not land in the middle of your measurement.

Time the right thing

Keep input and output away from the clock. Read every puzzle into memory first, solve inside the timed loop, and print results afterward. File reads and console writes are slow and noisy, and they have nothing to do with solver speed. Prefer a monotonic clock, exposed as perf_counter in Python, performance.now in JavaScript, or System.nanoTime in Java, rather than a wall clock that can jump when the system time is adjusted.

Report the median, not the mean

One slow run from a background process can wreck an average. Run each puzzle several times, keep the fastest or the median per puzzle, then summarize the set. The median is robust against outliers, so it describes the typical case honestly. It is also worth reporting the worst case separately, because a solver that is quick on average but occasionally stalls for seconds behaves very differently from one that is steadily fast.

Count work, not only seconds

Seconds depend on your laptop, your battery mode, and whatever else is running. To make results portable, also record machine-independent counters: the number of guesses placed, cells assigned by propagation, and backtracks performed. These stay stable across hardware, so another developer can reproduce and compare them. If your solver logs a million backtracks on a puzzle a person finds moderate, that points to a weak heuristic rather than a slow processor.

A simple benchmarking loop

1. Load all puzzles into memory and validate them once.
2. Run the whole set untimed as a warm-up pass.
3. For each puzzle, solve it several times and keep the median time and the operation counts.
4. Aggregate the results: median, worst case, total time, and total backtracks for the set.
5. Record your hardware, language version, and puzzle-set version alongside the numbers.

With this method you can benchmark Sudoku solver changes with confidence, because every result is reproducible and comparable. The same discipline underlies the free solver at shapedoku.com, where the logic treats nine glowing shapes exactly as a classic grid treats the digits one through nine. If you want intuition for why some grids are so much harder to time than others, solving a few by hand at app.shapedoku.com is a surprisingly good teacher.