Template matching was the first approach for digit recognition in my Sudoku Solver. The tool uses the phone camera to scan a Sudoku puzzle and overlay the solution in augmented reality. It finds the grid, corrects for perspective, isolates each cell, reads each digit, solves the puzzle, and renders the solution back onto the camera feed. Template matching handles the reading step. You compare each cell image against nine reference digit images and pick the closest one.
It produced enough recognition errors to make the solver unreliable.
I replaced it with a neural net.
Not a big one. A tiny MLP, trained in-browser with tf.js loaded from a CDN. Training data was generated synthetically, digit variants drawn to canvas and augmented into a usable dataset. The weights shipped compressed to int8, inference runs client-side. No backend needed.
There was one thing I had to figure out the hard way. Training with tf.js stalls when the tab goes to the background. The model.fit call yields back to the browser UI thread after every iteration, and in a backgrounded tab the thread never comes back. The fix is a training flag called yieldEvery: 'never', which stops tf.js from yielding during the fit loop. Training runs to completion regardless of tab focus.
One flag. Took longer to find than it should have.
The classifier works now. I added auto-capture on top of it: the solver only takes a snapshot when it reads a complete, valid, unique board. No manual trigger needed. You point the phone at the puzzle and wait for the overlay to appear.
What still doesn’t work is angle. Hold the phone over the puzzle at a heavy tilt and board detection fails. Perspective-corner rectification is the fix. It’s documented as Phase 3. I got the rest of it to a working state and stopped there.
The fix is written down. The code isn’t.