Reading VINs out of insurance claim video
Built at EY for an automotive client. The work is confidential, so there is no repository or demo to link.
problem
A vehicle insurance claim arrives as video and photographs. Somewhere in that footage is a vehicle identification number, seventeen characters stamped on a plate, and a person has to find it and type it in before the claim can be checked against anything at all.
That step is slow, it is repeated on every claim, and it is exactly the point a fraudulent claim relies on being handled carelessly.
approach
The pipeline runs in two stages. YOLOv8 locates the plate in a frame, then EasyOCR reads the characters out of that crop. Splitting it means the reader only ever sees a tight region instead of a whole dashboard, which is where most of the accuracy comes from. Extraction lands at 96 percent.
One read is not enough to act on. Qwen3-VL runs the same check across the rest of the evidence in a claim, the video, the documents, and the stills, and the results have to agree before a number is reported. It is served behind FastAPI and ends by generating audit ready PDF reports, because the output has to survive being read by someone who was not there when it ran.
The models were fine tuned on the client's own footage rather than run on stock weights. Manual review time per claim fell by roughly 80 percent.
tradeoffs
- choice Detect the plate first, then read it, instead of running OCR across the whole frame.cost Two models to maintain and two places to fail. A missed detection returns nothing at all, where whole frame OCR would at least have returned something wrong that a reviewer could correct.
- choice A second model has to agree before a VIN is reported.cost Slower and more expensive on every claim, and the pipeline returns nothing on genuinely ambiguous footage. That is the right answer in forensics and an irritating one for throughput.
- choice Fine tuned on the client's own footage rather than stock weights.cost None of the gain transfers. A different client with different cameras and lighting starts most of the way back at the beginning.
what broke
Reading the number reliably was most of the project. A VIN is stamped into metal in a vehicle bay, so a large share of the footage has a shadow lying across the plate, and the characters that survive a shadow badly are the ones the standard already warns about: a VIN never contains I, O or Q, because on a stamped surface they are indistinguishable from 1, 0 and 9. The reader returned them anyway.
Most of the fix was preprocessing rather than the model. Getting the crop, the contrast, and the thresholding right before EasyOCR ever saw the image did more than anything I changed downstream of it.
The rest was cutting my own validation rules back. I had written a set that were correct on paper and not achievable on real footage, and they were rejecting good reads more often than bad ones.
what I learned
Rules are only worth having if they can be enforced on the input you actually get. Fixed length and the restricted alphabet caught real errors. The more ambitious checks I wrote cost me more than they caught.
Most of the accuracy came from preprocessing, not from the model. I had assumed it would be the other way round, and I spent a while tuning the wrong end of the pipeline.