ML Pipeline

How AutoScan reads a number plate

A hybrid pipeline: a deep-learning detector finds the vehicle, classical computer vision isolates the plate, and a multimodal model reads it.

  1. 1. Image input

    A photo is uploaded or captured from the camera and decoded with OpenCV into a BGR pixel array.

    OpenCVNumPy

  2. 2. Vehicle detection — YOLOv8n

    The image is letterboxed to 640×640 and passed through YOLOv8n (COCO-pretrained), exported to ONNX and run on ONNX Runtime. Detections below 0.25 confidence are dropped and overlapping boxes are merged with Non-Max Suppression (IoU 0.45). The most confident car, motorcycle, bus or truck is kept.

    YOLOv8nONNX RuntimeNMS

  3. 3. Plate localisation — classical CV

    Inside the vehicle box, OpenCV searches for the plate: HSV colour masks for white, yellow, green (EV) and red plates, morphological closing, then contour filtering by aspect ratio (1.8–5.8) and area. If colour fails, a Sobel vertical-edge + Otsu threshold pass finds dense character regions. The crop is padded so trailing characters are never cut.

    HSV maskingMorphologySobelContours

  4. 4. Text recognition — Gemini Vision

    The plate crop is resized, denoised with a bilateral filter and contrast-enhanced with CLAHE, then read by Gemini's multimodal vision model. If the crop yields fewer than 6 characters, the full image is sent as a fallback. Regex post-processing fixes common OCR confusions (O→0, I→1, S→5, B→8) between digits.

    CLAHEGemini VisionRegex

  5. 5. Format analysis

    Gemini classifies the plate's jurisdiction (e.g. India – Maharashtra), registration type and canonical spacing (MH 12 AB 1234), returning structured JSON.

    LLMStructured output

  6. 6. Scoring & storage

    Overall confidence = 40% detection score + 60% recognition score. Each result is stored in PostgreSQL (Neon) so it can be reviewed, corrected and analysed on the dashboard.

    FastAPISQLAlchemyPostgreSQL

Model & system specs

Detector
YOLOv8n (COCO, 80 classes)
Runtime
ONNX Runtime, CPU
Input size
640 × 640, letterboxed
Model size
12.8 MB
Server memory
≈ 200 MB
OCR
Gemini Vision (multimodal LLM)
Backend
FastAPI on Render
Frontend
Next.js 15 on Vercel

Why this design?

  • Swapping PyTorch for ONNX Runtime cut server memory from over 2 GB to about 200 MB, so the model runs on a free-tier instance.
  • Classical CV for plate localisation needs no extra training data and works across Indian plate colours.
  • A vision LLM handles varied fonts, angles and lighting better than a small on-device OCR model, at the cost of an external API call.
  • Known limits: very small or heavily angled plates, and multiple vehicles in one frame (only the most confident vehicle is read).
Try the scanner