Scanning VINs off dusty tractors: what we learned building camera OCR three times

Why VIN entry kills onboarding funnels

The mySAME and myDeutz-Fahr apps - the white-label farmer platform we engineered for SDF Group, the Italian tractor manufacturer behind the SAME and DEUTZ-FAHR brands - hinge on one onboarding step: a farmer registers their tractor by its vehicle identification number. Everything downstream depends on it. Warranty, dealer contact, live telemetry over Bluetooth, service history - no VIN, no app.

And a VIN is a hostile thing to type. Seventeen characters of mixed letters and digits with no spaces, no checksum feedback in the UI, and no meaning to the person typing it. Get one character wrong and the lookup fails - or worse, quietly matches nothing and the farmer gives up. On a phone keyboard, standing next to a tractor, error rates are brutal. We watched the funnel: manual VIN entry was where onboarding went to die.

So we gave people ways to not type it.

Three ways to capture a VIN

The app ended up with three input paths, and all three survived to production because each one wins in a different situation.

QR is the best experience but only exists on newer machines. Manual entry covers everything but converts worst. OCR is the interesting middle: it works on a thirty-year-old tractor, and it only has to be better than typing to earn its place. That framing - OCR as a funnel optimization, not a computer-vision showcase - shaped every decision that followed.

Attempt 1: tesseract.js in the web layer

The app is Ionic + Angular on Capacitor (the same codebase we later carried through a major Capacitor and Angular modernization), so the path of least resistance was OCR in the web layer. We used tesseract.js v6: grab a photo with the Capacitor Camera plugin, hand the image URI to a Tesseract worker, recognize, done. One implementation, every platform, no native code.

It works - genuinely. For a printed label in decent light, tesseract.js reads the VIN fine, and it's the only option when the same code runs as a plain web app. But the honest scorecard from the field was mixed:

• Cold start is heavy. The recognition model has to load inside the web view before the first scan. On mid-range Android phones - which is what farmers actually carry - that pause reads as "the app is broken".
• Embossed metal is Tesseract's worst case. Stamped characters have no ink contrast, just shadows. Add dust, glare, and an off-axis photo, and accuracy falls off fast.
• It recognizes the whole photo. A VIN plate carries type approvals, weights, and part numbers. Tesseract returns all of it, and the VIN is somewhere in the soup.

We shipped it, measured it, and concluded it beat typing - barely. Good enough to validate the feature; not good enough to stop there.

Going native: one plugin interface, two platform recognizers

Both mobile platforms ship excellent on-device text recognizers: ML Kit text recognition on Android, the Vision framework on iOS. They're free, fast, offline, and - because they're built for exactly this kind of photo - dramatically better on stamped metal than a general-purpose OCR engine running in a web view.

The trick in a Capacitor app is to use them without forking your application code. We wrapped both recognizers behind a single shared Capacitor plugin interface, so the Angular layer calls one method and never knows which platform it's on:

// Illustrative - the shape of the shared plugin interface
interface TextDetection {
  text: string;
  // normalized corner points, clockwise from top-left
  topLeft: [number, number];
  topRight: [number, number];
  bottomRight: [number, number];
  bottomLeft: [number, number];
}

interface TextDetectorPlugin {
  detectText(options: { imagePath: string }):
    Promise<{ textDetections: TextDetection[] }>;
}

Behind that interface sit two small native implementations: a Kotlin class that feeds a bitmap to the on-device recognizer and maps each recognized line - text plus normalized corner coordinates - into the plugin result, and its Swift twin doing the same with Vision. Maybe a hundred lines each. Returning lines with geometry rather than one text blob matters: the VIN is one line on a busy plate, and per-line results let the UI show candidates and let the farmer tap the right one instead of editing a paragraph of noise.

One artifact of this phase still makes us smile: the code carries an unused image-orientation enum - up, down, left, right - the skeleton of a rotate-and-retry strategy we'd sketched for Tesseract's bad days. The native recognizers handled rotated and skewed text well enough that we never wired it up. Sometimes the best feature is the one you get to delete from the plan.

The post-processing that actually matters

Here's the part that looks too simple to be the answer. After recognition, we do this:

const candidate = rawText
  .replace(/[\n\t\r]/g, "")
  .replace(/[^A-Z0-9]/g, "")   // after uppercasing
  .trim();
// then: is the length plausible for a VIN?

Uppercase, strip everything that isn't A-Z or 0-9, check the length. That's it. No confidence thresholds, no fuzzy matching, no character-confusion tables. We considered cleverer things - the VIN spec even invites them, because a road-legal VIN never contains the letters I, O, or Q, precisely so nobody confuses them with 1 and 0. You could exploit that: auto-substitute a recognized O for a 0, an I for a 1.

We deliberately didn't, and the reason is the next section: any correction the client makes silently is a correction the user can't see and the server can't distrust. A wrong-but-plausible VIN is worse than an obviously wrong one. The client's job is to produce a clean candidate, show it to the human, and let them fix a character if needed. Judging whether it's a real VIN is somebody else's job.

Trust the registry, not the recognizer

This is the real lesson of building VIN capture three times: the recognizer is an input method, not an authority. Whatever the camera produced - or the QR code, or the farmer's thumbs - the candidate VIN goes to the backend and is checked against the actual product registry. That check has three distinct failure modes, and each gets its own UX, because they mean very different things:

• VIN not found. Probably a misread or a machine not yet in the registry. We don't dead-end: the farmer can continue by picking their tractor family manually, so a registry gap never blocks onboarding.
• VIN belongs to another brand. The platform is white-label - the same codebase ships as mySAME and myDeutz-Fahr - so a valid VIN can simply belong to the other brand's app. The error says so explicitly, by brand name, instead of a useless "not found".
• VIN already registered - HTTP 409. Someone else has claimed this machine. This is a conflict, not a typo, and it gets a distinct message, because telling a legitimate owner "invalid VIN" when the real story is "another account holds this tractor" generates support tickets that never resolve.

Once the registry is the source of truth, the pressure on the OCR evaporates. A misread costs the farmer one more scan or a one-character edit - it can't corrupt data, claim the wrong machine, or strand an account. That's what let us keep the client-side post-processing so boring.

Decision guide: when tesseract.js is enough vs when to go native

We've now shipped both, so here's the honest split:

• Stay with tesseract.js when your target is a plain web app (no native layer to call), the text is printed with real ink contrast, users scan indoors, and a few seconds of model loading is acceptable. It's one codebase and zero native maintenance.
• Go native (ML Kit / Vision) when you're already shipping through Capacitor, the text is embossed, engraved, or photographed in uncontrolled light, first-scan latency matters, or you need per-line geometry to pick a target out of a busy image. The native recognizers cost you two small platform classes and repay it every single scan.
• Do both when, like us, the same code runs as a web app and as native builds: one shared plugin interface, native recognizers on devices, tesseract.js as the web fallback. The application code doesn't branch; the plugin does.
• Either way, validate server-side. The recognizer choice affects conversion. The registry check protects correctness. Don't let a good recognizer tempt you into skipping the second one.

Frequently asked questions

Related: the full mySAME / myDeutz-Fahr white-label platform case study · live tractor telemetry over Bluetooth LE in the same app · modernizing a legacy Ionic Capacitor app · hire a dedicated development team