Arindam Chatterjee

@arindamchatterjee007

Ayush Khamrui

@ayushkhamrui

Valid Labels, Wrong Decisions: A Small-Model Reality Check

Submitted Oct 30, 2026

Abstract

A classifier can return only allowed labels and still make the wrong decision about when to defer to a human. In PocketRoute Lab, we evaluated a pinned Qwen2.5 1.5B model with constrained decoding and a reject-only policy for maintenance, delivery, inspection and review. The hybrid system reached 46/48 on previously exposed development/regression cases, but scored 73/96 on a separate AI-authored synthetic set. Four review-required requests were assigned a service category. Invalid labels fell to zero, yet the quality and safety gates still failed. We stopped short of a native release rather than promote the development score as readiness.

This joint talk explains the failed alternatives and the engineering around that decision: frozen inputs separated from answers, pinned model/protocol fingerprints, complete attempt accounting, and a portable scorer that rejects missing or malformed reports. A short walkthrough of the public evidence and GitHub-tested scorer shows why exact reproduction of a failing reference is not release acceptance. We will also explain why roughly 12-second cloud request latency and a 5.42 GiB process footprint cannot be presented as phone performance or energy savings. The contribution is a reproducible engineering investigation and practical test tooling, not a new foundation model or a claim that one model fits every device.

Takeaways

  • Separate output-format validity, accepted-label precision, coverage and false acceptance instead of promoting one development score.
  • Freeze data/protocol provenance and retain timeouts, missing cases and failed runs. Exposed regression cases are not a fresh quality test.
  • Share a report contract across integrations while qualifying actual runtimes, memory, cancellation, thermals and energy on each target device.

Audience

Engineers building local or edge-AI applications, small-model classifiers, evaluation tooling and native integrations. Familiarity with basic model inference and automated testing is helpful; no specialised hardware is required to follow the talk.

Format and outline

30-minute joint talk with a short evidence/scorer walkthrough, not an audience laptop workshop or a live native-device demonstration.

  • 3 minutes: the bounded task and intended device constraints.
  • 6 minutes: development versus confirmation, failed alternatives and the four false acceptances.
  • 7 minutes: public receipts, complete-case accounting and the portable scorer’s rejection of incomplete evidence.
  • 6 minutes: tokenizer/runtime boundaries, cancellation and device-specific resource qualification.
  • 3 minutes: reusable engineering lessons and limitations.
  • 5 minutes: questions and technical discussion.

Presenters

Arindam Chatterjee and Ayush Khamrui are proposing this session together for the independent PocketRoute Lab project. Both have approved being named as co-presenters.

Arindam is a software engineer with C++/Linux systems experience and maintains the experimental project. Ayush is a co-presenter and repository administrator. This proposal is made in an independent capacity, not on behalf of an employer; no employer endorsement is claimed.

Public evidence and supporting material

Draft slides and a two-minute elevator-pitch video are not yet available. The linked PDFs are technical references, not a slide deck.

Disclosure and limitations

Code, synthetic-data preparation and proposal drafting were AI-assisted; the source, run history and limitations are inspectable. The 96 cases are synthetic, not independently human-labelled customer data, and are now exposed. This is a Node/Linux experiment, not a validated phone/watch/MCU deployment. No physical-device energy measurement, production safety certification, new model training or customer deployment is claimed. All model suggestions remain subject to human review. The demonstrated reports will be labelled as recorded evidence or synthetic fixtures, not presented as live device inference.

Comments

{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}

Hosted by

Jumpstart better data engineering and AI futures