Exploring Standalone Small Classifiers: Qwen LoRA and BGE

02. Task Definition and Evaluation Protocol

The task uses 4 risk labels and 11 service intents. Training does not cover their full Cartesian product. Instead, it maps valid combinations into 13 joint classes such as emergency_symptom, routine_appointment, and uncertain_unknown. A fixed classification head produces 13-class logits in one forward pass, and a versioned mapping restores risk and intent.

Joint classification fixes the output space, eliminates JSON decoding failures, and makes confidence, calibration, and per-class errors directly measurable. Quality metrics include joint Macro-F1, risk Macro-F1, intent Macro-F1, emergency recall, ECE, and NLL.

Performance evaluation runs on an Intel Arc 130T. Single-request latency uses 20 warm-up iterations and 200 measured iterations, while throughput uses batch size 8. Peak memory comes from the PyTorch XPU allocator and represents only the tensor-allocation peak in this process.

Before-and-after comparisons must freeze the validation set, test set, label mapping, and measurement method. V2, V3, and V4 share 220 validation examples and 247 fixed test examples. The challenge set now serves as regression coverage for known difficult inputs rather than an independent blind test.

03. How the Two Base-Model Paths Are Compared

The Qwen path uses the same 13-class joint objective. Training first completes a frozen linear probe, then initializes the LoRA classifier from that head. The BGE path also emits 13-class logits, but fully fine-tunes about 24.0M parameters with separate learning rates for the encoder and random classification head.

The two paths share the label space, validation set, fixed test set, and performance protocol, but differ in trainable parameter count and optimization method, so the comparison covers complete engineering approaches rather than model names alone. Results must be read across quality, error patterns, latency, throughput, memory, and training time.

04. Optimization Path from V1 to V4

The iterations do more than add examples. Each round addresses the main bottleneck found in the previous one. V1 proves learnability, V2 changes the output mechanism, BGE V2 tests a small encoder, V3 improves data coverage, and V4 targets BGE's contextual boundaries.

Each optimization follows an observed problem: first generation latency, then classification-head training, then class coverage, and finally negation, reported context, and neighboring intents.

V1: Proving That Generative LoRA Can Learn the Task

V1 uses 263 training examples to fine-tune Qwen3-0.6B with LoRA for 4 epochs. The model autoregressively generates JSON containing risk, intent, route, and need_clarification. On the same 213 test examples, risk Macro-F1 rises from 0.1272 to 0.9217 and intent Macro-F1 from 0.0970 to 0.7017. Emergency recall rises from 0 to 0.875, while the invalid-JSON rate falls from 7.51% to 0.

V1 proves that a 0.6B model can learn the labels and output contract, but it does not solve inference cost or long-tail classes. It still misses 2/16 emergency examples, and unknown Recall remains 0. Generating about 39 output Tokens produces 2671.234 ms single-request p95 and only 2.587/s throughput at batch size 8.

The next round therefore changes the output mechanism instead of further tuning the JSON prompt. V2 removes the system prompt and autoregressive decoding in favor of a 13-class joint head. The goal is to emit logits in one forward pass while preserving deterministic risk and intent mappings.

V2: The Classification Head Fixes Latency but Exposes Training and Data Limits

V2 contains 424 training, 220 validation, and 247 test examples. Before formal training, it evaluates majority-class, TF-IDF, and frozen-Qwen linear-probe baselines. The linear probe reaches 0.7213 test Macro-F1. The first random-head LoRA run reaches only 0.3713, showing that the random head and LoRA backbone do not learn together effectively.

The second run initializes the classification head from the linear-probe weights before LoRA training and reaches 0.7254 test Macro-F1. After calibration and the established safety rules, fixed-test joint F1 is 0.698347, risk F1 is 0.855478, intent F1 is 0.675569, and challenge-regression joint F1 is 0.577211.

V2 reduces p95 from 2671.234 ms to 93.647 ms, a 28.52-fold improvement, but does not pass the quality gate. Batch=8 throughput rises from 2.587/s to 80.502/s. The result shows that a fixed head solves the main latency problem, while 424 training examples leave appointment, follow-up, navigation, and unknown boundaries unstable.

The next step splits into two investigations. One tests a smaller BGE encoder to see whether classification can become faster still. The other keeps validation and test data fixed to establish a comparable baseline for later data expansion.

BGE V2: The Small Encoder Gains Speed and Intent Quality

The first BGE run uses a uniform 2e-5 learning rate for the encoder and random classification head. After 112 steps, training remains near random and fixed-test joint F1 is only 0.093. The second run changes the encoder learning rate to 5e-5, the head learning rate to 1e-3, and trains longer, raising uncalibrated joint F1 to 0.704.

In the final calibrated result, BGE V2 records 0.686012 fixed-test joint F1, 0.664101 risk F1, 0.806377 intent F1, and 0.612410 challenge-regression joint F1. Single-request p95 is 7.863 ms, batch=8 throughput is 507.755/s, and training takes 24.91 seconds.

BGE V2 shows that an encoder with about 24.0M parameters can approach Qwen V2 joint quality with much higher speed, but its risk boundaries are substantially weaker. urgent_symptom F1 is only 0.243902, and routine_symptom F1 is 0. It is stronger on explicit service intents but more dependent on local symptom keywords.

The next round keeps the model structures unchanged and expands training coverage first. V3 retrains both Qwen and BGE on the same added data to determine whether the failures mainly reflect sample scarcity or the models' ability to represent compositional meaning.

V3: Data Expansion Produces Different Outcomes for the Two Models

V3 expands training from 424 to 2,740 examples by adding 2,316. The smallest class grows from 3 to 154 examples, polite phrasing is limited to 9.5% of the additions, and 157 training inputs contain “挂号.” The 220 validation and 247 test examples remain unchanged.

Qwen V3 fixed-test joint F1 rises from 0.698347 to 0.938188, while challenge-regression joint F1 rises from 0.577211 to 0.738095. BGE V3 fixed-test joint F1 rises from 0.686012 to 1.000000, but challenge regression moves only from 0.612410 to 0.612698, an increase of 0.000288.

The V3 data materially helps Qwen on complex inputs, while BGE's perfect fixed-test score mainly reflects saturation within the template domain. BGE still classifies stopped bleeding, fictional scenes, and symptoms explicitly attributed to someone else as emergencies. Adding more similar templates would improve the fixed score without resolving contextual boundaries.

The next round therefore applies both mechanism and data changes to BGE alone. V4 avoids copying challenge sentences and instead adds grouped contrast families. Training also adds marginal risk and intent supervision rather than relying only on the 13-class joint loss.

BGE V4: Contrast Data and Multi-Task Loss Improve Known Boundaries

V4 adds 479 training examples to the 2,740 V3 examples for a total of 3,219. Its 10 new families cover stopped bleeding and negated emergencies, fictional and quoted contexts, appointment versus follow-up, department versus navigation, and elliptical unknown inputs. The joint objective adds a 0.4-weight risk marginal loss, a 0.3-weight intent marginal loss, and 0.05 label smoothing.

BGE V4 challenge-regression joint F1 rises from 0.612698 to 0.899833, risk F1 from 0.829630 to 0.962865, intent F1 from 0.738344 to 0.951515, and errors fall from 5 to 2. Fixed-test joint F1 falls from the saturated 1.000000 to 0.986014 but remains above Qwen V3 at 0.938188.

V4 shows that grouped contrast data and marginal auxiliary supervision can repair known BGE weaknesses, but the gain remains a regression improvement. The remaining errors classify postoperative follow-up as appointment and stopped minor needle-site bleeding as emergency. The fixed test also contains 3 polite short appointment requests classified as unknown.

The next round should not generate more templates around these 2 errors. It should freeze Qwen V3, BGE V4, thresholds, and statistical methods, then create 500 to 1,000 source-independent blind examples. Only a new blind test can show whether V4 improvements transfer to authentic phrasing.

05. How Data and Training Changed Across Rounds

V1 data proved that LoRA could learn the labels, but minority classes were severely underrepresented: follow_up had only 4 training examples and hospital_navigation had only 3. V2 converted the dataset into a 13-class joint view with 424 training, 220 validation, and 247 test examples, then added an independent challenge regression set.

V3 targets insufficient class coverage, excessive courtesy phrases, and missing short inputs such as “挂号.” Training grows to 2,740 examples with at least 154 per class. Polite phrasing accounts for 9.5% of the added data, and 157 training inputs contain “挂号.” Validation and test files remain byte-identical, with 0 normalized duplicates.

After V3 fixes coverage, BGE saturates the fixed test while barely improving on the challenge set, indicating that the next step requires contextual contrasts rather than more similar templates. BGE V3 misclassifies stopped bleeding, fictional scenes, and symptoms explicitly attributed to someone else as emergencies.

V4 adds 479 training examples to V3 for a total of 3,219. Its 10 new families cover stopped bleeding, negated emergency signals, fictional or quoted contexts, appointment versus follow-up, department versus navigation, and elliptical unknown inputs.

BGE V4 combines 13-class joint cross-entropy with marginal risk and intent losses. The two auxiliary weights are 0.4 and 0.3, and the joint loss uses 0.05 label smoothing. Model output and the inference API remain unchanged.

06. Baselines, Training, and Reproduction

V2 first evaluates majority-class, TF-IDF, frozen-Qwen linear-probe, and random-head LoRA baselines. The frozen linear probe reaches 0.7213 test Macro-F1, while random-head LoRA reaches only 0.3713. Initializing the classifier from linear-probe weights raises the second LoRA run to 0.7254, showing that initialization matters more than simply adding training steps.

Baselines must run before formal training, with the same data, configurations, checkpoints, evaluation JSON, and benchmark JSON retained. The commands below reproduce the V3 and V4 runs and are byte-identical in both language versions.

# Train and evaluate the final Qwen V3 and BGE V4 classifiers
.\.venv-xpu\Scripts\python.exe -m med_router.v2.expand_training --config configs\v3\data.yaml
.\.venv-xpu\Scripts\python.exe -m med_router.v2.train --config configs\v3\qwen_linear_probe.yaml
.\.venv-xpu\Scripts\python.exe -m med_router.v2.train --config configs\v3\qwen_lora.yaml
.\.venv-xpu\Scripts\python.exe -m med_router.v2.evaluate --config configs\v3\qwen_eval.yaml
.\.venv-xpu\Scripts\python.exe -m med_router.v2.benchmark --config configs\v3\qwen_benchmark.yaml
.\.venv-xpu\Scripts\python.exe -m med_router.v2.expand_training --config configs\v4\bge_data.yaml
.\.venv-xpu\Scripts\python.exe -m med_router.bge.train --config configs\v4\bge_train.yaml
.\.venv-xpu\Scripts\python.exe -m med_router.bge.evaluate --config configs\v4\bge_eval.yaml
.\.venv-xpu\Scripts\python.exe -m med_router.bge.benchmark --config configs\v4\bge_benchmark.yaml

If PyTorch falls back to CPU or the XPU build is not loaded, quality evaluation can still run, but performance numbers are not directly comparable with this article. Any retraining should also regenerate hashes and reports rather than reuse old conclusions.

07. Final Quality Results

On the 247-example fixed test set, Qwen V3 records 0.938188 joint F1, 0.975096 risk F1, and 0.936507 intent F1. On the challenge regression set, it records 0.738095 joint F1, 0.952210 risk F1, and 0.694444 intent F1.

On the fixed test set, BGE V4 records 0.986014 joint F1, 0.983324 risk F1, and 0.983471 intent F1. On the challenge regression set, it records 0.899833 joint F1, 0.962865 risk F1, and 0.951515 intent F1.

The confirmed result is that BGE V4 outperforms Qwen V3 on both the fixed test and known challenge regression sets, but those sets influenced optimization direction and cannot establish absolute superiority on real inputs. Both models record 1.0 emergency recall on the challenge regression set.

BGE V4 reduces challenge-regression errors from 5 in V3 to 2. The remaining errors classify postoperative follow-up as appointment and stopped minor needle-site bleeding as emergency. On the fixed test, 3 polite short appointment requests are classified as unknown.

08. Final Performance and Training Cost

In the final rerun, BGE V4 records 5.548 ms single-request p95, 578.603 items per second at batch size 8, and 47.37 MiB peak allocated memory. Qwen V3 records 90.010 ms, 75.375 items per second, and 1,169.22 MiB.

Under the same local measurement protocol, BGE V4's p95 is about 1/16.22 of Qwen V3, throughput is about 7.68 times higher, and peak allocated memory is about 1/24.68. These numbers show a substantial engineering-cost advantage for the standalone small encoder.

Final Comparison Summary

The direct single-request p95 comparison is 2671.234 ms for generative Qwen V1, 90.010 ms for the Qwen V3 classifier, and 5.548 ms for the BGE V4 classifier.

Training cost also differs. Qwen V3 takes 126.91 seconds for the linear probe and 587.14 seconds for LoRA, totaling about 11.9 minutes. BGE V4 full fine-tuning takes 53.62 seconds, with the best checkpoint at epoch 6.

BGE V3 and V4 share the same model structure, so the difference between 8.338 ms and 5.548 ms should be treated as measurement variation rather than a structural speedup caused by training data. Performance results also cannot be directly extrapolated to different servers, CPUs, or mobile devices.

09. Result Boundaries and the Next Evaluation

This exploration answers two engineering questions. First, generative routing can be replaced with a fixed classification head to substantially reduce latency. Second, for the current labels and data, BGE V4 with about 24.0M parameters can exceed Qwen V3 regression quality while using less training and inference compute.

Independent generalization remains unanswered. The V3 fixed test and round2 challenge set both influenced later data design, so they now serve only as regression checks. Training data remains primarily synthetic, and the risk labels have not received clinical validation.

The final result supports further evaluation of BGE V4 but does not support recommending a single-model, dual-model, cascade, or rule composition from current scores alone. The next step is to freeze Qwen V3, BGE V4, thresholds, and statistical methods, then build 500 to 1,000 source-independent blind examples.

The blind set should cover authentic short queries, typos, dialects, speech transcription, negation, reported context, fictional scenarios, and compound requests. Qualified reviewers need to verify symptom-risk labels, with misses, over-escalation, and intent confusion evaluated according to their distinct error costs.