This Guide Shows How to Train Your Own Decision Model Like Jev Using Unsloth.
This guide fine-tunes Qwen3.5-4B with Unsloth into a CLINC150 intent decision model using Colab, 4-bit quantization, and low-rank adaptation.
Key Takeaways
- The guide shows how to turn Qwen3.5-4B into a decision model using Unsloth’s FastDecisionModel, which adds a decision head for selecting predefined answers instead of generating free text.
- It uses CLINC150, an intent classification dataset, and reshapes its records into decision-training format with input state, available choices, and gold labels.
- The workflow runs in Google Colab with 4-bit quantization, LoRA, and decision-head training to fine-tune the model efficiently.
- On a narrowed 10-intent task, validation accuracy improved from 18.5% before training to 100% after training, demonstrating the method as a proof of concept rather than full 150-category performance.
Meta description: Learn how to fine-tune Qwen3.5-4B with Unsloth into a decision model that classifies CLINC150 user intent, evaluates accuracy, and saves confidence scores.
Tags: decision models, Qwen3.5-4B, Unsloth, CLINC150, intent classification
Featured image: abstract, friendly illustration style with no real people, no logos, and no text
This tutorial is for developers who want to use Google Colab, Unsloth, and Qwen3.5-4B to build a decision model that classifies user intent from short messages. It shows how the CLINC150 dataset can be reshaped for decision training, how 4-bit quantization and Low-Rank Adaptation can fine-tune the model, and how accuracy, confidence, and saved outputs can be checked before wider use. In the described run, validation accuracy moved from 18.5% to 100% on a deliberately narrowed 10-intent task, and the model classified all 300 examples in the held-out test split correctly. That result is useful as a proof of concept, not as a claim that the full 150-category task is solved.
Why A Language Model Can Become A Decision Model
Language models usually generate text token by token, but some applications only need a selected decision, not an explanation. If a user asks about checking a savings account balance, a chatbot may describe the steps, while a decision model can simply return the matching intent and confidence scores. They are useful where choosing an action matters more than producing text: intent classification, content moderation, request routing, sentiment analysis, and agent workflows.
Models such as Jev, Laya, and Clef are built for structured decisions. Instead of writing text token by token, they choose from predefined options and return probability estimates. Unsloth can now train decision models from open-weight language models, including Qwen and Gemma. The official decision-model guide explains the method and includes ready-to-run notebooks.
Unsloth’s FastDecisionModel converts a pretrained language model into a decision model by adding a specialized decision head. The model processes the input and available choices, then scores possible answers without producing a conventional completion. It supports three question types: Choice, which selects one option from several; Noul, which returns a yes-or-no decision; and Score, which selects a level from ordered categories. This experiment uses Choice because the goal is to identify the intent behind a user message.
Preparing Colab And Qwen3.5-4B
- Open Unsloth’s official decision-model guide or the Qwen3.5-4B Decision Model notebook in Google Colab. Make a copy before changing anything. In runtime settings, choose Change runtime type and select an available NVIDIA GPU. The experiment used a Tesla T4 with approximately 14.6 GB of GPU memory. The setup can run on that GPU, but training takes longer than it would on a newer GPU.
- Run the dependency installation cell from the notebook. It installs uv, the compatible PyTorch and Transformers versions, Unsloth, and required GPU packages. The cell uses a capture directive so installation logs stay hidden from the notebook.
- Load Qwen3.5-4B with FastDecisionModel instead of the usual language-model loader. Use 4-bit quantization to reduce memory consumption. The decision head starts in a trainable state, and the pretrained model supplies language understanding.
- Configure Low-Rank Adaptation rather than updating every model parameter. A rank of 16 is used as a practical starting point for this experiment. The decision head is trained as well, and gradient checkpointing is enabled to reduce GPU memory use during training.
Reshaping CLINC150 Into Decision Training Data
- Download and inspect CLINC150. It is an intent classification dataset with 150 intent categories covering banking, travel, shopping, and everyday assistant tasks. The dataset has 15,000 in-scope training rows, 3,000 validation rows, and 4,500 in-scope test rows, with separate out-of-scope rows kept apart. Every record pairs a user message with the intent label for that message.
- For the first Colab experiment, select 10 intents from CLINC150. You can increase the number of intents, but a larger choice set increases input length and training requirements.
- Convert each record into the fields Unsloth expects for decision training: state, which is the input the model analyzes; questions, which are the decisions the model must make; and gold, which is the correct answer used during training. For this task, questions is a Choice question built from the selected intents.
- Exclude out-of-scope records from this first experiment. Their detection is a separate task that can be discussed after the classifier is evaluated. Each converted row now contains a natural-language user request, the set of available intents, and the correct intent label.
- Build the training dataset with Unsloth’s built-in dataset builder. It tokenizes the examples and converts them into the format required by the decision head. Use CLINC150’s original training, validation, and test partitions so the final test data stays separate from training and calibration.
Training And Measuring The Intent Classifier
- Evaluate the model before training. Since the decision head is fresh, its early predictions may resemble random guessing. On a balanced 10-class task, random guessing would reach roughly 10% accuracy. That score becomes the baseline for later comparison.
- Use Unsloth’s DecisionTrainer, which is designed for fine-tuning decision models. This experiment uses 60 training steps, matching the official Unsloth notebook’s quick-training example. For a more complete run, increase the number of steps or train for multiple epochs.
- Start training. As training runs, Unsloth logs metrics including training loss and runtime. In this run, validation accuracy climbed from 18.5% before training to 100% after training. That is strong for this small 10-intent subset, but it is not a claim of 100% accuracy on all 150 intents or on out-of-scope requests.
- Calibrate confidence scores with Unsloth’s built-in calibration method. Correct choices are not enough; the probabilities should match how often those choices are right. Calibration makes returned probabilities more trustworthy. Expected calibration error compares predicted confidence with observed accuracy; lower is better.
- Evaluate the trained model on the held-out 300-example test split. This is important because those test examples were not used for training or confidence calibration. In this reported run, all 300 examples were classified correctly, and the run reported zero loss and expected calibration error. Those are run outputs, not a promise of perfect generalization. Before deployment, test harder, ambiguous, and out-of-scope messages.
- Make predictions with a simple function that accepts a user query and returns the model’s predicted intent. For each message, the model returns an intent label and a probability for each option.
- Save the trained Low-Rank Adaptation adapters and the decision head. If you prefer a merged artifact, Unsloth can also save a full model. It can be reloaded with FastDecisionModel.from_pretrained() and used in Python applications or supported decision-model serving workflows.
Common Mistakes
- Treating 100% accuracy on the narrowed 10-intent test as proof that the model handles all 150 CLINC150 intents.
- Mixing out-of-scope examples into the first Choice experiment without defining out-of-scope detection as its own task.
- Skipping calibration and trusting confidence scores before checking expected calibration error.
- Increasing the number of intents without accounting for longer inputs and higher GPU memory demand.
- Loading Qwen3.5-4B as a normal language model instead of using FastDecisionModel, which prevents the decision head from being trained.
This approach matters for readers because it turns a general language model into a tool for routing user requests. The result is practical for small experiments, but deployment needs broader evaluation, ambiguous cases, and out-of-scope messages. You can use the same method with your own dataset and explore how decision models fit into applications where structured choices are more useful than long responses.
Comments (0)