Follow this step-by-step guide to fine-tune Qwen2.5 using Low-Rank Adaptation and publish your model on Hugging Face.
A guide to fine-tune Qwen2.5-0.5B-Instruct for fashion JSON attribute extraction via teacher labels, low-rank adaptation, Colab GPU, and Hugging Face.
Key Takeaways
- The tutorial fine-tunes Qwen2.5-0.5B-Instruct to convert clothing descriptions into structured JSON fashion attributes such as category, color, material, fit, pattern, sleeve_length, and neckline.
- Qwen2.5-7B-Instruct is used as a teacher model to generate labels with 4-bit quantization on a Google Colab T4 GPU, while the smaller student model is adapted using LoRA instead of full fine-tuning.
- A strict output schema and carefully defined prompt are essential to ensure consistent JSON-only responses and prevent the model from inventing unsupported attributes.
- The workflow includes loading and filtering an e-commerce product dataset, creating reproducible train/validation/test splits before label generation, and saving intermediate files to Google Drive.
- The final LoRA adapter can be published on Hugging Face for applications such as catalog enrichment, product filters, search, and recommendation pipelines.
Meta description: Learn how to fine-tune Qwen2.5-0.5B-Instruct for fashion attribute extraction using teacher labels, LoRA, and a Google Colab T4 GPU in this tutorial.
Tags: fashion attribute extraction, Qwen2.5, LoRA, Google Colab, Hugging Face
Featured image: A friendly abstract illustration of a small language model turning clothing descriptions into structured JSON fields on a laptop, with no people, logos, or text.
This tutorial is for developers and machine-learning learners who want a compact fashion attribute extractor that converts ordinary clothing descriptions into structured JSON fields. The workflow fine-tunes Qwen2.5-0.5B-Instruct with labels drafted by Qwen2.5-7B-Instruct, uses LoRA (low-rank adaptation) on a Google Colab GPU, and ends with an adapter published on Hugging Face. The original experiment also provides a GitHub notebook and a published model named Fashion Attribute Extractor on Hugging Face.
What the extractor produces
The task is intentionally narrow. A description such as "Relaxed-fit cotton hoodie with long sleeves, a drawstring hood, and a printed graphic on the front" should return an object with category, color, material, fit, pattern, sleeve_length, and neckline. In the example, category is hoodie, material is cotton, fit is relaxed-fit, pattern is printed, sleeve_length is long, while color and neckline are null.
Every field must contain either a string or null. The category field is not used to invent a new garment type; it preserves the catalog category supplied with the product description. A fashion e-commerce platform could use these structured attributes for catalog enrichment, product filters, product search, and recommendation pipelines.
Step-by-step build
- Start in Google Colab with a GPU. Create a new notebook and, when available, select Runtime, Change runtime type, and T4 GPU. Verify PyTorch and CUDA before loading models. The author used a Tesla T4 with approximately 14.56 GB of GPU memory. A GPU matters because the teacher model has 7 billion parameters and the student model has 0.5 billion parameters. Colab runtime availability can vary.
- Install the needed Python packages. The streamlined path uses transformers, datasets, accelerate, peft, bitsandbytes, huggingface_hub, and scikit-learn. The original notebook also used TRL, pandas, and PyArrow during preparation, but the simplified training path relies directly on Transformers Trainer. Successful runs used transformers 5.20.2, datasets 5.0.1, peft 0.21.2, accelerate 1.15.0, and bitsandbytes 0.50.2. For exact reproduction, pin the versions recorded in the notebook instead of depending on future defaults.
- Connect Google Drive early. Colab sessions can disconnect or reset. The author saved intermediate datasets, model checkpoints, and evaluation files in Drive under a fashion-distillation folder. This is optional, but useful when a project has several data-generation and training stages. The expensive work is not always training; generating and reviewing teacher labels took substantial effort, so preserving that work was important.
- Load the product dataset. The source used Qdrant/hm_ecommerce_products from Hugging Face, which contains product names, descriptions, and catalog metadata. Start by streaming the training split so the full dataset does not need to be loaded into memory. Print a few records first. This matters because the dataset has more fields than the extraction task needs.
- Clean and filter the descriptions. Raw catalog data can contain missing descriptions, repeated records, inconsistent categories, and products outside the task scope. The author selected an initial subset of 5,000 records and applied several filtering stages. A simplified filter keeps garment records with nonempty descriptions and valid categories. The original cleaned dataset contained 1,422 garment descriptions across 21 categories. A fresh run may produce a different count because the simplified filter does not implement every original duplicate-conflict check or garment-specific cleaning rule.
- Split the data before generating labels. Separate training, validation, and evaluation data prevents a model from being scored on examples it already saw. Use a reproducible train-test split with random_state 42 to create an approximately 80/10/10 division. In the original experiment, teacher labels were generated for an initial subset of 500 training-pool examples. After exclusions and review, 485 labeled examples remained. Those examples were divided into 436 student-training examples and 49 internal-validation examples. The outer test pool was kept separate for final exploratory evaluation.
- Define the output schema before asking the teacher for labels. Valid output must be one JSON object with all seven keys, strings or null values, no markdown fences, and no explanations. The rules also say to use the supplied catalog category, avoid inventing unsupported attributes, use null when an attribute is not established, identify the primary garment material rather than only a lining or trim, avoid confusing waist style with overall fit, and not treat a hood as a neckline shape. The tutorial prompt is shortened. The original experiment used a longer category-aware instruction prompt, and strict reproduction should use the prompt saved in teacher_config.json.
The prompt matters because the goal is consistent decisions across many descriptions, not merely plausible guesses. For example, a description that mentions a printed graphic but no color may have pattern printed while color remains null.
- Load the teacher model. The teacher is Qwen/Qwen2.5-7B-Instruct. Loading a 7B model directly in full 16-bit precision can exceed T4 memory, so the author used 4-bit quantization with nf4 weights, float16 compute dtype, and double quantization. The quantized teacher is used only for inference to draft labels; it is not fine-tuned in this workflow.
- Generate candidate labels. A helper function sends the catalog category and description to the teacher, applies the model's chat template, limits generation to 200 new tokens, uses greedy decoding, and parses the reply as JSON. For a real pipeline, parsing failures should be caught and the record should be retried or flagged for review. In the experiment, teacher predictions were processed in batches, saved to Drive, and questionable labels were reviewed.
- Train the student model. The student is Qwen2.5-0.5B-Instruct, trained with LoRA on the reviewed examples. This keeps the experiment smaller than full fine-tuning and makes the final artifact an adapter that can be published on Hugging Face.
- Evaluate and publish. The student model is evaluated against the original model. The GitHub notebook records the actual experiment, including debugging, repeated executions, and recovery checkpoints. The tutorial steps reorganize that work into a cleaner learning path. Because some data-cleaning and annotation rules are simplified, a fresh run will not necessarily reproduce the exact dataset or scores.
What this means for readers
This project is not a general fashion assistant. It is a practical distillation exercise: use a larger model to draft structured labels, review them, then teach a small model a narrow JSON format. That pattern is useful when a task needs many consistent outputs but does not need broad conversational ability. For readers, the value is a reproducible path from a product description to a smaller deployable model, while seeing where teacher labels, data cleaning, and schema rules affect the result.
Common mistakes
- Skipping the GPU check and then failing when a 7B model is loaded.
- Generating labels without a strict schema, which makes later evaluation noisy.
- Treating the teacher's JSON as automatically correct instead of reviewing questionable records.
- Evaluating the student on examples that were already part of training.
- Changing category extraction into independent garment classification.
- Not saving intermediate datasets, labels, checkpoints, and evaluation files before a Colab reset.
- Relying on default package versions when trying to reproduce the original notebook.
- Expecting exact dataset counts or scores from the simplified tutorial path.
Comments (0)