Creating a Retrieval-Augmented Generation Pipeline Using LLM Embeddings and Scikit-learn

Build a Qwen2.5-0.5B Retrieval-Augmented Generation Pipeline with scikit-learn and sentence-transformers

Meta description: Learn how to build a lightweight retrieval-augmented generation pipeline using scikit-learn, all-MiniLM-L6-v2, and Qwen/Qwen2.5-0.5B-Instruct for free.

Tags: retrieval-augmented generation, scikit-learn, sentence-transformers, Qwen2.5-0.5B, local embeddings

Featured image: abstract, friendly illustration of text documents becoming vector points and a small model answering a question, no real people, no logos, no text.

For developers who need a lightweight RAG pipeline, or retrieval-augmented generation pipeline, without paid APIs or heavyweight frameworks, this guide shows how to build an in-memory, end-to-end system using scikit-learn for vector retrieval, all-MiniLM-L6-v2 sentence-transformers for embeddings, and Qwen/Qwen2.5-0.5B-Instruct for text generation. The guide notes that readers can brush up on basic RAG concepts before following along.

Why a lightweight pipeline is enough

Modern RAG systems are often thought of as inseparable from an ecosystem of frameworks, including typically expensive managed vector databases used to store document collections for retrieval. At the core, however, a RAG system relies on a fairly simple notion: calculating distances between semantic vectors that represent texts. Because of that, a dedicated AI infrastructure is not strictly necessary when the document base is small, when a team is prototyping a new proof of concept, or when deployment costs must stay low. The method described here uses scikit-learn procedures together with open-source LLMs to build transparent, fast RAG pipelines that remain fully functional.

Build the pipeline in seven steps

1. Choose the runtime. The pipeline can be implemented and run in a local IDE, but the guide recommends Google Colab for the demonstration. Using Colab’s free T4 GPU makes execution more efficient. Start by installing and importing the required libraries and dependencies.

1. Load the miniature document base. The guide makes a small dataset available in a public GitHub repository. The dataset contains nine text files with brief descriptions of Asian countries. Download the files, extract them, and save the texts in a simple list. This list becomes the document base for the RAG system.

1. Chunk the texts. Before embeddings are created, the texts need to be put into the right format, meaning they must become embedding vectors. Chunking is common practice in RAG because it turns documents into smaller, semantic pieces that the embedding model, responsible for mapping pieces of text into numerical embeddings, can process more effectively. The source texts are short, so they do not really require chunking, but the pipeline simulates this step to illustrate best practices for real-world RAG systems. The guide defines a native chunking function with smaller-than-usual values for chunk size and overlap, both measured in words, so the effect can be observed on the small documents. In a real application, larger values for both arguments are likely more useful.

1. Generate embeddings. Once the chunks are ready, the pipeline creates embedding vectors with the all-MiniLM-L6-v2 sentence-transformer. This popular lightweight model maps text pieces into numerical embeddings, which can then be compared by the retriever.

1. Create the retriever. The retriever is one of the key components of a RAG pipeline. Instead of a vector database, the guide wraps sklearn.neighbors.NearestNeighbors, a simple yet effective distance-calculation algorithm, into a clean, reusable class. The class mimics vector-database functionality while using cosine distance, which is ideal for embedding vector representations.

1. Add the generator. The other major component is the generator. The pipeline uses a small, open-source LLM trained for natural language generation from Hugging Face: Qwen/Qwen2.5-0.5B-Instruct. The model is small enough to run smoothly in a Colab notebook without risking a RAM crash.

1. Wire the pieces and test. With the embeddings, retriever, and generator in place, the last step is to connect them into one in-memory pipeline. The guide asks the system a question about its knowledge base, which covers Asian countries. The text in test_question can be replaced to try different questions.

What the result means for readers

The completed pipeline is completely free to use and avoids the overhead of heavyweight frameworks or paid APIs. It relies solely on scikit-learn vector retrieval and open-source models for both embeddings and generation, producing a transparent system that runs in memory. For readers, the value is not only the final answer to a sample question, but also a solid starting point for prototyping, experimenting with different retrieval strategies, and validating concepts before scaling the system.

Related directions from the guide

The guide also points readers toward related pipeline work: building a RAG pipeline with llama.cpp in Python, combining LLM embeddings with tabular features, combining LLM embeddings with term frequency-inverse document frequency and metadata, sentiment analysis pipelines, robust machine learning pipelines, and context pruning for long-running agents.

Common mistakes

Expecting commercial LLM-level results from this setup. The guide uses a small model and a tiny knowledge base, so behavior and outputs are reasonable for the small-scale system, not for production-quality generation.

Skipping chunking in real documents. The nine short text files do not need aggressive chunking, but larger corpora do. The pipeline includes chunking to demonstrate the step that becomes important in real-world RAG systems.

Using chunk size and overlap values that are too small. The guide intentionally uses smaller-than-usual values to make the effect visible on short documents. Real applications usually need significantly larger values for both arguments.

Overlooking cosine distance. The retriever uses sklearn.neighbors.NearestNeighbors with cosine distance because cosine distance is well suited to embedding vector representations.

Assuming the free Colab setup will handle larger workloads. Qwen/Qwen2.5-0.5B-Instruct is chosen because it is small enough for a notebook without risking a RAM crash; bigger corpora or bigger models may not fit the same setup.