Creating an Art Recommender from the Ground Up Using CLIP and ChromaDB

Meta description: Learn how a tutorial uses CLIP and ChromaDB to embed art images, store vectors, and prepare semantic retrieval for an art recommender system.

Tags: CLIP, ChromaDB, vector database, art recommender, semantic search

Featured image: Abstract illustration of framed artworks linked by glowing lines to a grid of vector dots, friendly colors, no people, no logos, no text.

A new tutorial, Part II of a project, shows developers how to turn previously collected high-quality art data into a vector database using CLIP and ChromaDB. The goal is to store image embeddings so an art recommender can retrieve works by semantic similarity, not just by author, year, or color. It is aimed at readers who already have images and metadata and want a local, persistent data layer for later vector queries. The tutorial uses Rocks at Port-Goulphar, Belle Île by Claude Monet, retrieved from the API of the Art Institute of Chicago (ARTIC), as its visual anchor.

Why semantic embeddings matter for an art recommender

Embedding is not just a technical step. At the most basic level, machine learning models need structured numerical representations, but images can be represented in many ways. A metadata-driven embedding could use author, year, and colors; a raw pixel embedding could use intensity values for each pixel. If the project used only metadata, a SQL database could be a simple and widely adopted choice.

For this project, the chosen target is a recommender, and a basic recommender depends on similarity, especially semantic similarity. The tutorial gives a concrete test: if a user loves naturalistic artworks, recommending Monet paintings merely because they contain green is too shallow. Two paintings from different eras, regions, and artists may still belong together if both show a forest at dawn, because the shared theme is nature. The tutorial says encoding this kind of information in a vector is not trivial, but notes that AI labs have been tackling the problem and have released results as open-source models.

This is why the project uses CLIP, a well-known foundation model developed by OpenAI. CLIP is multi-modal: it can encode images and text in the same vector space. The tutorial explains the intuition as training a model to map images and their descriptions to the same vector, while mapping different images and different descriptions to different vectors. It also notes that the core mathematics can be described as matrix diagonalization. For readers who want more detail, the tutorial points to a Medium article, the original OpenAI blog post, and the scientific paper by Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., … & Sutskever, I. (2021, July), Learning transferable visual models from natural language supervision, published in the International conference on machine learning, pages 8748–8763. The tutorial also shows a figure from that paper illustrating how images and captions are mapped to the same vector space. It recommends reading separately about embedding spaces, vector similarity, and distance measures, while saying those concepts are not required to follow the steps.

What the vector database must do

ChromaDB is an open-source, easy-to-use library for building vector databases. It is designed for intuitive vector storage and retrieval and considers multi-modal use. The tutorial wraps ChromaDB's API in a class called ChromaDBGenerator. That class stores image data as vectors and retrieves it through text or vector queries. The wrapper imports ChromaDB utilities, JSON and image paths, embedding data models, and a helper for cosine similarity.

The class controls persistent storage, collection naming, similarity settings, embedding functions, and data loading. The persistent client, chromadb.PersistentClient(path=CHROMA_DATA_PATH), keeps the database on local disk, so a new session can use the data without re-embedding it. The collection is created with client.get_or_create_collection(...), using embedding_functions.OpenCLIPEmbeddingFunction(), ImageLoader(), and metadata from a vector_similarity dictionary. The dictionary defines three similarity options: cosine_similarity as hnsw:space: cosine, l2_norm as hnsw:space: l2, and inner_product as hnsw:space: ip. The tutorial says cosine similarity is the default and points to HNSW as a core retrieval concept, without requiring readers to master it at this stage.

The tutorial notes that its code may be older than current ChromaDB features and encourages readers to check the latest documentation. It also notes that, at the time of writing, OpenCLIPEmbeddingFunction is the only image embedding model available inside ChromaDB, and that other embedding methods can be generated externally and then stored.

The class also handles metadata. It reads .jpg files and matching .json files, then filters metadata to values ChromaDB accepts: strings, integers, floats, and booleans. Adding metadata is optional because None values can cause problems. If metadata is clean, the tutorial says adding it can make retrieval more nuanced. Because CLIP is multi-modal, the same collection can also accept text documents through add_text_data_to_collection(ids, documents), not only image files.

Step-by-step setup with CLIP and ChromaDB

1. Start from the collected art data. The previous tutorial part gathered high-quality images and related JSON files. This step assumes those files already exist.

1. Choose semantic embedding over metadata-only embedding. For a recommender, the project selects CLIP so image and text queries can live in the same vector space.

1. Configure ChromaDB's persistent storage. Create a path with CHROMA_DATA_PATH and give the collection a stable collection_name, because multiple collections can be handled through one instance.

1. Set the similarity method. Use similarity_measure="cosine_similarity" by default, or select l2_norm or inner_product through the vector_similarity dictionary.

1. Instantiate the client and embedding function. Use chromadb.PersistentClient and OpenCLIPEmbeddingFunction. The tutorial notes that, at the time of writing, this is the only available image embedding model inside ChromaDB, and that other embedding methods can be generated externally and then stored.

1. Include an image loader. Use ImageLoader() when include_data_loader is True, so images are preprocessed consistently before embedding.

1. Create or fetch the collection. Call get_or_create_collection with the collection name, embedding function, data loader, and similarity metadata.

1. Load image names and metadata. retrieve_images_and_jsons scans for .jpg files, opens the matching .json file for each image, removes values that do not conform to ChromaDB requirements, and stores the cleaned metadata in self.jsons.

1. Add images to the vector collection. add_image_data_to_collection builds ids from image names without .jpg, uris from images_path, and optional metadatas from the JSON files.

1. Add text data if needed. add_text_data_to_collection inserts documents with their ids, enabling text-to-image retrieval through CLIP's shared vector space.

1. Prepare for embedding retrieval. The tutorial identifies the next feature as retrieving embeddings corresponding to selected images, which supports later similarity queries.

What this enables next

This setup gives the art recommender a local vector layer that can retrieve images by visual meaning, compare images to text, and reuse stored embeddings across sessions. It also prepares the project for the next described feature: fetching embeddings for selected images. The tutorial notes that real database design is iterative, because more features may become desirable once the app is tested. That matters for readers because semantic retrieval can support recommendations across different artists, eras, and regions when works share themes, such as nature at dawn, without relying on shallow color or author filters. The source leaves more advanced embedding-space theory as optional background, so the practical path is clear: store CLIP vectors in ChromaDB, keep metadata clean, and build queries from there.

Common mistakes

• Treating embeddings as only a technical detail. The project needs semantic similarity, so metadata or raw pixel values alone may not match user intent.
• Switching collection names inconsistently. A persistent ChromaDB collection should have a stable name when multiple collections are used through the same instance.
• Adding messy metadata without checking values. None values and unsupported types can cause retrieval or storage problems.
• Assuming CLIP is only for images. Its multi-modal design lets text and images be queried in the same vector space.
• Ignoring similarity choice. The source defaults to cosine similarity, but l2_norm and inner_product are available through hnsw:space settings.
• Re-embedding data unnecessarily. A persistent client keeps local storage so a new session can reuse the vectors.