Create an art recommendation system from scratch using the Contrastive Language-Image Pre-training model and ChromaDB

Meta description: Learn how a user embedding can power a swipe-based art recommender, using liked and disliked artworks to suggest pieces and museums for visitors.

Tags: user embedding, recommender systems, art technology, museum apps, vector search

Featured image: abstract friendly illustration of artwork cards moving into a personalised museum route, with no real people, logos or text.

This guide is for developers building a swipe-based art recommender. Based on Part III: The recommender, it explains how a user embedding can summarise liked and disliked artworks, suggest new pieces, and connect preferences to museums in a city. The source frames the final chapter of a series that began in Part I and Part II, and opens with The Merry Fiddler by Gerard van Honthorst, retrieved from the Rijksmuseum API.

Why the system is framed as a “Tinder for art”

The concept gives users artworks to swipe through, with left and right movements feeding a user profile. That profile can then recommend more artworks and museums to visit in a specific city. The author notes that this is the most subjective chapter, because a different vision could have produced different components and features.

The intended capabilities include showing artworks for like or dislike responses, preventing repeated artwork appearances, balancing exploration of unseen pieces with exploitation of recommended pieces, providing a personalised analytics page for most liked author, style, material and location, and suggesting museums from preferences and location. The article does not implement all of this fully. The author says a complete version would supply enough material for three more articles at least, but basic versions can still fit with the work already built in Part I and Part II.

How a user embedding turns likes and dislikes into one vector

The starting idea is similarity: recommend artworks close to those the user liked. As interactions accumulate, scanning every liked artwork becomes harder. The user embedding compresses those signals into one object. The author acknowledges this choice may seem arbitrary at first, but says it becomes useful because it places users and artworks in the same vector space, giving a consistent way to compare entities and use similarity search for recommendations.

The method sits inside a user class with other functionality that can support the app workflow. The function create_or_update_user_vector creates or updates the vector. It first removes potential duplicates from newly liked and newly disliked image ids and vectors, then records the new likes and dislikes. Liked vectors are kept as they are, while disliked vectors are multiplied by -1.0. The combined vectors are normalised through create_normalized_vector. Inverting disliked vectors treats a dislike as support for the opposite direction in the embedding space. The author chose this to use information from disliked items, but says readers can replace it with another logic. The author also notes that a temporary numpy array conversion is used until the final structure is decided.

When a user already has self.vector, the update does not recompute the embedding from every old interaction. That would be inefficient. Instead, the existing vector is scaled by a weighing factor: the total number of liked and disliked artworks minus the number of new vectors being added. The scaled existing vector is then added to the new liked and inverted disliked vectors before normalisation, and the result is stored as user_vector in user_preferences_db at the user’s index.

The scalar example makes the arithmetic clear. If a user likes an artwork with embedding 4 and dislikes one with embedding 2, the initial user vector is (4 - 2) / 2 = 1. If the user then likes an artwork with embedding 1, the same result can be computed explicitly as 4 - 2 + 1 = 3, then 3 / 3 = 1. It can also be computed as (1 × 2 + 1) / 3 = 1, where the previous user vector is scaled by the two artworks used to create it. The author says more efficient embedding methods may exist and invites readers to explore them.

Build the basic recommendation flow

1. Use efficient_diverse_vectors_retrieval to get an initial selection of varied artworks through the efficient method for retrieving diverse vectors.
2. Get the image_path from each artwork and display the artwork.
3. After the user interacts with several pieces, compute the user embedding with create_or_update_user_vector.
4. Retrieve recommended images with find_n_most_similar_to_vector from the ChromaDB class.
5. In the source example, the user likes three artworks and dislikes one, then the top three recommended artworks from the collection are shown.
6. Repeat the flow, passing the liked and disliked vectors combined as image_paths_to_avoid to efficient_diverse_vectors_retrieval so previously seen artworks are not shown again.

The author warns that recommendation quality here is subjective. Without proper data collection from multiple users’ interactions, there is no way to confirm whether the top three artworks match the user’s taste. The method is a starting strategy, and experiments can adjust parameters and measure their impact on interactions.

Personalised queues and engagement temperature

The author says the basic workflow can feel repetitive. Many recommendation engines use continuous streams, where a feed adapts instantly as the user scrolls. The article does not reach that level, but it proposes a basic queue design with two parts: an exploration queue for diverse images and an exploitation queue for recommended images. New images can be appended at regular intervals while the user is swiping, and the queues can be adjusted from interactions.

A temperature parameter is meant to track engagement. If temperature falls, the user is considered less engaged, and the system should show more recommended artworks. If temperature rises, the author says the app can dare a bit more exploration, though the source passage stops before specifying the exact high-temperature behaviour.

Other experiments suggested in the source include changing the weight of disliked artworks from -1 to -0.5, giving recently liked or disliked artworks twice the weight in the final user embedding, and adding a super like that gives an artwork extra weight.

Common mistakes

• Treating the embedding result as objective truth. The author calls this subjective and notes that multi-user interaction data is needed before judging taste match.
• Recomputing the user vector from all previous embeddings every time. The scaled existing vector is meant to avoid that inefficiency.
• Forgetting duplicate handling. create_or_update_user_vector removes new image ids and vectors that already appear in liked or disliked records.
• Ignoring disliked signals. Inverted disliked embeddings are part of the method, but their weight can be changed.
• Letting exploration and exploitation queues become too long. The simple approach keeps the queue short so it can adjust quickly.
• Skipping the no-repeat rule. The source wants artworks not shown twice, and image_paths_to_avoid supports that basic requirement.