Can Text Embeddings Perfectly Capture Text?

Featured image: Abstract friendly illustration of vector numbers flowing into readable sentences, no real people, no logos, no text.

Meta description: A vec2text study shows text embeddings can reveal original sentences, recovering 92 percent of 32-token sequences and raising vector database privacy questions.

Tags: vector databases, embeddings, RAG, privacy, vec2text

For developers, security teams, and businesses using vector databases, a vec2text study presented at EMNLP 2023 delivers a practical warning: the numbers behind Retrieval Augmented Generation (RAG) systems may not hide the original text as well as many assume. In the paper Text Embeddings Reveal As Much as Text, the authors describe a method called vec2text that recovered 92% of 32-token sequences exactly and reached a BLEU score of 97 after 50 steps. That matters because many AI products store embeddings instead of documents and treat those embeddings as opaque enough to make privacy seem easier.

Vector databases grew because RAG made embeddings everywhere

The rapid rise of generative AI pushed many companies to build systems that answer questions from their own document collections. A common and inexpensive starting point is RAG, an approach associated with the paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In a typical setup, documents are stored in a database, the system retrieves the most relevant ones for a user’s question, and a language model generates an answer informed by the retrieved text.

Relevancy in RAG is usually determined by embeddings, vector representations produced by an embedding model. The working assumption is that relevant documents should be close together in embedding space, so similarity in numbers can stand in for similarity in meaning. As RAG became common, vector databases emerged as a new class of tools designed to store and search large numbers of embeddings. Startups claiming to make embedding search easier attracted hundreds of millions of dollars of funding, and many new applications began converting text into vectors for storage.

A software engineer building a RAG system may notice that the database does not contain the text itself. It contains embedding vectors, which can look like rows of random-seeming numbers. The original text may be protected by a privacy policy, but the system does not send the text to the database provider in the same readable form. That raises a question: if embeddings look random, can they still be turned back into text?

Embedding numbers look random, but they carry limits

Embeddings are not designed to be human-readable. Beyond the requirement of semantic similarity, there is no fixed rule saying which embedding must be assigned to a given text input. The numbers inside an embedding vector can vary based on their initialization, and people can interpret similarity between embeddings without understanding the individual numbers inside them.

A neural embedding model takes text as input and produces an embedding vector for search. Text embeddings are outputs of neural networks, built from sequences of matrix multiplications joined by nonlinear functions. In traditional text-processing neural networks, a string is split into token vectors that undergo repeated nonlinear operations. At the output layer, tokens are averaged into a single embedding vector.

A principle from signal processing known as the data processing inequality says functions cannot add information to an input; they can only sustain or decrease the information already present. Even though deeper layers of a neural network may appear to build higher-level representations, they are not adding new information about the world that did not come from the input. Nonlinear layers also destroy some information. A common nonlinear layer, ReLU, simply sets negative inputs to zero. After many such layers, a typical text embedding model cannot retain every detail of the original text.

The question is not new in all of machine learning. In computer vision, researchers have shown that deep representations can sometimes be inverted. A 2016 result associated with Dosovitskiy showed that images could be recovered from feature outputs of deep convolutional neural networks, producing blurry-but-similar versions of the original input. Work described in Understanding Invariance via Feedforward Inversion of Discriminatively Trained Classifiers showed that inversion models could reconstruct images given only the 1000 probability outputs of an ImageNet classifier, most of which are close to 0. Since 2016, image embedding inversion has improved, with higher-accuracy models and broader settings.

Vec2text turns a small vector example into readable text

The paper moves from that vision analogy to text. It starts with a toy setting: text inputs are limited to 32 tokens, about 25 words, and embedded into vectors of 768 floating-point numbers. At 32-bit precision, the paper’s example describes these embeddings as 32 * 768 = 24,576 bits, or around 3 kilobytes. The question is whether text can be perfectly reconstructed from that vector.

To evaluate success, the paper uses exact match, which asks how often the recovered text is identical to the input. It notes that no prior inversion methods had success on exact match, making this an ambitious target. It also uses BLEU score, a similarity measure that can be thought of as a rough percentage of how close the inverted text is to the original.

The first approach is straightforward: gather a large dataset of embedding-text pairs and train a model to output the text given the embedding as input. Prior work used a decoder-only architecture that inputs an embedding as a prefix. The paper initially trained an encoder-decoder model conditioned on an upscaled sentence embedding on the encoder side. This first model reached a BLEU score of around 30/100. In practice, it could guess the topic of the input and recover some words, but it lost their order and often got most words wrong. The exact match score was close to zero. The paper notes that asking one model to reverse the output of another model in a single forward pass is difficult, comparable to complicated generation tasks like producing perfect sonnets or satisfying multiple attributes.

A later architecture change made a difference. The final method includes an additional hypothesis text along with an upscaled hypothesis embedding. When the model generated text, the researchers re-embedded that hypothesis and compared it to the true embedding. The cosine similarity was around 0.97, meaning the generated text was close in embedding space but not identical to the original text. If the embedding model were lossy and mapped multiple different inputs to the same output, exact reconstruction would be impossible because the system could not tell which sequence produced the embedding. In practice, the paper says it did not observe these collisions.

That observation led to vec2text. The method is described as a kind of learned optimization. Given a ground-truth embedding, a current hypothesis text, and the hypothesis position in embedding space, a corrector model predicts a new text sequence closer to the true text. The system iteratively generates hypotheses, re-embeds them, and feeds them back as input. A single forward pass of correction raises the BLEU score from 30 to 50. With 50 steps and a few tricks, the method recovers 92% of 32-token sequences exactly and reaches a BLEU score of 97. The paper suggests that a BLEU score of 97 generally means nearly perfect reconstruction, perhaps with a few punctuation marks misplaced.

What this means for RAG builders and what remains unclear

For teams building RAG systems, the result changes how embeddings should be viewed. An embedding is not just a harmless search key. If a database is breached, or if a service provider could access embeddings and apply inversion techniques, the stored vectors may reveal original text in some cases. The paper’s strongest result is for 32-token sequences, so it does not claim that every long document can be reconstructed with the same reliability.

Several questions remain open. Because an embedding vector has a fixed number of bits, there must be a point where text length exceeds what the vector can perfectly preserve. The paper notes that some embedding models can embed documents up to thousands of tokens, and it leaves future work to analyze the relationship between text length, embedding size, and embedding invertibility. Another open question is defense: can embedding models be designed so embeddings remain useful for search while obfuscating the text that produced them?

The method itself is not limited to text. The core idea behind vec2text is iterative optimization in embedding space, and the paper says it does not use text-specific tricks. That suggests similar recovery problems may be studied across other modalities, as long as the system has black-box access to an embedding model. For now, the clearest lesson for RAG users is simple: embeddings may look random, but they are not automatically safe.