Embeddings are numerical representations of a token:

• They are also known as embedding vectors, vector representations, or vectors.
  
• They aim to capture the semantic (meanings, context, patterns) of the embedded text.
  
• They represent a token and its links with other tokens based on the contexts (contextualized tokens embeddings)
  
• They allow measuring the semantic similarity between two tokens.
  

Embeddings are used for:

• Text generation
  
• Text classification
  
• Text clustering
  
• ...
  

Embeddings have the following characteristics:

• The size (number of dimensions) of the embeddings is fixed and depends on the underlying embedding model.
  
• Each cell in the dimension row has a numerical value (property).
  
• The properties (numerical values) of the embeddings are used to represent tokens.
  

A tokenizer is used in the training process of its associated model:

• The model is linked with its tokenizer and can't be used with another tokenizer.
  
• The model holds an embedding for each token in the tokenizer's vocabulary.
  
• When a model is created, its embeddings and weights are randomly initialized.
  
• When a model is trained, its embeddings and weights are assigned proper values that capture the semantic of each token.
  

Types of embeddings:

• Embedding a token: creating one vector that represents the token.
  
• Embedding a sentence: creating one vector that represents the sentence (used for categorization, semantic search RAG).
  
• Embedding a document: creating one vector that represents the document (used for categorization, semantic search RAG).
  

When generating embeddings for text (longer than just one token) the embeddings should capture the meaning of the whole text. One way to generate those embeddings is to average the values of all the tokens' embeddings of the text.

Example: Generate tokens' embeddings

Download the model:
$ huggingface-cli download microsoft/deberta-v3-xsmall
Python code:
from transformers import AutoModel, AutoTokenizer model = AutoModel.from_pretrained("microsoft/deberta-v3-xsmall") tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-base") tokens = tokenizer('Hello Embeddings!', return_tensors='pt') for token in tokens['input_ids'][0]: print(tokenizer.decode(token)) # [CLS] [Hello] [Emb] [edd] [ings] [!] [SEP] output = model(**tokens)[0] print(output.shape) # torch.Size([1, 7, 384]) print(output)
Output:
[CLS] Hello Emb edd ings ! [SEP] torch.Size([1, 7, 384]) tensor([[[-3.3186, 0.1003, -0.1506, ..., -0.2840, -0.3882, -0.1670], [-0.5446, 0.7986, -0.4200, ..., 0.1163, -0.3322, -0.3622], [-0.1689, 0.6443, -0.0145, ..., 0.0207, -0.5754, 1.3607], ..., [ 0.0366, 0.0818, -0.0607, ..., -0.4793, -0.7831, -0.9185], [-0.0555, 0.3136, 0.2662, ..., 0.3092, -0.4876, -0.3294], [-3.1255, 0.1324, -0.0899, ..., -0.1426, -0.5295, 0.0731]]], grad_fn=<NativeLayerNormBackward0>)
Note that the created embeddings have the follwong size: 1, 7, 384

• 1: the batch dimenesion
  
• 7: seven tokens
  
• 384: each token is embedded in a vector of 384 values
  

The batch dimension can be larger than 1 in cases when multiple sentences are given to the model to be processed at the same time.

Example: Generate sentence embeddings

Install the Sentence Transformers library:
$ pip install sentence-transformers
Python code:
from sentence_transformers import SentenceTransformer model = SentenceTransformer("all-MiniLM-L6-v2") sentences = ["Hello Sentence Transformers library!.", "Generate sentence embeddings!"] embeddings = model.encode(sentences) print(embeddings.shape) # (2, 384)