Text clustering aims to cluster texts based on their semantic content.

INPUT (unstructured textual data): ['cats', 'dogs', 'elephants', 'birds', 'cars', 'trains', 'planes'] OUTPUT (clusters of semantically similar data): Cluster 1 ['cats', 'dogs', 'elephants', 'birds'], Cluster 2 ['cars', 'trains', 'planes']
To cluster documents we follow these three steps:

• Embedding documents:
  First we need to convert the documents to embeddings using an embedding model. The created embeddings will represent the features that we want to cluster. We need to choose a model that was trained for semantic similarity tasks.
  
  Example:
  INPUT: ['cats', 'dogs', ...] OUTPUT: cats [0,0,0,0,1], dogs [0,0,0,0,2], ...
  
• Reducing the dimensionality of the documents's embeddings:
  To ease clustering a large volume of data we can choose to reduce the dimensions of the embeddings using a dimensionality reduction library. This process might cause the loss of information and hence the clustering might not be very accurate.
  
  Example:
  INPUT (embedding with dimension 5): [0,0,0,0,1] OUTPUT (compressed embedding with dimensions 3): [0,0,1]
  See this page for more details about UMAP (Uniform Manifold Approximation and Projection) for dimension reduction:
  https://umap-learn.readthedocs.io/en/latest/index.html
  
  
• Clustering the reduced embeddings:
  The last step is to use a clustering library to find groups of semantically similar documents.
  
  Example:
  INPUT: (embeddings): [0,0,1], [0,0,2] OUTPUT: cluster 0 ['cats', 'dogs', 'elephants', 'birds'], cluster 1 ['cars', 'trains', 'planes']
  See this page for more details about the HDBSCAN clustering Library:
  https://hdbscan.readthedocs.io/en/latest/index.html
  
  

Example: Clustering a set of words

Required modules:
$ pip install umap-learn $ pip install hdbscan $ pip install matplotlib
Python code:
from sentence_transformers import SentenceTransformer from umap import UMAP from hdbscan import HDBSCAN from matplotlib import pyplot import numpy as np model = SentenceTransformer("sentence-transformers/all-MiniLM-L12-v2") # create the embeddings of a set of words sentences = ['cats', 'dogs', 'elephants', 'birds', 'cars', 'trains', 'planes'] embeddings = model.encode(sentences) print(f'Number of the embedded documents and their dimensions: {embeddings.shape}') # reduce the embeddings dimensions reduced_embeddings = UMAP(n_components=5, random_state=42).fit_transform(embeddings) print(f'Number of the embedded documents and their reduced dimensions: {reduced_embeddings.shape}') # Create an HDBSCAN object and fit the model to the data cluster_algorithm = HDBSCAN(min_cluster_size=2).fit(reduced_embeddings) # Get the cluster labels (Note: -1 means noise points) cluster_labels = cluster_algorithm.labels_ # Get the number of clusters n_clusters = len(set(cluster_labels)) - (1 if -1 in cluster_labels else 0) print(f'Number of clusters: {n_clusters}') # Print features in cluster 0 print("Cluster 1:") cluster = 0 for index in np.where(cluster_labels==cluster)[0][:10]: print(f'Feature {index}: {sentences[index][:10]}') # Print features in cluster 1 print("Cluster 2:") cluster = 1 for index in np.where(cluster_labels==cluster)[0][:10]: print(f'Feature {index}: {sentences[index][:10]}') # Plot the results pyplot.scatter(reduced_embeddings[:, 0], reduced_embeddings[:, 1], c=cluster_labels, cmap='Spectral', s=40) pyplot.colorbar() pyplot.savefig('hdbscan_cluster_plot.png')
Output:
Number of the embedded documents and their dimensions: (7, 384) Number of the embedded documents and their reduced dimensions: (7, 5) Number of clusters: 2 Cluster 1: Feature 0: cats Feature 1: dogs Feature 2: elephants Feature 3: birds Cluster 2: Feature 4: cars Feature 5: trains Feature 6: planes
Chart of the clusters: