t-SNE

t-SNE stands for t-Distributed Stochastic Neighbor Embedding.

The "t" in t-SNE comes from the t-distribution, a probability distribution that is used to compute the similarity between data points in the low-dimensional space. The t-distribution has fatter tails than the Gaussian distribution, which allows it to better handle the crowding problem that can occur in high-dimensional data when many points are close together.

The "SNE" in t-SNE stands for Stochastic Neighbor Embedding, which uses a different cost function to minimize the difference between the pairwise similarities in the high-dimensional space and the pairwise similarities in the low-dimensional space.

Unlike linear dimensionality reduction techniques such as PCA, t-SNE aims to preserve the pairwise distances between data points in the high-dimensional space by mapping them to a low-dimensional space.

How t-SNE works

1. Given a dataset of high-dimensional data points, the first step is to compute the pairwise similarities between each data point. t-SNE uses a Gaussian kernel to compute the similarity between two data points in the high-dimensional space, which can be interpreted as the probability that the two points are neighbors.

2. Next, t-SNE creates a low-dimensional map and computes pairwise similarities between data points in the low-dimensional space using a similar Gaussian kernel.

3. t-SNE then aims to minimize the difference between the pairwise similarities in the high-dimensional space and the pairwise similarities in the low-dimensional space which is achieved by minimizing the Kullback-Leibler divergence between the two distributions.

4. t-SNE uses gradient descent to optimize the embedding of the data points in the low-dimensional space, adjusting their positions until the pairwise similarities in the low-dimensional space are as close as possible to the pairwise similarities in the high-dimensional space. 

t-SNE Example

Here's an example of using t-SNE to visualize the MNIST handwritten digits dataset

python
from sklearn.datasets import load_digits 
from sklearn.manifold import TSNE 
import matplotlib.pyplot as plt 

# Load the MNIST dataset                     
x, y = load_digits(return_x_y=True)

# Apply t-SNE to reduce the dimensionality of the data to 2 dimensions
tsne = TSNE(n_components=2, random_state=42)
x_tsne = tsne.fit_transform(x)

# Plot the results 
plt.scatter(x_tsne[:, 0], x_tsne[:, 1], c=y, cmap=plt.cm.get_cmap('jet', 10))
plt.colorbar(ticks=range(10))
plt.clim(-0.5, 9.5)
plt.show()

In this example, we first load the MNIST dataset of handwritten digits then apply t-SNE with two components to reduce the dimensionality of the data and create a two-dimensional map. Finally, we plot the resulting low-dimensional embedding of the data points, coloring each point according to its true class label.

The resulting plot will shows the handwritten digits separated into distinct clusters, with similar digits (e.g 0 and 6, 2 and 7) located close to each other in the low-dimensional space. This demonstrates how t-SNE can be used to visualize high-dimensional data in a meaningful way, revealing underlying patterns and relationships that may be difficult to discern in the original high-dimensional space. 

What are the hyperparameters in t-SNE

There are several hyperparameters in t-SNE that can affect the quality of the embedding and the performance of the algorithm. 

Perplexity 

Perplexity is a measure of the effective number of neighbors for each point in the high-dimensional space. This hyperparameter maintains the balance between local and global structure in the resulting embedding. A good starting value for perplexity is usually between 5 and 50, and it is often recommended to try multiple values and inspect the resulting embeddings to choose the best one.

Learning rate 

The learning rate controls the step size of the optimization algorithm used to minimize the cost function. A higher learning rate can result in faster convergence but may also lead to unstable embeddings, while a lower learning rate can be more stable but may result in slower convergence. A good starting value for the learning rate is often around 200, but it may need to be adjusted depending on the dataset and the desired level of convergence.

Number of iterations 

The number of iterations determines the maximum number of steps the algorithm takes to optimize the embedding. A higher number of iterations can result in a better quality embedding, but it can also be more computationally expensive. A good starting value for the number of iterations is often between 500 and 1000, but it may need to be increased or decreased depending on the dataset and the desired level of convergence.

Metric 

The metric used to compute distances between data points can affect the quality of the embedding. Euclidean distance is often used by default, but other metrics such as cosine distance or Mahalanobis distance may be more appropriate depending on the dataset and the research question.

Choosing the optimal hyperparameters for t-SNE can be challenging, and it often requires a combination of experimentation and visualization. It is often recommended to try a range of values for each hyperparameter and evaluate the resulting embeddings using visualization techniques such as scatterplots or heatmaps. It is also important to keep in mind that the optimal hyperparameters may depend on the dataset and the research question, so it may be necessary to experiment with different values for each new dataset.

t-SNE Benifits

1. t-SNE is able to capture nonlinear relationships between high-dimensional data points that may be difficult to discern in the original space.

2. t-SNE is a flexible algorithm that can be used with a wide range of distance metrics and parameters, allowing users to tailor the algorithm to their specific needs.

3. t-SNE produces a low-dimensional representation of high-dimensional data that is easy to visualize and interpret. This can be particularly useful for exploring complex datasets and identifying patterns and relationships that may be difficult to discern in the original space.

t-SNE Limitations

1. t-SNE can be computationally expensive for very large datasets or high-dimensional data. For datasets with more than 10,000 data points, it may be necessary to use approximations or subsampling to reduce the computational cost.

2. t-SNE uses random initialization and optimization techniques, which can lead to different embeddings for the same dataset and hyperparameters. This can make it difficult to reproduce results or compare different embeddings.

3. t-SNE can be prone to overfitting, particularly for datasets with small sample sizes or noisy data. It is important to carefully choose the hyperparameters and perform cross-validation to ensure that the embedding is not overfitting to the data.

4. While t-SNE produces a low-dimensional embedding that is easy to visualize, interpreting the embedding can be challenging. The meaning of the distances between data points in the embedding is often unclear, and it can be difficult to draw meaningful conclusions about the underlying structure of the data.