Is Cosine Similarity Really About Similarity? 🤔
Cosine similarity is often used to measure semantic similarity in learned embeddings, but this new paper by Netflix researchers raises some important concerns.
Here are the key takeaways:
1. Cosine Similarity can give arbitrary or meaningless results in some cases, especially with learned embeddings.
2. Scaling Freedom: The flexibility in how embeddings are scaled during training can lead to non-unique, misleading cosine similarities.
3. Regularization Effects: Different regularization techniques can significantly affect the learned embeddings and distort cosine similarity.
4. Dot Product vs. Cosine: The unnormalized dot product often yields more stable and reliable results compared to cosine similarity.
5. Matrix Factorization Models: In linear models, cosine similarity can be particularly problematic due to scaling issues.
6. Deep Learning Impact: These issues may be even worse in deep models where multiple layers and different regularizations are applied.
7. Proposed Remedies: Train models specifically for cosine similarity or avoid using embedding spaces altogether to improve reliability.
8. Caution: The authors recommend carefully considering how embeddings are learned and normalized before applying cosine similarity.
Full paper here: arXiv:2403.05440v1
#seo
#Cosinesimilarity
#netflix
