Annoy Reviews and Details

Social recommendations and mentions

• How Spotify Decides What Plays Next

  Increased opacity: As models become deeper, interpretability drops. Building your own “transparent” playlist generators using open‑source tools (e.g., Annoy for nearest‑neighbor searches) can serve as a personal sanity check. - Source: dev.to / 3 months ago

• Should LLMs just treat text content as an image?

  Yeah, that's why Erik Bernhardsson came up with the Approximate Nearest Neighbors Oh Yeah algorithm (or ANNOY for short) > We use it at Spotify for music recommendations. After running matrix factorization algorithms, every user/item can be represented as a vector in f-dimensional space. This library helps us search for similar users/items. We have many millions of tracks in a high-dimensional space, so... - Source: Hacker News / 11 months ago

• Do we think about vector dbs wrong?

  The focus on the top 10 in vector search is a product of wanting to prove value over keyword search. Keyword search is going to miss some conceptual matches. You can try to work around that with tokenization and complex queries with all variations but it's not easy. Vector search isn't all that new a concept. For example, the annoy library (https://github.com/spotify/annoy), an open source embeddings database. - Source: Hacker News / about 3 years ago

• Vector Databases 101

  If you want to go larger you could still use some simple setup in conjunction with faiss, annoy or hnsw. Source: about 3 years ago

• Calculating document similarity in a special domain

  I then use annoy to compare them. Annoy can use different measures for distance, like cosine, euclidean and more. Source: over 3 years ago

• Can Parquet file format index string columns?

  Yes you can do this for equality predicates if your row groups are sorted . This blog post (that I didn't write) might add more color. You can't do this for any kind of text searching. If you need to do this with file based storage I'd recommend using a vector based text search and utilize a ANN index library like Annoy. Source: over 3 years ago

• [D]: Best nearest neighbour search for high dimensions

  If you need large scale (1000+ dimension, millions+ source points, >1000 queries per second) and accept imperfect results / approximate nearest neighbors, then other people have already mentioned some of the best libraries (FAISS, Annoy). Source: over 3 years ago

• [R] Unlimiformer: Long-Range Transformers with Unlimited Length Input

  Would be possible to further speed up the process with using something like ANNOY? https://github.com/spotify/annoy. Source: over 3 years ago

• Faiss: A library for efficient similarity search

  I like Faiss but I tried Spotify's annoy[1] for a recent project and was pretty impressed. Since lots of people don't seem to understand how useful these embedding libraries are here's an example. I built a thing that indexes bouldering and climbing competition videos, then builds an embedding of the climber's body position per frame. I then can automatically match different climbers on the same problem. It works... - Source: Hacker News / over 3 years ago

• How to find "k" nearest embeddings in a space with a very large number of N embeddings (efficiently)?

  If you just want quick in memory search then pynndescent is a decent option: it's easy to install, and easy to get running. Another good option is Annoy; it's just as easy to install and get running with python, but it is a little less performant if you want to do a lot of queries, or get a knn-graph quickly. Source: over 3 years ago

• [Discussion] NLP for products matching

  Probably I won't be bale to explain better than it's stated on annoy page: https://github.com/spotify/annoy But the bottom line is speed. Instead of computing similarities of embeddings one by one you do it via index that works way faster. Source: over 3 years ago

• Do i really need a vector database

  Perhaps you can store your embeddings anywhere (sql or even a file) and use Approximate Nearest Neighbors like https://github.com/spotify/annoy for comparison? Source: over 3 years ago

• Comparing millions of image hashes in rust

  Hi, I have a huge list of hashes of images, that I have to compare and find matching items and delete duplicates. Is there something similar to spotify/annoy in Rust or BK-Tree/VP-Tree implementation? Thanks. Source: over 3 years ago

• Leaving Spotify for Freer Pastures

  Is your music recommendation system open source? Would be down to check it out and learn a thing or two from it. On the topic of vector search, I'm fairly certain that Spotify still uses Annoy (https://github.com/spotify/annoy). Like Faiss, it's a great library but not quite a database, which would ideally have features like replication (https://milvus.io/docs/replica.md), caching, and access control, to name a few. - Source: Hacker News / almost 4 years ago

• [D] [R] Large-scale clustering

  To improve the running time you could try an approximate algorithm: https://github.com/spotify/annoy/. Source: almost 4 years ago

• Bitmap Indexes in Go: Search Speed

  Ducks, the story: I was using Python in-memory vector search engine called Annoy [1] to do semantic search on various kinds of data. It worked great for finding "similar" objects. Story A has similar text to story B, image A looks like image B, etc. But doing basic metadata lookups was surprisingly hard. How do I get all images matching some criteria (say, size range, or tags)? I'd have to serialize them all into... - Source: Hacker News / almost 4 years ago

• What do You Prefer?

  The actual data that is used by Spotify that is in fast storage is likely in a compressed feature vector format (see https://github.com/spotify/annoy) that makes no sense to humans. The process of getting the “raw” data likely isn’t optimized; and the business has no appetite in optimizing this process because no one has literally died from not getting their raw data in 10 seconds. Source: about 4 years ago

• [D] Any example for Novalty detection in RGB image Dataset with Pytorch?

  Oh, like spotify Annoy I am also like beginner level so I don't understand quickly but I am definitely getting there. Thanks. Source: about 4 years ago

• Create a Wine Recommender using NLP on AWS

  It'd definitely be a nice-to-have. Luckily it shouldn't be to hard to create a custom estimator using something like Spotify's Annoy library. I might try it out whenever I come back and revisit the project. Source: about 4 years ago

• Can anyone help me refine my model (music based CNN - artist recognition)?

  Add your examples to the index and build the trees in annoy. I feel like its straight forward. There you have to provide the dimension of the features which is the feature vector you get. In my case I am reusing (without fine-tuning) the effecientNetB3 without the last layer. Hence it results in feature vectors with 1536 dimensions. Https://github.com/spotify/annoy. Source: over 4 years ago

• Is doing knn on the output of a music artist classifier a good way to build a system that suggests new music?

  Approximate Nearest Neighbors is what Spotify uses for music recommendations: Https://github.com/spotify/annoy. Source: over 4 years ago