Annoy Reviews and details

Social recommendations and mentions

• Do we think about vector dbs wrong?

  The focus on the top 10 in vector search is a product of wanting to prove value over keyword search. Keyword search is going to miss some conceptual matches. You can try to work around that with tokenization and complex queries with all variations but it's not easy. Vector search isn't all that new a concept. For example, the annoy library (https://github.com/spotify/annoy), an open source embeddings database. - Source: Hacker News / 8 months ago

• Vector Databases 101

  If you want to go larger you could still use some simple setup in conjunction with faiss, annoy or hnsw. Source: 10 months ago

• Calculating document similarity in a special domain

  I then use annoy to compare them. Annoy can use different measures for distance, like cosine, euclidean and more. Source: 11 months ago

• Can Parquet file format index string columns?

  Yes you can do this for equality predicates if your row groups are sorted . This blog post (that I didn't write) might add more color. You can't do this for any kind of text searching. If you need to do this with file based storage I'd recommend using a vector based text search and utilize a ANN index library like Annoy. Source: 11 months ago

• [D]: Best nearest neighbour search for high dimensions

  If you need large scale (1000+ dimension, millions+ source points, >1000 queries per second) and accept imperfect results / approximate nearest neighbors, then other people have already mentioned some of the best libraries (FAISS, Annoy). Source: 12 months ago

• [R] Unlimiformer: Long-Range Transformers with Unlimited Length Input

  Would be possible to further speed up the process with using something like ANNOY? https://github.com/spotify/annoy. Source: 12 months ago

• Faiss: A library for efficient similarity search

  I like Faiss but I tried Spotify's annoy[1] for a recent project and was pretty impressed. Since lots of people don't seem to understand how useful these embedding libraries are here's an example. I built a thing that indexes bouldering and climbing competition videos, then builds an embedding of the climber's body position per frame. I then can automatically match different climbers on the same problem. It works... - Source: Hacker News / about 1 year ago

• How to find "k" nearest embeddings in a space with a very large number of N embeddings (efficiently)?

  If you just want quick in memory search then pynndescent is a decent option: it's easy to install, and easy to get running. Another good option is Annoy; it's just as easy to install and get running with python, but it is a little less performant if you want to do a lot of queries, or get a knn-graph quickly. Source: about 1 year ago

• [Discussion] NLP for products matching

  Probably I won't be bale to explain better than it's stated on annoy page: https://github.com/spotify/annoy But the bottom line is speed. Instead of computing similarities of embeddings one by one you do it via index that works way faster. Source: over 1 year ago

• Do i really need a vector database

  Perhaps you can store your embeddings anywhere (sql or even a file) and use Approximate Nearest Neighbors like https://github.com/spotify/annoy for comparison? Source: over 1 year ago

• Comparing millions of image hashes in rust

  Hi, I have a huge list of hashes of images, that I have to compare and find matching items and delete duplicates. Is there something similar to spotify/annoy in Rust or BK-Tree/VP-Tree implementation? Thanks. Source: over 1 year ago

• Leaving Spotify for Freer Pastures

  Is your music recommendation system open source? Would be down to check it out and learn a thing or two from it. On the topic of vector search, I'm fairly certain that Spotify still uses Annoy (https://github.com/spotify/annoy). Like Faiss, it's a great library but not quite a database, which would ideally have features like replication (https://milvus.io/docs/replica.md), caching, and access control, to name a few. - Source: Hacker News / over 1 year ago

• [D] [R] Large-scale clustering

  To improve the running time you could try an approximate algorithm: https://github.com/spotify/annoy/. Source: over 1 year ago

• Bitmap Indexes in Go: Search Speed

  Ducks, the story: I was using Python in-memory vector search engine called Annoy [1] to do semantic search on various kinds of data. It worked great for finding "similar" objects. Story A has similar text to story B, image A looks like image B, etc. But doing basic metadata lookups was surprisingly hard. How do I get all images matching some criteria (say, size range, or tags)? I'd have to serialize them all into... - Source: Hacker News / over 1 year ago

• What do You Prefer?

  The actual data that is used by Spotify that is in fast storage is likely in a compressed feature vector format (see https://github.com/spotify/annoy) that makes no sense to humans. The process of getting the “raw” data likely isn’t optimized; and the business has no appetite in optimizing this process because no one has literally died from not getting their raw data in 10 seconds. Source: over 1 year ago

• [D] Any example for Novalty detection in RGB image Dataset with Pytorch?

  Oh, like spotify Annoy I am also like beginner level so I don't understand quickly but I am definitely getting there. Thanks. Source: over 1 year ago

• Create a Wine Recommender using NLP on AWS

  It'd definitely be a nice-to-have. Luckily it shouldn't be to hard to create a custom estimator using something like Spotify's Annoy library. I might try it out whenever I come back and revisit the project. Source: almost 2 years ago

• Can anyone help me refine my model (music based CNN - artist recognition)?

  Add your examples to the index and build the trees in annoy. I feel like its straight forward. There you have to provide the dimension of the features which is the feature vector you get. In my case I am reusing (without fine-tuning) the effecientNetB3 without the last layer. Hence it results in feature vectors with 1536 dimensions. Https://github.com/spotify/annoy. Source: almost 2 years ago

• Is doing knn on the output of a music artist classifier a good way to build a system that suggests new music?

  Approximate Nearest Neighbors is what Spotify uses for music recommendations: Https://github.com/spotify/annoy. Source: about 2 years ago

• Anatomy of a txtai index

  Embeddings - The embeddings index file. This is an Approximate Nearest Neighbor (ANN) index with either Faiss (default), Hnswlib or Annoy, depending on the settings. - Source: dev.to / about 2 years ago

• Should we begin Linear Algebra with Matrices, or start with Vector Spaces?

  Take for example Spotifies implementation of ANN https://github.com/spotify/annoy. Source: over 2 years ago