Show HN: Improving RAG with Chess Elo Scores? (YC W25)

1

HackMD

We tried a bradley-terry loss function, as calculated with https://hackmd.io/@-Gjw1zWMSH6lMPRlziQFEw/SJ8sRl1Zge We found that MSE after elo-adjustment worked equally well. And, MSE lets you shuffle (q, d) across the dataset which has good statistical properties (Versus contrastive, which makes you sample the same query many times within a single minibatch) In this case "InfoNCE" isn't applicable because the reranker's output is a scalar, not a vector. So that's why we checked both bradley-terry and MSE.

#Task Management #Project Management #Productivity 71 social mentions

2

Medium

One trouble I could see with your approach is that you treat the information "Doc at pos i" beats "Doc at pos j" independently from I and j. Intuitively, it is not as critical when a bad doc is at rank 9 instead of rank 10; compared to bad doc landing at rank 1 instead of rank 10. LambdaMART's approach seems better in that respect. https://medium.com/@nikhilbd/pointwise-vs-pairwise-vs-listwise-learning-to-rank-80a8fe8fadfd.

#Blogging #Blogging Platform #CMS 2754 social mentions