- Conversational simulation is an emerging idea building on top of model-graded eval” - AI Startup Founder Things to consider when comparing options: “Types of metrics supported (only NLP metrics, model-graded evals, or both), level of customizability; supports component eval (i.e. Single prompts) or pipeline evals (i.e. Testing the entire pipeline, all the way from retrieval to post-processing)” “+method of... - Source: Hacker News / 9 months ago

Humanloop (YC S20) | London (or remote) | https://humanloop.com We're looking for exceptional engineers that can work at varying levels of the stack (frontend, backend, infra), who are customer obsessed and thoughtful about product (we think you have to be -- our customers are "living in the future" and we're building what's needed). Our stack is primarily Typescript, Python, GPT-3. Please apply at... - Source: Hacker News / over 1 year ago

https://humanloop.com/ Find the prompts users love and fine-tune custom models for higher performance at lower cost. - Source: Hacker News / over 1 year ago

PrivateGPT is a nice tool for this. It's not exactly what you're asking for, but it gets part of the way there. https://github.com/zylon-ai/private-gpt. - Source: Hacker News / 2 months ago