I focused on supporting Ollama, GPT4All, and LMStudio, all of which run smoothly on a Mac computer. Many of these tools are user-friendly wrappers around Llama.cpp, allowing easy model downloads and providing a REST interface to query the available models. Last week, I also added "👋🏼 Jan" support because HuggingFace has endorsed this provider out-of-the-box. - Source: dev.to / 4 days ago

Ollama is an open-source tool for using LLMs like Llama 3 on your computer. Thanks to new research, these models don't need a lot of VRAM, computing power, or storage. They are designed to work well on laptops. - Source: dev.to / 15 days ago

One thing I've noticed is that gemma is much less verbose by default. [0] https://github.com/ollama/ollama. - Source: Hacker News / about 2 months ago

A few weeks ago, I started using Ollama to run language models (LLM), and I've been really enjoying it a lot. After getting the hang of it, I thought it was about time to try it out on one of our real-world cases (I'll share more about this later). - Source: dev.to / 2 months ago

GitHub - ollama/ollama: Get up and running with Llama 2, Mistral, Gemma, and other large language models. - Source: dev.to / 3 months ago

The $0.47 bill seems reasonable for an experiment, but imagine someone doing a task of this complexity as a daily job - let's say 100x times, or a little more than 4 hours - the bill would be $47/day. It feels like there's still an opportunity for a cheaper solution. Have you or someone else experimented with e.g. https://localai.io/ ? - Source: Hacker News / 6 months ago

We're using LocalAI https://localai.io/ for inference on the back end amongst other tools. Source: 8 months ago

We recently added support to use open-source models by integrating with LocalAI (https://localai.io). With LocalAI, we can run open-source models like Llama2 and seamlessly build LLM applications using LLMStack and run everything on-prem. Source: 9 months ago

- Ability to use local open-source LLMs like Llama2 etc using LocalAI (https://localai.io) Background: We started as a closed source prompt management platform early this year (trypromptly.com) and eventually landed as an Enterprise LLM apps platform. In the process, we learned how hard it is to sell a horizontal SaaS platform. That combined with the concerns around data privacy (both with us hosting data as well... - Source: Hacker News / 9 months ago

LocalAI https://localai.io/ and LMStudio https://lmstudio.ai/ both have fairly complete OpenAI compatibility layers. llama-cpp-python has a FastAPI server as well: https://github.com/abetlen/llama-cpp-python/blob/main/llama_... (as of this moment it hasn't merged GGUF update yet though). - Source: Hacker News / 9 months ago