Ollama

Installation of Ollama engine takes about 6gb on my system. I have it running with Phrase Express. Tray icon keeps disappearing, but CLI not needed after the setup. A lot of people think it runs slowly.. myself included, but they say it has to do with many factors.

Since it doesn't have a GUI, you need to get used to the command-line interface (the commands are similar in style to Git, if you're familiar with it). There are no customization options when loading models, and it only supports models in the GGUF format. However, even GGUF models don't work out of the box—you'll need to manually adjust or convert them before use, due to its custom model structure.

Because it's CLI-based, it can even be compiled and run on a phone using Termux or similar tools. That said, it lacks a 'Chat with Documents' feature, meaning it doesn’t include built-in tools for embedding your own documents or performing RAG-like operations. These have to be set up manually. So overall, it’s not very user-friendly, but it’s a minimal and lightweight choice for running LLM/AI models.

Ollama is a great way to run open source AI models locally. The installation of models is very straightforward, and Ollama even has their own repository for the models so I don’t have to worry about getting them from a reputable source. Unfortunately, it’s not the easiest app for people who don’t like command-line-interfaces, but this does make it a really powerful tool. To get a graphical interface, I use Ollama-App. So far, it has been working pretty well even though it is in beta for desktop.