Conversational AI agents (eg: chatGPT, Gemini, etc.,) are the buzz over the Internet these days. But if you are someone like me who was hesitant to try out any of these due to privacy concerns or an aversion to such tech (may be a luddite-like mentality), then I want you to reconsider and try out something that's open, private and completely runs offline.
Yesterday, I attended Dev Days (it's the name for their meetups) by FSMK. They have invited Ranjith Raj from Swecha for an introductory session on LLM prompting. Thanks to Ranjith, who helped me overcome some of my above mentioned hesitations, because he introduced to us how we can install FOSS apps in our personal devices (Laptops & Smartphones) and download LLM/SLM models that are released as open source models (we can discuss in another thread, what does open and freedom means in AI model context, as it is not straightforward in AI/ML context) and use them totally private, offline and without restrictions.
This means, I don't have to worry about a model that's run by someone outside my control, instead it runs only in my device and it ends there. While LLMs are called Large Language Models, SLMs are called Small Language Models. As you can understand from it's name, SLMs are smaller in scale and narrowed to particular scope. Instead of trying to do many things like LLMs, SLMs try to excel in doing one thing is what I have understood.
Terminologies aside, let's get practical. Ranjith introduced two FOSS apps to us.
- Ollama- For PCs.
- PocketPal - For Android & iOS
Both these apps are capable of downloading one of the open models of our choice, loads them and provides a nice frontend (chat window) to prompt or chat with the model as an agent. Various run-time parameters like Temperature (i.e how factual or imaginative it can go), can be configured once the model has been loaded.
While PocketPal is MIT licensed, I found another app called ChatterUI which is AGPL licensed. ChatterUI seems much more usable, polished and feature-rich compared to PocketPal and I personally prefer AGPL/GPL licensed products compared to any of the permissive licensed products.
PocketPal & ChatterUI are my first interaction with real-world AI models. I got excited as I downloaded couple of models (llama-3.2-1b and gemma2-2b) and started using them in no time. I was also exposed to it's limitations in Day-1 itself, like though the responses are relevant, they are not accurate and factual all the times. The temperature parameter value of-course led to it being creative/imaginative/exaggeration. A model is only as good as the accuracy of the dataset used to train it. There could be factual errors, systemic social biases, low sample of data in the dataset which could lead to poor and even dangerous conclusions.
Though PocketPal and ChatterUI being FOSS apps, I found that they are not yet available in F-Droid. I wanted to make them available in F-Droid, I immediately checked if someone has already asked for or attempted to do so and then I found this and this.
ChatterUI maintainer wants to wait till it matures to version 1.0.0 before publishing to any store and PocketPal maintainer is ok to publish to F-Droid since the app is FOSS and it has already been published to other stores. Reading through the discussion, I found that they had no prior experience publishing to F-Droid and then I pitched in, since I have some experience on how publishing to F-Droid works. I have already started on it and submitted one Pull Request.
I have also asked it to be included in the IzzyOnDroid repo. Let's hope PocketPal atleast would be available through F-Droid / IzzyOnDroid repo sooner. And then there is also Obtanium way of installing them directly from the source.
Finally, Ranjith mentioned the efforts of Swecha to build a Telugu LLM which has potential to democratize open LLMs further. In the same lines, we as a movement should also focus on creating or strengthening any efforts towards Tamil SLM/LLM.
I hope you enjoyed reading it, if you have tried any of them before or if you know another FOSS app to utilize these models offline and private, do share them here. Thanks🫰.
