Local AI with OobaBooga

OobaBooga is an open source project that generates a simple web interface for LLMs. It runs on all common operating systems with a suitable Python interpreter. You can ask it anything because no data is transmitted to a provider on the Internet. If Miniconda, the leaner version of the popular Anaconda Python distribution, is not already available globally on your system, the simple one-click installer sets it up.

For the web user interface (UI) to answer questions in the style of ChatGTP, it also needs one or more LLMs, which must be quantized to work on regular PCs. Put simply, quantization compresses the model by reducing the accuracy of the vectors. In principle, two quantizations are suitable for use on the PC: GGML for models that work on the CPU, and GPTQ for GPUs (NVIDIA).

The most popular website for open source models is Hugging Face [5], which now has hundreds of freely usable LLMs of different quality for different areas of application, many of which are merges (i.e., models that combine several existing LLMs with different weightings to create a new model). In addition to LLMs with optimization for chat or text creation (Storytellers), "coder" models generate program code.

Hugging Face user "TheBloke" offers a large number of ready-made quantized models in his repository that work on standard PCs. However, their performance leaves much to be desired. To come to grips with the technology, you need a computer with a powerful NVIDIA GPU. The VRAM capacity is particularly important because a GPTQ model must fit completely into the GPU's memory. To be able to load a large LLM, your GPU must have 24GB of VRAM. That said, specialist coding models will get by with less because they can do without the general knowledge of the chat or storyteller LLMs.

For this article, I used the CodeUp-Llama-2-13B code generation model [6], which makes do with the 12GB of VRAM provided by the RTX 4090 in the lab computer. Again, I asked the coding model to generate the Bash script in Listing 1. Unlike ChatGPT, the self-hosted LLM initially provided a working approach by suggesting rpm -qa and rpm --queryformat as tools to determine the versions; however, it then messed up the Bash syntax and, later in the code, started using variables that it did not declare previously. Code generation for Ansible suffered a similar fate. Again, the first lines of the proposed playbook made perfect sense, but as the output progressed, the LLM produced confused output and mixed Ansible syntax with Bash syntax.

The reasons for degenerating code quality are probably the high degree of quantization and the limited token buffer, which is the LLM's memory. Put simply, if the buffer overflows during the LLM's response, the model forgets what the user asked in the first place.

I looked at other open models, including models that have not been optimized for code generation. All models started off on the right path, but then all failed in terms of syntax and output values. Most of the proposed | grep and | awk filters do not work. All of the Ansible code generated by generic models was useless.

Self-hosted open source LLMs on your own graphics card provide some surprisingly good answers, at least as long as you chat with them casually and do not ask for working program code. The quality of the quantized models and the limited token buffers simply are not up to the task right now.

Conclusions

As always, you'll find far more hype than reality when it comes to AI-generated scripts. The current tools cannot reduce your scripting workload. At best, they can provide some assistance. In the future, the models will certainly grow and become more powerful, but it is doubtful whether they will also become better because of the degenerating training data. What will therefore be of particular interest to application developers is helper LLMs with limited, but specialized, databases.

The Author

Andreas Stolzenberger worked as an IT magazine editor for 17 years. He was the deputy editor in chief of the German Network Computing magazine from 2000 to 2010. After that, he worked as a solution engineer at Dell and VMware. In 2012 Andreas moved to Red Hat. There, he currently works as principal solution architect in the Technical Partner Development department.