« Previous 1 2
Chatbots put to the scripting test
Beginner's Mistake
Local AI with OobaBooga
OobaBooga is an open source project that generates a simple web interface for LLMs. It runs on all common operating systems with a suitable Python interpreter. You can ask it anything because no data is transmitted to a provider on the Internet. If Miniconda, the leaner version of the popular Anaconda Python distribution, is not already available globally on your system, the simple one-click installer sets it up.
For the web user interface (UI) to answer questions in the style of ChatGTP, it also needs one or more LLMs, which must be quantized to work on regular PCs. Put simply, quantization compresses the model by reducing the accuracy of the vectors. In principle, two quantizations are suitable for use on the PC: GGML for models that work on the CPU, and GPTQ for GPUs (NVIDIA).
The most popular website for open source models is Hugging Face [5], which now has hundreds of freely usable LLMs of different quality for different areas of application, many of which are merges (i.e., models that combine several existing LLMs with different weightings to create a new model). In addition to LLMs with optimization for chat or text creation (Storytellers), "coder" models generate program code.
Hugging Face user "TheBloke" offers a large number of ready-made quantized models in his repository that work on standard PCs. However, their performance leaves much to be desired. To come to grips with the technology, you need a computer with a powerful NVIDIA GPU. The VRAM capacity is particularly important because a GPTQ model must fit completely into the GPU's memory. To be able to load a large LLM, your GPU must have 24GB of VRAM. That said, specialist coding models will get by with less because they can do without the general knowledge of the chat or storyteller LLMs.
For this article, I used the CodeUp-Llama-2-13B code generation model [6], which makes do with the 12GB of VRAM provided by the RTX 4090 in the lab computer. Again, I asked the coding model to generate the Bash script in Listing 1. Unlike ChatGPT, the self-hosted LLM initially provided a working approach by suggesting rpm -qa
and rpm --queryformat
as tools to determine the versions; however, it then messed up the Bash syntax and, later in the code, started using variables that it did not declare previously. Code generation for Ansible suffered a similar fate. Again, the first lines of the proposed playbook made perfect sense, but as the output progressed, the LLM produced confused output and mixed Ansible syntax with Bash syntax.
The reasons for degenerating code quality are probably the high degree of quantization and the limited token buffer, which is the LLM's memory. Put simply, if the buffer overflows during the LLM's response, the model forgets what the user asked in the first place.
I looked at other open models, including models that have not been optimized for code generation. All models started off on the right path, but then all failed in terms of syntax and output values. Most of the proposed | grep
and | awk
filters do not work. All of the Ansible code generated by generic models was useless.
Self-hosted open source LLMs on your own graphics card provide some surprisingly good answers, at least as long as you chat with them casually and do not ask for working program code. The quality of the quantized models and the limited token buffers simply are not up to the task right now.
Conclusions
As always, you'll find far more hype than reality when it comes to AI-generated scripts. The current tools cannot reduce your scripting workload. At best, they can provide some assistance. In the future, the models will certainly grow and become more powerful, but it is doubtful whether they will also become better because of the degenerating training data. What will therefore be of particular interest to application developers is helper LLMs with limited, but specialized, databases.
Infos
- Gamers fool AI news sites: https://www.polygon.com/23803152/world-of-warcraft-glorbo-ai-news-site-reddit
- ChatGPT: https://chat.openai.com/auth/login
- IBM watsonx Code Assistant for Ansible: https://www.ibm.com/products/watsonx-code-assistant
- OobaBooga: https://github.com/oobabooga/text-generation-webui
- Hugging Face: https://huggingface.co
- CodeUp-Llama-2-13B: https://huggingface.co/TheBloke/CodeUp-Llama-2-13B-Chat-HF-GGML
« Previous 1 2
Buy this article as PDF
(incl. VAT)
Buy ADMIN Magazine
Subscribe to our ADMIN Newsletters
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Most Popular
Support Our Work
ADMIN content is made possible with support from readers like you. Please consider contributing when you've found an article to be beneficial.
![Learn More”>
</a>
<hr>
</div>
</div>
<div class=](https://www.admin-magazine.com/var/ezflow_site/storage/images/media/images/learn-more/211417-1-eng-US/Learn-More.png)