Nextcloud Local Large Language Model

An on-premises text processing backend for the Nextcloud Assistant or any app that uses the text processing functionality.

This app uses llama.cpp under the hood and is thus compatible with any open-source model in GGUF format.

Installation

See the Nextcloud admin documentation for installation instructions and system requirements.

Development installation

0. (Optional) Install Nvidia drivers and CUDA on your host system.

1. Create and activate a Python virtual environment:

   python3 -m venv ./venv && . ./venv/bin/activate
   

2. Install dependencies:

   poetry install
   

3. (Optional) Enable hardware acceleration if your system supports it (check the llama.cpp documentation for your accelerator).

4. (Optional) Download any additional desired models into the models directory:

   Examples:

   wget -nc -P models https://download.nextcloud.com/server/apps/llm/llama-2-7b-chat-ggml/llama-2-7b-chat.Q4_K_M.gguf
   wget -nc -P models https://download.nextcloud.com/server/apps/llm/leo-hessianai-13B-chat-bilingual-GGUF/leo-hessianai-13b-chat-bilingual.Q4_K_M.gguf
   wget -nc -P models https://huggingface.co/Nextcloud-AI/llm_neuralbeagle_14_7b_gguf/resolve/main/neuralbeagle14-7b.Q4_K_M.gguf
   

5. Run the app:

   PYTHONUNBUFFERED=1 APP_HOST=0.0.0.0 APP_ID=llm2 APP_PORT=9081 APP_SECRET=12345 APP_VERSION=<APP_VERSION> NEXTCLOUD_URL=http://nextcloud.local python3 lib/main.py
   

6. Register the app with the manual_install AppAPI deploy daemon (see AppAPI admin settings in Nextcloud).

   With the Nextcloud Docker dev environment, you can just run:

   make register
   

   Example if Nextcloud is installed on bare metal instead:

   sudo -u www-data php /var/www/nextcloud/occ app_api:app:unregister llm2 --force
   sudo -u www-data php /var/www/nextcloud/occ app_api:app:register llm2 manual_install --json-info "{\"id\":\"llm2\",\"name\":\"Local large language model\",\"daemon_config_name\":\"manual_install\",\"version\":\"<APP_VERSION>\",\"secret\":\"12345\",\"port\":9081}" --wait-finish
   

Development installation using Docker

Note

Currently, running the Docker image requires that your host system have CUDA/NVIDIA drivers installed and is equipped with a GPU capable of performing the required tasks.

0. Install Nvidia drivers and CUDA on your host system and install NVIDIA Docker toolkit.

1. Build the Docker image:

   docker build --no-cache -f Dockerfile -t llm2:latest .
   

2. Run the Docker image:

   sudo docker run -ti -v /var/run/docker.sock:/var/run/docker.sock -e APP_ID=llm2 -e APP_HOST=0.0.0.0 -e APP_PORT=9081 -e APP_SECRET=12345 -e APP_VERSION=<APP_VERSION> -e NEXTCLOUD_URL='<YOUR_NEXTCLOUD_URL_REACHABLE_FROM_INSIDE_DOCKER>' -e CUDA_VISIBLE_DEVICES=0 -p 9081:9081 --gpus all llm2:latest
   

3. Register the app.

   With the Nextcloud Docker dev environment, you can just run:

   make register
   

   Example if Nextcloud is installed on bare metal instead:

   sudo -u www-data php /var/www/nextcloud/occ app_api:app:unregister llm2 --force
   sudo -u www-data php /var/www/nextcloud/occ app_api:app:register llm2 manual_install --json-info "{\"id\":\"llm2\",\"name\":\"Local large language model\",\"daemon_config_name\":\"manual_install\",\"version\":\"<APP_VERSION>\",\"secret\":\"12345\",\"port\":9081}" --wait-finish