LLM Server Setup

Codumentor works with any OpenAI-compatible LLM provider. You can use a cloud API or run a self-hosted LLM server. This guide covers both options.

Cloud Providers

The simplest approach is to use a cloud LLM API. Set the model name, base URL, and API key in your codumentor.yaml:

Provider	Base URL	Notes
OpenAI	`https://api.openai.com/v1` (default)	Set `OPENAI_API_KEY`
Google Gemini	`https://generativelanguage.googleapis.com/v1beta/openai/`	OpenAI-compatible endpoint
OpenRouter	`https://openrouter.ai/api/v1`	Aggregates multiple providers

Example: Google Gemini

models:
  agent: "gemini-2.0-flash"
  agent_base_url: "https://generativelanguage.googleapis.com/v1beta/openai/"
  agent_api_key: "${GOOGLE_API_KEY}"
  embedding: "text-embedding-004"
  embedding_base_url: "https://generativelanguage.googleapis.com/v1beta/openai/"
  embedding_api_key: "${GOOGLE_API_KEY}"

Example: OpenAI

models:
  agent: "gpt-4o"
  agent_api_key: "${OPENAI_API_KEY}"
  embedding: "text-embedding-3-small"
  embedding_api_key: "${OPENAI_API_KEY}"

Example: OpenRouter

models:
  agent: "anthropic/claude-sonnet-4"
  agent_base_url: "https://openrouter.ai/api/v1"
  agent_api_key: "${OPENROUTER_API_KEY}"
  embedding: "text-embedding-004"
  embedding_base_url: "https://generativelanguage.googleapis.com/v1beta/openai/"
  embedding_api_key: "${GOOGLE_API_KEY}"

Self-Hosted LLM Server

For organizations that require on-premises inference or want to reduce API costs at scale, Codumentor includes a self-hosted LLM server stack based on vLLM and LiteLLM.

Supported Models

Model	Config ID	Description
Qwen3-235B-A22B-AWQ	`qwen3`	Alibaba's efficient reasoning model with tool calling support
GLM-4.7 FP8	`glm47`	ZhipuAI's reasoning model with advanced tool calling

Both models require NVIDIA GPUs with CUDA 12.8+ and use FP8/AWQ quantization for efficient inference.

Installation

Automated Install (Recommended)

cd llm-server
sudo bash install-all.sh

The installer sets up vLLM with CUDA support, the LiteLLM proxy, and an optional systemd service. It prompts for model selection or accepts --model qwen3 / --model glm47 for non-interactive use.

Manual Install

Install components individually if you need more control:

# 1. System setup and vLLM
sudo bash install_and_prepare_vllm.sh --model qwen3

# 2. LiteLLM proxy
bash install-litellm.sh

# 3. Optional: systemd service
sudo bash install-llm-service.sh

Running the Server

Start All Services

cd llm-server
bash start.sh

This creates a tmux session with four panes: vLLM server, embedding service, LiteLLM proxy, and GPU monitoring.

Start Individual Services

bash start-llm.sh       # vLLM server only
bash start-embedder.sh  # Embedding service only
bash start-litellm.sh   # LiteLLM proxy only

Stop Services

Detach from the tmux session (Ctrl-b d) or kill it:

tmux kill-session -t llm_session

Using Systemd

sudo systemctl start llm-server
sudo systemctl stop llm-server
sudo systemctl status llm-server

Switching Models

Switch between installed models without reinstalling:

# Interactive selection
bash switch-model.sh

# Non-interactive
bash switch-model.sh --model glm47

The script updates the configuration, detects running services, and offers to restart them.

Port Defaults

Service	Port	Description
vLLM	8000	Raw vLLM OpenAI-compatible API
LiteLLM	1248	LiteLLM proxy (routes to vLLM and embedding)

Connecting Codumentor to a Local Server

Point your configuration at the LiteLLM proxy:

models:
  agent: "qwen3"
  agent_base_url: "http://localhost:1248/v1"
  embedding: "text-embedding-004"
  embedding_base_url: "http://localhost:1248/v1"

SSH Tunnel for Remote LLM Server

If the LLM server runs on a remote machine (e.g., a GPU instance in the cloud), create an SSH tunnel to forward the LiteLLM port to your local machine:

ssh root@<server-ip> -L 1248:localhost:1248

With the tunnel active, configure Codumentor to use http://localhost:1248/v1 as the base URL, just as with a local server.

For persistent tunnels, add to your SSH config:

Host llm-server
    HostName <server-ip>
    User root
    LocalForward 1248 localhost:1248

LLM Server Plugin

The LLM Server plugin automates server lifecycle management from the Codumentor web UI. When it detects that the LLM server is unreachable, it offers to start a GPU instance and monitors its health throughout the session. Key features:

Automatic detection of connection errors
Interactive UI card for starting/stopping the server
Health monitoring with periodic checks against /v1/models
Auto-shutdown after configurable inactivity period to control GPU costs
Multi-user synchronization to prevent duplicate instances

For full details on the plugin and its configuration, see the LLM Server Plugin documentation.