LLM Server Manager Plugin

The LLM Server Manager Plugin intercepts LLM connection errors and offers to start the LLM server via an interactive UI card. It provides multi-user synchronization, health monitoring, and automatic shutdown on inactivity.

Overview

The plugin:

Hooks into onLLMError to detect connection failures to the local LLM server
Checks for existing instances before starting new ones
Displays a UI card prompting users to start the server
Tracks server startup progress with real-time updates
Monitors server health with periodic checks
Automatically shuts down the server after configurable inactivity period
Uses a singleton state manager to prevent concurrent start requests
Broadcasts progress to all connected conversations

Backends

The plugin supports two backends for provisioning LLM servers:

Backend	Description	Use Case
DataCrunch (legacy)	Direct DataCrunch API integration with SSH tunneling	Single-provider setup, existing DataCrunch users
dstack	Multi-cloud orchestration via dstack	Multi-provider flexibility, automatic cost optimization

What is dstack?

dstack is an open-source GPU orchestration platform that:

Provisions GPUs across multiple cloud providers (Verda, Lambda Labs, RunPod, AWS, GCP, etc.)
Automatically selects the cheapest available option that meets your resource constraints
Exposes HTTPS endpoints via its gateway (no SSH tunnel needed)
Handles health checks and auto-scaling natively
Provides a shared state server so multiple Codumentor instances coordinate without conflicts

When using dstack, you run a dstack server that manages credentials for your cloud providers. The plugin connects to this server to allocate/release GPU instances.

Backend Comparison

Feature	DataCrunch	dstack
Providers	DataCrunch only	Multiple (Verda, Lambda, RunPod, AWS, GCP, etc.)
Connection	SSH tunnel to raw IP	Direct HTTPS endpoint
Cost optimization	Manual profile selection	Automatic cheapest-available selection
Multi-instance coordination	Via DataCrunch API	Via dstack server (PostgreSQL state)
Setup complexity	Lower (just API keys)	Higher (dstack server required)

Configuration

DataCrunch Backend (Legacy)

plugins:
  - module: codumentor.plugins.llm_server
    class: LLMServerPlugin
    priority: 5
    args:
      enabled: true
      auto_prompt: true
      default_profile: "qwen3-fin02"

      # Backend selection (default: datacrunch)
      backend: "datacrunch"

      # Health checks
      health_check_interval_seconds: 60
      health_check_timeout_seconds: 10
      health_check_max_failures: 3

      # Auto-shutdown
      auto_shutdown_enabled: true
      inactivity_timeout_minutes: 30

      # Instance lifecycle
      check_existing_instances: true

dstack Backend

plugins:
  - module: codumentor.plugins.llm_server
    class: LLMServerPlugin
    priority: 5
    args:
      enabled: true
      auto_prompt: true
      default_profile: "qwen3-72b"

      # Use dstack backend
      backend: "dstack"
      dstack_url: "http://localhost:3000"
      dstack_token: "${DSTACK_TOKEN}"
      dstack_profiles_path: "profiles.yaml"
      dstack_project: "main"

      # Health checks
      health_check_interval_seconds: 60
      health_check_timeout_seconds: 10
      health_check_max_failures: 3

      # Auto-shutdown
      auto_shutdown_enabled: true
      inactivity_timeout_minutes: 30

      # Instance lifecycle
      check_existing_instances: true

Configuration Parameters

Common Parameters

Parameter	Type	Default	Description
`enabled`	boolean	`true`	Whether the plugin is enabled
`auto_prompt`	boolean	`true`	Automatically show UI card when connection errors are detected
`default_profile`	string	`null`	Default server profile to use if none specified
`priority`	integer	`5`	Hook priority (lower = earlier execution)
`backend`	string	`"datacrunch"`	Backend type: `"datacrunch"` or `"dstack"`
`health_check_interval_seconds`	integer	`60`	Seconds between health checks
`health_check_timeout_seconds`	integer	`10`	Timeout for health check HTTP requests
`health_check_max_failures`	integer	`3`	Mark server as ERROR after N consecutive failures
`auto_shutdown_enabled`	boolean	`true`	Enable auto-shutdown on user inactivity
`inactivity_timeout_minutes`	integer	`30`	Minutes of user inactivity before auto-shutdown
`check_existing_instances`	boolean	`true`	Check for existing instances before starting

dstack-Specific Parameters

Parameter	Type	Default	Description
`dstack_url`	string	`"http://localhost:3000"`	URL of the dstack server
`dstack_token`	string	`null`	Authentication token for dstack (if server requires auth)
`dstack_profiles_path`	string	`null`	Path to YAML file with profile definitions
`dstack_project`	string	`"main"`	dstack project name

How It Works

Error Detection

The plugin detects LLM server connection errors by matching error messages against known patterns:

CONNECTION_ERROR_PATTERNS = [
    "connection refused",
    "econnrefused",
    "localhost:1248",
    "127.0.0.1:1248",
    "apiconnectionerror",
    "failed to connect",
    "connection error",
    "cannot connect",
]

Existing Instance Detection

Before starting a new instance, the plugin checks for existing servers:

DataCrunch backend:

Queries DataCrunch API for instances with status "running" or "starting"
Running instance found: Syncs local state, extracts IP address, broadcasts "started" event
Starting instance found: Waits for it instead of creating duplicate
No instances: Proceeds with new instance creation

dstack backend:

Queries dstack via FleetManager.list_servers()
Running server found: Syncs local state, uses existing endpoint URL
No running servers: Proceeds with new server allocation

This prevents duplicate GPU instances and associated costs.

Health Monitoring

The plugin runs a background scheduler that periodically checks server health:

Queries the /v1/models endpoint (more reliable than /health for model readiness)
Tracks consecutive failures in health_check_failures
After health_check_max_failures failures, marks server as ERROR
Broadcasts error event to all connected UIs

Health check targets by backend:

DataCrunch: Checks http://127.0.0.1:1248/v1/models (via SSH tunnel)
dstack: Checks the full endpoint URL directly (e.g., https://gateway.dstack.ai/v1/models)

This detects scenarios where:

The server is running but the model hasn't finished loading (~6 minutes)
The model has crashed or become unresponsive

Inactivity Detection

The plugin hooks into onInputReceived to track user activity:

Records timestamp of each user message (not LLM/agent activity)
Background task checks: (now - last_user_input) >= inactivity_timeout_minutes
If threshold exceeded, broadcasts auto-shutdown event and stops the server

This ensures expensive GPU instances are stopped when no one is actively using them.

UI Card States

The plugin displays an interactive card with different states:

Prompt: "Server not reachable" with Start/Dismiss buttons
Starting: Spinner with progress message and percentage
Success: Green checkmark with server IP address
Error: Red X with error message and Retry button
Auto-shutdown: Notification before shutdown due to inactivity

Multi-User Synchronization

The plugin uses a class-level singleton state manager to coordinate across multiple conversations:

Prevents duplicate starts: Only one server instance can be started at a time
Shared progress: All connected conversations see the same progress updates
Race-safe: Uses asyncio.Lock to handle concurrent start requests

Server States

UNKNOWN ──(startup check)──> STOPPED or RUNNING (found existing)
STOPPED ──(start request)──> STARTING
STARTING ──(health check pass)──> RUNNING
STARTING ──(timeout/error)──> ERROR
RUNNING ──(health failures)──> ERROR
RUNNING ──(inactivity timeout)──> STOPPING ──> STOPPED
RUNNING ──(stop request)──> STOPPING ──> STOPPED

State	Description
`UNKNOWN`	Initial state, server status not checked
`STOPPED`	Server is not running
`STARTING`	Server startup in progress
`RUNNING`	Server is running and accessible
`STOPPING`	Server shutdown in progress
`ERROR`	Server startup or health check failed

State Fields

The ServerState object tracks different fields depending on the backend:

Field	DataCrunch	dstack	Description
`ip_address`	✓	-	Server IP (used with SSH tunnel)
`instance_id`	✓	-	DataCrunch instance ID
`endpoint`	-	✓	Full HTTPS URL (e.g., `https://gateway.dstack.ai/...`)
`server_id`	-	✓	dstack run name
`profile`	✓	✓	Profile used for allocation
`started_at`	✓	✓	When server reached RUNNING state
`is_model_ready`	✓	✓	Whether `/v1/models` returns loaded model

REST API Endpoints

The plugin exposes REST endpoints under /ui/v0/plugins/llm_server/:

Endpoint	Method	Description
`/start`	POST	Start the LLM server
`/stop`	POST	Stop/hibernate the server
`/status`	GET	Get current server status
`/profiles`	GET	List available server profiles

Start Server

curl -X POST http://localhost:2638/ui/v0/plugins/llm_server/start \
  -H "Content-Type: application/json" \
  -d '{"profile": "qwen3-32b"}'

Response:

{
  "status": "starting",
  "profile": "qwen3-32b",
  "message": "Server startup initiated"
}

Get Status

curl http://localhost:2638/ui/v0/plugins/llm_server/status

Response (DataCrunch backend):

{
  "status": "running",
  "ip_address": "192.168.1.100",
  "profile": "qwen3-32b"
}

Response (dstack backend):

{
  "status": "running",
  "endpoint": "https://gateway.dstack.ai/services/llm-qwen3-72b-a1b2c3d4/",
  "server_id": "llm-qwen3-72b-a1b2c3d4",
  "profile": "qwen3-72b"
}

List Profiles

curl http://localhost:2638/ui/v0/plugins/llm_server/profiles

Response:

{
  "profiles": {
    "qwen3-32b": "Qwen 3 32B with FP8 quantization",
    "llama-70b": "Llama 3 70B instruct"
  }
}

UI Event Types

The plugin emits the following event types:

Event Type	Description
`plugin.llm_server.connection_failed`	Connection error detected, prompt to start
`plugin.llm_server.starting`	Server startup initiated
`plugin.llm_server.progress`	Progress update during startup
`plugin.llm_server.started`	Server started successfully
`plugin.llm_server.stopped`	Server stopped
`plugin.llm_server.error`	Error occurred during operation
`plugin.llm_server.auto_shutdown`	Auto-shutdown initiated due to inactivity

Prerequisites

DataCrunch Backend

API Credentials

export DATACRUNCH_CLIENT_ID="your_client_id"
export DATACRUNCH_CLIENT_SECRET="your_client_secret"

Server Profiles

Server profiles are configured in scripts/llm_server_manager/profiles.json:

{
  "qwen3-32b": {
    "description": "Qwen 3 32B with FP8 quantization",
    "instance_type": "1V100.6V",
    "location": "FIN-02"
  }
}

dstack Backend

1. dstack Server

You need a running dstack server. Use the automated installer:

/usr/local/lib/codumentor/scripts/install-dstack-server.sh install

This installs dstack as a systemd user service with Verda (DataCrunch) backend configured.

For detailed setup instructions, see the dstack Server Setup Guide.

2. Cloud Provider Configuration

Configure dstack with your cloud provider credentials in ~/.dstack/server/config.yml:

projects:
  - name: main
    backends:
      # Verda (if you have an account)
      - type: verda
        creds:
          type: api_key
          client_id: ${VERDA_CLIENT_ID}
          client_secret: ${VERDA_CLIENT_SECRET}

      # Lambda Labs
      - type: lambda
        creds:
          type: api_key
          api_key: ${LAMBDA_API_KEY}

      # RunPod
      - type: runpod
        creds:
          type: api_key
          api_key: ${RUNPOD_API_KEY}

dstack automatically selects the cheapest available provider that meets your resource requirements.

3. Profile Configuration

Create a profiles.yaml file with your model configurations:

profiles:
  qwen3-72b:
    model: "Qwen/Qwen3-72B"
    min_gpu_memory_gb: 80
    gpu_types: [H100, H200, A100_80G]

  qwen3-32b:
    model: "Qwen/Qwen3-32B"
    min_gpu_memory_gb: 48
    gpu_types: [A100, H100, L40S]

  llama3-70b:
    model: "meta-llama/Llama-3-70B-Instruct"
    min_gpu_memory_gb: 80

  codestral-22b:
    model: "mistralai/Codestral-22B-v0.1"
    min_gpu_memory_gb: 48

Profile fields:

model: Hugging Face model ID to serve via vLLM
min_gpu_memory_gb: Minimum GPU memory required (dstack filters to GPUs with this much memory or more)
gpu_types: Optional list of preferred GPU types (e.g., [H100, A100])

Troubleshooting

Plugin Not Loading

Check the logs for plugin loading errors:

grep "LLMServerPlugin" logs/codumentor.log

UI Card Not Appearing

Verify the plugin is enabled in your configuration
Check that auto_prompt is set to true
Ensure the error is a connection error to localhost:1248

Server Not Starting

Verify DataCrunch credentials are set
Check available profiles: python scripts/manage_llm_server.py start --profile list
Check service logs for detailed errors

Health Checks Failing During Startup

The LLM model takes ~6 minutes to load. During this time, health checks will report "No models loaded yet". This is normal—the plugin automatically detects when the model is ready via the /v1/models endpoint.

Auto-Shutdown Triggers During Active Use

Check that your activity is generating onInputReceived events. Only direct user input (not LLM responses or tool executions) resets the inactivity timer. This is by design to prevent agent loops from keeping the server running indefinitely.

Multiple Instances Created

Ensure check_existing_instances: true is set. If backend API calls fail (e.g., network issues), the plugin falls back to allowing start attempts.

dstack Server Connection Failed

Verify dstack server is running: curl http://localhost:3000/api/health
Check dstack_url in configuration matches your server
If using authentication, ensure dstack_token is set correctly
Check dstack server logs for errors

dstack Allocation Fails

Verify cloud provider credentials in ~/.dstack/server/config.yml
Check that at least one provider has available capacity
Ensure your profile's min_gpu_memory_gb is achievable (some providers may not have large GPUs available)
Check dstack server logs: dstack server output or ~/.dstack/server/logs/

Profile Not Found (dstack)

Verify dstack_profiles_path points to a valid YAML file
Check the profile name matches exactly (case-sensitive)
Verify YAML syntax: python -c "import yaml; yaml.safe_load(open('profiles.yaml'))"

CLI Interface

For command-line server management, see the LLM Server CLI.

Common commands:

# Start server with default profile
python scripts/manage_llm_server.py start

# Start with specific profile
python scripts/manage_llm_server.py start --profile qwen3-32b

# Stop/hibernate all instances
python scripts/manage_llm_server.py stop

# List available profiles
python scripts/manage_llm_server.py start --profile list

Architecture

The plugin consists of four layers:

Service Layer (src/codumentor/services/llm_server/): - Core logic for server management - Singleton state manager for coordination - Backend adapters (DataCrunch, dstack) - Instance lifecycle management

GPU Fleet Library (src/gpu_fleet/): - FleetManager protocol for server allocation/release - DstackFleetManager implementation wrapping dstack SDK - Profile loading from YAML - Provider-agnostic interface

Plugin Layer (src/codumentor/plugins/llm_server/): - Hook handlers for error interception and activity tracking - Background scheduler for health and inactivity checks - REST API endpoints - UI event emission

UI Layer (ui/plugins/llm_server/): - Custom element <x-llm-server-card> - Interactive controls for server management - Real-time progress display

Backend Flow

┌─────────────────────────────────────────────────────────────────┐
│                     LLMServerStateManager                        │
│  (Singleton state, backend dispatch, UI event broadcasting)     │
└─────────────────────────────┬───────────────────────────────────┘
                              │
              ┌───────────────┴───────────────┐
              │ backend="datacrunch"          │ backend="dstack"
              ▼                               ▼
┌─────────────────────────┐     ┌─────────────────────────────────┐
│   DataCrunch Backend    │     │    FleetManagerAdapter          │
│   - instance_manager.py │     │    - Wraps gpu_fleet library    │
│   - SSH tunnel required │     │    - HTTPS endpoints (no SSH)   │
│   - profiles.json       │     │    - profiles.yaml              │
└───────────┬─────────────┘     └───────────────┬─────────────────┘
            │                                   │
            ▼                                   ▼
┌───────────────────────┐         ┌─────────────────────────────────┐
│   DataCrunch API      │         │       gpu_fleet Library         │
│                       │         │   - FleetManager protocol       │
└───────────┬───────────┘         │   - DstackFleetManager impl     │
            │                     └───────────────┬─────────────────┘
            ▼                                     │
┌───────────────────────┐                         ▼
│   GPU Instance        │         ┌─────────────────────────────────┐
│   (via SSH tunnel)    │         │       dstack Server             │
└───────────────────────┘         │   - Multi-provider selection    │
                                  │   - Shared state (PostgreSQL)   │
                                  └───────────────┬─────────────────┘
                                                  │
                                  ┌───────────────┴───────────────┐
                                  ▼               ▼               ▼
                               Verda          Lambda          RunPod
                                  │               │               │
                                  ▼               ▼               ▼
                               GPU Instance   GPU Instance   GPU Instance
                               (HTTPS endpoint)

Module Structure

src/codumentor/plugins/llm_server/
├── __init__.py           # Package exports
├── llm_server_plugin.py  # Main plugin class with scheduler integration
├── error_detection.py    # Connection error pattern matching
├── health_check.py       # HTTP health probe utilities (IP and endpoint)
└── scheduler.py          # Background task scheduler

src/codumentor/services/llm_server/
├── __init__.py           # Service exports
├── server_state.py       # Singleton state manager with dual-backend support
├── fleet_adapter.py      # Adapter bridging gpu_fleet with state manager
├── instance_manager.py   # DataCrunch instance operations (legacy)
├── datacrunch_client.py  # DataCrunch API client factory
├── profiles_manager.py   # DataCrunch profile configuration
├── ssh_tunnel.py         # SSH port forwarding (DataCrunch only)
└── exceptions.py         # Custom exceptions

src/gpu_fleet/
├── __init__.py           # Public exports (create_fleet_manager, etc.)
├── interface.py          # FleetManager protocol, LLMServer, Profile dataclasses
├── manager.py            # DstackFleetManager implementation
├── profiles.py           # YAML profile loading
├── factory.py            # create_fleet_manager() entry point
└── exceptions.py         # FleetError hierarchy

Source Code

Plugin: src/codumentor/plugins/llm_server/llm_server_plugin.py
Service: src/codumentor/services/llm_server/
UI: ui/plugins/llm_server/

LLM Server Manager Plugin

Overview

Backends

What is dstack?

Backend Comparison

Configuration

DataCrunch Backend (Legacy)

dstack Backend

Configuration Parameters

Common Parameters

dstack-Specific Parameters

How It Works

Error Detection

Existing Instance Detection

Health Monitoring

Inactivity Detection

UI Card States

Multi-User Synchronization

Server States

State Fields

REST API Endpoints

Start Server

Get Status

List Profiles

UI Event Types

Prerequisites

DataCrunch Backend

API Credentials

Server Profiles

dstack Backend

1. dstack Server

2. Cloud Provider Configuration

3. Profile Configuration

Troubleshooting

Plugin Not Loading

UI Card Not Appearing

Server Not Starting

Health Checks Failing During Startup

Auto-Shutdown Triggers During Active Use

Multiple Instances Created

dstack Server Connection Failed

dstack Allocation Fails

Profile Not Found (dstack)

CLI Interface

Architecture

Backend Flow

Module Structure

Source Code

See Also