LLM Server Manager Plugin

The LLM Server plugin detects when the local (or fleet-hosted) LLM is unreachable and offers to start it from an interactive UI card. It coordinates across users so only one instance is started, watches health, and can shut the server down after inactivity.

Backends

Backend	Connection	When to use
datacrunch (default, legacy)	SSH tunnel to a DataCrunch GPU instance (localhost:1248)	Existing DataCrunch setups
dstack	HTTPS endpoint from a dstack server	Multi-cloud provisioning; no SSH tunnel
fleet_manager	Embedded in-process fleet, or a remote fleet-manager service	Newer GPU fleet with pluggable providers

dstack (and fleet_manager) pick a provider that meets the profile's GPU constraints. DataCrunch needs the SSH Tunnel plugin (or this plugin's own tunnel) to localhost:1248.

Configuration

Common + DataCrunch

plugins:
  - module: codumentor.plugins.llm_server
    class: LLMServerPlugin
    priority: 5
    args:
      enabled: true
      auto_prompt: true
      default_profile: "qwen3-fin02"
      backend: "datacrunch"

      health_check_interval_seconds: 60
      health_check_timeout_seconds: 10
      health_check_max_failures: 3
      startup_grace_period_minutes: 10

      auto_shutdown_enabled: true
      inactivity_timeout_minutes: 30

      check_existing_instances: true

dstack

plugins:
  - module: codumentor.plugins.llm_server
    class: LLMServerPlugin
    priority: 5
    args:
      enabled: true
      backend: "dstack"
      dstack_url: "http://localhost:3000"
      dstack_token: "${DSTACK_TOKEN}"
      dstack_profiles_path: "profiles.yaml"
      dstack_fleet_config_path: null
      dstack_project: "main"
      default_profile: "qwen3-72b"
      auto_shutdown_enabled: true
      inactivity_timeout_minutes: 30

fleet_manager

plugins:
  - module: codumentor.plugins.llm_server
    class: LLMServerPlugin
    args:
      enabled: true
      backend: "fleet_manager"
      fleet_profiles_path: "fleet-profiles.yaml"
      fleet_manager_mode: "embedded"    # or "remote"
      fleet_manager_url: null           # required when mode is remote
      fleet_manager_api_key: null

Parameters

Parameter	Type	Default	Description
enabled	bool	true	Enable the plugin
auto_prompt	bool	true	Show the start-server card when a connection error is detected
default_profile	string	null	Profile to start if the user does not pick one
backend	string	"datacrunch"	"datacrunch", "dstack", or "fleet_manager"
health_check_interval_seconds	number	60	Seconds between health checks
health_check_timeout_seconds	number	10	Timeout for each health HTTP request
health_check_max_failures	int	3	Consecutive failures before the server is marked ERROR
startup_grace_period_minutes	number	10	After start, health failures do not mark ERROR until the model has been ready once or this many minutes have passed (model load is often ~6 minutes)
auto_shutdown_enabled	bool	true	Stop the instance after user inactivity
inactivity_timeout_minutes	number	30	Minutes since the last user message before auto-shutdown. Agent/tool activity does not reset the timer
check_existing_instances	bool	true	Reuse a running/starting instance instead of allocating a second GPU

dstack-only

Parameter	Type	Default	Description
dstack_url	string	"http://localhost:3000"	dstack server URL
dstack_token	string	null	Auth token if the dstack server requires it
dstack_profiles_path	string	null	YAML file of model/GPU profiles
dstack_fleet_config_path	string	null	Optional dstack fleet YAML
dstack_project	string	"main"	dstack project name

fleet_manager-only

Parameter	Type	Default	Description
fleet_profiles_path	string	null	YAML profiles for the embedded fleet
fleet_manager_mode	string	"embedded"	"embedded" or "remote"
fleet_manager_url	string	null	Remote fleet-manager URL (remote mode)
fleet_manager_api_key	string	null	API key for the remote fleet manager

What users see

When the LLM connection fails (connection refused, localhost:1248 / 127.0.0.1:1248 unreachable, and similar), a card appears:

1. Prompt — Start or dismiss
2. Starting — progress while the instance boots
3. Success — instance reachable (IP or HTTPS endpoint)
4. Error — message plus Retry
5. Auto-shutdown — notice before inactivity stop

All connected conversations share the same progress. Duplicate start clicks do not spawn a second GPU.

Health checks hit /v1/models (DataCrunch via the SSH tunnel on 127.0.0.1:1248; dstack/fleet against the HTTPS endpoint). An empty model list means the weights are still loading — that is expected during the grace period.

Prerequisites

DataCrunch

export DATACRUNCH_CLIENT_ID="your_client_id"
export DATACRUNCH_CLIENT_SECRET="your_client_secret"

Profiles live in scripts/llm_server_manager/profiles.json (instance type, location, description). Pair with SSH Tunnel if the model is reached through localhost:1248.

dstack

Install and configure a dstack server — see the dstack Server Setup Guide. Point dstack_profiles_path at a YAML file:

profiles:
  qwen3-72b:
    model: "Qwen/Qwen3-72B"
    min_gpu_memory_gb: 80
    gpu_types: [H100, H200, A100_80G]

Cloud credentials belong in ~/.dstack/server/config.yml on the dstack host, not in the Codumentor plugin args.

HTTP API

Mounted at /ui/plugins/llm_server/ (authenticated like other plugin routes):

Method	Path	Description
POST	/start	Start (body: {"profile": "…"} optional)
POST	/stop	Stop / hibernate
GET	/status	Current status, IP or endpoint, progress
GET	/profiles	Available profiles
POST	/establish-tunnel	Re-open the SSH tunnel to a DataCrunch instance that is already running

CLI

See scripts/README.md. Common commands:

python scripts/manage_llm_server.py start
python scripts/manage_llm_server.py start --profile qwen3-32b
python scripts/manage_llm_server.py stop
python scripts/manage_llm_server.py start --profile list

Troubleshooting

Card does not appear — Plugin enabled, auto_prompt: true, and the error is a connection failure (not an auth/model error).

Health checks fail during startup — Model load often takes ~6 minutes. Failures inside startup_grace_period_minutes (default 10) do not mark ERROR.

Auto-shutdown during an agent-only loop — Only user messages reset the inactivity timer, by design.

Two GPU instances — Keep check_existing_instances: true. If the backend API is unreachable, a start may still be attempted.

dstack connection failed — curl $dstack_url/api/health, matching dstack_url / dstack_token, and dstack server logs.

dstack allocation fails — Provider credentials in ~/.dstack/server/config.yml, capacity, and a min_gpu_memory_gb that some provider can satisfy.

See also

• SSH Tunnel — persistent localhost:1248 forward for DataCrunch
• dstack Server Setup Guide