dstack Server Setup Guide

This guide describes how to install and configure a dstack server for use with Codumentor's LLM Server Plugin.

Overview

dstack is an open-source GPU orchestration platform that provisions GPUs across multiple cloud providers. When configured as the backend for the LLM Server Plugin, it:

Provisions GPU instances on Verda (DataCrunch) automatically
Exposes HTTPS endpoints directly (no SSH tunnel needed)
Enables future multi-provider support (Lambda Labs, RunPod, AWS, etc.)

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Codumentor Server                         │
│                                                              │
│  ┌─────────────────┐     ┌──────────────────────────────┐  │
│  │  LLM Server     │────>│  dstack Server (:3000)       │  │
│  │  Plugin         │     │  - Manages GPU provisioning  │  │
│  └─────────────────┘     │  - Verda backend configured  │  │
│                          └─────────────┬────────────────┘  │
└────────────────────────────────────────│────────────────────┘
                                         │
                                         ▼
                              ┌──────────────────────┐
                              │  Verda/DataCrunch    │
                              │  - 4xH200 instance   │
                              │  - vLLM + GLM-4.7    │
                              │  - HTTPS endpoint    │
                              └──────────────────────┘

Prerequisites

Python 3.11+
DataCrunch API credentials:
DATACRUNCH_CLIENT_ID
DATACRUNCH_CLIENT_SECRET

Set these in your shell profile (e.g., ~/.bashrc):

export DATACRUNCH_CLIENT_ID="your_client_id"
export DATACRUNCH_CLIENT_SECRET="your_client_secret"

Installation

Automated Installation

Use the provided install script:

# Ensure credentials are exported
source ~/.bashrc

# Run installer (as regular user for user service, or as root for system service)
/usr/local/lib/codumentor/scripts/install-dstack-server.sh install

The script will:

Create a Python virtual environment with dstack
Configure dstack with Verda backend using your credentials
Create and start a systemd service
Save the admin token

Manual Installation

If you prefer manual setup:

1. Install dstack

# Create directory
mkdir -p ~/.local/share/dstack
cd ~/.local/share/dstack

# Create virtual environment
python3 -m venv venv
source venv/bin/activate
pip install "dstack[all]"

2. Configure dstack Server

Create ~/.dstack/server/config.yml:

projects:
  - name: main
    backends:
      - type: verda
        creds:
          type: api_key
          client_id: "YOUR_DATACRUNCH_CLIENT_ID"
          client_secret: "YOUR_DATACRUNCH_CLIENT_SECRET"

Set permissions:

chmod 600 ~/.dstack/server/config.yml

3. Create systemd User Service

Create ~/.config/systemd/user/dstack.service:

[Unit]
Description=dstack GPU Orchestration Server
After=network.target

[Service]
Type=simple
WorkingDirectory=/home/YOUR_USER/.local/share/dstack
ExecStart=/home/YOUR_USER/.local/share/dstack/venv/bin/dstack server --host 127.0.0.1 --port 3000
Restart=on-failure
RestartSec=5

[Install]
WantedBy=default.target

Enable and start:

systemctl --user daemon-reload
systemctl --user enable dstack
systemctl --user start dstack

4. Get Admin Token

journalctl --user -u dstack | grep "admin token"

Save the token to the secrets file (used by Codumentor systemd services):

echo 'DSTACK_TOKEN=YOUR_TOKEN' >> ~/.config/codumentor/secrets.env

Also save locally for reference:

echo "YOUR_TOKEN" > ~/.local/share/dstack/admin-token
chmod 600 ~/.local/share/dstack/admin-token

Configuration

dstack Profiles

The LLM Server Plugin uses profiles to define GPU requirements. Create or edit /usr/local/lib/codumentor/config/dstack-profiles.yaml:

profiles:
  glm47:
    model: "zai-org/GLM-4.7-FP8"
    min_gpu_memory_gb: 564  # 4x141GB H200
    gpu_types: [H200]
    port: 8000
    image: "vllm/vllm-openai:latest"
    extra_args:
      - "--tensor-parallel-size=4"
      - "--max-model-len=190000"
      - "--gpu-memory-utilization=0.95"
      - "--trust-remote-code"
      - "--enable-auto-tool-choice"
      - "--tool-call-parser=glm47"
      - "--reasoning-parser=glm45"

Profile fields:

Field	Description
`model`	HuggingFace model ID
`min_gpu_memory_gb`	Minimum total GPU memory required
`gpu_types`	Preferred GPU types (e.g., H200, H100, A100)
`port`	Port vLLM listens on
`image`	Docker image for vLLM
`extra_args`	Additional vLLM command-line arguments

Codumentor Plugin Configuration

Update codumentor.yaml to use the dstack backend:

plugins:
  - module: codumentor.plugins.llm_server
    class: LLMServerPlugin
    priority: 5
    args:
      enabled: true
      auto_prompt: true
      default_profile: "glm47"

      # dstack backend
      backend: "dstack"
      dstack_url: "http://localhost:3000"
      dstack_token: "${DSTACK_TOKEN}"  # Loaded from ~/.config/codumentor/secrets.env
      dstack_profiles_path: "/usr/local/lib/codumentor/config/dstack-profiles.yaml"
      dstack_fleet_config_path: "/usr/local/lib/codumentor/config/dstack-fleet.yaml"
      dstack_project: "main"

      # Health checks
      health_check_interval_seconds: 60
      health_check_timeout_seconds: 10
      health_check_max_failures: 3
      startup_grace_period_minutes: 10

      # Auto-shutdown
      auto_shutdown_enabled: true
      inactivity_timeout_minutes: 30

      check_existing_instances: true

Service Management

Commands

# Check status
systemctl --user status dstack

# View logs
journalctl --user -u dstack -f

# Restart service
systemctl --user restart dstack

# Stop service
systemctl --user stop dstack

Verify Installation

# Check dstack is responding
curl http://localhost:3000/

# List projects (requires dstack CLI)
source ~/.local/share/dstack/venv/bin/activate
dstack project list

Uninstallation

/usr/local/lib/codumentor/scripts/install-dstack-server.sh uninstall

Or manually:

systemctl --user stop dstack
systemctl --user disable dstack
rm ~/.config/systemd/user/dstack.service
systemctl --user daemon-reload

To fully remove:

rm -rf ~/.local/share/dstack ~/.dstack

Troubleshooting

Service Won't Start

Check logs:

journalctl --user -u dstack -n 50

Common issues:

Invalid credentials in config.yml
Port 3000 already in use
Python environment issues

Plugin Can't Connect to dstack

Verify dstack is running: systemctl --user status dstack
Check URL in codumentor.yaml matches dstack server
Verify admin token is correct

GPU Allocation Fails

Check Verda credentials are valid
Verify requested GPU type is available
Check dstack server logs for detailed errors

Reverting to DataCrunch Backend

To switch back to the legacy DataCrunch backend:

Update codumentor.yaml: ``yaml args: backend: "datacrunch" # Changed from "dstack" ``

Restart Codumentor

The dstack service can remain running for future use.

Secrets Management

Codumentor systemd services load environment variables from:

~/.config/codumentor/secrets.env

Add the dstack token there:

echo 'DSTACK_TOKEN=your-token-here' >> ~/.config/codumentor/secrets.env

This file is referenced via EnvironmentFile= in systemd service units and should contain all sensitive configuration (API keys, tokens, passwords).

File Locations

File	Purpose
`~/.config/codumentor/secrets.env`	Secrets for Codumentor services (DSTACK_TOKEN, etc.)
`~/.local/share/dstack/venv/`	dstack Python environment
`~/.local/share/dstack/admin-token`	Admin authentication token (backup)
`~/.dstack/server/config.yml`	dstack server configuration
`~/.dstack/config.yml`	dstack CLI configuration (auto-generated)
`~/.config/systemd/user/dstack.service`	systemd service unit
`/usr/local/lib/codumentor/config/dstack-profiles.yaml`	LLM server profiles
`/usr/local/lib/codumentor/config/dstack-fleet.yaml`	Fleet configuration (auto-created if missing)
`/usr/local/lib/codumentor/scripts/install-dstack-server.sh`	Installation script

Adding More Cloud Providers

dstack supports multiple backends. To add additional providers, edit ~/.dstack/server/config.yml:

projects:
  - name: main
    backends:
      # Verda (DataCrunch)
      - type: verda
        creds:
          type: api_key
          client_id: "${DATACRUNCH_CLIENT_ID}"
          client_secret: "${DATACRUNCH_CLIENT_SECRET}"

      # Lambda Labs (example)
      - type: lambda
        creds:
          type: api_key
          api_key: "${LAMBDA_API_KEY}"

      # RunPod (example)
      - type: runpod
        creds:
          type: api_key
          api_key: "${RUNPOD_API_KEY}"

Restart dstack after changes:

systemctl --user restart dstack

dstack automatically selects the cheapest available provider that meets profile requirements.