EdgeCluster One — Architecture Diagram

Five-Layer Architecture Overview

Green solid border = completed and running; dashed border = planned

Completed & Running

Cloud upstream (current interim solution)

Planned, not yet implemented

✓ 14 components completed / 33 total · AMD 395 (Windows/Ollama) connected · L2 multi-node online

▶ Currently Implemented Scope (MVP)

L4 Client Layer ✓ Connected

Kilo Code

VS Code AI coding plugin

OpenAI Compatible + MCP SSE

✓ Configured

Claude Code

Anthropic CLI Agent

Anthropic SDK / OpenAI SDK

✓ Compatible Access

Hermes Agent

Enterprise Agent framework

OpenAI Compatible

⏳ Planned

OpenCode

Open-source coding Agent

OpenAI Compatible

⏳ Planned

↓ LLM Request POST /v1/chat/completions · Bearer ec-dev ↓ MCP Tool Call HTTP/SSE JSON-RPC 2.0

L3 LLM LiteLLM Proxy ✓ Running :8080

OpenAI Compatible

POST /v1/chat/completions

✓ Running

Model Routing

Alias → upstream mapping

✓ 5 aliases

API Key Auth

LITELLM_MASTER_KEY

✓ Running

Retry & Fallback

429/5xx max 2 retries

✓ num_retries=2

Param Cleanup

drop_params=true

✓ Running

Audit Log DB

Requires PostgreSQL (Prisma)

⏳ Planned

L3 MCP agentgateway ✓ Running :8081 / :15000

MCP SSE Endpoint

HTTP/SSE · JSON-RPC 2.0

:8081/sse

✓ Running

stdio Wrapper

npx MCP Server subprocess

✓ Running

Tool Audit UI

Admin observability

:15000/ui

✓ Running

filesystem × 14

read/write/search and other tools

✓ Verified

GitHub MCP

@mcp/server-github

⏳ Slot reserved

EdgeCluster Ops

Custom ops MCP Server

⏳ Planned

↓ OpenAI-Compatible API · BIGMODEL_API_KEY · Current Upstream

L2 Inference Layer ✓ AMD 395 Connected (Windows/Ollama)

BigModel API

Zhipu GLM-4.6 / GLM-4-Flash

open.bigmodel.cn

✓ Cloud Upstream

Ollama (AMD 395 · Windows)

10.253.212.178 · Ollama v0.21.1 · Connected ✅

gemma4:latest (8B Q4_K_M) · edgecluster-gemma4

:11434 · think=false / think=true dual mode

✓ Online

Ollama (AMD 395 · More Models)

Same node · ROCm acceleration pending verification

Llama 3.3 70B · Qwen2.5 72B to be pulled

:11434/v1 · OpenAI-compatible

⏳ Models pending pull

vLLM Node (PoC)

4× DGX Spark · TP=4 · 70B

:8000/v1 · Config reserved

⏳ Awaiting hardware

vLLM 300B Cluster

16 nodes · TP=4 PP=4

Multi-node load balancing

⏳ Planned

⏳ Planned Scope

↓ GPU Compute · Inference Engine · Tensor Parallel

L1 Hardware Layer ✓ AMD 395 Connected

AMD Ryzen AI Max 395

Strix Halo · 40-core RDNA3.5 iGPU · 128GB unified memory

Windows · Ollama v0.21.1 · gemma4:latest ✅

10.253.212.178:11434 · Registered EdgeCluster node

✓ Online · ROCm pending verification

DGX Spark (NVIDIA)

128GB unified memory / node

vLLM · TP=4

⏳ Procuring

AMD RX 9070 XT

16GB GDDR6 · ROCm

⏳ Planned

Moore Threads MTT S80

80GB · Domestic GPU首选

⏳ Planned

High-Speed Interconnect

25/100GbE · RDMA RoCE v2

⏳ Planned

L5 Operations Management Layer ⏳ Planned

Admin Console

React + Ant Design

⏳ Planned

SSO

LDAP / SAML / OIDC

⏳ Planned

Audit Log

PostgreSQL · Prisma

⏳ Planned

Monitoring & Alerting

Prometheus + Grafana

⏳ Planned

Token Statistics

Usage reports · Chargeback

⏳ Planned

L3 Gateway Layer — Internal Component Structure

LiteLLM (LLM traffic) and agentgateway (MCP tool traffic) run side by side, independent of each other

LiteLLM Proxy

port 8080

Ingress Endpoints

POST /v1/chat/completions ← Main entry

GET /v1/models

GET /health/liveliness ← Health check (no auth)

Request Processing Pipeline

Auth Verification

Bearer token = LITELLM_MASTER_KEY

401 → Rejected

Model Routing

model_name → litellm_params (api_base + key)

edgecluster-glm → GLM-4.6

Param Cleanup

drop_params=true: drop unsupported upstream fields

reasoning_content and other GLM-specific fields

Forward to Upstream

HTTP POST → BigModel / vLLM

Supports stream=true SSE passthrough

Retry Logic

429 / 5xx → max 2 retries, retry_after=0

⚠ Note: Anthropic Passthrough

/anthropic/v1/messages → direct connection to real Anthropic, bypasses model_list
To route to BigModel, use /v1/chat/completions + Claude alias

agentgateway

port 8081 · admin 15000

Ingress Endpoints

GET /sse ← SSE connection (client subscribes)

POST / ← JSON-RPC request endpoint

GET /ui ← Admin UI (:15000)

MCP Session Lifecycle

SSE Handshake

GET /sse → establish persistent event stream

Client receives endpoint event

initialize

Negotiate protocol version, declare capabilities

protocolVersion: 2024-11-05

tools/list

Returns 14 filesystem tool descriptions

name + description + inputSchema

tools/call (loop)

{"name":"read_file","arguments":{...}}

Forward to stdio MCP Server subprocess

stdio → Subprocess

npx @modelcontextprotocol/server-filesystem

Scope: /Users/xiaopingfeng

Extending MCP Servers

Append to agentgateway/config.yaml targets:
GitHub, EdgeCluster ops tools, etc. (commented slots already reserved)

End-to-End Data Flow

LLM inference requests (left) and MCP tool calls (right) run in parallel, independently

🧠 LLM Inference Request Flow OpenAI API

Kilo Code

POST /v1/chat/completions
Authorization: Bearer ec-dev
{"model":"edgecluster-glm","stream":true}

↓

HTTP · Local 127.0.0.1

LiteLLM Proxy :8080

① Auth Bearer token
② Lookup model_name → litellm_params
③ drop_params cleanup fields
④ Construct upstream request + BIGMODEL_API_KEY

↓

HTTPS · open.bigmodel.cn/api/paas/v4

BigModel API (GLM-4.6)

Inference execution
reasoning_content (internal consumption)
→ content token streaming output

↓

SSE stream · data: {"choices":[{"delta":{...}}]}

LiteLLM (SSE Passthrough)

Directly pass through SSE stream to client
No additional conversion overhead

↓

SSE stream · HTTP chunked transfer

Kilo Code

Real-time rendering of streaming tokens
Displayed inline in editor

🔧 MCP Tool Call Flow JSON-RPC 2.0

Kilo Code Agent

LLM decides tool call is needed
e.g. read file to assist reasoning

↓

GET /sse · Establish persistent SSE connection

agentgateway :8081

① initialize handshake
② tools/list → return 14 tool manifest
③ tools/call dispatch to target

↓

JSON-RPC over stdin/stdout

MCP Server (Node.js subprocess)

@modelcontextprotocol/server-filesystem
Scope: /Users/xiaopingfeng
Execute actual filesystem operations

↓

stdout · JSON-RPC result

agentgateway

Wrap result → SSE event push
Also write to audit log (Admin UI)

↓

SSE event · tool result JSON

Kilo Code Agent

Tool result injected into Agent context
Continue next inference round to LiteLLM

Startup Sequence

Stop Old Processes

lsof find :8080 / :8081 → kill PID

Start LiteLLM + agentgateway

nohup background start, PID written to /tmp/edgecluster/*.pid

Wait for LiteLLM Ready

curl /health/liveliness · timeout 30s → error

Wait for agentgateway Ready

MCP initialize probe (requires Content-Type + Accept headers)

Model Routing Mechanism

Client model name → LiteLLM route → actual upstream; upstream switching is transparent to the client

Client Model Name	Routes To	Actual Upstream	Usage
edgecluster-glm	openai/glm-4.6	BigModel	Primary inference, Kilo Code default config
edgecluster-glm-flash	openai/glm-4-flash	BigModel	Lightweight & fast, cost-saving scenarios
claude-3-5-sonnet-20241022	openai/glm-4.6	BigModel	Kilo Code / Claude Code default request name
claude-sonnet-4-5	openai/glm-4.6	BigModel	Anthropic SDK alias
claude-3-haiku-20240307	openai/glm-4-flash	BigModel	Mapped to Flash variant for cost savings
edgecluster-70b ⏳	openai/edgecluster-70b	vLLM :8000	PoC phase, enabled when hardware is ready
edgecluster-300b ⏳	openai/edgecluster-300b	vLLM multi-node	Production phase, same-name multi-backend → load balancing

Kilo Code Integration (OpenAI Mode)

        Provider : OpenAI Compatible

        Base URL : http://127.0.0.1:8080/v1

        API Key  : ec-dev

        Model ID : edgecluster-glm

        MCP URL  : http://127.0.0.1:8081/sse

Claude Code Integration (Environment Variables)

        ANTHROPIC_BASE_URL=

          http://127.0.0.1:8080

        ANTHROPIC_API_KEY=ec-dev

        ⚠ /anthropic/* passes through to real Anthropic

        ✓ Use /v1/ + claude alias to bypass

Same-Name Multi-Backend — Load Balancing Principle

Register multiple litellm_params with the same model_name in model_list; LiteLLM Router automatically simple-shuffles (random distribution):

# litellm/config.yaml — Planned: vLLM multi-node
- model_name: edgecluster-300b
  litellm_params:
    model: openai/edgecluster-300b
    api_base: http://192.168.1.11:8000/v1   # node-01
- model_name: edgecluster-300b   # Same name = automatic load balancing
  litellm_params:
    model: openai/edgecluster-300b
    api_base: http://192.168.1.12:8000/v1   # node-02

Evolution Path

Three phases; upstream switching only requires config changes, zero client-side changes

Current ✓

MVP

Local development validation

·LiteLLM :8080
·agentgateway :8081
·Kilo Code integration
·BigModel GLM-4.6 upstream
·filesystem MCP × 14 tools
·start-all.sh one-click startup

⏳ PoC Phase

Local vLLM

4× DGX Spark · 70B inference

·Uncomment vLLM config
·TP=4 tensor parallelism
·Llama-3 70B AWQ
·PostgreSQL audit log
·Prometheus + Grafana
·LiteLLM UI usage statistics

⏳ Production Phase

300B Cluster

16 nodes · TP=4 PP=4

·300B AWQ Q4 quantization
·Multi-node load balancing
·100GbE + RDMA interconnect
·SSO login (LDAP/OIDC)
·Compliance audit + Chargeback
·Flagship customer deployment

⚡ Switch to Local vLLM — Only 3 Config Lines to Change, Restart LiteLLM

# litellm/config.yaml

model_list:
  # Existing BigModel config remains unchanged (can serve as Fallback)
  - model_name: edgecluster-glm
    litellm_params:
      model: openai/glm-4.6
      api_base: https://open.bigmodel.cn/api/paas/v4
      api_key: os.environ/BIGMODEL_API_KEY

  # Uncomment below ↓ fill in real vLLM node IP
  - model_name: edgecluster-70b
    litellm_params:
      model: openai/edgecluster-70b
      api_base: http://192.168.1.11:8000/v1    ← Fill in actual IP
      api_key: none

# Restart: ./start-all.sh
# Kilo Code side: change model to edgecluster-70b, everything else unchanged

Milestones

Milestone	Target Cycle	Key Deliverables	Status
M0 — MVP	Completed	LiteLLM + agentgateway + Kilo Code full-stack connected	✓ Done
M1 — PoC	4 weeks	4-node DGX Spark + vLLM 70B + basic monitoring	⏳ Awaiting hardware
M2 — Alpha	8 weeks	Admin console + SSO + Prometheus + VS Code plugin PoC	⏳ Planned
M3 — Beta	12 weeks	300B cluster + compliance audit + first flagship customer	⏳ Planned

📋 Development Progress Log

Updated after each task completion · Auto-synced to GitHub Pages

2026-04-24

✓ Completed L5 Admin Console System Monitoring Fix & Upgrade

· Extracted deps.py to resolve circular import (routers → main → routers)
· Added routers/system.py: CPU/memory/GPU detection (macOS + Linux + ROCm + NVIDIA)
· Dashboard stat cards connected to real system data (memory usage, GPU model, process online count)
· Added "System Resource Monitoring" page: host resources + GPU cards + process status table

✓ Completed L2 + L1: Ollama & AMD Ryzen AI Max 395 Joined Architecture

· L2 inference layer added Ollama (:11434 OpenAI-compatible, supports Llama 3.3 70B / Qwen2.5 72B)
· L1 hardware layer added AMD Ryzen AI Max 395 (Strix Halo · 40-core RDNA3.5 · 128GB unified memory)
· litellm/config.yaml added Ollama commented config section (uncomment to connect)

✓ Completed GitHub Pages Deployment

· Created public repo fxp/edgecluster-arch
· Enabled GitHub Pages → fxp.github.io/edgecluster-arch
· Auto-sync after each task completion (update-pages.sh)

✓ Completed L5 Admin Console MVP Development

· FastAPI backend (port 9090): health / models / keys / spend / logs / system 6 routes
· Dark theme SPA frontend: dashboard, service health, model list, API Key management, usage statistics, audit log
· Integrated LiteLLM API + Ollama API + agentgateway health probes

✓ Completed Architecture Diagram Published + Completion Status Annotations

· 5-tab architecture diagram: System Overview, Gateway Layer Detail, Data Flow, Model Routing, Evolution Path
· Each component in overview annotated with completion status (green solid = running, dashed = planned)
· Legend, counters, "Currently Implemented Scope" divider

✓ Completed MVP Full-Stack Verification (LiteLLM + agentgateway + Kilo Code)

· LiteLLM :8080 — 5 model aliases, BigModel GLM-4.6, OpenAI compatible ✅
· agentgateway :8081 — MCP SSE, filesystem × 14 tools ✅
· Kilo Code MCP config written (mcp_settings.json), VS Code connected ✅
· start-all.sh one-click full-stack startup (LiteLLM + agentgateway + Admin)

2026-04-26

✓ Completed L5 Admin — Usage Statistics Dashboard

· Added /spend/summary today/month summary endpoint
· Added /spend/daily 7-day trend endpoint
· Frontend three-tab view: by model / by key / 7-day trend (CSS horizontal bar chart)
· Top stat cards: today's requests, token count, cost

✓ Completed L5 Admin — API Key Full Quota Management

· Added PUT /keys/{key} update budget/models/rate limits
· Added POST /keys/{key}/block|unblock disable/restore
· Frontend: budget progress bar, RPM display, copy key, edit modal, disable/restore buttons
· Key creation supports RPM + TPM rate limit fields

✓ Completed L3 agentgateway — MCP Gateway Management Page

· Added routers/mcp.py: parse config.yaml + detect agentgateway online status
· Frontend: MCP gateway status card, registered server list (type/command/URL)
· Audit log page improved (MCP JSONL + LLM log dual view)

✓ Completed Prometheus + Grafana Monitoring Integration

· Added GET /api/metrics (Prometheus text format)
· Exports: memory, service online status, platform info and other metrics
· Created monitoring/docker-compose.yml (Prometheus :9091 + Grafana :3000)
· Admin UI added "Prometheus Metrics" page (live preview + config examples)

✓ Completed SSO Login Page + Multi-Node Management

· Login overlay: enter Admin Key → localStorage persistence, supports logout
· Added routers/nodes.py: node registry CRUD (JSON file storage)
· Node register/remove/ping, real-time health detection (async concurrent)
· Frontend "Node Management" page: local status + remote node cards (latency, tags, status)

2026-04-26

✓ Verified Full-Chain Integration Test Passed (BigModel as Inference Provider)

· L2 LiteLLM → BigModel GLM: 5 model aliases, chat/stream/function-calling all passed ✅
· L3 agentgateway MCP: 14 filesystem tools, Session-ID mechanism normal ✅
· L2+L3 Closed Loop: LLM decides tool → MCP executes → result backfill → final answer (user query → read directory → generate answer) ✅
· L4 OpenAI Python SDK direct to EdgeCluster, streaming output, claude alias routing all normal ✅
· L5 Admin Console v0.2 all APIs show real data, fixed agentgateway health check false alarm ✅
· Finding: glm-4.6 is a reasoning model, requires max_tokens≥300 (reasoning consumes ~220tok); glm-4-flash has no reasoning latency

2026-04-28

✓ Completed AMD Ryzen AI Max 395 (Windows/Ollama) — Remote Node Connected

· Hardware: AMD Ryzen AI Max 395 (Strix Halo · 40-core RDNA3.5 · 128GB · 10.253.212.178) running Windows + Ollama v0.21.1, model gemma4:latest (8B Q4_K_M)
· LiteLLM Routing: Added edgecluster-gemma4 (think=false) and edgecluster-gemma4-think (think=true) model aliases
· Finding: Gemma4 triggers thinking mode on Chinese input, reasoning consumes all tokens, content is empty → fixed via extra_body: think: false
· Node Registration: Rewrote routers/nodes.py, supports auto-detecting node type (EdgeCluster Admin / Ollama-only)
· Detection order: GET /api/health (EdgeCluster) → GET /api/version (Ollama) → returns node_type / ollama_version / models
· Admin UI Node Card: Displays node_type badge (Ollama/EdgeCluster), Ollama version, model list, latency
· Fixed dedup bug: registering same ID or same URL triggers update (no longer inserts duplicate entries)
· Verified: after registration, ping returns online:true, latency_ms:~70, models:["gemma4:latest"] ✅

2026-04-28

✓ Completed Admin Console v0.3 — Operations Management Layer Fully Implemented

· Node Model Management: GET/POST/DELETE /api/nodes/{id}/models — list, pull (SSE streaming progress), delete remote Ollama node models
· Service Lifecycle Control: POST /api/system/services/{name}/restart|stop|start — UI restart LiteLLM / agentgateway
· Background Health Monitoring: FastAPI lifespan starts asyncio task, 60s probe all services, memory ring buffer stores 2h history, GET /api/health/history returns uptime + Sparkline data points
· Real-time Log Stream: GET /api/logs/stream/{service} — SSE tail logs, supports ?api_key= query param auth (EventSource doesn't support Headers)
· Config Editor: GET/PUT /api/config/litellm — online edit LiteLLM config.yaml, YAML validation before write, auto-restart LiteLLM after save
· Frontend Added 3 Pages: Service Control (with uptime Sparkline) / Real-time Log Stream (terminal emulation) / Config Editor (save triggers restart)
· Node card added "Manage Models" button → Modal for Pull/delete models with progress bar
· Dashboard added health history Sparkline (async injection from background monitor)
· Dual auth mode: Bearer Header + ?api_key= Query Param (deps.py unified support)

2026-04-29

✓ Completed Inference Layer Migrated to BigModel (AMD 395 Temporarily Not on Same Network)

· All model aliases switched to BigModel API: edgecluster-gemma4 / gemma4-think / claude-* all route to GLM-4.6 / GLM-4-Flash
· Ollama entries kept as comments (api_base: http://10.253.212.178:11434), one-line uncomment to restore when back on same LAN
· Fixed gemma4 offline alias token exhaustion issue: Original alias pointed to GLM-4.6 (reasoning model), with max_tokens≤200 reasoning_content exhausted budget, content was empty; changed to GLM-4-Flash (non-reasoning), all tests normal under max_tokens:60 ✅
· Finding: GLM-4.6 is designed for reasoning tasks requiring larger token budgets (recommended max_tokens≥500); daily conversation/tool calls recommended GLM-4-Flash

🔍 Root Cause gemma4-think Returns tool_call Instead of Text Reply — Root Cause Analysis

· Root Cause: Kilo Code auto-injects tool definitions to all requests (including answer tool); Gemma4 with thinking mode enabled is extremely inclined to call tools rather than reply directly, resulting in {"tool_calls":[{"function":"answer","args":{...}}]}
· Core Conflict: think mode + external tool injection → model wraps "how to answer" as a tool call
· Solution: Use edgecluster-glm-flash in Kilo Code; keep edgecluster-gemma4-think for direct API call scenarios without tool injection
· Route Verification Methods: ① Ask model identity (Gemma4 → "Google DeepMind", GLM → "ChatGLM"); ② Check response model field; ③ Check LiteLLM routing logs in real-time log stream page

✓ Fixed LiteLLM Process Startup — Environment Variable Passing Fix

· Issue: nohup with source .env doesn't reliably inherit environment variables on some shells, causing BIGMODEL_API_KEY not to be passed to LiteLLM process, returning AuthenticationError
· Fix: Changed to env $(cat .env | grep -v '^#' | xargs) nohup litellm ..., explicitly injecting .env key-values into subprocess
· Startup script acknowledged, pending unified update to start-all.sh

2026-04-30

✓ Completed PostgreSQL@16 Auto-Install & LiteLLM Database Initialization (setup-db.sh)

· Idempotent install script bash setup-db.sh: Homebrew install PG@16 → initdb cluster initialization → start → create user edgecluster + database → write DATABASE_URL to .env → Prisma DB Push — fully automated, no manual configuration needed
· Prisma migration: prisma db push --schema schema.prisma --skip-generate, first run creates 60 tables (LiteLLM_SpendLogs, LiteLLM_VerificationToken, etc.), idempotent and re-runnable
· start-all.sh integration: Added _ensure_postgres(), auto-checks and starts PG on each launch (prints prompt to run setup-db.sh if not installed)
· Key detail: macOS requires LC_ALL="en_US.UTF-8" to start PG properly (otherwise FATAL: postmaster became multithreaded); all variables in script use ${VAR} format to avoid Unicode character parsing bugs

✓ Fixed Admin Console Three Bug Fixes

· agentgateway health probe: Original logic POSTed MCP initialize to :8081/ → SSE streaming response caused httpx timeout → status ❌; changed to GET :15021/healthz/ready (agentgateway native readiness probe, ~21ms returns 200) ✅
· Create Key no response: httpx.HTTPStatusError str(e) doesn't contain response body, LiteLLM's "DB not connected" in body was dropped; fixed _check_db_error() to read e.response.text, throws 422 + descriptive message when no DB; after DB connected, Key creation works normally and returns full sk-... token
· Error Toast styling: showToast(msg, 'error') added red border + auto 6s close (success Toast is 3s), frontend doCreateKey() added try/catch and r.ok check

🔧 Hardened LiteLLM Startup Chain Hardening (Telemetry Hang / Env Vars / Path Spaces)

· Startup hang root cause: LiteLLM sends 3× SYN_SENT telemetry requests on startup (target 127.0.0.2:80), intercepted by local proxy → hangs 2-3 minutes; fix: .env add LITELLM_TELEMETRY=False, LITELLM_LOCAL_MODEL_COST_MAP=True, config.yaml add telemetry: false, all startup commands append --telemetry False
· Environment variable injection: env $(grep -v '^#' .env | grep '=' | xargs) nohup litellm ensures variables are explicitly injected into subprocess (avoids nohup dropping variables); note .env comments must be on their own line, no inline comments
· iCloud path spaces: system.py/config.py use shlex.quote() to wrap iCloud paths containing spaces, preventing LiteLLM CLI from splitting spaces into extra arguments

⏳ Next Steps

▸AMD 395 back on same network → Ollama route switchback + ROCm verification

▸AMD 395 pull Llama 3.3 70B / Qwen2.5 72B

▸Admin usage statistics page view real spend data (DB connected)

▸Grafana dashboard JSON pre-configuration (auto-import)

▸vLLM node integration (after DGX Spark arrives)

▸Multi-node LiteLLM route aggregation (cross-node load balancing)

EdgeCluster One — Architecture Diagram

Revision history

Related reading