ModelExpress is a model distribution cache layer in NVIDIA Dynamo. It sits between the workers and the model source (MatrixHub or Hugging Face). The first worker triggers a download into the ModelExpress cache. Every subsequent worker gets the model from that cache — no second download.
In this test we deploy two Dynamo vLLM workers for Qwen/Qwen2.5-1.5B-Instruct (~3 GB) and compare the model acquisition time of the first worker (cache miss) versus the second worker (cache hit).
Environment
| Component | Configuration |
|---|---|
| GPU | HAMi vGPU |
| Model | Qwen/Qwen2.5-1.5B-Instruct (~3 GB) |
| ModelExpress | v0.3.0 |
| MatrixHub | self-hosted, Hugging Face-compatible endpoint |
| Storage mode | NO_SHARED_STORAGE=1 (gRPC streaming) |
How it works
┌──────────┐ ┌──────────────┐ ┌────────────┐
│ Worker 1 │────▶│ ModelExpress │────▶│ MatrixHub │
│ Worker 2 │────▶│ (cache) │ │ (registry) │
│ Worker N │────▶│ │ └────────────┘
└──────────┘ └──────────────┘
- First request: ModelExpress downloads the model from MatrixHub into its local cache, then streams the files to the requesting worker over gRPC.
- Subsequent requests: ModelExpress streams directly from its local cache. No download from MatrixHub.
Compared to a plain MatrixHub setup (Blog 1), the decode component adds three environment variables:
VLLM_PLUGINS=modelexpress— enable the ModelExpress vLLM pluginMODEL_EXPRESS_NO_SHARED_STORAGE=1— use gRPC streaming instead of a shared filesystemMODEL_EXPRESS_URL— the ModelExpress server address
Deployment files
Worker 1: dgd-blog2-c-mx.yaml
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeployment
metadata:
name: vllm-7b-c
namespace: dynamo-system
spec:
components:
- name: Frontend
type: frontend
replicas: 1
podTemplate:
spec:
containers:
- name: main
image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:latest
workingDir: /workspace
env:
- name: HF_ENDPOINT
value: "http://<matrixhub-endpoint>"
command: ["python3", "-m", "dynamo.frontend"]
args: ["--http-port", "8000"]
resources:
requests:
cpu: "2"
memory: "4Gi"
limits:
cpu: "2"
memory: "4Gi"
- name: decode
type: decode
replicas: 1
podTemplate:
spec:
containers:
- name: main
image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:latest
workingDir: /workspace
env:
- name: HF_ENDPOINT
value: "http://<matrixhub-endpoint>"
- name: VLLM_PLUGINS
value: "modelexpress"
- name: MODEL_EXPRESS_NO_SHARED_STORAGE
value: "1"
- name: MODEL_EXPRESS_URL
value: "http://<modelexpress-service>:8001"
command: ["python3", "-m", "dynamo.vllm"]
args:
- --model
- Qwen/Qwen2.5-1.5B-Instruct
- --served-model-name
- Qwen/Qwen2.5-1.5B-Instruct
- --tensor-parallel-size
- "1"
- --gpu-memory-utilization
- "0.85"
- --max-model-len
- "8192"
- --no-enable-log-requests
resources:
requests:
cpu: "4"
memory: "16Gi"
nvidia.com/vgpu: "1"
nvidia.com/gpucores: "30"
nvidia.com/gpumem: "10000"
limits:
cpu: "4"
memory: "16Gi"
nvidia.com/vgpu: "1"
nvidia.com/gpucores: "30"
nvidia.com/gpumem: "10000"
Worker 2: dgd-blog2-c2-mx.yaml
Worker 2 is a separate DynamoGraphDeployment with the same configuration but a different name (vllm-7b-c2). The full YAML is identical except for metadata.name.
Clear the ModelExpress cache
ModelExpress stores model files on a PVC (local-path). Restarting the pod alone does not remove the cached files. To start from a clean state:
kubectl exec -n model-express deploy/model-express-modelexpress \
-- rm -rf /root/models--Qwen--Qwen2.5-1.5B-Instruct /root/blobs
kubectl rollout restart deployment/model-express-modelexpress -n model-express
kubectl rollout status deployment/model-express-modelexpress -n model-express
Verify:
kubectl exec -n model-express deploy/model-express-modelexpress \
-- ls /root/models--Qwen--Qwen2.5-1.5B-Instruct
# Expected: ls: cannot access ... No such file or directory
Deploy Worker 1
kubectl apply -f dgd-blog2-c-mx.yaml
kubectl get pods -n dynamo-system -o wide -w
Watch the decode pod logs:
kubectl logs -n dynamo-system -f <c-decode-pod>
2026-07-06T02:28:45 INFO dynamo_llm::hub: Successfully connected to ModelExpress server
2026-07-06T02:28:45 INFO modelexpress_client: Requesting model: Qwen/Qwen2.5-1.5B-Instruct from provider: HuggingFace
2026-07-06T02:28:45 INFO modelexpress_client: Model Qwen/Qwen2.5-1.5B-Instruct: Model download in progress
2026-07-06T02:29:24 INFO modelexpress_client: Model Qwen/Qwen2.5-1.5B-Instruct: Model download completed successfully
2026-07-06T02:29:24 INFO modelexpress_client: Shared storage disabled, streaming files from server for model Qwen/Qwen2.5-1.5B-Instruct
2026-07-06T02:29:24 INFO modelexpress_client: Streaming model Qwen/Qwen2.5-1.5B-Instruct files to "/home/dynamo/.model-express/cache" with chunk size 32768 bytes
2026-07-06T02:29:48 INFO modelexpress_client: Streaming complete: received 8 files (3098967011 bytes) for model Qwen/Qwen2.5-1.5B-Instruct
Worker 1 waited 38.3 seconds for ModelExpress to download the model from MatrixHub, then received the files over gRPC streaming in 24.2 seconds.
Deploy Worker 2
After Worker 1 is ready, deploy Worker 2:
kubectl apply -f dgd-blog2-c2-mx.yaml
kubectl get pods -n dynamo-system -o wide -w
Watch the decode pod logs:
kubectl logs -n dynamo-system -f <c2-decode-pod>
2026-07-06T02:32:35 INFO dynamo_llm::hub: Successfully connected to ModelExpress server
2026-07-06T02:32:35 INFO modelexpress_client: Requesting model: Qwen/Qwen2.5-1.5B-Instruct from provider: HuggingFace
2026-07-06T02:32:35 INFO modelexpress_client: Model Qwen/Qwen2.5-1.5B-Instruct: Model already downloaded
2026-07-06T02:32:35 INFO modelexpress_client: Shared storage disabled, streaming files from server for model Qwen/Qwen2.5-1.5B-Instruct
2026-07-06T02:32:35 INFO modelexpress_client: Streaming model Qwen/Qwen2.5-1.5B-Instruct files to "/home/dynamo/.model-express/cache" with chunk size 32768 bytes
2026-07-06T02:32:58 INFO modelexpress_client: Streaming complete: received 8 files (3098967011 bytes) for model Qwen/Qwen2.5-1.5B-Instruct
Worker 2 sees Model already downloaded — ModelExpress skips the download entirely and streams directly from its local cache in 22.8 seconds.
Verify the inference service
After both workers are ready, test that each can serve inference requests:
# Worker 1
kubectl exec -n dynamo-system <c-frontend-pod> -- \
curl -s http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"Qwen/Qwen2.5-1.5B-Instruct","messages":[{"role":"user","content":"hi"}],"max_tokens":20}' \
| python3 -m json.tool
# Worker 2
kubectl exec -n dynamo-system <c2-frontend-pod> -- \
curl -s http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"Qwen/Qwen2.5-1.5B-Instruct","messages":[{"role":"user","content":"hello"}],"max_tokens":20}' \
| python3 -m json.tool
Both return a normal response:
{
"choices": [
{
"message": {
"content": "Hello! How can I assist you today?",
"role": "assistant"
},
"finish_reason": "stop"
}
],
"model": "Qwen/Qwen2.5-1.5B-Instruct"
}
Results
| MX → MatrixHub download | gRPC streaming | Total model acquisition | |
|---|---|---|---|
| Worker 1 (cache miss) | 38.3 s | 24.2 s | 62.5 s |
| Worker 2 (cache hit) | 0 s | 22.8 s | 22.8 s |
For comparison, without ModelExpress (Blog 1):
| Source | Model acquisition |
|---|---|
| Public Hugging Face | ~10 min 32 s |
| MatrixHub direct | 29 s |
Worker 2 saved the full 38.3-second MatrixHub download. With more workers, the saving multiplies: N workers share one download.
Notes
First-worker overhead. The first worker through ModelExpress (62.5 s) is slower than a direct MatrixHub download (29 s), because the model passes through an extra gRPC streaming hop (~24 s). ModelExpress pays off when multiple workers need the same model.
Streaming throughput. The gRPC streaming stage transfers 3 GB in ~23 seconds (~131 MB/s). The current implementation uses a 32 KB chunk size with about 94,000 iterations. With shared storage (NO_SHARED_STORAGE=0), workers can mount the ModelExpress cache directory directly and skip streaming entirely — model acquisition drops to near zero for cached models.
When to use what.
| Scenario | Recommendation |
|---|---|
| Single worker | MatrixHub direct download (fastest) |
| Multiple workers scaling out | ModelExpress + MatrixHub |
| Shared filesystem available (NFS/Lustre) | ModelExpress shared_storage mode |
| No shared filesystem | ModelExpress NO_SHARED_STORAGE streaming mode |
