Warm caching and the economics of idle-free inference
Why dedicated GPU servers sit idle most of the day and why plain serverless makes users wait for cold starts. A transparent look at NeuroHub's three-tier model lifecycle and what you actually pay for.
Where Does the Cloud GPU Budget Go?
Utilization of dedicated servers under real-world traffic
Off-Peak Idle
Peak usage typically spans a few hours a day. The rest of the time a reserved node sits idle while you keep paying full price.
Cold-Start vs Cash Dilemma
Shutting servers down (plain serverless) means loading model weights from disk to VRAM takes tens of seconds. Users bounce. Keeping everything hot burns cash.
NeuroHub: 3-Tier Lifecycle
Model state lives on three tiers: Hot (VRAM), Warm (host RAM), Dormant (NVMe). While Warm, VRAM is released and weights stay in host memory — the first request is served in seconds with no idle-GPU cost.
How the Three-Tier Orchestrator Works
The orchestrator manages deployment lifecycles and moves a model between tiers based on demand.
Active Inference
The model is loaded into GPU VRAM (llama.cpp / vLLM). Requests are served immediately; time-to-first-token is on the order of tens of milliseconds.
Warm Host-RAM Hold
With no traffic, the deployment sleeps: VRAM is released while the process and weights stay resident in host RAM. The next request ramps up to ready in seconds — no disk reload, no GPU held.
Dormant Storage
Infrequently used weights rest on the node's local NVMe. On a cold start the orchestrator brings the container up while the user sees an estimated wait (ETA) in the interface.
Pre-warming is predictive: when a model sees sustained demand (a few requests within a rolling window), the scheduler proactively starts the deployment into a warm tier so later requests don't wait. Authors can also pin a model "always warm" for a limited period.
What You Pay For on NeuroHub
Instead of renting an idle server, you pay strictly for executed tokens. 1 credit = $1 (USD).
| Hosting Model | Billing | Idle Cost | DevOps |
|---|---|---|---|
| Dedicated GPU VM | Hourly rental | You pay for all idle time | High (CUDA, Docker) |
| Plain serverless | Per request | Cold starts: tens of seconds | Medium (scaling rules) |
| NeuroHub | Strictly per token | No VRAM held | Zero (one API key) |
Tier timings depend on the model, hardware and node configuration. The final inference price is computed by the platform and shown before you run a request.
Experience the Speed First-Hand
Catalog models are deployed on the cluster and ready via interactive chat, playground, and OpenAI-compatible API with transparent pay-as-you-go billing.