Local inference hardware is expensive. Organizations consolidate workloads onto shared GPUs rather than dedicating silicon per team. Efficiency is real. So is the governance surface that appears only when tenants share infrastructure.
Questions that do not arise in single-tenant setups:
- Who authorized this workload on this hardware?
- Which model version produced this output?
- Could one tenant’s work affect another’s results or state?
- Were the right access controls active at execution time?
In cloud, the provider’s multi-tenancy story answers those — hypervisor isolation, network segmentation, IAM. On local infrastructure, your team is the provider. The answers have to be designed.
Isolation layers
Compute. Shared GPU memory? Shared scheduling? Dedicated GPUs per tenant are simplest to reason about and least efficient. Time-slicing or MPS is more efficient and trusts the isolation the driver and scheduler actually provide.
Model / adapter. When tenants use different fine-tunes or adapters on shared base weights, each request must bind to the approved adapter for that tenant. The wrong adapter is a correctness failure — and a data-handling failure if adapters were trained under different access rules.
Data. Prompts, outputs, and intermediate state (KV cache, activations) from one tenant must not be readable by another. GPU memory needs explicit clearing between workloads or architectural separation that prevents cross-tenant access.
Audit. Inference logs without tenant attribution support aggregate monitoring, not tenant-specific incident response.
Adapters and change control
LoRA weights, prompt vectors, and specialized configs are often the practical unit of customization. In multi-tenant settings they become a governance object:
- Each inference should be attributable to a specific adapter version approved for that tenant. “The model” is not enough when base weights combine with adapters that change behavior.
- Adapters evolve. Historical versions matter when someone asks what produced an output last quarter.
- Loading or swapping an adapter is a deployment change. If it can happen without review, change control has a hole.
Shared infrastructure, enforced boundaries
Tenant boundaries. Hardware allocation, permitted models, permitted adapters, data-handling rules — documented and enforceable, not hallway agreements.
Request-level attribution. Tenant identity, model version, adapter version, hardware identifier on every inference event. That is the base for audit and investigation.
Change control for deployment state. New model, adapter swap, inference config change: treat as a deployment event. In CMMC-scoped environments, changes to systems handling CUI need documented change management.
Reviewable separation evidence. When an auditor asks how tenant A’s data stays out of tenant B’s reach, the answer must be concrete: isolation model, enforcement mechanism, verification procedure — not “we trust the GPU driver.”
Dedicated hardware
Sometimes the governance cost of multi-tenancy exceeds the hardware savings. Dedicated hardware per classification level, program, or tenant removes isolation questions at the cost of utilization.
That is a legitimate architectural decision when data-handling rules make shared risk unacceptable. Make it against the program’s threat model and obligations — not by defaulting into multi-tenancy and hoping informal separation holds.
Shared infrastructure with single-tenant assumptions does not work. Either enforce multi-tenant boundaries, or do not share.