On-prem AI deployment checklist

Data and network paths

List inputs, outputs, sources, users, external services and logs. Decide which outbound connections are allowed and how they are observed.

Workload, models and hardware

Measure expected use, concurrency, latency and memory. Test with representative data; a single benchmark cannot size operations.

Update, observe and recover

Define identity and permissions, model versions, service metrics, backups, restoration and rollback. Record owners and handover procedure.

Questions for acceptance testing

• Does the service respond within the agreed time at expected concurrency?
• Can an unauthorised user reach a source through answers, logs or cache?
• Can a model or index be updated and rolled back with traceability?
• Has restoring a backup been tested, rather than merely creating backups?

Record who runs each check and what evidence they retain. Acceptance is not just the development team showing a demo: the operating team should be able to repeat the steps. Adjust the checklist to the workflow’s criticality; some questions call for network or security specialists.

1. Classify the data and deployment boundary

List documents, recordings, prompts, outputs and operational logs that the system will handle. Identify owner, sensitivity, retention and the people allowed to access each. State whether the system will run in a customer data centre, a segmented network, an edge device or a hybrid arrangement; “on-prem” alone does not specify its boundary. Mark every service that may still be external, including updates, telemetry and support access.

2. Size from the workflow

Record expected concurrent users, burst periods, context length, document volume, latency objectives and failure tolerance. Test a representative model and retrieval pipeline with those inputs. Capture memory at start and under sustained load, queue depth and tail latency, not just a single demo response. Repeat when quantisation or model version changes; either may alter quality as well as capacity.

3. Decide who can reach what

Draw permitted network flows between clients, identity provider, inference service, search index, storage and monitoring. Name ports, protocols, outbound dependencies and administrator access. Specify service and user roles, credential rotation and revocation. For a document assistant, test that a restricted source cannot leak through a cited answer, cache or log. Have security and network owners confirm assumptions rather than treating an architectural sketch as approval.

4. Plan updates and recovery before release

Version the model, configuration and index-building process. Agree how a candidate version is evaluated and promoted, how to roll back both model and index, and who can authorise the change. Back up data and configuration, then restore a sample in a separate environment. Record recovery time observed for that sample and what remains untested. A backup file by itself is not a recovery plan.

5. Make operation transferable

Monitor health, inference latency, retrieval failures, storage, queues and access-related events without logging sensitive content by default. Write who receives alerts, handles incidents and maintains dependencies. The handover should include a system map, runbook, known limits, test set and ownership of future changes. Ask the operating team to repeat a deploy, rollback and recovery exercise before declaring the service ready.

This checklist supports the on-prem deployment service and the GPU capacity guide. Its acceptance thresholds must come from the actual workflow; none of the checks above constitutes a completed client test.