What is AI infrastructure?

To most boards, "AI infrastructure" means a room full of GPUs. That is one possible answer, for one kind of workload. AI infrastructure is really everything an AI application needs to run reliably: computing, storage and networks; the software that manages models and data; the data pipelines that feed them; and the governance that decides how their outputs are used. For buildings and estates it may also include local devices on site and secure connections to operational systems such as the building management system (BMS).

Many organisations asking the question need something much smaller than the GPU room, and some need nothing on site at all. The right place to start is the application, not the hardware.

Three questions about the workload

The first is whether you are training a model or using one. Training a large model from scratch needs concentrated computing and very fast networks for weeks. Running an existing model to answer questions or score data, known as inference, is usually far lighter.

Next comes how quickly the application must respond, and where. A report produced overnight can run anywhere, but a function that has to keep working when the internet link is down must run on site. Where that line falls changes the infrastructure.

Then there is the data the application needs, and where it lives. Moving large raw datasets is costly and sometimes not permitted, so it may be better to bring the processing to the data. Between them, these three answers shape almost everything else.

The layers to plan

Early projects often underestimate the second and last rows and overestimate the first.

Governance is an operating requirement

The US National Institute of Standards and Technology's AI Risk Management Framework (AI RMF 1.0, released in January 2023 and now being revised as part of the White House AI Action Plan) organises AI risk management into four functions: Govern, Map, Measure and Manage. It describes trustworthy AI as valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy-enhanced, and fair with harmful bias managed. The framework is voluntary, but it gives a brief a useful structure: who approves the application, how its performance is measured and what happens when it degrades.

Where outputs influence how a building runs, the boundary between advice and control has to be clear, and the consequence of a wrong answer should set the level of supervision.

The running costs nobody quotes

Models, data sources and software all change, so the budget has to cover monitoring performance, re-evaluating after changes, keeping a known working version to fall back on, and the staff time all of that takes. No compute platform, however capable, makes inaccessible or poorly labelled building data useful.