What should a smart building proof of concept test?

A proof of concept should test the assumptions the full investment depends on that cannot be checked on paper. Is the data good enough? Does the integration work with your actual systems? Can and will the operating team act on the results? Can the solution be secured and supported? Is the benefit large enough to measure? It should be small, time-limited and judged against success criteria agreed before it starts. That means writing down the one to three questions it must answer, designing it to answer them in your real environment, and deciding in advance what result would lead you to proceed, change course or stop.

Demonstration, proof of concept, pilot

The three terms are often used interchangeably, but they answer different questions. A demonstration shows what a product can do, usually in the supplier's environment, and proves very little about your building. A proof of concept tests whether a specific approach works in your environment at small scale, and identifies what would stop it working. A pilot runs a solution in normal operation across a defined part of the estate, to confirm the benefits and operating arrangements before wider rollout.

Projects can jump from demonstration straight to rollout, or run a proof of concept with no criteria, so that it never formally ends. Knowing which of the three you are running is the first safeguard.

Test the assumption that could sink the case

Every business case rests on assumptions. The ones worth testing are those that would most change the decision if they turned out to be wrong, and about which you are least certain. The usual candidates are these:

Not every proof of concept needs to test all of them. Testing too many at once makes the results hard to interpret.

Decide what success and failure look like

Each question needs an agreed definition of success before the work starts. That might be at least an agreed percentage of required points available with correct metadata; alerts reviewed weekly, with fewer than an agreed proportion judged false; or integration running through the production network without exceptions to the security policy. Equally important is the result that would lead you to stop. Without stop criteria, a proof of concept can run on indefinitely.

Your building, not a test bench

A proof of concept on a test bench, or in a supplier's cloud with sample data, does not test your buildings. It should use your systems, your network and your people, and wherever possible include the awkward parts: the older controllers, the floor with poor wireless coverage, the night shift. For new buildings, the RIBA Smart Building Overlay to the Plan of Work advises that the specified technologies be prequalified during technical design, by the master systems integrator, through a rigorous desktop exercise or factory mock-ups by the supply chain. For existing buildings, a proof of concept on site serves the same purpose.

Some answers take a year

Questions about data quality, integration and security can often be answered in weeks. Benefits such as energy savings usually cannot. The RIBA overlay recommends tuning and monitoring a new smart building for at least 15 months after handover, allowing three months of baseline data and twelve months to capture seasonal effects. A short proof of concept should therefore not be expected to prove annual energy savings. If energy is the question, the measurement boundary and baseline should be defined using a recognised method, such as the International Performance Measurement and Verification Protocol (IPMVP), and a longer pilot planned.

The commercial terms need the same clarity as the technical ones. Agree in writing who pays for what, who owns the data and configuration created, what happens to installed devices if you do not proceed, and that taking part does not commit you to buy. If the supplier provides the proof of concept free of charge, it matters all the more that the evaluation criteria remain yours.

The closing report

A proof of concept should close with a short report setting out the questions, the criteria, the results, what was learned and the recommendation to proceed, change or stop. It should also record the effort involved, because, as the theatre operator above found, that is the best evidence for the cost of scaling.