How do you compare smart building vendors and system integrators?

The first rule is to compare them separately, because you are judging two different things. A vendor sells a product; an integrator sells the ability to design, install, configure, document and support a working system. Products are tested on functional fit, openness, security and how long they will be supported. Integrators are tested on relevant experience, the people who will actually do the work, their delivery and testing methods, their own security practices and how they will support you after handover. In both cases the criteria, weightings and pass-or-fail conditions should be fixed before proposals arrive, claims should be verified against evidence rather than brochures, and demonstrations should follow scripts built from your own scenarios.

Two roles, two evaluations

A good product badly integrated produces a poor system, and even a strong integrator cannot fully compensate for a closed or insecure product. Some suppliers do both jobs, but the two roles should still be evaluated separately when one company bids for both. It also helps to know whether an integrator is independent or tied to particular manufacturers, because that shapes what it will propose.

Judging the product

For BACnet products, the BACnet Testing Laboratories (BTL) certification programme lists products independently tested against the standard test package. The listing should match the product model and firmware version actually being offered.

Security claims need the same scrutiny. CISA's Secure by Demand guidance for owners of operational technology suggests questions to put to manufacturers: whether default passwords have been eliminated; whether older, insecure protocols are disabled by default; whether the product logs logins and changes; whether multi-factor authentication comes in the baseline version; whether a software bill of materials is available; whether the manufacturer publishes end-of-life dates; and whether the product supports open standards that make replacing or adding products simpler. ISASecure runs certification schemes against parts of the ISA/IEC 62443 series for components, systems and development practices. A certificate is useful evidence, but check exactly what it covers.

Ask for each product's support end date as well. The EU Cyber Resilience Act will make manufacturers state it at the time of purchase from December 2027, but there is no reason to wait.

Judging the integrator

The contract topics in the September 2026 CISA and FBI fact sheet on third-party integrators of industrial control systems make good interview questions, not just contract clauses. Ask each bidder how it controls remote access, manages changes and patches, decides who may work on your systems, and how you would operate and recover without it. The fact sheet is written for critical infrastructure, but the answers reveal an integrator's working maturity more clearly than any policy document.

Running a fair evaluation

Some criteria should be gates, not scores: data export, minimum security requirements and compliance with the specification are pass or fail, and no low price should compensate for failing one. The remaining criteria need weighting before bids are opened, with the balance between quality and price agreed and written down in advance. Prices then have to be normalised so that like is compared with like over the same period, including recurring costs and exclusions.

Demonstrations are most revealing when every bidder works through the same scenarios, ideally with a sample of your own data, and is asked to run the export in front of the panel. References deserve more than a courtesy call: ask referees about problems, how they were handled and whether the named people really did the work. And the reasons for the decision should be recorded, which protects it and makes the next procurement easier.

The usual mistakes

Flawed comparisons fail in familiar ways. They evaluate feature lists and never test how products perform the owner's own tasks. They accept "open" and "secure" as self-descriptions without evidence. They compare first costs while recurring costs differ. And they allow the integrator who wrote the specification to bid against it.