Evaluating a pipeline inspection technology vendor: the questions that actually separate real capability from a demo

Evaluating a pipeline inspection technology vendor is a genuinely difficult procurement problem, precisely because the thing that matters most - whether the system actually works reliably on your network, under your conditions - is exactly what a sales demo is least equipped to show honestly.

Why demos are structurally misleading#

A vendor demo is, almost by definition, built to showcase the system performing well - typically on a curated dataset, chosen because it demonstrates the capability clearly, under conditions favourable to a clean result. This isn't necessarily dishonest, but it is structurally uninformative about how the same system performs on an unfamiliar network with its own terrain, coating history, data quality, and edge cases. An operator evaluating technology based primarily on demo performance is evaluating the vendor's best case, not their own expected case.

The questions that actually matter#

A more rigorous evaluation focuses on a specific set of questions. What exact claims has the vendor tested, and against what independent ground truth - not a general accuracy percentage, but the actual false positive rate (how often flagged findings turn out not to be real issues) and false negative rate (how much real risk goes undetected)? Is that validation work published or otherwise available for independent scrutiny, or does the operator have to simply trust an assertion? Does a human remain firmly in the decision loop, with the system supporting rather than replacing engineering judgement? And critically: is the vendor honest and specific about what the system does not do, or does every capability question get answered with an enthusiastic yes?

Integration is not an afterthought#

A technically strong inspection system that cannot feed its output into an operator's existing risk-management, compliance reporting, and integrity data systems creates a new silo rather than removing manual work - someone still has to manually reconcile the new tool's findings with everything else the operator already tracks. This risk is frequently underweighted in technical evaluations that focus heavily on detection capability and pay less attention to workflow fit. Integration compatibility - what systems and data formats the vendor's output can actually feed - should be assessed early in the evaluation, not discovered as a problem after a technical proof-of-concept has already concluded.

Data governance questions to ask before piloting#

Pipeline network coordinates, inspection findings, and integrity data are commercially sensitive and, in many cases, security-sensitive information. Before any pilot begins, an operator should have clear answers on data residency (where the data is stored and processed), access control (who at the vendor can see the operator's specific data), retention policy (how long data is kept and what happens to it after the engagement ends), and explicitly what the vendor is and is not permitted to do with the data - including whether aggregated or anonymised findings might be used in the vendor's own marketing or product development without specific permission.

A useful evaluation checklist#

Concretely, a rigorous vendor evaluation for pipeline inspection technology should cover: measured false positive and false negative rates against an independent dataset, not a vendor-curated demo; a clear, honest statement of what the technology does not do; a defined integration path into existing risk and compliance systems; explicit data governance terms agreed before any data changes hands; and a pilot structured to generate real, operator-verifiable evidence rather than simply showcase the product. Vendors willing to be evaluated on these terms - rather than only on a polished demo - are signalling a materially different, more credible relationship with their own claims.

Related reading#

This connects directly to what "AI-powered" actually means in inspection software and to why false positive rate, not aggregate accuracy, is the number that matters.