The authors of the paper below wrote that they were “encouraged to seek publicly available data that could be analysed to provide a comparison with the inter-laboratory trials we undertook.” They don’t say who encouraged them, but the UK Food Standards Agency (FSA) funded the work so we may assume that the encouragement came from them.
Jarvis’s paper is a small part of what this blog has been calling for.
The paper is a third-party analysis of datasets obtained from the Health Protection Agency (HPA) proficiency testing schemes and the Central Science Laboratory (CSL). The data confirm what generations of microbiologists have known: there may be great variability in trying to count bacterial colonies.
How much uncertainty do bacterial counts have? Often more than 10-20%. Sometimes much, much more:
“Hence, at a 95% probability level, an average Cl. perfringens count of 5·0 log10 CFU g−1 in one laboratory would not differ significantly from a count of 0·9 log10 CFU g−1 obtained on the same food sample when examined under different conditions and/or in other laboratories.”
An error, or uncertainty, of over 10,000! Well, the errors aren’t usually so large.
“With the exception of the aerobic colony count, the levels of M(easurement) U(ncertainty) estimated from repeatability and reproducibility data for many routinely used microbiological test procedures are far wider than has been generally recognized. This lack of precision has many ramifications in the practice of food microbiology. Laboratory accreditation, as part of a laboratory’s quality assurance scheme, requires the laboratory both to monitor and to seek to improve the levels of uncertainty associated with its results. Even more importantly, the results of food examinations obtained by microbiological testing are increasingly reviewed for compliance with legal, commercial or other criteria. If estimates of MU associated with the test results is to be taken into account, as required by various EU Reports and Directives (Anon 2003b, 2004; Commission Directive 2004), then the whole question of the way in which microbiological criteria for foods are set and used will need to be reviewed.”
Since this paper, and those in its references, UKAS has become even more obsessive about calculating uncertainty of measurement and witnessing staff performing tests. The organisation is never candid about criticisms that show how shaky its foundations are. Turning up the bureaucracy is its only answer when flaws in its system show up. It doesn’t want the fee-payers to catch on how little accreditation can control. Something outside control by the regulators scares them too.
Recognition of experimental limitations threatens not only the ISO management inspection system, but the EU regulations that the FSA exists to enforce.
That’s why they’ve built an international cartel of accreditation bodies. Government protection is essential because the ISO inspection system cannot compete with reality otherwise.
How much will quality have changed in a decade as labs implement the increased measures which UKAS requires to strengthen the illusion of all factors being under control?