The Historical Development of Test Accuracy Ratios
...And their use in Reporting Uncertainty at Agilent Technologies

Over the years, our service deliverables and supporting infrastructure has developed in different ways and at different rates in our many operating markets around the world. These variations resulted from the autonomy allowed to our service operations to satisfy local expectations and sensitivities but with increasing global standardization more customers, particularly multinationals, are less tolerant of such international inconsistency.
Two critical and related technical challenges of the calibration infrastructure are uncertainty and guardbanding. Both are presently addressed to varying degrees of rigor in our various services and territories. Since, in terms of volume, the majority of equipment certified by Agilent is tested using our proprietary software, an explanation for the qualification of its measurement adequacy is summarized here.

Historical Perspective

In general, the calibration procedures used in our support operations have been those recommended in the published service manuals. These were adopted without further analysis as the assumption was that the product support engineers responsible for the procedure had taken adequate care regarding the need for the standard to have a better specification than the test unit. The basis of such comparison (or the definition of specifications) may not have been consistent since, until 1992 and the publication of the ISO Guide to the Expression of Uncertainty in Measurement, there was no widely accepted methodology. Test software was devised which was simply the automation of these procedures with substitution of modern equipment where appropriate.

The introduction of the US Department of Defense Military Standard 45662A had a profound impact upon our operations, especially in North America. For the first time, this market-driving standard clearly required assessment of test standards and methods used for calibration to ensure that they were adequate. This was defined as being a minimum test accuracy ratio of 4:1 (i.e. the specification of the standard must not exceed one quarter of the unit-under-test), unless the customer was advised. The commercial need to be able to claim compliance with 45662A required the company to reappraise its calibration procedures.

In many cases the original justifications for the test recommendations were not available (particularly from other manufacturers) and so an evaluation program was established under the direction of our (then named) Product Support Division. European service centers, operating in a somewhat more mature environment at that time, were less involved in this project since independent accreditation for some capabilities had already been achieved. However, this was usually for laboratory-style measurement techniques rather than those used for most production work (i.e. systems).

Hence, the focus was for Test Accuracy Ratio (TAR) compliance rather than rigorous assessment of uncertainty. Service center technicians were identified to accomplish this analysis and to share their work. The efficiency of this program was hindered due to the different equipment configurations available at each service center, which meant that many analyses had to be redone to make them applicable. Unfortunately, some of this work was poorly documented particularly where the evaluation simply noted that, by inspection, the TAR was greater than 4 to 1. The reason for such limited evidence is understandable when considering, for example, tests upon a handheld DMM using a precision multifunction calibrator where the TAR was certainly greater that 4 to 1. However, the actual figure (or uncertainty) was not determined to minimize the time and cost-burden of the task.
The publication of the ISO "Guide" mentioned further stimulated this work by establishing clear guidance on the use of a single methodology, meaning that analyses would be more readily accepted in other countries.

In 1992 the project was restructured such that a smaller core team of metrologists would complete the review for all Hewlett-Packard manufactured instruments. However, it was estimated that it would take 70 man-years to complete for just the remaining 2000 HP products and an alternative strategy was adopted -- to categorize equipment into product families, review the tests involved and identify the best instrument(s) in a given subgroup. Uncertainty was then determined for this best representative instrument using the ISO methodology and the TAR calculated by comparing this uncertainty with the product's specification. Similar evaluation for the other instruments in the class having worse specifications was considered unnecessary because the TAR would obviously be larger, assuming the same (type-B only) contributors to uncertainty applied (e.g. test standards and calibration technique).

Reporting

Measurement adequacy for calibrations made using our software is reported on an addendum to the test data. Where a specific uncertainty / TAR assessment has not been made for the tested model, this information originates from the best representative analysis. It takes the form of a statement that the TAR is not less than 4 to 1 unless noted. Where the TAR is less than 4 to 1 and for single-sided specifications where this concept is inappropriate, the actual measurement uncertainty is shown.

The TAR Concept

The traditional metrology community typically expects to see an uncertainty attributed to each measured value. This is reasonable for artifacts having few parameters / testpoints such as the resistance and mass standard calibrations of the laboratory. The measured values of this type of reference is likely to be used to derive correction values and the uncertainty propagated in the error analysis for the next item in the calibration chain.

We believe that the majority of our customers primarily want assurance, in the form of a statement, that their equipment meets its specification since this is commonly used at the production or manufacturing level where it forms the basis of the uncertainty. Since the use of correctional values is rarely appropriate, the calibration data and uncertainty has little significance to such an equipment-user; it is only used by the cal lab to assess conformance with specification. The use of test data and uncertainty in attempt to "improve" upon the quoted specification of a product is not recommended unless crucial to the application and the results of substantial analysis is available (more than simply trending the historical performance, unless the usage and environment are the same as defined for the calibration).

Because of this, our belief is that the cost to re-engineer our existing automated systems in order to present the adequacy of measurement information in the form of traditional uncertainty would be far greater than any real value that customers would gain. The concept behind the definition of the 4 to 1 TAR is an equitable distribution of producer and consumer risk. At 4:1 and 95% confidence there is only 0.8% chance that a product declared as being in-specification (that is, measured value not exceeding the spec. limit) is actually out-of-specification.

Future Intent

In a cost-conscious commercial world, sufficient resources will probably never be available to undertake an academic assessment of all the procedures that have been developed (and generally proved adequate) over many years. A compromise is proposed based on "grandfathering" the analyses and without significant reworking of the procedures themselves. The best solution is to start anew rather than try to patch procedures created many years ago, but the period of transition will probable last several years.

Frequently Asked Questions

What is the uncertainty of measurement associated with the calibrations made by your software?
The uncertainty is reported as a TAR. It is assumed greater than 4 to 1 except where noted. It is found in the "Measurement Adequacy Addendum" (previously named MIL-STD-45662A addendum) to the results printout.
How can a TAR be used to meet the requirement to report measurement uncertainty?
Simply divide the specification of the unit under test by 4.
Aren't many TARs much greater than 4 to 1?
Yes they are, but in some cases early analyses failed to document the actual values. They were just checked-off as meeting the minimum 4-to-1.
How are Tars calculated?
Agilent determines the uncertainty and compares this value with the specification of the unit under test.
Are they just specification compared with specification?
No, Tars are not spec. versus spec. The original concept of TAR was developed by Jerry Hayes of the US Navy in the early 1960s.
What risk is associated with the propagation of Tars through several calibration levels?
The TAR of the last calibration has the most effect. Risk converges to some value of risk for the "Beginning of Period" value. The TAR doesn't propagate like measurement uncertainty.