A Note on the Reliability Tests of Estimates from ARMS Data

Agricultural and Resource Economics Review, Oct 2004 by Kim, C S, Hallahan, C, Lindamood, W, Schaible, G, Payne, J

USDA uses the concept of "publish-ability" rather than statistical reliability of an estimate for quality validation of USDA estimates, which is solely based on the sample size and the coefficient of variation (CV). We demonstrate conceptually how the reliability of the sample mean can be tested by estimating the upper and lower bounds of the confidence interval for an unknown population mean using the CV. However, the reliability test for the sample mean can be made only under the normality assumption. USDA multiple-way Agricultural Resource Management Survey (ARMS) estimates are used to illustrate the relative measure of precision for sample-based estimators.

Key Words: ARMS data, coefficient of variation, publish-ability, reliability

The National Agricultural Statistics Service (NASS) and Economic Research Service (ERS) of the U.S. Department of Agriculture (USDA) are striving to improve the availability and the quality of data on crop production practices as well as farm financial management. The Agricultural Resource Management Survey (ARMS)-Phase II is USDA's primary source of information on farm crop production practices for major crops, including corn, soybeans, wheat, grain sorghum, barley, oats, and cotton. This survey provides annual field-level data by crop on irrigation technology and water use, nutrient use and nutrient management practices, crop residue management practices, pesticide use and pest management practices, and crop seed varieties including genetically modified seeds. These data summaries, currently available for soybeans, wheat, and cotton for the period 1996-2000, and corn for the period 1996-2001, are invaluable to decision makers and analysts within government agencies and the public.1

Quality validation of USDA ARMS estimates is based solely on the sample size and the coefficient of variation (CV), which is also called the relative standard error. Some details can be found hi Dubman (2000), Kott (1997, 2001), and in Sommer et al. (1998). According to USDA's general guidelines for statistical reporting standards, no estimate should be suppressed simply because it is deemed statistically unreliable. Nevertheless, the presence of such an estimate in a published table should be noted. In particular, an estimator (mean or proportion) in a data summary table of an agency publication should be marked with an asterisk denoting it as potentially unreliable (in a statistical sense) if either the sample size is less than a fixed number of individuals or if the estimate's CV is greater than some designated limit (USDA, 1993). The designated CV can be set at the agency's discretion for an estimator based on commonly occurring events. For the ARMS-Phase II data, each estimator is identified as having a CV less than or equal to 25%, greater than 25% but less than or equal to 50%, greater than 50% but less than or equal to 100%, or greater than 100%.

The CV is an ideal measure for comparing variation across numerous sets of data expressed in different units, such as corn price per bushel and corn yield per acre. However, the CV is not a very meaningful measure without some assurance that the population mean, μ, lies within a preassigned precision level. For USDA ARMS measures, use of these estimates for policy analyses requires a broader statistically determined measurement of precision.

Therefore, the objectives of this paper are fourfold: first, to inform ARMS data users of the reasons why USDA provides the CV for each estimate; second, to explain conceptually how the CV can be used for testing the reliability of an estimator; third, to address the assumption of normality applied to our reliability tests; and finally, to demonstrate our reliability tests applied to a subset of 2001 ARMSPhase II estimates. While ARMS-Phase II summary data tables can contain estimates for both means and proportions, we concentrate on mean estimates in this paper.

Reliability versus Publish-ability of an ARMS Estimator

For the case of a delete-a-sample jackknife method, Miller (1964) demonstrated that the confidence interval for an estimator approaches the confidence interval for an estimator from the normal distribution as the sample size increases, i.e., as n [arrow right] ∞. However, for a delete-a-group jackknife method, an increase in sample size does not change the number of replications. So, at this time, it remains unclear how much the sample size affects the confidence interval for a jackknife estimator.

An Example from USDA-ARMS Estimates

To explain the reliability of estimates based on USDA ARMS data, we base our analysis on a summarized data table from the multiple-way ARMS tables posted on the USDA-ERS website under the title "Nutrient use by tillage system and irrigation system" associated with corn for all survey states (refer to website given in footnote 1).

The USDA ARMS information is represented in the first two columns of table 1, where the second column represents the sample means (percent of acres treated and pounds per treated acre) for the year 2001, and each estimator is identified with its CV. Estimates are marked based on a CV of less than or equal to 25%, greater than 25% but less than or equal to 50%, and greater than 50% but less than or equal to 100% (see table 1 footnote).