Uncertainty [MGD Sections]

UNFCCC decisions and requirements
IPCC good practice guidance
Relationship to UNFCCC
GHGI coverage, approaches, methods and tiers
Design decisions relevant to national forest monitoring systems
Land cover, land use and stratification
Forest reference emission levels and forest reference levels
Quality assurance and quality control
Guiding principles – Requirements and design decisions
Estimation methods for REDD+ activities
Integration frameworks for estimating emission and removals
Selecting an integration framework
Activity data x emission/removal factor tools
Fully integrated tools
Practical considerations in choosing an integration tool
Guiding principles – Methods and approaches
Remote sensing observations
Coarse resolution optical data
Medium resolution optical data
High resolution optical data
L-band Synthetic aperture radar
C-band and X-band SAR
LIDAR
Global forest cover change datasets
Ground-based observations
National forest inventories
Auxiliary data
Guiding principles – Remote sensing and ground-based observations
Activity data
Methods for estimating activity data
Maps of forest/non-forest, land use, or forest stratification
Detecting areas of change
Additional map products from remote sensing
Estimating uncertainty of area and change in area
Estimating total emissions/removals and its uncertainty
REDD+ requirements and procedures
Reporting forest reference emission levels and forest reference levels
Technical assessment of forest reference emission levels and forest reference levels
Reporting results of REDD+ activities
Technical analysis of the REDD+ annex to the BUR
Additional advice on REDD+ reporting and verification
Guiding Principles – Reporting and verification of emissions and removals
Financial considerations
Country examples – Tier 3 integration
Use of global forest change map data
Relative efficiencies
Developing and using allometric models to estimate biomass

Record Keeping [MGD Sections]

Integration + Estimation [MGD Sections]

Ground Based Observations [MGD Sections]

5.1.5.3   Model-assisted estimators Previous topic Parent topic Child topic Next topic

The essence of the estimators is that the relationship between a variable of interest such as crown cover and predictor variables such as map classes or spectral intensities may be used to predict the variable of interest (e.g. crown cover) for each map unit. The estimate obtained by adding or averaging all the map unit (pixel) predictions is then corrected for estimated bias resulting from systematic prediction error by comparing the reference and map data. Because the relationship is often estimated using a regression model, the estimator is characterized as the model-assisted, generalized regression (GREG) estimation. However, the estimators can be used with a large variety of methods for producing the map predictions, not necessarily involving regression (Sannier et al., 2014). The model assisted general regression estimators (GREG) are provided by Särndal et al., (1992, section 6.5) as,
Equation 12
image34.svg
and,
Equation 13
image35.svg
where N is the number of map units, n is the reference set sample size, yi is the observation for the ith reference set sample unit, image36.svg is the map class, image37.svg, and image38.svg. The first term in Equation 12, image39.svg, is simply the mean of the map unit predictions, image40.svg, for the area of interest, and the second term, image41.svg, is an estimate of bias calculated over the reference set sample units and compensates for systematic classification errors. The primary advantage of the GREG estimators is that they capitalize on the relationship between the reference observations and their corresponding map predictions to reduce the variance of the estimate of the population mean.
For continuous reference observations such as proportion of forest, the GREG estimators typically produce slightly greater precision than the STR estimators. However, when the map and the reference data represent the same classes of a categorical variable (such as activity classes), the STR estimators produce slightly greater precision than GREG estimators (McRoberts et al., 2016).
The decision tree in Figure 12 is intended to help users decide which sampling designs and estimators to use given the nature of the maps and available reference data. Decision point and subsection numbers in the following discussion correspond to the numbered decision points in the decision tree (Figure 12).
Figure 12: Guidance on choosing inference framework for estimation of activity data
Figure12.svg
Considerations at the decision points in the tree are as follows:
Decision Point 1: Do you plan to use a map for estimating activity data?
Although much of the literature on estimating activity data assumes that one or more maps will be used, there is no statistical requirement for doing so. Statistically rigorous and credible estimates can be obtained using only reference data(1). The primary advantages of using maps are (i) that spatially explicit analyses are possible and (ii) that when used with reference data and appropriate statistical estimators, the precision of estimates may be substantially increased, thereby complying with the IPCC good practice guidance that “uncertainties are reduced as far as practicable” (IPCC, 2003; preface) Furthermore decision 4/CP.15 Opens in new window requires Parties to establish NFMS that provide estimates that are transparent, consistent, as far as possible accurate, and that reduce uncertainties, taking into account national capabilities and capacities. An underlying assumption in Figure 12 is that if a map can be acquired, then it will be used.
Decision Point 2: Do you plan to use a change map?
Because by definition activity data pertain to change, maps that enhance the estimation of activity data typically relate to change, although the exact manner in which they do so can vary. Change maps often depict change in land cover in the form of discrete map categories but may also depict proportions of attributes assigned to change categories such as continuous classification schemes that represent proportions of pixel area covered by specific land cover types. For decision advice, an assumption is that a change map will be used (i.e. answering “yes” to Decision Point 2) under two conditions: (1) a change map can be acquired, preferably by comparing images produced on a consistent basis from data gathered on two dates, or else by comparing two compatible maps for two dates, and (2) reference change data in the form of observations of the same locations for dates comparable to the change interval can be acquired.
Decision Point 3: Do you have a reference sample of change observations?
The primary issue is whether a reference sample of change observations obtained using a probability sampling design is already available or if it must be acquired.
Decision Point 4: Select statistical estimators consistent with reference sample design.
In this case, a sample is available and the selection of a statistical estimator and inferential approach must correspond to the sampling design used for selecting the reference sample. For example, if the reference sample was acquired using an STR sampling design, then STR estimators must be used. At this point (and at points 3 and 7), it is assumed that the sample size is considered appropriate to accommodate the guiding principles of IPCC.”
Decision Point 5: Select sampling design and statistical estimators.
The selection of a sampling design and statistical estimator relies to a large extent on the nature of the map and the reference data. If the change map consists of forest/non-forest change/no change predictions, then a general recommendation is to use the map classes as strata and either SRS or SYS designs within strata (Olofsson et al., 2014). The primary advantage of STR sampling is that the precision of within-strata estimates (equivalent to activity data class estimates) can be controlled. In particular, for small or rare activity data classes, the number of observations obtained from overall SRS or SYS sampling can be too small to satisfy precision requirements(2). However, if the reference data are acquired using an SRS design or an NFI-based SYS design, then PSTR estimators may produce considerably greater precision than SRS estimators. In general, to minimize the standard error of the activity data estimate, a stratified estimator is recommended if the map identified in Decision Point 2 depicts change in the form of discrete map categories whereas a model-assisted GREG estimator is recommended if the map depicts change in the form of proportions of map categories (Stehman, 2013; McRoberts et al., 2016).
Decision Point 6: Will you use a reference sample of change observations?
The assumption underlying this and the succeeding decision points is that maps will not be used for estimating activity data. The substantive consequences are that opportunities for increasing the precision of the activity data estimates are not available and that spatial depictions of activity class locations cannot be constructed. For this decision point, the essential issue is whether reference change observations can be acquired; for these analyses, reference change observations consist of differences in observations of forest attributes acquired at the same locations for the two relevant dates. If such reference observations can be acquired, the assumption is that the reference sample of change observations will be used, primarily because the corresponding analyses are less statistically complex and less computationally intensive. If reference change observations cannot be acquired, such as when reference data are acquired from temporary NFI sample plots, separate analyses are required.
Decision Point 7: Do you have two reference samples of the same forest attribute?
The decision point assumes that at least one of the two conditions specified for Decision Point 2) is not satisfied, and therefore that it will not be feasible to use a change map for estimation of activity data. For example, acquisition of two forest attribute maps may be possible, but for some reason the maps cannot be compared to produce a change map. Additionally, acquisition of reference observations may be possible, but for some reason they cannot be acquired for the same spatial locations, perhaps because the reference data are acquired from an NFI that uses temporary ground plot locations. Three scenarios are possible: (a) both reference samples have been previously acquired, (b) one reference sample has been acquired previously and a second is yet to be acquired, and (c) both reference samples are yet to be acquired. For the first and second scenarios, the statistical estimators must be selected to be compatible with the sampling designs used to acquire the existing reference sample or samples. For the second and third scenarios, the assumption that reference change observations cannot be acquired precludes acquiring the two samples at the same locations. For these two scenarios, the combination of sampling design and statistical estimator for samples yet to be acquired can be either the same as, or differ from, a previously acquired sample, or from the other sample yet to be acquired.
Two examples presented in Box 24 and Box 25 illustrate methods for estimation of activity areas, one based on a stratification approach (Cochran, 1977; Olofsson et al., 2013, Olofsson et al 2014) for a map with categorical predictions, and the other based on a model-assisted approach (Särndal et al., 1992; Sannier et al., 2014) for a map with continuous predictions. These examples cover cases that are likely to be encountered in practice and illustrate how to generate unbiased estimates of activity areas with confidence intervals, thus satisfying the IPCC good practice criteria in Penman et al. (2003). As explained in Decision Point 5, the stratified approach illustrated in Example 1 is particularly useful when the strata correspond to activities. The model-assisted approach in Example 2 is more useful when the mapped response variable is continuous and when the relationship between reference data and map data used as auxiliary information can be exploited to increase precision.
An important distinction between the approaches illustrated in the two examples concerns the use of the map data. In the first example, the pixel-level map data are in the form of allocation to discrete classes and are used only to construct strata, to calculate stratum weights, and to reduce the variance of the area estimate relative to the variance of the estimate based only on the reference observations. Of importance, with the stratified estimator for the first example, the within-stratum estimates are based entirely on the reference observations. In the second example, the map data are used as a continuous, segment-level, auxiliary variable. The model-assisted estimator facilitates greater exploitation of the relationship between the segment-level reference proportion of area and the segment-level map proportion of area. Consequently the model-assisted estimator requires compensation for the effects of segment-level model prediction error, but it also exerts a greater influence on the final estimates via a greater reduction in the variance error of the area estimate.

Box 24: A stratified approach to accuracy assessment and area estimation

Data and sampling design
A 30-m x 30-m Landsat-based change map for 2000 to 2010 consisted of two change classes and two non-change classes: (1) deforestation with area of 18,000 ha, (2) forest gain with area of 13,500 ha, (3) stable forest with area of 288,000 ha, and (4) stable non-forest with area of 580,500 ha. Because we have a change map and we intend to use it, the answer is “Yes” to Decision Points 1 and 2. A sample of reference observations did not exist and needed to be collected, so the answer is “No” at Decision Point 3.
For Decision Point 5, because the areas of the map change classes are small, together comprising only 3.5% of the total area, a STR design with the four map classes as strata was selected for acquiring the reference sample to be used for accuracy assessment. Because the map depicts change in the form of discrete map categories with the strata corresponding to activities, stratified estimation is suitable, with strata taking account of likely drivers of change. The sample size must be large enough to yield sufficiently precise estimates of the areas of classes but small enough to be manageable. A sample size of 640 pixels was distributed randomly with 75 pixels to each of the two change classes, 165 pixels to the stable forest class, and 325 pixels to the stable non-forest class following the recommendations in Olofsson et al. (2014).
Estimation
The Landsat pixels randomly selected for the sample reference data were subject to high quality manual classifications. The same underlying Landsat data were used to produce both the map and reference classifications, with the assumption based on three independent assessments that the reference classifications were of greater quality than the map classifications. An error matrix was constructed based on a pixel-by-pixel comparison of the map and reference classifications for the accuracy assessment sample (see Matrix 1), which uses the numerical data provided in the two previous paragraphs).
Matrix 1: Error matrix of sample counts
     
Reference
         
 
Strata
Deforesta-tion
Forest gain
Stable forest
Stable non-forest
Total
Am,h [ha]
wh
Map
Deforesta-tion
66
0
5
4
75
18,000
0.02
Forest gain
0
55
8
12
75
13,500
0.015
Stable forest
1
0
153
11
165
288,000
0.32
Stable non-forest
2
1
9
313
325
580,500
0.645
Total
69
56
175
340
640
900,000
1
The cell entries of the error matrix are all based on the reference sample. The sample-based estimator (statistical formula) for the area proportion, image43.svg, is denoted as image44.svg, where h denotes the row and j denotes the column in the error matrix. The specific form of the estimator depends on the sampling design. For equal probability sampling designs, including SRS and SYS designs, and STR designs for which the strata correspond to the map classes, as is the case for this example,
Equation 14
Eqn005.svg
where image46.svg is the proportion of the total area in s stratum (map class) h, (see the final column in Matrix 1) and nh is nhj summed over j.
Accordingly, the error matrix may be expressed in terms of estimated area proportions, image47.svg (see Matrix 2), rather than in terms of sample counts, nhj (see Matrix 1).
Matrix 2: Error matrix of estimated area proportions
     
Reference
       
 
Strata
Deforesta-tion
Forest gain
Stable forest
Stable non-forest
Total (wh)
Am,h [ha]
Map
Deforesta-tion
0.0176
0
0.0013
0.0011
0.020
18,000
Forest gain
0
0.0110
0.0016
0.0024
0.015
13,500
Stable forest
0.0019
0
0.2967
0.0213
0.320
288,000
Stable non-forest
0.0040
0.0020
0.0179
0.6212
0.645
580,500
Total
0.0235
0.0130
0.3175
0.6460
1
900,000
Once image47.svg is estimated for each element of the error matrix; accuracies, activity areas, and standard errors of estimated areas can be estimated. User’s accuracy,Eqn006.svg, producer’s accuracy,image49.svg, and overall accuracy, image50.svg, where H denotes the number of strata (map classes), are all estimated area proportions.
For this example, the estimate of user’s accuracy is 0.88 for deforestation, 0.73 for forest gain, 0.93 for stable forest, and 0.96 for stable non-forest. The estimate of producer’s accuracy is 0.75 for deforestation, 0.85for forest gain, 0.93for stable forest, and 0.96for stable non-forest. The estimated overall accuracy is 0.95. Note that accuracy measures cannot be estimated using the sample counts in Matrix 1 because the sample is stratified.
The estimated area proportions in Matrix 2 are then used to estimate the area of each reference class. The row totals of the error matrix in Matrix 3 are the map class area proportions (wh) while the column totals are the estimated reference class area proportions.
Using the notation of Equation 6, and adding the subscript j to indicate reference class j,
Equation 15
image51.svg
but because
Equation 16
image52.svg
Equation 15 can be expressed as,
Equation 17
image53.svg
so that from Equation 7
Equation 18
image54.svg
The area for reference class j is estimated as the product of image55.svgand the total area, Atot. For example, the estimated area of deforestation the reference data is image56.svg.
Thus, the mapped area of deforestation image57.svg 18,000 ha is an underestimate by 3,158 ha.
The next step is to estimate a confidence interval for the estimated area of each class. Using the notation of Equation 10 and again adding the subscript to denote reference class j,
Equation 19
image58.svg
Noting from Equation 16 that yhji=0 or yhji=1, Equation 19 can be expressed as,
Equation 20
image59.svg
so that from Equation 8
Equation 21
image60.svg
and standard error,
Equation 22
image61.svg
From Equation 22, so that the standard error for the estimated area of forest loss is
image62.svg.
A 95% confidence interval of the estimated area of forest loss is ±1.96 × 3,142 = ±6,158 ha. Estimates and confidence intervals for all classes are shown in Matrix 3.
Matrix 3: Area estimates, standard errors and upper and lower 95% confidence interval limits.
Strata (j)
image63.svg
[proportion]
image64.svg
[proportion]
image65.svg[ha]
95% confidence interval
Lower [ha]
Upper [ha]
Deforestation
0.0235
0.0035
21,158
15,000
27,315
Forest gain
0.0130
0.0021
11,686
7,930
15,442
Stable forest
0.3175
0.0088
285,770
270,260
301,280
Stable non-forest
0.6460
0.0092
581,386
565,104
597,668
The stratified estimators presented in this section can also be applied if the sampling design is SRS or SYS where the map is used to define the strata (as identified above, this approach is sometimes referred to as “post-stratification” to distinguish the use of the strata for estimation from use of strata in the implementation of the sampling design)(3).

Box 25: A model-assisted approach to accuracy assessment and area estimation

Data and sampling design
In Example 2, a 100,000-km2 region of a tropical country was divided into 20-km x 20-km blocks with each block subdivided into 2-km x 2-km segments. A 30-m x 30-m, forest/non-forest classification was constructed for the entire region for each of 1990, 2000, and 2010 using Landsat imagery and an unsupervised classification algorithm. For each time interval, the map data for the ith segment consisted of the proportion of pixels, image66.svg, whose classifications changed from forest to non-forest. Reference data were acquired for each year by randomly selecting one segment within each block and visually interpreting each pixel within the segment as forest or non-forest using independent Landsat data, aerial photography, and other spatial data. Although both the map and reference data were based on Landsat imagery, the reference data were considered of greater quality because of the use by skilled interpreters with access to additional information. The sample of segments was denoted S, and for each time interval, the reference data for the ith segment consisted of the proportion of pixels, image67.svg, whose visual interpretations changed from forest to non-forest. The Decision Points are the same as in Example 1, but because the map shows change in the form of proportions of map categories (which vary continuously), a model-assisted generalized regression (GREG) estimator is more suitable than the stratified estimator used in Example 1. For a general description of GREG estimators see Section 5.1.5.
Estimation
For each time interval, consistent with the notation used for Equation 12 and Equation 13 above, the map-based estimate of proportion deforestation area was,
Equation 23
image68.svg
where N = 25,000 was the total number of segments in the study area. However, the map estimates are subject to classification errors which introduce bias into the estimation procedure. An adjustment term to compensate for estimated bias is,
Equation 24
image69.svg
where n = 250 is the number of segments in the sample. The adjusted (GREG) estimate is the map estimate with the adjustment term subtracted,
Equation 25
image70.svg
image71.svg
The standard error (SE) of image72.svg is,
Equation 26
image73.svg
where image74.svg and image75.svg
This estimator is based on an assumption of a SRS design. For a SYS design, the variances and standard errors may be over-estimated, yielding conservative estimates of confidence intervals. Estimates of deforestation area for each time interval are shown in Matrix 1 In the statistical literature, these estimators are characterized as the model-assisted GREG estimators even though prediction techniques other than regression may be used and the model may be implicit (Särndal et al., 1992; section 6.5).
Matrix 1: Area estimates, standard errors and upper and lower 95% confidence interval limits.
Interval
image76.svg
[proportion]
image77.svg
[proportion]
image78.svg
[ha]
95% confidence interval
Lower [ha]
Upper [ha]
1990-2000
0.0033
0.0012
33,000
9,480
56,520
2000-2010
0.0011
0.0012
11,000
0*
34,520
1990-2010
0.0044
0.0016
44,000
12,640
75,360
*Because the lower limit was negative, it was reset to 0.

 (1)
As noted in the glossary, reference data are generally collected according to probabilistic sampling design. This means that they can be used alone to produce estimates associated with REDD+ activities, or they can be used in combination with remotely-sensed mapping data to correct for classification bias, and this approach may be most resource-efficient.
 (2)
Good practice is defined by IPCC as applying to inventories that contain neither over- nor under-estimates so far as can be judged, and in which uncertainties are reduced as far as is practicable. Although there is no pre-defined level of precision, this definition aims to maximize precision without introducing bias, given the level of resources reasonably available for GHGI development.
 (3)