2.3.4 Quality assurance and quality control
Though not specifically required by REDD+ decisions, it is useful to implement QA/QC including internal review procedures in the development of estimates. A QA/QC system contributes to the objectives of good practice in inventory development(1), namely to improve transparency, consistency, comparability, completeness, and accuracy of national greenhouse gas inventories. The outcomes of QA/QC processes may result in a reassessment of inventory or category estimates or uncertainties, and to subsequent improvements in the estimates of emissions or removals. For example, the results of the QA/QC process may point to particular variables within the estimation methodology for a certain category that should be the focus of improvement efforts.
QA and QC are defined by IPCC as follows:
- Quality assurance (QA) – a planned system of review procedures conducted by personnel not involved in the inventory development process.
- Quality control (QC) – a system of routine technical activities implemented by the inventory development team to measure and control the quality of the inventory as it is prepared.
Section 5.5.2 of the GPG2003 introduces the idea of a QA/QC plan, which is described in more detail in volume 1, section 6.5 of the 2006GL . A written QA/QC plan is fundamental to a QA/QC system. This plan outlines QA/QC activities performed, the personnel responsible for these activities, and the schedule for completing these activities.
An effective QA/QC plan contains the following four core elements:
A QA/QC coordinator is responsible for implementing the QA/QC plan. In this role, the QA/QC coordinator:
- Clarifies and communicates QA/QC responsibilities
- Develops and maintains QA/QC checklists appropriate to various roles
- Ensures the timely and accurate completion of QA/QC checklists and related activities
- Develops an overall QA/QC timeline and when external reviews will occur
- Manages and delivers documentation of QA/QC activities for documentation and archiving
- Coordinates external reviews of estimates and reports and ensures that comments are incorporated.
- Review procedures
Although general QC procedures are designed to be implemented for all categories and on a routine basis(2), it may not be necessary or possible to check all aspects of input data, parameters and calculations every year. A representative sample of data and calculations from every category may be subjected to general QC procedures each year. In establishing criteria and processes for selecting sample data sets and processes, it is good practice to undertake QC checks on all parts of the system over an appropriate period of time as determined in the QA/QC plan.
When undertaking an internal review of MRV procedures, methodologies, and outputs it is recommended to ensure that(3):
- Sufficient independent expertise is available to conduct the internal review;
- Applied review methods are transparent, rigorous and scientifically sound;
- Review results are reasonable and well- explained;
- Review approach and findings are documented and considered in continuous improvement processes.
Box 8 suggests a checklist for internal verification purposes. When an internal review has been undertaken, it would be useful to report and document the following items:
- information that has been verified internally
- criteria that were used for the selection of verification priorities
- verification approaches, along with relevant data that were collected
- any limitations in the approaches that have been identified
- comparisons that have been performed with independent inventories, datasets, scientific literature, or other studies
- feedback received from external reviewers, with a summary of key comments, and reference to actions taken to address such comments have been addressed
- main conclusions of the verification
- actions taken as a result of the verification process
- any recommendations for inventory improvements or research at national/international level arising from the findings with their prioritization; together with identification of capacity building needs where relevant.
- Documentation and archiving procedures
A documentation and archiving system is needed for any monitoring system. Documentation and archiving allow estimates to be reproduced, safeguards against data and information loss, and supports internal and external verification processes.
An effective documentation and archiving system serves as institutional memory and should store information with enough detail to support new teams or team members in their roles. This will reduce duplication of work and make efficient use of resources. Archiving of material should enable easy access to the documentation and references. Where possible all information should be stored in a central location. Depending on the institutional arrangements established for REDD+ reporting (Chapter 1) the documentation and archiving system may be the same for national GHGI development and REDD+ reporting.
As a guide, the following information should be documented and archived related to REDD+ estimates:
- final reports (i.e. FREL/FRLs, REDD+ technical annexes to the BUR, other material submitted to the UNFCCC)
- activity data, also sources for information, contact persons, other contact information
- emission and removal factors and the reasoning for their choice
- methods used, including spreadsheets, models, instructions how to do the calculations, how to apply the models, reasoning for choices made
- archive by submission or inventory year
- expert judgment (documentation, contact information)
- changes made and recalculations
- results of key category analysis
- uncertainty analysis
- results of QA/QC measures
- improvement plan
- archiving plan
- review findings and responses.
A responsible person (an archive manager or archive coordinator) should be nominated to maintain the documentation and archiving system, and a plan made for updating the documentation and the archive. This plan could include operational elements such as what can be changed or updated, and by whom; when and how updates or changes are made, and who has access to change documentation within the archive, noting any special procedures for archiving of confidential data.
The system need not be expensive or complicated and may be electronic and/or hard copy, and should be located in a specified location, central to the NFMS. There are a number of sources available to assist in developing documentation and archiving systems. The ISO quality management and environmental management standard outlines a useful framework which can be built upon over time.
- Effective use of resources in delivering MRV requirements
Establishing and maintaining an NFMS required significant upfront and ongoing commitment and resources. When well designed, an NFMS can support a number of national and international reporting opportunities. In the context of REDD+, countries and international agencies should consider the most effective use of human and financial resources to deliver associated MRV requirements. This entails considerations such as:
- which pools and activities are likely to be significant in determining the level and trend in emissions and removals
- availability and cost of remote sensing data
- need for pre-processing and associated costs
- assessment of existing data sources and the costs associated with acquiring and processing new sources of data
- existence of ground-based data sets and need for new or supplementary surveys
- availability and suitability of existing tools for integration data and producing required reports
- national support resources, both human capacity and financial to implement, improve and operate the system in the long term.
- level of support and incentive payments and long-term costs
- co-benefits of taking action and opportunity cost of activities foregone
- opportunities for integration with broader land use monitoring systems for GHG inventory purposes, other reporting processes (such as FRA) or improving management of resources that will facilitate the flow of information, the co-ordination of different institutions and the consistency across reporting activities.
Effectiveness of finance requires consideration of long term monitoring costs. The design of a REDD+ policy framework can have a significant impact on the long term operational and improvement costs. REDD+ policies and MRV monitoring functions will co-evolve and therefore MRV processes need to be designed to serve known current and future policy requirements as well as being conditional on technical capabilities, initial development, and operational costs (Böttcher et al., 2009).
Long term improvement and operational costs, as well as short term implementation costs should be considered. Linkages to other permanent national monitoring activities, such as NFIs, for example should be prioritised. There should also be consideration of how to leverage existing data collection platforms and to establish systems to support other national and international reporting opportunities and requirements. The following considerations should therefore be part of the design process and will assist in reducing the risk of a financially unsustainable MRV program, based on sound science:
- MRV functions should be considered as a program, not a project, and will need to continue indefinitely.
- MRV design should be based on policy and reporting needs, country specific circumstances and definitions, financing mechanism, available technology and prospects for results-based payments. This will require close collaboration between policy makers and technical officers.
- evolution of annual budgets through all phases of the programme should be considered from the outset as part of the design and implementation stage to help ensure the program can be adequately funded.
- sources of funding is also a consideration as donors may be more likely to provide funds for design and to support implementation phases, but program funds for improvement will likely fall to countries in the longer term.
- the challenge of securing long term funding for the operational phase of the MRV program should not be underestimated given increasing pressure to show cost-effectiveness.
- integration of data in multipurpose data platforms (one data platform policy) should be considered as a way to seek for cost efficiency and long term sustainability.
The cost effectiveness of an MRV program design will depend on the balance between MRV and other REDD+ costs and the benefits of participating in REDD+ activities as well as the possibilities for using REDD+ as part of a broader land use monitoring platform. The outcome of these considerations will differ significantly from country to country. Cost effectiveness entails saving resources relative to alternative approaches, and not entailing disproportionate additional expenditure given the benefit anticipated.
If MRV monitoring costs are shared between sectors, an integrated monitoring system could have multiple benefits for non-REDD+ land use management (Böttcher et al., 2009). If the monitoring costs associated with co-benefits in other sectors such as optimized land management, improved fire management, agricultural monitoring, and monitoring other environmental values such as biodiversity are included, overall monitoring costs are likely to be lower than separate monitoring for each.
Appendix A gives information on establishment and operational costs based on FCPF budgets.
GFOI has improved international cooperation in the collection, interpretation, and sharing of earth observation information and sees this as an important way to increase cost-effectiveness to assist decision makers as they design their MRV programs.
Box 8: Suggested internal review checklist for REDD+
- Are all data and assumptions used for estimating emissions and removals transparently documented for all selected/important activities, carbon pools and gases?
- Are the methods applied consistent with methods used to calculate emissions and removals from the LULUCF sector reported in GHGIs to the UNFCCC?
- If some REDD+ activities or carbon pools have been omitted, does the report explain why?
- Are all gases required by the IPCC guidance and guidelines included? If not are explanations for the omission provided?
- Are emissions and removals reported as positive and negative terms, respectively?
Comparisons (one or more comparisons should be made)
- Compare REDD+ estimates with independently prepared estimates for the same areas/activities or compare regional sub- sets of national REDD+ estimates with independently prepared estimates for those regions.
- Compare activity data and/or emission estimates used in developing the REDD+ estimates with independent international databases and/or other countries.
- Compare REDD+ estimates with results calculated using another tier methodology, including IPCC tier 1.
- Compare REDD+ estimates with available high- intensity studies and experiments.
- Compare land areas and biomass stocks, and any other stock for which data are available, used in REDD+ global data sets.
Comparisons of uncertainties (one or more comparisons should be made)
- Compare uncertainty estimates with uncertainty reported in the literature.
- Compare uncertainty estimates with those from other countries and the IPCC default values.
- Cross check with available independent direct measurements (which may be available from local forest inventories (if not already used in the estimates), detailed growth measurements and/or measurements made on particular ecosystems for research purposes).
Many data checks can be automated both to allow more time for QC that needs to be done manually. Automated checks include checking ranges on input and output data against previous estimates, and checks against known points of truth. Automated checks often generate a list of suspicious data rather than producing a full pass/fail. This allows manual intervention to check the potential errors. Even with automated systems there should be a degree of random checks to provide confidence that the automated systems are not missing issues, and to improve them if they are.
Source: Adapted from GPG2003, Box 5.7.3 .
Adapted from GPG2003 section 5.7.3 .