4 Assessing

 ASSESSING WHAT ?

 As a crucial step in developing a strategy, it is desirable to carry out an in-depth assessment of the current status of the statistical system. The assessment is aimed at answering the question “Where are we?” through a full description of the National Statistical System (NSS). It should lead to an understanding of the adequacy of the statistical outputs and the organisation and management of the NSS as a whole.
  
The quality of this assessment will have direct consequences on the quality of the strategic choices to be made. Its results will provide the evidence for the development of appropriate strategies to bring the necessary change. This assessment should therefore be realistic, objective, detached, and critical. It should use best practices and be benchmarked against international standards and frameworks as appropriate. 
  
The assessment will identify gaps that need to be addressed. Most of these gaps are brought about by a mismatch between data demand and supply or between products and results (the effectiveness of the system from the user’s point of view), i.e. gaps in terms of what data are desired beyond what is available/gaps in terms of quality of available data and service provided. The assessment will also highlight the performances of the statistical system (efficiency - capacity/products), and present the main internal strengths of the system and its weaknesses, as well as the external threats that can affect the way in which it evolves and the opportunities that will have to be seized. Success will be evaluated against the starting situation. In fact, for countries which already have an NSDS, the assessing phase is equivalent to the evaluation phase of the previous NSDS.
  
Specifically, it should lead to an understanding of three main dimensions:
  • Assessment of statistical output
  • Assessment of user satisfaction and needs
  • Assessment of statistical NSS capacity  (governance and institutional arrangements, infrastructure, information technology, ...)
 
In practice
Preparation for the overall assessment requires a thorough analysis of existing documentation and review of the findings of any earlier assessments. Very few countries will be starting strategic planning from scratch (some countries are designing their second NSDS) and the purpose will normally be to improve an existing national statistical system.

The assessment will start with a review of policy documents to identify important areas and necessary indicators. These documents are likely to include national development policy frameworks and their reviews (such as poverty reduction and sectoral strategies), as well as sub-regional and international development policy documents (such as the Millennium Declaration and country MDG reports). It might also be appropriate at this stage to read the country policy documents of potential donors.

Analysis of all relevant documents, including existing reports on the statistical situation, will provide a first general picture of the country’s statistical development. Any fruitful analysis of the NSS must be undertaken and owned by the country itself.  
 
 
 
1. ASSESSING STATISTICAL OUTPUT
  
Existing and already planned outputs of the NSS will be assessed. Each key output should be gauged against agreed criteria, for instance using existing frameworks such as DQAF (Data Quality Assessment Frameworks). 
 
The General Data Dissemination System (GDDS) will play a central role at the heart of the NSDS in most countries. A large number of countries in all regions are currently participating in the GDDS and have therefore already carried out many of the steps that are required to develop a strategic approach to develop statistics. In fact, for those already committed to GDDS, this is an imposed framework. Those countries not yet participating would find the GDDS an important early step. 
 
 
In order to assess statistical output, one will consider the following questions:
  •  What statistics are available (inventory), their sources and how quickly they are made available to users (publication and dissemination policies and processes);
  • The quality of statistics and how they are produced (production processes, methods and procedures, use of international standards, constraints and problems) and processed, analysed and archived (IT policies, databases);
  • The possible improvement of the data management system to facilitate efficient data production, i.e. reduce duplication of effort and fill gaps in the system;
  • Are there clear definitions of all data produced?  Are they archived so that they can be accessed by all relevant users and producers throughout the NSS and beyond? Does the system produce the appropriate set of indicators to assess sector performance?
  • Are the existing dissemination policies and plans for statistical production adequate?
 
In practice
The NSDS design team, in close coordination with all other units (see PREPARATION) and with possible support, ideally from national consultants, will carry out the Assessment. Prior analysis of existing information, classification of statistical output and inventory of all the NSS units will facilitate the above exercise.
  
 
2. ASSESSING USER SATISFACTION AND NEEDS
  
Users need disparate statistics for various purposes and their capacity, literacy and sophistication in use of statistics will vary greatly. Some needs may have been suppressed by the lack of available statistics, and potential demand should be considered as well as current demand. User needs cannot be properly met unless these have been properly identified, synthesised, understood and prioritised and unless users are completely aware of the existing statistical production. It is important to emphasise that users invariably have a long list of statistical needs (the so-called shopping list), and every effort should be made to guide them to identify their priorities. The link of the user’s needs to national development plans and/or specific national programs will obviously be crucial when final decisions will be taken as to which statistical outputs will be prioritised. Moreover, user needs and priorities are always changing and tracking these changes requires that consultation and dialogue with users be an ongoing activity. 
 
 
Consultations and discussions with users should aim to answer the following main questions:
  • How do they use statistics in their own operations?
  • How far are required statistics available and how users have been constrained by lack of statistics; does the existing system contribute to produce the appropriate set of indicators to monitor national development goals and meet international requirements (MDG’s for example) and regional commitments?
  • How government and non-government users assess the adequacy of existing statistics in terms of relevance, accuracy, consistency, completeness, timeliness, level of disaggregation (geographic, gender, etc.), presentation or readability of publications, practices with respect to the revision of preliminary data and accessibility to data- meta-data and micro-data?
  • What are their relationships with main producers of statistics and how do they perceive their own role in the development of the NSS?
  • Are the current advocacy strategies sufficient to raise public awareness of the importance of the data produced; does the system provide adequate training to assist users to interpret data, develop indicators and make best use of statistics; does the system  offer tailored or on-demand studies?
  • What are their current and perceived future statistical needs and priorities? Are their needs linked to specific national programmes or development plans?
  • How do they think their needs can best be met in the context of the NSDS?
  • Are the users really aware of the constraints producers face and of the importance of sound methodology in statistical production?
 
 
In practice
Assessment of user needs can be done through various approaches. As with assessment generally, it is likely that the design team will be able to build on existing processes but a benchmark assessment of user needs (quality principles related to official statistics) is recommended for the NSDS. 

One approach is to identify those who are interested in particular data sets according to their preferred areas and arrange contact with these users. The list prepared through the PREPARATION process (see PREPARATION - Identification of stakeholders) is a good starting point in identifying the main data users. Selected institutions from each of the main user groups (see ADVOCATING) should be included in the consultation and discussions held with them, either individually or in small groups, whilst others might be invited to contribute in writing. The process should ensure that policy and decision makers as well as technical staff in user institutions are consulted. 

A second approach to user involvement that has met with success in a number of countries is to organise a country workshop bringing together data compilers, data users, and donor agencies. The workshops deal with specific statistical topics of interest to participants and in addition encourage dialogue among groups of compilers and users. The workshops have proven useful in sensitising participants to the importance of statistics, providing progress reports on data improvements, and discussing new issues. 

The views of the various users will be taken into account and compared with the inventory of official statistics. Information can also be obtained by means of a questionnaire or by actually visiting the stakeholders and interviewing them. The latter approach is usually preferred to avoid low response rates in administering questionnaires.
 
During the NSDS preparation process, the sector committees to be created will constitute an adequate framework for the participation of the users and the identification of their needs. 


 3. ASSESSING THE NSS CAPACITY 
 
This part of the assessment relates to the institutional and organisational framework of the NSS. The main themes to be reviewed will be:
  • Governance:
    o Statistical legislation (adequacy of the national statistical legislation; position and status of statistical agency, including the definition of appropriate levels of autonomy and independance…).
    o Coordination between line ministries and NSO, between NSS units
    o How priorities are set across the NSS
    o Organisational aspects, including how the NSS is managed
    o Adherence to professional ethics and international standards/ existing commitments 
    o Mechanisms for consultations between producers and users of statistics and how they are functioning, for instance committees and other arrangements for user - producer and producer - producer collaboration and consistency between sources, recognising that different models suit particular country situations.
  • Infrastructure and equipment: office buildings, vehicles, office material, etc. 
  • Information Technology: available software, dissemination, archiving tools, etc.
  • Human resources: policies in terms of adequacy (numbers, experience, skills, qualifications) and their management (recruitment and retention, incentive structures, existence of a statistics ‘cadre’); staff development (both initial statistical training and continuous professional development) and knowledge management;
  • Financial resources: is the budget allocated to the statistical system adequate to meet current and projected needs? What is the percentage of technical and financial external resources compared to the national budget?

 

 
In practice
Correcting widespread deficiencies in published statistics requires an understanding of their causes, direct and indirect. Whilst comparisons between products and results help identify shortcomings from the users’ point of view, comparisons between products and capacity will help identify the efforts that have to be made to increase the efficiency and effectiveness of the production process. 
In addition to DQAF, a SWOT analysis can help understand the general environment and highlight strengths, weaknesses (internal), opportunities and threats (external). This will lead to the desired evolution and help shape the vision (see ENVISIONING). 
 

 

Next: