Table VI.1 Parameters subjected to quality control - time of observation and code form

*According to specification in the Manual on Codes (WMO-No. 306) 1988 edition, Volume I.

Further, the following observational data should be involved according to the global exchange list in the Manual on the GTS (WMO-No. 386):

• 
  Data Validation
  
• 
  Data Cleaning
  
• 
  QC Monitoring
  

1. 
   Data Validation
   

Validation is a process used to determine if data are inaccurate, incomplete, inconsistent or unreasonable. The process may include completeness checks, plausible value checks, time and internal consistency checks. These processes usually result in flagging, documenting and subsequent checking of suspect records. Validation checks may also involve checking for compliance against applicable standards, rules, and conventions. A key stage in data validation is to identify the causes of the errors detected and to focus on preventing those errors from re-occurring.

2. 
   Data Cleaning (Remedial actions)
   

Data cleaning refers to the process of “fixing” errors in the data that have been identified during the validation process. The term is synonymous with “data cleansing”, although some use data cleansing to encompass both data validation and data cleaning. It is important in the data cleaning process that data is not inadvertently lost, and changes to existing information be carried out very carefully. It is reasonable to retain both the original data and the corrected data side by side in the database so that if mistakes are made in the cleaning process, the original information can be recovered.

• 
  Define and determine error types
  
• 
  Search and identify error instances
  
• 
  Correct the errors
  
• 
  Document error instances and error types
  
• 
  Modify data entry procedures to reduce future errors
  

3. 
   QC Monitoring
   

See Part III of the Guide, section 3.1.3.14 Network performance monitoring that can be applied and implemented by NMS generally for all types of observational data as well as the Appendix VI-1.

For these data quality control includes:
(a) Checking
(i) Gross error check;

(ii) Internal consistency check;

(iii) Time consistency check;

(iv) Accuracy of copying data;

(v) Accuracy of encoding.(b) Remedial action
(i) Before transmission: correct data and message;

(ii) After transmission: correct data and prepare 'COR' message;

inconsistencies occur in a reasonable time by observa­tion sites and NMC.

1. 
   Surface synoptic observations
   

Quality controlQC and monitoring of the observing programme is to be carried out at the NMC for all stations concerned, at least those included in the list of the rRegional bBasic sSynoptic nNetwork at main and intermediate stan­dard times.

COR - and RTD - messages are to be prepared and distributed according to the telecommunication plan:

• 
  For stations from which reports are to be exchanged glob­ally, up to 12 hours after the time of observation;
  

For stations from which reports are to be exchanged regionally, at main and intermediate standard times, up to three hours after the time of observation.

2. 
   Upper-air observations
   

Upper-air observations from stations of the rRegional bBasic sSynoptic nNetwork are processed and encoded either at the station itself or centrally. Quality controlQC and monitoring should be carried out starting at the station and the other centres responsible for communicating or processing or transmission of the data.

• 
  Occurrences of hyper adiabatic vertical gradients of temperature in the free atmosphere;
  

• 
  Comparison of some values in selected points against temporal change since the last observing time and against values obtained by interpolation from data of neighbouring stations;
  

• 
  Checking of the vertical wind profile against inhomogeneities.
  

Obviously erroneous TEMP data should be rejected from distribution either completely or partially, depending on the height where errors occur. In the case of minor errors detected after checking, these TEMP data are manually corrected. In any case, such corrected parts should be distributed nationally marked as "COR"corrected messages. Checked and controlled TEMP data are distributed internationally in accordance with the transmission schedule and are therefore only marked as "RTD"retard messages if data cannot be corrected and distributed in time.

In the second phase, after the data have been scrutinized and controlled checked at the station in the first phase, quality controlQC and correction (if necessary and possible) of all national meteorological data prior to their transmission over the GTS canshould be carried out manually at several centres concerned. , e.g.:

• 
  At data-collecting centres
  
• 
  At steering centres
  
• 
  At the NMC.
  

Monitoring of the process of observations, timing, encoding and preparation for transmission is performed by the NMC in both real time and non-real time, the latter aiming at ensuring improved quality for laternon-real time data. Checks and initiating appropriate remedial actions are made for:

• 
  Missing or delayed messages
  
• 
  Observational errors
  
• 
  Incorrect fFormats errors.
  

6.2.3.2 Automated methods

4. 
   Collecting
   
5. 
   Primary pre-processing
   
6. 
   Real-time control
   
7. 
   Non-real-time control
   
8. 
   Storage
   
9. 
   Retrieval.
   

The principles of organizing an automated quality controlQC of meteorological data depend on the stage of development of methods of algorithms for quality controlQC and automatic data acquisition. The algorithms used by Members for quality controlQC at the station are quite similar. In most cases they are based on physical and/or climatological interdependence and some statistical relations. There is, however, a requirement for further increasing the efficiency of the algorithms and programmes in use and it is recommended that Members exchange information on practical experience with the methods used which could be of great benefit for others.

3. 
   Utilization of general code concepts in support of data quality control