24 November 2003 Free Open Source idams project Terms of Reference Table of Contents

1. 
   Project outline
   
   1. 
      Background
      
   2. 
      Objective of the Free Open Source IDAMS Project
      
   3. 
      The final product
      
   4. 
      Justification of the project
      
   5. 
      Beneficiaries
      
   6. 
      Main actors
      
   7. 
      Expected impact
      
   8. 
      Resources
      
   9. 
      Methods of collaboration
      
   10. 
       Project evaluation
       

2. 
   Technical considerations
   
   1. 
      Components of the new IDAMS
      
   2. 
      Fundamental implementation principles
      
   3. 
      Operating system(s)
      
   4. 
      Architecture
      
   5. 
      Document types and their management
      
   6. 
      Application modules
      
   7. 
      Control modules
      
   8. 
      Programming language(s), developers’ toolkit(s)
      
   9. 
      Standards to follow
      
   10. 
       Use of existing IDAMS programs
       

3. Project implementation

   1. Project workflow

   2. Project management and organization

   3. Working methods

   4. 
      Project schedule and budget
      

IDAMS (Internationally Developed Data Analysis and Management Software) is a software package for the validation, manipulation and statistical analysis of data, developed by the UNESCO Secretariat in co-operation with experts from various countries. WinIDAMS, the current version of IDAMS for 32-bit Windows, is multilingual with a modern graphical user interface and an on-line Reference Manual. To answer user feedback and new advances in the field of statistics, as well as new computer technologies, it is developed and updated continuously. For more information please visit the www.unesco.org/idams site.

1. 
   Project outline
   

1. 
   Background
   

The IDAMS project has a long history in UNESCO; its results are among the most tangible products of the Organization. The project is part of the Regular Programme and has also received visible support from the Participation Programme as well as from national and international institutions. The project is implemented under the supervision of the CI/INF Division.

2. 
   Objective of the Free Open Source IDAMS Project
   

3. 
   The final product
   

The idea is that the final package should be an easy to use, permanently advancing information processing system designed to assist institutional, corporate and individual users (decision makers, managers, professionals, researchers, teachers, students, developers, etc.) in performing the standard three-phase statistical information analysis procedure which covers: 1) collection, organization, transformation and extraction of data from various sources; 2) exploration and synthesis of information using statistical and other mathematical procedures; 3) creation of ready to use reports that can be presented to all levels and types of information users, published on a web site, shared with other researchers or members of a community, etc..

4. 
   Justification of the project
   

The OpenIDAMS project will answer apparent needs identified in the course of the implementation of the current IDAMS project. Although the WinIDAMS user community is growing spectacularly, it was understood that there are several types of users whose requirements are not covered by the current package. The new IDAMS will considerably enlarge the user community benefiting from this UNESCO project (see 1.5. below). Another important demand is to keep pace with the advancement of information technology, maintaining the underlying technology up-to-date. This will be assured by the new software design and the applied implementation principles (see 2.1.-2.4.). It is obvious that, considering financial and human capacities, the execution of the project outlined in this document would not be possible by UNESCO and its current partners alone. Therefore, the enlargement of the developer capacity is foreseen by building a global statistical software developer community (see 1.6.). The guarantee for the success of this venture is the application of the revolutionary free/open source software development and dissemination methodology.

5. 
   Beneficiaries
   

The community taking advantage of the final product of the new project will be much wider than the one benefiting from the current IDAMS project. We can describe this enlarged community by type of users, independently of the social/economic sector to which they belong to, as follows:

• 
  End-users with general data management and exploration needs in science, education, culture, health, administration, management, etc., academic or non-academic, public or private sectors;
  
• 
  Statisticians, data exploration specialists involved in real-life projects and/or interested in the implementation of their new ideas;
  
• 
  Software developers with general or specific interest, seeking international visibility and recognition;
  
• 
  Software developers involved in the design and development of specific applications;
  
• 
  Future professionals, students and their teachers, interested in statistical information processing or in being involved in the development of advanced software products;
  
• 
  End-users of specific applications based on the toolkit that would result from the project.
  
• 
  The current WinIDAMS user community, which will be assured that the product used by them is maintained and kept up-to-date, and that they can switch to the new toolkit whenever they need to do so.
  

The following list gives an overview of the main types of institutions/organizations, be they public or private, in which users will take advantage of the final and intermediate products of the project:

• 
  Decision makers, managers and professionals in national and local governmental institutions as well as in international organizations;
  
• 
  Students, teachers, researchers and other professionals in universities and other educational institutions;
  
• 
  Managers and professionals in social, cultural and public service institutions and enterprises;
  
• 
  Professionals in research institutes in all fields of science;
  
• 
  Managers and professionals of private and public enterprises in all social and economic sectors (including production, trade, service, administration, banking and finance, etc.);
  
• 
  Professional staff in non-governmental organizations;
  
• 
  Software developer organizations involved in producing statistical information management tools;
  
• 
  Professionals involved in statistical information processing in any other institutions.
  

6. 
   Main actors
   

The new IDAMS will be developed and disseminated using the free/open source software practices and principles which materialize through the free/open source software developer and user community and which have already proved to be really effective and efficient. This approach encourages international solidarity, collaboration and voluntary community work amongst institutions, programmers and users. UNESCO is the originator and will be the motivator, organizer and supervisor of the project, assisted by an international pool of experts and institutions represented by an international governing board and a core network of institutions and individuals (for details see 3.2.(ii) and 3.2.(iii) in this document). Co-operation possibilities with the European Commission, Food and Agriculture Organization of the United Nations (FAO), Free Software Foundation Europe, Francophone Agency, International Organization for Migration (IOM) and other agencies as well as with universities, research institutes and with similar open source projects will be examined concerning software development, utilization (dissemination and use) and financing. Inter-sectoral co-operation in UNESCO with other sectors and departments (Science, Education, Africa Department, etc.) will be strongly encouraged.

7. 
   Expected impact
   

4. 
   Resources
   

The free software community (individuals and institutions worldwide) can be considered the main resource of the project. Considering the obvious demand for the present version of IDAMS, WinIDAMS must also be maintained and supported simultaneously with starting and keeping control of the OpenIDAMS project. This means that in spite of the mobilization of an external capacity larger than ever before, UNESCO must allocate sufficient financial and human resources (see Annex 4) to this promising and voluminous venture, which would have an inspiring effect on all information management tool related activities in UNESCO.

4. 
   Methods of collaboration
   

It is the existence of today’s Internet facilities that makes it possible to launch this project, which needs intensive communication within a broad virtual community. It should be stressed, however, that state-of-the-art services are far from the real “information highway”. If we really do the maximum for this kind of working method, there is good hope that instruments making the operation of virtual teams more efficient will really happen one day (many of them are already advertised but most of them can hardly be used in real-life situations). The lesson learnt during the project preparation is that although real meetings are needed anyway, UNESCO, being the promoter, is supposed to make efforts in providing tools (like conference servers) for facilitating the functioning of standing virtual meetings. This working method could contribute also to the efficiency of many other activities in UNESCO.

10. Project evaluation

2. 
   Technical considerations
   

1. 
   Components of the new IDAMS
   

The new IDAMS package will consist of several components, which are either standalone statistical information processing tools or building blocks for constructing specific applications:

1. 
   Basic and advanced, reasonably comprehensive general purpose data management, data archiving and data exploration package(s) with user-friendly graphical interface(s), as successor(s) of the existing IDAMS, based on (ii) and (iii) (to be distributed under GPL);
   
2. 
   Set(s) of standard application modules implementing data and document management, archiving and exploration functions (to be distributed under LGPL);
   
3. 
   Set(s) of recommended user interface primitives (to be distributed under LGPL);
   
4. 
   Taylor-made packages for specific applications based on (ii) and/or (iii) (to be distributed under GPL and other licenses);
   
5. 
   User and developers’ documentation (to be distributed under FDL).
   

2. 
   Fundamental implementation principles
   

The technological platform selected should guarantee the longevity of the new IDAMS, and assure that IDAMS will be usable on a broad range of machines and operating systems (portability), by a large user community. Project standards must be kept very high regarding quality, ease of use, reliability, accuracy and speed in the solution of problems. The source code should serve as a model of good programming and should be subject to continuous revision and improvement. The project should provide innovative approaches, challenges, state of the art software engineering, visibility and recognition. An important requirement is that the new IDAMS should sensitively follow key information technology developments.

3. 
   Operating system(s)
   

The portability, i.e. compatibility with different machines and operating systems, from mainframes to personal digital assistants and embedded devices, is of maximum importance. In particular Windows, Unix, Linux and their variants should be considered.

4. 
   Architecture
   

There must be a complete separation of the application control and the modules implementing the logic of the statistical, data management and archiving techniques, called application modules. Means for accessing all kinds of data, control information, results, etc., called documents hereafter, will be gathered in a library of document access methods, and should support the client-server architecture. In other words, documents and document access methods are on the server side forming a document server, application modules on the client side. Considering the relation between application and control modules, the former ones constitute the application server, the latter ones the controlling client (which is also a client in relation to the document server). This is a typical 3 Tier Client/Server Architecture. We should note that all servers and client(s) can be on the same computer. An overview of the planned system structure is shown in Annex 1. It should also be added that whenever speaking of and applying client/server architecture, it is understood that special attention is paid to the conformity with the Web Services Architecture concept and standards (W3C, Web Services Activity, http://www.w3.org/2002/ws/). Web Services, at a basic level, can be considered a universal client-server architecture that allows disparate systems to communicate with each other without using proprietary client libraries, effectively eliminating code dependencies between client and server (Web Services Architect, http://www.webservicesarchitect.com/) Web Services are in essence a collection of standards and protocols that allow us to make processing requests to remote systems by speaking a common, non-proprietary language and using common transport protocols.

5. 
   Document types and their management
   

The term “document” embraces all objects that serve for input to and are output from application and control modules. Their types are described below. For assisting the management of documents, we can foresee the use of a data base management system as an intermediate layer through which documents are accessed, in particular in archiving functions. The applied database can be a relational one like MySQL or Postgress, or a non-relational one like CDS/ISIS.

• 
  Data object
  

The data object type should cover old IDAMS datasets and matrices, their new table format representation with extended features as well as new data types represented by XML documents (giving more construct flexibility, e.g. multiple response variables, hierarchical structure in terms of variables and cases). A data object will always include a data array and corresponding metadata. Data objects are inputs to and outputs from application modules, and managed also by corresponding GUI modules, through data access methods. Data access methods return data according to the data view defined in the related control object (case and variable selection, data transformation).

• 
  Control object
  

This object type can be similar to the current IDAMS Setup, with extensions like data view instead of (in addition to) Data Definition. Macros and other advanced control structures (e.g. for…while…, run-time interaction), which will facilitate the formalization of analysis strategies. A control object (control script) is a sequence of control language statements for controlling the execution of application modules. They can be created/modified/deleted by using text editors and by corresponding GUI modules.

• 
  Report object
  

Report objects are output from application modules for displaying or for making hardcopies of results and execution records. They can be texts, tables, graphs, etc. (of standard format) or a composite of them. They can also be formatted reports made by using the Report Manager or coming from external sources.

• 
  Project description object
  

The Project concept would replace and generalize the Application concept (interrelated set of datasets, control scripts, results, temporary workspace, etc.) currently used in WinIDAMS. A full set of documents in a project can be taken for an item in an archive, and the related project type object would contain the additional metadata (in addition to all properties used so far) necessary for describing this item for the archive. Project description objects can be created/modified/deleted by using text editors as well as by corresponding GUI modules.

6. 
   Application modules
   

A version of independent application modules without GUI, to be run from the command line through the Supervisor, would be important for putting the Open source/Free IDAMS on track and also for technical reasons. Highly automated batch processes implemented by Windows .bat files or UNIX shells may also help users while awaiting for the availability of a GUI. They will remain useful for non-interactive use, even if a GUI is available. Nevertheless, application modules should foresee from their very first versions a later inclusion of facilities for the management of two-way information exchange with the GUI wherever applicable. The main groups of application modules are as follows:

• 
  Data Management
  

Includes data entry, validation and correction modules, as well as implementation of standard relational database operations (selection, projection, join, union, intersection, division, and difference) extended with aggregation and data transformation. Data management modules should provide intelligent data entry and presentation facilities, conversion services (e.g. hierarchical-to-table) and gateways to/from external format data, databases, etc. through data access methods.

• 
  Data Exploration
  

Covers basic and advanced statistical and other (fuzzy logic, visualization, etc.) data exploration techniques.

• 
  Report Management
  

Provides facilities for editing and publishing (in PDF, HTML, XML, and other standard document and graphical formats) report and data objects as well as importing external documents. An inherent IDAMS report composer is foreseen for allowing application-tailored presentation of data exploration results.

• 
  Archive Management
  

A full set of documents in a project can be taken for an item in an archive, with the related project description object, which contains additional information needed for describing the project for the archive. While normal data management operations are executed within a project, archive operations (browse, search, retrieve) are typically of inter-project character. Publication of data and projects on Internet can also be envisaged.

7. 
   Control modules
   

The application control, similarly to the application modules, should be modular in order to allow an incremental development of interface and control services, including support for the management of two-way information flow and exchange of computing resources between control and application modules. Control modules should also be able to work as client(s) of application and document access modules.

• 
  Supervisor
  

The Supervisor performs the interpretation of control objects including instructions received directly from the GUI, passes them to application modules, assures run-time connection between application modules and project environment, schedules and executes application modules, handles error messages and return codes.

• 
  Graphical User Interface (GUI)
  

According to the planned architecture, the GUI has obviously the role of a client. The only question is the technology (platform) to be used in the implementation model of this structure.

• 
  One of the alternatives is to use a dedicated client-server model, based on desktop or network services. This approach may result in a pretty efficient product but may require more effort to implement.
  

• 
  Another alternative is that the GUI be implemented on a Web browser based platform. Generalizing the use of the Web platform to the whole package, a software product that can be started and used from a browser, and that can display output using the same browser would be obtained.
  

Both alternatives have advantages and disadvantages. Separation of the control component, in particular separation of the GUI from the other constituents, permits the development of two different competing versions of the user interface without touching other components. Note that even the first alternative can be, and possibly will be based on Web Services (see subdivision 2.4.). The second (browser based) alternative would be practical for advanced non-standard and occasional users.

• 
  project management,
  
• 
  document (data, metadata, report) management,
  
• 
  application control (data manipulation, exploration, archiving and report generator services) management, including dialog with services
  
• 
  interaction management for assisting and keeping track of user actions.
  

• 
  Project Manager
  

The Project Manager will be a means for specifying the full environment of a project (a sociological survey, a series of scientific experiments, etc.):

• 
  language of interface, control scripts, metadata and reports,
  
• 
  location of resources (documents, application modules).
  

Projects are identified by names. The name of the last project used is stored by the system and the settings defined for this project are loaded at the beginning of the following session. These settings can be changed any time during the working session by selecting/creating and activating another project.

8. 
   Programming language(s), developers’ toolkit(s)
   

A mixed language open model would be attractive for several reasons (inclusion of existing modules, mobilizing programmers who know one or another programming language, etc.) but it has also drawbacks in many respects (maintenance, uniform document access methods, etc). In order to achieve the goal of maximum portability and to maintain a consistency, the use of one carefully selected main language is recommended in the reference implementation(s), leaving the door open also for the use of other languages which correspond to structural and functional requirements of the system.

9. 
   Standards to follow
   

• 
  A set of internal IDAMS application standards should be set and implemented in order to guarantee the continuity and the quality of the project. These standards should cover coding and documentation practices in a way that encourages the participation of a diverse set of developers and users that communicates through the Web and is based in many different countries.
  

• 
  The new product should be internationalized (Unicode), supporting different languages and regional settings (dates, decimal notation, currency, etc.). This requirement relates to both the interface(s) and results produced.
  

• 
  Standard file and document formats should be used in data and document management.
  

• 
  Standard graphical user interface features should be applied.
  

• 
  Web services interface standards (like SOAP) should be considered.
  

• 
  Communication standards should be observed.
  

• 
  Metadata language standards should be applied (DDI, RDF, …).
  

• 
  Project implementation and evaluation standards should be observed.
  

10. 
    Use of existing IDAMS programs
    

We should regard existing versions of IDAMS as legacy products. What is important in them is the functionality they provide. The code that is not protected by agreements can be published so that it serves as a guide to developers, or used by people interested in particular techniques. The code that is restricted by agreements would not be published but, together with existing documentation, could be used by the people with access to it (only UNESCO and ITAG members) to provide specifications of the functionality expected of the new products. Note that for the new project these specifications could be more valuable than the legacy source code.

1. Project implementation

1. Project workflow

Implementation of the Free Open Source IDAMS (OpenIDAMS) project will come to pass through the following phases:

1. 
   Project Initiation Phase;
   
2. 
   Project Opening Phase (Publishing officially OpenIDAMS as an Open Source Project);
   
3. 
   Permanent Project Execution Phase (Growth, Stability or Decline).
   

1. 
   Any Free Open Source Software (FOSS) project is launched by a group of initiators who provide an initial product for others to build upon — what we might call the project core, or "kernel." Consequently, the purpose of Project Initiation Phase (Phase 1) is
   

• 
  to establish formally who will be the OpenIDAMS project initiators, i.e. the architects, designers, coders and testers of the OpenIDAMS project "kernel", and also
  
• 
  to produce the “kernel”.
  

The initiation phase covers the following steps:

• 
  Creation and formalization of the OpenIDAMS Technical Steering Committee;
  
• 
  Setting up the initial core network of institutions;
  
• 
  Definition, Design and Refinement of the "kernel";
  
• 
  Implementation of the "kernel".
  

2. 
   The objective of the Project Opening Phase (Phase 2) is to publish officially OpenIDAMS as an Open Source Project. This phase covers the following steps and tasks:
   

• 
  Preparation (selection of a particular license for the product, establishment of adequate communication systems, selection and implementation of a suitable version control system, establishment of an effective involvement/recruitment strategy, establishment of appropriate governance structures and institutional designs, promotion of project credibility,);
  
• 
  Going open, publishing (the OpenIDAMS project will be officially launched as an OS project in this step).
  

3. 
   A successful Permanent Project Execution Phase (Phase 3) is the main objective of the OpenIDAMS initiative. Once a project officially becomes a FOSS, the hope is that it will generate enthusiasm and entice programmers and users in the global Internet community to utilize the existing product and contribute to its further development through programming, testing, or providing documentation.
   

At Phase 3, projects can

• 
  grow with new membership,
  
• 
  remain stable with about the same number of participants as they had before going open, or they can
  
• 
  gradually die from lack of participant interest.
  

Components of Phase 2 — credibility, communication systems, version control, recruitment strategies, and project governance — are predominant factors in enticing new members to the project. Learning will also be one of the motivating forces to attract software developers and users to participate in a FOSS development and to become members of FOSS communities.

2. Project management and organization

This part of the document defines the role of the different partners involved in the initiation phase of the Free Open Source IDAMS (OpenIDAMS) project to produce the "kernel". It must be kept in mind that most of these functions will remain valid also in the project opening and in the permanent project execution phase.

1. 
   the UNESCO Secretariat;
   

1. 
   the OpenIDAMS Technical Steering Committee;
   
2. 
   and a core network of institutions.
   

Besides them, there are two other groups of actors, the role of which is either different from the actors listed under (i)-(iii) or becomes significant only later in the permanent project execution phase:

4. 
   supporting institutions (their role is different);
   
5. 
   the global free/open source developer and user community.
   

1. 
   The UNESCO Secretariat, being the conceiver, will be the organizer and supervisor of the project. All related tasks will be assumed by the Information Society Division of the Communication and Information Sector (CI/INF), and within the Division by IT professionals specialized in software development.
   

2. 
   The OpenIDAMS Technical Steering Committee (OpenIDAMS TSC) in co-operation with the UNESCO Secretariat will be the initiator of the project and will take decisions about the tasks of the project and have authority to make changes to established operational rules.
   

The OpenIDAMS TSC will take the place of the OpenIDAMS Task Force and will enter in function the day after the ITAG 2003 Consultation. The TSC will initially be constituted of the Task Force members and, ex-officio, of the IDAMS Programme Manager.

• 
  To set and validate system specifications and features;
  
• 
  To design the OpenIDAMS "kernel", to monitor and participate in coding, testing and documentation
  

• 
  assure the conceptual integrity of the design and architecture,
  
• 
  make decisions about the design and the technical standards to be used in the project;
  
• 
  provide technical guidelines for coding, testing and documentation;
  

• 
  To define working method details (working rules, among others);
  
• 
  To assure peer-review;
  
• 
  To establish mechanisms for conflict resolution;
  
• 
  To review proposals for inserting new statistical, data mining, data management features;
  
• 
  To do and stimulate fund raising.
  

3. 
   The core network of institutions guided and supervised by the Technical Steering Committee will constitute the core of OpenIDAMS developers.
   

Any institution that wishes to be part of the OpenIDAMS core network of institutions should send a letter of commitment to UNESCO. The letter will then be transmitted to the OpenIDAMS Technical Steering Committee who will decide if the institution is accepted as a core network institution or with another status (like a "Reader" institution which not only tries to use the "kernel" but which also tries to understand how the system works by reading the code).

• 
  Escuela Colombiana de Ingeniería "Julio Garavito" (ECI) , Colombia,
  
• 
  Consorzio interuniversitario per le Applicazioni di Supercalcolo per Università e Ricerca (CASPUR), University of Rome "La Sapienza", Italy,
  
• 
  Cairo University, Faculty of Computers & Information, Egypt,
  
• 
  ALECSO, in co-operation with the Cairo University,
  
• 
  Baku Scientific and Training Centre, Azerbaijan republic,
  
• 
  Russian State Hydrometeorological University, Russian Federation.
  

Although their commitment has not been officialized yet, several African universities and free software associations declared their intention to participate in the project (Cameroon, Democratic Republic of Congo, Guinea, Ivory Coast, Togo, Tunesia).

4. 
   It is important to note that in the third, Permanent Project Execution Phase a fourth category of actors (the most decisive from the perspective of the future of the project), the global free/open source developer and user community will appear on the scene of the project.
   
5. 
   Supporting organizations, such as the United Nations Economic Commission For Africa (UNECA) and the Arab League Educational Cultural and Scientific Organization (ALECSO) (working already actively on setting up the project), give moral, intellectual, organizational and financial support to the project.
   

3. Working methods

Over the past several years, a group of new software development and project management methodologies have converged under the banner of Agile Software Development. These approaches - Extreme Programming, Adaptive Software Development, Feature-Driven Development, Lean Development, Crystal Methods, and others - have helped companies worldwide deliver results in complex problem domains from internal IT projects, to software products, to the exploding range of products containing embedded software. Agile Software Development combines specific software development and project management practices with an explicit organizational perspective that enables teams to deliver software products in volatile business and technology situations (see http://www.adaptivesd.com/learn.html). Agile Project Management and Software Development redefines how software will be delivered in our 21st century economy

1. 
   Operational rules direct the daily decisions and actions of actors involved in an open source project. They specify how new module submissions are managed, how new design strategies are defined, when new versions are released, etc. In these contexts, operational rules define what actors can or cannot do at certain levels of authority within the project community hierarchy.
   

2. 
   Collective-choice rules specify who is eligible to craft or change operational-level rules, and also define the procedures for changing established rules. In some instances, collective-choice rules may be defined by some project leader who has total authority over making and changing operational rules and can do this at any time. Other open source projects may have different sets of collective-choice rules that establish a project oversight body, such as a committee of core developers who, using some kind of voting system, make decisions about the operations of the project and have authority to make changes to established operational rules. From the two approaches the latter one (relying on a project oversight body) is the reasonable choice for the OpenIDAMS Project.
   

3. 
   At the Constitutional-choice level, rules are established to define who is eligible to create or change Collective-choice level rules — a kind of project Constitution, specifying who (or what positions in the project community) possess these rights, and outlining procedures for making changes in collective-choice arrangements.
   

Finally, with respect to open source institutional designs, there are two important areas of operational rules that may be especially important: (1) the system of peer-review and (2) established mechanisms for conflict resolution.

4. 
   Project schedule and budget
   

The project schedule outlined in this document covers

1. 
   the end of the current biennium and
   
2. 
   the next two (2004-2005 and 2006-2007) biennia.
   

According to III.1., the project is partitioned into phases, steps and tasks. The detailed project timetable is given in Annex 3. with

1. 
   description of activities, including expected deliverables and
   
2. 
   starting/ending date.
   

Wherever applicable, at the end of stages that are critical for the next activities, progress and quality evaluation is foreseen.

Annex 2: Acronyms used

Annex 3: Project Timetable

Annex 4: Budget of the OpenIDAMS Project for 2004-2005

Annex 5: Committed Institutions

Annex 6: Authors of this document