Performance Report for 2004
HDF Support for the ESDIS Project
and the EOSDIS Standard Data Format
Mike Folk, NCSA/University of Illinois
November 12, 2004
1Introduction
The year 2004 was the final year of a three-year project to provide HDF support for ESDIS. The work to be done on this project is described in the proposal entitled “Proposal to Renew the Cooperative Agreement Between NASA and NCSA To Provide HDF Support for the ESDIS Project And the EOSDIS Standard Data Format”. The proposal is attached as Appendix A. The following report describes the third year of work on this project.
2Project Goals and Activities
The primary goals of this cooperative agreement for the year beginning January 1, 2004 were:
-
To provide user support for the EOS community in the form of HDF consulting assistance, workshops and training, and documentation.
-
To maintain the HDF4 and HDF5 libraries and utilities and provide quality assurance. Maintenance includes making minor feature changes to address EOSDIS requirements, correcting errors, keeping current the software, test suites, configurations, and documentation, and conducting periodic releases of the software. Quality assurance involves upgrading and extending software testing, reviewing and revising documentation, improving the software development process, and strengthening software development standards.
-
To evolve the HDF5 library and utilities by extending and adapting the HDF5 library to meet evolving functional and high performance computing requirements demanded by EOSDIS, investigating and implementing promising new technologies to address EOSDIS needs, and continuing to develop the HDF5 Viewer/Editor.
-
To address issues of sustainability of the HDF formats and software. This includes identifying those aspects of the project that must continue and seeking mechanisms by which sustainability can be assured.
3Task-by-Task Description of Work
This section lists the tasks from the Program Plan, and the status and accomplishments related to each task.
3.1User Support Activities
Day-to-day user support continued to be a vital activity for the EOSDIS community. Improvements in the quality of the HDF libraries and utilities resulted in more staff time devoted to work on the HDF tutorials, to improving QA procedures, to maintaining the HDF web site and ftp server, and to providing more in-depth support for individual users and vendors.
Subtask
|
Status
| 3.1.1Provide helpdesk support
NCSA's HDF helpdesk provides support to DAAC programmers and analysts and other EOS science software teams by providing users with assistance in using HDF and NCSA tools, in mapping their data to HDF, and in installing, testing, and using the HDF library. The helpdesk helps users troubleshoot their programs, assists them with performance tuning for HDF4 and HDF5 applications, and assists users in making the transition from HDF4 to HDF5. The helpdesk gives assistance to vendors interested in adding HDF support for their products. It also maintains a suite of sample HDF5 files, to help users better understand the format and its capabilities.
|
This ongoing task continues at the required level. Response time continues to be very good. In the first 9 months of 2004, 38% of messages were resolved within four hours and only 3.8% were still open after 2 weeks. The number of helpdesk requests stayed at about the same level as in 2003. 75% of all questions received by HDF helpdesk were HDF5-related questions, comparing to 80% in 2003. The HDF helpdesk continues to work very closely with SIPS, DAACS and vendors whose support of HDF is important to the EOS community, especially with RSI and MathWorks.
During 2004, NCSA also expanded the collection of sample programs and datasets available on the web (http://hdf.ncsa.uiuc.edu/hdf5-files.html). This collection includes example programs and files with datasets of a variety of HDF5 datatypes, HDF and HDF-EOS files converted to HDF5, and HDF5 files converted from DOQ, DEM, GEOTIFF, and ESRI Shapefiles. This provides HDF users and HDF application developers with a substantial number of examples and test data.
| 3.1.2Support HDF-EOS development efforts
1. NCSA will continue to advise on the implementation of HDF-EOS 5, and help support DAACs that are beginning to use HDF5. In addition, NCSA will (a) build HDF-EOS with pre-releases of our library and advise the HDF-EOS team based on its findings, and (b) investigate the possibility of including HDF-EOS support in certain of the NCSA tools.
2. NCSA will also work with ESDIS to determine a strategy for supporting parallel I/O in a way that meets the needs of HDF-EOS users.
|
During this year, NCSA worked closely with HDF-EOS developers, DAACs and SIPS.
1. We periodically regression tested HDF-EOS5 with pre-releases of HDF5 on several platforms.
2. Information exchanges with HDF-EOS developers included these:
-
Discussions and email to diagnose possible memory leaks and other bugs.
-
Extensive discussions about performance issues for HDF-EOS5. (There have been more than 80 emails on this topic so far in 2004.)
-
There are still no clearly defined requirements from the users for parallel I/O support in the HDF-EOS library. The HDF team continues to work with the users and HDF-EOS library developers to determine the HDF-EOS community needs in parallel I/O. We presented an HDF5 Parallel Tutorial on the HDF-EOS VIII workshop, which was well attended and received.
|
3.1.3Conduct information outreach
NCSA will continue to maintain a web site, to publish an email newsletter, to give presentations to interested EOS groups such as DAACs and Working Groups, to participate in EOS-related meetings, and to host visitors from DAACs and other EOS-related projects.
|
The team was very active in this area in 2004:
-
MuQun Yang participated in the Weather Research and Forecasting Model (WRF) Workshop, June 26, 2004, and delivered the paper Performance Study of HDF5-WRF IO modules. Robert E. McGrath and Mike Folk were co-authors.
-
Staff reported on the collaboration with Unidata at the ESTC conference in the talk Merging the NetCDF and HDF5 libraries to Achieve Gains in Performance (ESTC 2004, June).
-
Mike Folk participated in an ESIP Federation meeting in August, in the process covering a number of important issues with colleagues and pushing forward collaborations involving HDF-GEO and OPeNDAP-HDF5-netCDF harmonization.
-
Elena Pourmal presented at several organizations in the UK (May 11-15, 2004):
British Met Office in Exeter, UK
canSAS-IV Workshop for small-angle scatterers at the Rutherford Appleton Laboratory (RAL), Didcot, UK
British Atmospheric Data Center at RAL, Didcot, UK
-
Elena Pourmal presented at the NOBUGS conference at the Paul Scherrer Institute (PSI), Villigen, Switzerland (October 18-24, 2004)
-
Elena Pourmal gave talks and tutorials to BMW Research and Development, Munich, Germany (October 14, 2004).
-
Quincey Koziol gave presentations to the XMS Group at BYU (hydroinformatics) (March 2004).
-
MuQun Yang gave presentations to two organizations in China (February 25-26, 2004):
National Satellite Meteorological Center of China, Beijing
Department of Atmospheric Sciences, Beijing Univ., Beijing
Laboratory for Climate Studies, National Climate Center, Beijing
-
Two extended abstracts will be presented at AMS in January 2005.
MuQun Yang, Robert E. McGrath, and Mike Folk, "HDF5: A High Performance Data Format for Earth Science (submitted)" American Meteorological Society, January, 2005.
MuQun Yang, Robert E. McGrath, and Mike Folk, "Atmospheric Sciences and Climate Applications Using HDF and HDF5 (submitted)" American Meteorological Society, January, 2005.
|
3.1.4Prepare and give tutorials and workshops
A major outreach activity is to prepare and give tutorials and workshops on HDF, and NCSA plays a key role in planning and participating in the annual HDF-EOS Workshop.
|
The HDF5 tutorials (http://hdf.ncsa.uiuc.edu/HDF5/doc/Tutor/) were enhanced in several respects, adding material on Property lists, file drivers, complex datatypes and utilities.
We participated actively in the HDF-EOS Workshop VIII in November 2004. In the workshop we conducted tutorials on the HDF5 and HDF5 Lite libraries and HDF tools, and consulted with users on effectively using HDF and HDF-EOS. We also made a number of presentations related to HDF and the HDF software development process, as well as presenting posters on topics ranging from HDF tools to research on geospatial applications of HDF.
|
Share with your friends: |
