Simulation-based engineering and science

Researchers in the PSN undertake a wide range of research activities in molecular modeling, ranging from quantum methods to mesoscale simulations. The group was generally upbeat about the state of support for their research from French and European funding sources.

Dr. Richard Blake, Acting Director, Computational Science & Engineering Dept. Email: r.j.blake@dl.ac.uk

Dr. Keith Refson, STFC Rutherford Appleton Laboratory, CSED
Email: k.refson@rl.ac.uk

Dr. Mike Ashworth, Advanced Research Computing Group, & Computational Engineering Group; Email: m.ashworth@dl.ac.uk

Dr. Martyn Winn, Team Leader, Computational Biology Team, SED
Email: m.d.winn@dl.ac.uk

Background

The Daresbury Laboratory (DL) began operations in 1962 and officially opened in 1967 as the Daresbury Nuclear Physics Laboratory. Today it is one of the major national laboratories of the UK with a staff of 500. The nuclear legacy today takes the form of the Synchroton Radiation Source (SRS) which is one of the major facilities of the Science & Technology Facilities Council (STFC), and accelerators ALICE and EMMA. A strong core of activities in SBES have evolved from DL’s historical roots and is enabled by a multi-institutional supercomputing consortium, and by HPCx and DL’s active participation in the UK’s Collaborative Computational Project (CCP) model for supporting community-scale software development. Furthermore, SBES activities, most notably an initiative to create the Hartree Centre (a CSE institute), figure prominently in the future of the Daresbury Laboratory.

The WTEC visiting team’s hosts described HPC hardware resources physically at the Daresbury site and also those more broadly in the STFC framework but having DL as the STFC-responsible entity. Located at Daresbury, the HPCx supercomputer is an IBM System P575 high-performance server, maintained by a consortium led by the University of Edinburgh, UoE HPCX Ltd., and funded by the UK’s Engineering and Physical Sciences Research Council (EPSRC).

More recently (2007), the STFC/DL joined a consortium (along with University of Edinburgh and NAG, Ltd.) that runs the High End Computing Terascale Resources (HECToR) at the Edinburgh supercomputer center (EPCC). In the first phase, HECToR featured 60 TFLOPs, but a subsequent phase (2009) will upgrade this to 250 TFLOPs. A third phase is planned for 2011. At the start of operations, the system featured 60 Cray XT4 cabinets connected to 576 TBs of storage with the UNICOS/lc operating system. The next upgrade is the addition of a Cray Black Widow supercomputer. For software, the discussion centered on the UK’s experiences with large-scale efforts in community code development in the CCPs. (CCPs are discussed in greater detail in the first presentation below, and the list of all CCP projects is provided at the end of this report.)

Presentations

The WTEC team’s hosts at DL prepared a most informative lineup of presentations that mapped well to the main themes of our SBES report and that addressed our list of questions. After a brief discussion of the morning’s agenda by our primary host, Prof. Nicholas Harrison, we listened to five presentations (PowerPoint files provided for each presentation), followed by a general discussion. We also had a poster session over the lunch break before departing.

Dr. Blake, Acting Director of the CSED (Computational Science and Engineering Department), presented plans for a new CSE institute, the Hartree Centre (named in honor of the prominent computational chemist). His talk is reported in detail because it provides a helpful guide to the organizational structure of the UK’s government funding bodies (the seven Research Councils of the UK’s funding body, the Office of Science and Innovation), as well as the strategic context for SBES at DL.

• 
  For funding of SBES activities, the three most relevant councils are the Science & Technology Facilities Council (STFC), the Biotechnology & Biological Sciences Research Council (BBSRC), and the Engineering & Physical Sciences Research Council (EPSRC).
  
• 
  The STFC supports large-scale experimental facilities for the UK research community, including neutron sources; lasers/light source science; HPC, CSE, and e-science; engineering instruments; and accelerators; it also funds the facilities for particle physics and astronomy. The STFC is a recent creation formed by the merger of the Council for the Central Laboratory of the Research Councils (CCLRC) and Particle Physics and Astronomy Research Council (PPARC) in 2007 and operates from Swindon, Rutherford, Daresbury, Edinburgh, and Chilton with an annual budget of UK£735 million.
  
• 
  DL benefits from strategic investments that resulted from a 2004 study that supported the creation of a “science and innovation in the campus model” for the Daresbury and Harwell locations. The plans for the Hartree Centre can thus be placed in the context of the 2004 plan. The recent growth of CSE (headcount of 90 full-time employees, operating budget of UK£7 million) would receive the additional impetus of UK£50 million of capital and UK£16 million operating costs and a machine room (10,000 sq. ft. and 10 MW power with UK£10 million machines refreshed on a two-year cycle).
  
• 
  Dr. Blake provided an overview of the CCP (Collaborative Computational Project) model and the history of CCPs. There have been 13 to date, with the CCP1 for the electronic structure of molecules. The 13 CCPs encompass activities of 370 groups and 53 HPC consortia involving 150 groups from the UK and 50 groups from the EU network. The CCP model advances CSE with “flagship” code development; maintenance and distribution of codes; periodic meetings and workshops; facilitating collaborations with overseas visitors; and issuance of regular newsletters. The governance is by the CCP Steering Committee, currently chaired by Prof. Peter Coveney, and membership is composed of the CCP chairs, the director of the European Center for Atomic and Molecular Simulation (CECAM), and international members.
  
• 
  Dr. Blake concluded his presentation with one-slide overviews of the main themes, each covered as subsequent presentations.
  

This was a presentation on the past, present, and future of the computational chemistry (quantum level) capabilities at DL and its integration with the UK and European frameworks for computational research and applications in industrial technology transfer. The presentation of the history of the CCP3 development of CASTEP (plane wave DFT code) and CRYSTAL (periodic solid state code) provided useful context illustrating the importance of the CCPs. CRYSTAL is used by 450 groups worldwide (46 in the UK), and there have been 1530 downloads of the CRYSTAL06 demo. CASTEP is distributed worldwide by Accelrys and had 314 citations in 2007. Thanks to CCP3, the high quality of the software engineering expertise invested in these codes is reflected in their flexibility and modular architecture. New mathematical/numerical methods, parallelism, and most importantly new science (NMR, phonons, hybrid XC functionals, PIMD, GA, E-field response, etc.) can all be readily incorporated into these codes.

An example of large-scale modeling of the simulation of hydrated polypeptide involving 300 atoms + 310 water as a CASTEP PW calculation (ultrasoft potentials) in 8 hours on 512 processors on HECToR will become a “routine calculation.” The talk also had a nice illustration of the complementary synergy between experiments and ab initio modeling: X-ray diffraction, XAS methods, IR and Raman spectroscopy, Neutron spectroscopy: all of these experimental methods provide complete information; the gaps in the structure-property relationships are filled in by ab initio simulations. This is essential to advances in materials research and design of next-generation catalysts.

Prof. Nicholas Harrison: Energy – The Materials Challenge

The presentation focused on the impact of SBES materials research in the energy field, e.g., fuel cells, hydrogen storage, nuclear containment, photovoltaics, novel electronics/spintronics, and solid state battery electrodes. The overarching and unifying theme is that the design space for the new materials in each of these applications is so large that experiments alone are unlikely to lead to the optimal (or even adequate) materials design, and that with recent and expected advances in algorithms and HPC hardware, the SBES approach is entering the “sweet spot” with respect to applicability and relevance for bridging the length and time scales encountered in foundational scientific predictions for structure & composition, thermodynamics of phases, reaction kinetics & dynamics, and electronic structure/correlation. The toolkit of codes—DL-POLY, CASTEP, CRYSTAL, kppw—provided a nice integration with the presentations on software development and CCP. The presentation on SBES in materials design for nuclear (fuels/waste) containment was especially memorable and timely in the context of solving the looming energy challenges in the post-fossil-fuel era. (Note: Prof. Harrison is also on the faculty of Imperial College and was on our Friday 2/29/2008 schedule at that campus.)

This presentation was on continuum scale modeling (CFD) with emphasis on geophysical fluid dynamics (ocean modeling). The presentation also included a discussion of consortia for combustion and aerodynamics. Pending advances in SBES resources and algorithms increase the resolution scale for ocean models. Most notable in terms of new science is the ability to incorporate continental shelves (shelf seas) in the model and the consequential impact on ecological and environmental sciences as we incorporate the most important oceanic habitats into the models. For CFD, in turbulence research such as higher resolution of eddy scales, the highlight is the impact on applications to combustion and aerodynamic simulations. In microscale flows, trends in SBES suggest closing of the gap between molecular-scale and continuum-scale models.

This overview was for the entire network of STFC, including activities at DL and a review of the importance of the CCP (and in this case, CCP4) in advancing the computational biology capabilities of the STFC. This talk also gave insights into the increasing presence of the BBRC in funding SBES activities in the biological sciences. CCP4 on X-ray crystallography revolved around software development for structure solution and refinement. But the STFC also realized the importance of associated developments in workflow: automation of structure determination, e-HPTX (management of workflow), DNA data collection, and PIMS (laboratory information management systems). This experience is especially relevant to our report chapter on “big data” and much of this was funded under the UK’s well known e-Science program. In terms of applications and impact on society, Dr. Winn gave a brief overview of the role of protein structure science in: (1) probing the causes of motor neuron disease (MND); and (2) target informatics in modern drug discovery research, e.g., the targets for cancer research (the Human EGFR family of receptors is the target for the oncology therapeutic Herceptin). The talk also gave a nice illustration of the multiscale approach for linking QM to QM/MM. He concluded his presentation with an overview of the increased scale of activity in computational biology that would be achieved in the context of a fully implemented vision of the Hartree Centre for CSE.

There were three overarching themes that registered with this author. First of all, the UK’s Science and Technology Facilities Council and its network of laboratories and facilities is of great interest as an organizational structure to sustain the deployment of cyber- as well as classical infrastructure. In comparison, it is noted that the United States has the mission-oriented national laboratories (e.g., DOE, NIH, etc.) and some examples in fundamental science (e.g., the National Center for Atmospheric Research, NCAR), but it has nothing on the systematic scale of the STFC. Especially in the context of software development for scientific research, the STFC structure in combination with the Collaborative Computational Projects allows the UK to unify and leverage resources to design, develop, and sustain community codes with a high degree of professional software engineering standards, as exhibited in the projects led by our hosts.

Secondly, our host department, the CSED of the STFC (Daresbury Laboratory and Rutherford Appleton Laboratory), has significant activities in SBES materials research from quantum to continuum scale. These are thoughtfully directed as some of the most promising thrusts to impact the present and future landscape of energy technologies. The presentation on optimal materials design for nuclear containment was an especially impressive illustration of the timeliness of SBES.

Third, the ambitious scope of the plans at Daresbury for the Hartree Centre, an institute for CSE, is an indication of the great strategic value placed on the future role of SBES. If this proposal moves forward, given the track record of achievements, the Hartree Centre is likely to become a magnet for global collaborations, including SBES researchers from the United States.

CCP1: Electronic structure of molecules (P. Knowles)

CCP2: Continuum states of atoms and molecues (E. Armour)

CCP3: Computational studies of surfaces (S. Crampin)

CCP4: Protein crystallography (J. Naismith)

CCP5: Computer simulation of condensed phases (M. Rodger)

CCP6: Molecular quantum dynamics (S. Althorpe)

CCP7: Astronomical spectra (D. Flower)

CCP9: Electronic structure of solids (J. Annett)

CCP11: Biosequences and function (D. Gilbert)

CCP12: High performance computing in engineering (S. Cant)

CCP14: Powder diffraction (J. Cockcroft)

CCPN: NMR in structural biology (E. Laue)

CCPB: Bio-molecular simulation (C. Laughton)

CCPP: Plasma physics CCP (tba)

Site: Technical University of Denmark (DTU)
Department of Chemical and Biochemical Engineering

Søltofts Plads Building 227

2800 Kongens Lyngby, Denmark

http://www.kt.dtu.dk/English.aspx
Date Visited: February 27, 2008
WTEC Attendees: A. Deshmukh (report author), P. Westmoreland
Hosts: Professor Rafiql Gani, Director, CAPEC
Computer-Aided Process Engineering Center
Tel: +45 4525 2882; Fax: +45 45932906
Email: rag@kt.dtu.dk

Erling Stenby, Director, IVC-SEP, Center for Phase Equilibria and Separation Processes (Former Chair of the Engineering Research Council for Denmark)

Jorgen Mollerup, Professor, IVC-SEP

The Chemical Engineering Department at TU Denmark is divided into six main research areas: CAPEC (process engineering), IVC-SEP (petroleum and separation), CHEC (combustion), BIO-ENG (food and bioreactors), Polymer Center (polymer engineering and kinetics), and Aerosol and Reaction Kinetics. Modeling and simulation activities are conducted by most groups, and the department is thinking of establishing a virtual center on product engineering. The WTEC panel members met with representatives of two of the groups from the Chemical Engineering Department: CAPEC and IVC-SEP.

CAPEC (Computer Aided Process-Product Engineering Center; http://www.capec.kt.dtu.dk/) was established in 1997 with the objective of developing and using a systems approach to solve and analyze problems related to Chemical and Biochemical Product-Process Modeling, Simulation, Synthesis, Design, Analysis, and Control/Operation for the Chemical, Petrochemical, Pharmaceutical, Agrochemical, Food, and Biochemical Industries. CAPEC has pioneered model-assisted process development methods that use simulations to identify the process parameters and then use experiments to verify the predictions. Center researchers have produced state-of-the-art models and databases that allow users to correlate and estimate pure component and mixture properties of importance in the chemical, petrochemical-pharmaceutical, and biochemical industries. The researchers have implemented results from the above models and databases to develop phenomena-based and/or parameter-based models for process, product, and operations. CAPEC has developed computer-aided systems for process integration, product-process synthesis & design, hybrid separation, waste reduction, pollution control, batch operations, advanced process control, and process analytical technology.

CAPEC is funded by annual fees from member companies. Each company pays €7000 and has access to CAPEC-developed software exclusively. The software developed under this arrangement is owned by CAPEC. It includes ICAS: Integrated Computer Aided System; CAPEC Database: Database of physical properties, solvents, specialty chemicals, reaction; UNIFAC-Utility: Groups, parameters, interface; Library: Solvers, MoT-based property models (UNIFAC, SRK, SAFT, PC-SAFT, CPA, Elec-UNIQUAC, GC-Flory), MoT-based process models (short-path evaporation, vacuum membrane distillation, pervaporation,); Design/visualization: Synthesis & design tools based on driving forces and phase diagrams, reverse method; CTSM: Continuous time stochastic modeling tool; GoLM: Grid of Linear Models toolbox for data driven modeling. The center sees future growth areas in pharmacology, agrobusiness, food, and aroma sectors.

Professor Gani, the director of CAPEC, is also involved in leading a discipline-wide effort on software integration. Computer-Aided Process Engineering Open Standards, or CAPE-OPEN, is a chemical-process-engineering community-based effort focusing on developing standards to facilitate integration of software and simulation packages. CAPE-OPEM is based on XML specifications, COM, and CORBA standards. This effort was initially funded by the EU. It has now evolved into Global CAPE-OPEN with academics, end user companies, and vendors from all across the world (U.S. researchers had to withdraw out of this effort due to lack of funding). A major setback for this effort has been the lack of support and adoption of open standards by commercial vendors, such as ASPEN.

The IVC-SEP group (Center for Phase Equilibria and Separation Processes; http://www.ivc-sep.kt.dtu.dk/) focuses on petroleum-related research activities. The faculty present at the meeting, Professor Stenby and Professor Mollerup had research interests in phase behavior in petroleum industry, protein purification, and modeling of chromatographic processes. This group is strong in algorithms for the exploration and extraction stages of oil and gas resources. They have significant activities connecting extraction efficiency and sustainability. The group consists of 7 faculty members, 15 PhD students, 6 researchers, and 8 technical assistants. The IVC-SEP center has collaborative research programs with the chemical and biochemical process industries, and petroleum technology and refining industries. The membership fee for industries to participate in the center’s programs is €8000 per year. Although there are some related programs in geophysics and geology, this is the only petroleum engineering research program in Denmark.

• 
  Computational Resources. The research groups mainly use workstations for model building and computations. They have access to supercomputers but do not use them for current studies. The CAPEC group is starting activities on molecular simulations, which might need more significant HPC resources.
  
• 
  Algorithms. The CAPEC director noted that they needed to implement their algorithms on the cutting-edge computational architectures to get the maximum payoff. The matching of algorithms and next-generation computational architecture was the key in scaling up, not necessarily more computational horsepower.
  
• 
  Uncertainty Quantification. The consensus was that there were significant errors in the models developed for chemical and biochemical processes. It was especially important in the bio area, where mode experimentation was necessary in order to develop models that would be suitable for model-assisted product development. Methods needed to be developed that would allow adaptive experimentation based on simulation models that are in turn refined based on partial experimental results.
  
• 
  Academic Environment. The Denmark Technical University (DTU) faculty highlighted the difficulty of attracting young researchers to academia due to the structure of the Danish academic system, where a new researcher is initially hired for three years as an assistant professor, after which an associate professor position may be created in that area by the university. However this position is open for everyone to apply. This creates significant uncertainty for young faculty in the system. The associate professor position is equivalent to a tenured position in U.S. academic world. The full professor positions are limited; the total number is fixed by the Education Ministry and DTU. Most faculty members participate in research centers on a voluntary basis.
  
• 
  Education and Training. The Chemical Engineering faculty noted that their PhD students are getting training in at least two of the following three areas: experiments, computations, and theory. The students at DTU are learning computational skills in the Informatics Department. The WTEC team’s hosts pointed out that the industry wants engineers well-trained in basics, not necessarily specialized in specific areas. They noted that the Danish students are very flexible and are being hired by the member companies in the consortia. Most students sponsored by member companies in centers go to work in the sponsoring company after graduation. The Chemical Engineering Department at DTU has collaboration with a French University (CS students), who come for 4 months to work on domain problems.
  
• 
  Funding Issues. Both CAPEC and IVC-SEP have strong collaborations with industrial partners. 40% of their total funding comes from member companies. Companies do not get anything specific for their membership in centers. Member companies can sponsor specific projects, PhD students, and post-docs. Typically the total cost of a research project for industry is the sum of the project cost plus 20% overhead if DTU owns the intellectual property (IP). The overhead is charged at 120% if the industrial partner intends to keep the ownership of any IP resulting from the research collaboration. DTU does not accept liability for any of the products. DTU has liability insurance for projects.