Simulation-based engineering and science

Modeling research for cell design is aimed at improvements in (1) cost, (2) power density, (3) longer durability, (4) cold start. Nissan researchers are looking at heat and mass transfer under dry and wet conditions; five types of transport are O₂, H₂, heat, electrons, and water. Numerical tools are used as thinking tools, not design tools. Numerical techniques include first-principle, MD, Monte-Carlo, and CFD. The key modeling target is MEA, but the present MEA model is not good enough to meet cost-reduction targets. The bottleneck is meso-nanoscopic-level modeling. The models, mostly 1D or 2D continuum models, range from component to MEA to cell in size. They believe they don’t need more computer power, rather more “thinking power.”

The Nissan model of the catalyst layer is derived from that of M. Eikerling, J. Electrochem. Soc. 153. Oxygen diffusivity vs. water saturation percent is based on a model by C.Y. Wang.

An important simulation problem is stack modeling under cold-start conditions. Thermal and water management with transient state is very important for cold-start. Nissan uses a non-isothermal cold-start model. For example, power density is affected by O₂ and H₂O concentration. A successful model must include frost formation

Nissan’s membrane material is Nafion. Goals are low cost and compactness, high performance, and high durability; proton conductivity is a limiting factor. Understanding of the mass transport mechanism in the PEM fuel cell is the key for a breakthrough. Issues in mass transport include: H+ conductivity in membrane, H₂O permeability in membrane, H+ conductivity in channels, volume fraction of channels, H₂O diffusivity in channels, H+ concentration, and H+ diffusivity. Nissan’s methods for mass transport are pore model MD for short-range diffusion and dissipative particle dynamics (DPD) for long-range diffusion. The water channel in the Nafion membrane includes some tightly bound water, some loosely bound, and some mobile. Problem size: 10 Å cubed box, 100K’s particles. Solved on workstation. Bigger box might help, but the problem is knowledge-limited. Charge transport model: proton diffusivity estimated by MD, including proton hopping model. The model consists of dimensional parameters, angle and distance between H₂O and H₃O+, fast diffusion and slow diffusion based on different mobility states of water. Challenges of modeling include incorporating longer-range movement using coarse-grained MD and improved DPD.

The primary issues in this area are cost and durability. The principal goal is reduction of Pt usage or substitution of Pt by a cheaper catalyst. Pt durability problems include Pt dissolution from electrode into membrane and corrosion of Pt by carbon. Nissan researchers are investigating these processes experimentally, but information on the chemical reactions is limited. They need to know the reaction path of the Pt electrode reaction, dissolution process of Pt, etc. Computational chemistry may be helpful here but may not be helpful for developing new catalysts. Challenges include the electric potential effect on electrode reactions and the complex structure of the catalyst surface. There are multiple elementary reactions and reaction pathways—at least 33 elementary reactions at one catalyst. Researchers tried to use the Earth Simulator for this, but it did not have enough computer power. Their plan for a computational chemistry approach includes: (1) calculate performance of catalysts on simple electrode reaction and compare results with experimental results, (2) apply computational chemistry to a more complex reaction (cathode reaction.), and (3) try to predict performance of unknown catalysts, including non Pt.

The WTEC panel’s hosts noted that their ideal candidate for a research position would be trained in computer science because fuel cell science can be learned on the job. Their ability to use simulation is limited by lack of fundamental knowledge. They need to work very closely with experimenters to cross-check results. Verification and validation are primarily by comparison with experiment. Future computation will be in computational chemistry. They don’t see any other areas for major investment at this time.

The Nissan fuel-cell research group is mostly doing small simulations on workstations. Modeling is mainly limited by lack of knowledge, not by lack of computer power. Their main target is the membrane-electrode assembly. They are planning a substantial future project: cost reduction and lifetime extension of the Pt catalyst, and substitution of Pt by a cheaper catalyst. They are considering an application of computational chemistry. Models would include rate equations, molecular dynamics, Monte Carlo transport, and dissipative particle dynamics. They plan to collaborate with an outside party to tackle a series of problems: (1) calculate performance of catalysts in a simple electrode reaction and compare results with experimental results, (2) apply computational chemistry to the more complex cathode reaction, and (3) try to predict performance of alternative catalysts, including non-Pt.

Site: Peking University Center for Computational Science and Engineering

Beijing 100871, P.R. China
Date: December 4, 2007
WTEC attendees: M. Head-Gordon (report author), S. Glotzer, S. Kim, J. Warren, P. Westmoreland
Hosts: Prof. Pingwen Zhang, School of Mathematical Sciences
Email: pzhang@pku.edu.cn

Prof. Wenjian Liu, College of Chemistry and Molecular Engineering

Prof. Luhua Lai, College of Chemistry and Molecular Engineering

Prof. Jingchu Luo, Center for Bioinformatics

Prof. Shaoqiang Tang, College of Engineering

Prof. Jingping Wang, College of Engineering

The Center for Computational Science and Engineering was established in 2001 by Peking University to provide a platform for interdisciplinary research related to large-scale scientific computing, the training of students in computational science and engineering, and the provision of high-performance computing resources to university research groups. At present, 29 faculty members have appointments in the center, in addition to their regular departmental appointments. The university gives the center an annual admission quota of 6–8 graduate students; they join the research groups of the affiliated faculty, who also supervise students admitted by their home departments.

The WTEC team heard a series of short presentations on the research of several faculty members who are participants in the center. A short summary of each is given below to provide some perspective on the range of research currently performed at the center.

• 
  Computational biology (Prof. Luhau Lai): In addition to her center affiliation, Prof. Lai is also a member of the Center for Theoretical Biology, which has the study of biological networks as its central theme Researchers in her group are active in structure-based drug design, in which they develop their own publicly distributed codes. Prof. Lai has also started computational modeling of the aggregation dynamics of amyloid fibrils, using both atomistic models and statistical potentials. Her research group has collaborations with a software company, but it has no direct links with the pharmaceutical industry at present.
  
• 
  Computational fluid dynamics in aerospace research (Prof. Jianping Wang): Prof. Wang had recently returned to China after more than 2 decades in Japan. He focuses on using CFD simulations (with experimental validation) to increase performance and usage of aerospace components, and additionally to shorten the development cycle and reduce risk. Within CFD, he has developed his own CFD approach, the “finite spectral method,” which he is developing into a multiscale approach.
  
• 
  Computational quantum chemistry (Prof. Wenjian Liu): Prof. Liu’s group has developed the Beijing Density Functional program, which treats the electronic structure of heavy elements where relativistic effects are important. Present research efforts are aimed at unifying the treatment of relativistic and non-relativistic electronic structure, with associated opportunities for developing new methods and algorithms.
  
• 
  Bioinformatics (Prof. Jingchu Luo): Prof. Luo summarized recent research on the rice genome.
  

The WTEC team was given a tour of the computer center by the system administrator, Mr. Fan Chun, who also discussed its operation with the team. The center houses a teraflop class machine, based on 128 nodes, each with 2 3.2 GHz Intel Xeon processors, and a total of 8 TB of storage. The interconnect is InfiniBand, and the machine was installed in 2004. It provides computing resources for about 20 groups at the university. The machine was purchased with university funds, and costs for administering the machine are also borne by the university. Users are charged 0.6 RMB per CPU hour to cover just the electricity costs of running the machine. There is a need for more energy-efficient computing, using multicore chips, to reduce this cost. The center provides basic commercial software tools, such as Intel compilers and IMSL software libraries, but otherwise users wishing to employ commercial application codes must purchase licenses themselves. At present a significant fraction of the jobs run on the cluster are serial calculations (perhaps 20–30% of capacity), while typical parallel jobs use moderate numbers of processors (8–16). The center is planning to offer short classes on parallel programming via MPI soon, as an alternative to full semester classes in academic departments.

Very little question and discussion time was available, due to the WTEC team’s tight schedule. However, in the course of the short presentations, a number of interesting issues were briefly raised:

• 
  In the center, there is an emphasis on programming by students, so that they learn to be capable of building their own computational tools and modifying existing ones. Existing programs may be used in cases where existing methods are adequate, while new ideas are the best basis for the construction of new programs. However for training, some groups do make sure that students go through the process of “reinventing the wheel” by making their own programs.
  
• 
  There is a cross-cutting issue of increasing computer power leading to a research focus on complex realistic systems, treated by very accurate simulation methods with the by-product of possible loss of physical insight. Several groups discussed this issue.
  
• 
  Prof. Liu discussed the overall progress of computational chemistry in China, stating that while most activity has traditionally been in applications of established tools, there is now a strong young community that appreciates the importance of developing new theory, algorithms, and software. Beyond Peking University itself, he mentioned specifically several strong groups in quantum dynamics, and the activity of the Nanjing and Xiamen groups in quantum chemistry.
  
• 
  Peking University’s Center for Computational Science and Engineering has a strong focus on international workshops and also on summer schools. Its staff members are additionally establishing a strong network of international collaborations, exemplified by a joint center of excellence in quantitative biomedical research with University of California, San Francisco, and computational quantum chemistry tie-ins that are planned with IACS (Kolkata, India), and the Royal Institute (Stockholm).
  

Tsukuba Central 2, Umezono 1-1-1

Tsukuba 305-8568 Japan

http://unit.aist.go.jp/rics/index-e.html

Dr. Tamio Ikeshoji, Director, RICS

Prof. Kazuo Kitaura, Kyoto Univ. and Prime Senior Researcher, RICS

Dr. Dmitri Fedorov, Senior Researcher, AIST

Dr. Seiji Tsuzuki, Senior Researcher, AIST

Dr. Eiji Tsuchida, Researcher, AIST

The Research Institute for Computational Sciences (RICS) is one of approximately 20 institutes that form part of AIST, which was founded in 2001 as part of a major reorganization of Japanese government research laboratories. The institutes are long-term entities that coexist with roughly 30 research centers that have fixed 7-year lifetimes. RICS currently has 29 permanent staff and 13 postdoctoral researchers, who are divided into 5 major groups that focus on quantum modeling, particle modeling, first principles simulations, fundamental analysis, and finally, other topics. Their focus is on simulation at the microscopic rather than the macroscopic level.

During a 3-hour morning meeting, the WTEC visiting panel heard an overview of RICS research from the Director, Dr. Ikeshoji, followed by technical presentations from Drs. Fedorov, Tsuzuki, and Tsuchida. Summaries of these presentations are given below.

RICS researchers have 3 primary missions: (1) development of computational methodologies and programs, (2) their applications to real systems (biological and nanoscale primarily), and (3) collaborations both inside and outside AIST. As a result, a significant number of programs are developed within RICS, including for the FMO method described by Fedorov below, FEMTEK described below by Tsuchida, and other codes for classical molecular dynamics, first principles molecular dynamics, and quantum simulation of materials. RICS researchers are participants in significant national research projects such as the Next Generation Supercomputing Project, including the fragment molecular orbital (FMO) project described below by Fedorov, a hydrogen storage initiative, and two nanoelectronics projects. Materials applications include calculations on proton transfer in fuel cells, aerosol deposition, and fracture and fatigue using molecular dynamics.

Dr. Fedorov described the basics of the FMO approach, as well as the latest FMO developments (Fedorov and Kitaura 2007). FMO calculations on a large biomolecule are a simplified divide-and-conquer approach in which calculations are performed on individual residues in the Coulomb field of others, followed by corrections based on calculations on pairs of residues (and, in principle, triples of residues, if needed). The approach is accurate and very efficient, as long as the division of the system is wisely chosen. FMO calculations are manifestly parallel, although there are challenges associated with load balancing. The FMO method was recognized with the Best Technical Paper Award at Supercomputing 05 (the 2005 meeting of the International Conference for High-Performance Computing, Networking, Storage, and Analysis). FMO methods are currently interfaced to the GAMESS program (which is a public domain U.S. code).

Dr. Tsuzuki described high-level electronic structure calculations for the interactions of aromatic groups with themselves and also with CH groups (Tsuzuki and Uchimaru 2006). Additionally, he discussed similar calculations of the interactions between the component ions of ionic liquids. These calculations are then used as inputs for the development of force fields that may be used for dynamics.

Dr. Tsuchida described the development of a new finite element method (FEM) for density functional theory (DFT), based on tensor product cubic polynomial basis functions (Tsuchida 2007). There are 4 key advances that make this approach promising: (1) the development of adaptive curvilinear coordinates that cause the mesh to deform as the atoms in the molecule move, (2) a fast iterative Poisson solver that uses a multigrid preconditioner, (3) a quasi-Newton solver for the self-consistent field problem, combined with a non-diagonal preconditioner, and (4) a new linear scaling approach based on orbital minimization with additional constraints to ensure linear dependence. The cross-overs for this method appear very competitive with rival methods. The resulting code (FEMTEK) is currently being applied to a first-principles simulation of the dynamics of liquid ethanol.

The main supercomputer is the “Supercluster,” which at the moment contains a 2144 CPU AMD Opteron cluster that performs at about 6 TFlops, and 2 smaller components (528 processor 1.3 GHz Itanium2, and 536 processor 3.06 GHz Intel Xeon). In 2008 RICS will purchase about 3 more small clusters consisting of about 200 CPUs, each performing in the teraflop range.

During the talks there was good opportunity for discussion of both scientific issues directly related to the content of the talks, and also more general issues surrounding simulation-based engineering and science. Some of the additional issues that came up included the following:

• 
  Potential postdocs with appropriate training in the development of algorithms and programs are hard to recruit. The main cause of this issue is that too many graduate students are being trained primarily to run existing codes to solve applied problems rather than learning the skills necessary to create new applications.
  
• 
  The development of codes within RICS is not always directed at achieving performance that exceeds rival public domain or commercial codes, but is also done to permit quick prototyping of new ideas and new functionality. In other cases, such as the tie-ins to the next-generation supercomputer project, very high (parallel) performance is the target.
  
• 
  National and international collaborations of RICS researchers are increasing significantly. This is important partly as a mechanism to raise additional research funds, and collaboration is increasingly also seen as part of the mission of the institute. There is also an active program of annual workshops on computational science, some international or regional in scope and others directly primarily at Japanese participants.
  

Fedorov, D.G., and K. Kitaura. 2007. Extending the power of quantum chemistry to large systems with the fragment molecular orbital method. J. Phys. Chem. A 111:6904-6914.

Tsuchida, E. 2007. Augmented orbital minimization method for linear scaling electronic structure calculations. J. Phys. Soc. Jap. 76(3) :034708-1–034708-7.

Tsuzuki, S., and T. Uchimaru. 2006. Magnitude and physical origin of intermolecular interactions of aromatic molecules: Recent progress of computational studies. Curr. Org. Chem. 10(7):745-762.

Site: RIKEN – The Institute of Physical and Chemical Research
Advanced Center for Computing and Communication (ACCC)

2-1, Hirosawa, Wako-shi

Saitama, 351-0198, Japan

http://www.riken.jp/engn/index.html

http://accc.riken.jp/E/index_e.html
Date Visited: December 3, 2007
WTEC Attendees: G. Karniadakis (report author), P. Cummings, L. Petzold, T. Arsenlis, C. Cooper, D. Nelson
Hosts: Dr. Ryutaro Himeno, Director and Senior Scientist, ACCC; Director, Development Group, NGSC R&D Center
Email: himeno@riken.jp

Dr. Takayuki Shigetani, Senior Technical Scientist, ACCC

Dr. Makoto Taiji, Deputy Project Director, Computational and Experimental Systems Biology Group

Dr. Toshiaki Iitaka, Senior Scientist, Computational Astrophysics Laboratory

RIKEN is an independent administrative institution under the Ministry of Education, Culture, Sports, Science, and Technology (MEXT) since 2003. It was originally founded as the first private research foundation in 1917; it was reorganized as a public corporation in 1958 under the Rikagaku Kenkyusho Law. Today, RIKEN carries out comprehensive research in science and technology in all fields except social sciences. It has seven campuses in Japan and five campuses abroad (2 in the United States, 1 in the UK, and 1 in Singapore). Its activities focus on brain sciences, accelerator-based research, developmental biology, biomimetic research, terahertz-wave research, genomics, research on allergy and immunology, bioresources, etc. The total funding in FY2007 was ¥89.4 billion (~US$805 million⁷) with about 3,000 researchers, about 1800 visiting Japanese scientists, and 685 visitors from abroad.

The Advanced Center for Computing & Communication (ACCC) is part of RIKEN and provides RIKEN researchers with computer resources and network services. Currently, it operates the RIKEN Super Combined Cluster (RSCC), which includes a mix of computer architectures, scalar, vector and accelerators. In particular, RSCC is composed of three subsystems, five Linux clusters with Xeon 3.06 GHz processors (512 dual nodes and 128 dual nodes x 4), a vector parallel computer (NEC SX-7/32) and an MD-GRAPE3 board (64Tflops). The RIKEN Principal Investigators proposed this hybrid concept to MEXT to develop the Next Generation of Super Computer (NGSC).

Computing Facilities

NGSC: Japan’s 10 Petaflop Supercomputer

The RIKEN researchers based their proposed design for Japan’s next-generation (10 petaflop) supercomputer on the need for multiscale and multiphysics simulation in life science and nanoscience, which, in turn, involves multiple computation components; hence, it requires the interaction of multiple architectures. To this end, they initially proposed a tightly coupled heterogeneous computer consisting of vector, scalar, and MD nodes with a fast interconnect between the heterogeneous nodes. The final design, however, does not have any accelerators.

MEXT leadership is committed to sustaining continuous development of supercomputers in Japan. Specifically, MEXT’s policy stems from the belief that by computational science, Japan’s competitiveness in science and technology can be maintained in a world-leading position and that successive development of supercomputers will be realized by maintaining the necessary technology inside Japan. To this end, the first priority is sustained performance in key applications at the level of at least ten percent of peak performance. The end result of such successive development of supercomputers will be the enhancement of information technology with novel new hardware (e.g., low-power CPUs) but at a reduced cost.