Table of Contents Executive Summary 3

2.Contributions to Science

2.1ATLAS

Based upon the results achieved during the computing challenges and at the start of ATLAS data taking we can confirm that the tiered, grid-based, computing model is the most flexible structure currently conceivable to process, reprocess, distill, disseminate, and analyze ATLAS data. We found, however, that the Tier-2 centers may not be sufficient to reliably serve as the primary analysis engine for more than 400 U.S. physicists. As a consequence a third tier with computing and storage resources located geographically close to the researchers was defined as part of the analysis chain as an important component to buffer the U.S. ATLAS analysis system from unforeseen, future problems. Further, the enhancement of U.S. ATLAS institutions’ Tier-3 capabilities is essential and is planned to be built around the short and long-term analysis strategies of each U.S. group.

An essential component of this strategy is the creation of a centralized support structure, because the considerable obstacle to creating and sustaining campus-based computing clusters is the continuing support required. In anticipation of not having access to IT professionals to install and maintain these clusters U.S. ATLAS at each institution has spent a considerable amount of effort to develop an approach for a low maintenance implementation of Tier-3 computing. While computing expertise in U.S. ATLAS was sufficient to develop ideas on a fairly high level only the combined expert knowledge and associated effort from U.S. ATLAS and OSG facilities personnel has eventually resulted in a package that is easily installable and maintainable by scientists.

Dan Fraser of the Open Science Grid (OSG) has organized regular Tier 3 Liaison meetings between several members of the OSG facilities, U.S. Atlas and U.S. CMS. During these meetings, topics discussed include cluster management, site configuration, site security, storage technology, site design, and experiment-specific Tier-3 requirements. Based on information exchanged at these meetings several aspects of the U.S. Atlas Tier-3 design were refined. Both the OSG and U.S. Atlas Tier-3 documentation was improved and enhanced.

OSG has hosted two multi-day working meetings at the University of Wisconsin in Madison with members from the OSG Virtual Data Toolkit (VDT) group, Condor development team, the OSG Tier-3 group and U.S. Atlas attending. As a result U.S. Atlas changed the way howthat user accounts were to be handled based on discussions with the system manager of the University of Wisconsin GLOW cluster; thus leading to a simpler design for smaller Tier-3 sites. Also, the method for Condor batch software installation using yum repositories and RPM files were developed. As a result the Condor and VDT teams created an official yum repository for software installation similar to how the Scientific Linux operation system is installed. Detailed instructions were written for the installation of U.S. Atlas Tier-3 sites, including instructions for components as complex as storage management systems based on xrootd.

Following the trips to Madison, U.S. Atlas installed an entire Tier 3 cluster using virtual machines on one multi core desktop machine. This virtual cluster is used for documentation development and U.S. Atlas Tier-3 administrator training. Marco Mambelli of OSG has provided much assistance in the configuration and installation of the software for the virtual cluster. Also, OSG supports a crucial user software package (wlcg-client with support for xrootd added by OSG) used by all U.S. Atlas Tier 3 users, a package that enhances and simplifies the user environment at the Tier 3 sites.

Today U.S. ATLAS (contributing to ATLAS as a whole) relies quite extensively on services and software provided by OSG, as well as on processes and support systems that have been produced or evolved by OSG. Partly this originates in the fact that U.S. ATLAS fully committed itself to relying on OSG for several years now. Over the course of the last about 3 years U.S. ATLAS itself invested heavily in OSG in many aspects – human and computing resources, operational coherence and more. In addition and essential to the operation of the worldwide distributed ATLAS computing facility the OSG efforts have aided the integration with WLCG partners in Europe and Asia. The derived components and procedures have become the basis for support and operation covering the interoperation between OSG, EGEE, and other grid sites relevant to ATLAS data analysis. OSG provides Software components that allow interoperability with European ATLAS sites, including selected components from the gLite middleware stack such as LCG client utilities (for file movement, supporting space tokens as required by ATLAS), and file catalogs (server and client).

It is vital to U.S. ATLAS that the present level of service continue uninterrupted for the foreseeable future, and that all of the services and support structures upon which U.S. ATLAS relies today have a clear transition or continuation strategy.

Based on experience and observations U.S. ATLAS made a suggestion to start work on a coherent middleware architecture rather than having OSG to continue providing a distribution as a heterogeneous software system consisting of components contributed by a wide range of projects. One of the difficulties we ran into several times was due to inter-component functional dependencies that can only be avoided if there is good communication and coordination between component development teams. A technology working group, chaired by a member of the U.S. ATLAS facilities group (John Hover, BNL) is asked to help the OSG Technical Director by investigating, researching, and clarifying design issues, resolving questions directly, and summarizing technical design trade-offs such that the component project teams can make informed decisions. In order to achieve the goals as they were set forth, OSG needs an explicit, documented system design, or architecture so that component developers can make compatible design decisions and virtual organizations (VO) such as U.S. ATLAS can develop their own application based on the OSG middleware stack as a platform.

As a short-term goal the creation of a design roadmap is in progress.

Regarding support services the OSG Grid Operations Center (GOC) infrastructure at Indiana University is at the heart of the operations and user support procedures. It is also integrated with the GGUS infrastructure in Europe making the GOC a globally connected system for worldwide ATLAS computing operation.

Middleware deployment support is an essential and complex function the U.S. ATLAS facilities are fully dependent upon. The need for continued support for testing, certifying and building a middleware as a solid foundation our production and distributed analysis activities runs on was served very well so far and will continue to exist in the future, as does the need for coordination of the roll out, deployment, debugging and support for the middleware services. In addition the need for some level of preproduction deployment testing has been shown to be indispensable and must be maintained. This is currently supported through the OSG Integration Test Bed (ITB) providing the underlying grid infrastructure at several sites with a dedicated test instance of PanDA, the ATLAS Production and Distributed Analysis system running on top of it. This implements the essential function of validation processes that accompany incorporation of new and new versions of grid middleware services into the VDT, the coherent OSG software component repository. In fact U.S. ATLAS relies on the VDT and OSG packaging, installation, and configuration processes that lead to a well-documented and easily deployable OSG software stack.

U.S. ATLAS greatly benefits from OSG's Gratia accounting services, as well as the information services and probes that provide statistical data about facility resource usage and site information passed to the application layer and to WLCG for review of compliance with MOU agreements.

One of the essential parts of grid operations is that of operational security coordination. The coordinator is provided by OSG today, and relies on good contacts to security representatives at the U.S. ATLAS Tier-1 center and Tier-2 sites. Thanks to activities initiated and coordinated by OSG a strong operational security community has grown up in the U.S. in the past few years, driven by the needs of ensuring that security problems are well coordinated across the distributed infrastructure.

Moving on to ATLAS activities that were performed on the OSG facility infrastructure in 2009 the significant feature, besides dissemination and processing of collision data late in the year, was the preparation for and the execution of the STEP’09 (Scale Testing for the Experimental Program 2009) exercise, which provided a real opportunity to perform a significant scale test with all experiments scheduling exercises in synchrony with each other. The main goals were to utilize specific aspects of the computing model that had not previously been fully tested. The two most important tests were the reprocessing at Tier-1 centers with the experiments recalling the data from tape at the full anticipated rates, and the testing of analysis workflows at the Tier-2 centers. In the U.S., and in several other regions, the results were very encouraging with the Tier-2 centers exceeding the metrics set by ATLAS. As an example, with more than 150,000 successfully completed analysis jobs at 94% efficiency U.S. ATLAS took the lead in ATLAS. This was achieved while the Tier-2s were also continuing to run Monte Carlo production jobs, scaled to keep the Tier-2 resources fully utilized. Not a single problem with the OSG provided infrastructure was encountered during the execution of the exercise. However, there is an area of concern that may impact the facilities’ performance in the future. As we continuously increase the number of job slots at sites the performance of pilot submission through CondorG and the underlying Globus Toolkit 2 (GT2) based gatekeeper has to keep up without slowing down the job throughput in particular when running short jobs. When addressing this point with the OSG facilities team we found that they were open to evaluate and incorporate recently developed components, like the CREAM CE provided by EGEE developers in Italy. There are already intensive tests conducted by the Condor team in Madison and integration issues were discussed in December between the PanDA team and Condor developers.

Other important achievements are connected with the start of LHC operations on November 23^rd. As all ATLAS detector components were working according to their specifications, including the data acquisition, trigger chain and prompt reconstruction at the Tier-0 center at CERN timely data dissemination to and stable operations of the facility services in the U.S. were the most critical aspects we were focusing on. We are pleased to report that throughout the (short) period of data taking (ended on December 16^th) data replication was observed at short latency, with data transfers from the Tier-0 to the Tier-1 center at BNL typically completed within 2.6 hours and transfers from the Tier-1 to 5 Tier-2 and 1 Tier-3 centers completed in 6 hours on average. The full RAW dataset (350 TB) and the derived datasets were sent to BNL, with the derived datasets (140 TB) further distributed to the Tier-2s. About 100 users worldwide started immediately to analyze the collision data, with 38% of the analysis activities observed to run on the resources provided in the U.S. Due to improved alignment and condition data that became available following first pass reconstruction of the RAW data at CERN a worldwide reprocessing campaign was conducted by ATLAS over the Christmas holidays. The goal of the campaign was to process all good runs at the Tier-1 centers and have the results ready by January 1^st, 2010. All processing steps (RAW -> ESD, ESD -> AOD, HIST, NTUP and TAG) and the replication of newly produced datasets to the Tier-1 and Tier-2 centers worldwide was completed well before the end of the year.