Guide to Advanced Empirical

2 Introduction in several other books and papers. To address these goals, the chapters in this book are grouped according to three primary themes:
“Research Methods and Techniques presents discussions on the proper use of various strategies for collecting and analyzing data, and the uses for which those strategies are most appropriate. Our aim in this section is to present ideas about strategies that are less often used in our field, which perhaps may provide some ideas about less conventional but still rigorous and useful ways of collecting data. We also aim to provide detailed information on topics such as surveys that in our experience often contain methodological errors.
“Practical Foundations provides a discussion of several basic issues that a researcher has to get right before a study becomes meaningful. This section discusses important, global issues that need to be considered from the very beginning of research planning. Chapters here discuss topics that are almost always going to be important, regardless of the specific choices that are made about running the study. Our aim in this section is to arm researchers with the information necessary to avoid fundamental risks. For example, an entire study maybe inappropriate if a researcher doesn’t understand enough about metrics and statistics to collect the right measures a researcher may not get the chance to run the study he/she wants if there is no good way to cooperate with industry or the results maybe jeopardized if incomplete data is collected by the study and the researcher don’t respond appropriately.
Finally, Knowledge Creation looks beyond the challenge of running an appropriate study to provide insight on what is becoming one of the most important challenges in empirical software engineering today–using a set of disparate studies all of which may employ different designs and metrics) to provide useful decision support on a question of interest. The conversion of discrete scientific results into a broadly useful body of knowledge on a topic is a difficult process, with many opportunities for introducing bias if done incorrectly. Refining and employing appropriate techniques in addressing this problem is one of the most important challenges for ensuring the relevance of empirical software engineering and showing its practical impact.
While we feel that all of these topics are of interest to many workers in this field, we do wish to direct the attention of certain readers to certain parts of the book. There are four target audiences for this book. Practising software engineering researchers, whether they reside in academia or in industrial research labs.
Of primary interest to such readers maybe the section on Knowledge Creation since building knowledge from multiple sources of data, suitable for providing higher level answers to problems, continues to be of more importance to the empirical software engineering community as a whole. At the same time, our methods for abstracting such knowledge are not yet well codified, and the chapters in this section raise awareness as well as inform researchers about the methods currently being employed.
Researchers maybe primarily either quantitatively or qualitatively inclined. Both types will find issues of direct relevance to the typical prob-

Introduction
3
lems that they encounter in the Research Methods and Tools section. That section is designed to provide a mix of relevant and interesting content of both types, as we feel strongly that the quantitative-qualitative distinction is an arbitrary one, and interesting and relevant conclusions will always need to combine a mix of both types of data. Overall, the methodological material will inform readers about advanced and defensible techniques that they can use in their research.
One last topic of special interest maybe that of reporting guidelines in Practical Foundations which can provide readers with guidelines that they can use for reporting their results, either internally within their organisations or in the scientific literature. Practising software engineers whose work involves data analysis. This category includes, for example, quality assurance personnel.
As budgets are squeezed, there is more pressure to provide stronger evidence and more convincing business cases to implement new technologies or make process changes, and even to justify decisions already made. Therefore, any knowledge that the engineers can use to help them achieve this goal is welcomed. The Research Methods and Tools section is of relevance to this target group. Insofar as readers maybe expected to address real problems of practical interest, we have tried to make available a set of techniques that maybe able to help them. As no two project environments and constraints are exactly alike, having as wide a variety of methods to apply will be beneficial. Moreover, the chapters in Practical Foundations that deal with the science of measurement and how to deal with common problems, such as missing data, may also be of help.
For example, one may consider using data collected about anew technology to estimate its effect on a larger process (simulation, understanding whether there are hidden costs of such changes by understanding more than just the dollar figure involved (qualitative methods, evaluating the impact of process changes within the organisation (through surveys of technical staff, customer surveys (survey design and focus groups, and performing appropriate analysis of factors affecting the incidence of operational defects when there are missing values in the defect databases (dealing with missing values. In particular, organisations following the improvement path stipulated by contemporary maturity/capability models are primary targets since these models emphasize measurement and quantitative control at higher levels of maturity/capability.
Other topics that maybe of particular relevance for individuals in industry performing empirical studies can be found in Practical Foundations For example, given that quality assurance personnel usually rely on the cooperation of the development and maintenance engineers, ethical behaviour will ensure that none of the engineers are alienated. Furthermore, there maybe legal ramifications for unethical behaviour, particularly in countries with strong labour laws (e.g., North European countries. The management of cooperation

4 Introduction with universities will be relevant for those involved in joint industry-university research projects. Since a successful collaboration depends on both parties, the industrial side would also contribute to this success if they are aware of these guidelines. Graduate students in software engineering. The book could serve as a text fora graduate level course on empirical methods, and/or as reference material for students embarking on a research project.
All of the material in the book will be of direct relevance to graduate students. Specifically, such readers may find valuable the coverage of the different types of studies that can be performed in order to make a decision on which approaches to follow during their research projects (Research Methods and Tools. Even more importantly, topics under Practical Foundations will help novice researchers recognize some of the background requirements in running successful studies, contribute towards ensuring that their research is well reported, and mitigate against the tendency of over-interpreting the results of individual studies.
The section on Knowledge Creation will help students understand the body of knowledge that may exist on their research topic and the importance of relating their work to existing theories that have been built up in the area. Reviewers of empirical research.
The overview of empirical methods with their strengths and weaknesses Research Methods and Tools, especially the discussion of appropriate issues that can be tackled with the various methods, should help reviewers make abetter judgement of the quality of an empirical study.
The section on Knowledge Creation is especially important to reviewers. First, it aims to inform such readers about, and increase the acceptance of, replication. Replication is critical for any discipline to progress, and reviewers are essentially the gatekeepers. The chapter on reporting guidelines would assist reviewers in ensuring that sufficient detail is reported in published manuscripts.
Perhaps the most relevant chapter under Practical Foundations for reviewers is the one concerned with ethics. Reviewers have to judge whether appropriate ethical behaviour was followed in published manuscripts. Again, being the gatekeepers fora discipline, they can encourage or discourage certain behaviours.
When we first set out to put this book together, we were motivated by what we as researchers felt was missing, a handy reference guide on some of the techniques we are called upon to apply as part of our work or to review in others work. Little did we understand at the time the kind of process we were embarking upon in trying to fill that gap. We wish to thank all of the chapter authors for their high-quality work and for helping to move this project along. We especially wish to thank all of

Introduction the external reviewers (listed below) for contributing their effort to improve the quality of the materials found here. We certainly hope that readers will find this, as we intended, a useful and practical reference for their own work.
Forrest Shull
Janice Singer
Dag Sjøberg

Download 1.5 Mb.

Share with your friends: