Guide to Advanced Empirical

Software Engineering 27(11), 1999–1013.
Novo, A. (2002), Analysis of multivariate normal datasets with missing values. Ported to R by Alvaro A. Novo. Original by J.L. Schafer.
R Development Core Team (2005), R: A language and environment for statistical computing, R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0. http://www.
R-project.org
Roth, PL, Missing data A conceptual review for applied psychologist, Personel
Psychology 47, Rubin, DB, Multiple Imputation for Nonresponse in Surveys, John Willey & Sons.
Schafer, J. L. (1997), Analysis of Incomplete Data, Monograph on Statistics ans Applied Probability, Chapman & Hall.
Schafer, J. L. & Olsen, MK, Multiple imputation for multivariate missing data problems, Multivariate Behavioural Research 33(4), 545–571.
Schafer, J. S. (1999), Software for multiple imputation. http://www.stat.psu.edu/˜jls/misoftwa.
html
Strike, K, Emam, K. E. & Madhavji, N. (2001), Software cost estimation with incomplete data,
IEEE Transactions on Software Engineering 27(10), Swanson, E. B. (1976), The dimensions of maintenance, in ‘Proc. 2nd Conf. on Software Engineering, San Francisco, pp. 492–497.
Twala, B, Cartwright, M. & Shepperd, M. (2006), Ensemble of missing data techniques to improve software prediction accuracy, in ‘ICSE’06’, ACM, Shanghai, China, pp. 909–912.
Weisberg, S. (1985), Applied Linear Regression, 2nd Edition, John Wiley & Sons, USA.

Index
abduction 318
abductive inference 317, 318
ACM code of ethics 230, 233, 237
ACM SigDoc action research 258, 301–303, 307, actor analytical phase of research 99, anomalies in data anonymity 238–239
ANOVA autonomy 232, available case 193
Basili’s framework of experimentation in software engineering 373–375, Bayesian analysis 90, behavioural sciences benchmark 65, beneficence between subjects design bias 77, 297, 299, 302, 306, 346, 347, 351,
352, 354, binomial data 90, 166, 169, 171, 172, blinding 207, blocks 295
Bonferroni procedure Brainstorming 11, 12–13, 97, Calibration 121, case study 50–51, 296–298, 313, 327, case study confirmatory 296, exploratory 296, 328
casewise deletion categorical data 166, 169, 171, causality 289, censoring central tendency 165, classical measurement theory 159
CMM 80, codes preformed 49
postformed coding phase of simulation 143, coding axial 49, selective 50, coercion 235–236, 247, cold fusion Common Rule the 241, compensation complete case, method compound metrics conceptual development conceptual modeling conclusion validity confidence in empirical results 337–338, confidence interval confidentiality 102, 216, 232, 238–240, 246,
248, 253, confidentiality of data 232, consent form 234, 250, 252, 253–254, constant comparison method construct validity 80, 161, 222, 223, 305
constructivism 291, 304
constructivists 290, 291, 293, 298, 302, constructs 223, 313, content analysis 59–60, content validity 80, context information 21, 24, 202, controlled experiment 291, 294–296,
306, convenience sampling correlation 166, cover letter 245 383

384 Index criterion validity 80, critical theorists 291–292, 298, critical theory 291–292, 302
Cronbach’s alpha 79, 161, 221
cross-case analysis cultural conflict data analysis 46, anomalies in binomial 90, 166, 169, 171, 172, categorical cleaning collection 37, 219, 279, 297, 303, confidentiality of 232, missing 89, 180–181, see also Chapter 7
multinomial nominal objective ordinal 90, 166–167, 169, quality 178–182, 189, subjective 58, validation 88–89, 178, visualization of deception 234–235, deductive methods 318 (also figure, pg. deletion techniques 186, dependent variable 187, 209, 217, descriptive statistics 97, 156, 161, 163, 164, design documents 118, direct observation 37, disclosure full partial discrimination validity dispersion 165, dynamic analysis effect size 221, 352, 354, 355, EMPEROR approach empirical research project 257, Empirical Software Engineering Journal empirical support 318, 319, 332
empirically-based theory 312, empiricists epistemology equivalence testing 168
ERB ethics review boards ethnographic research ethnography 21, 299, 300–301, evaluating theories evaluative phase of research 99, expedited review 244, Experience Factory 356, experience portal experimental unit 84, 181, 207, explanatory power 316, 319, 320, 332, explanatory theory EXTEND external validity 222, 223, 295, face validity 80, fairness 346
falsifiability field memo field notes 39, 45, 48, 49
FIML focus groups 12–13, 77, see also Chapter forecasting full information maximum likelihood function points 132, 133, fusion, cold 370
Goal/Question/Metrics (GQM) 206, 212, 215,
340, goodness of a theory Grounded theory 48, 293, 317, group mean substitution Group Support Systems 102–105
GSS 102–105
Hackystat Hawthorne effect 308
IEEE-CS/ACM code of ethics 230,
233, imputation hot deck 192, 194
k-nearest neighbor mean multiple 192, independent variable 217, 219, induction 293, 297, inductive analysis 372
industrially-sponsored project inferential statistics 156, 163, inflows 127, informational phase of research 99, informed consent 232–236, input parameter values Institutional Review Board 241
instrumentalism 316, 318

Index integrating evidence 352, intellectual property 262, 266, 276
inter-coder reliability (see also inter-rater reliability) internal validity 208, 223, 306
interpretivism 35–36, 291
inter-rater reliability 29, 79, 167, interval scale interview 13–15, 43–46, 82, 237, 268,
298–299, guide 45, scribe 45
semi-structured 14, 44, structured 13–14, 44, 105, unstructured 44, 45
Kaplan-Meier Estimate 190
Kendall’s tau 29, 90, 167
Kruskal-Wallis test laboratory package level variables 127, 128, 136, 138, life cycle 120
listwise deletion local theory 291, 292, 301, logistic regression 90, longitudinal study 14, 22, 68, 173, 190,
276, mapping studies MAR mathematical theory 160
MATLAB 126
MCAR mean absolute deviation mean absolute percentage error mean percentage error mean squared error mean substitution measurement 74, precision process scale 80, measurement theory 159, measures of association measures of dispersion median medical research, evidence in mental model 11, 16, 19, 1 18, member checking 53–54
meta-analysis 221, 225, qualitative 293, 309, see also Chapter quantitative 36, 60, 97, metric definition metric evaluation MI minimal risk minimize harm Mining Software Repositories, International Workshop on 25, missing at random missing completely at random missing data 89, 180–181, see also Chapter mixed methods 60, model calibration 121, model validation 120, moderator 94, modes 164
MSR 25, 26
multinomial data multiple imputation multiple regression 162, 192, multivariate regression 170
N-version programming negative case analysis 52
NMAR nominal data nominal scale 160
non-parametric model 163
non-parametric test 29–30, 163, 167, nonrandom sampling not missing at random null hypothesis 167, 168, 222, objective data objectivity 54, 48, 345, 346
object-oriented system, maintainability of observation 11, 12, 20–21, 37–43, 243, 244,
246, 250, ontology open coding 49, open questions 71
open-ended question 14, 44, 67, openness 345, ordinal data 90, 166–167, 169, ordinal scale 73–74, 76, 90, 160, organizational characteristics outflows 127
outlier 52–53, 164, output parameter values overrun of software projects 67, 350

pairwise deletion parametric models participant observation 21–22, 37–43, 234,
250, peer review of studies 275, 342, 349, 358, 366,
367
perspective-based reading 212, 215, 217, philosophical stance 36, physical theory pilot test 70, 78, 83, positivism 35–36, 291, pragmatism 292, 313, 316, pragmatists 292, prediction 163, 169–173, 175–176, 292, 314,
315, 319, probability sampling 86, problem statement 119–120, 204, 206, process control process improvement 120, 187, 258, 292, product quality 46, 122, 158, 226, 238, program comprehension 19, 315, project cost estimation project description 244, project management 66, 122, 123, propositional phase of research 99, proposition 48, 50–52, 54, 296, 305, 313, 316,
19, 320, 322, 323, publication bias 360
purposive sampling 94, 297
QSIM 125
QUAF qualitative analysis of causal feedback quasi experiment 202, questionnaire 11, 15–16, 65, 70, 74, 242, 245,
298–299, 329, 373, 376, see also Chapter questionnaire specification questions base-rate causality exploratory knowledge 289
open-ended 14, 44, 67, relationship quota sampling random sample randomization 85–86, 207, 215, rate variables 127, 128, 132, 136, rater agreement exercises ratio scale rationalists recruitment of subjects 239, 246, reductionist 291, reference behaviour 119, regression equation regression methodology 170, regression substitution reliability 59, 77–79, 80–81, 160, 161, repeated measures analysis replication 53, 214, 216, 240, 302, 326, see also Chapter replication close 375, criteria for differentiated exact external 365, information required to allow internal 365, partial reporting guidelines for reporting guidelines abstracts acknowledgements analysis analysis procedure appendices authorship background conclusion deviations from the plan discussion experiment design experiment planning goals hypotheses introduction keywords materials participants procedure references results task threats to validity title representative sample 68, 82, 295, requirements 11, 98, 104, 120, 123, 130, 144,
186, 317, 322, requirements prioritization 98, research design 294, 309, method question 100, 287–290, 309 386
Index

Index Research Ethics Board 233, response rate 15, 75, 87, 268, risk management 66, 98, 118, robustness sample size 87, 88, 94, 163, 168, 175, 185,
186, representative 68, 82, 295, bias 178, 181, 182, method 83, unit chain cluster based convenience 86
purposive 94, quota random snowball stratified 85, systematic scientific review scientific value 240, 244, scope 120, 319, 320, scope conditions 316, 323, searching for empirical evidence segmentation of participants sensitivity analysis 191, shadowing 11, 20–21, 38, 232, 237, synchronized 20, 38, similar response pattern imputation simple metrics 158–159
Simpson-traction replication simulation model deterministic dynamic 123
event-driven static stochastic 122–123
simulation
DE 124, discrete event 124, hybrid modeling tools qualitative quantitative SD 127–128, system dynamic 127–128, verification 121–122, skewed distribution 159, skill 267, software engineering theory 314, software industry software metrics 157, 160, 170, 171, software process 13, 21, 79, 117, 120, 148, software process improvement 13, software reliability software system 314, 321, specificity static analysis statistical analysis 29, 156–157, statistical power 168, 215, 353, statistical significance 222, 269, stratified random sample 85, structured abstract subject information sheet subject selection of vulnerable 233, 243, subjective data 58, substitution techniques survey design documentation instrument instrument evaluation reliability survey research 298–299, see also Chapter survival curve synchronized shadowing 20, 38, systematic literature review 340, target population 82, technology transfer 99, 100, 262–263
testability 319, theory 313, theory building 292, theory development 318, theory generation 48
think-aloud 11, 19–20, threats to validity 222–223, 269, time series 173, 174, time sheets 11, 17, transcription 19, 29, treatment triangulate 30, 52, triangulation 30, 52, 132, 304, concurrent truncation 159
t-test type I error 168, type II error 168, 223
UML unit of analysis 297

validity 52, 80, 161, 222–223, 305–307, construct 80, 161, 222, 223, content 80, conclusion criterion 80, discrimination external 222, 223, 295, face 80, internal 208, 223, 306
VENSIM 126, verification 121–122, visualization of data 54–57
voluntariness 232–233, 235, 236, 248,
252, vulnerable subjects 233, 243, weighting of evidence sources 354, 358
Wilcoxon/Mann-Whitney test within subjects design work diaries 11, 17–18 388
Index