1 Software Engineering Data Collection for Field Studies
23
Advantages: System monitoring requires no time commitment from software engineers. Since, people tend to be very poor judges of factors such as relative frequency and duration of the various
activities they perform, this technique can be used to provide such information accurately.
Disadvantages: It is difficult to analyze data from instrumented systems meaningfully that is, it is difficult to determine software engineers thoughts and goals from a series of tool invocations. This problem is particularly relevant when the working environment is not well understood or constrained. For example, software engineers often customize their environments by adding scripts and macros (e.g., in emacs). One way of dealing with this disadvantage is to playback the events to a software engineer and ask them to comment.
Although in many jurisdictions, employers have the right to monitor employees, there are ethical concerns if researchers become involved in monitoring software engineers without their knowledge.
Examples: Budgen and Thomson (2003) used a logging element when assessing how useful a particular CASE tool was. The logger element recorded data whenever an event occurred. Events were predetermined before. Textual data was not recorded. The researchers found that recording events only was a shortcoming of their design. It would have been more appropriate to collect information about the context of the particular event.
As another example, Walenstein (2003) used VNC (Virtual Network Computing) to collect verbatim screen protocols (continuous screen captures) of software developers engaged in software development activities. Walenstein also collected verbal protocols and used a theory-based approach to analyse the data.
More
recently, Storey et al. (2007) logged developers use of their TagSEA tool. The logs were stored on the client machine. The software engineers downloaded them to a server at specified intervals. The logs enabled Storey et alto understand how the tool was being used, and nicely complemented other data sources such as interviews and a focus group. Similar to this study,
Zou and Godfrey (2007) used a logger to determine which artifacts software maintainers were just viewing, and which were actually changed.
Reporting guidelines: The precise nature of the
logging needs to be reported, including any special instrumentation installed on the software engineer’s machines. This should include a description of what exactly is logged, with what frequency. Any special considerations with respect to data processing and analysis should also be detailed.
Share with your friends: