Supervised by Prof
III: Evaluation Techniques
Download 281.3 Kb.
|
- Navigate this page:
- IV: Real-world Experimentation
- Future Directions
III: Evaluation TechniquesThe following mechanisms will be used as evaluation tools for the experiments:
The first two techniques will give objective, directed measurements which will help assess the ease of perception or control of certain anticipated features of the interface. The latter two techniques incorporate subjective feedback and are more open-ended, in order to learn about new, unanticipated aspects of the system which might be discovered during the experimentation process. Depending on the results of the subjective evaluations, additional experiments might be added, in order to surgically isolate a certain feature or to examine a new aspect or direction which might not have been anticipated when the original tests were planned. IV: Real-world ExperimentationAfter the simplified scenarios have been studied, CMS will need to undergo similar evaluation in real-life environment, in order to evaluate how behavior might change under circumstances which more closely resemble those which sysadmins must deal with on a daily basis. The tests described above are all of a diagnostic nature; they set controlled conditions which test actual behavior against expected user activity, all while focusing on a single aspect of the system. They are useful as a tool for guiding the final system design, to know what features to tweak before making the next full prototype. In a broader sense, they cast light on the way users perceive certain features of tangible interfaces, and in so doing contribute their part to the broader field of TUI design as a whole. However, the overall goal of this project is to assess the suitability of the CMS system for its stated task – chiefly – the long-term monitoring of large-scale systems as part of the management of these systems. Therefore, upon completion of the diagnostic tests, the next step is to examine the Cube Management System under real-life or almost-real-life situations: namely, with a group of sysadmins monitoring a large-scale system. The following controlled situations are to be tested in such an environment:
Using these scenarios, we hope to observe the manner in which the system administrators perceive the system and work with it, as filtered through the "eyes" of CMS. In order to measure the effectiveness of CMS under these scenarios, user response with CMS will be compared to user response using a GUI or CLI under similarly-structured (but not exactly the same) scenarios, using the evaluation techniques outlined in the previous chapter. In order to correctly evaluate the advantage of CMS, each experiment set will involve 30 almost-repetitions (i.e. similar but not the exact same situation) of each scenario, and evaluate the average response times and error ratios over all the experiments. Statistical analysis of these measurements will also help pinpoint any significant deviations from the norm, which might arise in a certain configuration. Such deviations should then be analyzed to see what aspect of the particular deviant configuration might have brought about the deviation. The conclusions of the analysis (and of any other comments which might have arisen through the subjective analysis) would then need to be re-tested, and the system possibly adjusted to incorporate the new considerations. After several cycles of testing, evaluating, and tweaking, the details of the CMS interface will ultimately converge towards an optimal TUI for systems monitoring. At that point, enough will have been learnt about how systems administrators respond to CMS and to TUIs in general, to be able to come to significant conclusions for any future considerations in interface design. As CMS enters its testing and evaluation stages, input from the UI / Systems / Cognition communities is welcome, to help refine and optimize the plans for testing and evaluation. Future DirectionsSince CMS was first presented, much interest has been generated about the idea of a TUI for large-scale systems monitoring. Suggestions and inquiries have been made by members of the research community, systems administrators, and designers alike. Attitudes have ranged from skepticism along the lines of "if it ain't broke, don't fix it" to unadulterated gadget-lust. Alternative implementation suggestions have also been plentiful, including suggestions to use cards with embedded RFID chips that can pull up information from distributed displays or to use cellphones as cubes, with or without tray awareness. The novelty of the idea seems to have stimulated the creative juices of many researchers, and it is clear that CMS has opened discussions which could be carried on much further. As for us, the CMS research team: we are still looking to optimize the usability of the system, and then if all goes well and we see that that the system does indeed proffer a usability advantage, we hope to also incorporate implementation optimizations, to see if, the system can be produced cheaply enough but usefully enough to offer an attractive cost-advantage ratio. This means evaluating various materials, sizes of the system, types of light, display materials, power supplies, and other implementation factors. It is clear that the most immediate necessary step forward is to perform the lab experiments described in this paper, to be able to make informed decisions converging towards a design worth implementing. These observations also promise to contribute insights to the fields of cognitive science and usability research; we look forward to seeing what kind of insights will emerge from our trials. Another aspect of the CMS that we hope to develop and which promises to yield fruit for the larger scientific community is an analysis of the logic of systems representation. This relates also to the ideal of representing the abstraction hierarchy or any other logical construct of the system – it could very well be that as systems expand in complexity, they may even reach a saturation point beyond which there simply is no feasible way of representing all of the various possible functions of the system – and that the side-effects are thus inadequately represented in the interface. In order to properly assess this situation, thorough mathematical and experimental analyses are in order. This touches on theoretical mathematics such as multivariate analysis, as it investigates the question of how to map an issue in a complex, multilayered system, to a single or double- layered visual context. Or, more formally: given n multi-value variables per system element, we need to investigate what heuristic would best map these to a smaller subset of data. Instinctively, it would seem that the tangibility and ambient aspects of a TUI would "sneak in" additional dimensions of representation to the range of the mapping function, and thus enable more flexibility in the tradeoff between the expansion of the results set and the simplification of the mapping function – and even so, this might be insufficient to adequately represent the immense complexity that forms our computer systems. None of this intuition has yet to stand the test of mathematical rigor or of large-scale thorough experimental analysis. Putting it to this test serves to offer the most promising contribution that the study of CMS and systems administration in general have to offer the scientific, design, development, and general research communities. “… the world portrayed on our information displays is caught up in the two-dimensionality of the endless flatlands of paper and video screen … Escaping this flatland is the essential task of envisioning information – for all the interesting worlds …that we seek to understand are inevitably and happily multivariate in nature. Not flatlands.” - E. Tufte, Envisioning Information94 Download 281.3 Kb. Share with your friends:
|