The first evaluation of the prototype is now completed. The procedure was designed to test the individual components as well as the whole prototype (for playing music etc.). Raw data on selection times, error rates etc. was collected by custom development on the Microsoft .NET platform. Additional forms were displayed on-screen in between the steps of the procedure combined with standardized form based questionnaires to gather a rich set of data.
A more subjective approach was taken during the usage of the prototype as a whole to capture the aspects which could not be confined by automatic data collection or forms. Simply by observing participants and asking simple questions in normal spoken language. While perhaps being less scientifically valid this type of information is very valuable for understanding how the users think and react. Information that is crucial for improving the design for the next iteration. There is sometimes a difference the results gained by verbal utterance, questionnaires and measurable performance. For example, interfaces can be very efficient and fast but at the same time extremely demanding and stressful. Just measuring one performance factor would not tell the whole story.
This is why I chosen to use several methods to combine raw performance data, form based questionnaires and unconstrained verbal interviewing. Hoping that it can provide multiple aspects with potential for gathering a rich source of data.
For the evaluation I used a basic on-screen demographic form gathering age, sex, computer experience, conditions of vision etc. In-between the evaluation of the individual components I used the
NASA Task Load Index as a quick reflection form and at the end of the session I handed out both a
IBM Computer Usability Satisfaction Questionnaire and a
Q.U.I.S Generic User Interface Questionnaire. The only modification I performed was to remove a few questions that would not apply to my prototype (why ask about help pages when the prototype contains none)
I´ve found the
230 Tips and Tricks for Better Usability Testing guide to be really useful and should be read by anyone conduction HCI evaluation.
Questionnaires used in my evaluation, in PDF format:
