Humanist Evaluation Methods in Locative Media Design


  • Anders Fagerjord University of Oslo



Design, Methodology, Media Innovations


Media design can be used for research purposes if it includes a clearly defined research question, and clear evaluation to see whether an answer to the research question has been found. Using a project with locative media for classical music communication as our example, we discuss common evaluation methods from the User Experience field, observing that they all tend to test “interface” and not “content.” Instead we propose three other methods of evaluation, that have a basis in humanist theories, such as textual analysis and genre studies: (1) Qualitative interviews with evaluators after the evaluation, asking them to describe the service in their own words, followed by a semantic analysis to get at how they have understood the service. (2) Within-subject A/B tests with alternative versions that are different in key aspects. (3) Peer review by experienced design researchers, who are likely to have a more fine-tuned vocabulary to express their opinions.


Andersson, B.-E., & Nilsson, S.-G. (1964). Studies in the reliability and validity of the critical incident technique. Journal of Applied Psychology, 48(6), 398. doi:10.1037/h0042025

Bolter, J. D. (2003). Theory and Practice in New Media Studies. In G. Liestøl, A. Morrison, & T. Rasmussen (Eds.), Digital Media Revisited (pp. 15-33). Cambridge: MIT Press.

Flanagan, J. C. (1954). The critical incident technique. Psychological Bulletin, 51(4). doi:10.1037/h0061470

Gadamer, H.-G. (2004). Truth and method (W. Glen-Doepel, J. Weisheimer, & D. G. Marshall, Trans. Second, revised ed.). London: Continuum.

Hartson, R., & Pyla, P. S. (2012). The UX book: Process and guidelines for ensuring a quality user experience. Amsterdam: Morgan Kaufmann.

Hassenzahl. (2000). Hedonic and Ergonomic aspects determine a software’s appeal. CHI Letters, 2(1).

Hassenzahl, M. (2001). The effect of perceived hedonic quality on product appealingness. International Journal of Human-Computer Interaction, 13(4), 481-499.

Hevner, A. R., March, S. R., Park, J., & Ram, S. (2004). Design science in information systems work. MIS Quarterly, 26(1), 75-105.

Kohavi, R., Longbotham, R., Sommerfield, D., & Henne, R. M. (2009). Controlled experiments on the web: survey and practical guide. Data Mining and Knowledge Discovery, 18(1), 140-181. doi:10.1007/s10618-008-0114-1

Krippendorff, K. (2006). The semantic turn: A new foundation for design. Boca Raton: Taylor & Francis.

Krug, S. (2010). Rocket surgery made easy: The do-it-yourself guide to finding and fixing usability problems (Kindle ed.). Berkeley, California: New Riders.

Latour, B., & Woolgar, S. (1986). Laboratory work: The construction of scientific facts (Second ed.). Princeton University Press.

Lewis, C. (1982). Using the ‘thinking-aloud’ method in cognitive interface design. Research report RC 9265. IBM TJ Watson Research Center.

Liestøl, G. (1999). Rhetorics of Hypermedia Design. In Essays in Rhetorics of Hypermedia Design (Ph.D. Dissertation ed., p. 265). Oslo: Department of Media and Communication, University of Oslo.

Liestøl, G. (2006). Conducting Genre Convergence For Learning. Cont. Engineering Education and Lifelong Learning, 16(3/4), 255-270.

Løvlie, A. (2009). Textopia: Designing a locative literary reader. Journal of Location Based Services, 3(4), 249-276.

March, S. T., & Smith, G. F. (1995). Design and natural science research on information technology. Decision Support Systems, 15, 251-266.

Moulthrop, S. (2005). After the last generation: Rethinking scholarship in the age of serious play. Proceedings from Digital arts and culture, Copenhagen.

Nielsen, J. (2000). Designing Web Usability: The Practice of Simplicity. Indianapolis: New Rider.

Nielsen, J., & Loranger, H. (2006). Prioritizing Web Usability. Berkeley, California: New Rider.

Nyre, L. (2014). Medium design method: Combining media studeis with design science to make new media. The Journal of Media Innovation, 1(1), 86-109.

Osgood, C. E., Suci, G. J., & Tannenbaum, P. H. (1957). The measurement of meaning. University of Illinois Press.






opyright (c) 2014-2020 Simon Fraser University * Copyright (c) 2003-2020 John Willinsky * Distributed under the GNU GPL v3. For full terms see the file docs/COPYING. * * @brief Common site frontend footer. * * @uses $isFullWidth bool Should this page be displayed without sidebars? This * represents a page-level override, and doesn't indicate whether or not * sidebars have been configured for thesite. *}