OU blog

Personal Blogs

Christopher Douce

TM470 Considering evaluation

Visible to anyone in the world
Edited by Christopher Douce, Friday, 4 Oct 2024, 16:04

When planning your project, an important question to ask is: how can I tell if my project has been a success?

The extent of your evaluation very much depends on what you do during your project. For some projects, you might consider asking your stakeholders the question: “does this solve your problem?” You can ask this question in different ways: you could prepare a short survey (if you have a number of stakeholders), carry out an interview, or carry out a usability study.

If you consider design (rather than coding) to be your strong point, you may choose to carry out an interaction design project, where the output may well be a detailed high level prototype that could be presented using a prototyping tool. Since you may sidestep some of the important software development skills that you might otherwise demonstrate, is really important to ensure that you carry out your interaction design project in a really thorough way to clearly satisfy the requirement that your project is all about computing. If you are confident with software development and enjoy this aspect of your studies, you may well want to combine design with an aspect of software development; it is completely up to you.

Most interaction design projects are iterative. In fact, a practical recommendation is that a solid project that is based on the interaction design module should have three different development iterations, or phases. The thinking behind this is that this gives you the best opportunity to demonstrate your skills. When you get to the end you may want to ask the question: has my project solved the problem I set out to solve?

Consider an example of an app that could to pre-order a coffee from a local café, or to book a slot for your dog in a pet grooming parlour (you can choose another example if you prefer). 

Iteration 1: First initial sketches

Designs can be presented (and detailed) using different levels of fidelity, which (of course) relates to the idea of detail. A low fidelity design (or sketch) might be a rough sketch on paper (which is known as a paper-based prototype). A high fidelity design might be a design that closely resembles the final product. There are other terms that you might want to consider too: the idea of a vertical prototype (a prototype that covers a small bit of functionality in a lot of depth), or a horizontal prototype (a prototype that covers all the functionality of a product, but in a small amount of depth).

You might begin your design by creating some low fidelity designs, perhaps using paper-based prototyping. In other words, writing a bunch of sketches. You might want to combine this with other approaches from interaction design, such as the writing of user profiles and scenarios. It is up to you. Whatever you do needs to link back to the requirements of whatever it is you are designing. Different products will treat requirements in different ways.

When you have completed your first round of sketches (which could be made either using a pen or pencil, or using a tool), you should show them to whoever has the problem that needs to be solved (the café owner, or manager of the pet grooming parlour) to carry out your first round of evaluations.

Iteration 2: Wireframes

When you have received feedback, and have further refined your requirements, it is then a good idea to try to flesh out your design. You might have a second round of sketches, or you might move from pencil and paper to a prototyping tool. Choose whatever tool or product works best for you and your project.

When you have completed you second set of designs, go back to your stakeholders to get some further feedback. Here you should get some useful data (information about what works well, and what needs to be looked at in a bit more detail) to further refine you prototype further.

Iteration 3: Higher Fidelity prototypes

A high fidelity prototype is all about showing what a final (or polished) version of a product or system might look like. With this final iteration (or phase) you might get elements of your designing working. You may well develop software and create databases. 

As well as considering prototypes in terms of low and high fidelity, there is another couple of dimensions that can be considered helpful: horizontal and vertical prototypes. A horizontal prototype is a demonstration of a design across all its key functionality, but to a limited depth. A vertical prototype, on the other hand, means that you implement a small amount of functionality in a lot of depth. With a vertical prototype, you may well ‘go deep’ with the technology; you may have to choose appropriate software components and frameworks, and justify your choice.

When it comes to your project report, you should share your high fidelity prototype by walking your examiner through a series of screenshots.

Depending on the aim of your project, you might stop at this point. You may well have demonstrated a lot of technical skill and knowledge in a lot of detail. If your project is less about practical software development and more about design, it would be a good idea to carry out a final evaluation of your prototype to answer the question: does my design do what it supposed to do?

Using the DECIDE framework

The DECIDE framework can help you to plan (and run) an evaluation. It is featured within the interaction design module and its accompanying set text. Taking a letter at a time, here are the components of the framework: 

Determine the goals (of your evaluation): What are you aiming to get from your evaluation? Remember, this isn’t the goals of your project, it is the goals of your evaluation. What do you need to do to determine whether it is a success.

Explore the questions: What questions are you trying to find answers for? Or, alternatively, what questions do you need to ask to carry out an evaluation?

Choose the evaluation approach or paradigm: How are you going to approach your evaluation? You can choose a number of different approaches. You might wish to carry our a heuristic evaluation (which doesn’t involve users), a predictive evaluation (to predict how efficient your design is, perhaps using a cognitive walkthrough approach), or user testing (which can involve real users).

Identify the practical issues: This point relates to the detail of how you run your evaluation. If you need participants, how will you go about recruiting them? How will you ensure they are representative? How will you collect data? Will you be making notes on a data collection form, or if you’re carrying out an interview, will you be using the voice recorder app on your mobile phone? When you have captured data, where will you be storing your data, and how will you be making sure your data is secured.

Decide about the ethical issues: If you involve anyone in your evaluation, you need to think about ethics. Essentially, you need to gain permission, and you need to make sure that everyone knows what your evaluation is all about. An important point to emphasise is that you’re carrying out an evaluation of the performance of a product or a solution rather than the performance of your participants. Remember that your participants are always more important than the task of completing your evaluation. 

Evaluate: With your evaluation all planned, your data collection approach chosen, and permission gained, it is now time to carry out your evaluation. After you’ve gathered your data, it is time to analyse your data, interpret your data and then share your findings. In terms of your project report, this will mean the need to write a summary or conclusion.

Carrying out your evaluation

The DECIDE framework helps you to understand why you intend to carry out your evaluation, what criteria for success you consider to be important, and how you intend to carry out your evaluation.

If your project has an evaluation section, a recommendation is to show what you did in series of step by steps. In other words, adopt a narrative approach. Use a series of appendices to share the resources you may have created during the course of your evaluation. If participants are involved in your evaluations, consider taking a photograph of the environment in which an evaluation takes place, sharing that photograph within the body of your report.

Reflections

Although I have suggested three phrases or iterations is often appropriate for an interaction design project, there are no ‘right’ number of iterations. Every project is different. Whilst carrying you’re your project you may discover that you need more phases, and more evaluations.

References

Rogers, Y., Sharp, H., Preece, J. (2023) Interaction Design: Beyond Human-Computer Interaction, 6th Edition. Wiley.

Permalink Add your comment
Share post

This blog might contain posts that are only visible to logged-in users, or where only logged-in users can comment. If you have an account on the system, please log in for full access.

Total visits to this blog: 2284952