OU blog

Personal Blogs

Christopher Douce

TM470 Considering evaluation

Visible to anyone in the world
Edited by Christopher Douce, Friday, 4 Oct 2024, 16:04

When planning your project, an important question to ask is: how can I tell if my project has been a success?

The extent of your evaluation very much depends on what you do during your project. For some projects, you might consider asking your stakeholders the question: “does this solve your problem?” You can ask this question in different ways: you could prepare a short survey (if you have a number of stakeholders), carry out an interview, or carry out a usability study.

If you consider design (rather than coding) to be your strong point, you may choose to carry out an interaction design project, where the output may well be a detailed high level prototype that could be presented using a prototyping tool. Since you may sidestep some of the important software development skills that you might otherwise demonstrate, is really important to ensure that you carry out your interaction design project in a really thorough way to clearly satisfy the requirement that your project is all about computing. If you are confident with software development and enjoy this aspect of your studies, you may well want to combine design with an aspect of software development; it is completely up to you.

Most interaction design projects are iterative. In fact, a practical recommendation is that a solid project that is based on the interaction design module should have three different development iterations, or phases. The thinking behind this is that this gives you the best opportunity to demonstrate your skills. When you get to the end you may want to ask the question: has my project solved the problem I set out to solve?

Consider an example of an app that could to pre-order a coffee from a local café, or to book a slot for your dog in a pet grooming parlour (you can choose another example if you prefer). 

Iteration 1: First initial sketches

Designs can be presented (and detailed) using different levels of fidelity, which (of course) relates to the idea of detail. A low fidelity design (or sketch) might be a rough sketch on paper (which is known as a paper-based prototype). A high fidelity design might be a design that closely resembles the final product. There are other terms that you might want to consider too: the idea of a vertical prototype (a prototype that covers a small bit of functionality in a lot of depth), or a horizontal prototype (a prototype that covers all the functionality of a product, but in a small amount of depth).

You might begin your design by creating some low fidelity designs, perhaps using paper-based prototyping. In other words, writing a bunch of sketches. You might want to combine this with other approaches from interaction design, such as the writing of user profiles and scenarios. It is up to you. Whatever you do needs to link back to the requirements of whatever it is you are designing. Different products will treat requirements in different ways.

When you have completed your first round of sketches (which could be made either using a pen or pencil, or using a tool), you should show them to whoever has the problem that needs to be solved (the café owner, or manager of the pet grooming parlour) to carry out your first round of evaluations.

Iteration 2: Wireframes

When you have received feedback, and have further refined your requirements, it is then a good idea to try to flesh out your design. You might have a second round of sketches, or you might move from pencil and paper to a prototyping tool. Choose whatever tool or product works best for you and your project.

When you have completed you second set of designs, go back to your stakeholders to get some further feedback. Here you should get some useful data (information about what works well, and what needs to be looked at in a bit more detail) to further refine you prototype further.

Iteration 3: Higher Fidelity prototypes

A high fidelity prototype is all about showing what a final (or polished) version of a product or system might look like. With this final iteration (or phase) you might get elements of your designing working. You may well develop software and create databases. 

As well as considering prototypes in terms of low and high fidelity, there is another couple of dimensions that can be considered helpful: horizontal and vertical prototypes. A horizontal prototype is a demonstration of a design across all its key functionality, but to a limited depth. A vertical prototype, on the other hand, means that you implement a small amount of functionality in a lot of depth. With a vertical prototype, you may well ‘go deep’ with the technology; you may have to choose appropriate software components and frameworks, and justify your choice.

When it comes to your project report, you should share your high fidelity prototype by walking your examiner through a series of screenshots.

Depending on the aim of your project, you might stop at this point. You may well have demonstrated a lot of technical skill and knowledge in a lot of detail. If your project is less about practical software development and more about design, it would be a good idea to carry out a final evaluation of your prototype to answer the question: does my design do what it supposed to do?

Using the DECIDE framework

The DECIDE framework can help you to plan (and run) an evaluation. It is featured within the interaction design module and its accompanying set text. Taking a letter at a time, here are the components of the framework: 

Determine the goals (of your evaluation): What are you aiming to get from your evaluation? Remember, this isn’t the goals of your project, it is the goals of your evaluation. What do you need to do to determine whether it is a success.

Explore the questions: What questions are you trying to find answers for? Or, alternatively, what questions do you need to ask to carry out an evaluation?

Choose the evaluation approach or paradigm: How are you going to approach your evaluation? You can choose a number of different approaches. You might wish to carry our a heuristic evaluation (which doesn’t involve users), a predictive evaluation (to predict how efficient your design is, perhaps using a cognitive walkthrough approach), or user testing (which can involve real users).

Identify the practical issues: This point relates to the detail of how you run your evaluation. If you need participants, how will you go about recruiting them? How will you ensure they are representative? How will you collect data? Will you be making notes on a data collection form, or if you’re carrying out an interview, will you be using the voice recorder app on your mobile phone? When you have captured data, where will you be storing your data, and how will you be making sure your data is secured.

Decide about the ethical issues: If you involve anyone in your evaluation, you need to think about ethics. Essentially, you need to gain permission, and you need to make sure that everyone knows what your evaluation is all about. An important point to emphasise is that you’re carrying out an evaluation of the performance of a product or a solution rather than the performance of your participants. Remember that your participants are always more important than the task of completing your evaluation. 

Evaluate: With your evaluation all planned, your data collection approach chosen, and permission gained, it is now time to carry out your evaluation. After you’ve gathered your data, it is time to analyse your data, interpret your data and then share your findings. In terms of your project report, this will mean the need to write a summary or conclusion.

Carrying out your evaluation

The DECIDE framework helps you to understand why you intend to carry out your evaluation, what criteria for success you consider to be important, and how you intend to carry out your evaluation.

If your project has an evaluation section, a recommendation is to show what you did in series of step by steps. In other words, adopt a narrative approach. Use a series of appendices to share the resources you may have created during the course of your evaluation. If participants are involved in your evaluations, consider taking a photograph of the environment in which an evaluation takes place, sharing that photograph within the body of your report.

Reflections

Although I have suggested three phrases or iterations is often appropriate for an interaction design project, there are no ‘right’ number of iterations. Every project is different. Whilst carrying you’re your project you may discover that you need more phases, and more evaluations.

References

Rogers, Y., Sharp, H., Preece, J. (2023) Interaction Design: Beyond Human-Computer Interaction, 6th Edition. Wiley.

Permalink Add your comment
Share post
Christopher Douce

AL Development: Sketching and Prototyping, London

Visible to anyone in the world

On the evening of 8 December 2016, my staff tutor colleague, Asma, set up and ran an associate lecturer development event for tutors who were working on a number of design modules. Incidentally, this was also one of the last AL development events that were run in the London regional centre, before it closes at the end of January 2017.

I usually take notes during these AL development events, so I can share some notes to everyone afterwards, but I became pretty busy chatting to everyone which meant that I didn’t have the time. This blog post is, subsequently, a pretty short one, since I’m relying purely on my fallible memory.

The event was advertised to design tutors in two Open University regional areas: in London, and in the South East. Although design tutors were the main ‘target group’, the event was also open to tutors who worked on a module called TM356 Interaction Design and the User Experience (OU website). The aim of the event was to share tips and techniques about prototyping and sketching. These techniques could then, in turn, be shared with students during face to face tutorial sessions.

The session was really informal. It was, in essence, a kind of show case. Different activities and demonstrations were placed throughout the room on different tables, and participants were invited to ‘experience’ sets of different activities. One activity was all about sketching using shade, lines and texture (if I remember correctly). Another was a scene where we could practice still life drawing. In fact, we had a choice: a set of shells, or a set of objects which represented our location.

A collection of objects that represent London as a tourist attraction

I remember two other demonstrations or ‘stands’: one was about the creation of physical prototypes and another was a show and tell about how different drawing and sketching techniques could be used to represent different product designs. I was particularly taken by the physical prototyping demonstration: we were shown card, bendy steel wire (which could be easily bought in a hardware store), and masking tape. The wire, we were told, could be used to add structure to physical objects; pieces of wire could be bent and twisted together, and taped onto the back of segments of card, to create the surfaces of objects.

I tried my hand at sketching, but I have to confess that I didn’t get too far: I soon became engaged in discussions about how these different techniques might be useful during a longer tutorial about physical prototyping. Another thought was: how could we replicate these kinds of prototyping and interactive activities when we have to use online tools? Or put another way, how could we run sessions when students can’t physically get to a classroom. It is clear that there no easy answers; I now wish that I had made better notes of all the discussions!

Not only were we all exposed to a number of different techniques, some of the tutors also had an opportunity to catch up with each other and chat about how a new module was going.

An interesting question is: could it be possible to run an online equivalent of this session? The answer is: possibly, but it would be very different, and it would require a huge amount of planning to make it work: things don’t spontaneously happen in the online world like they can during a face to face session.

Although the office is closing, there are different planning groups that are starting up to try to make sure that essential associate lecturer development activities still continue. I’m not sure when there will be another face to face session quite like this, but I do hope we can organise another one.

Permalink Add your comment
Share post

This blog might contain posts that are only visible to logged-in users, or where only logged-in users can comment. If you have an account on the system, please log in for full access.

Total visits to this blog: 2364115