Personal Blogs

The term ‘software testing’ can be associated with a very simple yet essential question: ‘does it do what it supposed to do?’

There is, of course, a clear and obvious link to the topic of requirements, which express what software should do from the perspective of different stakeholders. A complexity lies in the fact that different stakeholders can have requirements that can sometimes conflict with each other.

Ideally it should be possible to trace software requirements all the way through to software code. The extent to which formal traceability is required, and the types of tests you need to carry out will depend on the character of the software that you are building. The tests that you need for a real-time healthcare monitor will be quite different to the tests you need for a consumer website.

Due to the differences in the scale, type and character of software, software testing is a large topic in software engineering. Chapter 5 of SWEBOK v4, the software engineering body of knowledge highlights different levels of test: unit testing, integration testing, system testing, and acceptance testing. It also highlights different types of test: conformance, compliance, installation, alpha and beta, regression, prioritization, non-functional, security, privacy, API, configuration, and usability.

In the article, The Practical Test Pyramid, Ham Vocke describes a simple model: a test pyramid.  At the bottom of the pyramid are unit tests that test code. These unit tests run quickly. At the top, there are user interface tests, which can take time to complete. In the middle there is something called service tests (which can also be known as component tests). Vocke’s article is pretty long, and quickly gets into a lot of technical detail.

What follows are some highlights from some Software Engineering radio episodes that are about testing. A couple of these podcasts mention this test pyramid. Although testing is a broad subject, the podcasts that I’ve chosen emphasise unit testing.

The first podcast concerns the history of unit testing. The last podcast featured in this article offers some thoughts about where the practice of ‘testing’ may be heading. Before sharing some personal reflections, some other types of test are briefly mentioned.

The History of JUnit and the Future of Testing

Returning to the opening question, how do you know your software does what it supposed to do? A simple answer is: you get your software to do things, and then check to see if it has done what you expect. It is this principle that underpins a testing framework called JUnit, which is used with software written using the Java programming language.

The episode SE Radio 167: The History of JUnit and the Future of Testing with Kent Beck begins with a short history of the JUnit framework (3:20). The simple idea of JUnit is that you are able to write tests as code; one bit of code tests another. All tests are run by a test framework which tells you which tests pass and which tests fail. An important reflection by Beck is that when you read a test, it should tell you a story. Beck goes on to say that someone reading a test should understand something important about the software code. Tests are also about communication; “if you have a test and it doesn’t help your understanding … it is probably a useless test”.

Beck is asked to explain the concept of Test Driven Development (TDD) (14:00). He describes it as “a crazy idea that when you want to code, you write a test that fails”. The test only passes when that code that does what the test expects. The podcast discussion suggests that a product might contain thousands of tiny tests, with the implication that there might be as much testing code as production code; the code that implements features and solves problems.

When considering the future of testing (45:20) there was the suggestion that “tests will become as important to programming as the compiler”. This implies that tests give the engineers useful feedback. This may be especially significant during periods of maintenance, when code begins to adapt and change. There was also an expression of the notion that engineers could “design for testability” which means that unit tests have more value.

Although the podcast presents a helpful summary of unit testing, there is an obvious question which needs asking, which is: what unit tests should engineers be creating? One school of thought is that engineers should create tests that cover as much of the software code as possible, also known as code coverage. Chapter 5 of SWEBOK shares a large number of useful test techniques that can help with the creation of tests (5-10).

Since errors can sometimes creep into conditional statement and loops, a well known technique is known as boundary-value analysis. Put more simply, given a problem, such as choosing a number of an item from a menu, does the software do what it supposed to do if the highest number is selected (say, 50)? Also, does it continue to work if the highest number just before a boundary is selected (say, 49)?

Working Effectively with Unit Tests

Another podcast on unit testing is SE Radio 256: Jay Fields on Working Effectively with Unit Tests. Between 30:00 and 33:00, there is an interesting discussion that highlights some of the terms that feature within Vocke’s article. A test that doesn’t cross any boundaries and focus on a single class could be termed a ‘solitary unit test’. This can be contrasted with a ‘sociable unit test’, where tests work together with each other; one test may influence another. Other terms are introduced, such as stubs and mocks, which are again mentioned by Vocke.

Automated Testing with Generative AI

To deliberately mix a metaphor, a glimpse of the (potential) future can be heard within SE Radio 633: Itamar Friedman on Automated Testing with Generative AI. The big (and simple) idea is to have AI helper to have a look at your software and ask it to generate test cases for you. A tool called CoverAgent was mentioned, along with an article entitled Automated Unit Test Improvement using Large Language Models at Meta (2024). A point is: you still need a software engineer to sense check what is created. AI tools will not solve your problems, since these automated code centric tools know nothing of your software requirements and your software engineering priorities.

Since we are beginning to consider artificial intelligence, this leads onto another obvious question, which is: how do we go about testing AI? Also, how do we make sure they do not embody or perpetuate biases or security risks, especially if they are used to help solve software engineering problems.

Different types of testing

The SWEBOK states that “software testing is usually performed at different levels throughout development and maintenance” (p.5-6). The key levels are: unit, integration, system and acceptance.

Unit testing is carried out on individual “subprograms or components” and is “typically, but not always, the person who wrote the code” (p.5-6). Integration testing “verifies the interaction among” system under test components. This is testing where different parts of the system are brought together. This may need different test objectives to be completed. System testing goes even wider and “is usually considered appropriate for assessing non-functional system requirements, such as security, privacy, speed, accuracy, and reliability” (p.5-7). Acceptance testing is all about whether it is accepted by key stakeholders, and relate back to key requirements. In other words, “it is run by or with the end-users to perform those functions and tasks for which the software was built”.

To complete a ‘test level’ a number of test objectives may need to be satisfied or completed. The SWEBOK presents 12 of these. I will have a quick look at two of them: regression tests, and usability testing.

Regression testing is defined as “selective retesting of a SUT to verify that modifications have not caused unintended effects and that the SUT still complies with its specified requirements” (5-8). SUT is, of course, an abbreviation for ‘system under test’. Put another way a regression test check to make sure that any change you have made hasn’t messed anything up. One of the benefits of unit testing frameworks such as JUnit is that it is possible to quickly and easily run a series of unit tests, to carry out a regression test.

Usability testing is defined as “testing the software functions that support user tasks, the documentation that aids users, and the system’s ability to recover from user errors” (5-10), and sits at the top of the test pyramid. User testing should involve real users. In addition to user testing there are, of course, automated tools that help software engineers to make sure that a product deployment works with different brewers and devices.

Reflections

When I worked as a software engineer, I used JUnit to solve a very particular problem. I needed to create a data structure that is known as a circular queue. I wouldn’t need to write it in the same way these days since Java now has more useful libraries. At the time, I needed to make sure that my queue code did what I expected it to. To give me confidence in the code I had created, I wrote a bunch of tests. I enjoyed seeing the tests pass whenever I recompiled my code.

I liked JUnit. I specifically liked the declarative nature of the tests that I created. My code did something, but my tests described what my code did. Creating a test was a bit like writing a specification. I remember applying a variety of techniques. I used boundary-value analysis to look at the status of my queue when it was in different states: when it was nearly full, and when it was full.

Reflecting Beck, I appreciated that my tests also told a story. I also appreciated that these tests might not only be for me, but might be useful for other developers who might have the misfortune of working with my code in the future.

The other aspect of unit testing that I liked was that it proactively added friction to the code. If I started to maintain it, pulling apart function and classes, the tests would begin to break. The tests became statements of ‘what should be’. I didn’t view tests in terms of their code coverage (to make sure that every single bit of software was evaluated) but in terms of simple practical tools that gave alternative expressions of the purpose of my software. In turn, they helped me to move forward.

It is interesting and useful to reflect on the differences between the test pyramid and the SWEBOK test levels. In some respect, the UI testing of the pyramid can be aligned with acceptance testing of the SWEBOK. I do consider the integration and system testing to be helpful.

An important point that I haven’t discussed is the question of when should a software engineer carry out testing? A simple answer is, of course, as soon as practically possible. The longer it takes to identify an issue, the more significant the impact and the greater the economic cost. The ideal of early testing (or early problem detection) is reflected in the term ‘shift-left’ testing, which essentially means ‘try to carry out testing towards the left hand side of your project plan’. Put even more simply: the earlier the better.

Returning to the overriding aim of software testing, testing isn’t just about figuring out whether your software does what it supposed to do. It is also about managing risk. If there are significant societal, environmental, institutional and individual impacts if software doesn’t work, you or your organisation needs to do whatever it can to ensure that everything is as correct and as effective as possible. Another point is that sometimes the weak spot isn’t the code, but in the spaces where people and technology intersects. Testing is socio-technical.

To conclude, it is worth asking a final question. Where is the software testing heading? Some of these podcasts suggest some pointers. In the recently past, we have seen the emergence of automation and the engineering of software development pipelines to facilitate continual deployment or delivery of software. I do expect that artificial intelligence, in one form or another, will influence testing practice, but AI tools can’t know everything about our requirements. There will be testing using artificial intelligence and testing of artificial intelligence. As software reaches into so many different areas of society, there will also be testing for sustainability.

Resources

JUnit is one of many bits of technology that can help to automate software testing. Two other  tools s I have heard of are called Cucumber which implements a language called Gherkin, a formal but human-readable language which is used to describe test cases. I’m also aware of something called Selenium which is “a suite of tools for automating web browsers”.

Since software testing is such an important specialism within software engineering, there are a series of industrial certifications that have been created by the International Software Testing Qualifications Board (ISTQB). As well as offering foundation level certifications, there are also certifications for specialisms such as agile, security and usability. Many of the topics mentioned in the certifications are also mentioned in Chapter 5 of SWEBOK v4.

I was alerted to a site called the Ministry of Testing which shares details of UK conferences and events about testing and software quality.

One of the points that I picked up from the podcasts was the point that, when working at forefront of an engineering subject, there is a lot of sharing that takes place through blogs. A name that was mentioned was Dan North who has written two articles that resonate: We need to talk about testing (or how programmers and testers can work together for a happy and fulfilling life), and Introducing BDD (BDD being an abbreviation for Behaviour Driven Development).

Acknowledgements

Many thanks to Josh King, a fellow TM354 tutor, who was kind enough to share some useful resources about testing.

From time to time, I dip into (and out of) a podcast series called Software Engineering Radio, which is produced by the IEEE. It’s a really useful resource, and one that I’ve previously mentioned in the blog post Software engineering podcasts.

This is the first of a series of blog posts that shares some notes I’ve made from a number of episodes that I have found especially interesting and useful. In some ways, these posts can be through of a mini course on software engineering, curated using the voices of respected experts.

Towards the end of each blog, I share some informal thoughts and reflections. I also share some links both earlier posts and other relevant resources.

I hope this series is useful for students who are studying TM354 Software Engineering, or any other OU module that touches on the topic of software engineering or software development.

Software engineering as a discipline

When doing some background reading (or listening) for TM113, I found my way to SE Radio 149: Difference between Software Engineering and Computer Science with Chuck Connell.

In this episode, there are a couple of sections that I bookmarked. The first is 10:20 through to 12:20, where there is a discussion about differences between the two subjects. Another section runs between 24:10 and 25:25, where there is an interesting question: is software engineering a science, an art, or a craft? The speaker in the podcast shares an opinion which is worth taking a moment to listen to.

According to the software engineering body of knowledge (SWEBOK), engineering is defined as “the application of a systematic, disciplined, quantifiable approach to structures, machines, products, systems or processes” (SWEBOK v4.0, 18-1). Put in my own words, engineering is all about building things that solve a problem, in a systematic and repeatable way that enables you to evaluate the success of your actions and the success of what you have created.

An early point in chapter 18 of SWEBOK is the need to: “understand the real problem” which is expanded to the point that “engineering begins when a need is recognized and no existing solution meets that need” (18-1). Software, it is argued, solves real world problems. This takes us to a related question, which is: how do we define what we are building? This takes us to the next post, which is all about requirements.

Before having a look at requirements, it is useful to break down ‘software engineering’ a little further. The SWEBOK is divided into chapters. The chapters that begin with the word ‘software’ are: requirements, architecture, design, construction, testing, maintenance, configuration management, engineering management, engineering process, models and methods, quality, security, professional practice, economics.

There are three others which speaks to some of the foundations (and have the word ‘foundation’) in their title. They are: computing, mathematical, and engineering.

Reflections

Software is invisible. It is something that barely exists.

The only way to get a grasp on this ‘imaginary mind stuff’ of software is to measure it in some way. The following bits of the SWEBOK is helpful: “Knowing what to measure, how to measure it, what can be done with measurements and even why to measure is critical in engineering endeavors. Everyone involved in an engineering project must understand the measurement methods, the measurement results and how those results can and should be used.” (SWEBOK v4, 18-10). The second sentence is particularly interesting since it links to the other really important element in software engineering: people. Specifically, everyone must be able to understand the same thing.

The following bit from the SWEBOK is also helpful: “Measurements can be physical, environmental, economic, operational or another sort of measurement that is meaningful to the project”.

Next up is software requirements. By writing down our requirements, we can begin to count them. In turn, we can begin to understand and to control what we are building, or working with.

In an earlier blog article, I described the structure of an undergraduate honours degree in terms of academic credits (or points). Degrees are organised into modules, and modules attract credit. To gain a degree you need 360 credits.

I am sometimes asked the following question: ‘is a degree from a brick university the same as a degree from the OU?’ The answer is, ‘of course it is’. A related question is, of course, ‘who defines what goes into a degree that makes them the same?’ 

This is where the QAA comes in. QAA, an abbreviation for the Quality Assurance Agency (for Higher Education) is a government quango (a quasi-non-governmental organisation) that defines what goes into a degree. Academics who are writing and delivering modules, and external examiners who help to maintain academic standards all need to know about QAA Benchmark Statements.

Here’s a definition of what they are, taken from the QAA website: “Subject Benchmark Statements describe the nature of study and the benchmark academic standards expected of graduates in specific subject areas, and in respect of particular qualifications. They provide a picture of what graduates in a particular subject might reasonably be expected to know, do and understand at the end of their course or programme.”

What follows is a mild ‘deep dive’ into a couple of subjects: Computing (which I’m involved with teaching), and English Literature (which I’m currently studying). If you are not interested in either of these, you can still find your own subject by having a rummage through the QAA website. An interesting activity would be to see how the learning outcomes for the module that you are tutoring are expressed on the QAA website.

I begin with computing, looking at computer science. I then move to look at Artificial Intelligent (AI) and then software engineering (which is a module I have some responsibility for), before looking at any accompanying QAA guidance that relates to project modules. This is followed by a contrasting look at English. I conclude with some reflections. To give you a good idea of what is contained within the QAA materials, I’m taking liberty of quoting extensively.

Computing

The QAA Computing benchmark statements are presented in two documents: a summary document (the basics), and a more detailed subject statement. At the time of writing, both documents were dated March 2022.

I began by looking at ‘the basics’ document. The purpose of a computing degree is to gives students “the opportunity to explore creative and dynamic technologies” and to “improve their employment prospects in a rapidly evolving global digital skills economy” (p.3). I was also drawn to a comment about assessment, which includes varied methods that are accessible to all students. Assessments should be authentic and tied to real-world contexts and constraints, allowing students to practically demonstrate the skills they have developed” (p.2).

Turning to the larger document, IEEE and ACM computing curricula (1.2, p.3). Before digging to the ACM curriculum and important subtopics within computing, below is a summary of points from the first three sections of the statement that struck me as being interesting and important. 

• Sustainability is to be addressed throughout the curriculum. The curriculum should make students aware of ‘the resource consumption of massive data centres used for cloud computing, and more generally, the environmental costs of both building hardware to support computing and disposing of electronic waste’ (1.16, p.6).
• On the topic ‘content, structure and delivery’, ‘educators should consider the balance between subject-specific and transferable skills development between educational and workforce requirements as courses are developed and maintained’ (3.4., p.15). In other words, learning activities should help students to develop skills that are useful in the workplace.
• On the topic of pedagogy, ‘teaching should encourage students to reflect, evaluate, select, justify, communicate and be innovative in their problem-solving; and prepare them to become adaptable independent learners throughout their lifelong learner-earner journey.’ (3.9, p.16)
• On the topic of defining what is to be learnt, computing ‘curriculum will define the knowledge students will gain and the course learning outcomes indicate the areas in which graduates will have knowledge competence or capability.’ (3.12, p.17)
• Computing is also a practical subject: ‘[t]he hardware and software resources available should facilitate a practical approach to the delivery of the course’ (3.13, p.17). 

Section 4 of the document summarises the benchmark standards, which are divided into the following areas: subject knowledge, understanding and skills, intellectual skills, computational problem solving, practical skills across the computing lifecycle, interpersonal and team working, and professional practice. The professional practice topic includes the importance of working with a legal and ethical framework, understanding the important of equality, diversity and inclusion, and sustainability.

Three levels of attainment are outlined: threshold (students must demonstrate a requisite understanding to get a pass), typical (students should demonstrate a sound understanding), excellent (students are to show an exceptional understanding, showing an ability to be creative and innovative).

Curricula summaries

Appendix 3 of the benchmark statement shares some further guidance about what should go in computer science degrees. Educators should be aware of the ACM (Association of Computing Machinery) curricula recommendations. From here, it is possible to find your way to a digital book, Computer Science Curricula 2023 by Kumar et al. (2023) which was published in January 2024. It is useful to note that the document summarises ‘core’ and ‘non-core’ subjects. 

There are connections between the subject of computing and related named degrees, such as artificial intelligence and software engineering, which are sketched below. This is followed by a brief summary about the importance and significance of project work.

Artificial intelligence

Artificial intelligence (AI) can be a named degree, but the QAA does not publish its own benchmark statement that is specific to AI. Instead, at the time of writing, it falls under the subject of computing. 

In the ACM 2023 Computer Science curriculum, AI is covered from pages 65 to 86. Two notable points summarise some recent changes: ‘importance of understanding and assessing the broader societal impacts and implications of AI methods and applications, including issues in AI ethics, fairness, trust, and explainability’ and ‘there are explicit goals to develop basic AI literacy and critical thinking in every computer science student’ (p.66).

Important AI topics include searching (for solutions), the use of heuristics, knowledge representation, machine learning, applications and (as mentioned) the societal impact of AI. Machine learning is a significant topic since it can relate to different approaches, and highlights the importance of working with data. Other areas (which are currently considered non-core) includes natural language processing, robotics, and computer vision.

Software engineering

Software engineering, like AI, can be a named degree. Appendix 3 of the QAA benchmark standard (p.34), contains a useful summary of software engineering, suggesting that students need to understand:

• problem definition, specification (including formal specification), design, implementation (including debugging) and maintenance, software testing, change management and documentation;
• cybersecurity, including information security, and safety-critical systems;
• understanding risk, reliability and scalability of the range of possible options and an appreciation of design trade-offs.  (QAA Computing Subject Benchmark Statement, 2022, p.34)

ACM offers some specific guidance about Software engineering courses, specifically SE2014 Curriculum Guidelines for Undergraduate Degree Programs in Software Engineering. In the recent 2023 ACM curriculum book software engineering is covered on pages 237 to 254. Topics that courses should cover includes teamwork, tools, requirements (functional and non-function, which can include sustainability), design, software construction, software refactoring, software reliability, and formal methods. It is interesting to note that formal methods has now been relegated to being a ‘non-core’ subject, which reflects both current trends and practices.

Another source of guidance for educators is something called the Software Engineering Body of Knowledge, which is also known as the SWEBOK. The most recent version, SWEBOK V4 was made available towards the end of 2024

Computing projects

When I was an undergraduate in computing in the 1990s, I had to do a computing project towards the end of my degree. The OU’s named Computing degree is formally accredited by the British Computer Society (BCS). This means that students must complete a project to gain a specific named degree.

The QAA benchmark guidance offers comments on the notion of a project in paragraph 3.19: “[c]omputing courses often conclude with a capstone activity, which brings together knowledge and practical and analytical skills that learners have developed throughout the course. This may take the form of a traditional project or end-point assessment, but other formats can be appropriate, whether research or practice-led” (p.18). Students who are studying on degree apprentices will work on a project that integrates together professional practice and independent study. 

The 2023 ACM curriculum book also offers a bit of guidance about projects, suggesting that ‘many of the fundamental topics of software engineering are best experienced in a hands-on fashion’ (p.238) whilst also emphasising the importance of feedback.

English Literature

The QAA benchmark standard for English was published in 2023

It shares the following characteristics of an English degree:

• 'English involves the study of language, literature and the practice of creative writing’ (p.3)
• ‘The study of language, of literary and cultural works, and the production of critical and creative work, enable students to interpret and interrogate past and present cultures, to anticipate their future transformations, and to enhance their ability to understand themselves, other people and our shared world.’ (p.3)
• English graduates should be able to: read critically, analyse texts, reflect critically, develop persuasive interpretations, articulate an understanding of complex texts, communicate, and apply a scholarly approach.

Just like the computing benchmark standards, there are three levels: threshold, typical, and excellent (p.14). To gain a pass, students must meet the threshold level. The benchmark is split into categories: generic skills; subject knowledge; understanding of EDI (equality, diversity and inclusion), access, sustainability and entrepreneurship.

Subject the subject knowledge is understandably quite descriptive, highlighting the importance of close reading of texts, helping students appreciate that different texts can carry multiple meanings, help students to understand the ‘rhetorical properties of texts’ (p.16), including pre-1800 texts, and how culture of the time would have influenced the production of texts. 

When looking through the benchmark standard, the following struck me as both interesting and important:

• ‘English courses often link literature, language or creative practice to sustainability and environmental challenges through exploring the relationship between humans and the natural environment across different periods and discourses, and between different modes and genres of writing. Ecocriticism places the environment at the centre of its interpretative focus.’ (1.15, p.5)
• ‘Creativity is fundamental to all aspects of English. Students of English demonstrate open-mindedness, initiative and independent thinking, as well as abilities in innovation, problem-solving and solution finding.’ (1.21, p.6)
• ‘The subject of English enhances the critical and analytic thinking that is crucial for enterprise and entrepreneurship, by encouraging curiosity, questioning, observation, pattern recognition, and problem identification.’ (1.22, p.6)

Unlike the computing subject benchmarks, there are no detailed appendices that provide pointers to other curriculum summaries.

Reflections

This has become more than a ‘quick’ look at QAA. To prepare this blog, I actually had about three goes of looking through it and reflecting on it before compiling this post.

There are some clear similarities and differences between the computing and English standards. Beginning with computing, I felt the QAA guidance tries to offer a balance of providing sufficient detail to be useful, whilst at the same time offering pointers to organisations which are able to offer a more comprehensive summary of the state of the curriculum in a fast moving field.

I was struck by the extent to which sustainability was emphasised, and I was impressed that it provided pointers toward helpful papers. When digging further into the computing curricula reports, I was struck that computing history is considered to be non-core. Computing clearly has an history, and understanding of how things are they way they are is important to understanding computing practices and tools.

Turning to English, the benchmark statement felt more complete, but I don’t think that should come as a surprise. It was striking was that nothing was mentioned about the canon; texts that are broadly recognised as being significant and worthy of study. This said, the standard wasn’t specific to English literature. This begs the question: do English academics have more of a free reign to choose texts? I have heard of instances where lecturers and students have worked together to study texts that they themselves identify as important.

There are clear points of similarity. There is the importance of skills, knowledge and critical thinking. Both subjects emphasise the importance of academic practice; what it means to study the respective subjects at a degree level.

It is interesting to see close reading emphasised in the English benchmark; it is certainly emphasised within the modules that I have been studying. Computing students have to do close reading too. Rather than analysing novels and poems, computing students need to analyse code. Code has, of course, two audiences: the machine, and software engineers working within the culture in which the software exists. There’s another important similarity, which is perhaps emphasised more in the English degree than it is in the computing degree: the importance of creativity. To get things done, computing students need to study a range of sources (different software components) and combine them together in creative ways.

So, what have I learnt from all this? I now know that the benchmark standards have a consistent structure, and I’ve given myself even more curriculum guidance to look through. I’m going to assume that’s a good thing.

Acknowledgements

I have drawn on the QAA subject benchmarks statements extensively in this post. They are worth spending a bit of time looking at. This article has been written as a part of an eSTEeM project about tutor practice.