Testing & Debugging
However carefully a program is written, mistakes creep in. A bug is any error in the code that stops the program working the way it should. Testing is the process of running a program with carefully chosen data to reveal these bugs so they can be fixed before real users ever see them.
The purpose of testing is to check that a program meets its requirements, produces the correct output for every likely input, and keeps working without crashing. Good testing also proves that a program is robust — it copes sensibly with unexpected or silly data instead of falling over. It is worth remembering that testing can reveal that bugs exist, but it can never prove that a program is completely free of them.
Testing checks that a program meets its requirements, gives correct outputs for all likely inputs, and is robust enough to handle unexpected data. It can show that bugs are present but never prove that none remain.
Testing as you go, and testing at the end
Programmers test in two different ways at two different stages. Iterative testing happens all the way through development. Each time a small section or module is written, the programmer runs it, checks it works, fixes anything wrong, then moves on. Because problems are caught early and in small pieces of code, they are far easier to locate and put right.
Final testing, also called terminal testing, is carried out once the whole program is finished. Now that the separate modules are joined together, the aim is to check that the complete system works as one and meets every requirement. Some faults only appear when parts of a program interact, so this final stage catches problems that module-by-module testing can miss.
If you write two hundred lines of code and only test at the very end, a single wrong result could be hiding anywhere. By testing each module as you build it, you already know the earlier parts work, so any new bug is almost certainly in the code you just added. This is why iterative testing saves so much time compared with leaving everything to the end.
Two kinds of error: syntax and logic
A syntax error breaks the grammatical rules of the programming language. A missing bracket, a misspelled keyword, or a colon left off the end of a line all count. Because the translator cannot understand the code, a syntax error stops the program from being translated or run at all. The good news is that these errors are usually flagged straight away, often with the line number where the problem was found.
A logic error is sneakier. The code obeys all the grammar rules, so it runs perfectly happily — but it produces the wrong output. Using a plus sign where you meant a minus, or checking ‘greater than’ when you meant ‘greater than or equal to’, will not stop the program, yet the answers will be wrong. Because nothing crashes, logic errors are the hardest to spot.
A syntax error breaks the rules of the language and stops the program running or being translated. A logic error lets the program run but produces unexpected, incorrect output.
Suppose a program should add VAT to a price. If you write prnt(total) instead of print(total), the translator does not recognise the command and reports a syntax error before the program even runs. But if you accidentally multiply the price by 1.02 when VAT is 20 per cent, the program runs fine and simply gives the wrong total — that is a logic error. Same program, two very different kinds of mistake.
Choosing good test data
Testing only with sensible, everyday values is not enough. A thorough test tries the edges and the rubbish too. There are four types of test data you should be able to choose for any scenario. Imagine a program that accepts a school year group, where only whole numbers from 7 to 11 are allowed.
- Normal test data is data that should be accepted without causing errors — sensible, typical values. Example: 9.
- Boundary test data is data of the correct type that sits right on the very edge of what is allowed. You test both sides of each edge: 7 and 11 should be accepted, while 6 and 12 should be rejected.
- Invalid test data is data of the correct type that should still be rejected. Example: 15 — it is a whole number, but it lies outside the allowed range.
- Erroneous test data is data of the wrong type altogether that should be rejected. Example: the word ‘nine’, because text cannot be a year-group number.
The difference between invalid and erroneous data is the data type. Invalid data is the correct type but out of range and should be rejected. Erroneous data is the wrong type entirely and should also be rejected.
Recording tests: the test plan
Professionals do not test at random and hope for the best. They write a test plan — a table listing each test, the data used, the type of test, the result expected, and the result that actually happened. If the actual result matches the expected result, the test passes; if not, a bug has been found and needs fixing. A strong test plan includes normal, boundary, invalid and erroneous data so that nothing is left unchecked.
| Test data | Type | Expected result |
|---|---|---|
| 9 | Normal | Accepted |
| 7 | Boundary | Accepted |
| 12 | Boundary | Rejected |
| 15 | Invalid | Rejected |
| ‘nine’ | Erroneous | Rejected |
Debugging: finding and fixing bugs
When a test fails, the next job is debugging — tracking down the exact cause of a bug and correcting it. Syntax errors are usually easy: the translator points to the line and often describes what is wrong. Logic errors take far more detective work, because the program runs without complaint and gives no clue where it went astray.
The most reliable way to hunt a logic error is to follow the code by hand, one line at a time, writing down how each variable changes. This is called a dry run, and the table you fill in is a trace table. Comparing the values the variables actually hold against the values they should hold reveals exactly where the program goes wrong. Adding temporary lines that print a variable’s value at key points does a similar job while the program is running.
Faced with a failing program, work methodically. First, read any error message carefully — it often names the line and the problem. Next, narrow down where the fault lies by checking the output of each module in turn. Then trace the suspect section by hand to see where a variable takes the wrong value. Finally, make one change at a time and re-test, so you always know which fix actually worked. Changing lots of things at once usually just creates new bugs.
Refining the solution
Testing and debugging feed into refining — improving the program once faults have been found. Refining might mean correcting an error, but it can also mean tightening up the logic, adding a check for input the programmer had not thought of, or rewriting a clumsy section so it runs faster or reads more clearly. Development is a cycle: write, test, debug, refine, then test again until the program is genuinely reliable.