9 Testing, Error Checking & Debugging
Writing code is easy. Finding out whether it works, and fixing it if it doesn’t, is the hard part. It is important to recognize that a program is “finished” only when it has been verified to work under all conditions that it may encounter. An often overlooked issue is handling user errors gracefully – a program should not “crash” if the user provides an invalid input. Rather, it should, whenever possible, provide useful feedback to the user and continue running.
This chapter covers some MATLAB features for testing, debugging and error checking to help make programs more reliable and robust.
9.1 Testing
The most important principle in verifying a large, modular program is to follow a bottom-up approach. That is, test each function prior to integrating them and testing the overall program. Honda would not assemble a car from untested parts and hope that the vehicle works. Likewise, trying to assemble and test a complex program without first verifying that any of its pieces work is a recipe for endless headaches and wasted time.
Consider this function, which converts a character array representing a Roman number into the equivalent decimal value. (This is essentially the same as an example from the previous chapter. It is not essential to fully understand the algorithm that the function uses to follow the point that we are making with the example.)
function [ value ] = romanToDecimal( in_string )
%Accepts a character array representing a Roman number
%Returns the equivalent decimal integer
%start with the last numeral
sl = numel(in_string);
%find its value
value = digit_val(in_string(sl));
%keep track of the value of the last numeral
last_digit = value;
%keep track of how many consecutive matching numerals there have been
rc = 1;
%Look at all numerals from the second to last, back to the first
for place = sl-1:-1:1
next_digit = digit_val(in_string(place));
%If the next digit is greater than the last one, add its value
if next_digit > last_digit
value = value + next_digit;
rc = 0;
%If the next digit is less than the last one, subtract its value
elseif next_digit < last_digit
value = value - next_digit;
%If the next digit is the same as the previous one, increment
%the number of consecutive matching numerals; if that number
%doesn't exceed three, add the value of the numeral; otherwise,
%the number isn't valid - return NaN
else
rc = rc + 1;
if rc > 3
disp('That is not a valid Roman number.')
value = NaN;
return;
else
value = value + next_digit;
end
end
last_digit = next_digit;
end
end
%Helper function to convert a single Roman digit to its decimal value
function [digval] = digit_val(digit_char)
digits = ['I', 'V', 'X', 'L', 'C', 'M'];
digit_vals = [1, 5, 10, 50, 100, 1000];
for i = 1:size(digits,2)
if digit_char == digits(i)
digval = digit_vals(i);
end
end
end
To test the function we might call the function on a representative example:
>> romanToDecimal ('XXII')
ans =
22
The function produced the correct answer, so it would be tempting to assume that it works. But it may not work for all valid inputs. For example, suppose the input is the same as in the previous example, but in lower case:
>> romanToDecimal ('xxii')
Output argument "digval" (and maybe others) not assigned during call to "roman3>digit_val".
Error in roman3 (line 10)
value = digit_val(in_string(sl));
The writer of this function did not anticipate that the user might pass the input in lower case, so it did handle this case correctly. (We will discuss how to fix the problem below.)
Similarly, the user might pass the input as a string rather than a character array:
>> romanToDecimal ("XXII")
Output argument "digval" (and maybe others) not assigned during call to "roman3>digit_val".
Error in roman3 (line 10)
value = digit_val(in_string(sl));
>>
In both of these cases, it could be argued that the user is at fault for not providing the input in the required format. However, a good program does not impose unnecessary constraints – if lower-case or string input can be accommodated without a large incremental effort, then it should be.
An improved version of the romanToDecimal() function is shown below. It corrects the two limitations mentioned above by converting the input string to a character array as needed and converting any lower case letters to upper case.
9.1.1 The Assert Statement and Unit Testing
A useful MATLAB feature for verifying code is the assert statement. This statement checks whether some specified condition is true, and if it isn’t, causes an error.[1] Here’s a very rudimentary demonstration of how assert( ) works:
>> assert (pi > 3.14)
>> assert (pi < 3.14)
Assertion failed.
If the assertion passes, the program simply continues. If the assertion fails, the program stops, unless the program “traps” the error, as discussed in a later section.
A typical set of unit tests for the romanToDecimal() function is given below. This is not necessarily an exhaustive set of tests – generating a comprehensive test set is a thorny problem – but it includes a variety of cases to verify that the function works with the range of expected inputs.
An error occurs during the last test, which asserts that the function should return 0 if the input is a null string. The function does not handle this case correctly, since in this line:
value = digit_val(in_string(sl));
it was assumed that sl (the length of the string) was non-zero. This special case could be handled by inserting a few lines as follows:
%start with the last numeral
sl = numel(in_string);
%Insert these lines to handle a null string
%****************************************
if sl == 0
value = 0;
return;
end
%****************************************
%find its value
value = digit_val(in_string(sl));
The return statement in MATLAB causes the function to terminate and return control to the script or function that called it. Unlike in other languages, the return statement does not return a value – remember that MATLAB functions return values through their output arguments.
Checkpoint
9.1.2 Constructing Unit Tests When the Answer is Unknown
In all of the previous examples, it was taken for granted that the correct answer was known. In more complex programs, that may not be the case. In a simulation, for example, you don’t know what answers to expect – otherwise there would be no need to run the simulation! However, there may be special cases for which the answer is known, and it is often possible to predict how a result will change if a parameter is varied. This can form the basis for a set of tests.
Consider a simulation of a falling skydiver that takes the drag coefficient and initial height as inputs and returns the height, velocity and time vectors as outputs.[2] The function definition might be
function [y, v, t] = parachute_simulation(initial_height, drag_coeff)
A set of unit tests can be based on the following physical principles:
- If the drag coefficient is zero, the problem reduces to free fall, and the exact solution is known.
- If the drag coefficient is increased, the maximum speed should decrease, and the time taken to reach the ground should increase.
A subset of the tests could be as follows:
% Assume that H0 and g have already been defined
% and that the function parachute_simulation( ) exists
% Drag-free case
[y0, v0, t0] = parachute_simulation (H0, 0);
% Check against exact solution - always allow for roundoff error
% To simplify, assume that v and g are positive numbers (i.e. +y axis is down)
assert ( abs(v(end) - sqrt(2*g*H0)) < 0.1);
assert ( abs(t(end) - sqrt(2*H0/g)) < 0.1);
%Test that max velocity and time to reach the ground follow the right trends
[y1, v1, t1] = parachute_simulation (H0, 0.5);
[y2, v2, t2] = parachute_simulation (H0, 1.0);
assert ( max(v1) < max (v0))
assert (t1(end) > t0(end))
assert (max(v2) < max (v1))
assert (t2(end) > t1(end))
9.1.3 Common Error Messages and Their Meaning
| Message | Typical Cause | Example |
| Unable to perform assignment because the indices on the left side are not compatible with the size of the right side. | Mismatched dimensions or type in an assignment statement | names(1) = input(‘Name?’,’s’);
|
| Too many output arguments. | Caller of a function expected the function to return a certain number of arguments, but the function returned less than that. Error may be in the caller or in the function. | %in script
|
| Not enough input arguments. | Function requires more input arguments than the caller provided | %in script
|
| Too many input arguments. | Caller provided more input arguments than the function expects | %in script
|
| Undefined function or variable x | Trying to use the value of x before assigning it. May be a typo or inconsistency in the variable name; did not initialize loop variable in a while loop | X = 5;
|
| Vectors must be the same length. | Plotting an (x,y) graph – x and y do not have the same dimensions; | t = 1;
|
| Array indices must be positive integers or logical values. | Trying to use 0, a negative number, or a non-number as an index | t = 0;
|
| Index exceeds array bounds. | Trying to access an array location that does not exist | t = 1;
|
Lecture Video 9.1 – Testing Using assert
9.2 Error Checking and Trapping
9.2.1 Checking Data Types
The improved version of romanToDecimal() used the built-in function isstring() to check if the input to the function needed to be converted from a string to character array. There are numerous similar functions in MATLAB, which can be used to verify that the input passed to a function has the correct data type or to alter the behavior of the function depending on the data type. A partial list is given in the table below. Each function returns true if the argument is of the specified type and false otherwise.
| Function | Description |
isnumeric() |
a numerical array, whether integer or floating point, including scalars |
isfloat() |
a floating point number or array |
isinteger() |
an integer number or array |
isrow() |
an array of dimensions 1xN, whether of numbers, characters, strings, or other data types |
iscolumn() |
an array of dimensions Nx1, whether of numbers, characters, strings, or other data type |
isscalar() |
an object of dimensions 1×1, whether numeric, character, string or other data type |
ismatrix() |
an array of dimensions NxM, where N and M are greater than or equal to 1 (By this definition, includes scalars and vectors) |
isempty() |
An array with a zero dimension (examples include a null string or empty cell) |
isnan() |
a numerical variable having the value NaN (not a number). May result from an undefined mathematical step such as dividing 0 by 0 |
iscell() |
a cell or cell array |
isstruct() |
a struct or struct array |
islogical() |
a logical (true / false) quantity or array |
ischar() |
a character array |
isstring() |
a string or string array |
isvector() |
an array having dimensions of 1xN or Nx1, whether numerical, character, string or other data type (By this definition, includes scalars) |
Checkpoint
9.2.2 Trapping Exceptions with Try / Catch
The try / catch structure allows a program to detect errors and handle them more gracefully than just halting the program – this is called “trapping” the error. The general structure of try / catch is
try
%block of code to execute, which could result in an error
catch
%block of code to execute if an error occurs
end
On application of this technique is handling cases in which the user of a program may have provided invalid inputs. As a simple example, consider a program that asks the user to enter the coefficients of a quadratic as an array, then calculates the roots. If the user does not enter an array, an indexing error will occur when the program tries to extract the individual coefficients. In that case, the program displays an error message, sets the roots to NaN, and continues.
coeffs = input ('Enter the coefficients in the form [a, b, c]: ');
try
a = coeffs(1);
b = coeffs(2);
c = coeffs (3);
r1 = (-b + sqrt(b^2 - 4*a*c))/(2*a);
r2 = (-b - sqrt(b^2 - 4*a*c))/(2*a);
catch
disp('Invalid coefficients: Must enter an array of 3 values')
r1 = NaN;
r2 = NaN;
end
fprintf('The roots are %.2f and %.2f \n', r1, r2)
The behavior of the program with valid and invalid inputs is as follows:
>> try_catch_example
Enter the coefficients in the form [a, b, c]: [1,4,1]
The roots are -0.27 and -3.73
>> try_catch_example
Enter the coefficients in the form [a, b, c]: 1
Invalid coefficients: Must enter an array of 3 values
The roots are NaN and NaN
>>
The try / catch structure can be used in conjunction with the assert statement in unit tests to allow subsequent tests to proceed after a failure. The unit tests from an earlier checkpoint questions could be restructured in the form:
num_failures = 0;
disp('Testing that pi is positive')
try
assert(pi > 0)
disp ('pi is positive')
catch
disp ('pi is not positive')
num_failures = num_failures + 1;
end
disp('Testing that pi is greater than 3')'
try
assert(pi > 3)
disp ('pi is greater than 3')
catch
disp ('pi is not greater than 3')
num_failures = num_failures + 1;
end
disp('Testing that pi is equal to 22/7')
try
assert(pi == 22/7)
disp ('pi equals 22/7')
catch
disp ('pi does not equal 22/7')
num_failures = num_failures + 1;
end
disp('Testing that pi is real')
try
assert (imag(pi) == 0)
disp('pi is real')
catch
disp ('pi is not real')
num_failures = num_failures + 1;
end
fprintf ('There were %i test failures \n', num_failures);
Assuming that the value of pi has not been changed from its normal value, the result of running this suite of tests is:
>> try_catch_w_assert
Testing that pi is positive
pi is positive
Testing that pi is greater than 3
pi is greater than 3
Testing that pi is equal to 22 / 7
pi does not equal 22/7
Testing that pi is real
pi is real
There were 1 test failures
>>
Notice that when an assert fails, the remaining instructions within that block are not executed. Thus “pi equals 22/7” is NOT printed.
Lecture Video 9.2 – Error Trapping with try/catch
Checkpoint
9.2.3 Advanced Topic – Exception Handling by Type
In previous examples of using try / catch, no attention was paid to what type of error occurred. In practice, it is useful to distinguish between different classes of errors and take different actions accordingly. This can be done by providing a variable in the catch statement to store information about the exception and using branching structure to choose an action based on the exception type.
The example below is taken from an auto-grading program. The assert statement compares the student’s answer (userVariable) to the correct answer (goodVariable) and raises an exception if they differ by more than a certain tolerance. However, there are several ways that the test could fail:
- If
userVariabledoes not have the correct dimensions, an error will occur when the subtraction is performed. The identifier for this error isMATLAB:dimagree - If
userVariableis not defined, again an error will occur on the subtraction. The identifier for this error isMATLAB:UndefinedFunction(undefined function and undefined variable produce the same exception) - If userVariable is defined and has the correct dimensions but an incorrect value, the assertion will fail. The identifier for this exception is MATLAB:assertion:failed
try
assert (abs((sum(userVariable - goodVariable))) < 1e-8)
disp('Test passed')
catch ME
switch ME.identifier
case 'MATLAB:assertion:failed'
fprintf('Calculation is incorrect\n');
case 'MATLAB:dimagree’
fprintf('XX Variable does not have correct dimensions\n')
case 'MATLAB:UndefinedFunction’
fprintf('XX Variable userVariable is not defined\n')
otherwise
fprintf('XX %s Error\n',ME.identifier);
end
end
The behavior of this code for each error type is shown below:
>> goodVariable = rand(1,5); %userVariable has not been defined
>> exception_handling_example
XX Variable userVariable is not defined
>> userVariable = rand(1,6); %dimensions are not consistent
>> exception_handling_example
XX Variable does not have correct dimensions
>> userVariable = rand(1,5); %values don't match
>> exception_handling_example
Calculation is incorrect
>> userVariable = goodVariable; %Arrays have the same value
>> exception_handling_example
Test passed
>>
A partial list of the most common exception identifiers is given below.
Table: Built-In Exception Identifiers for Common Errors
| Exception identifier | Cause |
MATLAB:assertion:failed |
Relational expression in the input of an assert statement evaluates to false. Example: assert (pi == 22/7) |
MATLAB:dimagree |
Incompatible array dimensions in an operation |
MATLAB:UndefinedFunction |
Variable of function is not defined |
MATLAB:assertion:LogicalScalar |
Input to an assert statement evaluates to a logical array rather than a logical scalar. Example: assert ([2, 4, 6, 8] > 1) |
MATLAB:scriptNotAFunction |
Attempt to call a script as if it were a function (i.e. by providing input arguments or expecting output arguments) |
MATLAB:TooManyInputs |
A call to a user-defined function had too many input arguments |
MATLAB:minrhs |
A call to a built-in or user-defined function did not have the minimum number of required input arguments. Example: y = atan2(1) |
MATLAB:maxrhs |
A call to a built-in function had more than the maximum number of input arguments. Example: y = atan2(1,2,3) |
MATLAB:TooManyOutputs |
A call to a user-defined function expected more output arguments than provided by the function. |
MATLAB:maxlhs |
A call to a built-in function expected more output arguments than provided by the function. Example: y = disp ('message') |
MATLAB:matrix:singleSubscriptNumelMismatch |
In an array assignment, the number of values on the right-hand side does not match the number of indices on the left-hand side. Example: V(1:3) = [1, 2] |
MATLAB:subsassigndimmismatch |
Similar to previous, but applies to a matrix. Example: M(2, 3:4) = [2, 4, 6] |
MATLAB:samelen |
x and y inputs to plot( ) do not have the same length |
9.3 Debugging
The most difficult part of programming is finding and fixing errors in code – what is known for historical reasons as “debugging”. In a way, the term “software bug” is unfortunate; a more accurate term is “programming error.” Bugs do not infest a program against the will of the programmer; they are incorporated by the programmer due to insufficient attention to detail. Nonetheless, a bug is not necessarily a sign of incompetence or negligence. It is extremely difficult to verify that an algorithm will work under all possible conditions until it has actually been implemented in software. Since few complex programs work perfectly on their first iteration, debugging is an essential skill.
Lecture Video 9.3 – Debugging
9.4 Mathworks Resources
Find an error? Have a suggestion for improvement? Please submit this survey.