XCTest Line by Line

Note: this article is up to date as of Swift 5.0 and Xcode 10.2 XCTest is Apple’s framework for testing code and user interfaces. It provides a variety of methods for testing equality, comparison, and error-throwing. However, the function names can be longer than the test input, making it hard to see what is going on. In this post, I’ll share a trick that I learned from Brian King, where we get clever with the built-in XCTest functions in order to provide clean line-by-line error reporting without the verbosity of using the built-in testing functions on every line.

A Type Worth Testing

First, we need to pick a type to use for our examples. I’m going to use a simple struct called Version, which represents software version numbers like “3.0.2”. It’s not as fancy as some version number types, but it will work well for our examples, because comparing version numbers is surprisingly complicated and easy to get wrong. Our Version type is pretty bare-bones: it represents a version number with major, minor, and bugfix components, and it can be initialized conveniently with a string.

A Buggy Function, or, The Reason We Write Tests

A common use for version numbers is to compare them. For example, a Web server might want to examine the version number of the app that made a request in order to send the appropriate response. Comparing version numbers is error-prone. Here’s what good version number comparison looks like (well, except for an intentional bug): This is a great example of code that is hard to get right. It’s all repetition with slight variation, so it’s tricky to see at a glance whether all the lhs and rhs are right, whether all the <s and >s are facing the right way, and whether we’re returning true and false in the right places. In fact, the code above has a bug: the third comparison should be >, but it is currently <. Let’s leave the bug in, because we need a failing test to show how this trick works.

The Status Quo: XCTAssert and Friends

Here is a typical test function, which ensures that various edge cases and permutations are covered. We are using XCTAssertLessThan, a function from XCTest that takes two Comparable values and fails the test if the first one is not less than the second one: There’s nothing wrong with this; it’s just that that the XCTAssertLessThan preamble is nearly as long as the values being compared, and the duplication of Version on every line is just more noise that we have to see past in order to focus on what’s important. Here is what it looks like when we run our tests and Xcode shows a failure on the fifth test, which is the one that runs afoul of the bug in our comparison code:

First Iteration: DRY It Up

Programmers love to keep it DRY—that is, “Don’t Repeat Yourself.” Let’s try to make the test more concise and readable by consolidating things that are repeated.
  1. Let’s shorten Version to just V by using a typealias. One-letter type names are usually a bad idea, but a name that exists only in a limited scope can have a shorter name because we don’t have to look far to find out what it means. See, for example, the common practice of using i as a loop counter.
  2. Why use XCTAssertLessThan on every single line? Instead, let’s stuff our values to be compared into an array, and then run XCTAssertLessThan in a loop!
  3. We can align the “left” and “right” values into columns so it’s easier to differentiate left from right.
Here’s the result of this cleanup (I’ll omit the test class from now on to keep things compact): Ah, much better! Now, the version numbers are the most prominent thing on each line, and we only need to call the assertion function once. This is great! Unfortunately, there’s a catch.
When tests fail, it should be as clear as possible which test failed, so that we can identify and fix the problem. And with this simplified approach, any test failure is going to be indicated on the line where the XCTAssertLessThan is called. Not ideal!

Second Iteration: Capturing Line Numbers

Ideally, we could use this clearer method of formatting the test, while still getting useful failure reporting in Xcode. This is where the trick comes in. Let’s start by looking at the function signature of XCTAssertLessThan, straight from the open source version on the official Swift GitHub repository: (Note that I’m using XCTAssertLessThan for this example, but all of the XCTAssert* methods have a similar signature.) From the top, we see:
  • The two values we are comparing (expression1 and expression2).
  • An optional message to include in the output if the test fails.
  • The file where the test failure occurred.
  • The line number where the failure occurred.
Note the default values of the file and line parameters: #file and #line, respectively. Normally, we don’t even see these values, thanks to the magic of default parameter values. However, we can pass our own StaticString and UInt values, respectively, and Xcode will dutifully show the error message in whatever file, and on whatever line, we specify. The trick is to capture the line where the array of input values is declared. We can then use enumerated() when looping over the array to get the index of each case as we test it. We can use this information to reconstruct the line number where the test case lives, which means we can point to that line if the test fails: Et voilà: compact, readable tests, but with any test failures reported on the line where the failing inputs were declared:
I love using this technique, and I try to employ it wherever I have a large number of test values that I want to compare with the same operator. It makes tests more fun to write, and increases their readability. However, it still requires doing a number of things correctly:
  • Capture the line number.
  • Get the Int to UInt type conversion right.
  • Get the math right—watch out for off-by-one errors!
  • Remember to pass the line number to the assertion function.
  • Ensure that there is one test case per line. Don’t put multiple cases on one line, and don’t wrap cases so they span multiple lines each.
  • Don’t comment out any of the inputs.
The last two are particularly important: violating the “one new test case per line” assumption will throw off the line numbering for any lines declared later, causing Xcode to report failures on the wrong line, which would be bad! Let’s see if we can do better.

Third Iteration: Helper Functions

We’ve been using tuples for our pairs of input values, and unfortunately, Swift tuples do not support default parameter values; only functions and initializers can do that. There are a few ways to overcome this limitation; I opted to create a Pair struct that uses a custom initializer to capture the file and line numbers: First, we define a Pair generic type that can wrap two arbitrary values. We give it a custom initializer that captures the file and line information for the line where it is called. Our previous example just used the line number, but capturing the file as well lets us define this type in a TestHelpers.swift file and use it throughout our tests. Second, we create a custom assertLessThan function that takes an array of pairs of a comparable type as input, and calls XCTAssertLessThan on all of them. This cleans up our test case considerably! Now, we don’t need to remember to capture the line where the array is declared or do any math in our test code, and we can safely comment any test cases we want to while debugging, because the line number is captured for every single line. And of course, we retain our nice reporting in Xcode:

One More Thing: Custom Comparison

One more thing: what if we want to do custom comparisons, in addition to the standard >, <, ==, etc? Well, here’s a general-purpose function for doing just that: Here’s an example of usage. On the left, we have a pair of Versions, and on the right, we have a Bool literal indicating whether they are expected to be equal: We can do whatever we want in the tests closure, as long as we pass the file and line parameters that it gives us into any XCTAssert* methods we might call. This ensures that test failures are still reported on the correct line. (Note that we have no failure screenshot here because we’re testing auto-generated Equatable conformance, which is always correct.)

Using Test Helpers

I’ve posted a gist with wrapped versions of all the built-in XCTAssert functions here. Try them out, and let me know what you think!

Leave a Reply

Your email address will not be published. Required fields are marked *