Retrying Failed Tests With Xcode 13 and Continuous Integration

As developers we never want our tests to fail, but there are often real world cases, especially with UI and integration tests against actual API (as opposed to a stable mock API or static data), where tests can and will occasionally fail. For example, API’s may go down, dependent data may change, or the Xcode Simulator may just crash. Now while none of these are good things to have to deal with and/or debug and testing against unstable API’s or databases is never best practice, this is also the real world. You may have to at least temporarily deal with legacy architecture, systems out of your control, QA environments with unstable data and code, or something similar.

If you are using a continuous integration (CI) enviroment such as Jenkins or Xcode Cloud to run your unit, integration, or UI tests, it’s also crucial that these test runs only fail in the correct scenarios. Nothing derails the developer acceptance of the need for automated testing more quickly than repeatedly chasing red herrings and getting stuck in the manual “just rerun the test to fix it” cycle. Consistently seeing failed yellow warning icons from tests will rapidly inure even the best developer to paying close attention to automation issues.

Up until recently, there was no way to handle test repetitions on failure in Xcode without creating custom script retry/fail logic. With Xcode 13 however, Apple finally provided developers with most of the tools they need to survive these situations, both through the UI and the command line with xcodebuild. As described in the WWDC 21 at the Diagnose unreliable code with test repetitions talk, you can now retry tests for a fixed number of repetitions, retry tests until a failure occurs, and retry tests upon failure. Each one of these scenarios has some specific use cases and can help flush out unstable and unreliable code; but I’m going to focus here on that last option to retry individual failed tests until they succeed.

Retrying Failed Tests through Xcode UI

If you right-click on the diamond test icon for the test or group of tests that you want to run in Xcode 13, you’ll see the Repeatedly… and Repeatedly Without Building… options available.

Selecting one of them will show the basic test repetition menu below. By default, you will see the following options selected.

To retry tests after failure until they succeed, we simply need to set Stop After: to Success like so and set the maximum repetitions if necessary.

Retrying Failed Tests through Xcode Test Plans

The above technique is great but gets old very quickly if you need to run tests over and over. So what do we do? Xcode Test Plans to the rescue! Introduced with Xcode 11 and also discussed back at WWDC 2019, test plans provide an easy way to run tests against multiple configurations for locale, debugging tools, environment settings, and more. With Xcode 13, Apple provided a number of new settings for each test plan configuration under the Test Execution section

We’ll be focusing here first on the Test Repetition Mode setting. Set to None by default, we will need to select it and set Retry on Failure. You would normally also want to specify the default number of test repetitions to try after failure to something reasonable.

In the screenshot below, we’ve set things up so that we will retry on failure up to 5 times.

That’s it! Once you’ve set this up, when you run tests against your test plan either through the Xcode UI, Xcode Cloud, or the command line, each unstable test will be retried up to the maximum number of times or until it succeeds. To use a test plan in a CI environment via the command line, it’s no different than normal.

xcodebuild -testPlan <my test plan name> -scheme <my scheme name> -destination <destination simulator or device> test

You can always override the test plan settings when testing manually by using the context menu method above as well.

Retrying Failed Tests with xcodebuild

If you are just working locally or are able to leverage Xcode Cloud, that’s great but I suspect most CI integrations out there these days are still leveraging the command line and the venerable xcodebuild tool to build, test, and archive iOS, iPadOS, and macOS projects. Luckily for us, Apple took this into account and added a number of new switches with Xcode 13. The following documentation is from the man page for xcodebuild.

• -test-iterations number. If specified, tests will run number times. May be used in conjunction with either -retry-tests-on-failure or -run-tests-until-failure, in which case this will become the maximum number of iterations.
• -retry-tests-on-failure. If specified, tests will retry on failure. May be used in conjunction with -test-iterations number, in which case number will be the maximum number of iterations. Otherwise, a maximum of 3 is assumed. May not be used with -run-tests-until-failure.
• -run-tests-until-failure. If specified, tests will run until they fail. May be used in conjunction with -test-iterations number, in which case number will be the maximum number of iterations. Otherwise, a maximum of 100 is assumed. May not be used with -retry-tests-on-failure.
• -test-repetition-relaunch-enabled [YES | NO] Whether or not each repetition of test should use a new process for its execution. Must be used in conjunction with -test-iterations, -retry-tests-on-failure, or -run-tests-until-failure. If not specified, tests will repeat in the same process.

For the basic “retry on failure” scenario that we’ve been discussing so far, we only need to be concerned with the first two options, -test-iterations and -retry-tests-on-failure. An example of building command line to retry each test test up to 5 times on failure of that test for an iOS simulator device might be something like this:

xcodebuild -retry-tests-on-failure -test-iterations 5 -scheme <my scheme name> -destination 'platform=iOS Simulator,name=iPhone 13 Pro Max' test

There isn’t that much else to repeating your failed tests other than the above switches. You can play around with the settings yourself. Be sure to use man xcodebuild or xcodebuild --help and review the documentation yourself if needed.

A Couple Warnings

There are two important caveats/warnings if you are trying to use test repetition that have caused some pain for me.

1. Relaunch with Test Repetition – You might expect that if you combine the -retry-tests-on-failure with the -test-repetition-relaunch-enabled setting that only the failed tests would launch in a new process. That is not currently the case and all of your selected tests – passed or failed – will rerun if just one test fails and the latter setting is set to YES. The run will still honor the maximum test repetitions but every single test will have to succeed for the test run to complete. This isn’t that big a deal for short test cycles but can be a killer if you’re running a UI test suite that takes, for example, an hour to run. This applies whether you are using the Xcode UI or command line.
2. Reporting Conversion – Third party tools such as xcpretty or trainer, which convert xcresult output to JUnit report format may not be set up to parse successful retried tests as test successes. This can result in still seeing the dreaded yellow status in Jenkins even though a retried test eventually succeeded. You may need to trigger your Jenkins pipeline result off of something other than the parsed test result such as the (** TEST SUCCEEDED ** in the output). I’m still working through the best way to handle this and looking at this in more depth (suggestions welcome here!).

Final Notes

I hope this helps someone out there needing to retry their tests on failure. I am well aware that even having to consider using these settings and tools mean that best practices may not be in place. If you are using retry on failure regularly for unit testing or situations where the entire test environment is under your control, you might consider digging into those issues sooner rather than later. That said, if you are fighting some real world scenarios or using Xcode’s testing capabilities for a budget status dashboard, for example, retrying on failure can make a positive difference.

Thanks for reading and as always please feel free to comment, share, follow. etc.