My client had started their mobile application UI test automation POC with Robot Framework and AppiumLibrary, but in practise, I could start implementing from scratch. We did not have much experience with other options, but we ended up choosing the XCTest library that is an integrated testing framework in Apple's Xcode IDE.
We found out immediate pros of preferring Apple's own tool set integrated to XCode over third-party tool Appium:
- XCTest demands no installations and setup top of Xcode,
- XCTest runs faster on integrated iPhone simulator, than Appium on Android real device,
- Test recording feature is available on Xcode, that speeds up scripting tests, as you don't need dedicated inspector tool to identify UI elements.
Other findings
Another expected advantage in using well integrated are apple tools, is that Xcode and Ios updates does not (usually) break test framework, which would apparently happen with Appium. Obvious downsides of choosing different tools for testing the same application (Android and IOS version), would be the doubled scripting work: same test cases written with Robot Framework and Swift. In addition to that, I had not ever used Swift language.
After gaining some experience with developing test automations with Swift, we faced more challenges. XCTest functions quite well usually, but when some strange failures occur, your best and only friend is Google. I still today don't personally know anybody else who uses these tools. With Appium and Robot Framework, you seem to have a better user community, at least in Finland.
Differences
Biggest difference in scripting work between AppiumLibrary and XCTest, is XCTest's lack of using XPaths. With XPaths, you can use more logic when identifying different UI elements. Then the test case does not break so easily when the application changes little. Here, the use of identifying properties of UI elements is necessary for the application developers. Without identifying property, the Xcode recording feature can recognize an UI element something like this in Swift language:
...and that is neither robust (when application changes) nor logically descriptive statement. But, it is better than nothing, which means that the recording does not always identify the desired UI element at all.
Challenges
The biggest challenge with XCTest was as simple as sharing reusable test functions between test suites. I did not have a strong programming background especially in Swift and our team’s developers did not have time and experience to help in examining the proper approach. For some reason I could not find a right example by googling or Xcode’s official site. Due to having also other projects, I did not have too much time to focus on the problem. We made a workaround by linking a resource file, where we could not use XCTest library functions, which enables waiting for testable properties to occur on UI. Hence, we used lots of sleeping functions, which increased run times. The things we had to validate by waiting for their occurrence, we used lots of copy-pasting same scripts to multiple test suite files.
Solving this workaround by better maintainable and robust architecture I could not find an answer earlier than a few weeks ago. I could not find it by googling Stackoverflow, but searching Github-codes with search terms XCTest and Swift. The solution was a simple test class inheritance example, so I wondered why I had not found it earlier. Refactoring 68 test cases took around a week and removed around 25% copy-pasted code from the code base and made it more robust and maintainable. Still I could not find empty templates for those files as an example in web, so I uploaded such myself to Github: xctest-resource-template
Summary
As summary on this journey to automate the same tests with different tools, I cannot prove it is a more efficient process than only one. We cannot compare the alternative we don’t have. But I can be sure that me and the rest of the mobile developer team have gained lots of new experiences with both tools and well functioning test set automatised.
Written by
Pekka Rantala