Parallel testing with UI Testing Xcode - xcode7

Question

Can I open multiple simulator on the same mac in order to do parallel testing using UI Testing from Xcode?

Answer 1

Update May 2018:
Bluepill allows tests to be split across multiple simulators and supports XCUITests:
we recently added support for Xcode UI Testing bundles. Thanks to the
inspiration from FBSimulatorControl. Bluepill now supports Xcode unit
test bundle and UI test bundles.
Original answer:
There doesn't seem to be a viable solution at the moment. However promising tools are coming into place. There's a framework called FBSimulatorControl and it's cli tool called fbsimctl which allows you to boot multiple simulators at once, and install & run apps.
With fbsimctl you could build you target application & install it on each simulator. The part I'm stuck on is how to build & install the companion XCTestUI runner app.
xcodebuild doesn't appear to allow building the XCTestUI runner app without invoking the test action (causing alerts that multiple simulators are open). If xcodebuild could allow the building the .app/.ipa for the runner app only, then it looks like fbsimctl could handle the rest.
There's also an open issue over at facebook/xctool trying to work through this problem.

Answer 2

Parallel UI testing is possible if you use multiple users on your mac and some kind of a continuous integration system. If done correctly you can run your UI test suite on a variety of device simulators at the same time. In other words you'd be able to run all your UI tests against many devices in the time it takes to run those tests on one device - which is what I think we're trying to accomplish here. The following are instructions on how I was able to get parallel UI tests up and running with gitlab-ci-multi-runner. These instructions assume you have never used any of the tools but are not instructions on how to write UI tests:
Install Xcode through the mac app store.
Open the mac app store.
Type "Xcode" into the search field
Navigate to the Xcode install page and click "Install"
Note: I used Xcode 8.2 at time of writing this.
Install the Xcode command line tools.
Open terminal and enter the following command xcode-select --install
Note: You will be needing Xcodebuild, so this step is critical.
Create a few new "test" users on your mac.
Open system preferences and click on "Users & Groups".
Click the lock icon on the bottom left of the window to make changes.
Enter a "User Name" and "Password" and then click "Unlock".
Click the "+" add button located at the bottom of the list of users on your mac.
Select "Administrator" in the "New Account" dropdown menu.
Give your user a name in the "Full Name" text field (i.e. Test User One)
Click on the "Account Name" text field and allow it to auto fill.
Enter a password of your choosing into the "Password" text field.
Re-enter the password into the "Verify" text field.
Optionally enter a password hint into the "Password Hint" text field.
Click "Create User".
Repeat step two several times. Note: the number of users you create will determine the number of devices will run through your UI test suite at a time. In this solution we will discuss just running tests against several different devices at once on a single mac.
Create an account on gitlab.com.
Create a new project on gitlab.
Here is a tutorial on how to do that.
Follow the "Command line instructions" on your new project detail page to move your Xcode project into a gitlab repository.
Create SSH keys for each test user:
Sign into each of the "test" user accounts you just created. Note: It is important each user be signed in so the gitlab-ci system can run tests on that user. If not the tests will fail.
Open terminal again and type ssh-keygen -t rsa
Follow this tutorial complete this process.
Navigate to your gitlab "Profile Settings" page and then to the "SSH Keys" tab.
Use pbcopy < ~/.ssh/id_rsa.pub in terminal to get your generated SSH key in each 'test' user account and then use cmd+v to paste each the SSH key into the "Key" text field on the "SSH Keys" tab in gitlab.
Note: It is critical for each 'test' user to have access to your project code in gitlab so don't skip this step for any of your test users.
Note: I personally like to test each user has access to the repo by doing a git clone while logged into each of the "test" users.
Create test "runners" on/for each of your "test" users on your mac.
Open your gitlab project page on gitlab.com.
Click on the project settings icon.
Click on the "Runners" link.
The important information on this page is the URL and registration token highlighted in red under the "How to setup a specific Runner for a new project" section of the page.
Open terminal and type sudo curl --output /usr/local/bin/gitlab-ci-multi-runner https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-ci-multi-runner-darwin-amd64 then hit return.
Type sudo chmod +x /usr/local/bin/gitlab-ci-multi-runner then hit return.
Type gitlab-ci-multi-runner install then hit return.
Type gitlab-ci-multi-runner start then return.
Type gitlab-ci-multi-runner status then hit return to ensure the service is running.
Type gitlab-ci-multi-runner register and hit return.
Copy and paste the URL from the gitlab page described above into terminal and hit return.
Copy and paste the token from the gitlab page described above into terminal and hit return.
Enter a description for this "test" user runner and hit return.
Enter a tag unique to this particular "test" user and hit return.
Type Shell and hit return.
Verify the runner has been added and has a green dot net to it on your gitlab runners page.
Note: It is critical for this solution to set up individual runners for each of your "test" users you set up on your mac to run tests in parallel on different devices. Equally critical is that you give different tags to each of your "test" user runners so gitlab can tell each "test" user to run your UI tests.
Open terminal and type nano /Users/{test user name}/.gitlab-runner/config.toml and hit enter.
On the first line replace the number "concurrent = 1" with how ever many "test" users you created. Then hit shift+x, y, then return to save changes.
Note: Make sure it's done on each user to ensure parallelism.
Create a UI test scheme for your Xcode project.
Click the scheme dropdown menu near where you would select which device to run the project on.
Select "New Scheme".
In the "Target" dropdown menu select option that contains "UITests" in the title. Then click "OK"
Click the same scheme dropdown menu as described above and then select "Manage Schemes".
Make sure the checkbox is checked to make the "UITests" scheme "Shared".
Click close.
In terminal type git add . and hit return to ensure the schemes file is included.
Type git commit -am "Added ui test scheme" and hit return.
Type git push and hit return.
Create a gitlab-ci.yml file for your project.
Create a gitlab-ci.yml file in the root folder of your project in the text editor of your choice.
Here is an example yml file for an example project I created:
before_script:
- carthage update
stages:
- test
iphone_4s_tests:
stage: test
script:
- xcodebuild test -configuration Debug -project parallel.xcodeproj -scheme parallelUITests -destination 'platform=iOS Simulator,name=iPhone 4s' | xcpretty --simple
tags:
- iphone_4s
iphone_5s_tests:
stage: test
script:
- xcodebuild test -configuration Debug -project parallel.xcodeproj -scheme parallelUITests -destination 'platform=iOS Simulator,name=iPhone 5s' | xcpretty --simple
tags:
- iphone_5s
iphone_6_tests:
stage: test
script:
- xcodebuild test -configuration Debug -project parallel.xcodeproj -scheme parallelUITests -destination 'platform=iOS Simulator,name=iPhone 6' | xcpretty --simple
tags:
- iphone_6
IMPORTANT NOTE: Each tag should correspond with a tag on each one of the runners installed on the "test" user. The key here is that your test suite will be run on individual device
While in your project directory in terminal type git add .gitlab-ci.yml and hit enter.
Type git commit -am "Added the .gitlab-ci.yml file to the project." and then return.
Type git push and then return.
BOOM! If you've done everything correctly then your wildest dreams of testing your UI against many iOS devices in parallel has finally been realized. You'll notice that when you navigate to your gitlab project page and then the "Piplines" tab your UI tests are now running across in parallel. IMO this is the cheapest and most efficient way to get all your UI testing done. In the time it takes your test suite to complete one round on one device you can - in theory - complete the UI testing for all the devices you wish to test against. This solution is much cheaper than trying to buy several devices and having them all test at the same time. Generally speaking a single mac, depending on the specs, can run 10 - 15 devices at a time if there is nothing else taxing the CPU.

Answer 3

As of Xcode 9, you can now boot multiple simulators together and also can run parallel tests on multiple devices.
See the What's New in Xcode 9 Apple Documentation page for more information.

Answer 4

Have a look at plu/pxctest tool to execute tests in parallel ios simulators. It is much easier to integrate with project when compared to blue pill.
https://github.com/plu/pxctest

Answer 5

Parallel testing in now possible with the Simulator as announced in the What's New in Testing WWDC 2018 session:
Hear about exciting improvements to code coverage, including how you can build your own automation on top of Xcode's coverage reports. Learn how to dramatically speed up the execution of your tests by leveraging distributed parallel testing, new in Xcode 10.

Answer 6

For running UITests in parallel using multiple simulators check out this amazing open source library by LinkedIn >
https://github.com/linkedin/bluepill