Deprecation Warning
Xcode 9 perfectly supports executing tests in parallel without any 3rd party tool like pxctest. We suggest moving to xcodebuild for running tests in parallel on multiple Simulators.
pxctest
Execute tests in parallel on multiple iOS Simulators.
Installation
To install via Homebrew you can use the plu/homebrew-pxctest tap:
$ brew tap plu/pxctest && brew install pxctest
Usage
Compile your tests with build-for-testing, example:
$ xcodebuild \
-IDEBuildLocationStyle=Custom \
-IDECustomBuildLocationType=Absolute \
-IDECustomBuildProductsPath="$PWD/build/Products" \
-scheme 'MyApp' \
-workspace 'MyApp.xcworkspace' \
-destination 'platform=iOS Simulator,name=iPhone 5,OS=10.1' \
build-for-testing
In build/Products you should find a .xctestrun file. This can then be passed to pxctest:
$ pxctest \
run-tests \
--destination 'name=iPhone 5,os=iOS 9.3' \
--destination 'name=iPhone 5,os=iOS 10.1' \
--testrun build/Products/MyApp_iphonesimulator10.1-i386.xctestrun
The --destination option can be passed in several times and will execute the tests in parallel on all Simulators.
run-tests options
To see a list of possible options, just run:
$ pxctest run-tests --help
Most of the options should be self-explanatory. If not, please open an issue or submit some pull request. However the --defaults option needs a special section here.
run-tests --defaults
This option expects a path to a file, which contains some JSON. After loading this file, its content gets applied to the Simulator's defaults. On the top level the keys must be either a relative path or a domain where the defaults are located.
Example: You can turn off all keyboard settings that you can find in the Simulator's Settings app by using following JSON content:
{
"com.apple.Preferences": {
"KeyboardAllowPaddle": false,
"KeyboardAssistant": false,
"KeyboardAutocapitalization": false,
"KeyboardAutocorrection": false,
"KeyboardCapsLock": false,
"KeyboardCheckSpelling": false,
"KeyboardPeriodShortcut": false,
"KeyboardPrediction": false,
"KeyboardShowPredictionBar": false
}
}
Headless testing
It's possible to execute the tests without actually launching a Simulator.app window. This can be convenient because it will separate your test workflow from your development. If the tests run in the Simulator.app window and you launch your app during the test run from Xcode, it will stop the tests. There's not much we can do about that, it's just the way Xcode is handling Simulator.app - it takes control over all instances.
First you need to pre-boot some Simulators in the background, with a custom device set path:
$ mkdir /tmp/test-simulators
$ pxctest boot-simulators \
--deviceset /tmp/test-simulators \
--destination 'name=iPhone 5,os=iOS 9.3' \
--destination 'name=iPad Retina,os=iOS 9.3'
Once the command has finished, the Simulators are in a usable state and we can begin running tests on them.
To launch your tests on these Simulators, you just need to pass the --deviceset option to the run-tests command:
$ pxctest run-tests \
--testrun build/Products/KIF_iphonesimulator10.1-i386.xctestrun \
--deviceset /tmp/test-simulators \
--destination 'name=iPhone 5,os=iOS 9.3' \
--destination 'name=iPad Retina,os=iOS 9.3' \
--only 'KIF Tests:TappingTests'
....................
KIF Tests - iPhone 5 iOS 9.3 - Finished executing 10 tests after 25.252s. 0 Failures, 0 Unexpected
KIF Tests - iPad Retina iOS 9.3 - Finished executing 10 tests after 25.119s. 0 Failures, 0 Unexpected
Total - Finished executing 20 tests. 0 Failures, 0 Unexpected
If you still see some Simulator.app window showing up, it might have different reasons:
- something went wrong pre-booting the devices earlier
- you forgot the
--devicesetoption - you didn't pre-boot all devices that you're using as
--destinationoptions now
You can verify the state of your devices via:
$ xcrun simctl --set /tmp/test-simulators list
== Devices ==
-- iOS 9.3 --
iPhone 5 (716A9864-08BD-4200-96ED-20EA1E81BE65) (Booted)
iPad Retina (D2EB2BB9-8862-4F0D-A933-079C9BA0342A) (Booted)
The Booted state here however does not mean "is ready for launching apps or running tests". After booting a device it enters the Booted state quickly, but still showing the loading bar above the Springboard (you can see that if you boot them via Simulator.app and keep watching the state that xcrun simctl reports).
When you're finished running tests, you may wish to shut down running simulators to release their resources.
$ pxctest shutdown-simulators \
--deviceset /tmp/test-simulators
Development
git clone --recursive https://github.com/plu/pxctest.git pxctest
cd pxctest
scripts/bootstrap.sh
NSUnbufferedIO=YES xcodebuild -scheme pxctest test | xcpretty -c
FBSimulatorControl
The functionality of pxctest would not be possible without the great FBSimulatorControl Framework provided by Lawrence Lomax at Facebook.
There are two command line tools that come with FBSimulatorControl:
- fbsimctl - command line interface to the FBSimulatorControl Framework
- fbxctest - test runner for running iOS testing bundles for the iOS Simulator Platform
Both of them are more flexible than pxctest, so it might be worth having a look at them.
Resource limits
By default the resource limitations (maximum processes per user, maximum open files) on Mac OS are quite small. There even seems to be a hard limit baked into the kernel of 2500 maximum processes per user. But there's some boot arguments that can be set to raise this limit to at least 5000.
So why is this important? A booted iPhone 7 Simulator is not just one process, it comes with a whole set of running processes. If you boot a couple Simulators at the same time, add your other running programs/processes on top of that, then you will hit these limits soon.
First we set the nvram boot argument:
$ sudo nvram boot-args="serverperfmode=1 $(nvram boot-args 2>/dev/null | cut -f 2-)"
If you're using Sierra, you will probably get an error message like
nvram: Error setting variable - 'boot-args': (iokit/common) general error
This is because starting from Sierra the nvram command should be executed from the recovery partition. more info here To restart a mac from the recovery partition press CMD+r during the boot procedure. From there open a terminal and insert the same command as before.
Next step is to make launchd set the new limits:
$ sudo cp config/limit.maxfiles.plist /Library/LaunchDaemons/limit.maxfiles.plist
$ sudo cp config/limit.maxproc.plist /Library/LaunchDaemons/limit.maxproc.plist
The two files can be found in this repository:
And we reboot! Afterwards you can check:
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 524288
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 5000
virtual memory (kbytes, -v) unlimited
The open files part should say 524288, and max user processes should be 5000.
If you are using Sierra, the output looks a bit different:
› ulimit -a
-t: cpu time (seconds) unlimited
-f: file size (blocks) unlimited
-d: data seg size (kbytes) unlimited
-s: stack size (kbytes) 8192
-c: core file size (blocks) 0
-v: address space (kbytes) unlimited
-l: locked-in-memory size (kbytes) unlimited
-u: processes 5000
-n: file descriptors 524288
The open files is now file descriptors and max user processes is renamed to processes.
Can't find the .xctestrun?
When code coverage is enabled on the project specifying IDECustomBuildProductsPath to xcodebuild has no effect. Instead you can provide derivedDataPath and get the .xctestrun file from the directory.
Compile your tests with build-for-testing, specifying derivedDataPath, for example:
$ xcodebuild \
-derivedDataPath "$PWD/derivedData" \
-scheme 'MyApp' \
-workspace 'MyApp.xcworkspace' \
-destination 'platform=iOS Simulator,name=iPhone 5,OS=10.1' \
build-for-testing
In derivedData/Build/Intermediates/CodeCoverage/Products you should find a .xctestrun file. This can then be passed to pxctest:
$ pxctest \
run-tests \
--destination 'name=iPhone 5,os=iOS 9.3' \
--destination 'name=iPhone 5,os=iOS 10.1' \
--testrun derivedData/Build/Intermediates/CodeCoverage/Products/MyApp_iphonesimulator10.1-i386.xctestrun
Release version update
- Update
Info.plist>CFBundleShortVersionString - Update
CHANGELOG.md - Ensure the latest stable Xcode version is installed and
xcode-selected. - From project root directory run
./scripts/release.sh - Create a GitHub release: https://github.com/plu/pxctest/releases/new
- Specify the tag you just pushed in the dropdown.
- Set the release title to the new version number.
- Add the changelog section to the release description text box.
- Upload the portable zip you just built to the GitHub release binaries.
- Click "Publish release".
- Publish to Homebrew and CocoaPods:
./scripts/publish.sh
510 Nov 24, 2022
2.6k Dec 17, 2022
234 Jan 9, 2023
70 Dec 30, 2022
0 Nov 1, 2021
2.6k Dec 28, 2022
565 Nov 16, 2022
0 Nov 30, 2021
18 Aug 19, 2022
845 Oct 30, 2022
20 Sep 27, 2022
8 Aug 17, 2022
3 May 25, 2022
1.1k Dec 16, 2022
8 Oct 11, 2021
92 Dec 16, 2022
22 Jan 3, 2023
129 Sep 6, 2022
23 Aug 29, 2021