Bluepill is a reliable iOS testing tool that runs UI tests using multiple simulators on a single machine

I managed to run bluepill against one of my apps with a bunch of UI tests based on Xcode UI testing. However I'm getting the following error. A bit googling suggests that this might be due to UI testing having different test template/config than "regular" XCTests. Is Xcode UI testing supported by bluepill? :)

Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'No target application path specified via test configuration: <XCTestConfiguration: 0x60800011a9d0>

Thanks!

The latest release fails to compile with:

Undefined symbols for architecture x86_64:
  "_OBJC_CLASS_$_SimDeviceFramebufferService", referenced from:
      objc-class-ref in SimulatorScreenshotService.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

The top of master branch fails with:

xcodebuild -workspace Bluepill.xcworkspace -scheme bluepill -configuration Release SYMROOT=../
[…]
In file included from /tmp/bluepill-20180930-3044-34c569/Bluepill-runner/Bluepill-runner/BPApp.m:10:
/tmp/bluepill-20180930-3044-34c569/Bluepill-runner/Bluepill-runner/BPApp.h:11:9: fatal error: 'BluepillLib/BPXCTestFile.h' file not found
#import <BluepillLib/BPXCTestFile.h>
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~

Hello, First time using bluepill.

I installed bluepill through brew install bluepill

I build my app by running the following command: xcodebuild -workspace MY.xcworkspace -scheme "UIABCTests" -sdk iphonesimulator -destination 'platform=iOS Simulator,name=iPhone 14 Pro Max' build-for-testing | xcbeautify

I got the xctestrun from Library/Developer/Xcode/DerivedData/name/Build/Products/UIABCTests_iphonesimulator16.0-arm64.xctestrun

AND the app from Library/Developer/Xcode/DerivedData/name/Build/Products/Debug-iphonesimulator/NameApp.app

Running this command:

     bluepill --xctestrun-path ./UIABCTests_iphonesimulator16.0-arm64.xctestrun -o ./output/

OR

{ 
   "app": "~/Desktop/NameApp.app",
   "xctestrun-path": "~/Desktop/UIABCTests_iphonesimulator16.0-arm64.xctestrun",
   "output-dir": "./build/" 
}

Also tried adding "unsafe-skip-xcode-version-check": "yes" to the abovejson` but did not work. and also tried

bluepill --unsafe-skip-xcode-version-check YES --xctestrun-path UIABCTests_iphonesimulator16.0-arm64.xctestrun -o ./output

no luck either :(.

gives me this error

2022-10-28 18:35:16.764 bluepill[37085:345815] Debug Enabled == 0
{37085} 20221028.183517 [  INFO  ] (BLUEPILL) Using xctestrun configuration
{37085} 20221028.183517 [  ERROR ] (BLUEPILL) Failed to read data for UIABCTests
ERROR: Failed to load some test bundles

bluepill version: Bluepill v5.12.1 xcode version: Version 14.0.1 (14A400) OS: 13.0 (22A380)

Hello!

We'd encountered an issue, connected with retry policy.

Given:

10 tests, where 5 always fail, 5 always pass.

--failure-tolerance = 1 --only-retry-failed = true

After first run we have Tests-results.xml in JUnit format. (5 passed, 5 failed) As failure-tolerance is enabled all failed tests are rerunned and as a result we have new Tests-results.xml only with these latests failed tests (5 failed) and previous file is removed and replaced with new one, built after latest test-run.

As a result we have in FINAL-REPORT.xml with only tests that were rerunned (only failed tests)

Is this behaviour expected? thx

Hi all

Bluepill is great tool for running tests in parallel, but I've got pretty weird error while starting iPhone simulators. Meanwhile it runs perfectly well on iPad. Any ideas on what is causing this issue?

[  INFO  ] Using xctestrun configuration
[  INFO  ] This is Bluepill v2.1.0-21-gb0944ab
[  INFO  ] Running with 2 simulators.
[  INFO  ] Packed tests into 3 bundles
[  INFO  ] Started Simulator 1 (PID 6902).
[  INFO  ] (BP-1) Running Tests. Attempt Number 1.
[  INFO  ] (BP-1) [Attempt 1] Create Simulator
[  INFO  ] 1 Simulator still running. [1]
[  INFO  ] Using 189 of 2048 processes.
[  INFO  ] Started Simulator 2 (PID 6923).
[  INFO  ] (BP-2) Running Tests. Attempt Number 1.
[  INFO  ] (BP-2) [Attempt 1] Create Simulator
[  INFO  ] (BP-1) Starting SimulatorScreenshotService for device:362A073F-EFC4-4B64-8265-B43494A55853
[  INFO  ] (BP-1) Booting a simulator without launching Simulator app
[  INFO  ] (BP-2) Starting SimulatorScreenshotService for device:15476C05-CD96-4F05-8102-ABE7271FE26B
[  INFO  ] (BP-2) Booting a simulator without launching Simulator app
 bp[6923:26705] CoreSimulatorService connection interrupted.  Resubscribing to notifications.
 bp[6902:26784] CoreSimulatorService connection interrupted.  Resubscribing to notifications.
 bp[6902:26697] *** Assertion failure in -[SimDeviceIOClient ioPorts](), /BuildRoot/Library/Caches/com.apple.xbs/Sources/CoreSimulator/CoreSimulator-494.33/CoreSimulator/SimDeviceIOClient.m:62
 bp[6923:26757] *** Assertion failure in -[SimDeviceIOClient ioPorts](), /BuildRoot/Library/Caches/com.apple.xbs/Sources/CoreSimulator/CoreSimulator-494.33/CoreSimulator/SimDeviceIOClient.m:62
 bp[6923:26757] *** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Failed to initialize IO ports for device IO client.'
 *** First throw call stack:
 (
 	0   CoreFoundation                      0x00007fffc267357b __exceptionPreprocess + 171
 	1   libobjc.A.dylib                     0x00007fffd78d21da objc_exception_throw + 48
 	2   CoreFoundation                      0x00007fffc2678132 +[NSException raise:format:arguments:] + 98
 	3   Foundation                          0x00007fffc4147da6 -[NSAssertionHandler handleFailureInFunction:file:lineNumber:description:] + 166
 	4   CoreSimulator                       0x000000010bf69dc1 -[SimDeviceIOClient ioPorts] + 412
 	5   SimulatorKit                        0x000000010c1aff67 +[SimDeviceFramebufferService portForDisplayClass:io:] + 98
 	6   SimulatorKit                        0x000000010c1b1581 __54-[SimDeviceFramebufferService registerClient:onQueue:]_block_invoke.194 + 220
 	7   libdispatch.dylib                   0x00007fffd8186524 _dispatch_call_block_and_release + 12
 	8   libdispatch.dylib                   0x00007fffd817d8fc _dispatch_client_callout + 8
 	9   libdispatch.dylib                   0x00007fffd81939a0 _dispatch_queue_serial_drain + 896
 	10  libdispatch.dylib                   0x00007fffd8186306 _dispatch_queue_invoke + 1046
 	11  libdispatch.dylib                   0x00007fffd817f6b5 _dispatch_root_queue_drain + 476
 	12  libdispatch.dylib                   0x00007fffd817f48c _dispatch_worker_thread3 + 99
 	13  libsystem_pthread.dylib             0x00007fffd83cc5a2 _pthread_wqthread + 1299
 	14  libsystem_pthread.dylib             0x00007fffd83cc07d start_wqthread + 13
 )
 libc++abi.dylib: 2018-03-21 17:31:03.666 bp[6902:26697] *** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Failed to initialize IO ports for device IO client.'
 *** First throw call stack:
 (
 	0   CoreFoundation                      0x00007fffc267357b __exceptionPreprocess + 171
 	1   libobjc.A.dylib                     0x00007fffd78d21da objc_exception_throw + 48
 	2   CoreFoundation                      0x00007fffc2678132 +[NSException raise:format:arguments:] + 98
 	3   Foundation                          0x00007fffc4147da6 -[NSAssertionHandler handleFailureInFunction:file:lineNumber:description:] + 166
 	4   CoreSimulator                       0x000000010eb84dc1 -[SimDeviceIOClient ioPorts] + 412
 	5   SimulatorKit                        0x000000010edcaf67 +[SimDeviceFramebufferService portForDisplayClass:io:] + 98
 	6   SimulatorKit                        0x000000010edcc581 __54-[SimDeviceFramebufferService registerClient:onQueue:]_block_invoke.194 + 220
 	7   libdispatch.dylib                   0x00007fffd8186524 _dispatch_call_block_and_release + 12
 	8   libdispatch.dylib                   0x00007fffd817d8fc _dispatch_client_callout + 8
 	9   libdispatch.dylib                   0x00007fffd81939a0 _dispatch_queue_serial_drain + 896
 	10  libdispatch.dylib                   0x00007fffd8186306 _dispatch_queue_invoke + 1046
 	11  libdispatch.dylib                   0x00007fffd817f6b5 _dispatch_root_queue_drain + 476
 	12  libdispatch.dylib                   0x00007fffd817f48c _dispatch_worker_thread3 + 99
 	13  libsystem_pthread.dylib             0x00007fffd83cc5a2 _pthread_wqthread + 1299
 	14  libsystem_pthread.dylib             0x00007fffd83cc07d start_wqthread + 13
 )
 terminating with uncaught exception of type NSException
 libc++abi.dylib: terminating with uncaught exception of type NSException
 [  INFO  ] Simulator 1 (PID 6902) has finished with exit code 6.
 [  INFO  ] Simulator 2 (PID 6923) has finished with exit code 6.
 [  INFO  ] PID 6902 exited 6.
 [  INFO  ] PID 6923 exited 6.

I am randomly facing this scenario -

15-Mar-2018 08:45:09 | {2741} 20180315.044509 [  INFO  ] 2 Simulators still running. [2, 5] 15-Mar-2018 08:45:09 | {2741} 20180315.044509 [  INFO  ] Using 722 of 2128 processes. 15-Mar-2018 08:45:23 | {4547} 20180315.044523 [ PASSED ] (BP-5) 100.155758s ViewMyActivity/testViewMyActivity 15-Mar-2018 08:45:23 | {4547} 20180315.044523 [  INFO  ] (BP-5) _XCT_didFinishExecutingTestPlan 15-Mar-2018 08:45:23 | {4547} 20180315.044523 [  INFO  ] (BP-5) All Tests Completed. 15-Mar-2018 08:45:23 | {4547} 20180315.044523 [  INFO  ] (BP-5) Socket transport disconneted 15-Mar-2018 08:45:23 | {4547} 20180315.044523 [  INFO  ] (BP-5) DTXConnection disconnected. 15-Mar-2018 08:45:24 | {4547} 20180315.044524 [  INFO  ] (BP-5) BPDEBUGGING finished 15-Mar-2018 08:45:24 | {4547} 20180315.044524 [  INFO  ] (BP-5) Number of Executions: 1 15-Mar-2018 08:45:24 | {4547} 20180315.044524 [  INFO  ] (BP-5) Final Exit Status: BPExitStatusTestsAllPassed 15-Mar-2018 08:45:24 | {4547} 20180315.044524 [  INFO  ] (BP-5) BP exiting 0 15-Mar-2018 08:45:24 | {2741} 20180315.044524 [  INFO  ] Simulator 5 (PID 4547) has finished with exit code 0. 15-Mar-2018 08:45:24 | {2741} 20180315.044524 [  INFO  ] PID 4547 exited 0. 15-Mar-2018 08:45:39 | {2741} 20180315.044539 [  INFO  ] 1 Simulator still running. [2] 15-Mar-2018 08:45:39 | {2741} 20180315.044539 [  INFO  ] Using 721 of 2128 processes. 15-Mar-2018 08:46:09 | {2741} 20180315.044609 [  INFO  ] 1 Simulator still running. [2] 15-Mar-2018 08:46:09 | {2741} 20180315.044609 [  INFO  ] Using 722 of 2128 processes. 15-Mar-2018 08:46:39 | {2741} 20180315.044639 [  INFO  ] 1 Simulator still running. [2] 15-Mar-2018 08:46:39 | {2741} 20180315.044639 [  INFO  ] Using 718 of 2128 processes. 15-Mar-2018 08:47:09 | {2741} 20180315.044709 [  INFO  ] 1 Simulator still running. [2] 15-Mar-2018 08:47:09 | {2741} 20180315.044709 [  INFO  ] Using 718 of 2128 processes. 15-Mar-2018 08:47:39 | {2741} 20180315.044739 [  INFO  ] 1 Simulator still running. [2] 15-Mar-2018 08:47:39 | {2741} 20180315.044739 [  INFO  ] Using 721 of 2128 processes. 15-Mar-2018 08:48:09 | {2741} 20180315.044809 [  INFO  ] 1 Simulator still running. [2] 15-Mar-2018 08:48:09 | {2741} 20180315.044809 [  INFO  ] Using 718 of 2128 processes. 15-Mar-2018 08:48:39 | {2741} 20180315.044840 [  INFO  ] 1 Simulator still running. [2] ...... 15-Mar-2018 10:12:53 | {2741} 20180315.061253 [  INFO  ] 1 Simulator still running. [2] 15-Mar-2018 10:12:53 | {2741} 20180315.061253 [  INFO  ] Using 727 of 2128 processes.

I want to know what causes a simulator to be in running situation , i have also tried running with -R (error-retries) as 0 but still i am facing this issue. Is this related to some test that hangs and causing the simulator to be in running state ?

How can i debug and solve this issue from recurring ?

Running this:

bluepill -a XXXX.app/ -s XXXX.xcscheme -o output/

crashes and returns this:

bluepill[78419:3224648] *** Terminating app due to uncaught exception 'NSRangeException', reason: '*** -[__NSSingleObjectArrayI objectAtIndex:]: index 1 beyond bounds [0 .. 0]'
*** First throw call stack:
(
	0   CoreFoundation                      0x00007fff7d8390db __exceptionPreprocess + 171
	1   libobjc.A.dylib                     0x00007fff924cba2a objc_exception_throw + 48
	2   CoreFoundation                      0x00007fff7d8a3b4f -[__NSSingleObjectArrayI objectAtIndex:] + 111
	3   bluepill                            0x000000010bb80448 +[BPXCTestFile BPXCTestFileFromExecutable:withError:] + 872
	4   bluepill                            0x000000010bb7f746 +[BPApp BPAppWithAppBundlePath:withExtraTestBundles:withError:] + 907
	5   bluepill                            0x000000010bb78b3c main + 739
	6   libdyld.dylib                       0x00007fff92dad255 start + 1
	7   ???                                 0x0000000000000007 0x0 + 7
)
libc++abi.dylib: terminating with uncaught exception of type NSException
Abort trap: 6

<failure type="Failure" message="UNKNOWN ERROR - PARSING FAILED: XCTestOutputBarrierTest Case '-[Target.TestSuite TestCase]' started.">
Unknown File:0
</failure>

• Removed a bunch of unnecessary abstractions and generally moved towards a bundle-centric instead of an app-centric world view.

• Clean up errors by reworking BP_ERROR to BP_SET_ERROR that handles the error == nil case.

• Changed the name of a bunch of properties to better match what's in the .xctestrun files.

• Check that the app we're given is actually built for the iphonesimualtor and not for iphoneos.

• Since we're getting environment variables and commandline arguments from the .xctestrun file now, they are properly expanded.

I have a few questions:

1. I have the options - "screenshots-directory": "./screen/". However the screen's folder are empty despite some of the tests failing. Is there anything else that i need to do?
2. What is the latest most updated fastlane plugin for Bluepill that you guys are using?

Below is my config file:

  {
 "xctestrun-path": "path to xctestrun", 
   "output-dir": "/Users/Testresults",
   "device": "iPhone 11 Pro",
   "num-sims": 2,
   "screenshots-directory": "./screen/",
   "reuse-simulator": true,
   "failure-tolerance": 3,
   "only-retry-failed": true
}

I got the latest release of bluepill 1.1.1, and ran bluepill with the new xctestrun-path option pointing to a valid xctestrun file generated by a build-for-testing compile of my project. It ran the correct test scheme, but it did not skip any of the tests specified in SkipTestIdentifiers.

This is with Xcode 8.3.2 and Bluepill 1.1.1 release

When I am trying to run xctestrun file using bluepill I got NSInvalidArgumentException error. Here is the screenshot. Here is xctestrun file xctestrun file.zip please help to find the solution. Thanks in advance.

bundleIdentifier != nil issue when running rules_ios/Bazel-built ios_ui_test that doesn't occur when using the standard xctestrunner.

In this simplistic example, I was getting it to happen always. But in our real project, this crash was happening most of the time (maybe 9/10 cases) but sometimes the app + UI test bundle would launch.

Reproduction

1. Checkout https://github.com/mattrobmattrob/rules_swift_testbed/tree/mr/bluepill/bundle.identifier.issue. This is a prebuilt "release" on ~master from this repo.
2. Use the mr/bluepill/bundle.identifier.issue branch
3. Run bazelisk test //bluepill:UI_Tests_Bluepill.

Runtime Error

{41232} 20221109.230820 [  INFO  ] (BP-1) Test plan started!
{41232} 20221109.230820 [  INFO  ] (BP-1) Test bundle is connected. Capabilities = <XCTCapabilities: 0x600002b573a0>: {
    "ubiquitous test identifiers" = 1;
}
{41232} 20221109.230820 [  INFO  ] (BP-1) _XCT_didBeginExecutingTestPlan
{41232} 20221109.230820 [  INFO  ] (BP-1) All Tests started.
{41232} 20221109.230820 [  INFO  ] (BP-1) Starting TestSuite All tests
{41232} 20221109.230820 [  INFO  ] (BP-1) Starting TestSuite SimpleTest.xctest
{41232} 20221109.230820 [  INFO  ] (BP-1) Starting TestSuite SimpleUITests
{41232} 20221109.230820 [ FAILED ] (BP-1)   0.002188s SimpleUITests/testPasses()
{41232} 20221109.230820 [  ERROR ] (BP-1) HOW DID WE GET AN ERROR THAT WASN'T PARSED? We received an error in a test case that wasn't started or did not parse properly.
Problem line: XCTestOutputBarrier/tests/ios/ui-test/simple.swift:10: error: -[SimpleTest.SimpleUITests testPasses] : Invalid parameter not satisfying: bundleIdentifier != nil (NSInternalInconsistencyException)
{41232} 20221109.230820 [  INFO  ] (BP-1) _XCT_didFinishExecutingTestPlan
{41232} 20221109.230820 [  INFO  ] (BP-1) All Tests Completed.
{41232} 20221109.230820 [  INFO  ] (BP-1) DTXSocketTransport disconnected
{41232} 20221109.230820 [  INFO  ] (BP-1) DTXConnection disconnected.
{41232} 20221109.230821 [  INFO  ] (BP-1) Finished

@ob @caniszczyk We are using bluepill to run tests and at the end we are using a framework to post the results in xml in UI format. During that process we are facing issue with GMT timestamp format. As Junit XML has to follow ISO8601_DATETIME_PATTERN for timestamp So is it possible to change GMT to UTC format?

I have installed xcode 13.1 as 13.2 is not compatible. i am trying to run bluepill for xcode 13.1 with config file. Note: I have also 13.2 installed i havent deleted.

{
  "xctestrun-path": "./CounterApp_iphonesimulator10.3-x86_64.xctestrun",
  "output-dir": "./build/",
  "xcode-path": "/Applications/Xcode13.1.app"

}

i have also attached an screenshot of my command. Is there any command i am missing?

Proposal

Apple has provided a few different ways to run tests. Logic tests without an application, logic tests within an application, and a UI Test target. Currently Bluepill doesn't support tests without an app bundle path associated with them, actively erroring out when one isn't found. This proposal is to add the ability to run logic tests without an associated application to Bluepill.

Benefits

There are 2 main benefits for allowing non-app hosted tests.

Faster Signal

Tests that run without an application start more quickly and run on a much smaller process compared to an app, usually the xctest process. This allows them to run before a simulator is fully booted (aka when the simulator hits the home screen) and doesn't require a full app install. Fully booting and installing these apps can take a large portion of time and aren't needed when performing tests outside the application context. Removing this need to be hosted will allow them to finish much quicker.

Booting and installing an application uses lot of resources on the accompanying machine (CPU and memory) slowing down other simulators that are running simultaneously. This could also alleviate some of that burden.

One thing to note, on a small scale this faster signal doesn't have a huge impact but as you grow your codebase to many hours and/or many modules worth of tests this can be a significant boost

Single Test Execution Path While I understand Bluepill was originally intended to be used for UI tests this creates a bifurcation of how tests are done and information about those tests are gathered. One for app hosted tests and one for non-hosted tests. Adding the ability to run non-hosted tests allows for developers to have a single way to run and gather information about their tests.