Xcode 11 uses a new format for xctest (version 3), which doesn't have a TestSummary.plist file, and uses Zstandard-compressed "database-like" files.

Xcode 11 Release Notes state:

The format of result bundles changed in Xcode 11. A result bundle is an asset produced by Xcode 11 with the xcresult file extension that contains information about the build, tests, code coverage, and more. Any xcresult files produced with Xcode 10 or earlier cannot be read by Xcode 11. A result bundle can be produced by passing -resultBundlePath ./Example.xcresult to an xcodebuild invocation and the Example.xcresult can then be opened in Xcode. Xcode also creates result bundles in Derived Data. The current result bundle version number is 3, which can be specified by passing the xcodebuild flag -resultBundleVersion 3. Version 3 is the default in Xcode 11, but it is still recommended for automation to explicitly pass the flag, so that any potential future versions that become the default do not cause issues to existing tools. Result bundles can be inspected using xcresulttool. A JSON representation of the root object of the result bundle can be exported using xcrun xcresulttool get --format json --path ./Example.xcresult and any nested object, identified by its reference found in the JSON output, can be exported by adding the flag --id REF. xcresulttool also provides the description of its format using xcrun xcresulttool formatDescription. (41633595)

Basically, to obtain results from an Xcode 11 xcresult, one would have to do the following:

// The output of this command will be useful to automatically parse the output of the next commands
$ xcrun xcresulttool formatDescription get --format json
[...]

$ xcrun xcresulttool get --format json --path Project.xcresult
[...]
          "testsRef" : {
            "_type" : {
              "_name" : "Reference"
            },
            "id" : {
              "_type" : {
                "_name" : "String"
              },
              "_value" : "0~YVEC1htpkY3jLmeu4DB8ah-sJueP7_vUdqA2rL1_R3xT_aftqqEgGUSCHZ0mmpojBSyy4mEL4LX2xyFlfr4pEQ=="
            },
[...]

// We take the value of the `testsRef` key (if any) for each `ActionRecord` entry
$ xcrun xcresulttool get --format json --path Project.xcresult --id '0~YVEC1htpkY3jLmeu4DB8ah-sJueP7_vUdqA2rL1_R3xT_aftqqEgGUSCHZ0mmpojBSyy4mEL4LX2xyFlfr4pEQ=='
{
  "_type" : {
    "_name" : "ActionTestPlanRunSummaries"
  },
  "summaries" : {
    "_type" : {
      "_name" : "Array"
    },
    "_values" : [
[...]

Although xcpretty and trainer parse different files to produce the Junit output, the content of the Junit file produced should be identical.

The testcase classname and name are used to identify uniquely a test across different test runs, for example in third party Tests managers like Xray for JIRA.

In the scenario of transition of xcpretty to trainer, it is required that the tests can be identified in a test run before the transition and a run after the transition.

In the Junit produced by xcpretty :

• the test classname is the test method classname prefixed by the test target in Xcode
• the test name is the name of the test method in the code, without () In the Junit produced by trainer :
• the test classname is is the test method classname without prefix
• the test name is the name of the test method in the code, including ()

This Pull Request includes changes to align the Junit test classname and name with xcpretty format.

This fixes #26. It will look for crash "activities", extract the related crash files, and dump them into the failure messages. The output looks kind of funny (i.e. the huge failure message), but I think this should work?

TODO:

• [x] Verify Jenkins reads junit xml files like this (shows up in the stack trace section)
• [x] Add option to opt into crash trace collection
• [ ] Double check this works on Jenkins at work

I'm using trainer 0.5.0 standalone (not as a fastlane plugin). I'm using RVM ruby and installing the gem with bundle.

I get this output:

$ bundle exec trainer
/Users/max.friedrich/.rvm/gems/ruby-2.3.1/gems/trainer-0.5.0/lib/trainer.rb:1:in `require': cannot load such file -- fastlane_core (LoadError)
	from /Users/max.friedrich/.rvm/gems/ruby-2.3.1/gems/trainer-0.5.0/lib/trainer.rb:1:in `<top (required)>'
	from /Users/max.friedrich/.rvm/gems/ruby-2.3.1/gems/trainer-0.5.0/bin/trainer:4:in `require'
	from /Users/max.friedrich/.rvm/gems/ruby-2.3.1/gems/trainer-0.5.0/bin/trainer:4:in `<top (required)>'
	from /Users/max.friedrich/.rvm/gems/ruby-2.3.1/bin/trainer:22:in `load'
	from /Users/max.friedrich/.rvm/gems/ruby-2.3.1/bin/trainer:22:in `<main>'
	from /Users/max.friedrich/.rvm/gems/ruby-2.3.1/bin/ruby_executable_hooks:15:in `eval'
	from /Users/max.friedrich/.rvm/gems/ruby-2.3.1/bin/ruby_executable_hooks:15:in `<main>'

fastlane_core is indeed required in lib/trainer.rb but was recently removed as a dependency in trainer.gemspec.

Looks like the latest beta now includes the duration of the tests so i've added support :) The time attribute is now present on both the test case and suite in the junit ouput 🎉

I also had to regenerate the Valid1.plist and Valid2.plist files from my own Xcode project so i've attached a copy incase anybody else needs to re-generate them in the future... Not sure if that should go into the repo though?

I've never really written any ruby before but the changes are minimal so it wasn't too hard but you might just want to sense check.. Tests all pass though.. Enjoy!

Unit.xcodeproj.zip

Fixes #32

Motivation

Support Xcode 11 xcresult format

Progress

• [x] Allowed reading of new Xcode 11 xcresult file
• [x] Run xcrun commands to get the JSON output
• [x] Run on small sample project to get mapping of ☝️ to match Xcode 10 junit file
• [x] Clean up logic from ☝️
• [x] Link up failure files and line number and error messages
• [ ] Test on bigger test suites
• [x] Write tests

Description

Files changed

• Edited test_parser.rb
• Added xcresult.rb
• The majority of the 79 files changed here by adding an .xcresult file for tests

Summary

• Added searching of path for *.xcresult
• Created all the models we needed based off of xcrun xcresulttool formatDescription in xcresult.rb
• Gets JSON formatted results with xcrun xcresulttool get --format json --path #{path}
  • Used to get an id of all the test summaries
  • Also used to get list of all the failures
• Uses ids of test summaries to get specific info for each test summary with xcrun xcresulttool get --format json --path #{path} --id #{id}
  • Used to get list, status, and duration of all tests run
  • Maps failures based upon target name and test name

Results

From Xcode 10

<?xml version="1.0" encoding="UTF-8"?>
<testsuites tests="7" failures="2">
    <testsuite name="TestUITests" tests="1" failures="0" time="17.92351496219635">
        <testcase classname="TestUITests" name="testExample()" time="17.92015504837036">
        </testcase>
    </testsuite>
    <testsuite name="TestThisDude" tests="6" failures="2" time="0.7454169988632202">
        <testcase classname="TestTests" name="testExample()" time="0.0010919570922851562">
        </testcase>
        <testcase classname="TestTests" name="testFailureJosh1()" time="0.006746053695678711">
            <failure message="XCTAssertTrue failed (/Users/josh/Projects/fastlane/test-ios/TestTests/TestTests.swift:37)">
            </failure>
        </testcase>
        <testcase classname="TestTests" name="testPerformanceExample()" time="0.4788789749145508">
        </testcase>
        <testcase classname="TestThisDude" name="testExample()" time="0.00041794776916503906">
        </testcase>
        <testcase classname="TestThisDude" name="testFailureJosh2()" time="0.0015239715576171875">
            <failure message="XCTAssertTrue failed (/Users/josh/Projects/fastlane/test-ios/TestThisDude/TestThisDude.swift:36)">
            </failure>
        </testcase>
        <testcase classname="TestThisDude" name="testPerformanceExample()" time="0.2513129711151123">
        </testcase>
    </testsuite>
</testsuites>

From Xcode 11

<?xml version="1.0" encoding="UTF-8"?>
<testsuites tests="7" failures="2">
    <testsuite name="TestUITests" tests="1" failures="0" time="16.05245804786682">
        <testcase classname="TestUITests" name="testExample()" time="16.05245804786682">
        </testcase>
    </testsuite>
    <testsuite name="TestThisDude" tests="6" failures="2" time="0.5279300212860107">
        <testcase classname="TestTests" name="testExample()" time="0.0005381107330322266">
        </testcase>
        <testcase classname="TestTests" name="testFailureJosh1()" time="0.006072044372558594">
            <failure message="XCTAssertTrue failed (/Users/josh/Projects/fastlane/test-ios/TestTests/TestTests.swift#CharacterRangeLen=0&amp;EndingLineNumber=36&amp;StartingLineNumber=36)">
            </failure>
        </testcase>
        <testcase classname="TestTests" name="testPerformanceExample()" time="0.2661939859390259">
        </testcase>
        <testcase classname="TestThisDude" name="testExample()" time="0.0004099607467651367">
        </testcase>
        <testcase classname="TestThisDude" name="testFailureJosh2()" time="0.001544952392578125">
            <failure message="XCTAssertTrue failed (/Users/josh/Projects/fastlane/test-ios/TestThisDude/TestThisDude.swift#CharacterRangeLen=0&amp;EndingLineNumber=35&amp;StartingLineNumber=35)">
            </failure>
        </testcase>
        <testcase classname="TestThisDude" name="testPerformanceExample()" time="0.2531709671020508">
        </testcase>
    </testsuite>
</testsuites>

Resolves: https://github.com/KrauseFx/trainer/issues/19

We are using Trainer at my company, and we realized that the following log comes from Trainer using a deprecated method from FastlaneCore::Helper:

This PR adds a constant to the Trainer that points to the gem path.

Please, let me know if you would prefer that method to be implemented differently.

Attempting to install trainer fails, as the required dependencies cannot be found.

Gemfile:

source "https://rubygems.org" gem 'trainer'

Running bundle install fails with the following error:

Bundler could not find compatible versions for gem "credentials_manager":   In Gemfile:     trainer was resolved to 0.1.1, which depends on       fastlane_core (< 1.0.0, >= 0.48.1) was resolved to 0.48.1, which depends on         credentials_manager (< 1.0.0, >= 0.16.0)

Could not find gem 'credentials_manager (< 1.0.0, >= 0.16.0)', which is required by gem 'fastlane_core (< 1.0.0, >= 0.48.1)', in any of the sources.

Similar errors occurred when I specified a version number in the Gemfile.

Similar to fail_build in scan, this defaults to true, which means scan will fail the build if unit tests fail. You can optionally set this option to false if you're handling this outside Fastlane.

This behaviour is helpful when you're using a reporting step on your CI pipeline, which will set the job status depending on the results of the report publishing.

For instance, on Jenkins:

pipeline {
  stages {
    stage('Build') {
      steps {
        sh 'bundle exec fastlane build'
      }
    }

    stage('Run unit tests') {
      steps {
        sh 'bundle exec fastlane test'
      }
    }
  }

  post {
    success {
      junit(testResults: '*_TestSummaries.xml') // This will set the job to unstable if unit tests failed
    }
  }
}

It turns out that target_name is not available when using xcodebuild test-without-building. test_name is always available though, so we can fall back to that. This ensures backward compatibility, although it looks like test_name could be used in any case (if both target_name and test_name are present, the values are the same).

(second pull request, I managed to delete the previous one when trying to fix a bad commit)

…test-without-building, target_name is not in the TestSummaries.plist. test_name seems to be always available, also when running xcodebuild test (in which case it has the same value as target_name).

In https://github.com/fastlane/fastlane/pull/19624, the code from this repo was copied directly into the Fastlane repo. Since then, there have been further updates to the trainer code within Fastlane, for example a new feature in https://github.com/fastlane/fastlane/pull/19957 a couple of weeks ago.

On the other hand, this repo has not been updated for several months. Is the intention that all further development of trainer will take place within the Fastlane fork of the code? If so, should this repo be updated with a message to inform people and avoid potential confusion? Perhaps it should even be archived.

I would like to be able to extract the total number of tests executed and total failures from the xcresult file. Is that an output available with trainer or can it be added to the lane_context

This may just be user error, but I'm not seeing a difference in the junit reports generated when xcpretty_naming is true vs when it is false. Looking at the code, it looks like this is the expected behavior when the testData is in a .xcresult file. Anyway, not sure if the bug is in the code, the documentation, or just good old-fashioned user error.

expected output:

<testsuites name='Target.xctest' tests='8' failures='0'>
  <testsuite name = 'Target.TestClass1' tests='4' failures='0'>
      <testcase classname='Target.TestClass1' name ='test1' time='.001'/>
      <testcase classname='Target.TestClass1' name ='test2' time='.001'/>
      <testcase classname='Target.TestClass1' name ='test3' time='.001'/>
      <testcase classname='Target.TestClass1' name ='test4' time='.001'/>
  </testsuite>
  <testsuite name = 'Target.TestClass2' tests='4' failures='0'>
      <testcase classname='Target.TestClass2' name ='test1' time='.001'/>
      <testcase classname='Target.TestClass2' name ='test2' time='.001'/>
      <testcase classname='Target.TestClass2' name ='test3' time='.001'/>
      <testcase classname='Target.TestClass2' name ='test4' time='.001'/>
  </testsuite>
</testsuites>

actual output:

<testsuites tests='8' failures='0'>
  <testsuite name = 'Target' tests='8' failures='0' time='.008>
      <testcase classname='TestClass1' name ='test1()' time='.001'>
      </testcase>
      <testcase classname='TestClass1' name ='test2()' time='.001'>
      </testcase>
      <testcase classname='TestClass1' name ='test3()' time='.001'>
      </testcase>
      <testcase classname='TestClass1' name ='test4()' time='.001'>
      </testcase>
      <testcase classname='TestClass2' name ='test1()' time='.001'>
      </testcase>
      <testcase classname='TestClass2' name ='test2()' time='.001'>
      </testcase>
      <testcase classname='TestClass2' name ='test3()' time='.001'>
      </testcase>
      <testcase classname='TestClass2' name ='test4()' time='.001'>
      </testcase>
 </testsuite>
</testsuites>

Some CI environments are capable of surfacing test attachments alongside the test report. XCTest supports attachments, so supporting viewing those attachments in the test reports would be fabulous. Supporting the known environments would seem to require dumping the attachments from the test results file format into images in a structured directory tree and adding a format option to massage the test results into NUnit 3 format in addition to the default of the JUnit report format.

Known CI Environments Supporting Test Attachments

Jenkins

Jenkins support for test attachments is available using the JUnit Attachments plugin. It can correlate files in a directory tree named to match the test tree and display them together.

Azure Pipelines

Azure Pipelines relies on NUnit output format support to surface test attachments. The attachments can then be accessed from a tab when the test failure info is pulled up.

There is support now for skipping tests in XCTest ( https://developer.apple.com/documentation/xctest/methods_for_skipping_tests?language=objc )

Would be great to have support for processing this output in trainer.

• Fix xcpretty_naming for xcresult
• Add test result object hierarchy
• Refactored group and name parsing function

These fixes came about due to the way the test result format has changed between Xcode 10 and 11, and how the logic to parse xcresult bundles in Trainer was not finding any of our failing tests.

In our use case, we were finding that test failures were not being parsed correctly because we were grouping our tests from a collection of other test suites like so:

class AllTests: XCTestCase {
    
    override class var testCaseClasses: [XCTestCase.Type] {
        return [
            TestsA.self,
            TestsB.self,
            TestsC.self,
        ]
    }
    
    override class var defaultTestSuite: XCTestSuite {
        let testSuite = XCTestSuite(forTestCaseClass: self)
        let testCases = testCaseClasses.flatMap { testCaseClass in
            testCaseClass.testInvocations.map { testCaseClass.init(invocation: $0) }
        }
        testCases.forEach(testSuite.addTest)
        return testSuite
    }
}

...

class TestsA: XCTestCase {
    
    func testFoo() {
        XCTAssertTrue(false)
    }
}

This results in find_failure incorrectly comparing AllTests/testFoo to TestsA/testFoo and returning false due to the identifier and sanitized_test_case_name values not matching in the test metadata.

Instead we should call xcresulttool get --format json and pass it the id we get from the summary reference in the metadata to get an accurate representation of the test failure summary. The benefit of this approach is that we also get explicit file name and line number information that we are able to format in exactly the same way as Trainer has been doing with the older test_result bundles in Xcode 10 and earlier.