Resolves #2

This PR adds a very naive support for reporting test failures in Xcode. Instead of using runtime to create tests, as suggested in #17 I've added a method XCTestCase.describe that will store a reference to this test case in custom reporter, to use for reporting later, calls global describe function with passed in closure and then runs all the tests. At the end all added tests are removed so that they are not run again later. All individual test failures will appear in Issues navigator.

Tests will be still run at the application exit (without reporting via Xcode) but if they were already run with XCTestCase.describe they will be removed from the global context and so will not be run twice.

Also added option to disable colours in output which make output less readable in Xcode console, that does not support ANSI colors anyway.

XCTestCase.recordFailure(..) method has been deprecated on Xcode12+ and it will cause compile error on Xcode12.

To avoid that, added @available block. In the future, XCTestCase. recordFailure(..) should be replaced with new API, XCTestCase.record(_:).

Video https://developer.apple.com/videos/play/wwdc2020/10687/ Document https://developer.apple.com/documentation/xctest/xctestcase/3546549-record

Some dependency of mine has Spectre as a dependency, and when Xcode 13 beta resolves the dependencies, it issues this warning:

Invalid Exclude '…/SourcePackages/checkouts/Spectre/Sources/Spectre/[email protected]': File not found.

I'm not sure what it means, but this seemed like a reasonable place to report the issue.

Nowadays, swift-corelibs-xctest exists, so it should be possible to use Spectre from XCTest on Linux. This change removes the #if guards around XCTest extensions, with the exception of some Xcode 12+ APIs that have not yet made it upstream.

Testing done

Builds and tests on a linux machine using the official Swift docker images:

$ docker pull swift
$ docker run -itv $PWD:/code swift swift test --package-path /code

Playground sources a still not available when using 0.8.0 as a dependency. More - various samples (like https://github.com/IBM/FoodTrackerBackend) cannot be correctly configured when using the Swift PM in the CLion (we are currently using CMake for integrating it and CMake complains about files missed). Please, bump the version so the most recent sources are included.

• beNotNil to validate that value is not nil
• notThrow to validate that function does not throw
• throw(_ match: (Error) -> Bool) to validate that function throws error using custom condition rather than Equatable
• throw and throw(_:) expectations to better match other APIs (where to is omitted)
• expect(...).to.not for not expectations, expect(...).to has now better scoped methods and code completion

When building with Xcode 9 GM, we get the following errors:

[ehyche@GM19403 Spectre (ehyche_fix_swift4)]$ swift build
Compile Swift Module 'Spectre' (8 sources)
/Users/ehyche/src/personal/Spectre/Sources/Expectation.swift:98:22: warning: redundant conformance constraint 'Key': 'Equatable'
public func == <Key: Equatable, Value: Equatable> (lhs: Expectation<[Key: Value]>, rhs: [Key: Value]) throws {
                     ^
/Users/ehyche/src/personal/Spectre/Sources/Expectation.swift:98:57: note: conformance constraint 'Key': 'Equatable' inferred from type here
public func == <Key: Equatable, Value: Equatable> (lhs: Expectation<[Key: Value]>, rhs: [Key: Value]) throws {
                                                        ^
/Users/ehyche/src/personal/Spectre/Sources/Expectation.swift:108:22: warning: redundant conformance constraint 'Key': 'Equatable'
public func != <Key: Equatable, Value: Equatable> (lhs: Expectation<[Key: Value]>, rhs: [Key: Value]) throws {
                     ^
/Users/ehyche/src/personal/Spectre/Sources/Expectation.swift:108:57: note: conformance constraint 'Key': 'Equatable' inferred from type here
public func != <Key: Equatable, Value: Equatable> (lhs: Expectation<[Key: Value]>, rhs: [Key: Value]) throws {
                                                        ^
/Users/ehyche/src/personal/Spectre/Sources/Expectation.swift:98:22: warning: redundant conformance constraint 'Key': 'Equatable'
public func == <Key: Equatable, Value: Equatable> (lhs: Expectation<[Key: Value]>, rhs: [Key: Value]) throws {
                     ^
/Users/ehyche/src/personal/Spectre/Sources/Expectation.swift:98:57: note: conformance constraint 'Key': 'Equatable' inferred from type here
public func == <Key: Equatable, Value: Equatable> (lhs: Expectation<[Key: Value]>, rhs: [Key: Value]) throws {

These changes fix those build errors. I also verified that it still builds against Swift 3.2 in Xcode 8.

Few last Swift development snapshots are yapping about this:

./Spectre/Sources/Expectation.swift:70:25: warning: string interpolation produces a debug description for an optional value; did you mean to make this explicit?
    throw lhs.failure("\(value) is equal to \(rhs)")
                        ^~~~~~~
./Spectre/Sources/Expectation.swift:70:26: note: use 'String(describing:)' to silence this warning
    throw lhs.failure("\(value) is equal to \(rhs)")
                        ~^~~~~~
                         String(describing:  )
./Spectre/Sources/Expectation.swift:70:26: note: provide a default value to avoid this warning
    throw lhs.failure("\(value) is equal to \(rhs)")
                        ~^~~~~~
                               ?? <#default value#>

This PR makes Swift a good tail-waggling boy... :wink: All builds and tests passed.

Swift 3.0
[x] - Build
[x] - Test
Swift 3.0.1
[x] - Build
[x] - Test
Swift DEV SNAPSHOT 2016-11-11
[x] - Build
[x] - Test

@kylef Please review and merge. If you do, please bump the Spectre version tag as well. Thanks.

All of Spectre tests added to global contexts start in atexit. XCode always says that they are passed. As really Spectre tests has not started yet.

If we start them manually with run. XCode also fails, as inside run exit is called.

Is there any plan to add ability testing with XCode?

First of all, thank you for working on this, very neat tool.

I just tried to compare the object type with my expectation. I got it to work with

https://github.com/ryuichis/swift-ast/blob/66bb167818d4ecc5b009f4f19085826e861d37f5/Tests/declaration/ParsingTypeAliasDeclarationSpec.swift#L119

However, if I change the line to

try expect(Mirror(reflecting: node.type).subjectType) == testType

then I got the following error:

Tests/declaration/ParsingTypeAliasDeclarationSpec.swift:119:71: error: binary operator '==' cannot be applied to operands of type 'Expectation<Any.Type>' (aka 'Expectation<protocol<>.Type>') and 'Type.Type'
                try expect(Mirror(reflecting: node.type).subjectType) == testType
                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^  ~~~~~~~~
Tests/declaration/ParsingTypeAliasDeclarationSpec.swift:119:71: note: overloads for '==' exist with these partially matching parameter lists: (FloatingPointClassification, FloatingPointClassification), (_MirrorDisposition, _MirrorDisposition), (Mirror.DisplayStyle, Mirror.DisplayStyle), (Bool, Bool), (Any.Type?, Any.Type?), (COpaquePointer, COpaquePointer), (Character, Character), (UInt8, UInt8), (Int8, Int8), (UInt16, UInt16), (Int16, Int16), (UInt32, UInt32), (Int32, Int32), (UInt64, UInt64), (Int64, Int64), (UInt, UInt), (Int, Int), (Float, Float), (Double, Double), (Float80, Float80), (ObjectIdentifier, ObjectIdentifier), (String, String), (Index, Index), (String.UnicodeScalarView.Index, String.UnicodeScalarView.Index), (String.UTF16View.Index, String.UTF16View.Index), (String.UTF8View.Index, String.UTF8View.Index), (UnicodeScalar, UnicodeScalar), (_SwiftNSOperatingSystemVersion, _SwiftNSOperatingSystemVersion), (Bit, Bit), (AnyForwardIndex, AnyForwardIndex), (AnyBidirectionalIndex, AnyBidirectionalIndex), (AnyRandomAccessIndex, AnyRandomAccessIndex), (AccessLevel, AccessLevel), (ContextualKeywordType, ContextualKeywordType), (KeywordType, KeywordType), (PunctuatorType, PunctuatorType), (Token, Token), (ContiguousArray<Element>, ContiguousArray<Element>), (ArraySlice<Element>, ArraySlice<Element>), (Array<Element>, Array<Element>), (AutoreleasingUnsafeMutablePointer<Memory>, AutoreleasingUnsafeMutablePointer<Memory>), (T, T), (LazyFilterIndex<Base>, LazyFilterIndex<Base>), (FlattenCollectionIndex<BaseElements>, FlattenCollectionIndex<BaseElements>), (FlattenBidirectionalCollectionIndex<BaseElements>, FlattenBidirectionalCollectionIndex<BaseElements>), (Set<Element>, Set<Element>), ([Key : Value], [Key : Value]), (SetIndex<Element>, SetIndex<Element>), (DictionaryIndex<Key, Value>, DictionaryIndex<Key, Value>), (_HeapBuffer<Value, Element>, _HeapBuffer<Value, Element>), (HalfOpenInterval<Bound>, HalfOpenInterval<Bound>), (ClosedInterval<Bound>, ClosedInterval<Bound>), (ManagedBufferPointer<Value, Element>, ManagedBufferPointer<Value, Element>), (T?, T?), (T?, _OptionalNilComparisonType), (_OptionalNilComparisonType, T?), (Range<Element>, Range<Element>), (ReverseIndex<Base>, ReverseIndex<Base>), (UnsafeMutablePointer<Memory>, UnsafeMutablePointer<Memory>), (UnsafePointer<Memory>, UnsafePointer<Memory>), ((A, B), (A, B)), ((A, B, C), (A, B, C)), ((A, B, C, D), (A, B, C, D)), ((A, B, C, D, E), (A, B, C, D, E)), ((A, B, C, D, E, F), (A, B, C, D, E, F)), (Self, Self), (E, E.ValueType)
                try expect(Mirror(reflecting: node.type).subjectType) == testType
                                                                      ^
make: *** [.build/debug/test_runner] Error 1

Tests could be even more readable if we could write something like try expect(anObject).to.beTypeOf(AClass) and try expect(anObject).to.beProtocolOf(AProtocol).

By adding an atexit hook when instantiating the global context, we can remove the need to call run manually.

There may be a better, more flexible solution to the problem, but I figured I'd throw the idea out there!

Similiar to mocha and other testing frameworks which allow you to see how long a test case takes. There should also be options to cause test failure when any test goes over some threshold. Mocha default reporter colours the time yellow/red to show slower tests.

Unfortunately this likely means we have to change the design of the reporters themselves.

Hi, @kylef

I want to test some async methods. Is it now supported in Spectre?

I just had a brief look at Spectre but didn't find this feature. Maybe something like this in other BDD testing framework:

$0.it("should do something asynchronously") {
    waitUntil { done in
        //.. async work
        done()
    }
}

If this is not yet in Spectre, I'd like to try to implement (although I have no experience on creating a test framework). Before I could get into it, I think it's better to confirm it. What do you think about it?