It seems that Apple Silicon machines generate snapshots that are slightly different from the ones generated on x86 machines. The problem is very similar to the one where the very same simulator generates different images depending on the OS version.

The issues persist when running the simulator using Rosetta.

Taking into account that it’ll be quite common in the near future to have a team of developers with Intel and Apple Silicon architectures… What’s the best way to handle it?

Thanks

Problem

The existing image matching precision strategy is not good at differentiating between a significant difference in a relatively small portion of a snapshot, and imperceivable differences in a large portion of the snapshot. For example, these snapshots below show that a 99.5% precision value fails an imperceivable background color change while allowing noticeable changes (text and color) to pass:

| Reference | Fails | Passes | |:--:|:--:|:--:| | | | | | | Imperceivable background color difference | Significant text and color changes |

Solution

This PR adds a new optional perceptualPrecision parameter to image snapshotting which determines how perceptually similar a pixel must be to consider it matching. This parameter complements the existing precision parameter that determines the percentage of pixels that must be considered matching in order to consider the whole image matching.

This approach is similar to https://github.com/pointfreeco/swift-snapshot-testing/pull/571 and https://github.com/pointfreeco/swift-snapshot-testing/pull/580 but uses perceptual distance rather than Euclidean distance of sRGB values. This is significant because the sRGB color space is not perceptually uniform. Pairs of colors with the same Euclidean distance can have large perceptual differences. The left and right colors of each row have the same Euclidean distance:

The perceptualPrecision parameter is the inverse of a Delta E value. The following table can be used to determine the perceptualPrecision that suites your needs:

Perceptual Precision Value | Description -- | -- 100% | Must be exactly equal ≥ 99% | Allows differences not perceptible by human eyes ≥ 98% | Allows differences possibly perceptible through close observation ≥ 90% | Allows differences perceptible at a glance ≥ 50% | Allows differences when more similar than opposite ≥ 0% | Allows any differences

This perceptual color difference calculation is performed by the CILabDeltaE CoreImage Filter which is available on macOS 10.13+, iOS 11+, and tvOS 11+. The use of CoreImage to accelerate the image diffing results in a +90% over the existing byte-by-byte comparison.

~~Additionally, when macOS 11.0+, iOS 14+, tvOS 14+ is available, the CILabDeltaE filter is joined with the CIColorThreshold and CIAreaAverage filters to accelerate the whole image comparison, resulting in a 97% speed improvement.~~ The CIColorThreshold filter has been backported to 10.13+, iOS 11+, and tvOS 11+ using MPSImageThresholdBinary so the 97% speed improvement is available in all OS versions.

Benchmarks

| Byte-Buffer Comparison | CILabDeltaE + MPSImageThresholdBinary + CIAreaAverage | |:--:|:--:| | macOS 10.10+, iOS 11+, tvOS 10+ | macOS 10.13+, iOS 11+, tvOS 11+ | | 0.906s - baseline | 0.025s - 97% speed improvement | | | |

Addressed Issues

• closes https://github.com/pointfreeco/swift-snapshot-testing/issues/313
  • A perceptualPrecsion of >=98% will prevent imperceivable differences from failing assertions while still catching noticeable differences
• closes https://github.com/pointfreeco/swift-snapshot-testing/issues/424
  • The small differences in Intel/Apple Silicon hardware accelerated anti-aliasing/shadows/blur rendering are under the 2% DeltaE value which is nearly imperceivable by the human eye
• closes https://github.com/pointfreeco/swift-snapshot-testing/issues/573
  • Thanks to CoreImage acceleration, the perceptual precision comparison is 90-97% faster than the exact pixel precision comparison

Related PRs

• https://github.com/pointfreeco/swift-snapshot-testing/pull/288
• https://github.com/pointfreeco/swift-snapshot-testing/pull/490
• https://github.com/pointfreeco/swift-snapshot-testing/pull/571
• https://github.com/pointfreeco/swift-snapshot-testing/pull/580

Hi folks!

Why it has no been releases lately? for those who use semantic versions in packages, we can't have the latest fixes unless we point to master, is there any reason behind it?

Thanks

Subpixel difference

Workaround for the issue described in #313 and #424.

The readme clearly states that the reference-device should be identical to the recorder-device, but as M1 Macs are becoming popular with developers at a different rate to CI & cloud providers, that is becoming increasingly difficult.

Allow each subpixel (R, G, B) to deviate by up to a certain degree. Currently, a precision: 0.9 means that "90% of the pixels must match 100%". The issues described in particularly #313, where some experience that the pixels deviate by only 1, can be resolved by comparing with subpixelThreshold: 1. Comparing with precision: 0.9, subpixelThreshold: 1 then means "90% of the subpixels' value must be ±1 of the reference".

Optimization

When an image comparison memcmp fails and we need to compare each byte of the failing image, the current loop iteration is slow when compiled without optimization on Intel machines.

My team defaulted to adding custom compiler flags to SnapshotTesting, but the problem can also be diminished significantly by using a more basic loop implementation.

The difference between looping over a range (current implementation) and looping with a condition can be seen on Godbolt.

func byteDiff1(reference: [UInt8], comparison: [UInt8]) -> Bool {
    let maxDiffs = 100
    var diffs = 0
    for byte in 0..<reference.count {
        if reference[byte] != comparison[byte] {
            diffs += 1
            if diffs >= maxDiffs {
                return false
            }
        }
    }
    return true
}

func byteDiff2(reference: [UInt8], comparison: [UInt8]) -> Bool {
    let maxDiffs = 100
    var diffs = 0
    var byte = 0
    while byte < reference.count {
        if reference[byte] != comparison[byte] {
            diffs += 1
            if diffs >= maxDiffs {
                return false
            }
        }
        byte += 1
    }
    return true
}

Hi everybody, I'm having an issue here with color discrepancies in an image view. I'm using KIF framework for UI Tests and SnapshotTesting to take screenshots between interactions (asserts made on the app key window). Everything works fine, except that I get color differences from snapshots taken on my local machine vs snapshots taken on CI machine (both running the same stack, Mojave + Xcode 11.3).

Some examples below:

This one is taken by CI machine

This one is taken by my local computer

This is the comparison of both images made by imagemagick

And these are the color discrepancies of a same pixel, measured by macOS Digital Color Meter

I know there are lots of variables and also using KIF, but do you guys have any idea of what might be causing this difference?

Thanks!

The problem is that rendering system on different hardwares with the same macOS Catalina version may produce not matched image snapshots. There may be several bytes that have difference just of 1, e.g. in one bitmap some byte has value 212 and in other bitmap the same byte has value 213. Such difference isn't visible by eye even in generated difference_<uuid>.png image. So snapshots generated on developer computer always fail on CI computer. The solution is to ignore the difference of 1 and fail only in case if difference greater than 1.

Describe the bug Podspec missing in version 1.10.0. Cocoapods is still widely used in the industry. Many other dependencies don't have SPM support, which makes it even harder if some others only support SPM..

To Reproduce run pod install with version 1.10.0

Expected behavior version 1.10.0 should support Cocoapods to prevent headaches. Please add the support back, we waited so long for the m1 fix 🙏

When a user sets a custom navigation delegate on a WKWebView and makes a snapshot of a view containing this web view, the navigation delegate may be replaced by a stub that makes sure the snapshotting lib knows when to snapshot the web view. This works great when users don't implement a custom WKNavigationDelegate themselves. But when a user implements a custom delegate themselves this fails.

Consider for example a custom delegate that limits the web view to github.com and opens the browser as soon as the user taps on a link outside of github.com

class GithubNavigationDelegate: NSObject, WKNavigationDelegate {
  override init() { }

  public func webView(
    _ webView: WKWebView,
    decidePolicyFor navigationAction: WKNavigationAction,
    decisionHandler: @escaping (WKNavigationActionPolicy) -> Void
  ) {
    // limit to webpages on to github.com
    guard navigationAction.request.url?.host == "github.com" else {
      decisionHandler(.cancel)
      // Warn user and open url in browser instead
      return
    }

    decisionHandler(.allow)
  }
}


webview.navigationDelegate = GithubNavigationDelegate()

This can't be snapshot tested at the moment because the delegate is removed as soon as a snapshot is made.

I think this could be fixed by retaining the original delegate, implementing all the possible delegate methods and forwarding all the calls to the original delegate. I'll see if I can draft up a PR. Please let me know if you think of another way of dealing with this issue.

Finally, many thanks for all the work on this great lib 👏🏼 so far, hopefully I can contribute back a bit.

What do you recommend for using this with continuous integration? If the images need to be stored and then subsequently re-run so a comparison can be made, then how would you typically configure your CI setup to keep these images around? I appreciate that there are many CI tools out there and I'm not suggesting specifics. Just some general guidance in the README would be great. Thanks.

Can not make SnapshotTesting work. When I use assertSnapshot(matching: view, as: .image) I get this message "The file "failed "Test.swift" doesn’t exist"

What I can do ?

Followup of https://github.com/pointfreeco/swift-snapshot-testing/pull/228, Fixes https://github.com/pointfreeco/swift-snapshot-testing/issues/230

This PR fixes inline snapshot tests containing special characters. AssertInlineSnapshot now checks whether the snapshot contains special characters or not. If it does, escapes the snapshot literal appropriately.

Known issue: AssertInlineSnapshot still searches only for "" and multiline string literals (it does not search #""" """# for example)

This will improve https://github.com/pointfreeco/swift-snapshot-testing/issues/625 a bit.

Without this patch snapshots crash, with this patch, only the background color of the webview is snapped.

With the addition of CleanCounterBetweenTestCases in 1.10 to support multiple runs of tests popular testing framework Quick no longer works.

Quick lets users specify test cases declaratively and creates multiple tests from the same function. So when I specify multiple snapshots within one spec() function the counterMap gets reset inbetween each one breaking the automatic naming.

I understand we want this feature to support the new feature of re running tests, but would it be possible to give users some more flexibility as to when the counterMap gets reset or the ability to turn it off?

Environment

• swift-snapshot-testing: 1.10
• Xcode: 14.2
• OS: iOS 16

When snapshot testing screenshots in landscape orientation, the diffs are very noisy as the original orientation of the reference image is ignored and it is treated, for the purpose of the visual diff, as a portrait orientation image. (The actual comparison seems to work correctly in a pass/fail sense, however.)

Unfortunately, this makes it very hard to see what actually changed. For example when removing a button in a drawing app I am currently working on, this is the diff I receive in the current version of the repository:

I believe this issue can be resolved by using the orientation from the new image for the visual diff which results in the following much more readable output:

I don't have enough context to know if my solution has drawbacks for the library/community as a whole so I would certainly appreciate review/discussion to see if there is a better solution.

Describe the bug SwiftUI, iOS16: LazyVGrid wrapped into NavigationView is not displayed correctly when using verifySnapshot. Works correctly on iOS15

To Reproduce SwiftUI, iOS16: Wrap LazyVGrid into NavigationView -> LazyVGrid displayed incorrectly.

FYI: Outside of the NavigationView, LazyVGrid displayed correctly.

verifySnapshot(matching: view, as: .image(layout: .device(config: .iPhoneX)), testName: testName)

Expected behavior LazyVGrid content is displayed correctly when placed in NavigationView on a snapshot

Screenshots Actual Result: Expected Result:

Environment

• swift-snapshot-testing version [1.9.0]
• Xcode [14.1]
• Swift [5.7]
• OS: [iOS 16.0]

I've had some issues with some objects, where a memory address would not be correctly erased. I think it would be better if this was customizable from the outside, because right now there doesn't seem to be any way to customize the output of the snap function. What do you think?

Describe the bug Form isn't being rendered properly in Snapshots.

To Reproduce Used this code

// And/or enter code that reproduces the behavior here.
class SampleSnapshotTest: XCTestCase {
    
    func testDefault() {
        let vc = UIHostingController(rootView: SampleForm())
        vc.view.frame = UIScreen.main.bounds
        
        assertSnapshot(matching: vc, as: .image)
    }
}

struct SampleForm: View {
    
    var body: some View {
        Form {
            Section {
                Text("Item1")
                Text("Item1")
                Text("Item1")
                Text("Item1")
                Text("Item1")
            } header: {
                Text("Header")
            }
        }
    }
}

Expected behavior

Screenshots

Environment

• swift-snapshot-testing version 1.10.0
• Xcode Version 14.2 (14C18)
• Swift 5.7.2
• OS: 16.2