Upload failing iOS snapshot tests cases to S3

Overview

Second Curtain

Build Status

If you're using the cool FBSnapshotTestCase to test your iOS view logic, awesome! Even better if you have continuous integration, like on Travis, to automate running those tests!

Purpose

Isn't it frustrating that you can't see the results of your tests? At best, you'll get this kind of error output:

ASHViewControllerSpec
  a_view_controller_with_a_loaded_view_should_have_a_valid_snapshot, expected a matching snapshot in a_view_controller_with_a_loaded_view_should_have_a_valid_snapshot
  /Users/travis/build/AshFurrow/upload-ios-snapshot-test-case/Demo/DemoTests/DemoTests.m:31

        it(@"should have a valid snapshot", ^{
            expect(viewController).to.haveValidSnapshot();
        });

    Executed 1 test, with 1 failure (1 unexpected) in 0.952 (0.954) seconds
** TEST FAILED **

Wouldn't it be awesome if we could upload the failing test snapshots somewhere, so we can see exactly what's wrong? That's what we aim to do here.

Usage

Usage is pretty simple. Have an S3 bucket that is world-readable (that is, include the following bucket policy).

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "AllowPublicRead",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::bucket-name/*"
    }
  ]
}

(Replace "bucket-name" with your bucket name.)

It's also a good idea not to use a general-purpose S3 user for your CI, so create a new one with the following policy that will let them list buckets, but only read from or write to the bucket you're using.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListAllMyBuckets"],
      "Resource": "arn:aws:s3:::*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:*"
      ],
      "Resource": "arn:aws:s3:::bucket-name"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:*"
      ],
      "Resource": "arn:aws:s3:::bucket-name/*"
    }
  ]
}

OK, so the hard part is mostly done. Now that we have a place to upload our images, let's configure our build.

I use a Makefile to build and run tests on my iOS project, so my .travis.yml file looks something like this:

language: objective-c
cache: bundler

env:
  - UPLOAD_IOS_SNAPSHOT_BUCKET_NAME=bucket-name
  - UPLOAD_IOS_SNAPSHOT_BUCKET_PREFIX=an/optional/prefix
  - AWS_ACCESS_KEY_ID=ACCESS_KEY
  - AWS_SECRET_ACCESS_KEY=SECRET_KEY
  - AWS_REGION=OPTIONAL_REGION_DEFINITION

before_install:
  - bundle install

before_script:
  - export LANG=en_US.UTF-8
  - make ci

script:
  - make test

(You can take a look at how to encrypt information in your config file, but this has limitations due to how ecrypted variables are accessed via PRs on forks.)

My Makefile looks like this:

WORKSPACE = Demo/Demo.xcworkspace
SCHEME = Demo

all: ci

build:
	set -o pipefail && xcodebuild -workspace $(WORKSPACE) -scheme $(SCHEME) -sdk iphonesimulator build | xcpretty -c

clean:
	xcodebuild -workspace $(WORKSPACE) -scheme $(SCHEME) clean

test:
	set -o pipefail && xcodebuild -workspace $(WORKSPACE) -scheme $(SCHEME) -configuration Debug test -sdk iphonesimulator | second_curtain | xcpretty -c --test

ci:	build

Notice that we're piping the output from xcodebuild into second_curtain.

Note also that we're using xcpretty, as you should, too. The xcpretty invocation must come after the second_curtain invocation, since Second Curtain relies on parsing the output from xcodebuild directly.

And finally, our Gemfile:

source 'https://rubygems.org'

gem 'cocoapods'
gem 'xcpretty'
gem 'second_curtain', '~> 0.2'

And when any snapshot tests fail, they'll be uploaded to S3 and an HTML page will be generated with links to the images so you can download them. Huzzah!

Sample diff

Note that when the S3 directory is created, it needs to be unique. You can provide a custom folder name with the UPLOAD_IOS_SNAPSHOT_FOLDER_NAME environment variable. If one is not provided, Second Curtain will fall back to TRAVIS_JOB_ID or CIRCLE_BUILD_NUM, and then onto one generated by the current date and time.

Comments
  • preview

    preview

    Does the mouse drag etc, feel done working on this for today. Currently ignores the uploads.

    Can you think of a way to get the github PR address and the travis url?

    opened by orta 8
  • Link Travis-CI builds automatically

    Link Travis-CI builds automatically

    I think the timestamp for a folder name may be a little fragile. Allow to specify the folder from the Makefile and document how to setup Travis-CI (as an example) and reuse its environment variables to name the folder the same as the build number,

    opened by dblock 6
  • Exception in second_curtain

    Exception in second_curtain

    ..................................................................................................................................../Users/travis/build/artsy/eigen/vendor/bundle/ruby/2.0.0/gems/second_curtain-0.2.0/lib/second_curtain/parser.rb:23:in `parse_line': undefined method `fails=' for nil:NilClass (NoMethodError)
        from /Users/travis/build/artsy/eigen/vendor/bundle/ruby/2.0.0/gems/second_curtain-0.2.0/bin/second_curtain:8:in `block in <top (required)>'
        from /Users/travis/build/artsy/eigen/vendor/bundle/ruby/2.0.0/gems/second_curtain-0.2.0/bin/second_curtain:7:in `each_line'
        from /Users/travis/build/artsy/eigen/vendor/bundle/ruby/2.0.0/gems/second_curtain-0.2.0/bin/second_curtain:7:in `each_line'
        from /Users/travis/build/artsy/eigen/vendor/bundle/ruby/2.0.0/gems/second_curtain-0.2.0/bin/second_curtain:7:in `<top (required)>'
        from /Users/travis/.rvm/gems/ruby-2.0.0-p353/bin/second_curtain:23:in `load'
        from /Users/travis/.rvm/gems/ruby-2.0.0-p353/bin/second_curtain:23:in `<main>'
        from /Users/travis/.rvm/gems/ruby-2.0.0-p353/bin/ruby_executable_hooks:15:in `eval'
        from /Users/travis/.rvm/gems/ruby-2.0.0-p353/bin/ruby_executable_hooks:15:in `<main>'
    F** TEST FAILED **
    
    

    https://magnum.travis-ci.com/artsy/eigen/builds/5392603

    bug 
    opened by orta 4
  • Uploading False Matches

    Uploading False Matches

    Due to the way that Facebook has structured their library and the way that Specta/Expecta work, it's possible for different images to be written to disk and logged out that don't fail the tests (they're compared before the will block's timeout completes). We should do heavier log parsing, like here.

    bug 
    opened by ashfurrow 2
  • Reference images by filename only

    Reference images by filename only

    In the current version, the HTML template references images by their public url, as retrieved from the AWS SDK through the property public_url. However, now that the images are stored in the same folder as index.html, they could be reference by their filename only.

    A little backstory behind this PR: while we use Amazon S3 to store the snapshot image testing artefacts, we do not want them to be publicly available. Hence, we use a private bucket and only expose this bucket through a service that adds HTTP Basic Authentication. This won't work if images are referenced by their public path, as there is none in our case.

    opened by larslockefeer 1
  • Location of images vs location of index.html

    Location of images vs location of index.html

    While working in #33, I found that while the index.html ends up inside a folder, the images are not in the same folder but in the level above. As far as i understand, this means that the images of test n may be overwritten by the images of test n + 1. Is this the expected behaviour?

    opened by larslockefeer 1
  • Fix for #12

    Fix for #12

    Hi Ash,

    While investigating #12, I found the following:

    • bucket.objects expects a path without a leading /
    • In some cases, too many slashes were added to a path in Upload and UploadManager, leading to the described bug

    With this PR, I introduce PathUtils, a small class that allows you to build a path from components, ensuring that the end result does not contain leading and trailing slashes.

    All tests still pass and the uploads work fine for me. There may be a Ruby standard-library function for this, but not that I am aware of 🤓

    opened by larslockefeer 1
  • Update to logic around deciding what snapshots to show

    Update to logic around deciding what snapshots to show

    https://github.com/orta/Snapshots/commit/1eb3b3880b58910cc3c5109e6da97a6a19cbde98#diff-540fbd6334e0d2f5602d5e0ab469111bR74

    Fixes this use case: https://github.com/orta/Snapshots/commit/1eb3b3880b58910cc3c5109e6da97a6a19cbde98#diff-3ee3a5b11f02a33f691495fc2633356cR17 - which is super common in most of our projects

    opened by orta 1
  • Fixes tests

    Fixes tests

    Hey @larslockefeer, this addresses the test changes I mentioned. Does it look OK to you? I've added you as a collaborate on this repo, so merge when you think it's ready. I can submit a new release to ruby gems today. Thanks again!

    Merges #36.

    opened by ashfurrow 0
  • Add support for passing in a file

    Add support for passing in a file

    using tee it's easy to output Xcode's build logs into a separate file - we can make second_curtain a separate process that happens after. Which should fix the weird positioning of the link for it.

    opened by orta 0
  • circleci

    circleci "support"

    Hi,

    I just discovered circleci artifacts, and this prevents you to need to upload to S3 bucket or other "third-part" service. You only needs to copy to move to $CIRCLE_ARTIFACTS folder, and it will be available from circleci panel, and you can share link of these files, because it's public! :dancer:

    NOTE: I don't have any relation with circleci, just I'm preparing a talk about fastlane, and I was playing with it :D

    opened by patoroco 1
  • Possible false positives with heterogeneous test cases

    Possible false positives with heterogeneous test cases

    Say you have a test case that tests a view with snapshots and then does some other test.

    expect(view).will.haveValidSnapshot(); // pass, but does emit a "ksdiff" line
    expect(1).to.equal(2); //fails, but the kaleidoscope command from the previous line is recorded as a failure in second curtain
    

    If the first expectation emits a "ksdiff" line to the log, as is common will will blocks, then the second, failing expectation will mark the first one, which passed, as a failure to be uploaded.

    /cc @orta

    bug 
    opened by ashfurrow 0
  • FTP support

    FTP support

    Hi,

    My use case would be to send it over to the QA or Design guys so they can see the result or monitor these kind of tests. I don't use S3, so why not publishing to a ftp server or something else.

    opened by netbe 2
Owner
Ash Furrow
Building software with my whole heart. He/him.
Ash Furrow
Snapshot testing tool for iOS and tvOS

SnapshotTest is a simple view testing tool written completely in Swift to aid with development for Apple platforms. It's like unit testing for views.

Pär Strindevall 44 Sep 29, 2022
iOS Simulator type agnostic snapshot testing, built on top of the FBSnapshotTestCase.

SnappyTestCase iOS Simulator type agnostic snapshot testing, built on top of the FBSnapshotTestCase. Why? Snapshot testing helps to deliver views that

Tooploox 15 Oct 11, 2019
📸 Delightful Swift snapshot testing.

?? SnapshotTesting Delightful Swift snapshot testing. Usage Once installed, no additional configuration is required. You can import the SnapshotTestin

Point-Free 3k Jan 3, 2023
Swift Package with examples of how to tests iOS apps

GimmeTheCodeTDD A swift package with examples of unit tests in iOS development. Requirements Xcode 13 Content Dependency Injection Constructor Injecti

Dominik Hauser 8 Oct 11, 2021
Library for unifying the approach to network mocking in iOS unit- & UI-tests.

TinkoffMockStrapping Example To run the example project, clone the repo, and run pod install from the Example directory first. Requirements Installati

Online financial ecosystem 22 Jan 3, 2023
Bluepill is a reliable iOS testing tool that runs UI tests using multiple simulators on a single machine

Bluepill is a tool to run iOS tests in parallel using multiple simulators. Motivation LinkedIn created Bluepill to run its large iOS test suite in a r

Mobile Native Foundation 3.1k Jan 3, 2023
A collection of useful test helpers designed to ease the burden of writing tests for iOS applications.

MetovaTestKit is a collection of useful test helpers designed to ease the burden of writing tests for iOS applications. Requirements Installation Usag

null 23 Aug 29, 2021
TestSchedulerDemo - Demonstration code for iOS Unit Tests and Asynchronous

TestSchedulerDemo This repository contains demonstration code for my Medium arti

Carsten Wenderdel 0 Mar 19, 2022
Erik is an headless browser based on WebKit. An headless browser allow to run functional tests, to access and manipulate webpages using javascript.

Erik Erik is a headless browser based on WebKit and HTML parser Kanna. An headless browser allow to run functional tests, to access and manipulate web

Eric Marchand 544 Dec 30, 2022
Mockit is a Tasty mocking framework for unit tests in Swift 5.0

Mockit Introduction Mockit is a Tasty mocking framework for unit tests in Swift 5.0. It's at an early stage of development, but its current features a

Syed Sabir Salman-Al-Musawi 118 Oct 17, 2022
Trying to implement Unit Tests for @Binding properties in a ViewModel

BindingTester Trying to implement Unit Tests for @Binding properties in a ViewModel ViewModel to be tested class SheetViewModel: ObservableObject {

Raphael Guye 0 Oct 22, 2021
Catching fatal errors in unit tests

Precondition Catching When running tests which hit fatal errors, often preconditions the built-in support with XCTest. One package which supports cach

Brennan Stehling 0 Nov 28, 2021
Small library to easily run your tests directly within a Playground

[] (https://developer.apple.com/swift/) Build Status Branch Status master develop About PlaygroundTDD enables you to use TDD directly on Xcode Playgro

Whiskerz AB 317 Nov 22, 2022
Write unit tests which test the layout of a view in multiple configurations

Overview This library enables you to write unit tests which test the layout of a view in multiple configurations. It tests the view with different dat

LinkedIn 565 Nov 16, 2022
UITest helper library for creating readable and maintainable tests

UITestHelper General information When creating UI tests you will see that you are often repeating the same pieces of code. The UITestHelper library wi

Edwin Vermeer 56 Feb 20, 2022
Swift framework containing a set of helpful XCTest extensions for writing UI automation tests

AutoMate • AppBuddy • Templates • ModelGenie AutoMate AutoMate is a Swift framework containing a set of helpful XCTest extensions for writing UI autom

PGS Software 274 Dec 30, 2022
Tool for generating Acceptance Tests in Xcode, inspired by Fitnesse

AcceptanceMark is a tool for generating Acceptance Tests in Xcode, inspired by Fitnesse. Read this blog post for a full introduction to AcceptanceMark

Andrea Bizzotto 64 Jun 18, 2022
Freezer is a library that allows your Swift tests to travel through time by mocking NSDate class.

Freezer Freezer is a library that allows your Swift tests to travel through time by mocking NSDate class. Usage Once Freezer.start() has been invoked,

Sergey Petrov 8 Sep 24, 2022
Marvel - Marvel Characters App using MVVM, and including unit tests

Marvel About The purpose of this project is to develop a app using MVVM, and inc

null 1 Mar 20, 2022