A SwiftUI component for launching custom picture-in-picture experiences

Overview

Pipify for SwiftUI

This library introduces a new SwiftUI modifier that enables a view to be shown within a Picture in Picture overlay. This overlay allows information to be displayed on the screen even when the application is in the background.

This could be great for applications like navigation apps and stock market apps which rely on real-time information being shown to the user.

Note: This project currently relies on SwiftUI 4 and as such has a minimum deployment target of iOS 16, tvOS 16, or macOS 13 - all of which are currently in beta. This library does not work on watchOS.

Getting Started

Installation

We currently support installation through Swift Package Manager.

https://github.com/getsidetrack/swiftui-pipify.git

Project Configuration

You will need to enable "Background Modes" in your project entitlements window. Specifically, you need the Audio, AirPlay and Picture in Picture option to be enabled. Without this picture in picture mode cannot launch.

Pipify View

The "pipify view" is the view which is actually shown within the picture-in-picture window. This can either be the view which you add the pipify modifier to, or a completely different view.

If you do not provide a custom pipify view, then we will use the view that the modifier was added to. By default this will use Apple's 'morph' transition which will animate the view into the picture-in-picture controller.

When a custom pipify view is provided, we will render this offscreen which causes picture-in-picture to simply fade in.

Your pipify view can be any SwiftUI view, but there are some important notes to be aware of:

  1. This view will be rendered invisibly when created, so closures such as task and onAppear may be called when you don't expect it.
  2. Most user interactions are not supported - so buttons, tap gestures, and more will not work.
  3. Animations and transitions may result in unexpected behaviour and has not been fully tested.

Usage

Simply add the pipify modifier to your SwiftUI view. There are two key signatures based on whether you want to provide your own custom pipify view (see above).

@State var isPresented = false

var body: some View {
    yourView
        .pipify(isPresented: $isPresented) // presents `yourView` in PIP
        
    // or

    yourView
        .pipify(isPresented: $isPresented) {
            SomeOtherView() // presents `SomeOtherView` in PIP
        }
}

In the example above, you can replace yourView with whatever you'd like to show. This is your existing code. The state binding is what determines when to present the picture in picture window. This API is similar to Apple's own solutions for example with sheet.

If you provide a custom SwiftUI view as your pipify view, then you may choose to add our pipify controller as an environment object in that separate view ("SomeOtherView" in the example code above).

Note that you cannot use the EnvironmentObject in the view which specifies the pipify modifier.

@EnvironmentObject var controller: PipifyController

This will give you access to certain variables such as the renderSize which returns the size of the picture-in-picture window.

Alternatively, you can attach our custom closures to your view to get informed about key events.

yourPipifyView
    .onPipRenderSizeChanged { size in
        // size has changed
    }

A basic example project is included in the repository. Want to share your own example? Raise a pull request with your examples below.

Testing

Pipify will not launch on unsupported devices and will return an error in the debug console stating that it could not launch. You can check whether a device is compatible by using PipifyController.isSupported which returns true or false. You may use this to show or hide the pip option in your application.

You must test this library on physical devices. Due to issues in simulators outside of our control, you will see various inconsistencies, lack of support and other bugs when run on simulators.

How does this work?

This library utilises what many may refer to as a "hack" but is essentially a feature in Apple's picture-in-picture mode (PiP).

Picture-in-Picture has been around for quite a while first launching with iOS 9 in 2015 and was later brought to macOS 10.15 in 2019 and tvOS 14 most recently in 2020. It provides users with the ability to view video content while using other applications, for example watching a YouTube video while reading tweets.

Pipify expands on this feature by essentially creating a video stream from a SwiftUI view. We take a screenshot of your view anytime it updates and push this through a series of functions in Apple's AVKit. From these screenshots which we turn into a video stream, we can launch picture-in-picture mode.

From a user's perspective, the view is moved in a window above the application. Video controls may be visible and can be hidden by tapping on them. Users can temporarily hide or close the picture-in-picture window at any time.

⚠️ There is no reason to believe that this functionality breaks Apple's App Store guidelines. There are examples of apps on the App Store which use this functionality (Example), but there has also been cases of apps being rejected for misuse of the API (Example).

We recommend that developers proceed with caution and consider what the best experience is for their users. You may want to explore ways of implementing sound into your application.

Thanks

Credit goes to Akihiro Urushihara with UIPiPView which was the inspiration for building this SwiftUI component.

Comments
  • Breaking Changes: API update and package rename

    Breaking Changes: API update and package rename

    • dockable has been changed to pipify
    • developer no longer manages the PipifyController (simplifying implementation)
    @State var showPip = false
    
    someView
      .pipify(isPresented: $showPip) // uses self as the view (morph effect)
    
    // or
    
    someView
      .pipify(isPresented: $showPip) {
        SomeOtherView() // uses view in closure as the view (fade effect)
      }
    
    // optional:
    // in your view you can add the controller as an environment object
    
    @EnvironmentObject var controller: PipifyController
    
    // this gives you access to the renderSize etc
    // alternatively:
    
    yourPipifyView
        .onPipRenderSizeChanged { size in
            // new size goes here
        }
    

    Closes #5 Closes #10

    opened by Sherlouk 4
  • View not being re-rendered

    View not being re-rendered

    Looking at the examples, I'm not seeing the view being updated with the width and height information. In my application that uses Pipify, I'm also not seeing the view being re-rendered even though the code in setView's renderer.objetWillChange is being called.

    opened by adamtow 3
  • View rendered is of low quality

    View rendered is of low quality

    I'm passing a simple view into the Pipify modifier, and the resulting rendered view (text and images) is of low quality. It's like it's scaling up from the view. I tried setting the renderer.scale property but that didn't seem to affect anything. This is evident in the Example project as well. Is there a trick to get high quality text and images in the Pip view?

    opened by adamtow 2
  • Add Support for Button Interactivity

    Add Support for Button Interactivity

    Progress on #10 Closes #13

    Work in progress, likely to require more structural changes to provide custom modifiers for this.

    Ideally the play/pause button would only 'work' (store changes) if a closure is provided. Likewise the skip buttons should only be active if a closure is provided for those.

    opened by Sherlouk 1
  • Seamless Docking

    Seamless Docking

    At present, we use a hidden view to transition into the PiP state. This can make the transition a bit jerky and potentially confusing. It would be nice to give the developers the option to dock an existing view instead of docking a hidden view.

    This would be achieved by removing the content: parameter in the existing dockable modifier. If no content is provided, the attached view is docked. If content is provided, that is docked instead.

    opened by Sherlouk 1
  • Examples

    Examples

    It would be good to share examples in the community (whether tweets, code samples, or released apps) so people can be inspired and see it in action.

    https://apps.apple.com/us/app/minispeech-live-transcribe/id1576069409 - released application https://twitter.com/jordibruin/status/1543247957482405888 - proof of concept

    documentation 
    opened by Sherlouk 0
  • Support Multiple Dockable Views Simultaneously

    Support Multiple Dockable Views Simultaneously

    We should explore the possibility of having a single controller being able to manage multiple views at the same time. It appears Apple does not like you have multiple picture in picture controllers in the lifecycle at once.

    opened by Sherlouk 0
  • Make Controls Usable

    Make Controls Usable

    Allow users to interact with the play/pause button and potentially even the skip forward/backwards button and have the view change as a result of that.

    opened by Sherlouk 0
  • hide playback controls

    hide playback controls

    The combination of the pipController.requiresLinearPlayback = true and the CMTimerange hides the backwards and forwards controls.

    I'm not sure what the CMTimeRange needs to be so let's triple check that stuff

    opened by jordibruin 0
  • Provide Render Size

    Provide Render Size

    Apple provides us with a delegate call whenever the render size of the PiP window is changed. This is because not only does Apple themselves mostly decide the size, but also the user can resize the window.

    Some clients may wish to dynamically change what content is shown based on how large the window is on the screen - as such it would be good to route this information through to the view when we render it. This could likely be done with a custom environment key.

    good first issue 
    opened by Sherlouk 0
  • Improved Diagnostics and Logging

    Improved Diagnostics and Logging

    We need to ensure we're providing a suitable amount of structured logging in order to aide with debugging and implementation.

    We should ensure that we're providing a good separation between warnings/errors and general information - perhaps with opt-ins through a log level.

    We may consider using swift-log.

    good first issue 
    opened by Sherlouk 0
  • Spinner instead of content

    Spinner instead of content

    When you build the sample app or an app based on the SPM using Xcode 14.0.1, it SwiftUI PIP view seems to have been replaced by a gray view with a spinner in it. If you close the PIP window, you see the correct SwiftUI view for a brief moment before the PIP disappears.

    opened by adamtow 1
  • `EXC_BAD_ACCESS` on startup in macOS 13 beta 5

    `EXC_BAD_ACCESS` on startup in macOS 13 beta 5

    Try to run the example project on macOS 13 Beta 5 (build number: 22A5321d), then it crashed on startup with EXC_BAD_ACCESS error, log below:

    2022-08-16 02:05:10.214520+0800 Sample[18556:300581] [User Defaults] Couldn't read values in CFPrefsPlistSource<0x600001008980> (Domain: com.apple.powerlogd, User: kCFPreferencesAnyUser, ByHost: Yes, Container: (null), Contents Need Refresh: Yes): accessing preferences outside an application's container requires user-preference-read or file-read-data sandbox access
    2022-08-16 02:05:10.495536+0800 Sample[18556:300581] [Pipify] creating pip controller
    2022-08-16 02:05:10.496087+0800 Sample[18556:300581] [Pipify] creating pip controller
    2022-08-16 02:05:10.532626+0800 Sample[18556:300581] *** Assertion failure in NSPoint _NSViewClampPointToLimits(NSView *, NSString *, NSPoint)(), NSView.m:640
    2022-08-16 02:05:10.533196+0800 Sample[18556:300581] Invalid parameter not satisfying: fabs(point.x) <= NSVIEW_GEOMETRY_LIMIT
    2022-08-16 02:05:10.538790+0800 Sample[18556:300581] [Pipify] setting view content
    2022-08-16 02:05:10.539346+0800 Sample[18556:300581] [Pipify] setting view content
    

    It looks like crash on makeNSView(_:) in LayerView. Maybe https://github.com/utmapp/UTM/commit/d5ba4fb7f32cc80a7489043b22d8e6c73a591851 helps.

    bug 
    opened by FradSer 1
  • Automated Tests

    Automated Tests

    This library definitely relies on a few 'tricks' and so it's going to be an interesting challenge to add a number of UI and snapshot tests to prove that it works as intended.

    We should look to use UI tests as a way to follow a step by step set of instructions and snapshot each step of the way to prove that the device looks and is responding as expected. Specifically, we should be clearly seeing the PIP window launch with the right content and prove that it's updating dynamically.

    opened by Sherlouk 0
  • Improve Documentation and Sample App

    Improve Documentation and Sample App

    Alongside a much richer sample app with numerous examples showing the different capabilities, we also want to provide a DocC documentation bundle with rich real-world examples with simple to follow instructions.

    We should look at adding one or more DocC tutorials to allow people to follow along at home.

    I also feel like from a contributing perspective I should add an architecture diagram which better explains exactly how all of this works.

    good first issue 
    opened by Sherlouk 0
  • Add Redraw Safety to Pipify Modifier

    Add Redraw Safety to Pipify Modifier

    SwiftUI redraws views often, this is a big part of it's architecture to allow for quick and dynamic updates to views. Unfortunately, this causes issues with Pipify because it regenerates the PIP controller every time.

    This has been seen in this Twitter thread (https://twitter.com/aoverholtzer/status/1546249575748358144) and my own experiences.

    We need to ensure we protect against this.

    opened by Sherlouk 2
  • Add Error for Unsupported Device

    Add Error for Unsupported Device

    Currently we do not enforce the isSupported check - it would be logical to move this up into the PipifyModifier so we're not adding extra load onto unsupported devices.

    We should also make sure we're adding a log message (warning) so developers are aware of this.

    Notably, this will detect simulators which causes some weird edge cases at the moment.

    good first issue 
    opened by Sherlouk 0
Releases(0.3.0)
  • 0.2.0(Jul 5, 2022)

    🚨 Breaking Change

    This release contains a rename from dockable to pipify and large changes to the public API.

    • You can no longer create the PipifyController (formerely DockableController) - we manage this for you, simplifying integration.
    • You can initialise a PIP experience by providing a isPresented binding, much like other Apple APIs.
    • You can now provide a separate SwiftUI view (which will fade in) or morph a view in your hierarchy.
    • You can now update your UI based on the render size of the PIP window.
    • You can now listen to PIP button presses (play, pause, skip forward, skip backward).
    • You can now change the percentage filled on the playback progress bar (useful for timers!)
    • Added SwiftUI modifiers to the Xcode library to help in development
    • Improved logging throughout to aide in debugging
    • Improved sample app to better showcase the functionality available

    Contributors

    Thanks to @jordibruin who has contributed massively to this release, both with testing, marketing, and feature contributions!

    Full Changelog: https://github.com/getsidetrack/swiftui-pipify/compare/0.1.0...0.2.0

    Source code(tar.gz)
    Source code(zip)
  • 0.1.0(Jul 1, 2022)

    This should not be used in production applications and has not been tested rigorously. More changes (including the public interface) may be made shortly.

    Source code(tar.gz)
    Source code(zip)
Owner
Sidetrack
An evolution to travelling by train
Sidetrack
Pipable is an iOS library to implement Picture in Picture for any UIView.

Pipable Pipable is an iOS library to implement Picture in Picture for any UIView. This is done just by conforming to a protocol. The "Audio, AirPlay a

Emma Cold 11 Nov 3, 2022
Space! – an iOS widget displaying NASA's Astronomy Picture of the Day

Space! NASA's Astronomy Picture of the Day – now on your Home Screen with widgets! Space! displays the latest APOD photo curated by NASA every day. Se

Jacob Bandes-Storch 72 Dec 17, 2022
This simple cordova plugin will download picture from an URL and save to IOS Photo Gallery.

Photo Viewer This plugin is intended to download a picture from an URL into IOS Photo library.. How to Install Cordova: cordova plugin add https://git

Alwin jose 1 Oct 23, 2021
A UIImageView extension to let the picture-cutting with faces showing better

UIImageView-BetterFace A UIImageView extension to let the picture-cutting with faces showing better Last update in v0.2_stable : add a UIImage+BetterF

Croath Liu 779 Sep 1, 2022
A apple music cover picture shadow style image library

ShadowImageView A apple music cover picture shadow style image library ShadowImageView is a iOS 10 Apple Music style image view, help you create elege

Old Donkey 794 Dec 17, 2022
Picture anonymiser using Vision face recognition

?? Anonymojizer [WIP] Anonymize people in photos by replacing their faces by emojis. How to use it ? Pick a photo from the gallery Choose an emoji The

Kaww 1 Dec 20, 2021
GIFImage component for your SwiftUI app!

SwiftUI GIF Lightweight SwiftUI component for rendering GIFs from data or assets, with no external dependencies. As a bonus, there's an extension that

Gordan Glavaš 17 Dec 6, 2022
AlamofireImage is an image component library for Alamofire

AlamofireImage AlamofireImage is an image component library for Alamofire. Features Image Response Serializers UIImage Extensions for Inflation / Scal

Alamofire 3.9k Dec 29, 2022
ImageView - Component for loading and displaying different images aka SVG/PNG/JPG/JPEG

ImageView Component that loads and displays images(.svg/.png/.jpg/.jpeg) form as

Sergei 1 Mar 23, 2022
A custom ImageView that is used to cover the surface of other view like a scratch card, user can swipe the mulch to see the view below.

MCScratchImageView GIF Showcase Requirments iOS 8.0+ Xcode 7.2+ Swift 4.0 Installation CocoaPods pod "MCScratchImageView" Manually Just drag MCScratch

Jaylen Bian 359 Dec 17, 2022
Image picker with custom crop rect for iOS written in Swift (Ported over from GKImagePicker)

WDImagePicker Ever wanted a custom crop area for the UIImagePickerController? Now you can have it with WDImagePicker. Just set your custom crop area a

Wu Di 96 Dec 19, 2022
📸 iMessage-like, Image Picker Controller Provides custom features.

RAImagePicker Description RAImagePicker is a protocol-oriented framework that provides custom features from the built-in Image Picker Edit. Overview O

Rashed Al-Lahaseh 14 Aug 18, 2022
A custom image view that implements device motion scrolling

YXTMotionView A custom image view that implements device motion scrolling Installation CocoaPods Add the dependency to your Podfile: platform :ios pod

Hanton Yang 79 Dec 17, 2022
ImagePickerSheetController replicates the custom photo action sheet in iMessage.

ImagePickerSheetController About ImagePickerSheetController is a component that replicates the custom photo action sheet in iMessage. It's very simila

Laurin Brandner 1.5k Jan 3, 2023
Custom iOS camera and photo picker with editing capabilities

Overview Paparazzo is a component for picking and editing photos. Key Features ?? Taking photos using camera ?? Picking photos from user's photo libra

avito.tech 757 Jan 4, 2023
Set of Extensions and Custom control for standard types and classes 🎨

OpencvQueen Set of Extensions and Custom control for standard types and classes. Just like Doraemon’s pocket, has an endless variety of props for us t

Condy 17 Dec 23, 2022
A Swift/SwiftUI utility for caching and displaying images in SwiftUI Views

A Swift/SwiftUI utility for caching and displaying images asynchronously. Built with Swift 5.5 and works with async/await.

王雪铮 Xuezheng Wang 1 May 5, 2022
🚀SwiftUI Image downloader with performant LRU mem/disk cache.

Progressive concurrent image downloader for SwiftUI, with neat API and performant LRU mem/disk cache.

Cheng Zhang 42 Sep 24, 2022
Lazy image loading for SwiftUI

A missing piece in SwiftUI that provides lazy image loading.

Alexander Grebenyuk 9 Dec 26, 2022