The best way to record your AR experience!

Overview

SCNRecorder

GitHub license Platforms Swift Cocoapods compatible Carthage compatible Swift Package Manager

SCNRecorder allows you to record videos and to capture images from ARSCNView, SCNView and ARView (RealityKit) without sacrificing performance. It gives you an incredible opportunity to share the media content of your augmented reality app or SceneKit based game.

Starting version 2.2.0 SCNRecorder supports Metal only.

Sample

Requirements

  • iOS 12.0+
  • Xcode 12.0+
  • Swift 5.0+

Installation

CocoaPods

pod 'SCNRecorder', '~> 2.7'

Carthage

github "gorastudio/SCNRecorder"

Usage

Import the SCNRecorder module.

import SCNRecorder

Call sceneView.prepareForRecording() at viewDidLoad.

@IBOutlet var sceneView: SCNView!

override func viewDidLoad() {
  super.viewDidLoad()
  sceneView.prepareForRecording()
}

And now you can use new functions to capture videos.

try sceneView.startVideoRecording()
sceneView.finishVideoRecording { (videoRecording) in 
  /* Process the captured video. Main thread. */
  let controller = AVPlayerViewController()
  controller.player = AVPlayer(url: recording.url)
  self.navigationController?.pushViewController(controller, animated: true)
}

To capture an image it is enough to call:

sceneView.takePhoto { (photo: UIImage) in
  /* Your photo is now here. Main thread. */
}

or

sceneView.takePhotoResult { (result: Result<UIImage, Swift.Error>) in
  /* Result is here. Main thread. */
}

Look at the Example project for more details.

Audio capture

ARSCNView

To capture video with audio from ARSCNView enable audio in the ARConfiguration.

let configuration = ARWorldTrackingConfiguration()
configuration.providesAudioData = true
sceneView.session.run(configuration)

SCNView

To capture audio from SCNView you have to implement it by yourself.

var captureSession: AVCaptureSession?

override func viewDidLoad() {
  super.viewDidLoad()
  sceneView.prepareForRecording()
  
  guard let recorder = sceneView.recorder else { return }
  let captureSession = AVCaptureSession()
  
  guard let captureDevice = AVCaptureDevice.default(for: .audio) else { return }
  
  do {
    let captureInput = try AVCaptureDeviceInput(device: captureDevice)
    
    guard captureSession.canAddInput(captureInput) else { return }
    captureSession.addInput(captureInput)
  }
  catch { print("Can't create AVCaptureDeviceInput: \(error)")}
  
  guard captureSession.canAddRecorder(recorder) else { return }
  captureSession.addRecorder(recorder)
  
  captureSession.startRunning()
  self.captureSession = captureSession
}

or, simply

var captureSession: AVCaptureSession?

override func viewDidLoad() {
  super.viewDidLoad()
  sceneView.prepareForRecording()
  
  captureSession = try? .makeAudioForRecorder(sceneView.recorder!)
}

Music Overlay

Instead of capturing audio using microphone you can play music and add it to video at the same time.

let auidoEngine = AudioEngine()

override func viewDidLoad() {
  super.viewDidLoad()
  
  sceneView.prepareForRecording()
  do {
    audioEngine.recorder = sceneView.recorder
    
    // If true, use sound data from audioEngine if any
    // If false, use sound data ARSession/AVCaptureSession if any
    sceneView.recorder?.useAudioEngine = true
    
    let player = try AudioEngine.Player(url: url)
    audioEngine.player = player
    
    player.play()
  }
  catch { 
    print(\(error))
  }
}

RealityKit

To support recording RealityKit, copy ARView+MetalRecordable.swift and ARView+SelfSceneRecordable.swift files to your project. Then look at RealityKitViewController.swift for usage.

That's it!

Look at the Example project for more details.

Author

Thanks to Fedor Prokhorov and Dmitry Yurlov for testing, reviewing and inspiration.

License

This project is licensed under the MIT License - see the LICENSE file for details

Comments
  • Crashing on iPad Pro 2020 and iOS 14+

    Crashing on iPad Pro 2020 and iOS 14+

    Hello, when I try to take a photo or a video using your library on iPad Pro 2020 and iOS 14, the whole app freezes. I am not able to determine what's causing it because the debugger doesn't tell you anything; simply the app completely freezes. This wasn't the case on iOS 13+, exactly the same code.

    What do you think it is? Would you mind looking into it? Many thanks.

    opened by zeman-88 14
  • ARView+SelfSceneRecordable Crash

    ARView+SelfSceneRecordable Crash

    When I set the minimum deployments 'iOS 10.0' and run the example ,

    _cancelable = scene.subscribe( to: SceneEvents.Update.self ) { [weak sceneRecorder] (_) in sceneRecorder?.render() } SceneEvents.Update will crash, can you help to me?

    opened by frank-1992 12
  • Take photo crash and record issue

    Take photo crash and record issue

    Hello, Thank you for wonderful project.

    I bump up the library version to 2.0.0 to 2.2.0. After I update version, I can't take photo with crash log. And no video after video recording.

    I have some devices and I test "SCNRecorder".

    • iPad mini 5(iOS 13.6)
    • iPhone 11 Pro Max(iOS 13.5)
    • iPhone XS(iOS 14.1)
    • iPad Pro 12.1 2020(iPadOS 14.1)

    I test by checkout following project tag 2.0.0, 2.1.3 and 2.2.0. And run Example Project. In ARSCNView Example

    On version 2.0.0

    Nice working

    • iPad mini 5(iOS 13.6)
    • iPhone 11 Pro Max(iOS 13.5)
    • iPhone XS(iOS 14.1) (NavigationItem bug)

    Not working

    • iPad Pro 2020(iPadOS 14.1) (NavigationItem bug)

    StartVideoRecording Not working

    On version 2.1.3

    Nice working

    • iPad Pro 12.1 2020(iPadOS 14.1)
    • iPhone XS(iOS 14.1)

    Not working

    • iPad mini 5(iOS 13.6) &
    • iPhone 11 Pro Max(iOS 13.5)

    TakePhoto Not working with error.

    2020-11-16 12:18:16.303563+0900 Example[529:45566] Metal API Validation Enabled
    2020-11-16 12:18:18.087331+0900 Example[529:45847] [CAMetalLayerDrawable texture] should not be called after already presenting this drawable. Get a nextDrawable instead.
    Fatal error: Unexpectedly found nil while unwrapping an Optional value: file SCNRecorder/ImageOutput.swift, line 60
    2020-11-16 12:18:18.089814+0900 Example[529:45768] Fatal error: Unexpectedly found nil while unwrapping an Optional value: file SCNRecorder/ImageOutput.swift, line 60
    

    StartVideoRecording Not working with black video.

    On version 2.2.0

    Nice working

    • iPad Pro 12.1 2020(iPadOS 14.1)
    • iPhone XS(iOS 14.1)

    Not working

    • iPad mini 5(iOS 13.6) &
    • iPhone 11 Pro Max(iOS 13.5)

    TakePhoto Not working with error.

    2020-11-16 12:28:26.076353+0900 Example[555:49819] Metal API Validation Enabled
    Fatal error: Unexpectedly found nil while unwrapping an Optional value: file SCNRecorder/ImageOutput.swift, line 60
    2020-11-16 12:28:27.593305+0900 Example[555:50010] Fatal error: Unexpectedly found nil while unwrapping an Optional value: file SCNRecorder/ImageOutput.swift, line 60
    

    StartVideoRecording Not working with black video.

    Clue

    I try to find clue I find TakePhoto VTCreateCGImageFromCVPixelBuffer function is not working correctly iOS 13.

    I can't find any clue StartVideoRecording

    I think OS version dependent on this project. Please check.

    Hope you find reason soon!

    opened by 0k1019 8
  • Recorded audio not continuous

    Recorded audio not continuous

    Hi,

    Thanks for the good work. I am trying to record video and audio in ARKit. I setup the audio device like you mentioned. However the audio in the recorded video is not continuous and keeps interrupting. I appreciate any help.

    I am running on iOS 13.7

    opened by bluepixeltech 6
  • Build error in XCode 13

    Build error in XCode 13

    Hi, I tried to build my project with your library as pod and it shows error in file VideoSettings.Code. Error text is "Enum cases with associated values cannot be marked potentially unavailable with '@available'". @available(iOS 13.0, *) case hevcWithAlpha(_ compressionProperties: HEVCCompressionProperties)

    I can build project only after deleting "@available(iOS 13.0, *)" , but this is hack.

    Please can you fix this bug?

    opened by juraam 5
  • got hanging in video (while recording video with animated model)

    got hanging in video (while recording video with animated model)

    Hello this is nice framework but i found an issue in video , its getting hanging in video with animated model which is provided in attachment so please solve this issue in updated framework , it most helpful for all developers . Thanks .

    opened by ramakrishnaios 5
  • Xcode 14 Circular Reference when using SelfRecordable extension methods

    Xcode 14 Circular Reference when using SelfRecordable extension methods

    Possibly a resurgence of https://github.com/gorastudio-ceo/SCNRecorder/issues/51

    I'm using the latest 2.8.0 version, with clean Derived Data, and Podfile.lock

    The IDE is not pinpointing the precise location of the circular reference:

    Screen Shot 2022-10-01 at 1 10 53 AM

    Below is a shortened version of the class:

    import Foundation
    import SCNRecorder
    import AVKit
    import AVFoundation
    import AssetsLibrary
    
    @objc public class SurfaceRecorder: NSObject {
        private var audioRecorder: AVAudioRecorder? =  nil
        private var audioURL: URL? = nil
        private var methodBCaptureSession: AVCaptureSession? = nil
        private var connection : AVCaptureConnection?
        private var fileOutput: AVCaptureMovieFileOutput?
        private var fileOutputURL: URL? = nil
        
        private var IARView: IARSurfaceView? = nil
        
        private var view: SCNView {
            guard let view = IARView?.internalSceneView() else {
                fatalError("Recording methods should be called after SurfaceView.load() method")
            }
            return view
        }
    
        // MARK: - Private Methods
        
        (...)
        
        private func startVideoRecording() -> NSError? {
            FileLogger.shared.log(content: "Recording started...")
    
            view.prepareForRecording()
            
            do {
                try view.startVideoRecording()
            } catch {
                FileLogger.shared.log(content: "Failed to start recording: \(error.localizedDescription)")
                return NSError.error(from: error)
            }
            
            return nil
        }
        
        (...)
        
        @objc public func stopRecording(finalFileName: String?, completionHandler handler: @escaping (NSURL?, Error?) -> Void) {
            
            if let audioRecorder = audioRecorder {
                audioRecorder.stop()
            }
            
            guard let audioURL = audioURL else {
                handler(nil, nsError(withText: "Failed to obtain audio file URL", code: IARErrorCode.errorRecordVideo))
                return
            }
            
            view.finishVideoRecording { [weak self] (videoRecording) in
                (...)
            }
        }
    }
    

    Circular references are likely to be caused by these lines:

            view.prepareForRecording()
    
           // and
            
            do {
                try view.startVideoRecording()
    
           // and
    
            view.finishVideoRecording { [weak self] (videoRecording) in
    
    

    any advices?

    opened by rogerioth 4
  • RealityKit in Deployment target below iOS 13

    RealityKit in Deployment target below iOS 13

    when I try to use SCNRecorder, it crashed

    1. set the project deployment target below iOS 13
    2. import RealityKit and write add code " @available(iOS 13.0, *) " in UIViewController class
    3. init a Entity class or ModelEntity class ( let _ = Entity() )
    4. start to run the project and it will crash in iOS 13 and iOS 14 but iOS 15 is OK, very strange (you can run my demo) do you know how to solve this problem , I think it is apple's bug. how do you think , hope you can reply , thank you
    opened by frank-1992 4
  • Requires NSCameraUsageDescription always

    Requires NSCameraUsageDescription always

    When I submit to appstore connect I get an error saying that it is required to have NSCameraUsageDescription key, even though I am not using the camera because I am recording only a ScnView. Is there a way to disable the part relative to the camera API in the SDK so that Apple does not require NSCameraUsageDescription key?

    opened by giomurru 4
  • Capture Pixel Buffers

    Capture Pixel Buffers

    Hi, I really like this package, but I'm trying to do something with it for which I can't see an easy route.

    I want to catch every frame before it's added to the video, and send it elsewhere instead of save it to a video file. Is this possible?

    I tried using the capturePixelBuffers method, but the handler in there doesn't get called, not sure if it's supposed to work that way though.

    Any help would be greatly appreciated 🙏

    opened by maxxfrazer 4
  • Low fps during recording on iPhone 12 Pro

    Low fps during recording on iPhone 12 Pro

    Hello, I use your library for recording ARSCNView and it works great. I record on the iPhone 11 and iPhone SE and it has 60 fps during recording. But when I try to record in new iPhone 12 Pro, fps drops down to 27-35 FPS. Do you know about that problem ? How can I fix it ?

    opened by juraam 4
  • How can I get timestamp from SCNRecorder when recording?

    How can I get timestamp from SCNRecorder when recording?

    Hello

    I want to save some data from ARFrame and associate the data to the video frame captured from SCNRecorder

    How I can not read the time when recorder is recording

    func append(pixelBuffer: CVPixelBuffer, withPresentationTime time: CMTime) throws {
      guard pixelBufferAdaptor.assetWriterInput.isReadyForMoreMediaData else { return }
      guard pixelBufferAdaptor.append(pixelBuffer, withPresentationTime: time) else {
        if assetWriter.status == .failed { throw assetWriter.error ?? Error.unknown }
        return
    }
    
      let seconds = time.seconds
      duration += seconds - lastSeconds
      lastSeconds = seconds
    }
    

    Here I want to get the CMTime here to save with my ARFrame data.

    Thanks a lot

    opened by DikeyKing 1
  • Wrong color profile on iOS 12, iPhone 6 device when recording at high resolution.

    Wrong color profile on iOS 12, iPhone 6 device when recording at high resolution.

    Hi, I encountered some problems with the output colors while using SCNRecorder on iPhone 6 running iOS 12.5.5. iPhone X and iPad 9th gen on iOS 15.5 are fine though. The recording made with iOS 12.5.5 and iPhone 6 has dull colors that appears less saturated than normal. I found that the issue can be solved if we do not set a custom value for the AVVideoColorPropertiesKey and we leave this key to its default value. I created a pull-request which includes this fix.

    opened by giomurru 12
  • Iphone X landscape video recording crashes app.

    Iphone X landscape video recording crashes app.

    Everything works fine on my iPad but when my iPhoneX is in landscape mode it crashes when starting the video recording (portrait works fine).

    Let me know if you need any particular debug info. I'm on Ios 15.1

    let sourceTextureDescriptor = self.makeSourceTextureDescriptor(basedOn: lastTexture) guard let sourceTexture = self.device.makeTexture( descriptor: sourceTextureDescriptor, iosurface: surface, plane: 0 ) else { throw Error.noSourceTexture }

    com.apple.scenekit.scnview-renderer (15): signal SIGABRT

    Thank you.

    opened by hizzlehoff 0
  • Landscape Video Recording Problem on iPhone 8 Running iOS 14.2

    Landscape Video Recording Problem on iPhone 8 Running iOS 14.2

    Hello,

    I encountered an unusual issue while working with SCNRecorder.

    Videos recorded in landscape orientation on iPhone 8 specifically running iOS 14.2 are rotated sideways and almost look as if they were recorded in locked portrait mode. Videos recorded in portrait orientation work just fine.

    It is a bit difficult to properly describe this issue so please refer to the three screenshots below.

    Preview of Video Recorded in Portrait Mode

    portrait

    Preview of Video Recorded in Landscape Left Mode

    landscape-left

    Preview of Video Recorded in Landscape Right Mode

    landscape-right

    I am using version 2.3.0 of SCNRecorder.

    I have failed to replicate this issue on the following devices:

    • iPhone 7 running iOS 13.6.
    • iPhone 11 Pro running iOS 14.2
    • iPad Pro 12.9'' (2015) running iPadOS 14.2

    I have also attempted to play around with the video settings (specifically the size and the scaling mode) rather than using the default video settings but that didn't seem to help.

    Thank you in advance for looking into this.

    opened by pandraos 16
Releases(2.8.1)
  • 2.7.0(Oct 3, 2021)

  • 2.6.0(May 3, 2021)

  • 2.5.0(Apr 14, 2021)

  • 2.4.0(Dec 7, 2020)

    • Use Metal Performance Shaders to copy and covert bytes. A most performant way regardless of iOS version and device model.

    • Add integration tests. They cover the most important parts of SCNRecorder to guarantee reliability Screenshot 2020-12-07 at 2 51 43 PM

    • Update internal audio settings to support mp4, m4v, 3gp from any source.

    • Cleanup.

    Source code(tar.gz)
    Source code(zip)
  • 2.2.0(Nov 6, 2020)

    • Drop OpenGL support.
    • Fix first time recording frame drop.
    • Move AWAssetWriter's initialization to a separate queue.
    • Remove a bunch of any more not used files.
    • Drop Filters support - with current implementation they produce wrong results, they should be redone if needed.
    • Minor performance improvements.
    Source code(tar.gz)
    Source code(zip)
  • 2.1.1(Nov 2, 2020)

  • 2.1.0(Oct 30, 2020)

    • Remove the need for setting framebufferOnly to false.
    • Start using IOSurface to gather texture data.
    • Fix iPhone 12 frame drop.
    • Set minimum deployment target to 12.0 as recommended.
    • Silence EAGL deprecation warnings.
    • Add the ability to capture pixel buffer and remove the recorder from AVCaptureSession.
    • Simplify public interface (require and assert prepareForRecording() call)
    Source code(tar.gz)
    Source code(zip)
  • 2.0.0(Sep 16, 2020)

  • 1.1.0(Jan 3, 2020)

    SCNRecorder is becoming more simpler, faster and stable. Now, to start recording a video it is enough to import the library and call one function.

    Other changes:

    • Reduce the number of internal queues.
    • Fix OpenGL support.
    • Add orientation changes support. (But since AVAssetWriter doesn't support orientation changes, you still have to lock orientation during video recording. Otherwise, the video will look strangely.)
    • Address memory leak issues.
    Source code(tar.gz)
    Source code(zip)
Owner
GORA Studio
We create best Augment Reality mobile apps with best practice for iOS + Android. And full stack of web-services and mobile development.
GORA Studio
Cut and paste your surroundings using AR

An AR+ML prototype that allows cutting elements from your surroundings and pasting them in an image editing software.

Cyril Diagne 14.5k Dec 27, 2022
The best way to use the Zora API in your Swift projects.

ZoraKit The best way to use the Zora API in your Swift projects. Disclaimer This is still very much a work in progress, and is really a proof-of-conce

MBLA 6 Sep 23, 2022
Smooch is the best way to have personal, rich conversations with people on your website or customers on any device

Smooch is the best way to have personal, rich conversations with people on your website or customers on any device. Our features, integrations and developer-friendly APIs empower companies to connect with their customers in a whole new way.

Zendesk 121 Aug 1, 2022
Simple and integrated way to customize navigation bar experience on iOS app.

NavKit Simple and integrated way to customize navigation bar experience on iOS app. It should save our time that we usually use to make abstraction of

Wilbert Liu 37 Dec 7, 2022
A powerful, beautiful way to experience Formula1

F1 Pocket Companion A powerful, beautiful way to experience Formula1, right on your iPhone Note This project will probably change it's name. I'm curre

Liam Doyle 3 Nov 5, 2022
Record your position and export your trip in GPX with GPS Stone on iOS.

GPS Stone Register your trips and export them as GPX files. Notes We currently have included a UIRequiredDeviceCapabilities with a location-services v

Frost Land 11 Sep 24, 2022
Best way to use UIGesture

NiceGesture The best way to use UIGesture #Usage ###TapGesture: because tapGesture is only have one state,so it's only have method whenTapped view.n

Zhuo 104 Sep 9, 2022
The best way to watch Put.io on iPhone and iPad

The best way to watch Put.io on iPhone and iPad

Fetch 89 Oct 27, 2022
Best way to dismiss Keyboard in a View Controller iOS (Swift)

Best way to dismiss Keyboard in a View Controller iOS (Swift) First way: Implement UITextFieldDelegate’s textFieldShouldReturn method and dismiss curr

null 0 Dec 18, 2021
Pager is the simplest and best way to implement sliding view controllers in Swift

Pager is the simplest and best way to implement sliding view controllers. Installation Drop in the Spring folder to your Xcode project. Or via CocoaPo

Lucas Oceano 234 Aug 5, 2022
CodeEdit App for macOS – Elevate your code editing experience. Open source, free forever.

CodeEdit for macOS CodeEdit is a code editor built by the community, for the community, written entirely and unapologetically for macOS. Features incl

CodeEdit 15.8k Dec 31, 2022
A guide on setting up Xcode with all the essential Applications, Tools, and Frameworks to make your development experience with Xcode great!

A guide on setting up Xcode with all the essential Applications, Tools, and Frameworks to make your development experience with Xcode great!

Michael Royal 24 Jan 4, 2023
Record your beats easily with DongZ DaZ

DongZ DaZ Record your beats easily with DongZ DaZ. Table of Contents Introduction Technique Features Feedback Introduction A new way to record and sha

Ivan 0 Dec 7, 2021
A collaborative Augmented Reality iOS experience using Real-time Messaging

Collaborative AR Experience This project creates an augmented reality experience where you can build a scene together with anyone around the world, se

Agora.io Community 9 Dec 1, 2022
Turning on a VPN is always a painful experience on an iOS device due to the deep nested menus.

VPN On Turning on a VPN is always a painful experience on an iOS device due to the deep nested menus. This App installs a Today Widget into Notificati

Lex Tang 4.4k Dec 26, 2022
BarcodeScanner is a simple and beautiful wrapper around the camera with barcode capturing functionality and a great user experience.

Description BarcodeScanner is a simple and beautiful wrapper around the camera with barcode capturing functionality and a great user experience. Barco

HyperRedink 1.6k Jan 7, 2023
A modern photo viewing experience for iOS.

NYTPhotoViewer NYTPhotoViewer is a slideshow and image viewer that includes double-tap to zoom, captions, support for multiple images, interactive fli

The New York Times 2.8k Jan 5, 2023
The first affordable AR headset experience

AR MultiPendulum: AR Headset Experience AR MultiPendulum allows users to interact with virtual objects directly with their hand instead of tapping an

Philip Turner 28 Nov 18, 2022
Better payment user experience library with cool animation in Swift

?? Preview ?? Features Easily usable Simple Swift syntax Cool flip animation Compatible with Carthage Compatible with CocoaPods Customizable Universal

yassir RAMDANI 177 Nov 20, 2022
Album CovAR: an interactive AR experience for viewing album covers

Album CovAR - iOS Album CovAR is an interactive AR experience for viewing album covers. The app displays additional information about an album includi

null 0 Dec 14, 2021