⬆️ Rad Media Capture in Swift

I've seen this in our implementation and in the sample app. When we record a single video clip, it seems like the last fraction of a second of the clip gets cut off. You can see this in the project's demo app as well.

To repro in the demo project, hold the capture button to record a clip and just before letting go of the button, mark the clip with an audio cue. You'll notice during playback, the audio cue is not present.

@piemonte Hey the previous issue #34 was closed so I thought it necessary to bring this up again since i failed to fix it.

For whatever reason despite setting videoConfigured to high quality, I still get lowQuality output video when I play back using your other library Player.

I have these settings in my viewDidLoad so would appreciate further hints as to what could possible cause the bad video. Im using the latest version of this library.

self.nextLevel.delegate = self
        self.nextLevel.videoDelegate = self
        self.nextLevel.photoDelegate = self
        self.nextLevel.deviceDelegate = self
        self.nextLevel.captureMode = .video
        self.nextLevel.focusMode = .autoFocus
        self.nextLevel.devicePosition = .front
        self.nextLevel.deviceOrientation = .portrait
        self.nextLevel.videoStabilizationMode = .auto
        self.nextLevel.videoConfiguration.bitRate = 2000000
        self.nextLevel.videoConfiguration.preset = AVCaptureSessionPresetHigh
        self.nextLevel.videoConfiguration.transform = CGAffineTransform(scaleX: -1, y: 1)```

Hi Piemonte,

We have below configuration for video recording:

1. Bit rate : 2000000
2. AVAssetExportPresetHighestQuality

but after video recording we are getting pixelated video file. Can you please help.

I'm having issues with the rotation of the preview layer, it's happening super slowly and with a lot of stuttering.

My set up looks something like this:

self.nextLevel.delegate = self
self.nextLevel.videoDelegate = self
self.nextLevel.captureMode = .videoWithoutAudio
self.nextLevel.automaticallyUpdatesDeviceOrientation = true
self.nextLevel.videoConfiguration.preset = .hd1920x1080

I have the preview set up like this:

/// The recording preview which contains the next level recorder
private lazy var previewView: UIView = {
    let view = UIView(frame: UIScreen.main.bounds)
    view.backgroundColor = .black
    self.nextLevel.previewLayer.frame = view.bounds
    view.layer.addSublayer(self.nextLevel.previewLayer)
    return view
}()

// In view did load
self.previewView.frame = self.view.bounds
self.previewView.autoresizingMask = [.flexibleWidth, .flexibleHeight]
self.view.insertSubview(self.previewView, at: 0)

Here is a GIF of the slow rotation:

Any ideas what could be going on?

Xcode 9.0

I encounter type issues (about 75 of them):

AVMediaType' (aka 'NSString') is not implicitly convertible to 'String!'; did you mean to use 'as' to explicitly convert?

In :

extension AVCaptureDeviceInput {
    
    /// Returns the capture device input for the desired media type and capture session, otherwise nil.
    ///
    /// - Parameters:
    ///   - mediaType: Specified media type. (i.e. AVMediaTypeVideo, AVMediaTypeAudio, etc.)
    ///   - captureSession: Capture session for which to query
    /// - Returns: Desired capture device input for the associated media type, otherwise nil
    public class func deviceInput(withMediaType mediaType: AVMediaType, captureSession: AVCaptureSession) -> AVCaptureDeviceInput? {
        if let inputs = captureSession.inputs as? [AVCaptureDeviceInput] {
            for deviceInput in inputs {
                if deviceInput.device.hasMediaType(mediaType) {
                    return deviceInput
                }
            }
        }
        return nil
    }
    
}

I've looked into it, but everything I find suggests taking the sample buffer and converting to a UIImage, which doesn't seem right, particularly when it would require updating the AVPreview layer each frame.

I'm currently looking at the VideoDelegate methods:

public func nextLevel(_ nextLevel: NextLevel, willProcessRawVideoSampleBuffer sampleBuffer: CMSampleBuffer) {}

public func nextLevel(_ nextLevel: NextLevel, renderToCustomContextWithImageBuffer imageBuffer: CVPixelBuffer, onQueue queue: DispatchQueue) {}

Any tips would be appreciated.

Motivation

Early this year Apple announced a lot of features related to the AVFoundation and one of them is the AVPortraitEffectsMatte

This new class is an auxiliary image used to separate foreground from background with high resolution for depth enabled photos.

I think this would be a great addition to the NextLevel, so I put some time and effort into this and came up with this pull request.

Changes

New flag variable on NextLevelPhotoConfiguration

/// Enables portrait effects matte output for the photo
public var isPortraitEffectsMatteEnabled: Bool = false

Enabling the AVPortraitEffectsMatte if available and desired on capturePhoto()

...
    if #available(iOS 12.0, *) {
        if photoOutput.isPortraitEffectsMatteDeliverySupported {
            photoOutput.isPortraitEffectsMatteDeliveryEnabled = 
            self.photoConfiguration.isPortraitEffectsMatteEnabled
        }
    } 
...

New Protocol method on NextLevelPhotoDelegate

The nextLevelDidFinishProcessingPhoto should be used to get an AVCapturePhoto that can have metadata for AVDepthData or AVPortraitEffectsMatte. In the future could be more useful for new capabilities that Apple introduce related to AVFoundation.

@available(iOS 11.0, *)
func nextLevelDidFinishProcessingPhoto(_ nextLevel: NextLevel, photo: AVCapturePhoto)

New Protocol called NextLevelPortraitEffectsMatteDelegate

// MARK: - NextLevelPortraitEffectsMatteDelegate

/// Portrait Effects Matte delegate, provides portrait effects matte updates
public protocol NextLevelPortraitEffectsMatteDelegate: AnyObject {
    
    @available(iOS 12.0, *)
    func portraitEffectsMatteOutput(_ nextLevel: NextLevel, didOutput portraitEffectsMatte: AVPortraitEffectsMatte)
    
}

Using the newer method of AVCapturePhotoCaptureDelegate

    @available(iOS 11.0, *)
    public func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?) {
        DispatchQueue.main.async {
            self.photoDelegate?.nextLevelDidFinishProcessingPhoto(self, photo: photo)
        }
        
        // MARK: Portrait Effects Matte Output
        if #available(iOS 12.0, *){
            if let portraitEffectsMatte = photo.portraitEffectsMatte {
                DispatchQueue.main.async {
                    self.portraitEffectsMatteDelegate?.portraitEffectsMatteOutput(self, didOutput: portraitEffectsMatte)
                }
            }
        }
    }

Considerations

Is there any way that I can keep contributing for the NextLevel? I've loved to implement this feature in the library and I will be using NextLevel for my photo app @fade-it on next releases.

Thank You! Gus

Hey there, Awesome library, but can we make it even better ? I was wondering for few things that might or might not be problems:

• Does this device.isSubjectAreaChangeMonitoringEnabled = true have to change on each focusEnded and exposureEnded call? https://github.com/NextLevel/NextLevel/blob/9cad174578d1986d2cecc6c230de79666764bebe/Sources/NextLevel.swift#L1663-L1691

• Another thing I've found that I can't explain to myself is why the following: Why is device.isAdjustingFocus always false in here https://github.com/NextLevel/NextLevel/blob/9cad174578d1986d2cecc6c230de79666764bebe/Sources/NextLevel.swift#L1611-L1624

• And finally: Why is focusAtAdjustedPointOfInterest being called instead of focusExposeAndAdjustWhiteBalance in the function above?

Has anyone encounter this issue when running in Xcode 12?

Looks like an iOS 14 SDK issue, but not 100% sure.

I could find availableVideoCVPixelFormatTypes in Objc header file, but somehow it is not visible in Swift.

And here is Apple's documentation for availableVideoPixelFormatTypes which is nonobjc but still, cannot find it in Swift header

https://developer.apple.com/documentation/avfoundation/avcapturevideodataoutput/2926555-availablevideopixelformattypes

NextLevel version: 0.16.3

Updates Checked in XCode 11.7, here is the definition for that property

Hey there,

Is there any way microphone input can be disabled and then re-enabled using this framework? I'm trying to create haptic feedback when the user taps on a button on the screen when the user is not recording. As playing haptics is disabled when an AVCaptureSession is active with an audio input, I'm thinking of disabling audio input while the user is not recording, and then enabling it just before they start recording.

Thanks for the help!

-William

Hi guys, really awesome project! I want to achieve slow motion video recording in the app I am working on. I was able to "stretch out" the video with AVFoundation only, but the quality sucked because the frame rate was unchanged while timescale was decreased. I saw that NextLevel supports slow motion recording on supported device, so I tried to look into the documentations and only found the bitrate and frameRate property. How should I use them to achieve slow motion video recording? Thanks!

I am integrating NextLevel into my project and I am using ARKit. Taking photoFromVideo works however, when I try to record, the video sample buffer is not updated, and no clips actually record. The didStartClipInSession is called however, the clip is never recorded/completed. This is the same issue in the Example App

Hi,

Thanks for the great framework you made. I am trying to capture photos and I am using the following settings:

NextLevel.shared.delegate = self NextLevel.shared.deviceDelegate = self NextLevel.shared.videoDelegate = self NextLevel.shared.photoDelegate = self NextLevel.shared.devicePosition = .back NextLevel.shared.captureMode = .photo NextLevel.shared.videoConfiguration.preset = .high NextLevel.shared.automaticallyUpdatesDeviceOrientation = true

I successfully capture photo by calling NextLevel.shared.capturePhoto(), but in didFinishProcessingPhoto photo: AVCapturePhoto the photo.pixelBuffer is nil.

Can you please help me with this?

Is there any place to set shouldOptimizeForNetworkUse, this is set to true in the source code, which causes the writer.finishWriting to take a long time when I record a long video.

If video is in landscape and in between change mode to portrait the video is zoomed . Like its not smoooth

    NextLevel.shared.delegate = self
    NextLevel.shared.deviceDelegate = self
    NextLevel.shared.videoDelegate = self
    NextLevel.shared.captureMode = .video
    NextLevel.shared.deviceOrientation = .portrait
    NextLevel.shared.automaticallyUpdatesDeviceOrientation = true
    NextLevel.shared.flipCaptureDevicePosition()
    NextLevel.shared.audioConfiguration.bitRate = 44000
    NextLevel.shared.devicePosition = .back

iOS 14, ARKit latest version and using the "nextlevel" example app.

occlusion switched on with

arConfig.frameSemantics.insert(ARConfiguration.FrameSemantics.personSegmentationWithDepth)

occlusion works fine in the live ARview, but the recorded video it does not