Reconstruct a 3D face with RGB-D images captured by iPhone X's TrueDepth camera.

Overview

!TrueDepth camera can only be accessed on iPhone X or later.

TODO

  • Get the RGB-D files (This repository)

  • Generate 3D face mesh with landmark detector and 3DMM

4859

  • Refine the mesh with depth jsons
  • Deploy AI model to iPhone (CoreML/ARKit/ARCore/MediaPipe)

Reference

This repository is modified on the sample code offered by Apple.

Below is the original README.md content:

Streaming Depth Data from the TrueDepth Camera

Visualize depth data in 2D and 3D from the TrueDepth camera.

Overview

The TrueDepth camera provides depth data in real time that allows you to determine the distance of a pixel from the front-facing camera. This sample demonstrates how to use the AVFoundation framework’s capture API to read data from the TrueDepth camera, and how to display it in an intuitive fashion onscreen.

The sample shows two different views: a 2D view that distinguishes depth values by mapping depth to color, and a 3D view that renders data as a point cloud.

To see this sample app in action, build and run the project in Xcode on an iOS device running iOS 11 or later. Because Xcode doesn’t have access to the TrueDepth camera, this sample will not build or run in the Xcode simulator.

Set Up a Capture Session

Set up an AVCaptureSession on a separate thread via the session queue. Initialize this session queue before configuring the camera for capture, like so:

private let sessionQueue = DispatchQueue(label: "session queue", attributes: [], autoreleaseFrequency: .workItem)

The startRunning method is a blocking call that may take time to execute. Dispatch session setup to the session queue so the main queue isn’t blocked, allowing the app’s UI to stay responsive:

sessionQueue.async {
 self.configureSession()
}

Setting up the camera for video capture follows many of the same steps as normal video capture. See Setting Up a Capture Session for details on configuring streaming setup.

On top of normal setup, request depth data by declaring a separate output:

private let depthDataOutput = AVCaptureDepthDataOutput()

Explicitly add this output type to your capture session:

session.addOutput(depthDataOutput)
depthDataOutput.isFilteringEnabled = false
if let connection = depthDataOutput.connection(with: .depthData) {
 connection.isEnabled = true
} else {
 print("No AVCaptureConnection")
}

Search for the highest resolution available with floating-point depth values, and lock the configuration to the format.

let depthFormats = videoDevice.activeFormat.supportedDepthDataFormats
let filtered = depthFormats.filter({
 CMFormatDescriptionGetMediaSubType($0.formatDescription) == kCVPixelFormatType_DepthFloat16
})
let selectedFormat = filtered.max(by: {
 first, second in CMVideoFormatDescriptionGetDimensions(first.formatDescription).width < CMVideoFormatDescriptionGetDimensions(second.formatDescription).width
})

do {
 try videoDevice.lockForConfiguration()
 videoDevice.activeDepthDataFormat = selectedFormat
 videoDevice.unlockForConfiguration()
} catch {
 print("Could not lock device for configuration: \(error)")
 setupResult = .configurationFailed
 session.commitConfiguration()
 return
}

Synchronize the normal RGB video data with depth data output. The first output in the dataOutputs array is the master output.

outputSynchronizer = AVCaptureDataOutputSynchronizer(dataOutputs: [videoDataOutput, depthDataOutput])
outputSynchronizer!.setDelegate(self, queue: dataOutputQueue)

The CameraViewController implementation creates and manages this session to interface with the camera. It also contains UI to toggle between the two viewing modes, 2D and 3D.

Visualize Depth Data in 2D

The sample uses JET color coding to distinguish depth values, ranging from red (close) to blue (far). A slider controls the blending of the color code and the actual color values. Touching a pixel displays its depth value.

DepthToJETConverter performs the conversion. It separates the color spectrum into histogram bins, colors a Metal texture from depth values obtained in the image buffer, and renders that texture into the preview.

var cvTextureOut: CVMetalTexture?
CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault, textureCache, pixelBuffer, nil, textureFormat, width, height, 0, &cvTextureOut)
guard let cvTexture = cvTextureOut, let texture = CVMetalTextureGetTexture(cvTexture) else {
 print("Depth converter failed to create preview texture")
 CVMetalTextureCacheFlush(textureCache, 0)
 return nil
}

Visualize Depth Data in 3D

The sample’s 3D viewer renders data as a point cloud. Control the camera with the following gestures:

  • Pinch to zoom.
  • Pan to move the camera around the center.
  • Rotate with two fingers to turn the camera angle.
  • Double-tap the screen to reset the initial position.

The sample implements a 3D point cloud as a PointCloudMetalView. It uses a Metal vertex shader to control geometry and a Metal fragment shader to color individual vertices, keeping the depth texture and color texture separate:

CVMetalTextureCacheRef _depthTextureCache;
CVMetalTextureCacheRef _colorTextureCache;

The depth frame’s depth map provides the basis for the Metal view’s depth texture:

depthTexture = CVMetalTextureGetTexture(cvDepthTexture);">
CVPixelBufferRef depthFrame = depthData.depthDataMap;
CVMetalTextureRef cvDepthTexture = nullptr;
if (kCVReturnSuccess != CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
                        _depthTextureCache,
                        depthFrame,
                        nil,
                        MTLPixelFormatR16Float,
                        CVPixelBufferGetWidth(depthFrame),
                        CVPixelBufferGetHeight(depthFrame),
                        0,
                        &cvDepthTexture)) {
    NSLog(@"Failed to create depth texture");
    CVPixelBufferRelease(colorFrame);
    return;
}

id<MTLTexture> depthTexture = CVMetalTextureGetTexture(cvDepthTexture);

The RGB image provides the basis for the Metal view’s color texture:

colorTexture = CVMetalTextureGetTexture(cvColorTexture);">
CVMetalTextureRef cvColorTexture = nullptr;
if (kCVReturnSuccess != CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
                        _colorTextureCache,
                        colorFrame,
                        nil,
                        MTLPixelFormatBGRA8Unorm,
                        CVPixelBufferGetWidth(colorFrame),
                        CVPixelBufferGetHeight(colorFrame),
                        0,
                        &cvColorTexture)) {
    NSLog(@"Failed to create color texture");
    CVPixelBufferRelease(colorFrame);
    return;
}

id<MTLTexture> colorTexture = CVMetalTextureGetTexture(cvColorTexture);

Track Thermal State

Processing depth data from a live stream may cause the device to heat up. Keep tabs on the thermal state so you can alert the user if it exceeds a dangerous threshold.

@objc
func thermalStateChanged(notification: NSNotification) {
    if let processInfo = notification.object as? ProcessInfo {
        showThermalState(state: processInfo.thermalState)
    }
}

func showThermalState(state: ProcessInfo.ThermalState) {
    DispatchQueue.main.async {
        var thermalStateString = "UNKNOWN"
        if state == .nominal {
            thermalStateString = "NOMINAL"
        } else if state == .fair {
            thermalStateString = "FAIR"
        } else if state == .serious {
            thermalStateString = "SERIOUS"
        } else if state == .critical {
            thermalStateString = "CRITICAL"
        }

        let message = NSLocalizedString("Thermal state: \(thermalStateString)", comment: "Alert message when thermal state has changed")
        let alertController = UIAlertController(title: "TrueDepthStreamer", message: message, preferredStyle: .alert)
        alertController.addAction(UIAlertAction(title: NSLocalizedString("OK", comment: "Alert OK button"), style: .cancel, handler: nil))
        self.present(alertController, animated: true, completion: nil)
    }
}
You might also like...
Kanvas is an open-source iOS library for adding effects, drawings, text, stickers, and making GIFs from existing media or the camera.
Kanvas is an open-source iOS library for adding effects, drawings, text, stickers, and making GIFs from existing media or the camera.

Kanvas Kanvas is an open-source iOS library for adding effects, drawings, text, stickers, and making GIFs from existing media or the camera.

Extract opening hours tags from a camera image

OpeningHoursPhoto Extract opening hours tags from a camera image This project is no longer active. The latest development is taking part as part of Go

Swifttty and easy camera framework for iOS.
Swifttty and easy camera framework for iOS.

SwiftttCamera Swifttty and easy camera framework for iOS. View Demo · Report Bug · Request Feature SwiftttCamera is a wrapper around AVFoundation that

📸 A wrapper for AVCaptureSession - The way easier to use the Camera.

Capturer A wrapper for AVCaptureSession - The way easier to use the Camera. Setting up let captureBody = CaptureBody( configuration: .init { $0.

Instant camera hybrid with multiple effects and filters written in Swift.
Instant camera hybrid with multiple effects and filters written in Swift.

Kontax Cam Download on the app store! No longer on the app store Kontax Cam is an instant camera built 100% using Swift for iOS. You can take your pho

Simple camera application for iOS that uploads pictures to WebDAV server or Dropbox quickly. Available on the AppStore.
Simple camera application for iOS that uploads pictures to WebDAV server or Dropbox quickly. Available on the AppStore.

Upupu Simple camera application for iOS that uploads pictures to WebDAV server or Dropbox quickly. Also available on the AppStore. Features Easy and f

Custom iOS camera and photo picker with editing capabilities
Custom iOS camera and photo picker with editing capabilities

Overview Paparazzo is a component for picking and editing photos. Key Features 📷 Taking photos using camera 📱 Picking photos from user's photo libra

ImagePicker : an all-in-one camera solution for your iOS app
ImagePicker : an all-in-one camera solution for your iOS app

Description ImagePicker is an all-in-one camera solution for your iOS app. It lets your users select images from the library and take pictures at the

The camera shoot a photo when animals apear in the frame
The camera shoot a photo when animals apear in the frame

AutoCatDogCam The camera shoot a photo when animals apear in the frame. How to u

Owner
null
Vision Camera 📸 The Camera library that sees the vision.

Vision Camera ?? The Camera library that sees the vision. npm i react-native-vision-camera npx pod-install Documentation Guides API Ex

Marc Rousavy 3.5k Jan 5, 2023
AnylineFaceAuthentication pairs identity document scanning with a real-time liveness check utilizing the iPhone's camera, best suited for authenticating users over the internet.

AnylineFaceAuthentication AnylineFaceAuthentication pairs identity document scanning with a real-time liveness check utilizing the iPhone's camera, be

null 0 Mar 7, 2022
ImagePicker - selecting images from the photo albums, with allowed permissions /on real device accesing the camera

ImagePicker - selecting images from the photo albums, with allowed permissions /on real device accesing the camera, permission also needed/after picking an image it has the possibility to rename it

Pavel Surový 0 Jan 26, 2022
FlaneurImagePicker is an iOS image picker that allows users to pick images from different sources (ex: user's library, user's camera, Instagram...). It's highly customizable.

FlaneurImagePicker is a highly customizable iOS image picker that allows users to pick images from different sources (ex: device's library, device's c

FlaneurApp 17 Feb 2, 2020
Turn a photo of your food into a face

Megabite Megabite is a mobile app that automatically turns a photo of your food into a face. Read more about it here. Installation This project uses C

Aaron Randall 361 Oct 26, 2022
Ken Burns effect with face recognition!

APKenBurnsView Ken Burns effect with face recognition! APKenBurnsView is UIView subclass which supports face recognition to beautifully animate people

Alty 96 Sep 3, 2022
Picture anonymiser using Vision face recognition

?? Anonymojizer [WIP] Anonymize people in photos by replacing their faces by emojis. How to use it ? Pick a photo from the gallery Choose an emoji The

Kaww 1 Dec 20, 2021
A demo of face recognition SwiftUI app on iOS. Based on Vision, OpenCV, Dlib and ResNet.

iOS-FaceRecognizer A demo of face recognition SwiftUI app on iOS, build for iPad. Based on Vision, OpenCV, Dlib and ResNet. Features Add face image an

js_john 11 Aug 20, 2022
Autocrop - A face-aware crop utility using OSX's Vision framework

autocrop A high-performance face-aware crop utility using OSX's Vision framework

Alex Dong 0 Jan 19, 2022
FlexibleImage is implemented with the hope that anyone could easily develop an app that provides features such as Camera Filter and Theme.

FlexibleImage is implemented with the hope that anyone could easily develop an app that provides features such as Camera Filter and Theme. When you wr

Jungwon An 815 Dec 30, 2022