Code examples for Depth APIs in iOS

Overview

iOS-Depth-Sampler

Platform Language License Twitter

Code examples of Depth APIs in iOS

Requirement

Use devices which has a dual camera (e.g. iPhone 8 Plus) or a TrueDepth camera (e.g. iPhone X)

How to build

Open ARKit-Sampler.xcworkspace with Xcode 10 and build it!

It can NOT run on Simulator. (Because it uses Metal.)

Contents

Real-time Depth

Depth visualization in real time using AV Foundation.

Real-time Depth Mask

Blending a background image with a mask created from depth.

Depth from Camera Roll

Depth visualization from pictures in the camera roll.

Plaease try this after taking a picture with the Camera app using the PORTRAIT mode.

Portrait Matte

Background removal demo using Portrait Effect Matte (or Portrait Effect Matte).

Plaease try this after taking a picture of a HUMAN with PORTRAIT mode.

Available in iOS 12 or later.

ARKit Depth

Depth visualization on ARKit. The depth on ARKit is available only when using ARFaceTrackingConfiguration.

2D image in 3D space

A demo to render a 2D image in 3D space.

AR occlusion

[WIP] An occlusion sample on ARKit using depth.

Author

Shuichi Tsutsumi

Freelance iOS programmer from Japan.

Support via PayPal
Comments
  • AVCaptureDevice+Extension selectDepthFormat parses empty availableFormats which results in fatalError

    AVCaptureDevice+Extension selectDepthFormat parses empty availableFormats which results in fatalError

    Hi,

    Thanks for this great repo. I've had a play around with the samples and it seems only the front camera is working properly. The device I am using to run the program is iPhone 11 and my XCode version is 12.2.

    When I try to run Real-time Depth and Real-time Depth Mask, I run into a fatal error in AVCaptureDevice+Extension line 39.

    I've tried to have a look at it myself but since I'm relatively new to Swift, I've been unable to resolve the problem myself. The availableFormats is always empty which results in the guard failing. I'm not too sure why since the format field is always populated.

    Am I doing something wrong?

    opened by E-Mac27 1
  • 4 compilation issues on Xcode 10.0, Mojave

    4 compilation issues on Xcode 10.0, Mojave

    I cloned master @106813e0860f381f1821baddf76b9cdeebfcc490, opened the .xcworkspace, and change the signing, then tried to compile, which resulting in four compilation errors:

    /Users/jon/git/iOS-Depth-Sampler/iOS-Depth-Sampler/Renderer/MetalRenderer.swift:71:39: 
    Value of type 'MTLDrawable' has no member 'texture'
    
    /Users/jon/git/iOS-Depth-Sampler/iOS-Depth-Sampler/Samples/Realtime-Depth/RealtimeDepthViewController.swift:38:54:
     Value of type 'MTLDrawable' has no member 'layer'
    
    /Users/jon/git/iOS-Depth-Sampler/iOS-Depth-Sampler/Samples/ARKit/ARKitDepthViewController.swift:34:28: 
    Value of type 'ARSCNView?' has no member 'device'
    
    /Users/jon/git/iOS-Depth-Sampler/iOS-Depth-Sampler/Samples/ARKit/ARKitDepthViewController.swift:39:54:
     Value of type 'MTLDrawable' has no member 'layer'
    
    opened by jondwillis 1
  • convert depth16 to depth32 before grayPixelData

    convert depth16 to depth32 before grayPixelData

    grayPixelDataメソッド内でindexを1/2していた箇所について、Depthの 16bitフォーマットを32bitでscanしていたためではないかと思いました。 converting(toDepthDataType: kCVPixelFormatType_DepthFloat32) メソッドを実行してz軸を逆方向にしてindexをそのまま使用できることを確認しましたので、pullrequestさせて頂きました。

    opened by tatsuya-ogawa 0
  • Can ARKit distinguish real world face from picture face?

    Can ARKit distinguish real world face from picture face?

    I am using the ARKit to detect face with depthData.

    However, I am not able to distinguish whether this face is a real world person or a picture placed in front of the device camera.

    opened by malhobayyeb 2
  • Adding example to rectify lens distortion in depth images

    Adding example to rectify lens distortion in depth images

    Hi there!

    Thank you very much for creating this open source repo, I plan to use it for future ios machine learning projects! In the WWDC 2017 Talk, Apple discusses that the depth output is geometrically distorted to align with images produced by the camera. They mention to get precise true depth measurements, you need to correct for lens distortion. the WWDC Talk says that a reference implementation to correct for lens distortion is commented in the AVCameraCalibrationData.h file.

    It would be great if you can add an example view controller that a user taps to see a recitfied depth image and enable developers to work with precise, true depth measurements. I attached the reference implementation from AVCameraCalibrationData.h for ease of reference and if anyone can add this it would be amazing!

    The following reference implementation illustrates how to use the lensDistortionLookupTable, 
        inverseLensDistortionLookupTable, and lensDistortionCenter properties to find points in the 
        lens-distorted or undistorted (rectilinear, corrected) space. If you have a distorted image (such as a photo taken by a camera) and want to find a particular point in a corresponding undistorted image, you would call the sample method below using the inverseLensDistortionLookupTable. If you have an undistorted (aka distortion-corrected) image and want to find a point in the distorted image's space, you would call the sample method below using the lensDistortionLookupTable.
     
        To apply distortion correction to an image, you'd begin with an empty destination buffer and iterate through it 
        row by row, calling the sample implementation below for each point in the output image, passing the 
        lensDistortionLookupTable to find the corresponding value in the distorted image, and write it to your 
        output buffer. Please note that the "point", "opticalCenter", and "imageSize" parameters below must be
        in the same coordinate system, i.e. both at full resolution, or both scaled to a different resolution but
        with the same aspect ratio.
     
        The reference function below returns floating-point x and y values. If you wish to match the results with 
        actual pixels in a bitmap, you should either round to the nearest integer value or interpolate from surrounding
        integer positions (i.e. bilinear interpolation from the 4 surrounding pixels).
     
    - (CGPoint)lensDistortionPointForPoint:(CGPoint)point
                               lookupTable:(NSData *)lookupTable
                   distortionOpticalCenter:(CGPoint)opticalCenter
                                 imageSize:(CGSize)imageSize
    {
        // The lookup table holds the relative radial magnification for n linearly spaced radii.
        // The first position corresponds to radius = 0
        // The last position corresponds to the largest radius found in the image.
     
        // Determine the maximum radius.
        float delta_ocx_max = MAX( opticalCenter.x, imageSize.width  - opticalCenter.x );
        float delta_ocy_max = MAX( opticalCenter.y, imageSize.height - opticalCenter.y );
        float r_max = sqrtf( delta_ocx_max * delta_ocx_max + delta_ocy_max * delta_ocy_max );
     
        // Determine the vector from the optical center to the given point.
        float v_point_x = point.x - opticalCenter.x;
        float v_point_y = point.y - opticalCenter.y;
     
        // Determine the radius of the given point.
        float r_point = sqrtf( v_point_x * v_point_x + v_point_y * v_point_y );
     
        // Look up the relative radial magnification to apply in the provided lookup table
        float magnification;
        const float *lookupTableValues = lookupTable.bytes;
        NSUInteger lookupTableCount = lookupTable.length / sizeof(float);
     
        if ( r_point < r_max ) {
            // Linear interpolation
            float val   = r_point * ( lookupTableCount - 1 ) / r_max;
            int   idx   = (int)val;
            float frac  = val - idx;
     
            float mag_1 = lookupTableValues[idx];
            float mag_2 = lookupTableValues[idx + 1];
     
            magnification = ( 1.0f - frac ) * mag_1 + frac * mag_2;
        }
        else {
            magnification = lookupTableValues[lookupTableCount - 1];
        }
     
        // Apply radial magnification
        float new_v_point_x = v_point_x + magnification * v_point_x;
        float new_v_point_y = v_point_y + magnification * v_point_y;
     
        // Construct output
        return CGPointMake( opticalCenter.x + new_v_point_x, opticalCenter.y + new_v_point_y );
    }
    
    opened by interactivetech 5
  • Sample of creating an custom depth image and adding it to an image

    Sample of creating an custom depth image and adding it to an image

    Hi I was wondering if you could add a sample that could load up a grayscale jpg image and add that as the depth map of another image that does not already have depth data?

    opened by dredenba 1
Owner
Shuichi Tsutsumi
Freelance iOS Programmer
Shuichi Tsutsumi
Source code for the iOS app Screenshotter, on the App Store

Screenshotter This is the source code for the iOS app Screenshotter, available on the App Store. General Project Info Screenshotter is an Objective C

Riz 94 Mar 25, 2022
Source code for iOS app "Photos for VK" — albums and photos manager for social network VKontakte

VK Photos (formally Photos for VK) VK Photos is an iOS app for manage albums and photos in social network VKontakte (vk.com) Screenshots Disclaimer ⚠️

Yury S. 29 Oct 8, 2022
A better way to operate QR Code in Swift, support iOS, macOS, watchOS and tvOS.

EFQRCode is a lightweight, pure-Swift library for generating stylized QRCode images with watermark or icon, and for recognizing QRCode from images, in

EFPrefix 4.3k Jan 2, 2023
A very useful and unique iOS library to open image picker in just few lines of code.

ImagePickerEasy A very simple solution to implement UIImagePickerController() in your application. Requirements Swift 4.2 and above Installation Image

wajeehulhassan 6 May 13, 2022
Scan the MRZ code of a passport and extract the firstname, lastname, passport number, nationality, date of birth, expiration date and personal numer.

PassportScanner Works with 2 and 3 line identity documents. What is this With PassportScanner you can use your camera to scan the MRZ code of a passpo

Edwin Vermeer 441 Dec 24, 2022
SwiftGen is a tool to automatically generate Swift code for resources of your projects (like images, localised strings, etc), to make them type-safe to use.

SwiftGen is a tool to automatically generate Swift code for resources of your projects (like images, localised strings, etc), to make them type-safe to use.

null 8.3k Jan 5, 2023
A simple macOS app to read code from images, written purely in Swift using Vision Framework.

CodeReader A simple macOS app to read code from images, written purely in Swift using Vision Framework. Usage Drag an image Click the convert button R

Md Ibrahim Hassan 44 Nov 20, 2022
The frontend (phone) code for the e-mission server

e-mission phone app This is the phone component of the e-mission system. ✨ This has now been upgraded to cordova [email protected] and [email protected] (details)

null 16 Nov 2, 2022
Swift ports of Apple's Objective-C / C++ sample code

MetalSampleCodeSwift Swift ports of Apple's Objective-C / C++ sample code Metal is a great API, but it can feel inaccessible for Swift developers due

Jonathan Hobson 4 Dec 15, 2022
Agrume - 🍋 An iOS image viewer written in Swift with support for multiple images.

Agrume An iOS image viewer written in Swift with support for multiple images. Requirements Swift 5.0 iOS 9.0+ Xcode 10.2+ Installation Use Swift Packa

Jan Gorman 601 Dec 26, 2022
APNGKit is a high performance framework for loading and displaying APNG images in iOS and macOS.

APNGKit is a high performance framework for loading and displaying APNG images in iOS and macOS. It's built on top of a modified version of libpng wit

Wei Wang 2.1k Dec 30, 2022
An iOS/tvOS photo gallery viewer, useful for viewing a large (or small!) number of photos.

This project is unmaintained. Alex passed away in an accident in late 2019. His love of iOS development will always be remembered. AXPhotoViewer AXPho

Alex Hill 596 Dec 30, 2022
A lightweight generic cache for iOS written in Swift with extra love for images.

Haneke is a lightweight generic cache for iOS and tvOS written in Swift 4. It's designed to be super-simple to use. Here's how you would initalize a J

Haneke 5.2k Dec 11, 2022
✂️ Detect and crop faces, barcodes and texts in image with iOS 11 Vision api.

ImageDetect ImageDetect is a library developed on Swift. With ImageDetect you can easily detect and crop faces, texts or barcodes in your image with i

Arthur Sahakyan 299 Dec 17, 2022
A lightweight and fast image loader for iOS written in Swift.

ImageLoader ImageLoader is an instrument for asynchronous image loading written in Swift. It is a lightweight and fast image loader for iOS. Features

Hirohisa Kawasaki 293 Nov 24, 2022
Kanvas is an open-source iOS library for adding effects, drawings, text, stickers, and making GIFs from existing media or the camera.

Kanvas Kanvas is an open-source iOS library for adding effects, drawings, text, stickers, and making GIFs from existing media or the camera.

Tumblr 267 Nov 24, 2022
Lightbox is a convenient and easy to use image viewer for your iOS app

Lightbox is a convenient and easy to use image viewer for your iOS app, packed with all the features you expect: Paginated image slideshow. V

HyperRedink 1.5k Dec 22, 2022
An image download extension of the image view written in Swift for iOS, tvOS and macOS.

Moa, an image downloader written in Swift for iOS, tvOS and macOS Moa is an image download library written in Swift. It allows to download and show an

Evgenii Neumerzhitckii 330 Sep 9, 2022