Using ARKit and LiDAR to save depth data and export point cloud, based on WWDC20-10611 sample code

Overview

Save iOS ARFrame and Point Cloud

This project improves the usability of the sample code from WWDC20 session 10611: Explore ARKit 4. Note that the sample code is also on the original branch and the original code from WWDC20 can be checked out at the first commit. The original project places points in the real-world using the scene's depth data to visualize the shape of the physical environment.

Usability functions

This project adds the following functions:

  • Add buttons to start/pause recordings.

  • Save ARFrame raw data asynchronously at each frame when recording. The selected data include

    // custom struct for pulling necessary data from arframes
    struct ARFrameDataPack {
        var timestamp: Double
        var cameraTransform: simd_float4x4
        var cameraEulerAngles: simd_float3
        var depthMap: CVPixelBuffer
        var smoothedDepthMap: CVPixelBuffer
        var confidenceMap: CVPixelBuffer
        var capturedImage: CVPixelBuffer
        var localToWorld: simd_float4x4
        var cameraIntrinsicsInversed: simd_float3x3
    }

    The captured images are stored in jpeg format and others are coded into json files of which the format is specified as below.

    struct DataPack: Codable {
        var timestamp: Double
        var cameraTransform: simd_float4x4 // The position and orientation of the camera in world coordinate space.
        var cameraEulerAngles: simd_float3 // The orientation of the camera, expressed as roll, pitch, and yaw values.
        var depthMap: [[Float32]]
        var smoothedDepthMap: [[Float32]]
        var confidenceMap: [[UInt8]]
        var localToWorld: simd_float4x4
        var cameraIntrinsicsInversed: simd_float3x3
    }

    They can be retrieved in Finder with USB connection. Those raw data make it possible to leverage photogrammetry techniques for various tasks.

  • Save the point cloud in PLY format when the recording is stopped.

  • Add low memory warning and file saving progress.

Discussions

  • This answer has been very helpful in exporting point cloud in PLY format.

  • According to this answer, releasing ARFrame memory pool holding is important. If ARFrame currentFrame is passed into time-consuming async tasks like converting data formats or saving files to disks, the memory pool used by ARFrame is retained and no more frames can be written to the pool. In this case, a warning will arise.

    ARSessionDelegate is retaining XX ARFrames.
    

    To solve this problem, copy over the selected data to custom struct and pass that to async tasks. To deep copy CVPixelBuffer, try this code.

  • To get warning on low memory, see this doc.

  • Similar projects

You might also like...
Augmented Reality image tracking with SwiftUI, RealityKit and ARKit 4.

ARImageTracking This is an Augmented Reality Xcode project that uses Apple's newest RealityKit framework and ARKit 4 features, to dynamically track a

ARKit Demo Application
ARKit Demo Application

ARKitNavigationDemo Work in progress. In Progress Region — For one, we could render far fewer nodes. In fact, it’s a bit distracting that the entire t

This library uses ARKit Face Tracking in order to catch user's smile.
This library uses ARKit Face Tracking in order to catch user's smile.

SmileToUnlock Make your users smile before opening the app :) Gif with the demonstration Installation Cocoapods The most preferable way to use this li

Power! Unlimited power for ARKit 2.0!
Power! Unlimited power for ARKit 2.0!

A long time ago in a galaxy, far, far away... It is a period when iPhone SE and iPhone X were destroyed from the apple store, the AR market was under

A simple application created for educational purposes for mastering ARKit
A simple application created for educational purposes for mastering ARKit

ARDrawing AR Drawing is a simple application created for educational purposes for mastering ARKit. The basis of the project is copied from the project

Trying TDD with ARKit
Trying TDD with ARKit

ARPlacer BDD Spec As a user I want to place a random object in real world. I also want to see the distance between AR object and my phone. Use Cases

Draw VR content over live camera feed with ARKit

funny-ar Exercise with ARKit: draw VR content over live camera feed: work is in

ARKit: Projecting 3D mesh to 2D coordinate
ARKit: Projecting 3D mesh to 2D coordinate

ARKitDemo ARKit: Projecting 3D mesh to 2D coordinate A simple utility to project 3D face mesh in 2D coordinate on device screen. Sources: https://deve

Furniture E-Commerce Augmented Reality(AR) app in iOS powered by ARKit
Furniture E-Commerce Augmented Reality(AR) app in iOS powered by ARKit

HomeMax-iOS Furniture E-Commerce Augmented Reality(AR) app in iOS powered by ARKit and SceneKit. Inspired by IKEA place app. Description Experience on

Owner
null
PlacenoteSDK Sample app in native iOS using ARKit, written primarily in Swift

Placenote SDK for iOS Placenote SDK lets you easily build cloud-based Augmented Reality (AR) apps that pin digital content to locations in the real wo

Placenote 93 Nov 15, 2022
Smart Online Shopping iOS App with Augmented Reality (AR) and simple Social Media features using SwiftUI and Google Firebase Cloud Services

Table of contents App Demo How to Run Context Content How it's written Inspiration App Demo AR.online.shopping.iOS.demo.mp4 How to Run First make sure

Ashkan Goharfar 13 Nov 1, 2022
ARKit + CoreLocation: Combines the high accuracy of AR with the scale of GPS data.

ARKit: Uses camera and motion data to map out the local world as you move around. CoreLocation: Uses wifi and GPS data to determine your global locati

Andrew Hart 5.3k Dec 27, 2022
ARKit Base Project. Place virtual objects based on WWDC example project

ARKit - Placing Virtual Objects in Augmented Reality Learn best practices for visual feedback, gesture interactions, and realistic rendering in AR exp

Ignacio Chiazzo Cardarello 338 Jan 5, 2023
A library that allows you to generate and update environment maps in real-time using the camera feed and ARKit's tracking capabilities.

ARKitEnvironmentMapper Example To run the example project, clone the repo, and run pod install from the Example directory first. Installation ARKitEnv

SV Hawks 91 Dec 4, 2022
A library that allows you to generate and update environment maps in real-time using the camera feed and ARKit's tracking capabilities.

ARKitEnvironmentMapper Example To run the example project, clone the repo, and run pod install from the Example directory first. Installation ARKitEnv

SV Hawks 91 Dec 4, 2022
ARDicee - Simple augmented reality app using SceneKit and ARKit

ARDicee Simple augmented reality app using SceneKit and ARKit Requirements Xcode

donggyu 3 Feb 4, 2022
AR Ruler - A simple iOS app made using ARKit and SceneKit

A simple iOS app made using ARKit and SceneKit.Which can try to simplify little things in your life such as measuring stuff.

Dishant Nagpal 5 Aug 31, 2022
ARID - Augmented Reality app using Apple’s ARKit framework which can recognise faces of famous scientists

ARID Augmented Reality app using Apple’s ARKit framework which can recognise fac

Hemanth 0 Jan 12, 2022
An iOS Framework Capture & record ARKit videos 📹, photos 🌄, Live Photos 🎇, and GIFs 🎆.

An iOS Framework that enables developers to capture videos ?? , photos ?? , Live Photos ?? , and GIFs ?? with ARKit content.

Ahmed Bekhit 1.5k Dec 24, 2022