Augmented Reality Movie Trailer made with ARKit and SceneKit.

Overview

ARTrailer

Identify the movie title from a poster, and use SceneKit to display the movie trailer in AR.

Overview

This app runs an ARKit world-tracking session with content displayed in a SceneKit view. It first uses the Vision framework to find regions of visible texts on camera images, and pass the detected regions to the Tesseract framework for OCR. After text recognition, it overlays a movie trailer in AR world space.

Demo

Getting Started

ARKit requires iOS 11.0 and a device with an A9 (or later) processor. ARKit is not available in iOS Simulator. Building the sample code requires Xcode 10.0 or later.

Installation

  1. Run pod install and open ARTrailer.xcworkspace.
  2. This app uses the YouTube Data API to search for movie trailers. Add the API key to Keys.plist.
<plist version="1.0">
<dict>
    <key>YouTubeDataAPI</key>
    <string></string>
</dict>
</plist>

Note: You need quota for the YouTube Data API to send a query request. To test video overlay without quota, you may hard code the videoId in the YouTube url.

Run the AR Session and Process Camera Images

The ViewController class manages the AR session and displays AR overlay content in a SceneKit view. ARKit captures video frames from the camera and provides them to the view controller in the session(_:didUpdate:) method, which then calls the processCurrentImage() method to perform text recognition.

func session(_ session: ARSession, didUpdate frame: ARFrame) {
    // Do not enqueue other buffers for processing while another Vision task is still running.
    // The camera stream has only a finite amount of buffers available; holding too many buffers for analysis would starve the camera.
    guard currentBuffer == nil, case .normal = frame.camera.trackingState else {
        return
    }

    // Retain the image buffer for Vision processing.
    self.currentBuffer = frame.capturedImage
    processCurrentImage()
}

Serialize Image Processing for Real-Time Performance

The processCurrentImage() method uses the view controlle's currentBuffer property to track whether Vision is currently processing an image before starting another Vision task.

// Most computer vision tasks are not rotation agnostic so it is important to pass in the orientation of the image with respect to device.
let orientation = CGImagePropertyOrientation(UIDevice.current.orientation)

let requestHandler = VNImageRequestHandler(cvPixelBuffer: currentBuffer!, orientation: orientation)

visionQueue.async {
    do {
    // Release the pixel buffer when done, allowing the next buffer to be processed.
        defer { self.currentBuffer = nil }
        try requestHandler.perform([self.rectangleRequest, self.textRequest])
        self.recognizeTexts(cvPixelBuffer: self.currentBuffer!)
    } catch {
        print("Error: Vision request failed with error \"\(error)\"")
    }
}

Implement the Text Detector

The code's textDetectionHandler() method and rectangleDetectionHandler() method detect regions of the movie poster and detect regions of visible texts on the poster.

func textDetectionHandler(request: VNRequest, error: Error?) {
    guard let observations = request.results else {print("no result"); return}

    let result = observations.map({$0 as? VNTextObservation})
    if result.isEmpty {
        return
    }

    textObservations = result as! [VNTextObservation]
}

Perform Text Recognition

The code's recognizeTexts() method performs text recognition on the detected text regions.

func recognizeTexts(cvPixelBuffer: CVPixelBuffer) {
    var ciImage = CIImage(cvPixelBuffer: self.currentBuffer!)
    let transform = ciImage.orientationTransform(for: CGImagePropertyOrientation(rawValue: 6)!)
    ciImage = ciImage.transformed(by: transform)
    let size = ciImage.extent.size
    
    var keywords = ""
    for textObservation in self.textObservations {
        guard let rects = textObservation.characterBoxes else {
        continue
        }
        let (imageRect, xMin, xMax, yMin, yMax) = createImageRect(rects: rects, size: size)
        
        var text = runOCRonImage(imageRect: imageRect, ciImage: ciImage, tesseract: tesseract)
        keywords += " \(text)"
    }
    
    createVideoAnchor()
    textObservations.removeAll()
}

The runOCROnImage() method uses the Tesseract framework to perform OCR on the preprocessed image.

func runOCRonImage(imageRect: CGRect, ciImage: CIImage, tesseract: G8Tesseract) -> String {
    let context = CIContext(options: nil)
    guard let cgImage = context.createCGImage(ciImage, from: imageRect) else {
        return ""
    }
    let uiImage = preprocessImage(image: UIImage(cgImage: cgImage))
    tesseract.image = uiImage
    tesseract.recognize()
    guard let text = tesseract.recognizedText else {
        return ""
    }
    return text.trimmingCharacters(in: CharacterSet.newlines)
}

The code's preprocessImage() method applis image processing to optimize the camera images of the text regions for OCR.

func preprocessImage(image: UIImage) -> UIImage {
    var resultImage = image.fixOrientation().g8_grayScale()?.g8_blackAndWhite()
    resultImage = resultImage?.resizeVI(size: CGSize(width: image.size.width * 3, height: image.size.height * 3))!
    return resultImage!
}
  • Note: The accuracy of text recognition depends on the input image. Refer to the Tesseract documentation for different techniques in preprocessing images. To implement other image processing methods, add them to the UIImage extension.

Add a Video in AR

The createVideoAnchor() methods adds an anchor to the AR session.

// Create anchor using the camera's current position
if let currentFrame = sceneView.session.currentFrame {
    self.cinemaFrame = currentFrame
    
    var translation = matrix_identity_float4x4
    translation.columns.3.z = -1.5
    let transform = matrix_multiply((cinemaFrame?.camera.transform)!, translation)

    let anchor = ARAnchor(transform: transform)
    sceneView.session.add(anchor: anchor)
}

Next, after ARKit automatically creates a SceneKit node for the newly added anchor, the renderer(_:didAdd:for:) delegate method provides content for that node. In this case, the addVideoToSCNNode() method creates a SpriteKit node for the newly added anchor and plays a video at the anchor.

func addVideoToSCNNode(url: String, node: SCNNode) {
    let videoNode = SKVideoNode(url: URL(string: url)!)

    let skScene = SKScene(size: CGSize(width: 1280, height: 720))
    skScene.addChild(videoNode)

    videoNode.position = CGPoint(x: skScene.size.width/2, y: skScene.size.height/2)
    videoNode.size = skScene.size

    let tvPlane = SCNPlane(width: 1.0, height: 0.5625)
    tvPlane.firstMaterial?.diffuse.contents = skScene
    tvPlane.firstMaterial?.isDoubleSided = true

    let tvPlaneNode = SCNNode(geometry: tvPlane)
    tvPlaneNode.eulerAngles = SCNVector3(0,GLKMathDegreesToRadians(180),GLKMathDegreesToRadians(-90))
    tvPlaneNode.opacity = 0.9

    videoNode.play()

    node.addChildNode(tvPlaneNode)
}

References

You might also like...
SceneKit-MetalShader-SCNProgram - SceneKit MetalShader SCNProgram
SceneKit-MetalShader-SCNProgram - SceneKit MetalShader SCNProgram

SceneKit-MetalShader-SCNProgram Tech SceneKit (SCNProgram - SCNGeometry or SCNM

 ARVoxelKit - Lightweight Framework for Voxel graphic using AR + SceneKit
ARVoxelKit - Lightweight Framework for Voxel graphic using AR + SceneKit

ARVoxelKit Lightweight Framework for Voxel graphic using AR + SceneKit Requirements ARVoxelKit requires iOS 11 and devices, which support ARKit Usage

Swift framework for loading various 3d models in SceneKit

AssetImportKit AssetImportKit is a cross platform library (macOS, iOS) that coverts the files supported by Assimp to SceneKit scenes. Features AssetIm

 A library that allows you to generate and update environment maps in real-time using the camera feed and ARKit's tracking capabilities.
A library that allows you to generate and update environment maps in real-time using the camera feed and ARKit's tracking capabilities.

ARKitEnvironmentMapper Example To run the example project, clone the repo, and run pod install from the Example directory first. Installation ARKitEnv

 A library that allows you to generate and update environment maps in real-time using the camera feed and ARKit's tracking capabilities.
A library that allows you to generate and update environment maps in real-time using the camera feed and ARKit's tracking capabilities.

ARKitEnvironmentMapper Example To run the example project, clone the repo, and run pod install from the Example directory first. Installation ARKitEnv

Using ARKit and LiDAR to save depth data and export point cloud, based on WWDC20-10611 sample code
Using ARKit and LiDAR to save depth data and export point cloud, based on WWDC20-10611 sample code

Save iOS ARFrame and Point Cloud This project improves the usability of the sample code from WWDC20 session 10611: Explore ARKit 4. Note that the samp

An iOS Framework Capture & record ARKit videos πŸ“Ή, photos πŸŒ„, Live Photos πŸŽ‡, and GIFs πŸŽ†.
An iOS Framework Capture & record ARKit videos πŸ“Ή, photos πŸŒ„, Live Photos πŸŽ‡, and GIFs πŸŽ†.

An iOS Framework that enables developers to capture videos πŸ“Ή , photos πŸŒ„ , Live Photos πŸŽ‡ , and GIFs πŸŽ† with ARKit content.

ARKit + CoreLocation: Combines the high accuracy of AR with the scale of GPS data.
ARKit + CoreLocation: Combines the high accuracy of AR with the scale of GPS data.

ARKit: Uses camera and motion data to map out the local world as you move around. CoreLocation: Uses wifi and GPS data to determine your global locati

ARKit Demo Application
ARKit Demo Application

ARKitNavigationDemo Work in progress. In Progress Region β€” For one, we could render far fewer nodes. In fact, it’s a bit distracting that the entire t

Comments
  • Resource initialization failed:

    Resource initialization failed:

    Hey This project is so cool. I just tried to test it out. but I got this error in Xcode when i pointed the phone at a movie poster with text. Any idea of how to fix this? ARTrailer[16501:4219137] Resource initialization failed: could not open file /System/Library/LinguisticData/RequiredAssets_en.bundle/AssetData/en.lm/variants.dat

    opened by feraco 4
Owner
Wai Ting Cheung
Wai Ting Cheung
Augmented Reality image tracking with SwiftUI, RealityKit and ARKit 4.

ARImageTracking This is an Augmented Reality Xcode project that uses Apple's newest RealityKit framework and ARKit 4 features, to dynamically track a

Richard Qi 198 Dec 7, 2022
Furniture E-Commerce Augmented Reality(AR) app in iOS powered by ARKit

HomeMax-iOS Furniture E-Commerce Augmented Reality(AR) app in iOS powered by ARKit and SceneKit. Inspired by IKEA place app. Description Experience on

Ikmal Azman 5 Oct 14, 2022
ARID - Augmented Reality app using Apple’s ARKit framework which can recognise faces of famous scientists

ARID Augmented Reality app using Apple’s ARKit framework which can recognise fac

Hemanth 0 Jan 12, 2022
AR Ruler - A simple iOS app made using ARKit and SceneKit

A simple iOS app made using ARKit and SceneKit.Which can try to simplify little things in your life such as measuring stuff.

Dishant Nagpal 5 Aug 31, 2022
Smart Online Shopping iOS App with Augmented Reality (AR) and simple Social Media features using SwiftUI and Google Firebase Cloud Services

Table of contents App Demo How to Run Context Content How it's written Inspiration App Demo AR.online.shopping.iOS.demo.mp4 How to Run First make sure

Ashkan Goharfar 13 Nov 1, 2022
It is a music and podcast creation system that enables users to create multiple tracks using AR(Augmented Reality).

AR-Studio It is a music creation system enables users to create multiple tracks using AR(Augmented Reality). Introduction AR Studio turns your iPad, a

Ayush Singh 5 Dec 3, 2022
Develop simple and fun Augmented Reality (AR) iOS apps

AR-Dice Simple and fun to use iOS app made to make Augmented reality (AR) be in

Dishant Nagpal 1 Feb 23, 2022
Browse the streets of Los Angeles on an Augmented Reality historical walking tour.

Ruscha AR 0.3 Browse the streets of Los Angeles on an Augmented Reality historical walking tour. Explore Hollywood through the photos of Ed Ruscha. Th

Rick van Voorden 2 Jul 9, 2022
An augmented reality (AR) ghost hunting simulation for iPhone

ARanormal An augmented reality (AR) ghost hunting simulation for iPhone About ARanormal was the first game by Jesdo Software, released for iPhone in O

Jesse Douglas 1 Nov 2, 2021
Placing Virtual Objects in Augmented Reality

Placing Virtual Objects in Augmented Reality Learn best practices for visual feedback, gesture interactions, and realistic rendering in AR experiences

Yuchao 279 Dec 11, 2022