CoreML-in-ARKit
This simple project detects objects in Augmented Reality and displays 3D labels on top of them. This serves as a basic template for an ARKit project to use CoreML.
Model: Inception V3
Language: Swift 4.0
Written in: Xcode 9.0 GM (9A235) (Updated) XCode 9 beta 3 (9M174d)
Content Technology: SceneKit
Tested on iPhone 7 plus running iOS 11 beta 3 (15A5318g)
Note: SceneKit can achieve a 60 FPS on iPhone7+ - though when it gets hot, it'll drop to 30 FPS.
Instructions
You'll have to download "Inceptionv3.mlmodel" from Apple's Machine Learning page, and copy it into your XCode project. (As depicted in the following gif)
If you're having issues, double check that the model is part of a target (source: stackoverflow).
Footnotes
-
SceneKit Text Labels are expensive to render. Too many polygons (too much text, smoothness, characters) - can cause crashes. In future, SpriteKit would be more efficient for text-labels.
-
Not entirely certain if the code is actually transferring RGB data to the Vision Model (as opposed to YUV). Proof-of-concept-wise, it appears to work just fine with the Inception V3 model for now.
-
Whilst ARKit's FPS , is displayed - CoreML's speed is not. However, it does appear sufficiently fast for real-time ARKit applications.
-
Placement of the label is simply determined by the raycast screen centre-point to a ARKit feature-point. This could be altered for more stable placement.
Building Blocks (Overview)
Get CoreML running in real time in ARKit
-
There are some good tutorials / sample projects for getting CoreML running. See: [ 1 ] [ 2 ] [ 3 ]
-
What we do differently here is we're using ARKit's ARFrame as the image to be fed into CoreML.
let pixbuff : CVPixelBuffer? = (sceneView.session.currentFrame?.capturedImage)
- We also use Threading to continuously run requests to CoreML in realtime, and without disturbing ARKit / SceneView
let dispatchQueueML = DispatchQueue(label: "com.hw.dispatchqueueml")
...
loopCoreMLUpdate() // on viewLoad
...
func loopCoreMLUpdate() {
dispatchQueueML.async {
// 1. Run Update.
self.updateCoreML()
// 2. Loop this function.
self.loopCoreMLUpdate()
}
}
Add 3D Text
- Add a Tap Gesture.
- On Tap. Get the raycast centre point, translating it to appropriate coordinates.
- Render 3D text at that location. Use the most likely object.
3 Feb 4, 2022
5 Aug 31, 2022
0 Oct 20, 2022
279 Dec 11, 2022
23 Dec 16, 2022
91 Dec 4, 2022
4 Dec 22, 2022
74 Dec 21, 2022
1.5k Dec 24, 2022
198 Dec 7, 2022
5.3k Dec 27, 2022
296 Dec 16, 2022
20 Oct 31, 2022
628 Oct 22, 2022
93 Nov 15, 2022
516 Dec 1, 2022
0 Dec 21, 2021
3 Dec 18, 2021