HandTalk

Tongji Univ. SSE IOS Application Development coursework (final project) : An Demo based on American Sign Language Classfier

Development information

• Environment: macOS Monterey 12.1
• Dev tool: Xcode 13, Xcode 13 Simulator , CreateML, Netron
• Language: Swift
• Interface: SwiftUI
• Third-party service: This application uses extentional package ACarousel (github url: https://github.com/JWAutumn/ACarousel ) to present carousel view
• requirements: IOS 12.0+ (for use) ; macOS 10.14+ (for source code debug)

Model Overview

The dataset link: https://www.kaggle.com/grassknoted/asl-alphabet

The final version of mlmodel originates from the latest techniques from WWDC 2021 (https://developer.apple.com/videos/play/wwdc2021/10039/). Previous version includes using CreateML Image Classifier model, and an transformed mlmodel from Pytorch model (using coremltools) , which gave disappointing performance on real-time debugging.

A diagram of the model structure can be seen in Snapshots.