Pose Estimation on iOS with TensorFlow Lite

I try to build the project locally but have issues with the TFLiteSwift-Vision dependeny. I get the error:

No such module 'TFLiteSwift_Vision'

and

/Package.swift has no Package.swift manifest for version 0.2.8 in https://github.com/tucan9389/TFLiteSwift-Vision

Using Xcode 13.4.1.

Any ideas what I am doing wrong?

I am trying to run the project in Xcode 11.4.1 but all the .tflite files are missing under imgclsmob-Pose folder in Xcode. Anyone, please help me out with this. I want to compile the project.

Thanks

Hi, I was trying to run your application, but I cant seem to find lightweight_baseline_choi.tflite model file, can you please add the link for downloading.

Hello, I am having trouble finding these files.

PoseEstimation-TFLiteSwift/PoseEstimation-TFLiteSwift/LiteBaseline3DPose/litebaseline_3dpose_real_S.tflite: No such file or directory

PoseEstimation-TFLiteSwift/PoseEstimation-TFLiteSwift/LiteBaseline3DPose/litebaseline_3dpose_real_L.tflite: No such file or directory

I found the 'imgclsmob-Pose' and the 'Baseline3DPose' tflite files but am still having trouble finding the 'LiteBaseline3DPose' files

Thank you in advance.

| demo | | --- | | |

Related Issue

• https://github.com/tucan9389/PoseEstimation-TFLiteSwift/issues/28

PR Points

• Implement the pre-process
  • Image normalization using in PyTorch usually
• Implement the post-process
  • Define the keypoints each index
  • Implement softmax
  • Implement sum function that could get dim as augment like numpy or pytorch
  • Optimize sum and mul with Accelerate framework
• Log the 3d points
• Draw the 3d points in the app screen
• Support real-time (later)
• Implement additional post-processing for removing some noises (later)

Hello,

I am playing with pose estimation models. I am trying to find a model which run on iOS and able to identify join when the body is horizontal. It seems most models (at least poseNet posenet_mobilenet_v1_100_257x257_multi_kpt_strippe) works better vertically.

do you have any suggestions ?

Thank you

Hi. Thanks for your awesome work, we refered your postprocess implementation about openpose+TFlite. When we'd like to use in a production code, we wondering if we can use it.

hello,

I am getting this error on iPhone8. No apps running on background. I think the msg is clear but any tips to make is work ? maybe load less models ?

Connection to assetsd was interrupted - assetsd exited, died, or closed the photo library
Message from debugger: Terminated due to memory issue

any idea ?

Related Issue

• https://github.com/tucan9389/PoseEstimation-TFLiteSwift/issues/5

PR Point

• Add TFLiteImageInterpretor for implementing pre-process of image input
• Add PoseEstimator protocol for abstraction
• Add PoseNetPoseEstimator for specifying input/output and model name of PoseNet tflite file

PR Point

• Implement shoulder fixing function for 3d pose keypoints
  • Implement to rotate 3d vector by x-axis, y-axis, z-axis
  • Implement custom operator for multiplying or moving 3d vector
  • Use the shoulder fixing before performing pose-matching algorithm
• Add a shoulder fixing switch for each 3d live demo page

Vector Rotation

In 3D rotating around the Z-axis would be

    |cos θ   −sin θ   0| |x|   |x cos θ − y sin θ|   |x'|
    |sin θ    cos θ   0| |y| = |x sin θ + y cos θ| = |y'|
    |  0       0      1| |z|   |        z        |   |z'|

around the Y-axis would be

    | cos θ    0  -sin θ| |x|   | x cos θ + z sin θ|   |x'|
    |   0      1       0| |y| = |         y        | = |y'|
    |sin θ    0    cos θ| |z|   |−x sin θ + z cos θ|   |z'|

around the X-axis would be

    |1     0           0| |x|   |        x        |   |x'|
    |0   cos θ    −sin θ| |y| = |y cos θ − z sin θ| = |y'|
    |0   sin θ     cos θ| |z|   |y sin θ + z cos θ|   |z'|

Reference

• https://developer.apple.com/documentation/accelerate/working_with_matrices
• https://stackoverflow.com/a/14609567/4160632

• [x] capture 3d pose by tapping a button or listening to "capture"
  • speech recognition: https://github.com/jen2/Speech-Recognition-Demo
• [x] match the inferred pose and captured pose
• [x] show the matching score and highlight which captured pose is matched

Hi, in the readme you also have the 2D Demo .gif: https://user-images.githubusercontent.com/37643248/77227994-99ba2a80-6bc7-11ea-9b08-9bb57723bc42.gif

But where is the matching code to get this result? I only see the 3D views?

Hi ,

I tried my own trained model, but the output shape of my model is 1 x 21 x 3, but the model that you gave has an output shape of 1 x 672 x 32 x 32. I trained the model using https://github.com/SangbumChoi/MobileHumanPose repo. Could you please tell me why is this issue happening?

Thanks and Regards, Ashish

The app sometimes crashes during the scale operation in vImageScale_ARGB8888(&croppedImage, &resizedImage, nil, vImage_Flags(0)) == kvImageNoError

Goal

• For robust pose matching algorithm, implement pre-processing before cosine similarity

How to make it

1. Adjust predicted pose keypoints by captured right and left shoulder's keypoint
   • transform the predicted keypoints as same shoulder keypoints coordinate between predicted and captured

Editing Point

https://github.com/tucan9389/PoseEstimation-https://github.com/tucan9389/PoseEstimation-TFLiteSwift/blob/da2931e39eea3065d72d4c83383387bbef482bc0/PoseEstimation-TFLiteSwift/Argmax.swift#L25-L41

Reference

https://github.com/hollance/CoreMLHelpers/blob/0bd7f983c3e73abcffb692b4428e889ed4cf01b6/CoreMLHelpers/Math.swift#L23-L82