The example project of inferencing Pose Estimation using Core ML

Try to use hourglass model in here.

• input_image_shape: [1, 192, 192, 3]
• output_shape: [1, 48, 48, 14]
• input_node_name: image
• output_node_name: hourglass_out_3

I think it will speed up 3x than old model and the precision is acceptable.

Tomorrow I will update my repo.

Look forward to your replay ( ? FPS on ipx )

Run Joint View sample in Xcode 10.3, the FPS is 15 ~ 20, But run same project in Xcode 11.3, the FPS dropdown to 6~7, my device is iPhone 7, Is anything setting I need to change ?

Keep running into

Unknown class _TtC21PoseEstimation_CoreML19JointViewController in Interface Builder file.

Error

Tried running on Iphone 6 and Iphone 11 simulators

Hello

I need to add normalization to the image in the preprocess

This is how it's being done in python (From here):

    parser.add_argument('--std', type=list, default=[0.229, 0.224, 0.225],
                        help='the std used to normalize your images')
    parser.add_argument('--mean', type=list, default=[0.485, 0.456, 0.406],
                        help='the mean used to normalize your images')

    normalize = transforms.Normalize(std=args.std,
                                     mean=args.mean)

    transformation = transforms.Compose([transforms.ToTensor()])

    man_normalize = transformation(man_resize)

This is how it's being done in Android:

@Override
  | protected void addPixelValue(int pixelValue) {
  | imgData.putFloat((((pixelValue >> 16) & 0xFF)/255f - IMAGE_MEAN[0]) / IMAGE_STD[0]);
  | imgData.putFloat((((pixelValue >> 8) & 0xFF)/255f - IMAGE_MEAN[1]) / IMAGE_STD[1]);
  | imgData.putFloat(((pixelValue & 0xFF)/255f - IMAGE_MEAN[2]) / IMAGE_STD[2]);
  | }

How can I do similar thing in your Swift application?

Thanks

Demo

| Pose Capturing | Pose Matching | | ---- | ---- | | | |

PR Point

• Add pose capturing and matching

How to make the matching rate

1. Vector based method

1. Get the connected points (p1_1, p1_2) from captured points
2. Get the connected points (p2_1, p2_2) from predicted points
3. Exception handling (nil check, confidence check)
4. Get two vector

let vec1: CGPoint = p1_2 - p1_1
let vec2: CGPoint = p2_2 - p2_1

5. Calculate the angle with vec1, (0, 0), vec2: angle loss
6. Sum all angle loss and divide by (numberOfLoss) * (π/2): loss rate
7. Check the 6th's result is on 0.0 to 1.0 range.
8. Flip the result

let matchingRate = 1 - lossRate

2. Angle based method

Implemented on source code

Hello @tucan9389 @taehyen ,

Thanks for creating wonderful library for pose detection. But I need to know that can we draw dots only selected part of body instead full body while full body visible in front of camera.

Thanks and Regards

Hello. Thanks for the amazing repository!

I built the project and it works great.

I am trying to understand the performance test. I would be glad if you can elaborate about what was measured.

Looking at Measure.swift:

func 🎬🤚() {
       🏷(for: index, with: "end")
       
       let beforeMeasurement = getBeforeMeasurment(for: index)
       let currentMeasurement = measurements[index]
       if let startTime = currentMeasurement["start"],
           let endInferenceTime = currentMeasurement["endInference"],
           let endTime = currentMeasurement["end"],
           let beforeStartTime = beforeMeasurement["start"] {
           delegate?.updateMeasure(inferenceTime: endInferenceTime - startTime,
                                   executionTime: endTime - startTime,
                                   fps: Int(1/(startTime - beforeStartTime)))
       }
       
   }

startTime is the moment that the image is received. It is a pixelBuffer, (640x480) image. endInferenceTime is the moment when the inference was stopped, receiving a (14, 96, 96) heatmap from the CPM model. beforeStartTime is the start time of the previous frame. Why do we measure it?

If so I understand what is:

• inferenceTime: endInferenceTime - startTime - from pixelBuffer, (640x480) image to (14, 96, 96) heatmap .
• executionTime: endTime - startTime - from pixelBuffer, (640x480), including post-process (convertToPredictedPoints, and moving average filter. BTW - what is this filter?)

But I don't understand what is fps: Int(1/(startTime - beforeStartTime))) - why is the beforeStartTime relevant?

To summarize, my questions are:

1. Did I understand correctly the inferenceTime and executionTime?
2. Why is the fps calculated with beforeStartTime?
3. Why do you have moving average filter in the post-process code? Is it instead the gaussian_filter in the original CPM code?

Thank you very much.

Hello

I am trying to use other model and it gets complicated.

I was wondering whether someone haev minimal example on how to run inference on existing image from memory? This way it will be simpler to understand how to adapt different model.

Thanks for the repo!

hello @tucan9389 i'm flutter developer i don't understand swift language and i just want to know about how you captured the poses with only skeleton drawn and compared it with live feed i'm using different posenet moel here please guide me with this.

Since this is not an issue so you can close this and please reply here email : [email protected]

Thanks :)