Source Model Link

https://github.com/zllrunning/face-parsing.PyTorch

Core ML Model Download Link

https://github.com/tucan9389/SemanticSegmentation-CoreML/releases/download/support-face-parsing/FaceParsing.mlmodel

Model Spec

• Input: 512x512 image
• Output: 512x512 (Int32)
  • Catetory index of each pixel
  • Defined 19 categories: ['background', 'skin', 'l_brow', 'r_brow', 'l_eye', 'r_eye', 'eye_g', 'l_ear', 'r_ear', 'ear_r', 'nose', 'mouth', 'u_lip', 'l_lip', 'neck', 'neck_l', 'cloth', 'hair', 'hat']
• Size: 52.7 MB
• Inference time: 30-50 ms in iPhone 11 Pro

Conversion Script

import torch

import os.path as osp
import json
from PIL import Image
import torchvision.transforms as transforms
from model import BiSeNet

import coremltools as ct

dspth = 'res/test-img'
cp = '79999_iter.pth'
device = torch.device('cpu')

output_mlmodel_path = "FaceParsing.mlmodel"

labels = ['background', 'skin', 'l_brow', 'r_brow', 'l_eye', 'r_eye', 'eye_g', 'l_ear', 'r_ear', 'ear_r',
            'nose', 'mouth', 'u_lip', 'l_lip', 'neck', 'neck_l', 'cloth', 'hair', 'hat']
n_classes = len(labels)
print("n_classes:", n_classes)

class MyBiSeNet(torch.nn.Module):
    def __init__(self, n_classes, pretrained_model_path):
        super(MyBiSeNet, self).__init__()
        self.model = BiSeNet(n_classes=n_classes)
        self.model.load_state_dict(torch.load(pretrained_model_path, map_location=device))
        self.model.eval()

    def forward(self, x):
        x = self.model(x)
        x = x[0]
        x = torch.argmax(x, dim=1)
        x = torch.squeeze(x)
        return x

pretrained_model_path = osp.join('res/cp', cp)
model = MyBiSeNet(n_classes=n_classes, pretrained_model_path=pretrained_model_path)
model.eval()

example_input = torch.rand(1, 3, 512, 512)  # after test, will get 'size mismatch' error message with size 256x256
preprocess = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(
        mean=[0.485, 0.456, 0.406],
        std=[0.229, 0.224, 0.225],
    ),
])

traced_model = torch.jit.trace(model, example_input)


# Convert to Core ML using the Unified Conversion API
print(example_input.shape)

scale = 1.0 / (0.226 * 255.0)
red_bias   = -0.485 / 0.226
green_bias = -0.456 / 0.226
blue_bias  = -0.406 / 0.226

mlmodel = ct.convert(
    traced_model,
    inputs=[ct.ImageType(name="input",
                         shape=example_input.shape,
                         scale=scale,
                         color_layout="BGR",
                         bias=[blue_bias, green_bias, red_bias])], #name "input_1" is used in 'quickstart'
)



labels_json = {"labels": labels}

mlmodel.user_defined_metadata["com.apple.coreml.model.preview.type"] = "imageSegmenter"
mlmodel.user_defined_metadata['com.apple.coreml.model.preview.params'] = json.dumps(labels_json)

mlmodel.save(output_mlmodel_path)

import coremltools.proto.FeatureTypes_pb2 as ft

spec = ct.utils.load_spec(output_mlmodel_path)

for feature in spec.description.output:
    if feature.type.HasField("multiArrayType"):
        feature.type.multiArrayType.dataType = ft.ArrayFeatureType.INT32

ct.utils.save_spec(spec, output_mlmodel_path)

Hi there - first off, great work on this repo! :D

I wonder if there's a way to segment the whole image by padding the sides - as it stands, since imageCropAndScaleOption is .centerCrop, we only get the center.

| face-parsing model | | --- | | |

video source: https://www.youtube.com/watch?v=D571qZzLfX4

PR Points

• Support the face-parsing semantic segmentation model
• Support to visualize multi-class semantic segmentation output
  • Add MultitargetSegmentationTextureGenerater.swift
  • Add new multitarget_segmentation_render_target shader function in Shaders.metal

Model Info

• original model repo: https://github.com/zllrunning/face-parsing.PyTorch

Related Issues

• #12
• #13
• https://github.com/zllrunning/face-parsing.PyTorch/issues/27

How to run the FaceParsing model?

1. Download the mlmodel from here
2. Import the FaceParsing.mlmodel into SemanticSegmentation-CoreML>mlmodel folder of the project
3. Change the minimum iOS version to 14.0+
4. Change the model class from DeepLabV3() to FaceParsing() in LiveImageViewController.swift or LiveMetalCameraViewController.swift
5. Build & run the project on the real device

PR Points

• Add apps tab based on UITableViewController
• Add app feature for tracking a face and segmenting it with Metal
  • Use Vision framework and Core ML for face detection and face-parsing semantic segmentation

Super thanks @jsharp83 for your MetalCamera

Related Issue

• #5

Changed

• Make post-processing on real-time segmentation extremely fast
  • AS-IS(CPU): > 240 ms (iPhone 11 Pro)
  • TO-BE(GPU): < 1 ms (iPhone 11 Pro)
• The main logic during generating textures:
  1. Get current pixelbuffer from camera
  2. Inference the pixelbuffer and get a segmentation map
  3. Generate a texture from the pixelbuffer (cameraTexture) https://github.com/tucan9389/SemanticSegmentation-CoreML/pull/7#discussion_r524046755
  4. Generate a texture from the segmentation map (segmentaitonTexture) https://github.com/tucan9389/SemanticSegmentation-CoreML/pull/7#discussion_r524047448
  5. Merge the textures and make a texture (overlayedTexture) https://github.com/tucan9389/SemanticSegmentation-CoreML/pull/7#discussion_r524047448
  6. Assign the overlayedTexture to metalVideoPreview: MetalVideoView
  7. Draw the overlayedTexture texture

PR Points

• Add LiveMetalCameraViewController post-processing with MetalKit
• Support extremely fast post-process
  • Add MetalRenderingDevice.swift from MetalCamera
  • Add MetalVideoView.swift from MetalCamera
  • Add CameraTextureGenerater.swift from MetalCamera
  • Add SegmentationTextureGenerater.swift from MetalCamera
  • Add OverlayingTexturesGenerater.swift from MetalCamera
  • Add other utils and shader code from MetalCamera

Reference

• https://github.com/jsharp83/MetalCamera
• https://github.com/jsharp83/MetalCamera/blob/master/Example/MetalCamera/SegmentationSampleViewController.swift

Model Size (MB), Minimum iOS Version

| Model | Size | Minimum
iOS Version | | ----: | :----: | :----: | | DeepLabV3 | 8.6 | iOS12 | | DeepLabV3FP16 | 4.3 | iOS12 | | DeepLabV3Int8LUT | 2.3 | iOS12 |

Infernece Time (ms)

| Model vs. Device | XS | X | | ----: | :----: | :----: | | DeepLabV3 | 135 | 177 | | DeepLabV3FP16 | 136 | 177 | | DeepLabV3Int8LUT | 135 | 177 |

Total Time (ms)

| Model vs. Device | XS | X | | ----: | :----: | :----: | | DeepLabV3 | 409 | 531 | | DeepLabV3FP16 | 403 | 530 | | DeepLabV3Int8LUT | 412 | 517 |

FPS

| Model vs. Device | XS | X | | ----: | :----: | :----: | | DeepLabV3 | 2 | 1 | | DeepLabV3FP16 | 2 | 1 | | DeepLabV3Int8LUT | 2 | 1 |

Hi, Thanks for the great repo!

Let's imagine the coreml network needs a cropped image (only a subpart of the camera feed). No matter how the crop is done (it can be hard coded for testing purpose), I wonder if there is a way to change the DrawingSegmentationView for example, to achieve this. Right now, if the input image is cropped, the output view resize the image to the viewport and the result isn't well registered.