Hi, thanks for the great work!

I'm trying to convert the AnimeGANv2 Shinkai.pb model to CoreML, and I've got some questions.

1. Did you use the check points to generate a .pb file first, maybe using this script then convert it to coreml using coremltools?
2. In coremltools, should I use ct.ImageType(shape=(1, 256, 256, 3)) as input format of the converter?

Here's my converter code:

image_inputs = ct.ImageType(shape=(1, 256, 256, 3))
mlmodel = ct.convert('./Shinkai_53.pb', inputs=[image_inputs],  source='tensorflow')
mlmodel.save('./output.mlmodel')

And then I use:

import coremltools.proto.FeatureTypes_pb2 as ft
spec = ct.utils.load_spec("output.mlmodel")
output = spec.description.output[0]
output.type.imageType.colorSpace = ft.ImageFeatureType.RGB
output.type.imageType.height = 256
output.type.imageType.width = 256
ct.utils.save_spec(spec, "new.mlmodel")

to convert the output format to Image. After I drag the new mlmodel to Xcode, and preview the model with an image, the stylized image can't be generated. It seems to be loading forever.

Do u have a cue on what's going wrong? Or could u please tell me how u convert the AnimeGANv2 models? Again, thanks for your great work. Really appreciate it.

Hi: Thank you for the great work! Can you please provide the code for converting the RealESRGAN model to CoreML Model?

I tried to build by myself but apparently the output is wrong.

import coremltools as ct
import torch
from basicsr.archs.rrdbnet_arch import RRDBNet
import coremltools.proto.FeatureTypes_pb2 as ft

example_input = torch.rand(1, 3, 256, 256)
example_output = torch.rand(1, 3, 1024, 1024)
model_path = "/Users/vaida/Downloads/Safari download/RealESRGAN_x4plus.pth"

model = RRDBNet(num_in_ch=3, num_out_ch=3, num_feat=64, num_block=23, num_grow_ch=32, scale=4)
loadnet = torch.load(model_path, map_location=torch.device('cpu'))
# prefer to use params_ema
if 'params_ema' in loadnet:
    keyname = 'params_ema'
else:
    keyname = 'params'
model.load_state_dict(loadnet[keyname], strict=True)

traced_model = torch.jit.trace(model, example_input)

image_input = ct.ImageType(name="input", shape=example_input.shape)
image_output = ct.ImageType(name="output", shape=example_output.shape)


mlmodel = ct.convert(
	traced_model,
	source = "pytorch",
	inputs = [image_input]
)

spec = mlmodel.get_spec()

ct.utils.rename_feature(spec, "var_4053", "image")
output = spec.description.output[0]
output.name = "image"

output.type.imageType.colorSpace = ft.ImageFeatureType.ColorSpace.Value('RGB')
output.type.imageType.width = 1024
output.type.imageType.height = 1024

mlmodel = ct.models.MLModel(spec)
mlmodel.save("newmodel.mlpackage")

Thanks for this great repo and all the work you have done! I tested this model with several images in preview of Xcode 13.2.1. It always produce black image. https://drive.google.com/file/d/1o15OO0Kn0tq79fFkmBm3PES93IRQOxB-/view?usp=sharing

Hi, thanks for converting IS-Net models to Core ML. As I tried with Xcode 13.4.1 mlmodel preview function, both two models process forever and never return results. Could you please check it?

I am writing here to say thanks. I am new to iOS development, and just recently launched my app PixAI.

I used your scripts for A-ESRGAN and MM-RealSR conversion , the process was very smooth and it saved me a lot of time.

Best regards.

Hi, I converted a object detection model from pytorch to coreml with the code

def main(config, model_path, output_path, input_shape=(320, 320)):
    logger = Logger(-1, config.save_dir, False)
    model = build_model(config.model)
    checkpoint = torch.load(model_path, map_location=lambda storage, loc: storage)
    load_model_weight(model, checkpoint, logger)

    dummy_input = torch.autograd.Variable(
        torch.randn(1, 3, input_shape[0], input_shape[1])
    )

    traced_model = torch.jit.trace(model, dummy_input)

    logging.info("convert coreml start.")
    core_model = ct.convert(
        traced_model,
        inputs=[ct.ImageType(shape=dummy_input.shape, name='input', scale=0.017429, bias=(-103.53 * 0.017429, -116.28 * 0.017507, -123.675 * 0.017125))],
        outputs=[ct.TensorType(name="output")],
        debug=True
    )
    core_model.save(output_path)
    logging.info("finish convert coreml.")

the infer code with nms

image = Image.open(img_path).resize((320, 320)).convert('RGB')
model = ct.models.MLModel(mlmodel_path)
preds = model.predict({'input': image})
#post-process nms for preds

and the coreml result seems not correct as below the pytorch results the coreml results

could you please give me some advice about hot to get a correct coreml model, and it is so much better if you can convert the model for me if it is convenient, many thanks.

Dear author: I found you can set both the input and output type in Super resolution project to Image type in coreml. Thus it could use preview function in xcode very easily. However, I can not find any ways to set the output multiarray tensor as image type. The online resouce only give details on how to set input as image type. None of those tell how to set output. Thank you.

Hi @john-rocky:

Thanks so much for this repo. I wondered if you could describe how you went about exporting the yolov5s model to use CoreML?

I have tried this repeatedly on the most recent version of the yolov5 repo.

export.py --weights yolov5s.pt --include coreml to get the default values but whenever I try to import the resulting yolov5s.mlmodel into Xcode, the preview panel does not render and if I try to deploy it into an iOS app like this: https://github.com/shu223/MLModelCamera, it fails to function.

When I downloaded your yolov5s.mlmodel file, however Xcode was happy and it worked fine in MLModelCamera.

I'm running macOS v12.3.1 (Monterey), on a 2019 Intel MacBook Pro. Any thoughts you might have on why this is failing would be most welcome.

GFPGAN fails to load on ANE w/ CoreML, runs CPU only.

2022-09-09 00:46:25.563463-0700 CoreML Workbench[7881:699189] [espresso] [Espresso::handle_ex_plan] exception=ANECF error: ANECCompile(/Library/Caches/com.apple.aned/tmp/com.***.CoreML-Workbench/6E5190711B5E7CF18E3503087B04E28C40B3367120EBE1F811A6E081F667851C/D1A3E180F5A89B0B922655B966D949D288EC451818F77B38B6EC76EBA01859D8/) FAILED: err=(
    CompilationFailure
)
2022-09-09 00:46:25.564051-0700 CoreML Workbench[7881:699189] [coreml] Error plan build: -1.
2022-09-09 00:46:25.578371-0700 CoreML Workbench[7881:699189] [client] doUnloadModel:options:qos:error:: nil _ANEModel
2022-09-09 00:46:25.578522-0700 CoreML Workbench[7881:699189] [espresso] ANECF error:

Can you please share your convert script and how you converted it to coreml?

Hi, They appear to have updated their ISNET model for general use. https://github.com/xuebinqin/DIS

I really want to wait until V2, but they say it will take months.

Yolov7 trained with 1280 x 1280 images (pertained weights / model used: yolov7-d6) Trained with 5 custom labels. Inference runs fine, I added --img-size 1280 to the detect command: !python detect.py --weights my_custom_weights.pt --conf 0.25 --img-size 1280 --source inference/images/my_image.jpg

I updated classLabels[] to my 5 labels (removing the rest) I added --img-size 1280 to the export command ie !python export.py --img-size 1280 --weight my_custom_weights.pt

but on the last cell, output is:

IndexError Traceback (most recent call last) in () 1 # run the functions to add decode layer and NMS to the model. ----> 2 addExportLayerToCoreml(builder) 3 nmsSpec = createNmsModelSpec(builder.spec) 4 combineModelsAndExport(builder.spec, nmsSpec, f"big_data_best.mlmodel") # The model will be saved in this path.

in addExportLayerToCoreml(builder) 45 f"{outputName}_multiplied_xy_by_two"], output_name=f"{outputName}subtracted_0_5_from_xy", mode="ADD", alpha=-0.5) 46 grid = make_grid( ---> 47 featureMapDimensions[i], featureMapDimensions[i]).numpy() 48 # x,y * 2 - 0.5 + grid[i] 49 builder.add_bias(name=f"add_grid_from_xy{outputName}", input_name=f"{outputName}_subtracted_0_5_from_xy",

IndexError: list index out of range

I tried changing cell 8:

featureMapDimensions = [640 // stride for stride in strides] to featureMapDimensions = [1280 // stride for stride in strides]

as well as: builder.add_scale(name=f"normalize_coordinates_{outputName}", input_name=f"{outputName}_raw_coordinates", output_name=f"{outputName}raw_normalized_coordinates", W=torch.tensor([1 / 640]).numpy(), b=0, has_bias=False) to builder.add_scale(name=f"normalize_coordinates{outputName}", input_name=f"{outputName}_raw_coordinates", output_name=f"{outputName}_raw_normalized_coordinates", W=torch.tensor([1 / 1280]).numpy(), b=0, has_bias=False)

neither attempt worked.

Also, Yolov7 export.py CoreML was recently updated. I have previously exported a 1280 x 1280 image size Yolov5 custom trained model to Core ML using this repo.

Any thoughts or ideas would be greatly appreciated!!