A Swift library for creating and exporting CoreML Models in Swift

Related tags

Machine Learning SwiftCoreMLTools
Overview

SwiftCoreMLTools

Swift Swift Package Manager compatible ver License: MIT Documentation

A Swift Library for creating CoreML models in Swift.

Work in progress

This library expose a (function builder based) DSL as well as a programmatic API (see examples below).

The library also implement Codable protocol allowing to print and edit CoreML model in JSON format.

The library is not "official" - it is not part of Apple CoreML and it is not maintained.

This library use the Apple Swift Protocol Buffer package and compile and import to Swift the CoreML ProtoBuf datastructures defined from the GitHub Apple CoreMLTools repo - https://github.com/apple/coremltools/tree/master/mlmodel/format

This package could be used to export Swift For TensorFlow models or to generate new CoreML models from scratch providing a much swifty interface compared to directly using the Swift compiled CoreML protobuf data structures.

CoreML models generated with this library could be potentially personalized (trained) partially or entirely using the CoreML runtime.

CoreML support much more then Neural Network models but this experimental library is only focused, at the moment, on Neural Network support.

End to end test case exporting a real S4TF model at: https://github.com/JacopoMangiavacchi/TestSwiftCoreMLTools

Sample projects using this library to create and train Core ML models on device

LeNet Convolutional Neural Network for MNIST dataset

Transfer Learning with Categorical Embedding

Documentation

Documentation

Neural Network Support (work in progress)

Layers

  • InnerProduct
  • Convolution
  • Embedding
  • Flatten
  • Pooling
  • Permute
  • Concat

Activation Functions

  • Linear
  • ReLu
  • LeakyReLu
  • ThresholdedReLu
  • PReLu
  • Tanh
  • ScaledTanh
  • Sigmoid
  • SigmoidHard
  • Elu
  • Softsign
  • Softplus
  • ParametricSoftplus
  • Softmax

Loss Functions

  • MSE
  • CategoricalCrossEntropy

Optimizers

  • SGD
  • Adam

Export Swift for TensorFlow sample scenario

Trivial Swift for TensorFlow model

struct LinearRegression: Layer {
    var layer1 = Dense<Float>(inputSize: 1, outputSize: 1, activation: identity)

    @differentiable
    func callAsFunction(_ input: Tensor<Float>) -> Tensor<Float> {
        return layer1(input)
    }
}

var s4tfModel = LinearRegression()
// Training Loop ...

Export to CoreML using DSL approach

let coremlModel = Model(version: 4,
                        shortDescription: "Trivial linear classifier",
                        author: "Jacopo Mangiavacchi",
                        license: "MIT",
                        userDefined: ["SwiftCoremltoolsVersion" : "0.1"]) {
    Input(name: "dense_input", shape: [1])
    Output(name: "output", shape: [1])
    NeuralNetwork {
        InnerProduct(name: "dense_1",
                     input: ["dense_input"],
                     output: ["output"],
                     weight: s4tfModel.layer1.weight.transposed().flattened().scalars,
                     bias: s4tfModel.layer1.bias.flattened().scalars,
                     inputChannels: 1,
                     outputChannels: 1)
    }
}

Export a CoreML personalizable (re-trainable) model using DSL approach

let coremlModel = Model(version: 4,
                        shortDescription: "Trivial linear classifier",
                        author: "Jacopo Mangiavacchi",
                        license: "MIT",
                        userDefined: ["SwiftCoremltoolsVersion" : "0.1"]) {
    Input(name: "dense_input", shape: [1])
    Output(name: "output", shape: [1])
    TrainingInput(name: "dense_input", shape: [1])
    TrainingInput(name: "output_true", shape: [1])
    NeuralNetwork(losses: [MSE(name: "lossLayer",
                               input: "output",
                               target: "output_true")],
                  optimizer: SGD(learningRateDefault: 0.01,
                                 learningRateMax: 0.3,
                                 miniBatchSizeDefault: 5,
                                 miniBatchSizeRange: [5],
                                 momentumDefault: 0,
                                 momentumMax: 1.0),
                  epochDefault: 2,
                  epochSet: [2],
                  shuffle: true) {
        InnerProduct(name: "dense_1",
                     input: ["dense_input"],
                     output: ["output"],
                     weight: s4tfModel.layer1.weight.transposed().flattened().scalars,
                     bias: s4tfModel.layer1.bias.flattened().scalars,
                     inputChannels: 1,
                     outputChannels: 1,
                     updatable: true)
    }
}

Example code to export and save to a CoreML model data file

let model = Model(...){ ... }
let coreMLData = model.coreMLData
try! coreMLData.write(to: URL(fileURLWithPath: "model.mlmodel"))

CoreML model creation with programmatic API

var model = Model(version: 4,
                  shortDescription: "Trivial linear classifier",
                  author: "Jacopo Mangiavacchi",
                  license: "MIT",
                  userDefined: ["SwiftCoremltoolsVersion" : "0.1"])

model.addInput(Input(name: "dense_input", shape: [1]))
model.addOutput(Output(name: "output", shape: [1]))
model.addTrainingInput(TrainingInput(name: "dense_input", shape: [1]))
model.addTrainingInput(TrainingInput(name: "output_true", shape: [1]))
model.neuralNetwork = NeuralNetwork(losses: [MSE(name: "lossLayer",
                                                 input: "output",
                                                 target: "output_true")],
                                    optimizer: SGD(learningRateDefault: 0.01,
                                                   learningRateMax: 0.3,
                                                   miniBatchSizeDefault: 5,
                                                   miniBatchSizeRange: [5],
                                                   momentumDefault: 0,
                                                   momentumMax: 1.0),
                                    epochDefault: 2,
                                    epochSet: [2],
                                    shuffle: true)

model.neuralNetwork.addLayer(InnerProduct(name: "layer1",
                                         input: ["dense_input"],
                                         output: ["output"],
                                         weight: [0.0],
                                         bias: [0.0],
                                         inputChannels: 1,
                                         outputChannels: 1,
                                         updatable: true))

YAML / JSON Format model persistence (Codable)

Example CoreML model in YAML format

version: 4
shortDescription: Trivial linear classifier
author: Jacopo Mangiavacchi
license: MIT
userDefined:
  SwiftCoremltoolsVersion: '0.1'
inputs:
  dense_input:
    name: dense_input
    shape:
    - 1
    featureType: float
outputs:
  output:
    name: output
    shape:
    - 1
    featureType: float
trainingInputs:
  dense_input:
    name: dense_input
    shape:
    - 1
    featureType: float
  output_true:
    name: output_true
    shape:
    - 1
    featureType: float
neuralNetwork:
  losses:
  - type: mse
    base:
      name: lossLayer
      input: output
      target: output_true
  optimizer:
    type: sgd
    base:
      learningRateDefault: 1e-2
      learningRateMax: 3e-1
      miniBatchSizeDefault: 5
      miniBatchSizeRange:
      - 5
      momentumDefault: 0e+0
      momentumMax: 1e+0
  epochDefault: 2
  epochSet:
  - 2
  shuffle: true
  layers:
  - type: innerProduct
    base:
      name: layer1
      input:
      - dense_input
      output:
      - output
      weight:
      - 0e+0
      bias:
      - 0e+0
      inputChannels: 1
      outputChannels: 1
      updatable: true

Example CoreML model in JSON format

{
  "author" : "Jacopo Mangiavacchi",
  "shortDescription" : "Trivial linear classifier",
  "version" : 4,
  "license" : "MIT",
  "userDefined" : {
    "SwiftCoremltoolsVersion" : "0.1"
  },
  "inputs" : {
    "dense_input" : {
      "name" : "dense_input",
      "shape" : [1],
      "featureType" : "float"
    }
  },
  "outputs" : {
    "output" : {
      "name" : "output",
      "shape" : [1],
      "featureType" : "float"
    }
  },
  "trainingInputs" : {
    "output_true" : {
      "name" : "output_true",
      "shape" : [1],
      "featureType" : "float"
    },
    "dense_input" : {
      "name" : "dense_input",
      "shape" : [1],
      "featureType" : "float"
    }
  },
  "neuralNetwork" : {
    "layers" : [
      {
        "type" : "innerProduct",
        "base" : {
          "output" : [
            "output"
          ],
          "outputChannels" : 1,
          "weight" : [0],
          "input" : [
            "dense_input"
          ],
          "bias" : [0],
          "inputChannels" : 1,
          "updatable" : true,
          "name" : "layer1"
        }
      }
    ],
    "optimizer" : {
      "type" : "sgd",
      "base" : {
        "momentumDefault" : 0,
        "momentumMax" : 1,
        "learningRateMax" : 0.3,
        "miniBatchSizeRange" : [5],
        "miniBatchSizeDefault" : 5,
        "learningRateDefault" : 0.01
      }
    },
    "losses" : [
      {
        "type" : "mse",
        "base" : {
          "name" : "lossLayer",
          "input" : "output",
          "target" : "output_true"
        }
      }
    ],
    "shuffle" : true,
    "epochDefault" : 2,
    "epochSet" : [2]
  }
}
You might also like...
A simple deep learning library for estimating a set of tags and extracting semantic feature vectors from given illustrations.
A simple deep learning library for estimating a set of tags and extracting semantic feature vectors from given illustrations.

Illustration2Vec illustration2vec (i2v) is a simple library for estimating a set of tags and extracting semantic feature vectors from given illustrati

Masaki Saito 661 Dec 12, 2022
The Swift machine learning library.
The Swift machine learning library.

Swift AI is a high-performance deep learning library written entirely in Swift. We currently offer support for all Apple platforms, with Linux support

Swift AI 5.9k Jan 2, 2023
A lightweight library to calculate tensors in Swift, which has similar APIs to TensorFlow's
A lightweight library to calculate tensors in Swift, which has similar APIs to TensorFlow's

TensorSwift TensorSwift is a lightweight library to calculate tensors, which has similar APIs to TensorFlow's. TensorSwift is useful to simulate calcu

Qoncept, Inc. 323 Oct 20, 2022
Artificial intelligence/machine learning data structures and Swift algorithms for future iOS development. bayes theorem, neural networks, and more AI.

Swift Brain The first neural network / machine learning library written in Swift. This is a project for AI algorithms in Swift for iOS and OS X develo

Vishal 331 Oct 14, 2022
Resource monitor - A flutter plugin for Android and IOS to monitor CPU and RAM usage of device.

resource_monitor A flutter plugin for Android and IOS to monitor CPU and RAM usage of device. TODO Implement Android Side of this plugin. Add listener

Skandar Munir 1 Nov 11, 2022
Accelerated tensor operations and dynamic neural networks based on reverse mode automatic differentiation for every device that can run Swift - from watchOS to Linux
Accelerated tensor operations and dynamic neural networks based on reverse mode automatic differentiation for every device that can run Swift - from watchOS to Linux

DL4S provides a high-level API for many accelerated operations common in neural networks and deep learning. It furthermore has automatic differentiati

Palle 87 Dec 29, 2022
Realtime yoga pose detection and classification plugin for Flutter using MLKit

ML Kit Pose Detection Plugin Flutter plugin for realtime pose detection using MLKit's Blazepose. License Copyright (c) 2021 Souvik Biswas, Bharat Bira

Souvik Biswas 8 May 5, 2022
Sample code for Core ML using ResNet50 provided by Apple and a custom model generated by coremltools.
Sample code for Core ML using ResNet50 provided by Apple and a custom model generated by coremltools.

CoreML-samples This is the sample code for Core ML using ResNet50 provided by Apple. ResNet50 can categorize the input image to 1000 pre-trained categ

Yuta Akizuki 39 Nov 11, 2022
DL4S provides a high-level API for many accelerated operations common in neural networks and deep learning.
DL4S provides a high-level API for many accelerated operations common in neural networks and deep learning.

DL4S provides a high-level API for many accelerated operations common in neural networks and deep learning. It furthermore has automatic differentiati

DL4S Team 2 Dec 5, 2021
Comments
  • GAN models

    GAN models

    Hi, amazing to see an initiative for ML trainning on Mac! ; )

    Would it be possible to export a model from Pytroch to coreml, and then train (not inference) this coreml model on a mac?

    opened by alelordelo 5
  • Might work on improving this in the future

    Might work on improving this in the future

    Very good work @JacopoMangiavacchi, I have been eyeing this repository for over a year.

    I may make PRs to PythonKit as a result of side-loading Swift on Colab such as subclassing and passing lambdas between Swift and Python. That would instantly make this repo feature-complete on macOS (by importing coremltools as a Python library). Then, we could slowly wean the fork off of Python until it can run on iOS.

    I'm currently resurrecting Swift for TensorFlow so that it can be used to train on iOS, and a feature-complete Swift counterpart for Python CoreMLTools would make Swift even stronger in the data science work.

    It all depends on how much time you have to dedicate to this, because I'm racing against time to beat PyTorch at a Metal backend. Expect at least one month before any of these changes can happen.

    opened by philipturner 0
  • Support for CVPixelBuffer input?

    Support for CVPixelBuffer input?

    When creating a neural network using the DSL approach, for example

     let model = Model(version: 4,
                                        shortDescription: "custom model",
                                        author: "sgttwld",
                                        license: "MIT",
                                        userDefined: ["SwiftCoremltoolsVersion" : "0.0.12"]) {
                    Input(name: "image", shape: [3, 416, 416])
                    Output(name: "output", shape: [80], featureType: .float)
                    TrainingInput(name: "image", shape: [3, 416, 416])
                    TrainingInput(name: "output_true", shape: [1], featureType: .int)
                    NeuralNetwork(losses: [CategoricalCrossEntropy(name: "lossLayer",
                                               input: "output",
                                               target: "output_true")],
                                  optimizer: Adam(learningRateDefault: 0.0001,
                                                 learningRateMax: 0.3,
                                                 miniBatchSizeDefault: 128,
                                                 miniBatchSizeRange: [128],
                                                 beta1Default: 0.9,
                                                 beta1Max: 1.0,
                                                 beta2Default: 0.999,
                                                 beta2Max: 1.0,
                                                 epsDefault: 0.00000001,
                                                 epsMax: 0.00000001),
                                  epochDefault: UInt(epoch),
                                  epochSet: [UInt(epoch)],
                                  shuffle: true) {
                        Convolution(name: "conv1",
                                     input: ["image"],
                                     output: ["outConv1"],
                                     outputChannels: 16,
                                     kernelChannels: 3,
                                     nGroups: 1,
                                     kernelSize: [3, 3],
                                     stride: [1, 1],
                                     dilationFactor: [1, 1],
                                     paddingType: .valid(borderAmounts: [EdgeSizes(startEdgeSize: 0, endEdgeSize: 0),
                                                                         EdgeSizes(startEdgeSize: 0, endEdgeSize: 0)]),
                                     outputShape: [],
                                     deconvolution: false,
                                     updatable: true)
                        ReLu(name: "relu1",
                             input: ["outConv1"],
                             output: ["outRelu1"])
                        Pooling(name: "pooling1",
                                     input: ["outRelu1"],
                                     output: ["outPooling1"],
                                     poolingType: .max,
                                     kernelSize: [2, 2],
                                     stride: [2, 2],
                                     paddingType: .valid(borderAmounts: [EdgeSizes(startEdgeSize: 0, endEdgeSize: 0),
                                                                         EdgeSizes(startEdgeSize: 0, endEdgeSize: 0)]),
                                     avgPoolExcludePadding: true,
                                     globalPooling: false)
                        Flatten(name: "flatten1",
                                     input: ["outPooling1"],
                                     output: ["outFlatten1"],
                                     mode: .last)
                        InnerProduct(name: "hidden1",
                                     input: ["outFlatten1"],
                                     output: ["outHidden1"],
                                     inputChannels: 685584,
                                     outputChannels: 80,
                                     updatable: true)
                        Softmax(name: "softmax",
                                input: ["outHidden1"],
                                output: ["output"])
                       }
}

    then the resulting model requires MLMultiArray inputs. Any chance we can get support for CVPixelBuffer inputs?

    opened by sgttwld 0
Owner
Jacopo Mangiavacchi
Microsoft Senior Data Scientist - Intrapreneur at the Microsoft Garage - Google ML Developer Expert (GDE) - Former +IBM Senior Engineer
Jacopo Mangiavacchi
GitHub
Largest list of models for Core ML (for iOS 11+)

Since iOS 11, Apple released Core ML framework to help developers integrate machine learning models into applications. The official documentation We'v

Kedan Li 5.6k Jan 3, 2023
Easily craft fast Neural Networks on iOS! Use TensorFlow models. Metal under the hood.

Bender Bender is an abstraction layer over MetalPerformanceShaders useful for working with neural networks. Contents Introduction Why did we need Bend

xmartlabs 1.7k Dec 24, 2022
WhatPet - A basic app that classifies images of dogs, cats and rabbits using CoreML

WhatPet ✓ A basic app that classifies images of dogs, cats and rabbits using Cor

Micaella Morales 0 Jan 6, 2022
CoreMLSample - CoreML Example for in app model and download model

CoreMLSample Sample for CoreML This project is CoreML Example for in app model a

Kim Seonghun 2 Aug 31, 2022
A powerful SwiftUI Architecture that merges Redux to the functional world of Swift. While bringing powerful workflows to streamline CoreML/Metal/IPFS usage in the Apple ecosystem.

GraniteUI - v0.0 - WIP A powerful SwiftUI Architecture that merges Redux event handling and state management with functional programming. While bringi

Kala 2 Dec 22, 2022
Hand-gesture recognition on iOS app using CoreML

GestureAI-CoreML-iOS Hand-gesture recognizer using CoreML Demo Screenshots This app is using RNN(Recurrent Neural network) with CoreML on iOS11. The m

null 151 Nov 2, 2022
Tiny YOLO for iOS implemented using CoreML but also using the new MPS graph API.

YOLO with Core ML and MPSNNGraph This is the source code for my blog post YOLO: Core ML versus MPSNNGraph. YOLO is an object detection network. It can

Matthijs Hollemans 900 Dec 31, 2022
An example of CoreML using a pre-trained VGG16 model

CoreMLExample In this example we use AVFoundation to continuously get image data from the back camera, and try to detect the dominant objects present

Aleph Retamal 34 Apr 22, 2022
Matft is Numpy-like library in Swift. Function name and usage is similar to Numpy.

Numpy-like library in swift. (Multi-dimensional Array, ndarray, matrix and vector library)

null 80 Dec 21, 2022
A Swift deep learning library with Accelerate and Metal support.

Serrano Aiming to offering popular and cutting edge techs in deep learning area on iOS devices, Serrano is developed as a tool for developers & resear

pcpLiu 51 Nov 17, 2022