Stable Diffusion inference on iOS / macOS using MPSGraph

Overview

🍁 Maple Diffusion

Maple Diffusion runs Stable Diffusion models locally on macOS / iOS devices, in Swift, using the MPSGraph framework (not Python).

Maple Diffusion should be capable of generating a reasonable image in a minute or two on a recent iPhone (I get around ~2.3s / step on an iPhone 13 Pro).

To attain usable performance without tripping over iOS's 4GB memory limit, Maple Diffusion relies internally on FP16 (NHWC) tensors, operator fusion from MPSGraph, and a truly pitiable degree of swapping models to device storage.

On macOS, Maple Diffusion uses slightly more memory (~6GB), to reach <1s / step.

Projects using Maple Diffusion

Device Requirements

Maple Diffusion should run on any Apple Silicon Mac (M1, M2, etc.). Intel macs seem to not support Float16 or something.

Maple Diffusion should run on any iOS device with sufficient RAM (β‰₯6144MB RAM definitely works; 4096MB might but I wouldn't bet on it; anything lower than that won't work). Recent iPads should work out of the box; recent iPhones should work if you can get the Increase Memory Limit capability working (to unlock 4GB of RAM).

iOS 16.1 (beta) is reportedly broken and always generating a gray image :(

Usage

To build and run Maple Diffusion:

  1. Download a Stable Diffusion PyTorch model checkpoint (sd-v1-4.ckpt, or some derivation thereof)

  2. Download this repo

    git clone https://github.com/madebyollin/maple-diffusion.git && cd maple-diffusion
  3. Setup & install Python with PyTorch, if you haven't already.

    # may need to install conda first https://github.com/conda-forge/miniforge#homebrew
    conda deactivate
    conda remove -n maple-diffusion --all
    conda create -n maple-diffusion python=3.10
    conda activate maple-diffusion
    pip install torch typing_extensions numpy Pillow requests pytorch_lightning
  4. Convert the PyTorch model checkpoint into a bunch of fp16 binary blobs.

    ./maple-convert.py ~/Downloads/sd-v1-4.ckpt
  5. Open the maple-diffusion Xcode project. Select the device you want to run on from the Product > Destination menu.

  6. Manually add the Increased Memory Limit capability to the maple-diffusion target (this step might not be needed on iPads, but it's definitely needed on iPhones - the default limit is 3GB).

  7. Build & run the project on your device with the Product > Run menu.

You might also like...
macOS app for Hype Machine
macOS app for Hype Machine

Plug Discover and listen to music from Hype Machine Download Requires macOS 10.15 or later. Screenshot FAQ Can I contribute localizations? We do not h

A macOS app to visualise your iTunes library as graphs.
A macOS app to visualise your iTunes library as graphs.

iTunes Graphs iTunes Graphs is a Cocoa-based macOS app which visualises your iTunes library in a series of pie charts. Currently, it supports the foll

A simple Spotify lyrics viewer menu bar app for macOS in Swift 3
A simple Spotify lyrics viewer menu bar app for macOS in Swift 3

lyricsify This is a simple macOS menu bar application that shows you the lyrics of current playing spotify track. All the lyricses are from Wikia webs

Simple command line utility for switching audio inputs and outputs on macOS

Switch Audio Simple command line utility for switching audio inputs and outputs

PTStations - MacOS App to gather the information on the Train Station of the Portuguese Railway system

PTStations A simple macOS app that gathers information on the Portuguese Railway

TTSLanguage: Text To Speech commandline executable for macOS

TTSLanguage Text To Speech commandline executable for macOS. It can detect sentence language and read it using proper voice. example: $ TTSLanguage "H

iOS framework that enables detecting and handling voice commands using microphone.

iOS framework that enables detecting and handling voice commands using microphone. Built using Swift with minumum target iOS 14.3.

Beautiful Music Player app built using SwiftUI to demonstrate Neumorphic design pattern and MVVM architecture.
Beautiful Music Player app built using SwiftUI to demonstrate Neumorphic design pattern and MVVM architecture.

Beautiful Music Player app built using SwiftUI to demonstrate Neumorphic design pattern 🎨 and MVVM architecture πŸ— . Made with love ❀️ by Sameer Nawaz

The DoubanFM for iPhone,using AFN and MPMoviePlayer
The DoubanFM for iPhone,using AFN and MPMoviePlayer

DoubanFM github:@XVXVXXX csdn:@XVXVXXX please open DoubanFM.xcworkspace The DoubanFM for iPhone,using MPMoviePlayer [email protected]/AFNetwo

Comments
  • Added an icon for maple diffusion,

    Added an icon for maple diffusion,

    Thought you'd get a kick out the fact I used the app to auto generate it's own icon. I tried many times but the first one was the best by far so it feels like fate.

    opened by Halopend 0
Releases(0.3)
Owner
Morten Just
Isn't it cool how we all work together?
Morten Just
AudioPlayer is a simple class for playing audio in iOS, macOS and tvOS apps.

AudioPlayer AudioPlayer is a simple class for playing audio in iOS, macOS and tvOS apps.

Tom Baranes 259 Nov 18, 2022
Swift audio synthesis, processing, & analysis platform for iOS, macOS and tvOS

AudioKit AudioKit is an audio synthesis, processing, and analysis platform for iOS, macOS (including Catalyst), and tvOS. Installation To add AudioKit

AudioKit 8.7k Sep 30, 2021
A minimal chromatic tuner for iOS & macOS.

Zen Tuner A minimal chromatic tuner. Works well with any instrument, whether it's woodwinds, brass, strings or voice. Dark Light Features Beautiful, c

JP Simard 438 Nov 21, 2022
MuVis is a macOS, iOS, iPadOS app for real-time music visualization.

MuVis MuVis is an open-source multiplatform app (using SwiftUI, Swift, and Xcode) for music visualization. It renders informative (and musically usefu

Keith Bromley 7 Oct 10, 2022
An iOS and macOS audio visualization framework built upon Core Audio useful for anyone doing real-time, low-latency audio processing and visualizations.

A simple, intuitive audio framework for iOS and OSX. Deprecated EZAudio has recently been deprecated in favor of AudioKit. However, since some people

Syed Haris Ali 4.9k Nov 4, 2022
A sound fader for AVAudioPlayer written in Swift for iOS, tvOS and macOS.

Cephalopod, a sound fader for AvAudioPlayer written in Swift - iOS, tvOS and macOS This library can help fading sounds in and out with AvAudioPlayer.

Evgenii Neumerzhitckii 108 Jun 21, 2022
An original digital art work for macOS, iOS and web

WordClock WordClock versions of various vintages, for various platforms. https://www.simonheys.com/wordclock/ Downloads Currently avaiaible to downloa

Simon Heys 61 Nov 17, 2022
MuVis is a macOS, iOS, iPadOS app for real-time music visualization.

MuVis MuVis is an open-source multi-platform app (using SwiftUI, Swift, and Xcode) for music visualization. It renders informative (and musically usef

Keith Bromley 4 Oct 14, 2022
🎢 Control Apple Music right from your macOS menu bar.

About Music Bar is macOS application that places music controls right in your menu bar. Installation Download and unzip the latest release. Drag the a

Musa 197 Oct 29, 2022
macOS app that allows the control of Spotify and AppleMusic/iTunes music playback from the menu bar.

PlayStatus is a simple macOS app that allows the control of Spotify, Apple Music(macOS 10.15+) and iTunes including iTunes Radio/Beats1 playback from

Nikhil Bolar 109 Nov 11, 2022