High-performance and flexible video editing and effects framework, based on AVFoundation and Metal.

BearRuan

Last update: Dec 30, 2022

Overview

VideoLab

README 中文版本框架设计与实现介绍

High-performance and flexible video editing and effects framework, based on AVFoundation and Metal.

Framework design and implementation

Features

High-performance real-time video editing and exporting.
Highly free combination of video, image, audio.
Support audio pitch setting and volume adjustment.
Support CALayer vector animations, so complex text animations are supported.
Support keyframe animation.
Support After Effect-like pre-compose.
Support transitions.
Support custom effects. Such as LUT filter, zoom blur, etc.

The following are some GIFs of features(multiple layers, text animation, keyframe animation, pre compose, and transition)

Requirements

iOS 11.0+
Swift 5.0+

Installation

VideoLab is available through CocoaPods. Specify the following in your Podfile:

source 'https://github.com/CocoaPods/Specs.git'
platform :ios, '11.0'
use_frameworks!

target '
   
    '
    do
  pod 'VideoLab'
end

Usage

Basic Concept

RenderLayer

RenderLayer is the most basic unit in the VideoLab framework. A video, image, audio can be a RenderLayer, or even just an effect can be a RenderLayer. RenderLayer is more like the concept of the layer in After Effect.

RenderComposition

RenderComposition works as a composite, can set frame rate, canvas size, contains multiple RenderLayers, can set CALayer to support vector animations.

VideoLab

VideoLab can be considered as a lab where AVPlayerItem, AVAssetExportSession, AVAssetImageGenerator can be generated according to RenderComposition.

Basic Usage

// 1. Layer 1
var url = Bundle.main.url(forResource: "video1", withExtension: "MOV")
var asset = AVAsset(url: url!)
var source = AVAssetSource(asset: asset)
source.selectedTimeRange = CMTimeRange(start: CMTime.zero, duration: asset.duration)
var timeRange = source.selectedTimeRange
let renderLayer1 = RenderLayer(timeRange: timeRange, source: source)
    
// 1. Layer 2
url = Bundle.main.url(forResource: "video2", withExtension: "MOV")
asset = AVAsset(url: url!)
source = AVAssetSource(asset: asset)
source.selectedTimeRange = CMTimeRange(start: CMTime.zero, duration: asset.duration)
timeRange = source.selectedTimeRange
timeRange.start = CMTimeRangeGetEnd(renderLayer1.timeRange)
let renderLayer2 = RenderLayer(timeRange: timeRange, source: source)
    
// 2. Composition
let composition = RenderComposition()
composition.renderSize = CGSize(width: 1280, height: 720)
composition.layers = [renderLayer1, renderLayer2]

// 3. VideoLab
let videoLab = VideoLab(renderComposition: composition)

// 4. Make playerItem
let playerItem = videoLab.makePlayerItem()

Create RenderLayer
Create RenderComposition, set renderSize and layers
Create VideoLab with renderComposition
Make AVPlayerItem or AVAssetExportSession

More Advanced Usage

Transform

var center = CGPoint(x: 0.25, y: 0.25)
var transform = Transform(center: center, rotation: 0, scale: 0.5)
renderLayer1.transform = transform

Create Transform with center, rotation and scale
RenderLayer set transform

Audio Configuration

let audioConfiguration = AudioConfiguration()
let volumeRampTimeRange = CMTimeRange(start: CMTime.zero, duration: CMTime(seconds: 5, preferredTimescale: 600))
let volumeRamp1 = VolumeRamp(startVolume: 0.0, endVolume: 0.0, timeRange: volumeRampTimeRange)
audioConfiguration.volumeRamps = [volumeRamp1]
renderLayer2.audioConfiguration = audioConfiguration

Create AudioConfiguration
Create VolumeRamp with startVolume, endVolume and timeRange
AudioConfiguration set volumeRamps
RenderLayer set audioConfiguration

CALayer Animation

For exporting set your customized CALayer for RenderComposition

composition.animationLayer = <Your customized CALayer>

For playback add AVSynchronizedLayer to your view's layer, See more detail in Text Animation Demo.

Keyframe Animation

// 1. Keyframe animation
let keyTimes = [CMTime(seconds: 2, preferredTimescale: 600),
                CMTime(seconds: 4, preferredTimescale: 600),
                CMTime(seconds: 6, preferredTimescale: 600)]
let animation = KeyframeAnimation(keyPath: "blendOpacity",
                                  values: [1.0, 0.2, 1.0],
                                  keyTimes: keyTimes, timingFunctions: [.linear, .linear])
renderLayer1.animations = [animation]
    
var transform = Transform.identity
let animation1 = KeyframeAnimation(keyPath: "scale",
                                   values: [1.0, 1.3, 1.0],
                                   keyTimes: keyTimes, timingFunctions: [.quadraticEaseInOut, .quadraticEaseInOut])
let animation2 = KeyframeAnimation(keyPath: "rotation",
                                   values: [0, Float.pi / 2.0, 0],
                                   keyTimes: keyTimes, timingFunctions: [.quadraticEaseInOut, .quadraticEaseInOut])
transform.animations = [animation1, animation2]
renderLayer1.transform = transform

Create KeyframeAnimation with keyPath, values, keyTimes and timingFunctions
Set animations for a struct or class that implements the Animatable protocol (e.g. Transform struct, RenderLayer class)

RenderLayerGroup (After Effect-like pre-compose)

let layerGroup = RenderLayerGroup(timeRange: timeRange)
layerGroup.layers = [renderLayer1, renderLayer2]

Create RenderLayerGroup with timeRange
Set sub layers for layerGroup. See more detail in Layer Group Demo.

Transition

We don't have a transition layer, so instead, you can add a transform or operations to each RenderLayer to create a transition. See more detail in Transition Demo.

Custom Effects

// Filter
var filter = LookupFilter()
filter.addTexture(lutTextures[0], at: 0)
renderLayer.operations = [filter]

// Zoom Blur
var zoomblur = ZoomBlur()
animation = KeyframeAnimation(keyPath: "blurSize",
                              values: [0.0, 3.0],
                              keyTimes: keyTimes, timingFunctions: [.quarticEaseOut])
zoomblur.animations = [animation]
layerGroup1.operations = [zoomblur]

Create customize Operation inherited from BasicOperation. BasicOperation also conforms to the Animatable protocol
Set operations for RenderLayer.

TODO

Support Open GL render
Add speed adjustment for RenderLayer.
Provide a more convenient way to use transitions, possibly providing TransitionLayer.
Add log system.
Improve the demo and provide UI interactions.

Author

BearRuan, [email protected]

License

VideoLab is available under the MIT license. See the LICENSE file for more info.

Comments

Can't add an audio-only layer

I have a RenderComposition with a video RenderLayer. I want to mix an audio file with this video. I tried to create another RenderLayer with an AVAsset created from an AAC file and add it to my RenderComposition. After that, I could hear two audio streams from my video and audio files together, but the image disappeared. I got a black screen instead of my video. Also, the length of the resulting video equals the length of the AAC file, even though I set a shorter CMTimeRange.

So, how do I accomplish this? Is this supported by VideoLab? If not, will it be supported and how can I work around this problem now?

opened by devoln 4
simpleDemo()的视频合成有问题

你好，我从相册取三个视频出来，url类似这样的：file:///var/mobile/Media/DCIM/105APPLE/IMG_1999.MOV。 AVURLAsset *urlAsset = (AVURLAsset *)asset; NSURL *url = urlAsset.URL; let asset = AVAsset(url: url) let source = AVAssetSource(asset: asset); 后面走的代码跟simpleDemo()的是一模一样的，最终三个视频只播放了一个，VideoLab是不支持相册资源么？你们的演示示例只有加载项目里的资源。想请教一下，这个问题出现的原因？

opened by G-Singularity 0
Adding better support for Transitions

This is a general callout for input / advice / support on adding better Transitions to the Framework.

I've reviewed the demo project and see the limited options for Transitions available. I would like to expand on this and implement full support for Transitions but am having trouble finding the best approach for this. A lack of technical documentation also makes this more challenging.

My initial thought was to add a TransitionLayer which would hold any RenderLayers which you'd like the Transition applied on. But this felt too restrictive. Ex: What if you want different Transitions for each cut within your composition? I think this is a deadend idea but happy to hear other ways it might work at this level.

My next idea / approach was to add the Transition as a property directly to RenderLayer. The Transition should be applied / rendered when the current RenderLayer's content finishes playback. This felt like the best approach but curious if I'm overlooking any downsides.

The biggest issue is that even adding these classes, I haven't been able to figure out how to correctly apply the Filter effects to the composition. The LayerCompositor class seemed like the obvious place but, again without any technical docs, it's all remaining a bit cryptic for me at the moment.

I would love if I could get some input from @ruanjx here or anyone else who's used this extensively.

opened by joeypatino 0
How to use for adding captions? i.e multiple text layers on same composition

Hey, love this library!

How would you go about adding multiple text layers (for example captions) to the composition?

The composition can only take a single animationLayer value, so no place to add an array.

My thoughts are to create a base CALayer, then add all of the subsequent text animation layers to that layer, adjust their animation (from and duration values), and then just assign that base layer to composition.

It hasn't worked for me yet, but is that the right approach at all?

Thanks!

opened by royherma 0
Create video from uiimages

I know how to create video from some images using AVFoundation. But exporting the video using AVFoundation takes time. I want to make this export time faster. So I want to use metal for exporting a video from UIImage. Is there any way to achieve this using this framework?

opened by siam009 0
Example project keeps crashing on clicking on any table view cell with below error

Error:

Fatal error: Could not create render pipeline state for vertex:oneInputVertex, fragment:lookupFragment, error:Error Domain=CompilerError Code=2 "reading from a rendertarget is not supported" UserInfo={NSLocalizedDescription=reading from a rendertarget is not supported}

What is wrong?

opened by ElrB 3
How to mute asset audio and add custom audio?
Hi @ruanjx ,

First of all thank you for your effort, this is awesome framework that make work with AVFoundation easier.

[X] I read documents

[X] I read open or closed issues

Secondly I have a question about muting original audio of AVAsset and add custom audio to RenderComposition? I already tried to mute videoLayer with audioConfiguration like this

var audioConfiguration = AudioConfiguration() let volumeRampTimeRange = source.selectedTimeRange let volumeRamp1 = VolumeRamp(startVolume: 0.0, endVolume: 0.0, timeRange: volumeRampTimeRange) audioConfiguration.volumeRamps = [volumeRamp1] videoLayer.audioConfiguration = audioConfiguration

and then added new renderLayer for custom audio (mp3 file) named audioLayer and passed it to render composition like this

let composition = RenderComposition() composition.layers = [videoLayer, audioLayer]

videoLayer sound plays 1 second and then its being muted, audioLayer audio works well but no video on composition entirely it becomes black screen. Actually I couldn't see how to manage audio files in your documentation. Any helps will be appreciated .
opened by Coder-ACJHP 1