A video composition framework build on top of AVFoundation. It's simple to use and easy to extend.

Hi Vito, I'm wondering what the process looks like for processing a filter now that filterProcessor was removed in version 0.2. The VideoCat demo used this when applying Lookup table filters but I'm wondering how I should go about doing this now it's gone. Do you have an example I could try? Thank you for your work and time!

I have a list of videos AVAssets. I want to merge them into 1 video with their corresponding audios.

But the videos sometimes are portrait, sometimes square, sometimes landscape (they may have infinite different width and heights). I want the videos to merge and stay aspectFit to the size frame of the first video.

Is this possible with Cabbage ? Im having a hard time understanding your ¨timeline¨ concept.

Hi vitoziv, I cannot play audio in a specific time range after added. This is my source code. I cannot play audio1.mp3 after set startTime and duration, if I don't set this, it plays well. Thank you.

    let video1TrackItem: TrackItem = {
        let url = Bundle.main.url(forResource: "video1", withExtension: "mp4")!
        let resource = AVAssetTrackResource(asset: AVAsset(url: url))
        let trackItem = TrackItem(resource: resource)
        trackItem.videoConfiguration.contentMode = .aspectFit
        return trackItem
    }()

   let mp3TrackItem: TrackItem = {
        let url = Bundle.main.url(forResource: "audio1", withExtension: "mp3")!
        let resource = AVAssetTrackResource(asset: AVAsset(url: url))
        let trackItem = TrackItem(resource: resource)
        trackItem.startTime = CMTime(value: 1, timescale: 1)
        trackItem.duration = CMTime(value: 10, timescale: 1)
        return trackItem
    }()

   let timeline = Timeline()
    timeline.videoChannel = [video1TrackItem]
    timeline.audioChannel = [video1TrackItem]
    timeline.audios = [mp3TrackItem]

你好，首先非常感谢vitoziv的无私分享！大致把Cabbage的中文说明和源代码梳理了一下，但对Cabbage的设计思路和使用还有一些不太清楚的地方希望请教讨论一下。目前，项目需要实现给视频添加字幕和动画贴图的功能，我不太清楚是通过timeline上的overlays实现，还是使用CALayer去实现。但如果使用CALayer但的话，预览和渲染需要维护两套业务逻辑感觉比较麻烦。想请教一下vitoziv，Cabbage对这方面需求的支持是怎样考虑的以及你的建议？

DEBUG 后发现 VideoCompositionInstruction 的方法 open func apply(request: AVAsynchronousVideoCompositionRequest) -> CIImage? 在 16 秒后 request 的 sourceTrackIDs 为 nil，调用 sourceFrame(byTrackID trackID: CMPersistentTrackID) 也返回 nil。

求助问题原因所在。

下面是我通过 images 创建 AVPlayerItem 的代码：

func makePlayerItemFromImages(_ images: [UIImage]) {
        let ciImages = images.compactMap { $0.cgImage }.map { CIImage.init(cgImage: $0) }
        
        let resources = ciImages.map { ImageResource(image: $0) }

        for item in resources {
            let timeRange = CMTimeRange(start: kCMTimeZero, duration: CMTimeMake(100, 100))
            item.duration = timeRange.duration
            item.selectedTimeRange = timeRange
        }
        
        let items = resources.map { TrackItem(resource: $0) }
        
        var timeLineDuration = kCMTimeZero
        items.forEach {
            $0.configuration.videoConfiguration.baseContentMode = .aspectFit
            
            let timeRange = CMTimeRange(start: timeLineDuration, duration: $0.resource.duration)
            $0.configuration.timelineTimeRange = timeRange
            timeLineDuration = CMTimeAdd(timeLineDuration, timeRange.duration)

        }
        
        let timeline = Timeline()
        timeline.videoChannel = items
        
        let compositionGenerator = CompositionGenerator(timeline: timeline)
        compositionGenerator.renderSize = CGSize(width: 480, height: 480)
        let playerItem = compositionGenerator.buildPlayerItem()
        
        let controller = AVPlayerViewController.init()
        controller.player = AVPlayer.init(playerItem: playerItem)
        controller.view.backgroundColor = UIColor.white
        present(controller, animated: true, completion: nil)
    }

Context: TrackConfiguration's transform property lets developers applying custom transformations on TrackItem while TrackConfiguration's contentMode is responsible for how the TrackItem's content is displayed on screen (aspectFit/aspectFill/custom).

Issue: The CGAffineTransform stored in TrackConfiguration's transform property gets applied only after the contentMode transformations, so if a user transformation (e.g. rotation by 90 degrees) is set then .aspectFit and .aspectFill content modes won't work as expected:

https://github.com/VideoFlint/Cabbage/blob/e65b80fc04c026d14ca0a3a3ed0567d597233f86/Cabbage/Sources/Track/Configuration/TrackConfiguration.swift#L61-L84

Proposed solution: Move the user transform implementation just before the contentMode transformations.

@vitoziv what do you think? I would expect the content modes to work even if the TrackItem is e.g. rotated via the transform property, but I also don't know how you specify the purpose of transform property.

I'll create a pull request for this, please let me know what's your take on this proposal.

I make a simple demo to add text overlay on video as your suggestion:

(About text overlay, I suggest you add text's image to Timeline.passingThroughVideoCompositionProvider)

It works but I got the issue: the text overlay always had a black background.

I understand you used 1 black video to render image as video frame but in this case how can I delete this black background

P/s: How can I custom the video resource to create a track item from different resource type as a GIF file?

我参考VideoCat的Demo从相册中选择了图片

let resource = PHAssetImageResource.init(asset: asset, duration: CMTime.init(value: 3000, 600))
            guard let task = resource.prepare(progressHandler: progressHandler, completion: { (status, error) in
                if status == .avaliable {
                    resource.selectedTimeRange = CMTimeRange.init(start: CMTime.zero, end: resource.duration)
                    let trackItem: TrackItem = TrackItem(resource: resource)
                    let transition = CrossDissolveTransition()
                    transition.duration = CMTime(value: 900, timescale: 600)
                    trackItem.videoTransition = transition
                    let audioTransition = FadeInOutAudioTransition(duration: CMTime(value: 66150, timescale: 44100))
                    trackItem.audioTransition = audioTransition
                    if resource.isKind(of: ImageResource.self) {
                        trackItem.videoConfiguration.contentMode = .custom
                    } else {
                        trackItem.videoConfiguration.contentMode = .aspectFill
                    }
                    complete(trackItem)
                } else {
                    Log.error("image track status is \(status), check prepare function, error: \(error?.localizedDescription ?? "")")
                    complete(nil)
                }
            })

然后检查了，图片的选择，在 resource.prepare中，Image是存在的，status也是 available的。

public func reloadPlayerItem(_ items: [TrackItem]) -> AVPlayerItem {
        let timeLine = TimeLineManager.current.timeline
        let width = UIScreen.main.bounds.width * UIScreen.main.scale
        let height = width
        timeLine.videoChannel = items
        timeLine.audioChannel = items
        do {
            try Timeline.reloadVideoStartTime(providers: timeLine.videoChannel)
        } catch {
            assert(false, error.localizedDescription)
        }
        timeLine.renderSize = CGSize.init(width: width, height: height)
        let compositonGenerator = CompositionGenerator.init(timeline: timeLine)
        return compositonGenerator.buildPlayerItem()
    }

这是buildItem的方法，视屏和livephoto都没有问题，只有图片无法正常播放，在播放器的时长也不对。

master分支。 xcode 10.2.2 swift 5.0

没有找到关于overlayers轨道的demo。研究了下你的源码，不知道理解的对不对，请教一下。如果要使用overlays轨道，实现当前track中x:50,y,50的坐标中放一个size为50*50的overlay。参考了了ImageOverlayItem的实现方法，可以通过trackItem.configuration.videoConfiguration.transform配置来传相应的transform实现，这样可以实现，但使用起来不是很方便。

我的理解，overlays: [VideoProvider] ,overlays不只是VideoProvider协议，更合适的是进一步封装了frame的协议。

当我通过以下方式去填充音频轨道时，我遇到了以下问题： 1、对resouorce实例不进行深拷贝，所有音频资源的selectTimeRange属性总是会与最后一次循环的设置一致。 2、对resource实例进行深拷贝，音频资源的scaledDuration属性，总是为音频长度。 代码如下：

private func caculateMusicTrack(resource: AVAssetTrackResource, duration: CMTime) -> [TrackItem] {
        Log.out(">> Total VIDEO DURATION: \(duration.seconds)")
        Log.out(">> Music File Duration: \(resource.duration.seconds)")
        let numOfLoops = (duration.seconds - currentMusicStartOffset) / resource.duration.seconds
        let numOfLoopsRoundedUp = numOfLoops.rounded(.up)
        var sumPartsTotals = CMTime.zero
        var endS = CMTime.zero
        var result: [TrackItem] = []
        //Audio Trim
        for i in 0..<Int(numOfLoopsRoundedUp) {
            let mResource = resource.copy() as! AVAssetTrackResource
            Log.out(mResource)
            guard let musicAsset = mResource.asset else {
                continue
            }
            //Audio Trim
            let start = CMTimeMake(value: Int64(0.0 * 600), timescale: 600)
            if i == Int(numOfLoopsRoundedUp) - 1 { //is the last chunk of audio
                let lastChunkTimeFrac = numOfLoops.truncatingRemainder(dividingBy: 1) // ex 1.5 will give 0.5
                let lastChunkTimeSecs = musicAsset.duration.seconds * lastChunkTimeFrac //music from 0 to this value
                endS = CMTimeMake(value: Int64((lastChunkTimeSecs-0.05) * 600), timescale: 600)
            } else {
                endS = CMTimeMake(value: Int64(musicAsset.duration.seconds * 600), timescale: 600)
            }
            let timeOffset = CMTime.init(seconds: currentMusicStartOffset, preferredTimescale: 600)
            if i == 0 {
                let startTime = currentMusicStartOffset < 0 ? start - timeOffset : start
                mResource.selectedTimeRange = CMTimeRange.init(start: startTime, end: endS)
            } else {
                mResource.selectedTimeRange = CMTimeRange(start:start , end: endS)
            }
            mResource.selectedTimeRange = CMTimeRange(start:start , end: endS)
            Log.out("selectedStart:\(mResource.selectedTimeRange.start.seconds) - totalPart:\(mResource.selectedTimeRange.end.seconds)")
            let partMyTrackItem = TrackItem(resource: mResource)
            let zeroOffsetTime = CMTimeMultiply(musicAsset.duration, multiplier: Int32(i))
            if i == 0 {
                 partMyTrackItem.startTime = zeroOffsetTime + (currentMusicStartOffset < 0 ? CMTime.zero : timeOffset)
            } else {
                partMyTrackItem.startTime = zeroOffsetTime + timeOffset
            }
            partMyTrackItem.startTime = zeroOffsetTime
            Log.out("start:\(partMyTrackItem.startTime.seconds) - totalPart:\(mResource.scaledDuration.seconds)")
            sumPartsTotals = CMTimeAdd(sumPartsTotals, mResource.scaledDuration)
            result.append(partMyTrackItem)
        }
        return result
    }

Hi im back :)

So previously I successfully implemented your suggestions to merge videos with their corresponding audio tracks. Now Im wondering if it's possible 2 things.

1- Is it possible to define separate audio for each video with specific range of that audio ? example:

let tLine = Timeline()
var vChannel = [TrackItem]()
var aChannel = [TrackItem]()
//VIDEO Tracks
... trackVideoItem1, trackVideoItem2, trackVideoItem3...
//AUDIO Tracks
        let musicUrl = Bundle.main.url(forResource: "HumansWater", withExtension: "MP3")!
        let musicAsset = AVAsset(url: musicUrl)
        let resourceA = AVAssetTrackResource(asset: musicAsset)
        let trackAudioItem1 = TrackItem(resource: resourceA)
... same for trackAudioItem2, trackAudioItem3...
But how do I specify the start-end and duration of those tracks.  ??

tLine.videoChannel = [trackVideoItem1,trackVideoItem2,trackVideoItem3]
tLine.audioChannel = [trackAudioItem1, trackAudioItem2, trackAudioItem3]
try! Timeline.reloadVideoStartTime(providers: tLine.videoChannel)

Currently the above creates an unreadable video.

2- The other question is, If it's possible to define 1 music track for the entire video composition. example:

let tLine = Timeline()
var vChannel = [TrackItem]()
var aChannel = [TrackItem]()
//VIDEO Tracks
... trackVideoItem1, trackVideoItem2, trackVideoItem3...
//AUDIO Track for everything
        let musicUrl = Bundle.main.url(forResource: "HumansWater", withExtension: "MP3")!
        let musicAsset = AVAsset(url: musicUrl)
        let resourceA = AVAssetTrackResource(asset: musicAsset)
        let trackMusicItem = TrackItem(resource: resourceA)
But how do I specify the start-end of the audio (trimming the audio)

tLine.videoChannel = [trackVideoItem1,trackVideoItem2,trackVideoItem3]
tLine.audioChannel = [trackMusicItem]
try! Timeline.reloadVideoStartTime(providers: tLine.videoChannel)

Is it possible to set 1 audio for everything? and what happens if the audioTrack is shorter than the entire video composition or the video composition is shorter than the audioTrack, would it repeat the audioTrack ?

Thanks in advance :) :)

I noticed that while exporting video using AVAssetExportSession, the result appears to be darker. Attached below are the original and exported screenshot of the videos.

Is this expected? Any suggestions/workarounds to preserve as much of the visual quality of the original videos in the exported?

Original:

Exported:

The code:

        let timeline = Timeline()
        
        var videoChannels = [TrackItem]()
        var audioChannels = [TrackItem]()
        
        var currentTime = CMTime.zero
        for asset in self.assets {
            let resource = AVAssetTrackResource(asset: asset)

            let trackItem = TrackItem(resource: resource)
            trackItem.videoConfiguration.contentMode = .aspectFit
            trackItem.startTime = currentTime
            currentTime = CMTimeAdd(currentTime, asset.duration)
            
            videoChannels.append(trackItem)
            audioChannels.append(trackItem)
        }
        
        timeline.videoChannel = videoChannels
        timeline.audioChannel = audioChannels
        timeline.renderSize = CGSize(width: 1080, height: 1920)

        let compositionGenerator = CompositionGenerator(timeline: timeline)
        let exportSession = compositionGenerator.buildExportSession(presetName: AVAssetExportPresetHighestQuality)
        exportSession?.outputFileType = .mov

        let outputURL = URL(fileURLWithPath: NSTemporaryDirectory().appending("test.mov"))
        exportSession?.outputURL = outputURL
        exportSession?.exportAsynchronously {
            if let error = exportSession?.error {
                print("Failed to export: \(error)")
            } else {
                print("Movie file generated: \(outputURL)")
            }
        }

The Cabbage framework looks so nice. I have a question, Is Cabbage support multiple layers blend mode? Some basic blend modes like overlay, multiply, screen.

            let voiceResource = AVAssetTrackResource(asset: asset)
            
            if videoTimelineView.duration <= ((voiceOvers[currentVoiceOver]?.startTime ?? 0.0) + asset.duration.seconds) {
                voiceResource.selectedTimeRange = CMTimeRange(start: .zero, duration: CMTime(seconds: asset.duration.seconds, preferredTimeScale: 10))
            }
            item.identifier = voiceOvers[currentVoiceOver]?.name ?? ""
            item.startTime = CMTime(seconds: voiceOvers[currentVoiceOver]?.startTime ?? 0.0, preferredTimescale: 10)

Hello,

just wanted to ask a question. Is it possible to create multiple timelines and create single composition + videoComposition + audioMix? I'm trying to lay two videos on top of each other (e.g. like facetime calls). However every timeline consists of multiple subsequent video clips.

So it would look like this with two timelines and every letter representing a video file (mp4): aaaaaaabbbccccc dddeeefffffffffffff

Thank you

When i try to scale video speed it up or slow it the voice is gone but the videos works

let resource = AVAssetTrackResource(asset: asset)
                    resource.scaledDuration = CMTime(seconds: 5.0, preferredTimeScale: 600)
                    resource.duration = CMTime(seconds: 5.0, preferredTimeScale: 600)
                    let trackItem = TrackItem(resource: resource)
                    trackItem.videoConfiguration.contentMode = .aspectFill