ios - How do you modify an AVFoundation video copy (multiple splits and segment deletions)?

Question

I've been looking through Apple's sample code Building a Feature-Rich App for Sports Analysis and its associated WWDC video to learn to reason about AVFoundation and VNDetectTrajectoriesRequest. My goal is to allow the user to import videos (this part I have working, the user sees a UIDocumentBrowserViewController, picks a video file, and then a copy is made), but I only want segments of the original video copied where trajectories are detected from a ball moving.

I've tried as best I can to grasp the two parts, at the very least finding where the video copy is made and where the trajectory request is made.

The full video copy happens in CameraViewController.swift (I'm starting with just imported video for now and not reading live from the device's video camera), line 160:

func startReadingAsset(_ asset: AVAsset) {
    videoRenderView = VideoRenderView(frame: view.bounds)
    setupVideoOutputView(videoRenderView)
    
    // Setup display link
    let displayLink = CADisplayLink(target: self, selector: #selector(handleDisplayLink(_:)))
    displayLink.preferredFramesPerSecond = 0 // Use display's rate
    displayLink.isPaused = true
    displayLink.add(to: RunLoop.current, forMode: .default)
    
    guard let track = asset.tracks(withMediaType: .video).first else {
        AppError.display(AppError.videoReadingError(reason: "No video tracks found in AVAsset."), inViewController: self)
        return
    }
    
    let playerItem = AVPlayerItem(asset: asset)
    let player = AVPlayer(playerItem: playerItem)
    let settings = [
        String(kCVPixelBufferPixelFormatTypeKey): kCVPixelFormatType_420YpCbCr8BiPlanarFullRange
    ]
    let output = AVPlayerItemVideoOutput(pixelBufferAttributes: settings)
    playerItem.add(output)
    player.actionAtItemEnd = .pause
    player.play()

    self.displayLink = displayLink
    self.playerItemOutput = output
    self.videoRenderView.player = player

    let affineTransform = track.preferredTransform.inverted()
    let angleInDegrees = atan2(affineTransform.b, affineTransform.a) * CGFloat(180) / CGFloat.pi
    var orientation: UInt32 = 1
    switch angleInDegrees {
    case 0:
        orientation = 1 // Recording button is on the right
    case 180, -180:
        orientation = 3 // abs(180) degree rotation recording button is on the right
    case 90:
        orientation = 8 // 90 degree CW rotation recording button is on the top
    case -90:
        orientation = 6 // 90 degree CCW rotation recording button is on the bottom
    default:
        orientation = 1
    }
    videoFileBufferOrientation = CGImagePropertyOrientation(rawValue: orientation)!
    videoFileFrameDuration = track.minFrameDuration
    displayLink.isPaused = false
}

@objc
private func handleDisplayLink(_ displayLink: CADisplayLink) {
    guard let output = playerItemOutput else {
        return
    }
    
    videoFileReadingQueue.async {
        let nextTimeStamp = displayLink.timestamp + displayLink.duration
        let itemTime = output.itemTime(forHostTime: nextTimeStamp)
        guard output.hasNewPixelBuffer(forItemTime: itemTime) else {
            return
        }
        guard let pixelBuffer = output.copyPixelBuffer(forItemTime: itemTime, itemTimeForDisplay: nil) else {
            return
        }
        // Create sample buffer from pixel buffer
        var sampleBuffer: CMSampleBuffer?
        var formatDescription: CMVideoFormatDescription?
        CMVideoFormatDescriptionCreateForImageBuffer(allocator: nil, imageBuffer: pixelBuffer, formatDescriptionOut: &formatDescription)
        let duration = self.videoFileFrameDuration
        var timingInfo = CMSampleTimingInfo(duration: duration, presentationTimeStamp: itemTime, decodeTimeStamp: itemTime)
        CMSampleBufferCreateForImageBuffer(allocator: nil,
                                           imageBuffer: pixelBuffer,
                                           dataReady: true,
                                           makeDataReadyCallback: nil,
                                           refcon: nil,
                                           formatDescription: formatDescription!,
                                           sampleTiming: &timingInfo,
                                           sampleBufferOut: &sampleBuffer)
        if let sampleBuffer = sampleBuffer {
            self.outputDelegate?.cameraViewController(self, didReceiveBuffer: sampleBuffer, orientation: self.videoFileBufferOrientation)
            DispatchQueue.main.async {
                let stateMachine = self.gameManager.stateMachine
                if stateMachine.currentState is GameManager.SetupCameraState {
                    // Once we received first buffer we are ready to proceed to the next state
                    stateMachine.enter(GameManager.DetectingBoardState.self)
                }
            }
        }
    }
}

Line 139 self.outputDelegate?.cameraViewController(self, didReceiveBuffer: sampleBuffer, orientation: self.videoFileBufferOrientation) is where the video sample buffer is passed to the Vision framework subsystem for analyzing trajectories, the second part. This delegate callback is implemented in GameViewController.swift on line 335:

        // Perform the trajectory request in a separate dispatch queue.
        trajectoryQueue.async {
            do {
                try visionHandler.perform([self.detectTrajectoryRequest])
                if let results = self.detectTrajectoryRequest.results {
                    DispatchQueue.main.async {
                        self.processTrajectoryObservations(controller, results)
                    }
                }
            } catch {
                AppError.display(error, inViewController: self)
            }
        }

Trajectories found are drawn over the video in self.processTrajectoryObservations(controller, results).

Where I'm stuck now is modifying this so that instead of drawing the trajectories, the new video only copies parts of the original video to it where trajectories were detected in the frame.

question from:https://stackoverflow.com/questions/65895134/how-do-you-modify-an-avfoundation-video-copy-multiple-splits-and-segment-deleti

Answer 1

If you know the number of seconds into the current video, and the duration, you can transcode/export parts of the content original video using AVAssetExportSession.

Here's a piece of code I used a few years back to do something similar. It's swift 3 so the syntax might be slightly different than now.

let asset = AVURLAsset(url: originalFileURL)
let exportSession = AVAssetExportSession(asset: asset, presetName: AVAssetExportPresetHighestQuality)!

exportSession.outputURL = someOutputURLYouWant
exportSession.outputFileType = AVFileTypeMPEG4 // can choose other types than mp4 if you want

let start = CMTimeMakeWithSeconds(secondsIntoVideoFloat, 600)
let duration = CMTimeMakeWithSeconds(numberOfSecondsFloat, 600)
exportSession.timeRange = CMTimeRange(start, duration)

exportSession.exportAsynchronously {
  switch exportSession.status {
    // handle completed, failed, cancelled states.
  }
}