Vapoursynth - Trimming Video and Audio and keeping it in sync

[ENunn]

i've been trying vapoursynth off and on after being an avisynth main for years. other than certain plugins, the main thing holding me back from switching is trimming. i know that it's a different scripting language, but i can't get it right at all without the audio going out of sync.

when i edit commercial compilations for youtube, i do all the editing in avisynth. it's really easy in avspmod. it involves multiple trims.

example avisynth script:

SetFilterMTMode("DEFAULT_MT_MODE", 2)
AVISource("F:\king5 nbc - news and days of our lives feb 2009 d-vhs.avi", pixel_type="yv12")
Trim(3839, 82862)
Trim(0, 4625) ++ Trim(14964, 79023)
Trim(0, 9281) ++ Trim(13820, 68685)
Trim(0, 15277) ++ Trim(25747, 64147)
Trim(0, 20674) ++ Trim(30971, 53678)
Trim(0, 25159) ++ Trim(31951, 43382)
Trim(0, 29645) ++ Trim(36206, 36591)
normalize(0.8912)
assumetff()
par=getparity()
SeparateFields().PointResize(width,height)
Deblock_QED()
AssumeFrameBased()
SeparateFields()
Merge(SelectEven(),SelectOdd())
par ? AssumeTFF() : AssumeBFF()
Weave()
QTGMC(Preset="slow", ezdenoise=0.0, tuning="dv-hd", sourcematch=3, MatchEnhance=1, NoiseProcess=1, NoiseRestore=1.0, Sigma=0.0 , sharpness=0, border=false)
Trim(0, 18563) ++ Trim(18571, 60063)
SuperRes(matrixin="rec709", 2, .43, 0, """nnedi3_rpow2(2, nns=4, cshift="Spline36Resize")""", matrixout="rec709")
prefetch(8)

is there not an easy way to do video and audio trimming at the same time in vapoursynth?

[_Al_]

Vapoursyth does not have build in system where trimming video would automatically trim audio as well. And vapoursynth audio trims actually audio samples, not frames. Avisynth does that underneath its hood, calculating approximate samples for frames.

Also vapoursynth does not have normalize audio filter, so you'd need to normalize for example using ffmpeg normalize filter while encoding, not within vapoursynth script.

But while ago I constructed a python script that does that, using that new class to trim video where audio is trimmed along as well. So that could be imported to your vapoursynth script. It even support slices as well (vapoursynth way of trimming using slices instead of classic trim function)
I posted it here, it is that first script on that page.

So your script might look something like this:

import vapoursynth as vs
from vapoursynth import core
from some_script import Clip

video = core.bs.VideoSource(r"F:\king5 nbc - news and days of our lives feb 2009 d-vhs.avi")
audio = core.bs.AudioSource(r"F:\king5 nbc - news and days of our lives feb 2009 d-vhs.avi")
video = video.resize.Bicubic(format=vs.YUV420P8) #YV12

clip = Clip(video, audio)

clip = clip.trim(3839, 82862)
clip = clip.trim(0, 4625) + clip.trim(14964, 79023)
clip = clip.trim(0, 9281) + clip.trim(13820, 68685)
clip = clip.trim(0, 15277) + clip.trim(25747, 64147)
clip = clip.trim(0, 20674) + clip.trim(30971, 53678)
clip = clip.trim(0, 25159) + clip.trim(31951, 43382)
clip = clip.trim(0, 29645) + clip.trim(36206, 36591)

#getting back video and audio from a Clip class, or you can still use clip.video or clip.audio instead
video = clip.video
audio = clip.audio

#here you have other video filters

video.set_output(0)
audio.set_output(1)

[_Al_]

note that script is old so in that class Clip you change loading audio as core.bs.AudioSource as oppose core.bas.Source, if loading audio as attribute audio (you are not though)

[_Al_]

never mind, I just tested some trimmings and audio tends to go out of sync, using that Clip class, so forget it, it would need some work ...

Correction, I just used short video and just multiplied it, like 30 times , same with audio and I got out of sync anyway, so something is not right, like delay of simple wav, not sure. I'm just not having a long video available atm., if someone can try a 45min long video or so and trim it, and find out, what it is going to do ...

[Selur]

if someone can try a 45min long video or so and trim it, and find out, what it is going to do ...

I don't usually use Vapoursynth for audio, so I can't really tell from experience, but I tested it with:

# Imports
import math
import sys
import os
import vapoursynth as vs
# using Vapoursynth R78
# getting Vapoursynth core
core = vs.core
# Limit frame cache to 48449MB
core.max_cache_size = 48449
# Import scripts folder
scriptPath = 'F:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# loading plugins
core.std.LoadPlugin(path="F:/Hybrid/64bit/Vapoursynth/Lib/site-packages/vapoursynth/plugins2/libbestsource.dll")
# Import scripts
import validate
# Source: 'g:\Test.mkv'
# clip current meta; color space: YUV420P8, bit depth: 8, resolution: 1920x1080, fps: 23.976, color matrix: 709, color primaries: Unspecific, color transfer: BT.709, yuv luminance scale: limited, scanorder: progressive, full height: true (Source)
# Loading 'g:\Test.mkv' using BestSource
clip = core.bs.VideoSource(source="G:/Test.mkv", cachepath="J:/tmp/Test_bestSource", track=0, hwdevice="opencl")
frame = clip.get_frame(0)
# setting color matrix to 709.
clip = core.std.SetFrameProps(clip, _Matrix=vs.MATRIX_BT709)
# setting color transfer (vs.TRANSFER_BT709), if it is not set.
if validate.transferIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Transfer=vs.TRANSFER_BT709)
# setting color primaries info (to vs.PRIMARIES_BT709), if it is not set.
if validate.primariesIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Primaries=vs.PRIMARIES_BT709)
# setting color range to TV (limited) range.
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange'
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_LIMITED})
# making sure frame rate is set to 23.976fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# making sure the detected scan type is set (detected: progressive)
video = core.std.SetFrameProps(clip=clip, _FieldBased=vs.FIELD_PROGRESSIVE) # scan type: progressive

# Format: ac-3
# Length: 2567.840
# Bit rate: 384
# Channels: 6
# Sample rate: 48000
# Raw Audio: true
# Codec: pcm
# Channel Mapping: 3/2/0.1
# Dialog Normalization: -27
# Video delay: 0
# New Video delay: 0
# Audio only delay: 0
audio = core.bs.AudioSource("G:/Test.mkv")


def trim_av(video: vs.VideoNode, audio: vs.AudioNode, first: int = 0, last: int = None) -> tuple:
    """
    Trims both video and audio simultaneously based on video frame numbers.
    Follows VapourSynth's core.std.Trim behavior (inclusive of 'last').
    """
    # 1. Trim the video node
    v_trimmed = core.std.Trim(video, first=first, last=last)
    
    # 2. Calculate frame rate and sample rate
    fps = video.fps.numerator / video.fps.denominator
    sample_rate = audio.sample_rate
    
    # 3. Calculate corresponding audio samples
    # First sample starts at the beginning of the 'first' frame
    audio_first = math.floor(first * sample_rate / fps)
    
    if last is not None:
        # Last sample ends at the end of the 'last' frame
        audio_last = math.floor((last + 1) * sample_rate / fps) - 1
    else:
        audio_last = None
        
    # 4. Trim the audio node
    a_trimmed = core.std.AudioTrim(audio, first=audio_first, last=audio_last)
    
    return v_trimmed, a_trimmed

# Cut List frame numbers based on progressive input
cut_list = [
    (12995, 13035),
    (13071, 13123),
    (13180, 13270),
    (13369, 13413),
    (13541, 14944),
    (17796, 18033),
    (19986, 20060),
    (31344, 31553),
    (60053, 60764)
]

# Do the cutting
video_cuts = []
audio_cuts = []

for start, end in cut_list:
    v_cut, a_cut = trim_av(video, audio, first=start, last=end)
    video_cuts.append(v_cut)
    audio_cuts.append(a_cut)

# combine cuts
video = core.std.Splice(video_cuts)
audio = core.std.AudioSplice(audio_cuts)

# adjusting output color from YUV420P8 to YUV420P10 for NVEncModel
video = core.resize.Bicubic(clip=video, format=vs.YUV420P10)
# set output frame rate to 23.976fps (progressive)
video = core.std.AssumeFPS(clip=video, fpsnum=24000, fpsden=1001)
# output

video.set_output(0)
audio.set_output(1)

and these calls for encoding:

"F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vapoursynth\vspipe.exe" -o 1 "C:\Users\Selur\Desktop\Vapoursynth_Scripts\test2.vpy" \\.\pipe\vsaudio -c wav
"F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vapoursynth\vspipe.exe" -o 0 "C:\Users\Selur\Desktop\Vapoursynth_Scripts\test2.vpy" - -c y4m | "F:\Hybrid\64bit\ffmpeg.exe" -i - -i \\.\pipe\vsaudio -c:v libx264 -crf 18 -c:a aac -b:a 192k "g:\Output\output.mkv"

encoding does seem to work fine:

Information: VideoSource track #0 using CPU decoding fallback
Input #0, yuv4mpegpipe, from 'fd:':
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: rawvideo (Y3[11][10] / 0xA0B3359), yuv420p10le(progressive), 1920x1080, 23.98 fps, 23.98 tbr, 23.98 tbn
Input #1, wav, from '\\.\pipe\vsaudio':
  Duration: N/A, bitrate: 9216 kb/s
  Stream #1:0: Audio: pcm_f32le ([3][0][0][0] / 0x0003), 48000 Hz, 5.1(side), flt, 9216 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
  Stream #1:0 -> #0:1 (pcm_f32le (native) -> aac (native))
[aac @ 00000237032da8c0] Using a PCE to encode channel layout "5.1(side)"
[aac @ 00000237032da8c0] 32 kb/s per channel at 48000 Hz: consider resampling the input to 32000 Hz or lower for better quality.
[libx264 @ 00000237032d6200] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 00000237032d6200] profile High 10, level 4.0, 4:2:0, 10-bit
[libx264 @ 00000237032d6200] 264 - core 165 r3223 0480cb0 - H.264/MPEG-4 AVC codec - Copyleft 2003-2025 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=34 lookahead_threads=5 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=23 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=18.0 qcomp=0.60 qpmin=0 qpmax=81 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, matroska, to 'g:\Output\output.mkv':
  Metadata:
    encoder         : Lavf63.5.101
  Stream #0:0: Video: h264 (H264 / 0x34363248), yuv420p10le(tv, progressive), 1920x1080, q=2-31, 23.98 fps, 1k tbn
    Metadata:
      encoder         : Lavc63.7.100 libx264
    Side data:
      CPB properties: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
  Stream #0:1: Audio: aac (LC) ([255][0][0][0] / 0x00FF), 48000 Hz, 5.1(side), fltp, 192 kb/s
    Metadata:
      encoder         : Lavc63.7.100 aac
Output 2869 frames in 20.50 seconds (139.96 fps)=00:01:54.66 bitrate=6602.4kbits/s speed=5.85x elapsed=0:00:19.59
[out#0/matroska @ 00000237032d0cc0] video:94185KiB audio:2806KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 0.063982%
frame= 2869 fps=141 q=-1.0 Lsize=   97053KiB time=00:01:59.66 bitrate=6644.3kbits/s speed=5.89x elapsed=0:00:20.30
[libx264 @ 00000237032d6200] frame I:44    Avg QP:26.21  size:174382
[libx264 @ 00000237032d6200] frame P:1203  Avg QP:28.54  size: 48936
[libx264 @ 00000237032d6200] frame B:1622  Avg QP:30.43  size: 18435
[libx264 @ 00000237032d6200] consecutive B-frames:  7.1% 51.7%  2.7% 38.5%
[libx264 @ 00000237032d6200] mb I  I16..4: 18.5% 52.7% 28.8%
[libx264 @ 00000237032d6200] mb P  I16..4:  6.3% 20.6%  1.5%  P16..4: 37.9% 11.0%  5.6%  0.0%  0.0%    skip:17.2%
[libx264 @ 00000237032d6200] mb B  I16..4:  1.1%  3.6%  0.2%  B16..8: 37.9%  3.9%  0.8%  direct: 6.0%  skip:46.4%  L0:42.5% L1:52.0% BI: 5.5%
[libx264 @ 00000237032d6200] 8x8 transform intra:70.5% inter:85.9%
[libx264 @ 00000237032d6200] coded y,uvDC,uvAC intra: 56.4% 56.4% 12.8% inter: 21.1% 19.3% 1.0%
[libx264 @ 00000237032d6200] i16 v,h,dc,p: 30% 29% 14% 27%
[libx264 @ 00000237032d6200] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 18% 19% 23%  6%  7%  7%  8%  6%  7%
[libx264 @ 00000237032d6200] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 20% 21% 12%  6%  9%  8%  9%  7%  7%
[libx264 @ 00000237032d6200] i8c dc,h,v,p: 56% 22% 18%  4%
[libx264 @ 00000237032d6200] Weighted P-Frames: Y:0.1% UV:0.1%
[libx264 @ 00000237032d6200] ref P L0: 82.9% 11.1%  6.0%
[libx264 @ 00000237032d6200] ref B L0: 91.5%  6.5%  2.0%
[libx264 @ 00000237032d6200] ref B L1: 98.8%  1.2%
[libx264 @ 00000237032d6200] kb/s:6447.85
[aac @ 00000237032da8c0] Qavg: 125.393  Tr: 0.6%  TNS(L): 1.8%  TNS(S): 5.1%  M/S: 0.0%  I/S: 9.6%  PNS: 6.4%

and the output seems to be sync.


Cu Selur

[_Al_]

thanks,
it all looks the same, except using math.floor() instead of int() while rounding up samples.
I have to try it again more thoroughly later.