[Python] Extract clips from different videos, and merge them into one video, using ffmpeg

I want to extract multiple clips from different videos (in different encoding schemes/formats), and then merge them into one video.

The inputs are a list of files and precise timestamps of the clips:

[
("1.mp4", ["00:05:02.230", "00:05:05.480"]),
("4.mp4", ["00:03:25.456", "00:03:28.510"]),
("2.mp4", ["00:12:23.891", "00:12:32.642"]),
("2.mp4", ["00:12:44.236", "00:12:46.920"]),
("3.mp4", ["00:02:06.520", "00:02:11.324"]),
("1.mp4", ["00:06:23.783", "00:06:25.458"]),
("2.mp4", ["00:03:53.976", "00:03:56.853"]),
...
]


Option 1: Use ffmpeg -filter_complex and concat.

ffmpeg -y -i ./f19dbe55-b4cd-4cb5-a4f1-701b6864fea5.mp4 -filter_complex "[0:v]trim=start=1009.24:end=1022.53,setpts=PTS-STARTPTS[v0];[0:a]atrim=start=1009.24:end=1022.53,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a0];[0:v]trim=start=904.49:end=921.3,setpts=PTS-STARTPTS[v1];[0:a]atrim=start=904.49:end=921.3,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a1];[0:v]trim=start=1718.95:end=1737.19,setpts=PTS-STARTPTS[v2];[0:a]atrim=start=1718.95:end=1737.19,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a2];[0:v]trim=start=1692.64:end=1712.53,setpts=PTS-STARTPTS[v3];[0:a]atrim=start=1692.64:end=1712.53,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a3];[0:v]trim=start=322.34:end=330.13,setpts=PTS-STARTPTS[v4];[0:a]atrim=start=322.34:end=330.13,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a4];[0:v]trim=start=333.97:end=342.38,setpts=PTS-STARTPTS[v5];[0:a]atrim=start=333.97:end=342.38,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a5];[0:v]trim=start=352.33:end=359.74,setpts=PTS-STARTPTS[v6];[0:a]atrim=start=352.33:end=359.74,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a6];[0:v]trim=start=364.11:end=375.99,setpts=PTS-STARTPTS[v7];[0:a]atrim=start=364.11:end=375.99,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a7];[0:v]trim=start=395.56:end=407.88,setpts=PTS-STARTPTS[v8];[0:a]atrim=start=395.56:end=407.88,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a8];[0:v]trim=start=417.95:end=437.45,setpts=PTS-STARTPTS[v9];[0:a]atrim=start=417.95:end=437.45,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a9];[0:v]trim=start=443.81:end=452.68,setpts=PTS-STARTPTS[v10];[0:a]atrim=start=443.81:end=452.68,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a10];[0:v]trim=start=473.17:end=489.4,setpts=PTS-STARTPTS[v11];[0:a]atrim=start=473.17:end=489.4,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a11];[0:v]trim=start=506.35:end=510.53,setpts=PTS-STARTPTS[v12];[0:a]atrim=start=506.35:end=510.53,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a12];[0:v]trim=start=521.5360000000001:end=527.6610000000001,setpts=PTS-STARTPTS[v13];[0:a]atrim=start=521.5360000000001:end=527.6610000000001,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a13];[0:v]trim=start=542.7:end=558.58,setpts=PTS-STARTPTS[v14];[0:a]atrim=start=542.7:end=558.58,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a14];[0:v]trim=start=566.19:end=568.81,setpts=PTS-STARTPTS[v15];[0:a]atrim=start=566.19:end=568.81,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a15];[0:v]trim=start=588.01:end=602.13,setpts=PTS-STARTPTS[v16];[0:a]atrim=start=588.01:end=602.13,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a16];[0:v]trim=start=614.56:end=618.98,setpts=PTS-STARTPTS[v17];[0:a]atrim=start=614.56:end=618.98,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a17];[0:v]trim=start=725.62:end=737.95,setpts=PTS-STARTPTS[v18];[0:a]atrim=start=725.62:end=737.95,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a18];[0:v]trim=start=753.11:end=774.83,setpts=PTS-STARTPTS[v19];[0:a]atrim=start=753.11:end=774.83,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a19];[0:v]trim=start=778.85:end=782.14,setpts=PTS-STARTPTS[v20];[0:a]atrim=start=778.85:end=782.14,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a20];[0:v]trim=start=788.61:end=806.68,setpts=PTS-STARTPTS[v21];[0:a]atrim=start=788.61:end=806.68,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a21];[0:v]trim=start=835.81:end=856.58,setpts=PTS-STARTPTS[v22];[0:a]atrim=start=835.81:end=856.58,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a22];[0:v]trim=start=872.96:end=893.08,setpts=PTS-STARTPTS[v23];[0:a]atrim=start=872.96:end=893.08,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a23];[0:v]trim=start=937.9:end=951.32,setpts=PTS-STARTPTS[v24];[0:a]atrim=start=937.9:end=951.32,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a24];[0:v]trim=start=1045.13:end=1052.19,setpts=PTS-STARTPTS[v25];[0:a]atrim=start=1045.13:end=1052.19,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a25];[0:v]trim=start=1065.84:end=1101.57,setpts=PTS-STARTPTS[v26];[0:a]atrim=start=1065.84:end=1101.57,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a26];[0:v]trim=start=1137.44:end=1176.84,setpts=PTS-STARTPTS[v27];[0:a]atrim=start=1137.44:end=1176.84,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a27];[0:v]trim=start=1204.0:end=1212.93,setpts=PTS-STARTPTS[v28];[0:a]atrim=start=1204.0:end=1212.93,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a28];[0:v]trim=start=1272.17:end=1283.93,setpts=PTS-STARTPTS[v29];[0:a]atrim=start=1272.17:end=1283.93,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a29];[0:v]trim=start=1302.97:end=1331.42,setpts=PTS-STARTPTS[v30];[0:a]atrim=start=1302.97:end=1331.42,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a30];[0:v]trim=start=1389.77:end=1438.87,setpts=PTS-STARTPTS[v31];[0:a]atrim=start=1389.77:end=1438.87,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a31];[0:v]trim=start=1477.55:end=1491.53,setpts=PTS-STARTPTS[v32];[0:a]atrim=start=1477.55:end=1491.53,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a32];[0:v]trim=start=1503.42:end=1524.9,setpts=PTS-STARTPTS[v33];[0:a]atrim=start=1503.42:end=1524.9,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a33];[0:v]trim=start=1552.35:end=1608.12,setpts=PTS-STARTPTS[v34];[0:a]atrim=start=1552.35:end=1608.12,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a34];[0:v]trim=start=1616.93:end=1636.14,setpts=PTS-STARTPTS[v35];[0:a]atrim=start=1616.93:end=1636.14,asetpts=PTS-STARTPTS,afade=t=in:st=0:d=0.05[a35];[v0][a0][v1][a1][v2][a2][v3][a3][v4][a4][v5][a5][v6][a6][v7][a7][v8][a8][v9][a9][v10][a10][v11][a11][v12][a12][v13][a13][v14][a14][v15][a15][v16][a16][v17][a17][v18][a18][v19][a19][v20][a20][v21][a21][v22][a22][v23][a23][v24][a24][v25][a25][v26][a26][v27][a27][v28][a28][v29][a29][v30][a30][v31][a31][v32][a32][v33][a33][v34][a34][v35][a35]concat=n=36:v=1:a=1[outv][outa]" -map [outv] -map [outa] -c:v libx264 -c:a aac out.mp4


Note: afade=t=in:st=0:d=0.05 is used to mitigate the cramp video in the transition between clips.

Drawback: very slow, memory intensive (cause OOM)

Option 2: use ffmpeg -ss to extract, and then use -concat to merge.

ffmpeg -y -ss 00:00:10.550 -i .\remastered_video.mp4 -to 00:00:10.710 -c:v h264_qsv -global_quality 20 -c:a aac -af afade=t=in:st=0:d=0.05 ./o1.mp4


ffmpeg -y -f concat -safe 0 -i videos.txt -c copy out.mp4


Drawback: the audio and video are not progressing synchronously. They start synchronously but then diverge over time. It seems the tiny time difference inside each clip gets accumulated over time.

Trials we've made (but didn't help):

• "-vf setpts=PTS-STARTPTS", "-af afade=t=in:st=0:d=0.05,asetpts=PTS-STARTPTS", "-shortest", "-avoid_negative_ts make_zero", "-start_at_zero", ts format+"-bsf:v", "h264_mp4toannexb"
• Some suggests to put -ss after -i. But we don't want it because it will take a long time to position the frame (from the beginning of the video).

Option 3: Use Python (pyav) and seek.

• The intuition is simple: extract clips by timestamps, and then merge together.
• However, the complexity is beyond our capability. We will have to handle different frames (PTS/DTS), frame resolutions, audio sampling rates, from different video files.
• We've tried to convert all clips into the same resolution, audio sampling rate (48k), and format (mp4/h264). But the output video still has time mismatch (due to mis-positioned PTS).
• We're stuck at this point, and not sure if it's on the right track either.

Any advice will be greatly appreciated!

Continue reading...