Matching the frames of lower FPS video

[Mephesto]

I have two videos, one 23.976 fps without a logo, 25 fps with one and higher resolution. I'm matching the logo area from the other video to the new one but they aren't matching because I can't get the frame orientation to match between the two.

Is there a tool that can do this automatically?

[poisondeathray]

Are you only interested in the logo area? I'm assuming the logo area is clean on one version so you want to cover it up on the other ?

Couldn't you just resize one to match the other ? Why would you need to do this automatically? (It should only need to be done once) , unless the position or characteristics change , or the frames don't match (maybe different cuts of the video)?

Maybe a screenshot would help describe it better

[Mephesto]

Are you only interested in the logo area? I'm assuming the logo area is clean on one version so you want to cover it up on the other ?

Couldn't you just resize one to match the other ? Why would you need to do this automatically? (It should only need to be done once) , unless the position or characteristics change , or the frames don't match (maybe different cuts of the video)?

Maybe a screenshot would help describe it better

You misunderstood. It's a temporal, not spatial problem. The framerates are different so even downsampling the 25 fps video to 23.976 to match won't get the orientation correct. Frame 50 will really be 49 or 51 etc.

The videos do have some spatial differences like the one without logo being half the resolution, having brightness differences but that's not too important. I fixed that part best I could. The temporal part is what I can't.

It would be preferable actually if I could keep the higher fps but the logo-removed area would not match if I upscaled the fps so...

[Cornucopia]

Chg FR from 23.976 to 25 (that's how it's usually done - speedup). If timing matches (it should if it was done right originally), you just resize and then you can swap/extract, etc.

Scott

[poisondeathray]

If it's just a simple slow down or speedup, but same framecount, then use AssumeFPS() to match the other. If it's some other type of conversion you need to provide more details

[Mephesto]

You're not getting it. Let me illustrate:

First column: lower resolution, blown up, no logo, 23.976fps.
Second column: higher res, with logo, fps 25 > 23.976.

1 1
2 2
3 3
4 4
5 5
6 7
7 7
8 8
9 9
10 11
11 12
12 12
13 13
14 15
15 16
16 17
17 17
18 19
19 20
20 21
21 22
22 23
23 (a dropped frame inbetween 23 and 24, dropped due to 25 > 23.976)
24 24
25 25
26 26
27 27
28 28
29 29
30 31

I need all frames to match otherwise the logo area from the first video I can't paste on the second without mismatch artifacts because logo area from frame 30 is pasted on 31 while there's movement/camera shake. But the test that got me to illustrate this example just showed me that downsampling the 25fps vid to 23.976 to match the frames is not an option because frame23 from logoless video will have no matching frame to match to on the second vid because it has been dropped.

[poisondeathray]

You're missing frame 10 in the 2nd column , is that a dropped frame as well ? or did you goof ?

Are you sure frames are dropped? Or did you drop them from the conversion? Because a 23.976 original source is usually sped up to 25p (same number of frames) . Then YOU do the conversion to 25 => 23.976 with AssumeFPS(24000,1001) . This will match up EXACTLY, because it's the precise reversal . Or you speed up the 23.976 version to 25p with AssumeFPS(25) - it depends on what you want to do with the audio

Or was there some other conversion , like blends, or inserted duplicates ? There are many different ways for frame rate conversions, you should post a sample or describe it better

Start with what is the framecount and duration of the videos ?

[Mephesto]

You're missing frame 10 in the 2nd column , is that a dropped frame as well ? or did you goof ?

Yes, frame 10 from the lower-res video is the same as frame 11 in the higher-res video.

Are you sure frames are dropped? Or did you drop them from the conversion?

I dropped them in the conversion. Since I'm pasting the top-right corner of the first video to cover the logo of the second, the frames all have to match. I can't paste from frame 10 to frame 11 because the camera moves.

Because a 23.976 original source is usually sped up to 25p (same number of frames) .

No, the 25fps video is closer to the source than the 23.976fps MPG vid. There are no duplicate frames in the higher-quality 25fps vid, if only the attention-whoring idiot didn't paste a huge logo of his name on it I wouldn't have a lower-quality MPG version of it on my computer at all.

Then YOU do the conversion to 25 => 23.976 with AssumeFPS(24000,1001) . This will match up EXACTLY, because it's the precise reversal . Or you speed up the 23.976 version to 25p with AssumeFPS(25) - it depends on what you want to do with the audio

Yeah that won't work, I need to match the frame orientation 100%, remember? The 23.976 fps video has been downsampled from the original. AssumeFPS will change length.

Or was there some other conversion , like blends, or inserted duplicates ? There are many different ways for frame rate conversions, you should post a sample or describe it better

I used ChangeFPS, don't know what the author of the MPG vid did. No blends. I can't post a sample because this is a rare vintage vid that I was requested to find and preserve. Those two versions was all I could find on the net. I'm trying to get rid of that gay logo.

Start with what is the framecount and duration of the videos ?

31026 (HQ logo'd vid)
29741 (LQ logoless MPG vid)

Both same duration.

[poisondeathray]

Start with what is the framecount and duration of the videos ?

31026 (HQ logo'd vid)
29741 (LQ logoless MPG vid)

Both same duration.

This is the crux of the problem.

Assuming you want to keep the HQ version, 25FPS, but just remove the logo. You don't have enough "good" frames to "cover" up the logo . Even if you matched every LQ frame manually, you'd still be missing 1285 frames

I'm assuming you want to keep the HQ version, since you say it has the original frames and the 23.976 version is missing frames (it's possible that it's an original 25FPS source, eg. a BBC production)

[Mephesto]

The LQ version is half the resolution of the HQ version. I don't need to keep 25FPS, it's not a huge difference, I just want the frame orientation to be correct so I can properly remove the logo.

[poisondeathray]

The LQ version is half the resolution of the HQ version. I don't care if I have to deal with a lower framerate, it's not a huge difference, I just want the frame orientation to be correct so I can properly remove the logo.

The framerate isn't the problem; the framecount is the problem. Even if you have every frame matched up, there will be 1285 frames that still have the logo

If it was originally 25FPS production, and they are the same video duration, then the 23.976 version is missing frames and will playback "jerky" . It was a decimation conversion, not a simple slowdown .

So I take it you think that it's acceptable to drop frames and have jerky playback and still want it matched up ?

[Mephesto]

Yes and I can downsample the framerate to get the same count but the orientation of the frames aren't perfect, like I illustrated. Frames of the HQ video get dropped that shouldn't.

That's why I asked if there was an automated way to force the frames of both videos to match.

[poisondeathray]

Ok , now it's clear what you want - you don't mind wrecking the HQ video by dropping frames just to match frames and get rid of the logo LOL . IMO jerky playback is way worse than some logo. Or even other logo removal methods would be better

The language was confusing at first, because "orientation" of the frames usually refers to spatial orientation x,y coordinates (especially when prefaced with "logo" ) , not "temporal" orientation. Normally people just say my "frames don't match up" LOL

To answer the question, it depends on what the pattern is. If the cadence is symmetrical and repeating (e.g. every 25th frame is dropped in the HQ version to generate the LQ version) , you can use SelectRangeEvery() or similar filters (one of Stickboy's functions)
http://avisynth.org/mediawiki/SelectEvery

[poisondeathray]

If you want to count frames, one way is to use showframenumber() to overlay the current frame number (Starting from zero).

You want to determine which frames are missing from the LQ version compared to the HQ version, and establish if a pattern exists

A helpful way is to use avspmod and load each video in different tabs (you can use the number keys to toggle tabs)

e.g. is it 25, 50, 75, .... in the HQ version that are missing ?

[Mephesto]

Ok , now it's clear what you want - you don't mind wrecking the HQ video by dropping frames just to match frames and get rid of the logo LOL . IMO jerky playback is way worse than some logo. Or even other logo removal methods would be better

25 to 24 fps, WHAT A HUGE DIFFERENCE.

The language was confusing at first, because "orientation" of the frames usually refers to spatial orientation x,y coordinates (especially when prefaced with "logo" ) , not "temporal" orientation. Normally people just say my "frames don't match up" LOL

I did say exactly that. I said I had two videos which I wanted to paste over the logo area from and thus needed the frames to match up.

To answer the question, it depends on what the pattern is. If the cadence is symmetrical and repeating (e.g. every 25th frame is dropped in the HQ version to generate the LQ version) , you can use SelectRangeEvery() or similar filters (one of Stickboy's functions)
http://avisynth.org/mediawiki/SelectEvery

I posted the frames up to 30. Do you see a pattern? I don't.

[poisondeathray]

Ok , now it's clear what you want - you don't mind wrecking the HQ video by dropping frames just to match frames and get rid of the logo LOL . IMO jerky playback is way worse than some logo. Or even other logo removal methods would be better

25 to 24 fps, WHAT A HUGE DIFFERENCE.

It is. A jerk every second.

To answer the question, it depends on what the pattern is. If the cadence is symmetrical and repeating (e.g. every 25th frame is dropped in the HQ version to generate the LQ version) , you can use SelectRangeEvery() or similar filters (one of Stickboy's functions)
http://avisynth.org/mediawiki/SelectEvery

I posted the frames up to 30. Do you see a pattern? I don't.

Maybe because you did the numbering wrong? Did you see that you are missing frame 10 in the 2nd column ? And you can't determine a "pattern" by looking at 1 cycle... at minimum you need 2 cycles

[Cornucopia]

...It is. A jerk every second...

Is that like a sucker every minute?

Scott

[Mephesto]

It is. A jerk every second.

It's not really noticeable. I'll be deshaking later anyway. On top of that, I can interpolate it back to 25fps later.

Maybe because you did the numbering wrong? Did you see that you are missing frame 10 in the 2nd column ? And you can't determine a "pattern" by looking at 1 cycle... at minimum you need 2 cycles

We r not communicate so very good here. The frame 10 is missing for a reason. It means the frame 10 in the second (HQ) video doesn't exist in the first (LQ, no logo) video, therefore it's useless because I won't have a proper frame to cover the logo with on that frame.

Frame #9 in LQ vid is the same frame as 9 in the second, but frame 10 is the same as frame 11 in second vid.

[poisondeathray]

It is. A jerk every second.

It's not really noticeable. I'll be deshaking later anyway. On top of that, I can interpolate it back to 25fps later.

The usual interpolation methods don't work that way, every frame will be resampled not just the 1 missing frame (every frame will become degraded) . An interpolation from 24p => 25p using optical flow will retime the whole clip, not insert specifically where the drops are .

Deshaking won't help, it's temporal problem, not spatial

Maybe because you did the numbering wrong? Did you see that you are missing frame 10 in the 2nd column ? And you can't determine a "pattern" by looking at 1 cycle... at minimum you need 2 cycles

We r not communicate so very good here. The frame 10 is missing for a reason. It means the frame 10 in the second (HQ) video doesn't exist in the first (LQ, no logo) video, therefore it's useless because I won't have a proper frame to cover the logo with on that frame.

Frame #9 in LQ vid is the same frame as 9 in the second, but frame 10 is the same as frame 11 in second vid.

OK, I'm misreading it. The 2nd column refers to the corresponding HQ's version frame number when aligned to frame 0 (which you called "1") . Then the problem is a lot worse - Not only do you have many missing frames, some of those are duplicates in the LQ version - in effect you're missing more than 1285 unique frames .

Eitherway , you need to look at more than 1 cycle to see if a pattern of drops / duplicates exist (at least 50 frames, probably a lot more)

[Mephesto]

The usual interpolation methods don't work that way, every frame will be resampled not just the 1 missing frame (every frame will become degraded) . An interpolation from 24p => 25p using optical flow will retime the whole clip, not insert specifically where the drops are .

I remember InterFrame.avs, something you introduced me to which was awesome. I doubled framerates of some videos I watched, many of them DVD Xvid rips with blocking artifacts galore. The interpolated frame did not have blocking artifacts but original frames always did. Scrolling thru them in virtualdub would give me a blocking - no blocking - blocking - noblocking pattern. So I don't think all frames were affected.

On the other hand, I tried to upscale this 29fps video to 30 with InterFrame once and noticed that a couple frames before and after the interpolated one were changed so I left it alone.

I can specifically interpolate the frames I want with frame replacement RX though, but then I'd have to enumerate them in Excel, another extra step to consume my time. I might not upscale at all in the end.

Deshaking won't help, it's temporal problem, not spatial

Deshaking is a temporal procedure that fixes shakiness/jerkiness, remember?

OK, I'm misreading it. The 2nd column refers to the corresponding HQ's version frame number when aligned to frame 0 (which you called "1") . Then the problem is a lot worse - Not only do you have many missing frames, some of those are duplicates in the LQ version - in effect you're missing more than 1285 unique frames .

Yes, **** you for reminding me. :/

Eitherway , you need to look at more than 1 cycle to see if a pattern of drops / duplicates exist (at least 50 frames, probably a lot more)

Ok I'll give you some more in a minute. Hold up.

[poisondeathray]

The usual interpolation methods don't work that way, every frame will be resampled not just the 1 missing frame (every frame will become degraded) . An interpolation from 24p => 25p using optical flow will retime the whole clip, not insert specifically where the drops are .

I remember InterFrame.avs, something you introduced me to which was awesome. I doubled framerates of some videos I watched, many of them DVD Xvid rips with blocking artifacts galore. The interpolated frame did not have blocking artifacts but original frames always did. Scrolling thru them in virtualdub would give me a blocking - no blocking - blocking - noblocking pattern. So I don't think all frames were affected.

Yes, not all frames, only the lowest common denominator/FPS (24 and 25 are evenly divisible into 120) so every 4th or 5th frame . And yes, the interploation methods have their issues with quality (edge morphing)

I can specifically interpolate the frames I want with frame replacement RX though, but then I'd have to enumerate them in Excel, another extra step to consume my time. I might not upscale at all in the end.

R, RP, RX are written to replace frames ("write over bad frames" by interpolating frames) , not insert; ie. framecount is unchanged. You can modify it, or I think there might be some that already do this

Deshaking won't help, it's temporal problem, not spatial

Deshaking is a temporal procedure that fixes shakiness/jerkiness, remember?

It won't help much with frame drops . Deshaker move the frame in x,y space (it repositions the frame), it might be marginally better, but not much

OK, I'm misreading it. The 2nd column refers to the corresponding HQ's version frame number when aligned to frame 0 (which you called "1") . Then the problem is a lot worse - Not only do you have many missing frames, some of those are duplicates in the LQ version - in effect you're missing more than 1285 unique frames .

Yes, **** you for reminding me. :/

Eitherway , you need to look at more than 1 cycle to see if a pattern of drops / duplicates exist (at least 50 frames, probably a lot more)

Ok I'll give you some more in a minute. Hold up.

Seriously, it's probably not worth it. If that 1 cycle is representative with dupes, you effectively have 3 missing per cycle (since you have 2 sets of duplicates in the 1st cycle), probably equivalent to ~ 1285x3 missing frames...

[Mephesto]

well I can't give you the numbers now because I cannot use Ctrl+C. The batch I set in photoshop for that other video was to paste new cyan layer from clipboard because creating new layer and filling color would be an extra step and waste of time. Ugh.

So yeah I can't quote all your paragraphs now but I think it's worth it because that logo is distracting as **** and I hold no respect for ******** trying to stroke their egos by butchering rare videos.

22fps is pushing it but it's still not bad. if it went down to 20 i probably wouldnt do it

[poisondeathray]

Seriously ? Is the logo that bad that you can't use other methods to remove/hide the logo ? Even if you had matching frames, upscaling the patch probably won't even match very well (compression and quality differences to the HQ version)

Jerky playback (really jerky if you're dropping 3 per 25) is waaaay worse. Unless this a very, very terrible, kill-small-kittens terrible logo ? or something stupid like covers 50% of the screen and positioned in the middle




BTW , one way you can do the insert-interpolation frame every nth frame , is by inserting a blank frame as a "placeholder" by using InterleaveEvery() from stickboy , then interpolating to replace that "placeholder" frame (and you don't need an excel list)

https://forum.videohelp.com/threads/352741-Frame-interpolation?p=2222196&viewfull=1#post2222196

[Mephesto]

Its a 200x125 logo on a 640x480 video, that's what like 20% of the video? 20% of video someone basically saying "hai guys i'm Tsutomu Miyazaki, I have the world's largest lolicon and guro collection because i'm a needle-dick and practically never see the sun and i'm gonna force you to acknowledge my existence with the only HQ version of this you'll find on the net. I really wanted to moon you instead of typing out all this bullshit but I have a bad case of nearly every STD known to man that i've added to my collection with all the unwilling homeless girls I porked and i'm kinda self-concious 'bout it. so yeah um, hi!"

That's why the logo's so big, yo. This isn't the first time I dealt with this either. But at least with the last one the frame orientation was correct with the logoless version I had. Blow up the resolution, do a little deconvolution and it was practically unnoticeable.

The video won't be jerky. I'll figure something out. Oh right you needed the numbers, hold on...

[Mephesto]

31 31
32 32
33 33
34 35
35 36
36 36
37 37
38 39
39 40
40 41
41 41
42 43
43 44
44 45
45 46
46 (dropped frame from fps downscale)
47 47
48 48
49 49
50 50
...
5000 5001
6000 6000
7000 7000

[Mephesto]

BUMP. Any ideas?

I thought of interleaving every frame of the first video with the second and using stats from dedup.avs as a tool for determining matching frames and thus which frames of second video to keep, but I can't think of any intuitive way to do this.

[b5fan]

I'm just a video noob so I'm not sure if this is even possible, but mathematically speaking, could you upscale both videos to the least common framerate (in your case 600 fps) or whatever the highest rate possible is, then every frame will match. Isn't this basically what the newer TVs with super high Hz ratings do. That way they can take all different types of videos and they all look good. Then scale it back down to whatever you want for the final video.

[manono]

Its a 200x125 logo on a 640x480 video, that's what like 20% of the video?

Closer to 8%. Have you tried a logo remover? Yes, you'll get mostly a blur, but still better than a lousy logo.

I've done something like this before where the frame count and framerate didn't match up and I wanted to get rid of a nasty logo right in the middle. I had a better version with a big opaque logo right in the middle and YouTube had a poor quality but logoless version. I wound up, whenever the frames no longer matched, duplicating frames using the Loop function (if memory serves). Or maybe it was Trim, if the one I wanted had a lower framecount. These were usually done at scene changes so it wasn't very obvious. Even if you don't have the exact frames to match, often the background is the same (unless the camera is moving), and the replacement works out pretty well. It was only a 5 minute or so music video, though, and not a 25 minute episode. Still took a long time to do. Here's the post and thread:

https://forum.videohelp.com/threads/331626-Logo-removal-by-using-2-sources?p=2055312&vi...=1#post2055312

Of course, you already know how to do all that and your problem is the framecount doesn't match between your 2 sources. I think you should be trying to equalize the framecounts, either through interpolation or ChangeFPS, or something. Then replace the logo as best you can. After having done that, go through the whole thing and when you come to frames that don't match up, then create a new one by using the good frames on either side. As has been discussed already, though, it would be easier if you could find a pattern to the missing frames. As pdr mentioned, it could take a whole lot of framecounting to find one, but I think a pattern should exist, at least one where almost all of the frames will line up correctly. 2 or 3 cycles isn't near enough.

[jagabo]

I'm not absolutely sure this will work but I would look into using ConditionalFilter() to pick from multiple phase shifted versions (ie, plus and/or minus one or more frames temporally) of the secondary video. Match the overall frame count first. Then create some phase shifted versions of the secondary video (Trim (1,0), Trim(2,0), etc.). Use Subtract() of the main video vs phase shifted videos to create some test videos. Rectify the results of Subtract() and normalize based at zero, ie, abs(a-b) -- the brighter the result the worse the match. Then use the runtime feature to locate the phase shifted video with the closest match (lowest average intensity of the rectified test videos) and select the frame from that video.

[jagabo]

By the way, you can use InsertFramesMC() to insert individual motion interpolated frames where there are missing frames:

https://forum.videohelp.com/threads/352741-Frame-interpolation?p=2226119&viewfull=1#post2226119

You have to locate the missing frames manually though.