Per-frame metadata overlay?

[cheyrn]

For some parts of a video I want to stretch and compress parts of it to change the apparent speed, non-linearly and possibly in some non-formulaic way.

Right now, I have created videos with 1 digit per frame that matches a source video, then I change the apparent speed of these videos overlayed. The result is that I can see which source frames were duplicated or dropped. But, that means looking at each frame to see what source frame it was, which is too much work.

I would like to get that sequence of as an array and then transform the array using programs and copy the corresponding frames from the source video frame by frame from the transformed array.

For example:

A video segment goes:

101, 101, 101, 101, 102, 102, 102, 102, 103, 104, 105

I would like to have that as digits in a file. Then I could convert it to something like

x1 = spline(0, \
0, 101, \
1, 101, \
2, 101, \
3, 101, \
4, 102, \
5, 102, \
6, 102, \
7, 102, \
8, 103, \
9, 104, \
10, 105)
x2 = spline(1, \
0, 101, \
1, 101, \
2, 101, \
3, 101, \
4, 102, \
5, 102, \
6, 102, \
7, 102, \
8, 103, \
9, 104, \
10, 105)
...
x10 = spline(10, \
0, 101, \
1, 101, \
2, 101, \
3, 101, \
4, 102, \
5, 102, \
6, 102, \
7, 102, \
8, 103, \
9, 104, \
10, 105)

Is there something that will write per-frame metadata and then something that will read per-frame metadata?

[poisondeathray]

If you're still doing these types of manipulations in avisynth, use ShowFrameNumber() on the original sequence to overlay the original frame numbering

You can use another instance (and change the color or position or display options), to demonstrate the "after" effect of your manipulations to overlay the "new" frame numbering of the new clip

[Cornucopia]

Or get AE, do a time remap (with an expression) and you'd probably already be finished by now. Again, use the correct tool for the job.

Scott

[cheyrn]

Is AE after effects? I am using avisynth now for tasks that when done manually turns me into a mouse clicking robot. And as I go along I keep thinking of things that will accomplish something, and even if some interactive tool can do that, I would have to go figure out what sequence of actions that the computer wants me to perform to do it. I want to program the computer, not have it program me.

It's worse than that though, because I often decide to change things that I have already done, which means I would have to keep repeating a sequence of movements to replay what I did. With avisynth, it's in a script which I can edit.

I have digits in an overlay, what I want is the digits in a file as text, so that I can manipulate them. But, it sounds like nothing like that comes to mind.

[poisondeathray]

Is AE after effects? I am using avisynth now for tasks that when done manually turns me into a mouse clicking robot. And as I go along I keep thinking of things that will accomplish something, and even if some interactive tool can do that, I would have to go figure out what sequence of actions that the computer wants me to perform to do it. I want to program the computer, not have it program me.

It's worse than that though, because I often decide to change things that I have already done, which means I would have to keep repeating a sequence of movements to replay what I did. With avisynth, it's in a script which I can edit.

Yes, AE is after effects

Maybe there is some misunderstanding about what you're trying to do? Not even the most hardcore programmer would use avisynth for these types of editing tasks that involve syncing things . If you didn't have to sync anything, or test playback of anything (e.g to see what frames were dropped) , and just happen know exactly what you want in a script, then your approach migth make some sense . Basically if you didn't have to EDIT anything . Or how did you plan on testing your script for sync? It's avisynth that will end up requiring tedious script editing and take many times longer to do. Just trying to be helpful and brutally honest .

Some programmatic tasks , for example selecting every nth frame are handled better and easier in avisynth, certain tasks like high quality deinterlacing, several other things.... But syncing and non linear editing are NOT some of the strong points for avisynth . (NLE = non linear editor. That's why they are called NLE's, they are used for non linear editing)

I have digits in an overlay, what I want is the digits in a file as text, so that I can manipulate them. But, it sounds like nothing like that comes to mind.

If your digits are ALREADY overlaid into the video, then there is no way you can READ the overlay from the video with any accuracy and write it to a text file




Have you had a chance to read the previous post with RemapFrames() ? It basically does what you want for seeing which frames are dropped, which remain. You can specify specific numbers, patterns, or even read the frame numbers and replacement mappings from an external TEXT file if you really wanted to.

In the example code in the other thread, I used ShowFrameNumber() like I suggested here. This is a common technique used for debugging scripts etc...
https://forum.videohelp.com/threads/368301-AviSynth-Animate-syntax?p=2355683&viewfull=1#post2355683





It gets messy in a hurry if you have many files / clips. Unlike NLE's, avisynth doesn't have clip metadata handling, and organizational features. It's just a bad way of doing things with a separate external text file for each clip

If you still want to do this, but in more general terms - you'd have to list the types of software you are using. NLE's have EDL's , AAF's , OMF's , XML's that can do these types of things, but they are almost always application specific (an EDL from "x" software might not be able to be read by "y" software because of slight differences).

[poisondeathray]

So the text file , or string in avisynth if you didn't want to use external text file, wouldn't be comma separated . The format is [input frame range ] [replacement frame range] . You can display the frame overlay by using ShowFrameNumber() on the source clip . You can always comment it out # when encoding the final video

Your "x1" would look like this in RemapFrames in simplifed fashion ( you can still do line by line entries), but [0 3] 101 means those frames 0-3 inclusive in the output file have been replaced by 101 from the input (source clip)

[0 3] 101
[4 7] 102
8 103
9 104
10 105

[cheyrn]

The mappings file idea is very useful yes. I didn't quite take in what is useful about remapframes in the other thread because I was busy being confused about animate.

Essentially, I want something that would output the mappings file that remapframes takes as input, as far as it being text with frame numbers in a file. Here is an example:

Video A is 280 frames long. I have edited the video and now have video B containing what was frames 149-201 as frames 0-51. Now, I decimate or whatever the term is parts of the video and have video C. Video C has 97 frames. Of the 97 frames all of them are copies of one of the 52 frames from video B.

One month later, I am going to include video C in another video. I want to have a program, a python script for example, be able to tell that the frames 0-3 in the clip were frames 149-152 in video A.

A hacky solution that is not practical that I have now is overlaying pictures of digits and rerunning the scripts that created videos B and C. It's not practical because it means I would have to look at each frame and read the digit that is on it. I want the digits in a file as text. Or, some other way in which to tell what frames from video A are in video C.

About the interactive video editing tools, if I tried to do exactly the same thing, a month after creating video A, I would have to figure out how to manually recreate videos B and C. The way I would do that is by thinking of the steps to accomplish the task and moving my hands around to cause the software to do it. Essentially, I came up with a program to accomplish the task. If I do it manually, I am acting out the steps of the program.

I would rather write that program into a script that does it since I get tired and bored when acting as a robot.

If I were you reading about what I said i was doing, I would be unsure about what the person is actually trying to accomplish and think it all seems very complicated. What I am actually doing is reacting to the pain of having to go through the physical motions of making software do something, over and over again.

Thank you for the advice about easier ways to do things using an NLE. Even if they don't exactly match what I was trying to do, I learn from it. And, it is likely that there is something I don't yet know how to do in an interactive program that I only need to do once that would be easier using the tool than me writing a script.

[poisondeathray]

Essentially, I want something that would output the mappings file that remapframes takes as input, as far as it being text with frame numbers in a file.

Right now, you have write the mappings file, I don't know how to "automate" writing that in avisynth

Video A is 280 frames long. I have edited the video and now have video B containing what was frames 149-201 as frames 0-51. Now, I decimate or whatever the term is parts of the video and have video C. Video C has 97 frames. Of the 97 frames all of them are copies of one of the 52 frames from video B.

One month later, I am going to include video C in another video. I want to have a program, a python script for example, be able to tell that the frames 0-3 in the clip were frames 149-152 in video A.

A hacky solution that is not practical that I have now is overlaying pictures of digits and rerunning the scripts that created videos B and C. It's not practical because it means I would have to look at each frame and read the digit that is on it. I want the digits in a file as text. Or, some other way in which to tell what frames from video A are in video C.

The "one month later" usually doesn't really matter, normally you do your edits in avisynth (e.g. using Trim, or RemapFrames, or whatever), and you save the .avs. Later on , you edit the .avs. If you include that one month later and want to know where it was from - well presumably you have to look at it to do edits anyways - but I get the impression you're not even looking at the video to do edits

Even custom parser for the python script won't be able to tell you the original heritage of the frames from the remapframes external mappings text file alone, because there are no external references to specifics - like which clips are you referring to, where the clips reside physically (what drive location), where the frames originally came from. It would have to parse both the mappings text file and the .avs script to get the clip references. If you're a python wizard you might be able to do this

There are some write text file commands in avisynth e.g writefileif(), but I don't know how to use them to do what you want. You might ask in the doom9 avisynth forum if someone here doesn't have any ideas

About the interactive video editing tools, if I tried to do exactly the same thing, a month after creating video A, I would have to figure out how to manually recreate videos B and C. The way I would do that is by thinking of the steps to accomplish the task and moving my hands around to cause the software to do it. Essentially, I came up with a program to accomplish the task. If I do it manually, I am acting out the steps of the program.

No you wouldn't for how editing is normally done - it's same thing with an NLE, you save the project file. An additional benefit of NLE, is if you move the folder of source files (lets say you put them on another HDD from drive E:\folder\test to H:\somefolder) , you can relink automatically all the references to various files, instead of manually having to change each reference, or using find/replace . You can tell where the source clip was from, just by looking. All the metadata is saved, all the references. You can just click it and a wealth of information is presented. You can even swap clips with a click. Lets say you want all versions of clipA to be changed to clipB. Easy 2 clicks. For some reason it seems like you don' t want to look at the video when editing - but that' s how editing is done. It's VIDEO editing - you need to SEE things . With a music video, you need to HEAR things too, in realtime

An NLE can use scripts , saved as text based edits like EDL - it basically converts your edits to a list of reference of clip in/end points, edits, positions, various data. Unlike the mappings file, it contains all the data of the clip references, editing points, everything. So you might be able to use a NLE script with your python script, but you seem adverse to that

I would rather write that program into a script that does it since I get tired and bored when acting as a robot.

If I were you reading about what I said i was doing, I would be unsure about what the person is actually trying to accomplish and think it all seems very complicated. What I am actually doing is reacting to the pain of having to go through the physical motions of making software do something, over and over again.

Thank you for the advice about easier ways to do things using an NLE. Even if they don't exactly match what I was trying to do, I learn from it. And, it is likely that there is something I don't yet know how to do in an interactive program that I only need to do once that would be easier using the tool than me writing a script.

I see what you're saying, but you didn't mention anything about audio SYNC this time. Basically if you have to do any type of syncing to music, audio - avisynth (or any script based software) rates poorly compared to even a basic NLE. AV sync becomes tedious and robotic, it's a menial task. There are auto sync programs, but they have limits. Basically it's the opposite to what you think - you become a robot when using scripts because it's so tedious to check for sync. Very fiddly. Easily 20-30x faster in a NLE. Obviously you haven't done a project like making a music video, but it's literally insane to do this in scripts . And that's only 1 track. Usually you have complex compositions, multiple tracks. Complex music videos are almost impossible to do in avisynth or any scripting language, or they take a hell of a lot longer to do, and a lot of more work

I think the problem is that you haven't described clearly what you want to do, or everyone assumed you were making a music video or using audio. Or your definition of AV "sync" is very loose , and there is no real relationship between timing or beats or events

Some tasks are suitable for scripting, batch encoding, batch conversions, batch rewrapping, etc... automation. Those are suitable for using scripts, CLI scripts. Some tasks are NOT suitable. Typical video editing (it involves AV sync, and usually creative user input) is NOT suitable

[Cornucopia]

Sounds to me like you are trying to create your own EDL format. Good luck with reinventing that wheel!

Scott

[cheyrn]

Thank you for the advice. I will pay attention to whether or not I am doing things in a script that I can see how to do interactively more practically.