Avisynth - Open DPX files

[poisondeathray]

i did a little more digging. i was testing my dpx sequence with a tiny 4 frame sequence. i realized that vdmf was only showing 2 frames in the previewed sequence. so i created a new sequence that was 24 frames long. now vdmf is only showing a sequence of 14 frames. the first 9 or 10 frames are correct, and frames 11-14 are frames toward the end of the original 24 frame sequence.

so i tried a 24 frame jpg sequence and also a 24 frame tif sequence. the jpg and tif sequences are previewing correctly in vdmf. also, they work fine with my "film restoring" script applied in avisynth and are working fine frameserved a second time by avfs.

et voila, the problem seems to stem with using dpx in vdmf. i think i can live with converting my dpx files to tif if it comes to that. unless you have another suggestion maybe i should try vapourSynth again. as stated though, i so far unable to get imageMajick installed properly so no dpx files there either!

I'm wondering if something peculiar about your dpx sequence ? There many different dpx variations, are they from the same batch and type? Same characteristics, dimensions too ? Also are named and numbered correctly ?

Can you upload a sample , and zip(archive) it up ? For example the 4 frame sequence . If it's a bug with vdfm / vdub, then perhaps something can be done about it. because it seems to work ok here

i tried to autoload imageMajick (using this directory: C:\Program Files (x86)\VapourSynth\plugins64\libimwri.dll)
with this code:

import vapoursynth as vs
core = vs.get_core()
video = core.imwri.Read('D:/test.%06d.dpx')
video.set_output()

i get this error line 3

Failed to evaluate the script: Python exception: No attribute with the name imwri exists. Did you mistype a plugin namespace?

imwrif not imwri

also specify firstnum=whatever for the first number in the sequence

[poisondeathray]

I don't think anyone has "translated" videofred's script into a native vapoursynth equivalent, but it should be possible, all the building blocks are there.

But you can actually load vapoursynth scripts into avisynth with vapoursource using vsimport() . So if the only way you can get a dpx sequence (easily) loaded is with vapoursynth, you can still apply the existing videofred script in avisynth

If it's just a short sequence, the ffms2 and single splices isn't that bad in avisynth

[procedure]

imwrif not imwri

what the!???
the documentation shows imwri:
http://www.vapoursynth.com/doc/plugins/imwri.html
and imwri is all over the main doom post:
https://forum.doom9.org/showthread.php?t=170981

my god, how did you know it was changed? (don't answer that )

so i have gotten rid of the previous error, and now it appears to be simply a matter of how to bring in a proper image sequence.
i'm using this now:

video = core.imwrif.Read(filename="D:/Data/Downloads/VapourSynth/test/footage/test.%06d.dpx", firstnum=1)

and getting this error:

Python exception: Read: No files matching the given pattern exist

my dpx files look like this:
test.01.dpx
test.02.dpx
test.03.dpx
test.04.dpx
i uploaded a 44mb zip of these files. my dpx sequence is coming from a single transfer source. i can't imagine or see that there is a difference between each frame.
thank you!

[poisondeathray]

It should be "%02d" , because you have 2 placeholder digits (not "%06d", which would mean 6 digits)

Yes, the documentation needs updating

[procedure]

i'm sorry i should have researched that one. its just that there have been so many problems that that one got buried.

i will try to get into vapourSynth and frame serving it to adobe premiere as well as studying the possibilities of finding a similar script for film restoration. (in other words, i'll be back!)
i need deflicker, stabilization, degraining, sharpening, and most importantly (and i can live without all of them but this one) temporal dirt cleaning.

thank you so much. you are amazing.
(and thanks to everyone helping tirelessly on this thread!)

[poisondeathray]

I can reproduce the vdfm sequence issue . shekh is the author of that fork, and he's posted earlier in this thread, so I think he should be investigating the issue . The dpx themselves look ok, and import fine in other programs that support dpx - nothing looks out of the ordinary. So it probably is an issue specific to vdfm



If you just want something done now, I would go the VSImport() route in avisynth and use your existing workflow/script (basically just using the vpy to "read" the sequence.). There are a lot of "gotchas" in vapoursynth (I guess quite a few in avisynth too, but you've been using that so are at least familiar with a workflow that works for you) . But you can revisit vapoursynth when you have more time

But one "gotcha" you're going to have right off the bat is IM plugin imports as float in vapoursynth, so you would probably need to convert to a compatible colorspace/depth before sending it somewhere . Avisynth+ supports some additional depths and colorspaces, but not float

[sneaker]

.....

[shekh]

I can reproduce the vdfm sequence issue . shekh is the author of that fork, and he's posted earlier in this thread, so I think he should be investigating the issue . The dpx themselves look ok, and import fine in other programs that support dpx - nothing looks out of the ordinary. So it probably is an issue specific to vdfm

This is caused by some weirdness in ffmpeg.
I can see it with ffprobe.exe -i "test.%%02d.dpx" -start_number 1 -show_streams
...
r_frame_rate=15/1
time_base=1/25

These numbers don't make sense for me. Maybe will make a workaround in few days.

[jagabo]

I can reproduce the vdfm sequence issue .

I saw the same thing here -- only the first two of the four images loaded. Oddly, changing the filenames to start with zero instead of one loaded six frames (0,1,1,2,2,3). Appending two extra images (1,2,3,4,5,6) got the entire four frame sequence (1,2,3,4).

[shekh]

dpx sequence problem fixed in build 40412 https://sourceforge.net/projects/vdfiltermod/files/VirtualDub%20pack/version%2017/
Caused by framerate stored in pictures (15 in the example)

[procedure]

thanks, guys. you are amazing. i had to go back to work monday and today. hopefully i'll get some time to test this tomorrow. thank you!

[jagabo]

I can confirm the new version of VirtualDub FilterMod autoloads all four DPX files, 1 through 4.

[procedure]

Finally able to get off work to test this.

i can confirm that large dpx sequences (hundreds of frames) are importing normally into vdfm now. thank you so much shekh.

frameserving a 58 frame dpx sequence from vdfm 32bit to avisynth is causing a crash in vdfm

An out-of-bounds memory access (access violation) occurred in module 'VirtualDub'... ...reading address 00000000

however, vdfm 64bit is not crashing with any of my sequences.


i can import this vdr into a simple avs scipt and use avfs to frameserve this a 2nd time to vlc or adobe premiere.

unfortunately, if i use my restore film script from videoFred, vlc still works, but adobe premiere throws this error:

import failure. unsupported format or damaged file

when i get some time, i'll try and see if a specific plugin in the restore film script is causing the problem. but i'm guessing its actually just too many filters, perhaps.


i've been testing your idea to use VSimport of a vapourSynth script using the vapourSource plugin in avisynth. however, i can't get the darn plugin to load.
i've got vapourSource 0.1.0 in my plugin folder but
LoadPlugin("plugins/VapourSource.dll")
is throwing the usual error:

unable to load. module not found. Install missing library?

i'm not sure i need this now that vdfm is working anyway, but...
thank you!

[poisondeathray]

Finally able to get off work to test this.

i can confirm that large dpx sequences (hundreds of frames) are importing normally into vdfm now. thank you so much shekh.

frameserving a 58 frame dpx sequence from vdfm 32bit to avisynth is causing a crash in vdfm

An out-of-bounds memory access (access violation) occurred in module 'VirtualDub'... ...reading address 00000000

however, vdfm 64bit is not crashing with any of my sequences.

Can you check with taskmanager the memory usage? Just curious

i can import this vdr into a simple avs scipt and use avfs to frameserve this a 2nd time to vlc or adobe premiere.

unfortunately, if i use my restore film script from videoFred, vlc still works, but adobe premiere throws this error:

import failure. unsupported format or damaged file

when i get some time, i'll try and see if a specific plugin in the restore film script is causing the problem. but i'm guessing its actually just too many filters, perhaps.

You might need to convert to RGB24 or 32 at the end of the script. There are a few versions floating around but IIRC, most of that script is running in YV12 or YUY2

i've been testing your idea to use VSimport of a vapourSynth script using the vapourSource plugin in avisynth. however, i can't get the darn plugin to load.
i've got vapourSource 0.1.0 in my plugin folder but
LoadPlugin("plugins/VapourSource.dll")
is throwing the usual error:

unable to load. module not found. Install missing library?

i'm not sure i need this now that vdfm is working anyway, but...
thank you!

Not sure about x86 but it works with x64 and avisynth+ MT x64

Also, not sure if fred posted the updated version yet, but IIRC I think he mentioned updating his script for avisynth+ MT x64 . It's significantly faster and more stable than the old avisynth MT

[procedure]

as for the vdfm 32bit frame serving crash when opening the vdr in avisynth, the task manager did not show a peak in memory or cpu over 25 or 30%.

[procedure]

I think we have full resolution here to everything! You are a madman, Mr. Deathray! I added a converttoRBG24 at the end of the script and it opened and ran in adobe premiere fine.
Here are the full details of the workflow:

In order to open and filter (remove dirt and scratches, etc) a DPX (or tiff, etc) sequence in AviSynth and then frame serve that to Adobe Premiere:

----

1. Open the .dpx sequence in VDFM
2. Use VDFM's built-in frame server to serve the file as a .vdr file (or .avi). (I could only get this to work in VDFM 64bit see this link to get the frame server working:
https://github.com/stax76/staxrip/issues/159#issuecomment-299569068)
3. Create an AviSynth .avs script that opens that .vdr file (To import the file later into Adobe Premiere make sure the final file is converted to RGB24 or 32 in the script)
4. Use AVFS to frame serve the .avs script into a virtual .avi file (I used the simple command line: "avfs test.avs" to build the virtual .avi file)
5. Open the .avi file in Adobe Premiere

A filtered virtual avi file will barely function in the premiere timeline. So I plan to edit the film using the original dpx sequence. Then after the edit is complete, I hope to replace all the footage with AviSynth filtered (restored and cleaned) virtual avi files. Then I'll hit render and hope it doesn't take 3 weeks!

Seems so easy now, but i had a tragic head-scratcher at every step.

Thanks so much to everyone that has helped with this project. You are incredible!

(I'm realizing this whole plan is probably a crazy dream anyway. I have about 50+ dpx source sequences to edit in premiere. So to replace then all with virtual avi files, I'll have to open 50+ instances of VDFM and start frame servers. Then do the same with AVFS, if that is even possible!)

[poisondeathray]

as for the vdfm 32bit frame serving crash when opening the vdr in avisynth, the task manager did not show a peak in memory or cpu over 25 or 30%.

What is the system memory amount on that machine ? (25% of what ?)

A 32bit process can usually only access 2GB of memory . Maybe shekh can make some tweaks to the caching or memory handling

[procedure]

hi. obviously this is not going to be too scientific.
i have an i-7-6700k cpu with 16gb of ram running windows 10.
i tried it again and shut down every other program on my computer and noticed that before launching VDFM 32bit:
cpu 2-5%
Mem 27%
when i hit play on the avs with a simple vdr import in AvsPmod, around the moment of the crash it was around:
cpu 25%
mem 35%

[poisondeathray]

Is that memory figure for total or the vdfm process individually ? Check the breakdown in the details tab

[procedure]

ahhr. i should have checked it that way. ok.
before & during the crash, VDMF
CPU 0%
MEM 82mb

before crash, AvsPmod
CPU 0%
MEM 1700mb
during crash, AvsPmod
CPU 7%
MEM 1900mb

[shekh]

Fixed more bugs including the crash in 32bit. https://sourceforge.net/projects/vdfiltermod/files/VirtualDub%20pack/version%2017/

[procedure]

hi shekh. this seems to be the same release as the previous release build 40412. however, the crash appears to be gone now. not seeing a crash anymore. thanks very much!