Can't get DGIndex to work properly

[iTayb]

My Canon HV20 camera tape I recorded into the computer came up with .m2t files. A bunch of.
More like 100 files per hour (Clip 001.m2t, Clip 002.m2t, Clip 003.m2t, etc...)

I understood that in order to load in into Avisynth and IVTC it, I'll have to create .d2v index file with DGIndex.

When I load all the clips (212 clips, 2 hours) it looks well on DGIndex GUI. The frames seems to be suitable for 2 hours clip.

But when I try to save the project - It seems that only the first clip, which is 11seconds long, is being indexed.
When I try to demux audio ONLY, it demuxs the proper 2hours audio .mp2 file, which is actually what I needed.

How can I get it to work & index all the clips, one after each other?

[manono]

But when I try to save the project - It seems that only the first clip, which is 11seconds long...

What makes you say that? It seems to me if you got the audio for the whole thing, you're also getting a D2V project file for the whole thing. Did you then set up and open an AviSynth script using the D2V and the DGDecode.dll?

I understood that in order to load in into Avisynth and IVTC it...

Eh? I don't think I've ever heard of a camera that produces video that can be IVTC'd. I had a look at the specs and they say this:

1080/60i, 1080/30F, 1080/24F, 1080/24P

The only one that can possibly lend itself to being IVTC'd is this 30F, but I have no idea what the 'F' means in this context, and I seriously doubt it can be IVTC'd. 60i (more properly 30i) is an interlaced mode, I would guess. No IVTC there.

[poisondeathray]

IIRC, HV20 does 24p wrapped in 30i

iTayb - are you using most recent version of DGIndex?

Test on some different clips, like #30 to #40. (maybe there is something wrong with clip #1 causing it to abort)

[manono]

IIRC, HV20 does 24p wrapped in 30i

It comes out of the camera already hard telecined? If so, calling it 24p is a lie, if what you say it true. Or if I'm understanding what you're saying correctly.

[iTayb]

But when I try to save the project - It seems that only the first clip, which is 11seconds long...

What makes you say that? It seems to me if you got the audio for the whole thing, you're also getting a D2V project file for the whole thing. Did you then set up and open an AviSynth script using the D2V and the DGDecode.dll?

I say this because the length of the project is way too short, and DGIndex is done withing seconds (I guess 28GB 2hours videos should take some more to index)

I'm using version 1.5.5.

I tried doing it on Clips number 203, 204 and 205.
When I do "Save Project" I get Clip 203 PID 814 L2 2ch 48 384 DELAY -225ms.mp2 and Clip 203.d2v.
And as the filename says - It's only clip number 203. For some reason, it's ignoring the other clips.


Maybe I misunderstood what IVTC is actually is. The video itself is interlaced and has lots of artifacts which I wish to fix.
I thought IVTC = deinterlace? Sorry for my poor understanding.

This is my video specs:

General
ID : FF
Complete name : G:\EncodeTries\HV20\USA 2008\Media\Clip 001.m2t
Format : MPEG-TS
File size : 26.9 MiB
Duration : 8s 279ms
Overall bit rate : 27.3 Mbps
Maximum Overall bit rate : 33.0 Mbps
Encoded date : UTC 2008-08-02 19:14:48

Video
ID : 2064 (0x810)
Menu ID : 100 (0x64)
Format : MPEG Video
Format version : Version 2
Format profile : Main@High-1440
Format settings, Matrix : Default
Duration : 8s 266ms
Bit rate mode : Constant
Bit rate : 25.0 Mbps
Width : 1 440 pixels
Height : 1 080 pixels
Display aspect ratio : 16/9
Frame rate : 29.970 fps
Standard : Component
Colorimetry : 4:2:0
Scan type : Interlaced
Scan order : Top Field First
Bits/(Pixel*Frame) : 0.536

Audio
ID : 2068 (0x814)
Menu ID : 100 (0x64)
Format : MPEG Audio
Format version : Version 1
Format profile : Layer 2
Duration : 8s 279ms
Bit rate mode : Constant
Bit rate : 384 Kbps
Channel(s) : 2 channels
Sampling rate : 48.0 KHz
Resolution : 16 bits
Video delay : -64ms

Thank you very much for your help!

[poisondeathray]

manono - yes, my understanding is that model doesn't shoot true 24p

http://eugenia.gnomefiles.org/2007/07/13/canon-hv20-24p-pulldown/

I think iTayb is shooting 30i however, not 24p mode

iTayb - what is your end goal format? e.g. if it was to make a dvd, you should leave it interlaced.

I've never seen this happen with DGIndex; are you sure all the clips were shot in the same mode and have the same specs?

[iTayb]

If I wanted it to be DVD, I wouldn't buy an HD camera.
My goal format is x264.

Yes, all the clips has the same specs.

[manono]

I say this because the length of the project is way too short, and DGIndex is done withing seconds (I guess 28GB 2hours videos should take some more to index)

But you also said you got complete audio. For the same reason you can't create the D2V in mere seconds, neither can you demux the audio in seconds.

You might have to bring this up with the author, neuron2 (Donald Graft). You can post about your problem in the DVD2AVI/DGIndex Forum over at Doom9. You'll have to wait 5 days between the time you register and the time you can post.

That link you provided is interesting, poisondeathray, as at the bottom he explains how to do it using AviSynth. I wonder how Canon can get away with calling what the camera produces 24P (or 24FP, I guess), as there's nothing 'P' about it.

[iTayb]

When I try to create the .d2v file, the job is done in mere seconds, and it skips all clips but the first.

When I try to demux audio only, the job is done in a few minutes, and the length of the audio is suitable for all the clips together.

[manono]

Ordinarily you get the audio at the same time as you create the D2V file. That's why what you said didn't make much sense to me. I didn't realize you were demuxing the audio separately from creating the D2V. It's sounding more and more like either there's something wrong with the video, or (perhaps more likely), you're doing it incorrectly. Did you read the included docs thoroughly? I've never worked with camera video so maybe someone else can help. Or, as I suggested, you might try posting at the forum the DGIndex author moderates.

[iTayb]

I shall post there in five days.

My question is, is it even possible to load multiple clips and index them together into one .d2v project?

Because if it's not even possible, I should use directshowsource() for every separate clip and join them. (Is that method unrecommended for some reason?)

And one more question I should ask. What Bitrate should I grant for (to be) 1080p for (almost) same quality?
I don't have a problem with huge files, but 14.4GB per hour is just way too much.

Thank you very much!

[poisondeathray]

My question is, is it even possible to load multiple clips and index them together into one .d2v project?

Absolutely. There are even steps explaining how to do it in the manual

Because if it's not even possible, I should use directshowsource() for every separate clip and join them. (Is that method unrecommended for some reason?)

Possible to use DirectShowSource(). But not recommended. It's not frame accurate, and less consistent because it relies on system installed codecs & filters. It can be destructive if you forget to turn something off in ffdshow for example. If you are very diligent, it can work fine however. The problem is the more clips you have, the higher likelihood of sync issues with DirectShow.

Another alternative would be you use FFMpegSource2() , you could concatenate you clips that way. But it's still tedious if you have lots of small clips, although you maybe able to code a wildcard (*) function to append them in avisynth based on ascending number - that's beyond my scripting capabilities you would have to ask at Doom9 or start a new thread here.

And one more question I should ask. What Bitrate should I grant for (to be) 1080p for (almost) same quality?
I don't have a problem with huge files, but 14.4GB per hour is just way too much.

Impossible to answer. It depends on source complexity. Things with noise and motion require more to encode. If it's just a simple still movie, it might take very little. You could use 1 pass CRF mode as a quality based encoding method , instead of using a 2pass bitrate method. Most people use between 18-22, lower is better quality/bigger filesize.

[iTayb]

Because if it's not even possible, I should use directshowsource() for every separate clip and join them. (Is that method unrecommended for some reason?)

Possible to use DirectShowSource(). But not recommended. It's not frame accurate, and less consistent because it relies on system installed codecs & filters. It can be destructive if you forget to turn something off in ffdshow for example. If you are very diligent, it can work fine however. The problem is the more clips you have, the higher likelihood of sync issues with DirectShow.

Another alternative would be you use FFMpegSource2() , you could concatenate you clips that way. But it's still tedious if you have lots of small clips, although you maybe able to code a wildcard (*) function to append them in avisynth based on ascending number - that's beyond my scripting capabilities you would have to ask at Doom9 or start a new thread here.

Actually, the IVTC code provided in this "HV20 video pulldown" is as followed:

Directshowsource("__vid__",pixel_type="auto")
tdecimate()
AssumeFPS(24000,1001,sync_audio=true)
Lanczos4Resize(1920,1080)

Where TDecimate is TIVTC.dll's function and directshowsource has to have "Deinterlace" option on.

They do use directshowsource in their scripts. Is that somewhat wrong? Should I use different deinterlace function?

[poisondeathray]

Actually, the IVTC code provided in this "HV20 video pulldown" is as followed:

Directshowsource("__vid__",pixel_type="auto")
tdecimate()
AssumeFPS(24000,1001,sync_audio=true)
Lanczos4Resize(1920,1080)

Where TDecimate is TIVTC.dll's function and directshowsource has to have "Deinterlace" option on.

They do use directshowsource in their scripts. Is that somewhat wrong? Should I use different deinterlace function?

DirectShowSource doesn't have a detinterlace filter per se, it's done through the filter chain. e.g. you activate it through you decoder e.g. ffdshow, or whatever decoder you have for your source material. I've mentioned above potential problems using directshow, and it's usually only used as a last resort type method. You can also deinterlace in the .avs script

Something really doesn't make sense here to me; it doesn't make sense to decimate and deinterlace the footage, it's usually one or the other. If it's progressive footage in an interlaced wrapper, you extract the progressive footage by IVTCing (some form pulldown removal with different phase or cadence possibilities), not by deinterlacing. If it's pure interlaced, you deinterlace it, not decimate it....But I don't have this camera, and I haven't looked at the footage, so I can't be sure

[iTayb]

I'm not sure.
If I upload 5 seconds clip, would you care to check what this video actually is?

[poisondeathray]

I'm not sure.
If I upload 5 seconds clip, would you care to check what this video actually is?

I can try...but you might need a longer duration clip , preferrably with horizontal panning. Manono is much better at this than me he'll be able to tell for sure

Do you know what record mode you used when making the clips?

[iTayb]

I don't remember what record mode I was using. It was more than a year before.

Here is a horizontal panning 47.3MB / 14 sec clip: http://files.anime-il.com/Clip%20010.m2t
Thank you very much.

[jagabo]

It's straight 3:2 pulldown:

Mpeg2Source("filename.d2v")
TFM()
TDecimate()

DirectShowSource() will probably work in place of Mpeg2Source().

[poisondeathray]

Yes, i agree with jagabo, it looks like standard 3-2 pulldown (the sample clip on eugina's site on the link above has the same charateristics)

The other problem with directshowsource is if you load multiple files, you memory usage will balloon into a few GB or even more depending on the number of clips. It's a huge memory hog
e.g.
a=directshowsource("file01.m2t")
b=directshowsource("file02.m2t")
c=directshowsource("file03.m2t")
a++b++c
TFM().TDecimate()
LanczosResize(1920,1080)

Instead of directshowsource() I would use DGIndex (even if you have to index them all), or something like ffmpegsource2 which automatically creates an index for you
a=ffmpegsource2("file01.m2t",atrack=-1).assumefps(30000,1001)
b=ffmpegsource2("file02.m2t",atrack=-1).assumefps(30000,1001)
c=ffmpegsource2("file03.m2t",atrack=-1).assumefps(30000,1001)
a++b++c
TFM().TDecimate()
LanczosResize(1920,1080)

Interestingly, DGIndex when used to append Eugina's testclip caused pixellation and corruption at the join sites, but using the other methods to append don't seem to cause this. (The DGIndex append function works here when I copied the clip multiple times). Not sure if it's an issue with that test clip

Again, I don't know the wildcard function syntax how to append sequentially numbered clips, maybe jagabo , manono or the others do, otherwise doing a 100 or so clips can get tedious

Another approach would be to join the clips first into 1 big clip , e.g. using videoredo., before using avisynth

[iTayb]

VideoReDo caused pixellation and corruption at the join sites.

[poisondeathray]

VideoReDo caused pixellation and corruption at the join sites.

Indeed, that would have been far too easy, right?

Well it looks like you'll be joining this through scripts then

[iTayb]

I programmed C++ script that will create AVS automatically for me (with loops of course).
I'll try now encoding it (and indexing, which takes like forever)

I'll be back with results!
Thank everyone for all the help.

[Gavino]

I programmed C++ script that will create AVS automatically for me (with loops of course).

You can now write loops directly in Avisynth with the aid of the GScript plugin.
http://forum.doom9.org/showthread.php?t=147846

[jagabo]

DgIndex has a command line interface so you could write a program to index all your files too.

[iTayb]

I've just indexed all the files successfully.
Problem was that only the first clip was selected. I didn't know that I have to select otherwise than default.

So now I have one d2v project that needs IVTC/interlacement. I'll try the TFM().TDecimate() combination.

edit: My AVS script is as followed:

Mpeg2Source("Clips.d2v").AssumeFPS(30000,1001)
TFM().TDecimate()
AssumeFPS(24000,1001)
LanczosResize(1920,1080)

And there are pixellation and corruption at the join sites. I guess the d2v is corropted for some reason.
Example of bad join frame: http://i27.tinypic.com/2d6jcys.jpg

[jagabo]

All those AssumeFPS() commands shouldn't be needed.

I've seen Mpeg2Source() generate block errors like that but I'm not sure what causes it. Open GOPs maybe? What happens if you play the individual files in DgIndex? Do you see blocks there too?

[iTayb]

When you say DGIndex, do you mean the GUI?
If so - there are no block errors at all - even with all the files included.
If you mean by loading the d2v project with Mpeg2Source() - individual files are fine - no block errors.

It seems that the block errors are found at the join sites, if any are exist.

I tried inserting two copies of the same individual file as well. It's totally corrupted.

[poisondeathray]

^I already reported this behaviour in the above post with DGIndex's append on Eugina's testclip, sorry if it wasn't more clear

FFMpegSource2 , when I tried on a few copied files, didn't have the pixellation

[iTayb]

ffms2 doesn't support d2v projects, though. Plus it seems to have problems when combining more than 50 different clips - it gives random "Cannot open file" errors without practical reason. Maybe it wasn't made for this kind of job.

[poisondeathray]

ffms2 doesn't support d2v projects, though. Plus it seems to have problems when combining more than 50 different clips - it gives random "Cannot open file" errors without practical reason. Maybe it wasn't made for this kind of job.

ok. I can't say that I have ever tried >50 clips before...maybe you ran into a memory issue? Check taskmanager and maybe use SetMemoryMax(1024) ?

My observation was there were errors in the d2v project when using the append function on every cut/join, so I didn't see support for a d2v project being that useful, at least on these clips. Perhaps neuron2 will figure it out, when you get to post at Doom9

It would be even worse with DirectShowSource(), as each instance consumes a lot of memory, and avisynth 32bit has a max 2GB limit AFAIK