Saving space with MKV H.264 files

[Ahari]

Interesting. I just tried to add a folder with only 3 MKV files inside (nothing else) and Hybrid did the same thing.

[Selur]

I dropped over night a folder with ~600 files on it without a problem,.. :/
-> a debug output where the problem occures with 2 or 3 files would be nice, this way the amount of debug output I have to read is far less.

[Ahari]

How annoying!! Maybe it is the 64 bit thing rearing it's head again? (I am using the 32 bit ffmpeg in case you are wondering)

[Selur]

Don't think it's 64bit related, somewhere the 'enable GUI counter' get's lowered but not upped again if some actions interfere,.. I'm just not sure where and why

[Ahari]

Sherlock Holmes is on the job!! Super Sleuth is on the case.... (I mean you , not me - I wouldn't know where to start!! LOL)

[Ahari]

Don't know if it helps, but I just noticed that MediaInfo reports all 3 files in the folder as interlaced, but they appear to be progressive. Hybrid also says it is progressive.

[Selur]

Don't think so the problem is related to the Interlace checking routine,... they are started a lot more times than they are finished,..

What I think happens is that most of them get cancelled before they are finished and the GUI is not enabled on cancellation,...

07:49:00 - HMainTab disable GUI(17),..   ANALYSER START
07:49:00 - enableGui 0 + -1 = -1
07:49:33 - HResizeTab disable GUI(19),..   AUTOCROP START
07:49:33 - enableGui -1 + -1 = -2
07:49:33 - HFilteringTab disable GUI(8),..                   INTERLACE CHECK START
07:49:33 - enableGui -2 + -1 = -3
07:49:34 - HMainTab disable GUI(11),..   SUBITLE EXTRACTION START
07:49:34 - enableGui -3 + -1 = -4
07:49:34 - HMainTab endable GUI(16),..   ANALYSER END
07:49:34 - enableGui -4 + 1 = -3
07:49:34 - HResizeTab enable GUI(18),..   AUTOCROP END
07:49:34 - enableGui -3 + 1 = -2
07:49:35 - HMainTab enable GUI(13),..   SUBITLE EXTRACTION END
07:49:35 - enableGui -2 + 1 = -1
07:49:35 - HResizeTab disable GUI(19),..   AUTOCROP START                            
07:49:35 - enableGui -1 + -1 = -2
07:49:35 - HFilteringTab disable GUI(8),..                   INTERLACE CHECK START
07:49:35 - enableGui -2 + -1 = -3
07:49:36 - HFilteringTab disable GUI(8),..                   INTERLACE CHECK START
07:49:36 - enableGui -3 + -1 = -4
07:49:37 - HMainTab disable GUI(11),..   SUBITLE EXTRACTION START
07:49:37 - enableGui -4 + -1 = -5
07:49:38 - HResizeTab enable GUI(18),..   AUTOCROP END
07:49:38 - enableGui -5 + 1 = -4
07:49:38 - HMainTab enable GUI(13),..   SUBITLE EXTRACTION END
07:49:38 - enableGui -4 + 1 = -3
07:49:38 - HResizeTab disable GUI(19),..   AUTOCROP START
07:49:38 - enableGui -3 + -1 = -4
07:49:39 - Start interlace analyzer title03.mkv
07:49:39 - HFilteringTab disable GUI(8),..                   INTERLACE CHECK START
07:49:39 - enableGui -4 + -1 = -5
07:49:40 - HFilteringTab disable GUI(8),..                   INTERLACE CHECK START
07:49:40 - enableGui -5 + -1 = -6
07:49:41 - HMainTab disable GUI(11),..   SUBITLE EXTRACTION START
07:49:41 - enableGui -6 + -1 = -7
07:49:41 - HResizeTab enable GUI(18),..   AUTOCROP END
07:49:41 - enableGui -7 + 1 = -6
07:49:42 - HMainTab enable GUI(13),..   SUBITLE EXTRACTION END
07:49:42 - enableGui -6 + 1 = -5
07:49:42 - HResizeTab disable GUI(19),..   AUTOCROP START
07:49:42 - enableGui -5 + -1 = -6
07:49:42 - HFilteringTab disable GUI(8),..                   INTERLACE CHECK START
07:49:42 - enableGui -6 + -1 = -7
07:49:44 - HFilteringTab disable GUI(8),..                   INTERLACE CHECK START
07:49:44 - enableGui -7 + -1 = -8
07:49:45 - HMainTab disable GUI(11),..   SUBITLE EXTRACTION START
07:49:45 - enableGui -8 + -1 = -9
07:49:45 - HResizeTab enable GUI(18),..   AUTOCROP END
07:49:45 - enableGui -9 + 1 = -8
07:49:45 - HMainTab enable GUI(13),..   SUBITLE EXTRACTION END
07:49:45 - enableGui -8 + 1 = -7
07:49:45 - HResizeTab disable GUI(19),..   AUTOCROP START
07:49:45 - enableGui -7 + -1 = -8
07:49:46 - HFilteringTab disable GUI(8),..                   INTERLACE CHECK START
07:49:46 - enableGui -8 + -1 = -9
07:49:47 - HFilteringTab disable GUI(8),..                   INTERLACE CHECK START
07:49:47 - enableGui -9 + -1 = -10
07:49:49 - HMainTab disable GUI(11),..   SUBITLE EXTRACTION START
07:49:49 - enableGui -10 + -1 = -11
07:49:49 - HResizeTab enable GUI(18),..   AUTOCROP END
07:49:49 - enableGui -11 + 1 = -10
07:49:49 - HMainTab enable GUI(13),..   SUBITLE EXTRACTION END
07:49:49 - enableGui -10 + 1 = -9
07:49:49 - HResizeTab disable GUI(19),..   AUTOCROP START
07:49:49 - enableGui -9 + -1 = -10
07:49:50 - HFilteringTab disable GUI(8),..                   INTERLACE CHECK START
07:49:50 - enableGui -10 + -1 = -11
07:49:51 - HFilteringTab disable GUI(8),..                   INTERLACE CHECK START
07:49:51 - enableGui -11 + -1 = -12
07:49:52 - HMainTab disable GUI(11),..   SUBITLE EXTRACTION START
07:49:52 - enableGui -12 + -1 = -13
07:49:53 - HResizeTab enable GUI(18),..   AUTOCROP END
07:49:53 - enableGui -13 + 1 = -12
07:49:53 - HMainTab enable GUI(13),..   SUBITLE EXTRACTION END
07:49:53 - enableGui -12 + 1 = -11
07:49:53 - HResizeTab disable GUI(19),..   AUTOCROP START
07:49:53 - enableGui -11 + -1 = -12
07:49:54 - HFilteringTab disable GUI(8),..                   INTERLACE CHECK START
07:49:54 - enableGui -12 + -1 = -13
07:49:55 - HFilteringTab disable GUI(8),..                   INTERLACE CHECK START
07:49:55 - enableGui -13 + -1 = -14
07:49:56 - HResizeTab enable GUI(18),..   AUTOCROP END
07:49:56 - enableGui -14 + 1 = -13
07:50:30 - HFilteringTab enable GUI(9),..                       INTERLACE CHECK END
07:50:30 - enableGui -13 + 1 = -12

This would also explain why your analysis is so fast,..

[Ahari]

I'm glad I am not a programmer - you have to be really good at puzzles!

Btw: My machine is very fast and uses all cores at up to 95% in auto turbo mode (up to 4500Mhz) when processing this stuff. It is a brand new custom build and really kicks some serious butt!! I am in awe of the new technology! My old machine was steam powered

[Selur]

Okay, I think I know why this happens,.. and how to fix it.
At the moment at the end of the basic analysis Autocrop, Interlace Checking and Subtitle Extraction is triggered for each source.
In parallel but for only one source at a time, as long as the subtitle extraction is the slowest, everything is fine, but if the interlaced check takes longer, it gets canceled because the input source gets changed. This is the bug. The switching of the input source should wait for the interlaced check to be finished first.
-> I changed the code such that the source switching routine will check each second if the interlaced routine is running and only switch the source, when the interlaced routine stopped. This should fix the problem, but since like I mentioned earlier, the interlace check takes a huge a mount of time the total analysis time will increase accordingly.

Cu Selur

[Ahari]

You're getting there!

Tried the PM'd version:
The command window opens and stays open when I open Hybrid.

Interestingly it says that it is checking if ffmpeg is 32/64bit and then decides after a few seconds that it is 64bit! It carries on. I checked and my version is the 32 bit one!?

I then enabled my global settings, enabled the debug log and threw a folder at it.

During the analysis (which does now take much longer) the GUI was re-enabled, but then audio analysis started and the GUI was left disabled when it finally stopped.

Here is the debug log:
http://dl.dropbox.com/u/3334489/Hybrid/HybridDebugOutput%20%284%29.txt

[Selur]

Interestingly it says that it is checking if ffmpeg is 32/64bit and then decides after a few seconds that it is 64bit! It carries on. I checked and my version is the 32 bit one!

Okay, than I have to recheck the ffmpeg 32/64bit check,.. (but that does not concern the current problem, this is only interesting if one needs to use ffmpeg with avisynth)

----

-> will look at the debug output in a bit,.. (working on something else atm)

[Ahari]

Just to be sure I had the 32 bit version, I went to the ffmpeg download site and got the latest versions of both 64bit and 32 bit (released yesterday).

I tried the new 64 bit version, but it still crashes Hybrid!

I copied the new 32 bit version into the hybrid folder, but Hybrid still says it is 64 bit in the command window that stays open.

[Selur]

I tried the new 64 bit version, but it still crashes Hybrid!

problem is that you are the only one I ever saw having a problem with ffmpeg 64bit (is your system overclocked?) and I can't make a proper bug report since I can't really reproduce it on any of my machines,..

[Ahari]

I made absolutely sure that the new downloaded version of ffmpeg is the 32 bit one, but Hybrid still reports that it is the 64 bit one.

Funny thing is the whole issue we had with the 64bit ffmpeg is back again - namely Hybrid's GUI crashing when adding a file with more that one audio track!

I tried with both the old ffmpeg (32bit) and the new one (also 32bit).

Something fishy is going on here!?

I am now using the new release of ffmpeg (32bit). Of course it still crashes as explained above.

[Selur]

I am now using the new release of ffmpeg (32bit). Of course it still crashes as explained above.

Okay, now I'm totally confused,..
The 32bit ffmpeg also crashes for you? (I thought only the 64bit version did,..)

about the detection: I already wrote, that I will look at it,...

[Ahari]

(is your system overclocked?)

No. ===>Well, I suppose in theory it may be overclocked a bit as the auto turbo function does slightly overclock the default CPU speed when needed (I think).

I suppose it must be boosting performance somehow with the turbo speed and my CPU monitor does report 4400mhz when processing intensive stuff (the chip is sold as a 3400mhz chip - or therabouts) and then drops right down to 1600mhz for mundane stuff like typing.

[Selur]

Okay, normaly turbo from the bios/cpu should not cause the problem,..

[Ahari]

Okay, now I'm totally confused,..
The 32bit ffmpeg also crashes for you? (I thought only the 64bit version did,..)

So did I!! It was working fine the other day, but when I noticed the Hybrid command window saying that it wasn't sure what version ffmpeg it was and then decided that it was 64bit (even when it wasn't), I thought I would check it again.

I will play a bit more to try and track down what version works! Just got some things to do first.

"I'll be baaack!"

[Ahari]

Okay, normaly turbo from the bios/cpu should not cause the problem,..

It is a standard auto tuning thing that the bios did when I set up the machine. It went and did it's thing rebooting the machine 3 or 4 times and then auto setting the bios to do this. Don't know exactly how or what it did, but the machine is rock solid and very cool (only 54 deg C flat out at 100% on all cores!) Right now the CPU is idling along at a very happy 21 deg C and the fans are turning really slowly!

I never told it to do anything extreme - just used the "normal" settings. I like fast, but not at the cost of very expensive gear!!

[Ahari]

I don't understand the Turbo function fully, but I think it is based on Intel turbo or something. It can also boost the graphics card, but I left that alone!

There is another thing I am using called Lucid Virtu technology (part of the motherboard software) which uses the graphics card and the CPU in built graphics chip to help share processing things like video conversion - even comes with some of it's own conversion software, but I haven't played with that yet (I've got Hybrid ). Haven't a clue how Lucid works, but it seems to!

I am also using Intel Rapid Storage Technology (IRST) which uses an SSD together with a hard drive to give SSD speed to frequently used programs, while also having the capacity of a hard drive available. Really cool stuff!

I have been using all this stuff for a month or so without a single problem, so I really doubt that could be the problem.

Then again, no one else seems to be having this problem so who knows?...

[Ahari]

Okay, now I'm totally confused,..
The 32bit ffmpeg also crashes for you? (I thought only the 64bit version did,..)

So did I!! It was working fine the other day, but when I noticed the Hybrid command window saying that it wasn't sure what version ffmpeg it was and then decided that it was 64bit (even when it wasn't), I thought I would check it again.

I will play a bit more to try and track down what version works! Just got some things to do first.

"I'll be baaack!"

I got it working again!!

With all this fiddling with subtitles, I thought I don't really need Hebrew subtitles - I only had them in a few videos, just for fun anyway! Of course the videos that I am playing with have them!

So, late last night or this morning, I spotted the setting to only extract English subtitles and thought: "What a good idea! ). So I selected it (Config>Input settings>All subtitle - extract/add>only add/extract: and clicked on English).

I didn't think anymore about it.

Anyway, when I unticked that box a few minutes ago, everything started working again!! - except the folder issue!!

So that is something else to look at - boy trouble just seems to find me. Maybe I just try to tweak everything too much. It must be the perfectionist in me always trying out everything and trying to get the best out of everything!! It drives the wife nuts!

So for now I am back with the latest PM'd version with the latest ffmpeg (32 bit) and everything but the folder issue (and new subtitle thing) is working!

[Ahari]

Funny thing just happened!! I tossed the latest 64 bit version of ffmpeg into my Hybrid folder and the video with 3 audio tracks and 4 sub titles processed perfectly!! That was the same video that gave us all the trouble earlier.

Maybe ffmpeg 64 bit is fixed? If so - yayy! One less thing to worry about... I was thinking that maybe since Hybrid seems to think it was a 64 version anyway and I found that "subtitle english only" switch, why not give it a try, and it worked! Will double check everything again to make sure.

The folder still stops the GUI though.

[Selur]

The folder still stops the GUI though.

I know and I'm working on it,...

[Ahari]

I know and I'm working on it,...

I know - I was just summarising. You are doing a great job

I am pleased with the 64 bit thing though... one less thing to keep in the back of my brain every time I update in the future! I checked it again and it is working beautifully! Must be something with that "english only switch" and the new version of ffmpeg. As soon as I select that switch, Hybrid crashes on analysis.

Good luck with the folder thing...

I'm off for lunch.

[Ahari]

It works! You're the MAN!!



No crashes. No freezing. No funny windows popping up. No files left lying about... Awesome!

Now I can play a bit at last!

A quick check on the output - the first video was perfect, but the second one was auto cropped only at the top of the video.

Will do some more checking later.

[Ahari]

Just saw the new PM version. will give it a go later...

[Selur]

regarding cropping: using a higher threshold might help,... can't really say more without a small sample where the crop detection fails,..

[Ahari]

What program does the autocrop? I have no idea what each setting does.

Also, when I open just the video on its own, then the autocrop is perfect. It only gets "lost" when the whole folder is done together.

[Selur]

Hybrid rev 2012.05.10.1:

*fixed*

• audio: ac3 sample rate/channel count/bitrate combinations, before some non-valid combinations were offered.
• analysis: automation - subtitle extraction + interlaced check could cause to Hybrid to stay grayed out
• subs: Hybrid now automatically creates necessary mplayer/fontconfig files; otherwise subtitle preview and embedding does not work (mac only)
• muxing: timecode handling for vfr->vfr

*changed*

• analysis: speed up interlaced analysis by adding lavthreads

*added*

• input: option to specify which files Hybrid will support as input (by extension)

-> downloads: http://www.selur.de/downloads

Cu Selur

[Selur]

What program does the autocrop?

MPlayer,.. try with the latest version of Hybrid (added a autocrop cache)

cropdetect[=limit:round[:reset]]
Calculates necessary cropping parameters and prints the recommended parameters to stdout.
<limit> Threshold, which can be optionally specified from nothing (0) to everything (255) (default: 24).
<round> Value which the width/height should be divisible by (default: 16). The offset is automatically adjusted to center the video. Use 2 to get only even dimensions (needed for 4:2:2 video). 16 is best when encoding to most video codecs.
<reset> Counter that determines after how many frames cropdetect will reset the previously detected largest video area and start over to detect the current optimal crop area (default: 0). This can be useful when channel logos distort the video area. 0 indicates never reset and return the largest area encountered during playback.

source: http://www.mplayerhq.hu/DOCS/man/en/mplayer.1.txt
(if you want to the the command line Hybrid uses, enabled the debug output and look into the debug file )