if you double the frame rate must you double the bit rate...

[deadrats]

...IF you are going from a less efficient codec to a more efficient codec?

there's a project i've been working on for years, over 10, namely the restoration, for lack of a better word of some circa 90's adult dvd's as well as some greek comedies. both of these types of dvd's share the following in common: they're interlaced, with most of them as some sort of hybrid progressive interlaced encoding, they used low bit rates, almost all are in the 4mb/s range and they also have quite a bit of noise.

i still have the dvd's and intent to keep them but over the years it became an obsession of mine to see if i could clean them up, de-interlace them and end up with a clearer image than the source dvd's. i've tried tons of tools and methods and the two best ways i found of restoring them was to bob de-interlace with a good bobber and use neat video. some years ago i remember reading a post by jagabo were he said that qtgmc was the best de-interlacer bar none.

i eventually settled on using xvid4psp 5.10 because a) it was the only one that seems to detect the interlace patterns corrects and it allows for easy use of qtgmc. i must have done thousands of test encodes using both xvid and x264, with every possible combination of settings one can think off, eventually i settled on tesa, sub me 9, trellis 2, all psy disabled, weighted p and b disabled, 4 b frames and 5 reference frames, b-pyramid = normal, auto, adaptive b frames optimal, a gop of 250, no deblocking, no fast p skip, no dct, deterministic and all partitions disabled and qtgmc set to double the frame rate and slow.

today i was playing around with some of the settings and did a test encode and the results were incredible, i couldn't believe my eyes, i had never achieved such a nice output, easily better than the source.

then i realized that the file size of the encode was twice what it normally is and when i look at the settings i realized that i had inadvertently used double the bit rate of the source.

see, up until now i had reasoned it like this: the source is interlaced mpeg-2, h264 is twice as efficient as mpeg-2, i'm doubling the frame rate ergo i should be using the same bit rate for my h264 encode as the mpeg-2 source.

but after seeing the results i'm wondering if i've been doing it all work, i don't care about size, i only care about achieving the highest quality restoration, does it make sense that if i double the frame rate with qtgmc that i should be using twice the bit rate of the source, regardless of the target codec?

[Selur]

no deblocking <- you are aware that this is a BAD idea, since this disables a major part of H.264 (the inloop-deblocker; this is not a simple deblocker like used for postprocessing, but an integral part of H.264)

does it make sense that if i double the frame rate with qtgmc that i should be using twice the bit rate of the source, regardless of the target codec?

a. the bit rate of the source is in no way an indicator what bit rate is needed to archive a decent result with encoder X
b. the encoder gets uncompressed content as input (its an en- not a transcoder )
c. bob deinterlacing normally adds artefacts (= before not existing 'new' details) and might also smooth the output

Also with modern codecs doubling the frame rate should not require to double the bit rate since:
a. the amount of information that needs to be compressed should not have doubled
b. yes, a bobber adds new 'details', but a lot of these details should be easily compressible due to similarities

Cu Selur

[manono]

but after seeing the results i'm wondering if i've been doing it all work, i don't care about size, i only care about achieving the highest quality restoration, does it make sense that if i double the frame rate with qtgmc that i should be using twice the bit rate of the source, regardless of the target codec?

You might try doing CRF encodes so you don't have to worry about the bitrates needed as compared to a source that used a different codec.

[deadrats]

no deblocking <- you are aware that this is a BAD idea, since this disables a major part of H.264 (the inloop-deblocker; this is not a simple deblocker like used for postprocessing, but an integral part of H.264)

i disable it because of a conversation i had with his darkness and lady mulder over at the dick9 forums a while back. i had offered the suggestion that maybe codecs should be designed/implemented with high quality in-loop denoising and sharpening filters built in and both of them responded that it would kill image quality because the effects of the such an in-loop filter on one frame would be propagated across the entire video, with the artifacts getting progressively bigger as each frame reference a previous frame.

i responded by pointed out that if this was the case the same should hold true for the in-loop deblocking filter and i also pointed out that with a closed gop no frame can reference any frame outside that gop anyway so the effects of the propagation would be limited.

DS responded by ignoring the second part but confirming that indeed the effects of the in-loop deblocking filter do propagate across frames, with the effects on one frame getting more pronounced as the gop progresses.

when you add that to the explanations i have read about deblock that says that positive values smooth out details as the value gets bigger and negative values act like a sharpening filter as the values get lower, coupled with DS' comments about the effects of in-loop filters propagating across frames, i have concluded that deblocking should be disabled for maximum detail retention.

likewise, one of the reasons i disabled all psy optimizations, including aq, the psy brothers and mb-tree is because of comments made by DS and others that sometimes these algorithms may drop small details it would otherwise have retained but it will probably retain details that would otherwise be dropped. i've read a similar explanation about trellis.

to me it seems like almost all of these features are meant for the bit rate starve crowd, the guys that get their jollies by seeing how low the can drop the bit rate before the video is unwatchable; in other words it seems to be a compromise meant for low bit rate situations. since i have a real job and can afford another hard drive (i have 7 terabytes of storage space), i don't feel the need to bit rate starve my encodes, let the water flow baby.

now on a related note, there was a time the xvid4psp gui offered the option for x262 encoding but i've noticed that now it doesn't show up even though it's installed in the folder so i was wondering if a) your software supports x262, b) if it doesn't why no, c) does your software support qtgmc and d) why doesn't your software support only encoding a portion of an inputted video, i.e. as determined by starting and stopping time or frame the way other software like handjob, i mean handbrake does?

basically i'm saying would you consider adding those features to your software?

[deadrats]

You might try doing CRF encodes so you don't have to worry about the bitrates needed as compared to a source that used a different codec.

i know you have undertaken similar projects, i believe pdr once commented that you did the same thing with some indian dvd's, so let me ask you this: what work flow did you finally settle on and what tool? were the results satisfactory or was it a case of "you can't make a silk purse out of a sow's ear"?

with regards to crf, the usual suggestions i have seen for dvd sourced content is to go with crf 22 or somewhere thereabouts but since crf is technically telling the encoder to achieve a certain quality can i just do something like crf 10 and the encoder will only use the bit rate it needs to achieve that quality without overshooting. in other words, is the crf algorithm intelligent enough to say i only need about x amount of bit rate for this quality, so i won't let the bit rate go too much over that or does it throw bit rate at the encode blindly up to the limits defined in bit rate x?

i hope you understand the idea i was trying to express.

[Selur]

a. this makes only sense if you use need lossless compression, otherwise the penalty for not using the inloop deblocker will hurt you more
b. " the explanations i have read about deblock that says that positive values smooth out details as the value gets bigger and negative values act like a sharpening filter" <- where's that explanation, since this does not correspond the how I have the actual code in x264 and the H.264 standard in my head?
c. I agree that psy-ops be wrong and that it might make sense to disable them

> a) your software supports x262, 2.) if it doesn't why no
nope, hybrid does not support x262
a. isn't stable enough to be useful (main reason)
b. I would need to write a mpeg-2 bit rate calculator and have no clue about mpeg-2 container overheads
> c) does your software support qtgmc
yes, if the avisynthExtension is also installed
> d) why doesn't your software support only encoding a portion of an inputted video
it does, if you enabled the 'Cut Support'

> basically i'm saying would you consider adding those features to your software?
only missing feature would be x262, which from my testing and from what I last read on doom9 really isn't usable atm.

[jagabo]

i don't care about size, i only care about achieving the highest quality restoration

Then you would use lossless encoding.

does it make sense that if i double the frame rate with qtgmc that i should be using twice the bit rate of the source, regardless of the target codec?

No. In my experience double the frame rate requires 50 percent more bitrate to achieve the same quality on a frame-by-frame basis. Of course, when frames are flying by at 60 per second you see the problems less so that reduces the bitrate requirement to somewhere between 10 and 50 percent more. x264's CRF mode takes the frame rate into account.

but since crf is technically telling the encoder to achieve a certain quality can i just do something like crf 10 and the encoder will only use the bit rate it needs to achieve that quality without overshooting. in other words, is the crf algorithm intelligent enough to say i only need about x amount of bit rate for this quality, so i won't let the bit rate go too much over

That's the whole point of CRF encoding. Use only the bitrate required to achieve the requested quality.

with regards to crf, the usual suggestions i have seen for dvd sourced content is to go with crf 22 or somewhere thereabouts

I use CRF=18 or less for DVD rips, with film tuning for grainy sources. Since the small frame is going to be enlarged to 1920x1080 on our TV small defects become more noticeable. But even at CRF=18 you can see loss of quality if you know what to look for. At CRF=10 you'd be hard pressed to say which was the source and which was the x264 encoding, even looking at enlarged still frames.

[Selur]

if you really do not care about bit rate and you want to only lose a specify amount of detail, use 1pass constant quantizer (the extreme would be -qp 0 which is lossless)

[manono]

i know you have undertaken similar projects, i believe pdr once commented that you did the same thing with some indian dvd's, so let me ask you this: what work flow did you finally settle on and what tool? were the results satisfactory or was it a case of "you can't make a silk purse out of a sow's ear"?

Mine are quite different from yours, I'm sure. For one thing yours don't have to be unblended, as most of mine do. Most have the levels all screwed up. Some can turn out much improved, others are improved but the sources are so bad that they still look like s**t. In fact, it's kind of discouraging, as in my current project, that I've put in 60 hours on a 2.5 hour long film and it'll always look like garbage - much improved garbage, but garbage nonetheless. It just depends on the source.

I do them for DVD but make MP4s for upload to YouTube as well. All is done with AviSynth scripts, encoded with CCE or XviD4PSP, mostly. Audio worked on in Audacity before encoding to AC3 for DVD with SoftEncode, or whatever XviD4PSP uses to make AAC audio.

[deadrats]

" the explanations i have read about deblock that says that positive values smooth out details as the value gets bigger and negative values act like a sharpening filter" <- where's that explanation, since this does not correspond the how I have the actual code in x264 and the H.264 standard in my head?

i'm going to look for where i read that exact phrasing but in the meantime i would point you to the following discussion and in particular Lord Mulder's response:

http://forum.doom9.org/archive/index.php/t-163182.html

hm how is -3 -3 more aggressive than -2 -2 or -1 -1? doesnt the lower number mean lower deblocking strength so less attempts to use deblocking?

I mean that it is a rather "aggressive" attempt to improve sharpness/details at the cost of blockyness.

and read further down for where DS admits that psy-rd is a "hack", LOL.

there is also this:

http://komisar.gin.by/x264info.html

https://sites.google.com/site/linuxencoding/x264-ffmpeg-mapping

edit: i was googling and googling to find where i read that exact phrasing and then i realized that i got it from one of Lord Mulder's posts, here he says almost the exact same thing:

http://forum.doom9.org/archive/index.php/t-144817.html

There is no better! Higher deblocking means lower risk of visible blocks, but results in smoother (less detailed picture). Lower deblocking results in sharper image, but also means a higher risk to get ugly blocks...

i can't seem to find it but i know for a fact i read it from what was presented as an authoritative source that positive values cause the deblocking filter to act like a smoother resulting in a blurry picture and negative values cause it to act like a sharpening filter and i also know that DS has explicitly said to me that the effects of the filter propagate across the entire gop and he and Lord Mulder claimed that this is true for all filters, regardless of whether they are in-loop or not.

this isn't something i imagined.

[deadrats]

if you really do not care about bit rate and you want to only lose a specify amount of detail, use 1pass constant quantizer (the extreme would be -qp 0 which is lossless)

size is not a factor for me, what i care about is not only not losing any detail but actually enhancing what detail is already there, this is why i was using neat video because it seemed to do just that and qtgmc seems to try and do that as well with the way it deinterlaces, sharpens and denoises.

i really believe that is a codec developers, any codec developer really wanted to make a high quality encoder they would spend considerable amounts of time creating a neat video caliber denoise filter for their encoder and maybe even a really high quality de-interlace filter. and yes i know it's not the job of an encoder to denoise and/or de-interlace content, i've used the same argument myself against x264's psycho visual enhancement algorithms but if we're going to applaud an encoders psy capabilities and judge them by how well they implement them then it seems to me that maybe we should be demanding that a much more basic functionality be added first.

[deadrats]

All is done with AviSynth scripts

do you have a base script you normally go to or is each a custom affair? if you have a basic template you always use can you post it, i want to see if i can adapt it to my uses, maybe just comment out the parts about color correction and the like.