VC-1 to H264, best method & settings. Quality is #1 priority vs file size

[TechHero]

I have some lossless Blu-Ray rips (from discs I own that I want to keep on my server with metadata & all in one place). I am a Mac user, so I currently use "Make MKV" to rip the AVC / H264 stream and lossless audio to MKV, then I use Subler to mux the file into M4V, leaving video stream untouched.

The problem is I have quite a few Blu-Ray rips that are encoded in "VC-1" and you can't just mux those, they must be transcoded to H.264.

I've been researching the HIGHEST QUALITY way to transocde VC-1 to H264 but all I can find it references to using Handbrake or "iVI", both which use x264 to transcode. Problem is when I use the "lossless" option in iVI, it doesn't seem to be of very high quality regarding BFrames, Ref Frames, and Motion Estimation. When I read about Handbrake, most people are converting DVDs or really don't care much about super-high quality.

So far this is what i put together in Handbrake:
CRF: 14
Framerate: Same as source
Reference Frames: 3
Maximum B-Frames: 3
CABAC: Checked
8x8: Checked
Weighted P-Frames: Checked
Pyrmidal B-Frames: Off
Adaptive B-Frames: Optimal
Adaptive Direct Mode: Spatial
Motion Estimation Method: Uneven Multi-Hexagon
Subpxel ME & Mode Decision: 9: RD refine in all frames
Motion Estimation Range: 32
Partition Types: Default (most)
Trellis: Encode Only
Deblocking: 0
Picture Filters: OFF
Cropping: NONE
Anamorphic: Strict.

I've read a lot about the VC-1 spec that seems to imply it uses 1-2 ref frames and b-frames at most. Is using 3 ref frames and 3 b-frames in this scenario overkill? Also, what's the opinion on Pyramidal B-Frames in this situation? What about Motion Estimation Range and Subpixel ME settings? Also, is Trellis necessary in this case?

Since I'm basically attempting to keep as much of the quality as possible, not really caring about file sizes, can anyone recommend something else, or revised Handbrake settings to convert a full quality VC-1 Blu-Ray rip to as close to full quality H264 as possible? I tried CRF: 1 but the output file size was nearly double the source, plus I'm still unsure about the other settings. Most of what I read only recommends CRF of 18-22 which to me is too low quality.

Thanks!

[Selur]

The problem is I have quite a few Blu-Ray rips that are encoded in "VC-1" and you can't just mux those..

I guess your player doesn't support VC-1 inside mp4.
VC-1 is a valid, since SMPTE RP 2058-4-2011 iirc., video stream for the mp4 container. (supported by e.g. L-Slash source: https://code.google.com/p/l-smash/ win binary: http://x264.fushizen.eu/builds/l-smash/)

Is using 3 ref frames and 3 b-frames in this scenario overkill?

No

Also, what's the opinion on Pyramidal B-Frames in this situation?

If your playback software has not problem with it, use it.

What about Motion Estimation Range and Subpixel ME settings?

The higher the better (Motion Estimation Range above 64 doesn't help enough to be normally worth the effort) and the slower the encoding.

Also, is Trellis necessary in this case?

Trellis is required is you use sub-pixel precision > 9.

or revised Handbrake settings to convert a full quality VC-1 Blu-Ray rip to as close to full quality H264 as possible?

lossless x264 encoding, would be the best, but that a. takes a lot of space and b. is probably not supported by you player

I tried CRF: 1 but the output file size was nearly double the source, plus I'm still unsure about the other settings.

the lower the CRF/CQ the more unimportant the other settings get, if file size is not important

Most of what I read only recommends CRF of 18-22 which to me is too low quality.

personally I go for 15-18 normally, so if a lower CRF than 18 is fine for you (quality&space wise), use it

Cu Selur

[jagabo]

Around CRF=12 is where x264 encoding becomes nearly transparent for almost all material (and is roughly the quality used on most Blu-ray discs). Very clean sources can look pretty good around CRF=18. You can use the table in the following post to see what settings are used by the presets:

https://forum.videohelp.com/threads/349386-Target-bitrate-for-720p-%28From-Blu-ray-sour...=1#post2188503

Settings above "slow" will make very very little difference in most material, but will require much more encoding time.