Boxee Box issue ... need to convert DV AVI to H264

[njitgrad]

According to Boxee Box tech support, my DV AVI files are not supported but for some strange reason the device attempts to play the files anyway resulting in jumpy video and audio being out of sync.

Last night I decided to take a stab at my first attempt to convert a DV AVI file to H264 MKV, because they indicated this would be my best chance at getting playback.

First of all, would this codec and container be considered the next best codec/container that Boxee Box will support? File size is not an issue for me. I want the best quality possible at all costs.

I used the trial version of AVS Video Converter to convert a 5 minute DV AVI clip (~ 1GB).
The application was very simple to use. I used all default settings for my first file, except that I maxed out the bit rate at 16000...I figured the higher the bit rate, the higher quality the image.

It produced a 600MB MKV file that the Boxee Box played back with no problem. The quality was pretty OK, but there were a few observations I made:

1) It did not fill my HDTV screen in either direction. The footage appeared as a 16:9 rectangle in the middle of the screen, probably occupying about 1/2 to 2/3 of the total surface area of the 46" screen. The footage was shot at 16:9 so I would have expected it it fill the screen.

2) The primary difference between the AVI file and MKV file (when both are played on my PC) is that the former appears a bit clearer, while the latter appears a bit grainier. However in the AVI file you can see very very faint but noticeable vertical lines (which I assume is due to interlacing) whereas in the MKV file I see nothing like it. It is more noticeable in the AVI when there is camera or subject motion.

3) The color tone of the AVI file was definitely more vibrant. The MKV file appears duller. Is there a way to improve this?

I then proceeded to experiment making a 2nd MKV file, this time the only thing I changed from the previous attempt was the dimensions of the output. Rather than using defaults (the size of the source), I specified 1920x1080.

The resulting file was only minimally larger (< 1 MB) but this time it filled my TV screen completely. The only difference between this file and the 1st MKV file appeared to be the graininess, likely due to the fact that the original graininess was just being magnified.

Can anyone recommend anything I can do to improve the quality of my conversion or is this the best that I can get? I want to use the best possible conversion even if it requires the purchase of a converter application.

Below are the MediaInfo details for my AVI file, the 1st MKV and the 2nd MKV I created.

Thanks.

AVI FILE
General
Complete name : G:\Boxee AVI\2009.04.11 - Easter Egg Hunt.avi
Format : AVI
Format/Info : Audio Video Interleave
Format_Commercial_IfAny : DVCPRO
Format profile : OpenDML
File size : 1.07 GiB
Duration : 5mn 3s
Overall bit rate : 30.3 Mbps
Recorded date : 2009-04-11 19:08:40
Video
ID : 0
Format : DV
Format_Commercial_IfAny : DVCPRO
Codec ID : dvsd
Codec ID/Hint : Sony
Duration : 5mn 3s
Bit rate mode : Constant
Bit rate : 24.4 Mbps
Width : 720 pixels
Height : 480 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 29.970 fps
Standard : NTSC
Color space : YUV
Chroma subsampling : 4:1:1
Bit depth : 8 bits
Scan type : Interlaced
Bits/(Pixel*Frame) : 2.357
Stream size : 1.02 GiB (95%)
Encoding settings : ae mode=full automatic / wb mode=automatic / white balance= / fcm=manual focus
Audio
ID : 1
Format : PCM
Format settings, Endianness : Little
Format settings, Sign : Signed
Codec ID : 1
Codec ID/Hint : Microsoft
Duration : 5mn 3s
Bit rate mode : Constant
Bit rate : 1 536 Kbps
Channel(s) : 2 channels
Sampling rate : 48.0 KHz
Bit depth : 16 bits
Stream size : 55.7 MiB (5%)
Interleave, duration : 467 ms (13.99 video frames)

1st MKV FILE
General
Complete name : G:\Boxee AVI\2009.04.11 - Easter Egg Hunt 720x480.mkv
Format : Matroska
File size : 585 MiB
Duration : 5mn 3s
Overall bit rate : 16.2 Mbps
Encoded date : UTC 2010-11-23 04:11:50
Writing application : AVSMatroskaFile
Writing library : libebml v0.7.7 + libmatroska v0.8.1
Video
ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : Main@L5.1
Format settings, CABAC : Yes
Format settings, ReFrames : 1 frame
Codec ID : V_MPEG4/ISO/AVC
Duration : 5mn 3s
Bit rate : 15.6 Mbps
Width : 720 pixels
Height : 480 pixels
Display aspect ratio : 3:2
Frame rate : 29.970 fps
Standard : NTSC
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 1.510
Stream size : 567 MiB (97%)
Audio
ID : 2
Format : MPEG Audio
Format version : Version 1
Format profile : Layer 3
Codec ID : A_MPEG/L3
Codec ID/Hint : MP3
Duration : 5mn 3s
Bit rate mode : Constant
Bit rate : 192 Kbps
Channel(s) : 2 channels
Sampling rate : 44.1 KHz
Stream size : 6.96 MiB (1%)
Writing library : LAMELAME

2nd MKV FILE
General
Complete name : G:\Boxee AVI\2009.04.11 - Easter Egg Hunt 1920x1080.mkv
Format : Matroska
File size : 586 MiB
Duration : 5mn 3s
Overall bit rate : 16.2 Mbps
Encoded date : UTC 2010-11-23 04:46:48
Writing application : AVSMatroskaFile
Writing library : libebml v0.7.7 + libmatroska v0.8.1
Video
ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : Main@L5.1
Format settings, CABAC : Yes
Format settings, ReFrames : 1 frame
Codec ID : V_MPEG4/ISO/AVC
Duration : 5mn 3s
Bit rate : 15.7 Mbps
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate : 29.970 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.252
Stream size : 567 MiB (97%)
Audio
ID : 2
Format : MPEG Audio
Format version : Version 1
Format profile : Layer 3
Codec ID : A_MPEG/L3
Codec ID/Hint : MP3
Duration : 5mn 3s
Bit rate mode : Constant
Bit rate : 192 Kbps
Channel(s) : 2 channels
Sampling rate : 44.1 KHz
Stream size : 6.96 MiB (1%)
Writing library : LAMELAME

[njitgrad]

I guess my post was too long-winded.

Lets start with a simple question: bit rate.

I selected 16000 as my bit rate, because it is the max value allowed by the application.

If I had selected 15000 instead, I assume the file would be slightly smaller, but does that translate to any loss in quality?

Basically what I want to know is if there is an ideal video bit rate such that anything higher would be unnoticeable?

[jagabo]

file size = bitrate * running time (note that bitrate includes both audio and video bitrate)

So using 15000 instead of 16000 will give you a smaller size. The smaller you make the file the lower the quality will be. Exactly how low you can go before the quality loss becomes obvious depends on the codec you use, the settings used, and the nature of the video. Handheld camcorder video usually won't compress as well as professionally shot film.

I don't know the program you're using but try Xvid4PSP. It's free, easy to use, and works very well. Try one of the "x264 Q18" presets. Those are constant quality encodings -- you pick the quality (Q18 is very good, nearly indistinguishable from the source, the smaller the value the better the quality) and the encoder uses whatever bitrate is needed at each frame to deliver that quality.

[njitgrad]

file size = bitrate * running time (note that bitrate includes both audio and video bitrate)

So using 15000 instead of 16000 will give you a smaller size. The smaller you make the file the lower the quality will be. Exactly how low you can go before the quality loss becomes obvious depends on the codec you use, the settings used, and the nature of the video. Handheld camcorder video usually won't compress as well as professionally shot film.

I don't know the program you're using but try Xvid4PSP. It's free, easy to use, and works very well. Try one of the "x264 Q18" presets. Those are constant quality encodings -- you pick the quality (Q18 is very good, nearly indistinguishable from the source, the smaller the value the better the quality) and the encoder uses whatever bitrate is needed at each frame to deliver that quality.

If I choose a higher bit rate than the source file, will it make the playback better, worse, or no effect?

[jagabo]

I don't know how the Boxee Box handles very high bitrate files. h.264 (x264) encoding is much more efficient than DV encoding so it won't require nearly as much bitrate. I ran a 151 MB interlaced DV AVI file through Xvid4PSP with "x264 Q18 Fast" preset and it generated a 22 MB MKV file that was nearly indistinguishable from the source. If you don't like the preset you can fine tune it to your liking (eg, turn the quantizer down to get even higher quality).

Using a higher bitrate than your source will not improve its quality.

You should not throw away your original DV AVI files. They should be your masters in case you need to go back to them someday.

[njitgrad]

I don't know how the Boxee Box handles very high bitrate files. h.264 (x264) encoding is much more efficient than DV encoding so it won't require nearly as much bitrate. I ran a 151 MB interlaced DV AVI file through Xvid4PSP with "x264 Q18 Fast" preset and it generated a 22 MB MKV file that was nearly indistinguishable from the source. If you don't like the preset you can fine tune it to your liking (eg, turn the quantizer down to get even higher quality).

Using a higher bitrate than your source will not improve its quality.

You should not throw away your original DV AVI files. They should be your masters in case you need to go back to them someday.

Jagabo,

Last night I took your advice and experimented with several flavors of the same file using Xvid4PSP. Out of three containers (MKV, TS, and M2TS) I found that Boxee Box worked best with MKV files. All three containers resulted in decent "normal" playback. The MKV format won out because it was the only format of the three that the BB could fast-forward and rewind without hiccups.

Once I narrowed it down to MKV, I then experiment with several of the Q18 presets. To my surprise, they all seem to produce roughly the same file size, its just that some of the Q18 presets are quick to run, others take MUCH longer to run. However, they all seemed to produce the same quality video, which again surprised me.

What I would like to know is what specific settings (or preset) I need to use in order to produce an MKV file that most closely resembles my source DV AVI file. The MKV files I created with the Q18 settings were MUCH smaller than the DV AVI files. Keep in mind that my goal is not to save on disk space, its to maximize quality even if it means converting a single file takes several hours. Some of earlier my camcorder footage (prior to the acquisition of a video lamp) is a bit grainy, as evidenced in my DV AVI files. I need to make sure that my conversions with H.264 are as similar to the source DV AVI as possible. I do not expect perfect quality.

On the same topic, I did not see any settings for bit rate. Or is it hidden in some other menu?

Also, question about the "insane" presets...why are there several? There is a Q18 insane preset and Q22 insane preset. What makes one more insane than another?

I am much obliged for all your help.

[jagabo]

There are two basic methods of encoding:

1) Bitrate. You select the bitrate and the encoder gives you whatever quality it can for that bitrate. The more time the encoder spends working on the encoding the better the quality will be. But the bitrate (and hence file size) will be the same. In general, high bitrates give high quality, low bitrate give low quality.

2) Constant Quality. You select the quality and the encoder uses whatever bitrate is needed to give you that quality. The more time the encoder spends workiing on the video the smaller the resulting file will be. The quality will be the same. In general, hiqh quality results in high bitrates, low quality results in low bitrates.

These are basically two sides of the same coin. If you were to encode a video at Q18 and got a file that averaged 5000 kbps, then performed a two pass bitrate encoding with an average bitrate fo 5000 kbps, the two resulting videos would be about the same size and quality. The basic rule is to use quality based encoding when you care more about the quality than the file size, use bitrate based encoding you care more about the file size than the quality.

You get diminishing returns with encoding time. Going with a setting of Q18 HQ rather than Q18 Fast will get you a little more quality but it might take twice as long to encode. Going with Q18 Insane may take 10 times longer to encode but will only get you a tiny bit more quality than Q18 HQ. During the longer encodings x264 is looking for more ways to compress each frame, searching wider areas for motion vectors and over more frames.

The Q modes are constant quality. The smaller the Q value the higher the quality you will get. Q18 is very high quality but if you want to get higher quality customize the settings (the little gear icon next to the video encoding pulldown) and select a smaller quantizer. You can go down all the way to 0 to get lossless encoding but the file will be gigantic -- larger than your DV AVI file. You'll probably barely see any difference between Q15 and below.

If you want to make your files a little easier to seek you can go with smaller GOP sizes. The default max is 250. Reducing that to 100 or 50 will increase the file size a little bit but seeking will be faster during playback.

If you need to correct the aspect ratio go to the CLI tab (in the custom settings) and add the --sar command:

NTSC 4:3: --sar 10:11
NTSC 16:9 --sar 40:33
PAL 4:3: --sar 12:11
PAL 16:9: --sar 16:11

[njitgrad]

You can go down all the way to 0 to get lossless encoding but the file will be gigantic -- larger than your DV AVI file. You'll probably barely see any difference between Q15 and below.

That's what exactly what I wanted to know. This morning before I left for work I queued up a Q15, Q10, and Q5 to see the differences. I'm hoping by the time I get home they'll be done.

Can you explain the lossless preset to me in more detail? Sounds very intuitive but I just wanted to make sure. Sounds likes its more or less a re-packaging of the footage in my DV AVI file into an MKV H264 file. When I did this yesterday it created the file rather quickly. Does that sound about right? Does lossless imply the best possible copy of the original, or are there other factors? This is a long shot but could you actually improve the picture? As you may recall some of my footage is grainy due to low light conditions.

Also...why are there different types of lossless presets? If memory serves, there's at least one for Q18 and Q22. Doesn't lossless imply Q0? Maybe I'm just misremembering what the presets are. When I get home tonight I'll check again.

[jagabo]

Lossless is lossless. But to convert DV to x264 the DV must first be decompressed. That will cause it to grow about 5x in size (this is done internally by the software, frame by frame while converting, not saved to an intermediate file). From there the x264 lossless compression will be bit for bit identical when decompressed for display. With some exceptions: An NTSC DV source may com out of the DV decoder as YUY2 or RGB video. That will be converted to YV12 for x264 encoding. That reduces the vertical color resolution from 480 lines to 240 lines. YUY2 has a grayscale resolution of 720x480 and color resolution of 360x480 (NTSC DV has color resolution of 180x480). YV12 has has a grayscale resolution of 720x480 and color resolution of 360x240.

I only see one lossless preset in the Video Encoding pulldown. You can customize the setting but they don't make much difference in file size.

As a quick experiment I encoded the earlier referenced 151 MB DV AVI file as lossless x264 and it grew to 238 MB. My WDTV Live wouldn't play it. I doubt the Boxee Box will either.

[njitgrad]

Lossless is lossless. But to convert DV to x264 the DV must first be decompressed. That will cause it to grow about 5x in size (this is done internally by the software, frame by frame while converting, not saved to an intermediate file). From there the x264 lossless compression will be bit for bit identical when decompressed for display. With some exceptions: An NTSC DV source may com out of the DV decoder as YUY2 or RGB video. That will be converted to YV12 for x264 encoding. That reduces the vertical color resolution from 480 lines to 240 lines. YUY2 has a grayscale resolution of 720x480 and color resolution of 360x480 (NTSC DV has color resolution of 180x480). YV12 has has a grayscale resolution of 720x480 and color resolution of 360x240.

I only see one lossless preset in the Video Encoding pulldown. You can customize the setting but they don't make much difference in file size.

As a quick experiment I encoded the earlier referenced 151 MB DV AVI file as lossless x264 and it grew to 238 MB. My WDTV Live wouldn't play it. I doubt the Boxee Box will either.

Good info. So what would you suggest for my purpose? A preset, possibly modified by some specific configuration values? I will see what happens with a lossless file tonight in terms of playback on the BB.

[edDV]

Are these DV-AVI files from a camcorder?

A key issue for h.264 conversion is deinterlace or don't deinterlace. What I have found when quality is the objective and interlace camcorder DV is the source, interlace compression to h.264 does not yield much better compression than MPeg2. MPeg2 has the advantage that it can also be used for DVD.

If you deinterlace, picture quality issues can be severe depending on the type of camcorder video. Tripod stable video does well, hand held is problematic. Any noise in the picture will impact the quality of the deinterlace.

In general I like to keep DV camcorder material interlace and mostly use MPeg2 at > 8 Mb/s. Similar quality interlace h.264 is achieved around 6 Mb/s. A quality DVD player or HDTV will do a better deinterlace job than all but the most time consuming software deinterlacers. I'm not sure how the Boxee box handles interlace source. Ideally it will pass 480i to your HDTV for deinterlace and upscale.

[njitgrad]

I've confirmed that Boxee Box will not play lossless. However it did play my Q10 and Q5 version of my MKV files.
To create these files I simply selected the Lossless preset first and then only changed the Quantizer value (to 10 in one case, then 5 in another).

I noticed a significant quality difference between the Q18 and Q10 files.
I also notificed a quality difference between the Q10 and Q5 files.


My source DV AVI was 222MB (~1 min duration)
The various attempts at Q18 resulted in files approx 50MB in size give or take.
The Q10 was about 165MB.
The Q5 was about 228MB (not much larger than the source).

I may end up sticking with the Q5 files because they played just fine (including navigation forwards and back), they're about the same size as my source, and because they appear to be better than the Q5 files.

Now I'd like to find out what I can do to give the playback a little pop. The colors seem dull. When I selected vivid color correction, the colors were over-exaggerate. Any pointers here?

Also, do I need to change the resolution and aspect ratio setting? My source was 720x480 (16:9)...should I change my output resolution and ratio? There's also a choice for resize filter. OMG, there are just SO MANY settings I don't know what to do.

[edDV]

...
Also, do I need to change the resolution and aspect ratio setting? My source was 720x480 (16:9)...should I change my output resolution and ratio? There's also a choice for resize filter. OMG, there are just SO MANY settings I don't know what to do.

DV-AVI is non-square pixel (like DVD) for either 4:3 or 16:9. Not a problem for players that respect aspect ratio flags. If the player doesn't then square pixel conversion is required. For 4:3 that would be 640x480 (a nice divide by 16 in both H and V). For 16:9, 852 does not divide by 16. You need to fudge aspect ratio a bit to 848x480 or 864x480. If you go for deinterlace (not recommended for camcorder, broadcast or VHS source) then you can play with vertical rescales such as 720x400 (divisible by 16).

[jagabo]

In Xvid4PSP use Video -> Resolution/Aspect. Set the Input Aspect Ratio to 16:9. Set the Output Aspect Ratio to 16:9. Leave the resolution unchanged. The Boxee Box should play that properly.

After selecting the Vivid color correction use the custom tool to tone down the saturation.

[nimco]

njitgrad, it is been awhile you were experimenting with x.264 settings to transcode DV to H264.
Can you share about your latest achievements, did you get satisfied results?

I was working/experimenting on the same task - play my DV from camcorder on WDTVLive and possibly use encoded H264 files as a long term backup files.

I was playing with Mainconcept Reference HD decoder, experimented with several different settings.
What I found that by using X.264 encoder it coud be possible to achieve a higher PQ results than Mainconcept, since it uses H.264 type encoder.
I found that the best results are by keeping video as interlaced, encoding at 1920x1080 as 59.94fps.

[njitgrad]

njitgrad, it is been awhile you were experimenting with x.264 settings to transcode DV to H264.
Can you share about your latest achievements, did you get satisfied results?

I was working/experimenting on the same task - play my DV from camcorder on WDTVLive and possibly use encoded H264 files as a long term backup files.

I was playing with Mainconcept Reference HD decoder, experimented with several different settings.
What I found that by using X.264 encoder it coud be possible to achieve a higher PQ results than Mainconcept, since it uses H.264 type encoder.
I found that the best results are by keeping video as interlaced, encoding at 1920x1080 as 59.94fps.

I got distracted and haven't had a chance to get back to it. Can you give details on your work flow? I assume you're also using DV AVI footage you captured from a Sony camcorder.

[_Al_]

I would listen to edDV what he has to say,

keep it mpeg2, keep it interlaced, use the best encoder you can get (Procoder- excellent mpeg2 encoder, then perhaps MainConcept), CBR, 8000kbps, if you don't intend to make any DVD's out of it even more

H.264 needs to be put into 50p and square pixel, otherwise you get some kind of crappy and confusing result. So you need bob deinterlace to 50p from 50i and resize. Thats why mpeg2 is preferred in this case in my opinion. This is doable job, I have tested this myself: http://www.ulozto.cz/5370383/dv-deinterlace.zip no resize though, just tried to encode it through Avisynth, some TempGaussMC_scripts with NNEDI with some clean up of video, I got it from DVinfo forum then, and MeGui (x264 encoder) but too much hassle for a tons of videos.
Still hoping for a device that will play DVavi (DV25 codec, from our old good camcorders), perhaps Xtreamer Ultra, who knows, I have backed up mpeg2 files so no hurry.

Do not try to encode and upscale, not worthy, hardware upscalers do it for free, and perhaps do a better job.

[njitgrad]

keep it mpeg2, keep it interlaced, use the best encoder you can get (Procoder- excellent mpeg2 encoder, then perhaps MainConcept), CBR, 8000kbps, if you don't intend to make any DVD's out of it even more

I'd certainly like to try MPEG-2 but even if I like these encoders' trial-versions, I am in no position to dish out $400-$500 to get the full version. Any suggestions?

[jagabo]

HcEnc and HcGUI are free.

[edDV]

MPeg2 720x480i works best for me with DV source for either DVD or media players.

VC-1 interlace also is nice but only at high bit rate. Same goes for interlace h.264. Might as well use MPeg2 and be DVD compatible.

[_Al_]

I'd certainly like to try MPEG-2 but even if I like these encoders' trial-versions, I am in no position to dish out $400-$500 to get the full version. Any suggestions?

as was said, HC Encoder, Install Avisynth, load this Avisynth Script into HC Encoder:
AviSource("C:\yourvideo.avi")
converttoyv12(interlaced=true)

16:9, make DVD compliand, (to be sure perhaps), bit rate - max 8200kbps, average 7800, not sur if you have to set interlace manually or not. It will make m2v video

AC3 sound (perhaps 224kbps) you will generate elsewhere , Aften Gui, whatever works

Put that m2v and AC3 into IfoEdit (DVD Author - Author New DVD) or put them into MkvMerge Gui to mux mkv, in this case you can mix even your srt subtitles (generated in Subtitle workshop) whatever works for you.

Remember, you still have originals, so you do not judge quality under microscope, it is not top priority, archive those new videos right next those DV avi original files, hopefully, we will be able to play our originals in the future.