How to correct the video colour range tag

[iKron]

I think mkv file tag has wrong color range information, and the video encoder messing up with the colour


Original
https://forum.videohelp.com/images/imgfiles/Etts94A.png
Mediainfo
https://pastebin.com/0WNdF0fJ

-------------------

Encoded
https://forum.videohelp.com/images/imgfiles/SxkhpGn.png
Mediainfo
https://pastebin.com/758wPqy7

Here is the original file and encoded file (less than 500kb)
https://www103.zippyshare.com/v/1on31Awl/file.html


is there anyway, ffmpeg or other app will analyze the video file and add the correct colour range?

[poisondeathray]

The problem is the video was encoded with handbrake. Handbrake will range compress a full range video to limited range. Reflagging it with a bitstream filter won't help, because the actual values are changed from 0-255 to 16-235 in 8bit code values (or scaled to 64-940 in 10bit instead of 0-1023)

hevc can be reflagged to full without re-encoding , if it was encoded full but flagged limited. (That is not your case)
https://ffmpeg.org/ffmpeg-bitstream-filters.html#hevc_005fmetadata

At this point in time, there is no way to bypass or disable that behaviour by handbrake. It's a shame because many consumer videos these days can be full range to begin with e.g. go pro, action cams....

Don't use handbrake, problem solved. There are many other options

[jagabo]

There's also a 601 vs. 709 colormatrix problem. Though that may be the way your images were prepared. For example, VirtualDub usually uses a rec.601 matrix to display the video, so rec.709 videos display with the wrong colors (greens too bright, reds too dark, etc.).

[iKron]

just a stupid question. is there anyway we can delete the color range data and HB will encode it properly?

[poisondeathray]

just a stupid question. is there anyway we can delete the color range data and HB will encode it properly?

You could change the flag to limited, encode, change the flag to full... You can actually use "fullrange=on" in the HB advanced options and it will flag the output "full" , so no need for the 2nd extra step - it does work

But it's extra work. It's not inplace editing by ffmpeg - you have to stream copy (write another file). You could just just avoid handbrake/vidcoder . There are other reasons to avoid HB, such as the 8bit pipeline. If you feed a 10bit source it gets downgraded to 8bit. If you were going to change the flag with ffmpeg anyways, wouldn't it make more sense to use it to encode all in one go without changing the flag ?

eg.

ffmpeg -i Full_Range.mkv -bsf:v h264_metadata=video_full_range_flag=0 -c copy Full_Range_flagged_limited.mkv

The -bsf:v h264_metadata=video_full_range_flag=0 with modern ffmpeg builds leaves a color range full, and colour_range_Original flag as limited.
Color range : Full
colour_range_Original : Limited
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709

But it works ok to "trick" HB in my testing, and "fullrange=on" in HB tags the output. (Actual levels are full, flagged full)
Color range : Full
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709

This old ffmpeg build by roozhou does not leave the other flag; and the flagged range is clearly only limited . But it's so old that it probably does not support newer formats like AV1, HEVC....I didn't test it farther in HB. But if source file was AVC it should be ok
http://forum.doom9.org/showthread.php?t=152419
Color range : Limited
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709

[iKron]

post edited:

thank you @poisondeathray for giving time to help me.

i used "range=full" for x265. but still there is colour problem.

HB settings:
https://i.postimg.cc/xTxT9p2B/image.png

here is the dl link
https://www38.zippyshare.com/v/bO5nbwZI/file.html

ID                                       : 1
Format                                   : HEVC
Format/Info                              : High Efficiency Video Coding
Format profile                           : Main 10@L4@Main
Codec ID                                 : V_MPEGH/ISO/HEVC
Duration                                 : 30 s 0 ms
Bit rate                                 : 56.8 kb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 25.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0 (Type 0)
Bit depth                                : 10 bits
Bits/(Pixel*Frame)                       : 0.001
Stream size                              : 208 KiB (98%)
Writing library                          : x265 3.5+1-f0c1022b6:[Windows][GCC 10.2.0][64 bit] 10bit
Encoding settings                        : cpuid=1111039 / frame-threads=3 / numa-pools=12 / wpp / no-pmode / no-pme / no-psnr / no-ssim / log-level=2 / input-csp=1 / input-res=1920x1080 / interlace=0 / total-frames=0 / level-idc=0 / high-tier=1 / uhd-bd=0 / ref=3 / no-allow-non-conformance / no-repeat-headers / annexb / no-aud / no-hrd / info / hash=0 / no-temporal-layers / open-gop / min-keyint=25 / keyint=250 / gop-lookahead=0 / bframes=4 / b-adapt=2 / b-pyramid / bframe-bias=0 / rc-lookahead=20 / lookahead-slices=6 / scenecut=40 / hist-scenecut=0 / radl=0 / no-splice / no-intra-refresh / ctu=64 / min-cu-size=8 / no-rect / no-amp / max-tu-size=32 / tu-inter-depth=1 / tu-intra-depth=1 / limit-tu=0 / rdoq-level=0 / dynamic-rd=0.00 / no-ssim-rd / signhide / no-tskip / nr-intra=0 / nr-inter=0 / no-constrained-intra / strong-intra-smoothing / max-merge=3 / limit-refs=1 / no-limit-modes / me=1 / subme=2 / merange=57 / temporal-mvp / no-frame-dup / no-hme / weightp / no-weightb / no-analyze-src-pics / deblock=0:0 / sao / no-sao-non-deblock / rd=3 / selective-sao=4 / early-skip / rskip / no-fast-intra / no-tskip-fast / no-cu-lossless / b-intra / no-splitrd-skip / rdpenalty=0 / psy-rd=2.00 / psy-rdoq=0.00 / no-rd-refine / no-lossless / cbqpoffs=0 / crqpoffs=0 / rc=abr / bitrate=4000 / qcomp=0.60 / qpstep=4 / stats-write=0 / stats-read=2 / cplxblur=20.0 / qblur=0.5 / ipratio=1.40 / pbratio=1.30 / aq-mode=2 / aq-strength=1.00 / cutree / zone-count=0 / no-strict-cbr / qg-size=32 / no-rc-grain / qpmax=69 / qpmin=0 / no-const-vbv / sar=1 / overscan=0 / videoformat=5 / range=1 / colorprim=1 / transfer=1 / colormatrix=1 / chromaloc=1 / chromaloc-top=0 / chromaloc-bottom=0 / display-window=0 / cll=0,0 / min-luma=0 / max-luma=1023 / log2-max-poc-lsb=8 / vui-timing-info / vui-hrd-info / slices=1 / no-opt-qp-pps / no-opt-ref-list-length-pps / no-multi-pass-opt-rps / scenecut-bias=0.05 / hist-threshold=0.03 / no-opt-cu-delta-qp / no-aq-motion / no-hdr10 / no-hdr10-opt / no-dhdr10-opt / no-idr-recovery-sei / analysis-reuse-level=0 / analysis-save-reuse-level=0 / analysis-load-reuse-level=0 / scale-factor=0 / refine-intra=0 / refine-inter=0 / refine-mv=1 / refine-ctu-distortion=0 / no-limit-sao / ctu-info=0 / no-lowpass-dct / refine-analysis-type=0 / copy-pic=1 / max-ausize-factor=1.0 / no-dynamic-refine / no-single-sei / no-hevc-aq / no-svt / no-field / qp-adaptation-range=1.00 / scenecut-aware-qp=0conformance-window-offsets / right=0 / bottom=0 / decoder-max-rate=0 / no-vbv-live-multi-pass
Default                                  : Yes
Forced                                   : No
Color range                              : Full
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709
DURATION                                 : 00:00:30.000000000

[poisondeathray]

It looks like fullrange=on only works for x264;
but range=full seems to work for x265 ("actual full levels but limited flag" used as input, and the output levels and flag are both full too)

[poisondeathray]

i used "range=full" for x265. but still there is colour problem.

HB settings:
https://i.postimg.cc/xTxT9p2B/image.png

here is the dl link
https://www38.zippyshare.com/v/bO5nbwZI/file.html

This is range compressed, but flagged full

Did you use the "trick" file as input into HB ? You need to change the flag to limited for the input file, to prevent HB from compressing the range

[iKron]

but range=full seems not solving the issue. i am not sure if it's my player problem or video. i tried virtualdub app both is showing a light black bar at bottom

virtualdub:
https://i.postimg.cc/C1xwyb2w/Full-Range-original-00.png

edit:

oh i have to use this one first

ffmpeg -i Full_Range.mkv -bsf:v h264_metadata=video_full_range_flag=0 -c copy Full_Range_flagged_limited.mkv

something is not fully perfect. i mean if i want to do batch encode i have to check file one by one first.

**can i run this command for all video file even which is not full ranged? will it make any issue if video is not full ranged?

[poisondeathray]

but range=full seems not solving the issue. i am not sure if it's my player problem or video. i tried virtualdub app both is showing a light black bar at bottom

virtualdub:
https://i.postimg.cc/C1xwyb2w/Full-Range-original-00.png

The problem is your video. The levels are 16-235, not 0-255. And as mentioned above by jagabo - vdub uses Rec601 for the RGB preview conversion, not 709

When I did the test, with the command line in post #5 to generate the input file, and HB x265 using range=full, it works ok (meaning it's similar to the original video, which was full range data, flagged full range)

[iKron]

okay thank you, i understand now.

[poisondeathray]

edit:

oh i have to use this one first

ffmpeg -i Full_Range.mkv -bsf:v h264_metadata=video_full_range_flag=0 -c copy Full_Range_flagged_limited.mkv

something is not fully perfect. i mean if i want to do batch encode i have to check file one by one first.

**can i run this command for all video file even which is not full ranged? will it make any issue if video is not full ranged?

Yes you can batch - It won't affect a limited range flagged video, and all full range videos that are AVC will be flagged twice with additional flag (I don't like it, but it works for HB)
Color range : Full
colour_range_Original : Limited
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709

IMO there shouldn't be "colour_range_Original" field . It should just say color range: limited. You can use the other roozhou tool if the videos are AVC

[iKron]

@poisondeathray if the source video is x265 what will be the command line for

h264_metadata=video_full_range_flag=0

i know the source code colour range is not full, but i think before i start encode i will run the command line first. original video file media info is:
https://pastebin.com/dy6Adu8t

BTW, after using your suggested command....it seems without adding "range=full" into HB, HB giving correct output colour.

another question, exactly why "Color range : Full" is using? i assume the example video which i gave, probably the source file added the full tag wrongly or something. but some video might add "Full" tag correctly for a reason right?

=============

edit:

another interesting thing i found:

i encode the original source file using using HB. encoded file size become 241KB. HB gave incorrect colour.

then i run the command line which you gave. output file size become 91KB. colour is incorrect though.

[poisondeathray]

@poisondeathray if the source video is x265 what will be the command line for

h264_metadata=video_full_range_flag=0

https://ffmpeg.org/ffmpeg-bitstream-filters.html#hevc_005fmetadata

hevc_metadata=video_full_range_flag=0

i know the source code colour range is not full, but i think before i start encode i will run the command line first. original video file media info is:
https://pastebin.com/dy6Adu8t

BTW, after using your suggested command....it seems without adding "range=full" into HB, HB giving correct output colour.

That source from (https://pastebin.com/dy6Adu8t) is flagged "limited" , so there is no range compression in HB, so you get the expected results normally . The problem with HB is whenever there is a "full range" flag , regardless of the actual data, it applies range compression

another question, exactly why "Color range : Full" is using? i assume the example video which i gave, probably the source file added the full tag wrongly or something. but some video might add "Full" tag correctly for a reason right?

Your 1st file "Full_Range.mkv" was full range data, flagged full range. It's tagged correctly for the actual range.

edit:

another interesting thing i found:

i encode the original source file using using HB. encoded file size become 241KB. HB gave incorrect colour.

then i run the command line which you gave. output file size become 91KB. colour is incorrect though.

I'm not sure which ones you are referring to.

Full actual range ( 0-255, or 0-1023) in general requires more bitrate than compressed range, because there is more data represented with full range. When you compress the range like HB does on a full range source - you are throwing out data before you even start lossy compression

Also, HB will dither a high to lower bitdepth conversion, this add "noise", so it increase the filesize.

You might not be viewing "Full_Range.mkv" (the 1st source video) correctly or some the videos. A full range video, flagged full range, should be converted to RGB for display using full range, not limited range . The matrix needs to be correct to 709 vs. 601 (lvdub uses 601). For example, "Full_Range.mkv" should be displaying Red as RGB 193,0,0 on a computer monitor. (Perfect would be 191,0,0, but there are 8bit colorspace conversions and lossy encoding, rounding errors) . The ramp area should show 0-255 in converted RGB display values

[iKron]

i just wondering if we can use mkvmerge batch script to change the colour range.

is this following give the same result like the ffmpeg command line you provided?

--colour-range TID:n 	

Sets the clipping of the color ranges (0: unspecified, 1: broadcast range, 2: full range (no clipping), 3: defined by MatrixCoefficients/TransferCharacteristics).

i tried following batch script

set "mkv=C:\Program Files\MKVToolNix\mkvpropedit.exe"
for /R "N:\video" %%I in (*.mkv) do "%mkv%" "%%I" --edit info --set "colour-range=3"

but getting error

Error: The name 'colour-range' is not a valid property name for the current edit specification in '--set colour-range=3'.

it will be nice if we can use mkvmerge because for that command ffmpeg require to create another output file. for mkvmerge we can simply edit the source file.

[poisondeathray]

I don't know, maybe --colour-range TID:n needs mkvmerge, so remuxing to create another file too ? I don't see it listed in the mkvpropedit docs, only mkvmerge

[iKron]

my bad mkvpropedit -l showing the colour range parameter though. i google and probably i checked mkvmerge doc page...sorry

[poisondeathray]

For mkvpropedit, this gave similar results to ffmpeg (EDIT: actually it's' reversed to ffmpeg, the original field is now "FULL")

--edit track:v1 --set colour-range=1

Color range : Limited
colour_range_Original : Full
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709

But the levels were still range compressed after HB - ie. it didn't trick HB like ffmpeg did


And --set colour-range=0 to strip the flag (or 3 gives same result), but HB still compresses the range - it must be looking at the colour_range_Original flag. So apparently you need to get the original flag to say Limited like ffmpeg
colour_range_Original : Full
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709

So it looks like mkvpropedit won't work for that workaround.

Common sense says you might as well skip HB alltogether... Is there a reason to use HB instead ?

[iKron]

HB has auto black bar crop detection. and i think in background HB use some kind of filter which make the video quality little bit better. i am not sure may be they using some kind of own modified version of FFMPEG

well it's fine i guess, i made a simple batch script for running the command line which you gave me.

you already gave multiple times this answer but just to make a 101% confirmation :

1. before encode any video, it does not matter what colour range it has...i running this command first

fmpeg -i Full_Range.mkv -bsf:v h264_metadata=video_full_range_flag=0 -c copy Full_Range_flagged_limited.mkv

then i encode the video using HB. there should not be any problem right?

by the way, thanks a lot! you gave a lot time and thought to fig out my problem! i really appreciate it.

[poisondeathray]

-bsf:v h264_metadata will only work on AVC source videos

You might be able to make a smarter batch, calling mediainfo, or ffprobe, and if full range is yes, then run that ffmpeg command, otherwise don't. It should be possible and faster. You can make another thread , someone here knows how . One problem might be *which* flag, because "colour_range_Original" vs. "colour range" might cause some confusion

[poisondeathray]

Another issue would be how would you know which ones to use range=full in HB ? You'd have to keep track, otherwise video could be flagged incorrectly and playback incorrectly . Maybe it could be included the batch with handbrakecli; if full range is detected go down one path, otherwise encode normally

[iKron]

i will try to make one but i have very little knowledge about making batch, if you can help me to create it, it will be really helpful for me.

[iKron]

final confirmation please

if any media info has "Color range : Full"

i simply use the command line you provided and everything is fine right?

[jagabo]

if any media info has "Color range : Full"

i simply use the command line you provided and everything is fine right?

If you want to be absolutely sure you should look at the video and decide for yourself if it's flagged correctly. For example, many camcorders theoretically shoot limited range video but routinely have brights above Y=235. VHS caps often have blacks well above Y=16. MJPEG is usually full range but may not be flagged as such.

[poisondeathray]

final confirmation please

if any media info has "Color range : Full"

i simply use the command line you provided and everything is fine right?

or apparently
colour_range_Original : Full


But the ones that have actual full range data need the extra command in HB to flag it correctly as full, so you still have to distinguish between them

Otherwise the HB export will have full range data, but flagged as limited, and definitely wrong playback

And if you flag everything as full, it will be wrong for the ones that have limited range data

[pinoco]

hi everybody,
i know this thread is old but i think i have same problem. this is the specs (as seen in mediainfo) of my original video:

[Attachment 76357 - Click to enlarge]

this video was recorded by a mobile phone. this video file is having faded colours when converted thru handbrake. i am keeping resolution AND codec SAME as original (i.e. 1080p with h264).
is this problem because of this 'colour range' issue , as discussed on this page?
@poisondeathray u were suggesting ikron to not use handbrake. then which good software do u suggest for video conversion? i tried many diff softs ( ffmpeg batch converter, ffmpeg.exe with batch .cmd script, shana encoder) BUT all of them give less quality/clarity videos as compared to handbrake or WinX video converter (very good paid video converter). handbrake and WinX give less file size with more quality/clarity!!! as i observed; all softwares making use of ffmpeg libraries (unmodified, stock) give results that are subpar as compared to modified ffmpeg ones (i.e. handbrake) or completely proprietory (WinX) ones.
which video converter do u suggest that is as good as handbrake? i think there is none.
BTW , staxrip is also as good as handbrake or winX. staxrip is fading the colours of other videos also (which are kept as original by handbrake).

[poisondeathray]

is this problem because of this 'colour range' issue , as discussed on this page?

Nobody can tell for certain without a video sample

@poisondeathray u were suggesting ikron to not use handbrake. then which good software do u suggest for video conversion?

I just use the encoders directly. You can preprocess with avisynth or vapoursynth. Much more flexible this way without the problems.

Note that the OP's problems were caused by Handbrake . Not sure if the full range issues have been been fixed . The general advice is "Use what works." HB didn't work for him

But the HB 8bit pipeline issue has been fixed since then

i tried many diff softs ( ffmpeg batch converter, ffmpeg.exe with batch .cmd script, shana encoder) BUT all of them give less quality/clarity videos as compared to handbrake or WinX video converter (very good paid video converter).

handbrake and WinX give less file size with more quality/clarity!!! as i observed; all softwares making use of ffmpeg libraries (unmodified, stock) give results that are subpar as compared to modified ffmpeg ones (i.e. handbrake) or completely proprietory (WinX) ones.

I doubt it. Maybe there were flaws in your comparison, such as different settings or preprocessing

which video converter do u suggest that is as good as handbrake? i think there is none.

They are just GUIs - front ends . If you use the same preprocessing , the same encoder core version, same settings, you get the same results