RGB values range from 0 to 255, and JPEG compression converts this first into full range YUV (full range meaning not the limited range of 16 to 240 like normal YUV). The bytes of RGB data are then converted to floating point numbers (as needed for the DCT algorithm) and passed to a DCT encoder to get spacial frequency data. This spacial frequency data then gets rounded back from floating point data to one-byte-per-sample integer values. This is done before compressing.
The reverse is done for decoding. After decompression the integer DCT data it is converted back to floating point values and passed to an IDCT algorithm. There result is pixel values that are then rounded back to one-byte-per-sample integer values.
My question is how does the rounding work exactly? There's 10 rounding algorithms I know of.
Floor (always round down)
Ceiling (always round up)
Always round toward zero (negative rounds up and positive rounds down)
Always round away from zero (positive rounds up and negative rounds down)
Round toward nearest number with halfway tie-breaker always rounding up
Round toward nearest number with halfway tie-breaker always rounding down
Round toward nearest number with halfway tie-breaker always rounding toward zero
Round toward nearest number with halfway tie-breaker always rounding away from zero
Round toward nearest number with halfway tie-breaker always rounding toward nearest odd number
Round toward nearest number with halfway tie-breaker always rounding toward nearest even number
Now both the post-DCT and post-IDCT steps require rounding of some kind to get one-byte-per-sample values. I don't know what the correct rounding algorithm is, nor do I know if the same algorithm is supposed to be used for both the post-DCT and post-IDCT rounding steps. Also I know it needs to have the DCT and IDCT outputs scaled to fit within the byte range (0 to 255). Is this scaling usually applied with floating point math, prior to converting to integer values? Or is the scaling usually performed with integer math after converting to integer values?
I would like to know these things, because I'm trying to implement my own JPEG compatible DCT/IDCT steps, and float-to-integer conversion steps in a program I'm writing. And I notice that after I do the sequence of steps ByteToFloat->DCT->Round->ByteToFloat->IDCT->Round that my final pixel values do not cover the full range of 0 to 255 anymore. The output pixel values only cover the range 0 to 253 when I use the Floor rounding method (which I thought I had previously read that JPEG used). This is why I need some help here.
+ Reply to Thread
Results 1 to 1 of 1
Similar Threads
-
RAW to JPEG converter
By addi II in forum Newbie / General discussionsReplies: 13Last Post: 20th May 2019, 22:56 -
TV requires M-JPEG
By 1voyager1 in forum Video ConversionReplies: 21Last Post: 28th Jun 2018, 11:17 -
Mp3+jpeg to video
By catago in forum Newbie / General discussionsReplies: 8Last Post: 14th Nov 2017, 08:49 -
JPEG broken for high res photos?
By Gurd99 in forum ComputerReplies: 17Last Post: 21st Sep 2017, 05:17 -
jpeg -> video -> jpeg
By agatek in forum Video ConversionReplies: 22Last Post: 12th Jan 2017, 22:06