Transcoding, subsampling chroma multiple times (x265 specific?)

[_Al_]

Why would 2 bytes be used for 10 bit values?

You might be right, I simply assumed, memory allocates 32bit chunks with four bytes. But maybe three 10bit values could be allocated in 32bit chunk, or something similar somehow, not sure on that one. I just remember Vapoursynth's author mentioned that if data was something between 8 bit and 16bit is a waste of memory (assuming it allocates 16bit for 9bit to 16bit). But that would be uncompressed planes and particular software.

[jagabo]

DCT: https://en.wikipedia.org/wiki/Discrete_cosine_transform
ringing: https://en.wikipedia.org/wiki/Ringing_artifacts
compression artifacts: https://en.wikipedia.org/wiki/Compression_artifacts

Thanks. Although I was more trying to point out that while chroma subsampling and colorspace conversions have been explained in excruciating and excessive detail, with several example pictures and diagrams... the "actual compression" has not.

And if you read through the links, and the articles they link to, you'll have excruciating and excessive detail to wade through.

[non-vol]

Why would 2 bytes be used for 10 bit values?

You might be right, I simply assumed, memory allocates 32bit chunks with four bytes. But maybe three 10bit values could be allocated in 32bit chunk, or something similar somehow, not sure on that one. I just remember Vapoursynth's author mentioned that if data was something between 8 bit and 16bit is a waste of memory (assuming it allocates 16bit for 9bit to 16bit). But that would be uncompressed planes and particular software.

Are you talking about processing or storage? Modern computers work with 64 bit data internally, but you can store a stream of 9-bit values on your hard drive if you want to. And you would if you don't want to waste space.

[non-vol]

DCT: https://en.wikipedia.org/wiki/Discrete_cosine_transform
ringing: https://en.wikipedia.org/wiki/Ringing_artifacts
compression artifacts: https://en.wikipedia.org/wiki/Compression_artifacts

Thanks. Although I was more trying to point out that while chroma subsampling and colorspace conversions have been explained in excruciating and excessive detail, with several example pictures and diagrams... the "actual compression" has not.

And if you read through the links, and the articles they link to, you'll have excruciating and excessive detail to wade through.

Do they also have have the extremely helpful example pictures and diagrams that make it all easy to understand?

[jagabo]

DCT: https://en.wikipedia.org/wiki/Discrete_cosine_transform
ringing: https://en.wikipedia.org/wiki/Ringing_artifacts
compression artifacts: https://en.wikipedia.org/wiki/Compression_artifacts

Thanks. Although I was more trying to point out that while chroma subsampling and colorspace conversions have been explained in excruciating and excessive detail, with several example pictures and diagrams... the "actual compression" has not.

And if you read through the links, and the articles they link to, you'll have excruciating and excessive detail to wade through.

Do they also have have the extremely helpful example pictures and diagrams that make it all easy to understand?

Yes, the do.

[jagabo]

DCT: https://en.wikipedia.org/wiki/Discrete_cosine_transform
ringing: https://en.wikipedia.org/wiki/Ringing_artifacts
compression artifacts: https://en.wikipedia.org/wiki/Compression_artifacts

Thanks. Although I was more trying to point out that while chroma subsampling and colorspace conversions have been explained in excruciating and excessive detail, with several example pictures and diagrams... the "actual compression" has not.

And if you read through the links, and the articles they link to, you'll have excruciating and excessive detail to wade through.

Do they also have have the extremely helpful example pictures and diagrams that make it all easy to understand?

Yes, the do. Though "easy" is a relative term.

[non-vol]

Though "easy" is a relative term.

You're right.

Seriously though, what would be useful was a simplified and condensed version of what x265 does. In visual, not so much technical terms. Pictures, not words (the subject is video compression after all). I don't know if anything like this exists.

[_Al_]

you can take a mp4 or mov file, choose number of generation copies in the script below to see it for yourself:
dependencies: python 3.6 or above, vapoursynth, x265, mp4box

#Python3.6 or above

'''
loads source video file (mp4 or mov) and creates number of copies
where each copy is made from previous copy
effectively getting 'x' generation copy for study generation loss for x265
use as vapoursynth script or run python in cmd prompt:
python "generation_copy.py" 10 "F:/_lighthouse_lossless.mp4"
using mp4box, x265 and vapoursynth python module
'''

import vapoursynth as vs
from vapoursynth import core
import subprocess
import shutil
import os
import sys

core.max_cache_size = 800  #comment it out if having plenty of RAM
cwd = os.getcwd()          #current working directory
TEMP_DIR = cwd             
OUTPUT_DIR = cwd           #or select some:  OUTPUT_DIR = r'E:\some_dir'

MP4BOX, X265 = None, None
#MP4BOX = shutil.which('Mp4box') #windows,  mp4box.exe must be in working directory or in PATH
#MP4BOX = shutil.which('MP4box') #linux , I have it there with uppercase "P" after installing 
MP4BOX = r'C:\tools\MP4box-64bit\mp4box.exe'  #win , manualy giving PATH
X265 = shutil.which('x265')    #x265.exe must be in working directory or in PATH



for executable, path in dict(Mp4box=MP4BOX, x265=X265).items():
    if not path:
        raise FileNotFoundError(f'{executable} executable was not found, should be in working directory or in the path,'
           '\nif using vsedit, working directory is directory  where this script is saved')
    elif not os.path.isfile(path):
        raise FileNotFoundError(f'{executable} executable is not a file, check for errors in a path string')        
    elif not os.access(path, os.X_OK):
        raise FileNotFoundError(f'no administrative rights to run {executable} executable')

def stdin_frames(current, total):
    print(f'{current} of {total}')

def encode_copy(index, file_path):

    temp_output = os.path.join(TEMP_DIR, 'temp.hevc')  #mp4box does not support stdin
    new_path = os.path.join(OUTPUT_DIR, f'copy_{index}.mp4')
   
    #make vapoursynth clip
    clip = core.lsmas.LibavSMASHSource(file_path)
    
    if index == 1:
        #some filtering, slicing, crop could be applied to source clip only
        #clip = clip[0:50]
        #clip = core.std.CropAbs(clip, width=538, height=446, left=64, top=50)
        pass

    #encode HEVC
    print(f'encoding {index} of {NUMBER_OF_COPIES},  {new_path}')
    x265_cmd = [X265,
             '--frames', f'{len(clip)}',
             '--y4m',
             '--input-depth', f'{clip.format.bits_per_sample}',
             '--output-depth', '10',
             '--input-res', f'{clip.width}x{clip.height}',
             '--fps', f'{clip.fps_num}/{clip.fps_den}',
             '--crf',    '18',
             '--output', temp_output,
             '-']  
    process = subprocess.Popen(x265_cmd, stdin=subprocess.PIPE)
    clip.output(process.stdin, y4m = True) #progress_update=stdin_frames
    process.communicate()

    #muxing hevc stream to mp4
    print(f'muxing {index} of {NUMBER_OF_COPIES},  {new_path}')
    fps = f'{str(clip.fps_num/clip.fps_den)}'
    mp4box_cmd = [MP4BOX,
                 '-add' , temp_output+f':fps={fps}',
                 '-par', '1=1:1',
                 '-new',  new_path]
    process = subprocess.Popen(mp4box_cmd)
    process.communicate() 

    #delete temp
    if os.path.exists(temp_output):   os.remove(temp_output)

    return new_path


def run():

    new_path = SOURCE 
    for index in range(1, NUMBER_OF_COPIES+1):
        new_path = encode_copy(index, new_path)        


   
if __name__ == "__main__" and not len(sys.argv) == 1:

    '''
    running python script in command line:
    python "generation_copy.py" 10 "F:/_lighthouse_lossless.mp4"
    '''

    NUMBER_OF_COPIES = int(sys.argv[1])
    SOURCE = sys.argv[2]
    run()

            
elif __name__ == "__vapoursynth__" or len(sys.argv) == 1:
    
    '''
    this runs as vapoursynth script or if executing this script without arguments (in python console)
    it can be used using vspipe as well, example:
    "vspipe.exe" "gereration_copy.py" .
    or
    "vspipe.exe" "gereration_copy.vpy" .
    '''

    NUMBER_OF_COPIES = 10   #1 or more
    SOURCE = r'F:/_lighthouse_lossless.mp4'
    run()