x264 [error]: x264_encoder_encode failed

[_Al_]

interestingly, we have a problem on one PC that does not want to run x264.exe, it starts and silently exits, but if run in cmd as administrator it gives more readings:

raw [info]: 3840x2160p 0:0 @ 2997/50 fps (cfr)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX AVX2 FMA3 LZCNT BMI2
x264 [info]: profile High, level 5.2
x264 [error]: malloc of size 26199520 failed
x264 [error]: x264_encoder_encode failed

PC info:

		Operating System
			Windows 10 Home 64-bit
		CPU
			AMD Ryzen 9 3900X	45 °C
			Matisse 7nm Technology
		RAM
			64.0GB Dual-Channel Unknown @ 1599MHz (16-18-18-38)
		Motherboard
			ASUSTeK COMPUTER INC. TUF GAMING X570-PLUS (WI-FI) (AM4)	32 °C
		Graphics
			LEN C32q-20 (2560x1440@60Hz)
			2048MB ATI AMD Radeon HD 7700 Series (ATI)	43 °C
		Storage
			465GB Seagate FireCuda SCSI Disk Device (SATA (SSD))
			4657GB TOSHIBA HDWE150 SCSI Disk Device (SATA )
			447GB SanDisk Ultra II SCSI Disk Device (SATA (SSD))
		Optical Drives
			No optical disk drives detected
		Audio
			Realtek High Definition Audio

Has anyone some idea what could be the problem?

[BugMaster]

Which build of x264.exe are you using? Are you trying to encode 4K video with 32-bit x264.exe? If you need 64-bit x264.exe you can download it from: https://artifacts.videolan.org/x264/release-win64/

[_Al_]

Thanks.
Using 64bit build. Tested even with that latest 64bit build from that page, the same problem.

[Gus.Ghanem]

May be totally irrelevant, but there's a setting in the BIOS called Above 4G decoding (something to do with memory allocation). Maybe the video card needs to have more on-board VRAM; I have no experience with encoding, so I'm just guessing.

[El Heggunte]

............Has anyone some idea what could be the problem?

According to this post in Doom9's forum ( https://forum.doom9.org/showthread.php?p=1333784#post1333784 ),
MAYBE the problem is in the encoding settings that you gave to x264.exe

The memory usage due to the insane --bframes value, combined with tesa, combined with b-adapt 2, is much more of an issue.

[_Al_]

Thanks, I tried to limit buffers, did not help,

but I narrowed it down to vapoursynth line:

from nnedi3_resample import nnedi3_resample
clip = nnedi3_resample(clip, clip.width * 2, clip.height * 2, mode='znedi3')

without nnedi3_resample it works. Dll's are used the same, portable solution, compiled to exe, but on that particular computer it just does not work.

Is there some alternative for a decent quick double upscale HD to 4K roughly for vapoursynth, for real videos, not anime?

EDIT: Heck, it does not work even with simple upscale:

clip = core.resize.Lanczos(clip, clip.width * 2, clip.height * 2)

so it must be something with RAM related or so, it just cannot handle higher resolution ...

[_Al_]

The whole python script is here, but not sure if it helps. It seams that encoding to higher resolution crashes, if I remove that upscaling line, it works.

#python code, not php
import vapoursynth as vs
from vapoursynth import core
import sys
import os
import loadDLL
from nnedi3_resample import nnedi3_resample
import load
from pathlib import Path
import subprocess

try:
    path = sys._MEIPASS
    IS_FROZEN=True
except AttributeError:
    IS_FROZEN=False
    TOOL_DIR       = r'F:\path_to_\ENCODER_12\tools'
    VS_PLUGINS_DIR = ''
    is_ok, dlls_log = loadDLL.vapoursynth_dlls(core, r'F:\path_to_\e2\vapoursynth64\plugins')
    
if IS_FROZEN:
    TOOL_DIR = os.path.join(path,'tools')
    VS_PLUGINS_DIR = os.path.join(path,'vapoursynth64\plugins')
    isDLLS_LOADED, DLLS_LOG = loadDLL.vapoursynth_dlls(core, VS_PLUGINS_DIR)

def to_4k(sources):
    my_loader = load.Sources()
    x264   = Path(TOOL_DIR) / "x264.exe"
    MP4BOX = Path(TOOL_DIR) / "MP4Box.exe"
    for source_path in sources:
        data = my_loader.get_dataclasses([source_path])
        clip = data[0].clip
        if data[0].load_isError:
            print(data[0].load_log_error)
            continue
        clip = nnedi3_resample(clip, clip.width * 2, clip.height * 2, mode='znedi3')
        path = Path(source_path)
        output = path.parent / f'{path.stem}_4k.mp4'
        temp_output = path.parent / f'{path.stem}_4k.264'
        fps = clip.fps.numerator/clip.fps.denominator
        print(f'\nencoding:     {path}\nnew 4K file:  {output}')
        cmd = [x264, '--frames',           f'{len(clip)}',
                     '--input-csp',         'i420',
                     '--demuxer',           'raw',  
                     '--input-depth',       '8',
                     '--input-res',        f'{clip.width}x{clip.height}',
                     '--fps',              f'{fps}',
                     '--crf',               '17',
                     '--vbv-maxrate',       '70000',
                     '--vbv-bufsize',       '70000',
                     '--keyint',            '50',
                     '--tune',              'film',
                     '--colorprim',         'bt709',
                     '--transfer',          'bt709',
                     '--colormatrix',       'bt709',
                     '--output',             temp_output,
                     '-']
        
        process = subprocess.Popen(cmd, stdin=subprocess.PIPE)
        clip.output(process.stdin)
        process.communicate()

        mp4box_cmd = [MP4BOX,
                         '-add' , f'{temp_output}:fps={fps}',
##                         '-add',  f'{source_path}#audio',
                         '-new',  output]
        process = subprocess.Popen(mp4box_cmd)
        process.communicate()
        try:
            os.remove(temp_output)
        except:
            pass


if __name__ == '__main__':
    
    sources    = []
    error_text = ''

    if len(sys.argv) > 1:
        for arg in sys.argv[1:]:
            sources.append(arg)
    to_4k(sources) #rather just double resolution 

[El Heggunte]

deleted

[jagabo]

EDIT: Heck, it does not work even with simple upscale:

clip = core.resize.Lanczos(clip, clip.width * 2, clip.height * 2)

Remember, doubling the width and height of the frame requires four times as much memory.

so it must be something with RAM related

It literally told you that already:

x264 [error]: malloc of size 26199520 failed

malloc = Memory ALLOCation

The question is why a 26 MB allocation is failing on a computer with 64 GB of RAM. You have only 2 GB of video memory. If you're using the graphics card for any filtering or encoding that might run into a problem -- but I don't see any of that in your posts. As was mentioned before, each 32 bit process is limited to 2 to 3 GB of RAM usage (depending on setup), even on 64 bit Windows. So 32 bit encoders with long GOPs and large frame sizes will run into problems. 64 bit encoders shouldn't run into problems with 2K or 4K video and "normal" settings. Another possibility is a bug in the software -- a "memory leak" where it fails to deallocate memory it's no longer using but keeps allocating more as processing continues. The system doesn't realize the program is no longer using the old memory and keeps giving the program more. Eventually the system runs out of memory.

[poisondeathray]

"problem on one pc" implies that the same script works on another pc - is that right ? the problem is isolated to that one specific pc with 3900x , 64gb ?

Did you try vspipe method with -c y4m to rule out some raw issue ?

[_Al_]

Thanks,guys, yes it works on couple of PC's just one gives this error.
Tested x264.exe --demuxer y4m ...., but on that PC it failed (4k resolution, not HD). For now I was not able to test: vspipe command line loading script and piping it to x264 using command prompt.

But was able to test it quick like binaries compiled from Python and it failed again: vspipe loading vpy script, piped to x264 , it failed again. All scripts work, just that one PC gives problem.
That last main script, to simplified it a bit, was this:

import vapoursynth as vs
from vapoursynth import core
from pathlib import Path
import os
from subprocess import Popen, PIPE

source_path = r'F:/video.mp4'
#getting clip just for some video properties, otherwise clip gets processed in 'double_resolution.py'
clip = core.lsmas.LibavSMASHSource(source_path)
fps  = clip.fps.numerator/clip.fps.denominator

PATH        = Path(os.getcwd()) #Path(sys._MEIPASS) #if frozen
vpy_script  = PATH / 'double_resolution.py'
MODULES_DIR = PATH / 'python_modules'
TOOL_DIR    = PATH / 'tools'
VSPipe      = PATH / 'VSPipe.exe'
x264        = TOOL_DIR / 'x264.exe'
Mp4box      = TOOL_DIR / 'MP4Box.exe'
##import loadDLL #if frozen
##VS_PLUGINS_DIR = PATH / 'vapoursynth64/plugins'
##isDLLS_LOADED, DLLS_LOG = loadDLL.vapoursynth_dlls(core, VS_PLUGINS_DIR)
path        = Path(source_path)
output      = path.parent / f'{path.stem}_4k.mp4'
temp_output = path.parent / f'{path.stem}_4k.264'

vspipe_cmd = [VSPipe,   '--container',  'y4m',
                        '--arg',       f'source_path={source_path}',
                        '--arg',       f'modules_dir={str(MODULES_DIR)}',
                        vpy_script,
                        '-']

x264_cmd = [x264,       '--frames',           f'{len(clip)}',
                        '--demuxer',           'y4m',  
                        '--crf',               '17',
                        '--vbv-maxrate',       '70000',
                        '--vbv-bufsize',       '70000',
                        '--keyint',            '50',
                        '--tune',              'film',
                        '--colorprim',         'bt709',
                        '--transfer',          'bt709',
                        '--colormatrix',       'bt709',
                        '--output',             temp_output,
                        '-']

mp4box_cmd = [Mp4box,   '-add' , f'{temp_output}:fps={fps}',
                        '-add',  f'{source_path}#audio',
                        '-new',  output]

p1 = Popen(vspipe_cmd, stdout=PIPE, stderr=PIPE)
p2 = Popen(x264_cmd, stdin=p1.stdout, stdout=PIPE, stderr=PIPE)
p1.stdout.close()
p2.communicate()

p = Popen(mp4box_cmd)
p.communicate()

and its attribute "double_resolution.py" vapoursynth script,where vspipe passes variables (sets globals) source_path and modules_dir:

import vapoursynth as vs
from vapoursynth import core
import sys

sys.path.append(modules_dir)

import mvsfunc
from nnedi3_resample import nnedi3_resample

clip = core.lsmas.LibavSMASHSource(source_path)
clip = nnedi3_resample(clip, clip.width * 2, clip.height * 2, mode='znedi3')
clip.set_output()

That works, sometimes as python vspipe crashes for some reason, but if repeating it is ok. Binaries with no problem with similar setup adjusted for portable vs and python. Except that one PC.
Divers were updated , not remotely sure what the problem could be. That PC does not have python or vapoursynth installed, so everything is a portable setup or binaries, but I don't think it matters, it is just that higher resolution problem.