Best Compressed Storage of mts files ?

[NewTwoVideo]

Hello Everyone,

I have been using Clever FFmpeg GUI to convert 1080/59.94p (PSF) AC-3 mts files from my Panasonic camcorder to 1080/29.97p AAC-LC mp4 files.
Clever FFmpeg GUI works great and is easy once you get the workflow memorized.
My videos are just family memories and I am just trying to make them easier to view using PC instead of connecting the camcorder to the TV.

I was wondering, since Clever FFmpeg GUI has so many options for output, WHAT IS THE BEST STORAGE CONTAINER / CODEC / FILE FORMAT etc. to save / archive the mts files in a way that takes up less disk space?

As you know mts files are pretty big, 150 ~ 200 MB per minute. Most alternatives are smaller. But what combination of settings will preserve the most information and quality of the original?

I know that mts files have a timestamp in each frame. Preserving that would only be needed if I was documenting something for an insurance claim, like storm damage, so there I think I would save the original mts file.

So if that is not a factor, what is the best archiving format to save space but retain the quality and video and audio information?

TIA

[rgr]

1. Use Neat Video to "smooth the image" from noise or compression artifacts (no profile, just temporal and spatial denoising).
2. Save with crf=19.
This can be done in VirtualDub as batch processing.
You will get down to about 1/3 to half the size of the original file, by the way in Neat you can set sharpening and improve the quality.

I haven't found a better method to save mts yet.
(Additionally, I also use Deshaker to remove shakings).

PS. Check if the original mts is not saved in the range 16-255 - if so, you should also reduce the range to 16-235 or extend it to 0-255.

[Bwaak]

I have been using Clever FFmpeg GUI to convert 1080/59.94p (PSF) AC-3 mts files from my Panasonic camcorder to 1080/29.97p AAC-LC mp4 files.

PsF is not 59.94p, it is 29.97i.

As you know mts files are pretty big, 150 ~ 200 MB per minute. Most alternatives are smaller. But what combination of settings will preserve the most information and quality of the original?

200 MB/min is peanuts. Most modern cameras have the bitrate which is at least twice higher. Just keep them as they are. 1TB external USB3 drive is $50, this is good for 80 hours of footage.

[NewTwoVideo]

Thank you for the recommendations.

[NewTwoVideo]

I thought Progressive Segmented Frames referred to the version of progressive used by Panasonic and some of the other Camera camcorder manufacturers.
Thank you for the correction.

[4kblurayguru]

WHAT IS THE BEST STORAGE CONTAINER / CODEC / FILE FORMAT etc. to save / archive the mts files in a way that takes up less disk space?

video 360p 24fps 500kps avc
audio 64 kbps aac

for original files, follow 3 2 1 backup rules

[rgr]

I have been using Clever FFmpeg GUI to convert 1080/59.94p (PSF) AC-3 mts files from my Panasonic camcorder to 1080/29.97p AAC-LC mp4 files.

PsF is not 59.94p, it is 29.97i.

And in reality it is 29.97p.

PsF stands for "Progressive Segmented Frame". It is where a camera sends a progressive video format (such as 1080p29.97 or 1080p30) as an interlaced video signal (such as 1080i59.94 or 1080i59.94)
Some cameras do this for compatibility reasons, as a lot of older equipment does not support progressive video signals, especially when using SDI.

[Alwyn]

Saving home videos at 200MB/min is too much IMO; just not necessary.

video 360p 24fps 500kps avc audio 64 kbps aac

And that is going to the other extreme.

I'd go for 1080 AVC video at 5000kbps, audio 192kbps AAC = ~39MB/minute.

Trial them all, see what you like.

[NewTwoVideo]

Thank you, all, for the recommendations and the education.

[black-dvd-archiver]

I do this on transcodes and on my Panasonic Camcorder - running ~60FPS Progressive - it seems, although we know its not, to be transparent, Input looks same as output. mp4 are much easier to handle than MTS when viewing, joining and editing.

def Transcode_H26x(
    input_file: str,
    output_folder: str,
    frame_rate: float,
    width: int,
    height: int,
    interlaced: bool = True,
    bottom_field_first: bool = True,
    h265: bool = False,
    high_quality: bool = True,
) -> tuple[int, str]:
    """Converts an input video to H.264/5 at supplied resolution and frame rate.
    The video is transcoded to a file in the output folder.

    Args:
        input_file (str): The path to the input video file.
        output_folder (str): The path to the output folder.
        frame_rate (float): The frame rate to use for the output video.
        width (int) : The width of the video
        height (int) : The height of the video
        interlaced (bool, optional): Whether to use interlaced video. Defaults to True.
        bottom_field_first (bool, optional): Whether to use bottom field first. Defaults to True.
        h265 (bool, optional): Whether to use H.265. Defaults to False.
        high_quality (bool, optional): Use a high quality encode. Defaults to True.

    Returns:
        tuple[int, str]:
            arg 1: 1 if ok, -1 if error
            arg 2: error message if error else ""
    """
    assert (
        isinstance(input_file, str) and input_file.strip() != ""
    ), f"{input_file=}. Must be a non-empty str"

    assert (
        isinstance(output_folder, str) and output_folder.strip() != ""
    ), f"{output_folder=}. Must be a non-empty str"
    assert (
        isinstance(frame_rate, float) and frame_rate > 0
    ), f"{frame_rate=}. Must be float > 0"
    assert isinstance(width, int) and width > 0, f"{width=}. Must be int > 0"
    assert isinstance(height, int) and height > 0, f"{height=}. Must be int > 0"
    assert isinstance(interlaced, bool), f"{interlaced=}. Must be bool"
    assert isinstance(bottom_field_first, bool), f"{bottom_field_first=}. Must be bool"
    assert isinstance(h265, bool), f"{h265=}. Must be bool"
    assert isinstance(high_quality, bool), f"{high_quality=}. Must be bool"

    file_handler = file_utils.File()

    if not file_handler.path_exists(output_folder):
        return -1, f"{output_folder} Does not exist"

    if not file_handler.path_writeable(output_folder):
        return -1, f"{output_folder} Cannot Be Written To"

    if not file_handler.file_exists(input_file):
        return -1, f"File Does Not Exist {input_file}"

    _, input_file_name, input_file_extn = file_handler.split_file_path(input_file)

    output_file = file_handler.file_join(output_folder, f"{input_file_name}.mp4")

    if not utils.Is_Complied():
        print(
            "DBG Trans_H26x"
            f" {input_file=} {frame_rate=} {width=} {height=} {interlaced=} {bottom_field_first=} ER"
            f" {'5M' if height <= 576 else '35M'=}"
        )

    # Construct the FFmpeg command
    if h265:
        encoder = "libx265"
    else:
        encoder = "libx264"

    if high_quality:
        quality_preset = "slow"
    else:
        quality_preset = "superfast"

    black_border_size = 12

    if interlaced:
        # black_box_filter, most likely dealing with analogue video, head switching noise best removed, and edges
        # are best covered for optimal compression
        filter_commands = [
            f"drawbox=x=0:y=0:w=iw:h={black_border_size}:color=black:t=fill",
            f"drawbox=x=0:y=ih-{black_border_size}:w=iw:h={black_border_size}:color=black:t=fill",
            f"drawbox=x=0:y={black_border_size}:w={black_border_size}:h=ih-{black_border_size*2}:color=black:t=fill",
            f"drawbox=x=iw-{black_border_size}:y={black_border_size}:w={black_border_size}:h=ih-{black_border_size*2}:color=black:t=fill",
        ]
        black_box_filter = ",".join(filter_commands)

        field_order = f"fieldorder={'bff' if bottom_field_first else 'tff' }"
        video_filter = [
            "-vf",
            f"{black_box_filter},scale={width}x{height},{field_order}",
            "-flags:v:0",  # video flags for the first video stream
            "+ilme+ildct",  # include interlaced motion estimation and interlaced DCT
            "-alternate_scan:v:0",  # set alternate scan for first video stream (interlace)
            "1",  # alternate scan value is 1,
        ]
    else:
        video_filter = [
            "-vf",
            f"scale={width}x{height}",
        ]

    command = [
        sys_consts.FFMPG,
        "-threads",
        str(psutil.cpu_count() - 1 if psutil.cpu_count() > 1 else 1),
        "-i",
        input_file,
        *video_filter,
        "-r",
        str(frame_rate),
        "-c:v",
        encoder,
        "-pix_fmt",
        "yuv420p",  # Ensure the pixel format is compatible with Blu-ray
        "-crf",
        "18",
        "-preset",
        quality_preset,
        "-c:a",
        "aac",
        "-b:v",
        (
            "5M" if height <= 576 else "25M"
        ),  # SD Get low-bit rate everything else gets Blu-ray rate. Black Choice for now
        "-muxrate",
        "25M",  # Maximum Blu-ray mux rate in bits per second
        "-bufsize",
        "15M",
        "-g",
        "15",  # Set the GOP size to match the DVD standard
        "-keyint_min",
        "15",  # Set the minimum key frame interval to Match DVD (same as GOP size for closed GOP)
        "-sc_threshold",
        "0",  # Set the scene change threshold to 0 for frequent key frames
        output_file,
        "-y",
    ]

    if not file_handler.file_exists(output_file):
        result, message = Execute_Check_Output(commands=command, debug=True)

        if result == -1:
            return -1, message

    return 1, output_file

[Bwaak]

I do this on transcodes and on my Panasonic Camcorder - running ~60FPS Progressive ...

25 Mbps is pretty much the same as 26-28 Mbps employed in AVCHD Progressive. If you feed it with 60 Mpbs or 100 Mbps file XAVC or XAVC-S file, it would make more sense. In fact, I have a Sony camcorder that records to XAVC-S @ 60 Mbps from a tiny 1/6-inch sensor, and the quality is worse than from my Panasonic AVCHD Progressive camcorder. So, it this case I feel that these 60 Mbps are excessive and would not mind compressing them to, like, one third of their size. But then again, I don't have too many of these files, this is my son's camcorder, and he does not shoot much.

OTOH, 28 Mbps sometimes is not enough for the Panasonic, which is super-high detail, so when there is a lot of detail, it breaks into macroblocks. Later models have 35 Mbps mode, and even later models have 60 Mbps mode. I think that 35 Mbps would be the best middle ground.

[Swedaniel]

Hello Everyone,

I have been using Clever FFmpeg GUI to convert 1080/59.94p (PSF) AC-3 mts files from my Panasonic camcorder to 1080/29.97p AAC-LC mp4 files.
Clever FFmpeg GUI works great and is easy once you get the workflow memorized.
My videos are just family memories and I am just trying to make them easier to view using PC instead of connecting the camcorder to the TV.

I was wondering, since Clever FFmpeg GUI has so many options for output, WHAT IS THE BEST STORAGE CONTAINER / CODEC / FILE FORMAT etc. to save / archive the mts files in a way that takes up less disk space?

As you know mts files are pretty big, 150 ~ 200 MB per minute. Most alternatives are smaller. But what combination of settings will preserve the most information and quality of the original?

I know that mts files have a timestamp in each frame. Preserving that would only be needed if I was documenting something for an insurance claim, like storm damage, so there I think I would save the original mts file.

So if that is not a factor, what is the best archiving format to save space but retain the quality and video and audio information?

TIA

Have you concidered using an compression program such as PeaZip/Zipware/QuickZip or 7-Zip to pack your files into an Zip or Rar file. With theese programs you can set custom compression rates and you can add multiple files into an Zip/Rar Package and open the file and it will list all files in this package together with their file name, file size, date, time etc, and you can easily drag an file to unpack it back to the computer aswell you can add file into this package at any time. Most standard Windows systems have an built in support for opening Zip files. All listed programs up above are free programs, if you want to check the original program which is called WinZip there is many versions with Free Trials avalible, any file created with WinZip trial will still be valid after trial expire. Allso you can test WinRar which offer trial versions and the one im using basicly all it say is after the trial expire you must remove it or buy it but basicly you can continue to use it forever.. WinRar can handle Zip files and Rar files, PeaZip can allso handle Zip and Rar files.

If you would want to encrypt your videos to protect them, it is allso possible to encrypt and add passwords with theese programs, ellse some encryption programs offers built in compression. AES Crypter Professional v.1.0.3 have built in compression when encrypting files. Depending on the video codec the compression will vary since some video files are allready compressed, but an video with MJPEG Compressor Codec @ 5MB File size is now 300KB after encrypting it with AES Crypter Professional v.1.0.3. The AES Crypter Pro program is free to use, supports drag and drop operations and you can select and use any password for each file. With this program you can allso rename the file after it have been encrypted and when decrypting it, it will remember the original file name and put the name back when decrypting it.

Edit: When using the programs mentioned above, it wil restore the original file , fully to 100%, no loss of frames, quality or anything! It is basicly like putting an candy in the candy bag! Using AES Crypter i guess would be like putting the candy in your secret wallet space and hide it with an special locker to it.

[Cornucopia]

No. Almost all compression technologies already use some form of lossless compression similar to Zip, built into their algorithms. Further packing compressed files will yield no further gains in compression and in fact will often grow instead of shrink, due to the fact that what is remaining has more entropy/randomness in the code.


Scott

[pandy]

No. Almost all compression technologies already use some form of lossless compression similar to Zip, built into their algorithms. Further packing compressed files will yield no further gains in compression and in fact will often grow instead of shrink, due to the fact that what is remaining has more entropy/randomness in the code.


Scott

Usually you can squeeze some single % at a cost of heavy computation and of course all inconvenience associated with necessity to pack/unpack archive.

[Cornucopia]

In my experience, and I have tested this multiple times (assuming possible changes in archiving code efficiency), you get at best 1% improvement and at worst 5-10% regression, with VERY long compression & decompression times regardless, so unless you happen to luck out, it is almost NEVER worth it.


Scott

[pandy]

In my experience, and I have tested this multiple times (assuming possible changes in archiving code efficiency), you get at best 1% improvement and at worst 5-10% regression, with VERY long compression & decompression times regardless, so unless you happen to luck out, it is almost NEVER worth it.


Scott

Once again - my experience shows that around single % it is possible - never saw 5..10% regression, for transport streams depends on context gain can be significant - sometimes even close to 50%. But i agree - it is impractical (perhaps except transport streams where stuffing packet content is not random)

[Cornucopia]

Exception where it makes sense to me: workflow which requires encapsulation in archive format for uploading/sharing.


Scott

[Swedaniel]

In my experience, and I have tested this multiple times (assuming possible changes in archiving code efficiency), you get at best 1% improvement and at worst 5-10% regression, with VERY long compression & decompression times regardless, so unless you happen to luck out, it is almost NEVER worth it.


Scott

Some codecs compress the video files very much allready so sometimes the compression is less effective. But the question was allso about storing the mts files, which according to the main post those video files are 150-200MB per minute, so chances are there is alot of storage to be saved by using compression software..

Using WinRar to compress an 195MB Video File with MJPEG Compressor Codec took about 10-15 seconds and the rar package is now 9MB..

[Swedaniel]

In my experience, and I have tested this multiple times (assuming possible changes in archiving code efficiency), you get at best 1% improvement and at worst 5-10% regression, with VERY long compression & decompression times regardless, so unless you happen to luck out, it is almost NEVER worth it.


Scott

Once again - my experience shows that around single % it is possible - never saw 5..10% regression, for transport streams depends on context gain can be significant - sometimes even close to 50%. But i agree - it is impractical (perhaps except transport streams where stuffing packet content is not random)

Still the question is allso about archiving the mst files indicating long term storage to be able to use the video files for later.. An compression software or encryption software is an good choice to preserve any file and to protect it from unwanted changes during the long term storage, aswell to password protect the file together with encryption to defend the data in case of theft and aswell to decrease the file size.

[pandy]

Still the question is allso about archiving the mst files indicating long term storage to be able to use the video files for later.. An compression software or encryption software is an good choice to preserve any file and to protect it from unwanted changes during the long term storage, aswell to password protect the file together with encryption to defend the data in case of theft and aswell to decrease the file size.

Are you sure? - compression focus on size and this is contradictory to error protection - single error, single bit flip make whole archive unusable - scrambling may offer prevention of some errors cumulation and may provide rudimentary encryption but still - if your goal is long term archival storage then some form of error correction coding seem to be mandatory... usually such coding is applied by storage but this is beyond user control

[Cornucopia]

I agree @pandy.


Scott

[Swedaniel]

Still the question is allso about archiving the mst files indicating long term storage to be able to use the video files for later.. An compression software or encryption software is an good choice to preserve any file and to protect it from unwanted changes during the long term storage, aswell to password protect the file together with encryption to defend the data in case of theft and aswell to decrease the file size.

Are you sure? - compression focus on size and this is contradictory to error protection - single error, single bit flip make whole archive unusable - scrambling may offer prevention of some errors cumulation and may provide rudimentary encryption but still - if your goal is long term archival storage then some form of error correction coding seem to be mandatory... usually such coding is applied by storage but this is beyond user control

Haha yeah im sure! The only people i know who's against this that ive met was some sort of cyber criminals, mostly looking to steal or destroy peoples stuff

Alltough for myself i prefer using WinZip and tend to skip the encryption and password protction! But since WinZip require money im using PeaZip which is great, but tbh this WinRar that packed an 195MB video file to 9MB with default settings had me whop out of surprice! Sadly i dont use this codec very much but use an high compression codec (TechSmith) so the video files are allready pretty small and packing them is less effective.. but for me i use my video files on an file server and launch them directly in the web browser daily so it is pretty annoying to use encrypted or packed video files then, alltough im sure i know a few web coders who can set me up with live and automated encryption + compression + decryption + de-compression for normal and daily usage..

[pandy]

Haha yeah im sure! The only people i know who's against this that ive met was some sort of cyber criminals, mostly looking to steal or destroy peoples stuff

Alltough for myself i prefer using WinZip and tend to skip the encryption and password protction! But since WinZip require money im using PeaZip which is great, but tbh this WinRar that packed an 195MB video file to 9MB with default settings had me whop out of surprice! Sadly i dont use this codec very much but use an high compression codec (TechSmith) so the video files are allready pretty small and packing them is less effective.. but for me i use my video files on an file server and launch them directly in the web browser daily so it is pretty annoying to use encrypted or packed video files then, alltough im sure i know a few web coders who can set me up with live and automated encryption + compression + decryption + de-compression for normal and daily usage..

Good for you - luckily i've never met anyone i could call "cyber criminals" - my point was associated with coding theorem not crime - personally prefer open source software and 7z is my favorite archivers from many years.

btw to detect alteration of file you can use hash function - SHAxxx, MD5 are quite popular
https://en.wikipedia.org/wiki/List_of_hash_functions?useskin=vector
https://en.wikipedia.org/wiki/Comparison_of_cryptographic_hash_functions?useskin=vector

[Swedaniel]

Haha yeah im sure! The only people i know who's against this that ive met was some sort of cyber criminals, mostly looking to steal or destroy peoples stuff

Alltough for myself i prefer using WinZip and tend to skip the encryption and password protction! But since WinZip require money im using PeaZip which is great, but tbh this WinRar that packed an 195MB video file to 9MB with default settings had me whop out of surprice! Sadly i dont use this codec very much but use an high compression codec (TechSmith) so the video files are allready pretty small and packing them is less effective.. but for me i use my video files on an file server and launch them directly in the web browser daily so it is pretty annoying to use encrypted or packed video files then, alltough im sure i know a few web coders who can set me up with live and automated encryption + compression + decryption + de-compression for normal and daily usage..

Good for you - luckily i've never met anyone i could call "cyber criminals" - my point was associated with coding theorem not crime - personally prefer open source software and 7z is my favorite archivers from many years.

btw to detect alteration of file you can use hash function - SHAxxx, MD5 are quite popular
https://en.wikipedia.org/wiki/List_of_hash_functions?useskin=vector
https://en.wikipedia.org/wiki/Comparison_of_cryptographic_hash_functions?useskin=vector

Yeah, 7-zip is pretty nice, i havent used the acctual program very much! It seem like the best choice to use when it is necessary to encrypt the file names within the package for delivery to an customer or possibly even an friend.. Using Encryption software such as AES Crypter seem more like an good solution for personal usage! Alltough it compress file surprisingly well allso.. Allso i use QuickZip mostly for fun to scan Zip archives with the built in ClamAV Virus Scanner... Cyber criminals i have met quite many of them, some hang around in forums and chats and it is pretty common for them to link to viruses, scams or come with advice to people that is not so good advice