Current status of ld-decode / vhs-decode (true "backup" of RF signals)

[Titan_91]

If anyone would be interested in making a custom PCI-E card, it could follow the design of this DVR capture card on eBay:

https://www.ebay.com/itm/Dvr-video-capture-card-TE-104C-conexant-cx23883-39/164496634636



This has a few desirable qualities. A relatively simple layout, BNC connectors, and clear to follow widely spaced traces going to each of the video input (vmux) pins on the chip. There are existing PCI-E cards that use the CX2388X chip:

https://www.aliexpress.com/item/4000386216474.html?spm=a2g0s.9042311.0.0.27424c4dT579Xd

And the chip can be purchased in bulk here:

https://elcodis.com/parts/5909894/CX23881-19.html

[Zcooger]

For anyone without legacy PCI socket in their PC this card is a good choice because it was confirmed to work with CXADC driver under linux.
It's important to point that this card is an exception because of design. It uses additonal bridge chip for PCI-PCI express interface conversion.

Ultrasonic VT-600EX



https://www.aliexpress.com/item/4000386216474.html

[dellsam34]

If those cards are designed to only capture CCTV or composite signal how can they be capable of capturing the video drum RF output in the range of 30-40 MHz signal? I'm I missing something here?

[Zcooger]

If those cards are designed to only capture CCTV or composite signal how can they be capable of capturing the video drum RF output in the range of 30-40 MHz signal? I'm I missing something here?

Basicaly analog tuner is useless for Domesday decoder.
You use S-Video/CVBS pinouts for CXADC capture.

[Titan_91]

Basically. Also, these cards have low pass filters in the form of capacitors that terminate to ground. These stock filters cut off at around 5MHz. For best results, you want to remove any capacitors along your signal line and use a separate low pass filter inline to the input. Otherwise the signal will be heavily attenuated. VideoMem previously pointed out the NTSC VHS upper sideband goes past 7MHz. I have a 13MHz filter I'll test with VHS.

I'm still working with that guy on the Laserdisc Database forums to come up with a standard modding procedure for that PCI-E card that will give anyone great results who doesn't want to spend $350+ on a Domesday Duplicator. I've also found these cards can be overclocked to allow 40MSPS 10-bit captures, which matches the Domesday Duplicator. All that is needed is to replace the crystal oscillator that drives the chip clock.

Good observation about the PCI-E bridge chip. Maybe with enough interest something can be made in the future. I believe VideoMem came up with the idea a few months ago.

[Zcooger]

Anyway, here's first demo of software tape dropout correction in video form: https://open.lbry.com/@Zcooger:1/Domesday86-VHS-Decode--First-Tape-Dropout-Compensatio...o-24.11.2020:e
Lbry.tv provides great flexibility with bitrate so I uploaded comparison in 18Mb/s.
I stacked both results from newest version, dropout settings are default.

Overengineered script I used to drive the software:

#!/bin/sh
cpu_threads=$(grep -c ^processor /proc/cpuinfo)

./vhs-decode -p -t $cpu_threads --doDOD --cxadc3 "$1.r8" "$1" -s 0 -l 3000

test -f "$1_docd.tbc"       && rm -f "$1_docd.tbc"      | test -f "$1_docd.tbcc"        && rm -f "$1_docd.tbcc"
test -f "$1_docd.tbc.json"  && rm -f "$1_docd.tbc.json" | test -f "$1_docd.tbcc.json"   && rm -f "$1_docd.tbcc.json"

ld-dropout-correct --threads $cpu_threads "$1.tbc" "$1_docd.tbc"
ld-dropout-correct -i --threads $cpu_threads "$1.tbcc" "$1_docd.tbcc"

test -f "$1.rgb"            && rm -f "$1.rgb"           | test -f "$1chroma.rgb"        && rm -f "$1chroma.rgb"

ld-chroma-decoder -f mono -b --threads $cpu_threads "$1_docd.tbc" "$1.rgb"
ld-chroma-decoder -f pal2d --threads $cpu_threads "$1_docd.tbcc" "$1chroma.rgb"

ffmpeg -f rawvideo -r 25 -pix_fmt rgb48 -s 928x576 -i "$1chroma.rgb" -r 25 -pix_fmt rgb48 -s 928x576 -i "$1.rgb" -filter_complex "[0:v]format=yuv444p[chroma];[1:v]format=yuv444p[luma];[chroma][luma]mergeplanes=0x100102:yuv444p[out]" -map "[out]" -c:v ffv1 -threads $cpu_threads -top 1 -color_range tv -color_primaries bt470bg -color_trc gamma28 -colorspace bt470bg -pix_fmt yuv444p -an -aspect 4:3 -y "$1_doc.mkv"

[oln]

That's neat. It seems the inter-frame DOC as it is doesn't can look a bit off when there is movement (I haven't looked at the code if there is movement detection or something), so it may work better to use the -i option for luma as well for now.

[Zcooger]

Here's short sample with intra-field DOC enabled for luma and much stronger tape damage:

Stacking command:

ffmpeg -i jetix.mkv -i jetix_doc.mkv -filter_complex "[0:v][1:v]hstack=inputs=2[a];[a]bwdif=1[b];[b]scale=2880x1080:in_color_matrix=bt470bg:in_range=tv:out_color_matrix=bt709:out_range=tv[c]" -map "[c]" -pix_fmt yuv420p jetix_stacked.mp4

[dellsam34]

Is this going to be ready before Christmas?...Just kidding. The DOC filter looks pretty damn good, It would be great to post a sample from the VCR out to compare it to, Even the chroma is not that bad for a dark scene.

[Titan_91]

You know what I just realized? The fact that the "wavy" lines scrolling down screen on a physical VCR are a result of digital dropout correction. Except for large areas of static, you never really see the streaks of static, only that pixelated shimmering artifact. So unless you had a really old VCR, you would otherwise not know what actual dropouts look like. Pretty amazing.

Oln, I'm especially impressed with how your DOC algorithm handles the full half a field of static early in the boxing video. I just set the playback speed to 0.25x and compared that specific frame. A lot of picture information is preserved after that really bad field has been corrected. During playback in general, I can't notice any dropouts in the corrected version about 80% of the time. Your approach looks fantastic. It appears luma is a best guess and chroma is repeating on every dropped out line, resulting in a subtle red vertical smear.

[VideoMem]

Only here to say I'm following this thread, and diving deep into the jitter thing. It only turned to be a challenge to deal with.
Maybe I'm over engineering the thoughts a bit.

That DOC is the best, as it takes care of inter-frame data, not only the previous line as in VCR hardware.
The chroma seems to use the older approach.

Yes, the conexant chip with the PCIE bridge glue chip seems the way to go.

Does somebody have success capture more than one channel at a time in RAW mode with CXADC?

I mean for the HiFi part and/or the downconverted chroma path.

Currently I have got lost a bit in the too many paths to get lost in, all equally marvellous.

The other is the lack of a tool to easy, let say:
I have this sample of a signal (original) that is what I want to decode from that RF with all of these parameters (decoder parameters) for optimization.
I can enumerate all the parameters, put them in a range of adjustment, and give the initial conditions of the search.
I can write a cost function to evaluate the SNR between the original and the decoded.
I wish to have some AI toolkit that auto-search in the parameter space for the better parameter combinations that makes the output look as close as the original. Maybe I will end up witting a gradient descent search algo.
AI is new to me, in the sense I know what it does, but some parts on larger networks beyond the perceptron theory are still obscure to me, and I never set up it before.

I have been experimenting with the CX decoder (expander part of the compander noise reduction in LD).
There is an implementation already done, but this one is pure GNUradio flowchart.
Zcooger told me there is DBX in some HiFi VHS recorders too.
But halted the expander there until I get a way to optimize parameters in a semi-automated way.

By the way, I started a repository here with the tools I'm using for tests besides the vhs-decode / ld-decode.

https://github.com/VideoMem/Superheterodyne-decoding-tools

The title is ambitious, it currently have some test decoders and some code boilerplate I'm doing for the signal standards auto-detection.
It involves observing the peak distances in the RF spectra to classify if the capture is PAL/NTSC LD or a VHS capture.
The way the players autodetects the standards by hardware is still partially unknown to me.

Currently it only counts how many times it detect a peak from a size N FFT in the frequencies range +- some error parameter to build an histogram (and it does it poorly, because I'm not 100% sure that's the way).
Also I stepped in the fact the way Python performs in small read sizes is the worst, but nothing that can't be adressed.
Ignore the RFprobe folder for now.

It includes the Popular Electronics (January 1982) CX decoder schematic for LP (as I guess it is the same as the LD, but with the envelope control input filter moved from 100Hz to 500Hz and without the de-emphasis.)
It turned to be the expander analogue of a parallel compressor.

Thank you for the new discoveries update!
I will add some history here when I discover new things too.

[dellsam34]

Zcooger told me there is DBX in some HiFi VHS recorders too.

I have never heard of this before, I know most VCR's and pre-recorded tapes used some sort of NR on the audio linear mono track, but I wasn't aware of any NR for the Hi-Fi audio track.

[Zcooger]

Use of DBX II compander is mentioned in specification of some VHS HiFi capable decks. In official IEC specification it's called as an compander, encoder, decoder, compressor and expander. It's connected with "some" noise reducion system and it's present on every tape with helical scan audio implemented better or worse.

[dellsam34]

I start seeing them now:
https://www.amazon.com/gp/product/B009VTJFOS?SubscriptionId=AKIAJ4INUWPJ5KNDG3&tag=50kusd00605-20

Given the fact that capturing is mostly geared towards the home video tapes I don't think any consumer camcorders offered DBX for recording audio therefore I don't think it should be a concern for now.

[Titan_91]

Going back to that $50 PCI-E CX2388 card, I was pointed to one that is half the price of even that, with two BNC inputs! If someone can figure out a way to do simultaneous video and Hi-Fi captures at 8fsc (28.6MSPS) that would be awesome! Low pass filter caps going to ground will need need to be removed in place of a better filter for VHS, and any inline resistors potentially bypassed to get the strongest signal.

https://www.aliexpress.com/item/4001286595482.html

I know combining the video and Hi-Fi signals into a single feed has been done, but having separated signals will 100% ensure no crosstalk from image frequencies from one or the other. Plus, VCRs have separate test points for PB FM and Hi-Fi FM anyway. The Hi-Fi signal is so strong in fact, that post-amplification may not be needed. So in theory someone could connect an HF amplifier to the video input only and get a balanced result, without having to attenuate the needlessly amplified Hi-Fi signal (which could create crosstalk on the same cable).

In addition to those benefits, I wouldn't personally combine the feeds on my VCR. It could cause problems with the Hi-Fi signal back feeding into the PB FM circuit and vice-versa.

[jfoust]

I'm excited to hear about the existence of this project. I've been hoping it will rescue some old tapes I've been unable to play.

I bought a consumer Sony 8mm camcorder in 1992. In 1995 or so, it started having the well-known problem with surface-mount electrolytic capacitors that began to decay.

In 1997 or so, I discovered that although I could playback the video I had just recorded in the camcorder's LCD preview, that I was unable to play video I'd recorded months or years before. I also had a Sony Video Walkman and it couldn't play these tapes, either.

I'd always hoped that some alternative new low-level method of reading these tapes would emerge, something that could scan the recorded signal and reconstruct in software. This would rescue video of the first few years of my kids growing up. Perhaps there are other people who have tapes in the same situation that they'd like to rescue.

[Zcooger]

Hello, it's nice to see this project got your interest. We're hoping to meet more people like you.

Let's finally post something new after small break - first 2021 sample of software decoded tape with current version, have a watch. I'll try to describe it in detail later.

https://lbry.tv/@Zcooger:1/vhs-decode-2021-test:2?r=8CHypuz7Dp743zh8cYmwe4g8eC9soFfU

[Mischa]

delete me

[Mischa]

delete me

[Mischa]

Going back to that $50 PCI-E CX2388 card, I was pointed to one that is half the price of even that, with two BNC inputs! If someone can figure out a way to do simultaneous video and Hi-Fi captures at 8fsc (28.6MSPS) that would be awesome! Low pass filter caps going to ground will need need to be removed in place of a better filter for VHS, and any inline resistors potentially bypassed to get the strongest signal.

https://www.aliexpress.com/item/4001286595482.html

This seems to be CX23885 (PCIe) based. Should not work according to this?

If the information in the link is false or outdated let me know. I'm looking to get into some raw RF capturing and helping with the development of vhs-decode.

[oln]

Yeah the isn't any driver for doing raw capture with the PCIe ones. Whether it's possible to develop one or not I don't know, but the chip differs a bit from the PCI cx88x chips.

EDIT: The linked card may actually have the pci version + a pcie bridge. It's a bit hard to see chip number on the cx chip from the image image, but there is a second chip there that sits in between the cx chip and the pcie connector. We've discovered that there are a fair bit of cards with that setup and BNC connectors being sold. I don't know what they're used for, maybe some legacy hardware/software DVR stuff that depends on drivers for the older chip.

A user on the ld-decode/vhs-decode discord has also done some experiments with replacing the oscillator on a cx card which apparently made higher sample rates more usable.

In 1997 or so, I discovered that although I could playback the video I had just recorded in the camcorder's LCD preview, that I was unable to play video I'd recorded months or years before.

I suspect your problem tapes may have been recorded with the camcorder being out of alignment. It is not something software decoding will fix on its own as the problem is that the playback device isn't picking up all of the signal properly. One way to test is to see if there is any image when seeking (i.e fast-forward or rewind, hold the button after starting playback.). Sony's camcorders will blue-screen when the signal is too bad, but not when seeking so it allows you to see if there is something there. The way to rescue such a recording is to intentionaly misalign the playback device to how the recording one was, but it can be a but fiddly and hard to do (and also makes it hard to fully restore the alignment to standard afterwards.) Some camcorders and vcrs, like e.g Hitachi camcorders do not blue screen which I've found helpful for dealing with tapes that are so messed up you can't recover everything even with adjusting alignment.

Software decoding can help help get the most out of the signal that is read, but it won't be able to do anything if the signal isn't being read in the first place.

[Mischa]

Yeah the isn't any driver for doing raw capture with the PCIe ones. Whether it's possible to develop one or not I don't know, but the chip differs a bit from the PCI cx88x chips.

EDIT: The linked card may actually have the pci version + a pcie bridge. It's a bit hard to see chip number on the cx chip from the image image, but there is a second chip there that sits in between the cx chip and the pcie connector. We've discovered that there are a fair bit of cards with that setup and BNC connectors being sold. I don't know what they're used for, maybe some legacy hardware/software DVR stuff that depends on drivers for the older chip.

You are right. Taking a second look at the photo, zooming in to decipher the (barely readable) text, I'm able to make out the smaller chip to be a ASM1083. Which is indeed a PCI to PCIe bridge.

Then the other chip must be an CX23883. Which is indeed a PCI Audio/Video Broadcast Decoder.

Looks like a fine deal for €20.

I'd be interested to see why the PCIe variants of this chip won't work but I have other battles to fight for now

[dellsam34]

Hello, it's nice to see this project got your interest. We're hoping to meet more people like you.

Let's finally post something new after small break - first 2021 sample of software decoded tape with current version, have a watch. I'll try to describe it in detail later.

While the picture looks very stable, The quality is not there yet, Chroma noise, dot crawl, less details... I can still capture better than this using a broadcast analog to SDI capture device. Maybe the recording VCR is low quality, We would like to see a capture using the normal capture card to compare the two samples and have an idea of the original recording quality.

[Titan_91]

I've been working quite a bit with the person on Discord regarding various capture cards and the 40MHz crystal swap. I am actually using one of the overclocked cards now with the new crystal. It's very stable, good performance and no dropped samples that I can see. I haven't posted in this thread lately due to various issues with getting a better signal to noise ratio out of my workflow (and now compatiblity issues with my LaserDisc player). When I'm done messing with the LaserDisc stuff I'll try a new amplifier I have with my VCR and post another update.

[pitterbrayn]

Not really much easier. On laserdiscs the big challenge is having a good comb filter to filter out chroma/luma crosstalk. VHS (and most other videotape formats besides betacam and some of the early open reel systems) use a "color-under" system. The color is downconverted to a lower frequency and phase rotated (different for each of the two used video heads) when recording to tape, when decoding one has to undo that. Additionally, one has to compensate for tape jitter (not a large issue on laserdiscs) as the phase of the color signal determines the hue of the color.

[dellsam34]

Not really much easier. On laserdiscs the big challenge is having a good comb filter to filter out chroma/luma crosstalk. VHS (and most other videotape formats besides betacam and some of the early open reel systems) use a "color-under" system. The color is downconverted to a lower frequency and phase rotated (different for each of the two used video heads) when recording to tape, when decoding one has to undo that. Additionally, one has to compensate for tape jitter (not a large issue on laserdiscs) as the phase of the color signal determines the hue of the color.

Line jitter which usually corrected by the VCR's line TBC (if equipped) can be corrected I believe in software, The duplicator can capture the entire scan line including the horizontal sync data and just change its timing to get a perfect alignment with the adjacent lines. Then later it's just a matter of cropping like we normally do when capturing with a conventional capture device and crop from 720 to 704 to get rid of the padded 16 pixels (8 on each side).

As to the chroma under, I think the software compensate for that, and I'm pretty sure Zcooger and others are aware of that.

[oln]

Not sure what pittenbray was trying to say, the text in their post was just copy-pasted from one of my earlier posts.

[Titan_91]

Chroma under is mixed up to 3.58MHz/4.43MHz chroma and the existing ld-decode chroma processing code is used at that point, as a real VCR does. There's a current bug in the NTSC code that causes a phase rotation issue with every field, and wrong colors as a result. If oln is able to resolve that at some point I believe vhs-decode will be ready for real applications of decoding NTSC tapes. But PAL support is his main focus, which is fine as hopefully improvements made on the PAL side can also be implemented to NTSC.

Here's a really good PAL decode, and it's a year old:

https://www.youtube.com/watch?v=TcdhJsssdcs

[Titan_91]

My hard drive died. After installing Linux Mint 20.1 (Ubuntu 20.04 base) with a new drive I can no longer build cxadc. I didn't have this issue before. Here's what I get following the GitHub instructions:

user@mintiplex:~/Downloads/cxadc-linux3-master$ make && sudo make modules_install
make -C /lib/modules/`uname -r`/build M=$PWD
make[1]: Entering directory '/usr/src/linux-headers-5.4.0-62-generic'
  Building modules, stage 2.
  MODPOST 1 modules
make[1]: Leaving directory '/usr/src/linux-headers-5.4.0-62-generic'
make -C /lib/modules/`uname -r`/build M=$PWD modules_install
make[1]: Entering directory '/usr/src/linux-headers-5.4.0-62-generic'
  INSTALL /home/user/Downloads/cxadc-linux3-master/cxadc.ko
At main.c:160:
- SSL error:02001002:system library:fopen:No such file or directory: ../crypto/bio/bss_file.c:69
- SSL error:2006D080:BIO routines:BIO_new_file:no such file: ../crypto/bio/bss_file.c:76
sign-file: certs/signing_key.pem: No such file or directory
  DEPMOD  5.4.0-62-generic
Warning: modules_install: missing 'System.map' file. Skipping depmod.
make[1]: Leaving directory '/usr/src/linux-headers-5.4.0-62-generic'
user@mintiplex:~/Downloads/cxadc-linux3-master$ sudo insmod cxadc.ko vmux=2 level=1
insmod: ERROR: could not insert module cxadc.ko: Invalid module format

dmesg log shows:

[   85.295496] cxadc: version magic '5.4.0-58-generic SMP mod_unload ' should be '5.4.0-62-generic SMP mod_unload '
[   91.645527] cxadc: version magic '5.4.0-58-generic SMP mod_unload ' should be '5.4.0-62-generic SMP mod_unload '

I did try a reboot, same issue. I built the module again and tried to insert and got the same error again. For some reason make wasn't building against the active kernel, it was building against another version of the kernel that was installed. I figured it out, what happened was I built the module then installed a kernel update. Following the reboot, it wouldn't insert because of the kernel version mismatch. Running make again should have worked, and it used to for me in the past when I installed kernel updates. But it did not this time.

The root cause was make was re-using one or more files in the cxadc source folder. I had to delete the folder and re-extract it clean from the GitHub zip file. Ran the same command again and cxadc built, inserted, and is working now no problem.

[uff]

So, I found this project and really want to try it out. I have 3 VCRs. One "modern" JVC (HR-J668EU), which does not seem to have a pre-amp output (and I don't have an amplifier either). One older Panasonic, which has no LP mode (which makes it impossible for me to test all tapes) and finally an Akai VS-G712.

The Akai has a dedicated pre-amp board, at least so it seems. But... I can't seem to figure out where to tap it. The service manual lists "PB Y FM", which should be the proper spot, right? Probing that pin with my multimeter only yields that there's no AC at all (not even in the mV mode) and a constant ~3.2VDC, regardless of what I play (iirc, I don't have it set up right now). The other pin, "PB Envelope" suffers from the same thing.

Sadly, I don't have a scope so I can't figure out the actual waveforms, but connecting it to my cxadc kind of confirms there is no signal present (also plugging it into the card drops the voltage on the multimeter to like 0.4VDC, which is kinda scary ). I then tried probing random other pins to check if there is any AC one some pin at all, and it doesn't seem like it.

Am I missing something or is this VCR a lost cause? If needed I can provide the SM.