Any chance to get raw data from camcorder?

[alex-kas]

Hello folks,

Maybe I'm bad in googling, but. Is there any way to extract real raw data from a camcorder? Primarily from Panasonic x900 or GoPro, or from Nikon d600 camera in the video recording mode.
I mean data BEFORE h264 encoding. FULL UNCOMPRESSED DATA STREAM. For x900 in full HD progressive mode it is:
yuv420 with 1920x1080x25fps which is 6bytes/4pixels times 2073600 pixels times 25 times per second which is 77 760 000 bytes per second (about 74MB/sec) or roughly 622 Mbit/sec.

Practically USB3 is more than capable for that nowadays (its limit is about 400MB/sec, about 5 times more than I ask for.)

Any suggestions are welcome:
preferences tweaking
custom firmware
hand-crafted connection in the middle
soldering of extra chips
...

I know that for some Nikon cameras there exist custom firmwares which allow to significantly increase the output bitrate but yet not about capturing it all, just 2-3 times compared to the standard.

I can accept connections of any controllers in between (like usb3, sata, ...), any external wiring/cabling, extra wall socket connections, etc.
A naive idea is to organize a usb3 controller in between, power it with some 5V adapter and send to its data input the signal from CCD. This should then go to a storage, say, external HDD with proven USB3 speed.
But, there are few issues on the way:
where to connect a wire inside a camcorder?
what exactly raw data are and how to process them?
if (VERY UNfortunately) the built in encoder gets several data streams simultaneously, how to synchronize them?

Any thoughts,
Alex

[aedipuss]

no. not those cams. if you want raw buy something that already can record it like a canon 5d3 modified with magic lantern firmware.

no camera company is going to release schematics so forget mods. the best you can hope for is programmers hacking the firmware.

[Cornucopia]

Incorrect.

All 3 of those cams, and many others, can output "clean hdmi" which bypasses the compression & storage stages, and allow one to connect up with an external 3rd party recorder (usually still compressed, but more like visually-lossless wavelet using 4:2:2 or 4:4:4). Atomos ninja is one such device. Immensely better than 4:2:0 heavily-compressed h264.
Note that this does not mean "raw" in the truest, photog-scientific sense of the term: raw actually means prior to even beyer processing, without subsequent mapping of dynamic range LUTs, and with higher bit depths, and results in HUGE file sizes and bitrates. Like often 1.5-2.5x (or more) uncompressed 24bit rgb.

Scott

[aedipuss]

i was referring to a cam that can record actual raw internally on it's own storage media.

if i'm not mistaken the best a atmos ninja can record is prores 422, better than h264 but not nearly raw

[Bernix]

If only I could recognize any difference between 4:4:4 and 4:2:2

Bernix

[Cornucopia]

Look at the OP's bits-per-second calculation. He's not talking about true raw but uncompressed fhd yuv4:2:0.

Scott

[Bernix]

Misunderstanding just I am unable to see difference in 4:2:2 and 4:4:4.

Bernix

[Cornucopia]

Most people cannot, unaided. That is why 4:2:0 is considered acceptable for many consumer uses.
But trained eyes can see the difference, and if you do any editing/compositing/fx work (esp. with something like greenscreen), it makes a BIG difference. And it makes a difference when zooming/scaling.

Scott

[alex-kas]

I must accept that I'm bad in googling. I had no any idea that hdmi equipped camcorders output by product uncomressed video while recording. It is just incredible. Yes, I confused raw and uncompressed. For the moment I will be more than satisfied with the uncompressed video. Maybe having real raw (i.e. before applying filters, white balance, curves) is interesting but let me play with uncompressed for the moment. Technically speaking, I'm satisfied with white balance, colours, etc., when it comes to my devices.

However, when the stream goes uncomressed from hdmi, how do I know is it 444, 422, 420? By its size? I cannot influence it, I guess, just for my interest. Or can I?
I found that this is undocumented officially at ll (as well as the fact that hdmi outputs clean stream while recording).

Seems that I need ninja or so and some BIGGGG usb3 hard drives.

Thanks for educating myself, hope to try that soon.
Alex

[Bernix]

Thanks for explanation too. It make senses for editing and for zooming. Thats for sure true.
Isnt it sad, that such cameras (in pro bussines) are still using bayer istead of something like Foveon X3. I think there is no money limitation in this industry.

Bernix

[Cornucopia]

Google "clean hdmi" (aka fullscreen hdmi without any overlays, etc) along with your model in question to see if it is compatible.

The recorder it is hooked up to should tell you the specs of the incoming signal, so that should be a good indicator of color subsampling as well as the more obvious resolution and framerate. I would guess if doesn't tell you, the most likely color stream type would be 4:2:2 followed by 4:2:0 and then rarely 4:4:4.

Btw, Foveon was a great idea - perfect registration & no interpolation - , but is hampered in practice by sensitivity issues especially compared to modern beyer sensors. Plus, with the advent of 2-size hdr sensors and extended gamut/extra-primary color options making use of beyond-accuity resolutions, it's a continued big win for beyer methods even with its compromises. And they have made big enough strides in their matrixing algorithms that quality really can be quite good. What I am more bothered by these days is rolling shutter.
But if you ever want to see some SPECTACULAR images with beautiful contrast and perfect registration due to no need for debeyering, go take a look at the Leica Monochrom camera! Even at $3k-$5k it makes me want to go back to greyscale. Some of that stuff still looks great at ISO10000!

Scott

[poisondeathray]

However, when the stream goes uncomressed from hdmi, how do I know is it 444, 422, 420? By its size? I cannot influence it, I guess, just for my interest. Or can I?
I found that this is undocumented officially at ll (as well as the fact that hdmi outputs clean stream while recording).

Usually it's 422, even if it's actually 420 in 422. ie. the signal sent is UYVY 422, but it might only contain 420 worth of information . People measure the chroma resolution on a chroma resolution chart. These are specialized zone plate tests specifically designed to measure chroma resolution. If it's a popular model, possibly someone has already performed these tests. Google it, look specifically on videographer forums, and your camera model forums. Either way, uncompressed (if it's clean on your model(s)) is much better than the onboard compression. But be careful, some models have cadence issues with the HDMI out (some duplicate frames)