AI Interpolation of NASA footage (Smoothing/Increasing Fps)

[Dutchsteammachine]

For some months i've been messing about with the DAIN-APP. This program uses a trained artificial intelligence neural network that takes two real frames, and 'renders' in-between frames with its neural network.

A lot of early NASA film footage was shot at 6 or 12Fps, and interpolating this to 24 or even higher does wonders to the footage. It becomes much smoother, easier to watch, and you even notice smaller details that were hidden by the previous jittering appearance.

Here are some comparisons of original framerate VS interpolated framerate:













This was interpolated from 6fps to 24:





More coming soon!

[aedipuss]

LOL - smaller details that might not be real. you are making frames up that are "created", as in fictional by a.i. it doesn't seem to belong being used on documentary material.

[KarMa]

LOL - smaller details that might not be real. you are making frames up that are "created", as in fictional by a.i. it doesn't seem to belong being used on documentary material.

Interpolation isn't anything new to fear, otherwise we couldn't even deinterlace anything besides doing SeparateFields + nearest pixel upscaling. While the tools to certainly have expanded. Anyway in the examples provided, with the interpolated version the "created" frames are noticeably softer and actually pulses with grain whenever you hit an original frame then goes soft again. Not really seeing any added detail besides added fluidity. There are however some instances where large color blotches form during high speed camera panning, which certainly detract from the source material.

[jagabo]

You need to raise the duplicate frame decimation threshold in the Apollo 15 EVA clip. There are still lots of duplicate frames making the video choppy.

[Dutchsteammachine]

You need to raise the duplicate frame decimation threshold in the Apollo 15 EVA clip. There are still lots of duplicate frames making the video choppy.

In that test I still just exported the raw 59.95fps MXF file to 12fps and expected it to work. It was because of that video and its strange choppyness that I discovered discovered duplicated frames, which took some time to figure out how to correct.

Here is an EVA properly interpolated. I used a YT source for this one, I eventually want to re-make it with a HQ MXF source.

[Dutchsteammachine]

Do you guys prefer with music:




Or without:




And should I crop the 4:3 16mm to 16:9 or leave it as-is?

[KarMa]

I'd just leave the aspect ratio with what it started with.

[lingyi]

No music and no cropping. Opposed to the frame interpolation also.

I'm curious as to the intent and audience of your project. Is it a proof of concept? It it supposed to provide additional (artificial) information? It is an artistic project?* *This seems likely with the addition of music?

Edit: I see you have a Patreon for "...creating an archive of audio, film, video and photography." It seems your definition of "archive" is different from what most people define as the preservation and/or restoration of materials. I can see an artistic aspect to what you're doing, but I don't see it as archiving, especially with material that's already well archived like the NASA footage.

[Dutchsteammachine]

No music and no cropping. Opposed to the frame interpolation also.

I'm curious as to the intent and audience of your project. Is it a proof of concept? It it supposed to provide additional (artificial) information? It is an artistic project?* *This seems likely with the addition of music?

Edit: I see you have a Patreon for "...creating an archive of audio, film, video and photography." It seems your definition of "archive" is different from what most people define as the preservation and/or restoration of materials. I can see an artistic aspect to what you're doing, but I don't see it as archiving, especially with material that's already well archived like the NASA footage.

Hello lingyi,

Its meant to make viewing easier, and also be of really high quality. There is a lack of good quality, audio/photo/video synchronized footage online, this is what I want t do. The footage in my OP were just some tests. A lot of vids online also are short. I like showing the entire recordings.

As for my Patreon, I have to admin I mostly post stuff about interpolation there. But in the following months I want to focus more on my other works, that include scanning of late 1800s to early 1900s photography, and early to mid 1900s amateur film footage.

I got several films digitized in 2K already and more is coming. Most films will be provided on YT in their original framerate. I am just thinking on how to present footage like this, and make a trailer for it.

You can see photography and film works here: https://www.flickr.com/photos/steamarchive/albums

[Dutchsteammachine]

Apollo 11: First Steps & Moonwalk.





Apollo 14: Landing, EVA& Liftoff:



Apollo 16: Deep Space EVA:



Apollo 13: The Complete 16mm collection:

[lordsmurf]

That music is horribly distracting.
Don't do that.

I click off of videos that play random obnoxious music.

[lingyi]

Okay, I think I understand your intention. But since I was around when the original footage was aired, I highly prefer the original footage as is for it's historical content.

That said, are you going to provide your restored footage outside of YouTube which reencodes your work and loses quality in the process. Part of archiving is making the "original" restored material available in the event that something happens to you or your copies of your work.

IMO, restoration of photos is different in that they're static, and fine as long as you're not adding data that isn't there (i.e. what you're doing by interpolating frames in your videos).

[lingyi]

I took a look at the photos on you account and again I'm confused as to your intent.

You state "Preserving the past for the future.", but don't give any information (other than vague terms: Unknown WWII, Egypt 1937, etc.) about the photos. What is the original source, when where they taken, where were they taken, why were they taken? I'm also confused about why in some of your photos you don't remove scratches and debris. It could be argued that this introduces artificial artifacts, but if you're going to restore a photo, IMO, removal of artifacts that shouldn't be there is part of the process.

I wouldn't beat on you so harshly if you didn't have links to your Patreon on everything you do as it smacks of spam advertising of a hobby under the pretense "archival restoration".

Edit: As an amateur without the level of skill and knowledge you have, I applaud your efforts, but take offense at taking what are apparently public domain properties and seeking to profit off them. At the very least, the original sources should be given credit.

[Cornucopia]

Also, those NASA originals are public domain. Are yours?


Scott

[lingyi]

Nope! His images on Flickr are copyrighted. LOL!

I'd love to see him try to go after someone using "his" work and claiming it as their own!

Edit:

https://www.nasa.gov/multimedia/guidelines/index.html

"It is unlawful to falsely claim copyright or other rights in NASA material."

[Dutchsteammachine]

I took a look at the photos on you account and again I'm confused as to your intent.

You state "Preserving the past for the future.", but don't give any information (other than vague terms: Unknown WWII, Egypt 1937, etc.) about the photos. What is the original source, when where they taken, where were they taken, why were they taken? I'm also confused about why in some of your photos you don't remove scratches and debris. It could be argued that this introduces artificial artifacts, but if you're going to restore a photo, IMO, removal of artifacts that shouldn't be there is part of the process.

I wouldn't beat on you so harshly if you didn't have links to your Patreon on everything you do as it smacks of spam advertising of a hobby under the pretense "archival restoration".

Edit: As an amateur without the level of skill and knowledge you have, I applaud your efforts, but take offense at taking what are apparently public domain properties and seeking to profit off them. At the very least, the original sources should be given credit.

Hello Lingy,

The sources for my material are the original, physical camera negatives/slides. Same goes for any 8mm, 16mm or 35mm films. I own the original celluloid.

(Its more complicated for the NASA stuff, check my reply below to cornucopia)

I buy those from collectors for hundreds of dollars. I then proceed to scan them with my thousands of dollars worth of scanning equipment. Finally they are put online in full resolution, without watermark, and without strong internet compression. You aren't going to find better scans of this kind of material anywhere online, period.


Nowehere do I mention I restore my material before putting it online. I prefer to put the raw, as-is image online. Partially because most of my material is in decent condition anyway, and because hand-restoring thousands of photos would take way too long for a single person. Not to mention motion picture film...

Photos and films like this should be seen. Not stuck, hidden from view inside some collector's basement, gathering dust forever.


I disagree that leaving scratches on could introduce artefacts. They are a physical part of the original medium in its current state. How is removing this physical property, not exactly that: Adding/changing data that is not there on the original. Removing it looks visually more appealing of course.


The context of the source material is not always known. The collector who I buy them from does not always know. I am no professional historian that makes a living on researching this kind of material. Sometimes the material has dates on the film canisters, writing, names. Or on the back of printed photos.

But most of the time, it is my and the job of my watchers to figure it out. I post films and photos on tons of different websites, and usually within a few days enthusiast comment about what they think the date is, location, context, what is going on in the footage. Crowdsourcing like this is really powerful. But I am hesitant to add this to image description, as none of us are experts and could easily be wrong...


As for being in public domain or not... yes, my NASA films are in public domain. And I do give credit to NASA in the video description. However, I can claim ownership of my processing I believe.

I have never looked into establishing copyright on my still and film material. I wouldn't know where to begin. But since most families aren't known (I have been in contact with the family of the photographer of the 1937 egypt photos) Its possible I can get the footage into public domain, or have their copyright granted to me. There are companies who you can pay to research this. These companies try to find the company that holds copyright, or the families.

[Dutchsteammachine]

Also, those NASA originals are public domain. Are yours?


Scott

I see several people have brought up this concern. Let me clarify:

https://www.flickr.com/photos/steamarchive/albums/72157709206151053
Does NOT come from NASA archive. Amateur photography.
Original slides bought from eBay. Scanned with own equipment.
Photographer unknown.
Ever figuring out who owns copyright is highly unlikely.
Put up as owned copyright so companies don't run away with it and profit of my hard work and that of others.


https://www.flickr.com/photos/steamarchive/albums/72157709390757626
Does NOT come from NASA archive. Amateur photography.
Original slides bought from eBay. Scanned with own equipment.
Talked with photographer and family over mail. No transfer of copyright has been discussed.
Photographer and family very happy I restored their photos and shared with them.
If any licensing money is made from the material it will be split 50/50.
Put up as owned copyright so companies don't run away with it and profit of my hard work and that of others.


https://www.flickr.com/photos/steamarchive/albums/72157709066529347
Does NOT come from NASA archive. Amateur photography.
Collection I am borrowing from original photographer so I can scan and share them.
Note description: Scanned slides from American Apollo TV broadcasts. Photography © Bill. Scanning work Niels.
Put up as owned copyright so companies don't run away with it and profit of my hard work and that of others.


https://www.flickr.com/photos/steamarchive/albums/72157709700081717
Does NOT come from NASA archive.
Original prints bought from eBay. Scanned with own equipment.
NASA does not have images online in equivalent quality. In fact, they only scanned film of these photos in their own archive after I requested it.
Processed by me and Paul.
Put up as owned copyright so companies don't run away with it and profit of my hard work and that of others.


https://www.flickr.com/photos/steamarchive/albums/72157711192138486
Roscosmos, not NASA.
Original prints bought from RRauction. Scanned with own equipment.
Not online anywhere else. Though luck finding it at Roscosmos because its virtually impossible to find any photos there.
Put up as owned copyright so companies don't run away with it and profit of my hard work and that of others.


https://www.flickr.com/photos/steamarchive/albums/72157672325898307
Source images/files from NASA archives.
Own processing.
Put up as owned copyright so companies don't run away with it and profit of my hard work and that of others.


As for the NASA film interpolations, they are my own processing, including synchronizing audio and other film/TV sources. I believe my video's are transformative enough. I don't want some company running off with my days of processing and editing work.

[Cornucopia]

Nuff said - Out.

Scott

[Dutchsteammachine]

Nuff said - Out.

Scott

Hopefully I have answered your concerns adequately.

[lingyi]

I'm out too. Don't agree 100% with the OP's choices, but I'll leave it at that.

[Dutchsteammachine]

No, please do continue.

These kind of conversations are important. Lets try narrowing things down to the interpolated NASA footage.

After how much transformation of the original, public-domain content does material become derivative work on which the transformer can apply his own copyright?

[lingyi]

Explained here: https://www.planetary.org/blogs/guest-blogs/20120608-rush-copyright-nasa-images.html

It's not clear which side your work falls on. Consult a lawyer if you want to claim copyright ownership.

I'm definitely far from being a lawyer of any kind, just did a little research because I found it interesting. Which a few minutes of searching brought a possible answer. I'll leave it to you to continue as you wish.

[Dutchsteammachine]

Thanks for the link, it was an interesting read.

I don't think my interpolations are transformative enough for me to claim copyright.

I got to clarify I do not intent to sell any of the interpolations. They will always be online for free.

But I don't want some documentary maker or company to use my many days of hard work just for free. This is why I put them on YT as my copyright.

(I also can't tell you enough how many copyright strikes I get from songs that used NASA audio samples.)

[Dutchsteammachine]

What is your opinion on the photography I put on Flickr?

Say, the launch photos of the Saturn-V, or the World War II photos.


I buy these collections for a substantial amount of money. I then proceed to scan them in high quality with high-end equipment and put them online. This takes a lot of time and effort.

I do this because I think these historically significant photos need/deserve to be seen. I enjoy it when people react with "Wow!" and "Stunning work!"


With many collections like this, it will be very difficult if not impossible to narrow down the photographer and their family.


However, say a documentary maker or book writer wants to use some of my online work in my content. Should I just let them profit off MY hard work for free?

Any money earned will be put into preserving and sharing more unique collections like the ones online now.

[lingyi]

These are questions that a copyright lawyer has to answer. I'd be concerned about the ownership and reproduction rights to any photo or films you buy, especially those with people. IMO, it's bad enough asking people to support through Patreon what for now is apparently a hobby, but becomes really sticky and potentially costly for you when you try to monetize your efforts. People come out of the woodwork when there's money involved.

Questions that I'd think to ask include:

When you bought the prints, slides, films, did you receive written permission to reproduce /sell them for commercial use? Even if you got the negatives, do any other prints exist? What if they do and owner of those prints claims the reproduction and sale of any other prints are rightfully theirs. This is less of an issue with film as the actual negative is developed and for home use viewed. But again, there may be copies of the film in the hands of others who may claim them as their own.

If there are people in the photo/film, there's the additional issue of privacy rights as discussed here: https://commons.wikimedia.org/wiki/Commons:Photographs_of_identifiable_people What if someone recognizes someone in the photo and says "Hey, that's my grandfather!" Do they have the right of ownership and profit from the reproduction of the image? What if the person or their relatives don't want the photos/film/video viewed publicly? It's one thing to stop and recall the publication of a printed book, another, impossible task to remove all instances of them off the internet.

If you do find someone using your "hard work" in their website or printed material, where do you stand on copyright infringement? As I said earlier, it seems to me that unless you have solid written proof that the work is truly in within the realm of your ownership and rights of copyright, you'll likely stand little change of having a judgement in your favor. And even if you are within your rights, are you ready and willing financially to pursue a lawsuit?


Again, I am not in any sense an authority on copyright, just a layman with some thoughts that MAY be rooted in copyright rights and laws. And with that, I'm completely done with this thread as I've probably already said too much about a subject I know little about other than what I think is common knowledge, backed with a few search citations.

[Dutchsteammachine]

Thank you for your time Lingy.

Who owns copyright on orphaned works is indeed a big question.

Best option is always to get in contact with the family, which I managed to do in two occasions. But this is not always possible...

[lingyi]

Came back because what you don't know and assume is scary. Best consult a copyright lawyer before you think about selling your work. They'll be able to tell you what you need before you can claim copyright of any material. AFAIK, in any country, possession doesn't make you the rightful owner of something. What you 'think' is yours, may not rightfully be and you may be liable for claimed damages in addition to any money you made off the sale.

There are copyright scammers out there that claim you've infringed on their copyrighted material and threaten to sue unless you pay to stay out of the courts.

Fact is people either have faulty memories or outright lie, especially when money is involved. What may be claimed to be grandpa's photos, may be something he got from someone else who really owns the right to them. Or they may be something bought at a garage sale and the seller is claiming it for their own. Do a Tineye search on your images and you may find an exact match owned by someone else. As I said above, as long as there is no money exchanged, it's unlikely anyone will come after you. But the instant a sale and money is smelled, people will suddenly come out of the woodwork!

Watch Pawnstars and you'll see lots of people saying grandpa said that something was in the family for generations, but is no older than is. When my grandfather passed away, my Dad and I were looking through his things and my Dad picked up a broken Timex watch. He waxed nostalgic over my grandfather having owned and used it, but I'm sure it was the same watch I had tossed in the trash years before. My grandfather probably picked it up from the trash.

[Dutchsteammachine]

And that is the big question.

How does anyone prove they own copyright to that footage? Be it photos or film. The chances of a film copy are very small, same for prints. Unless a portrait of somebody is clearly filmed and confirmed to be part of the family, I dont see any other way.

I am working on digitizing a large collection of footage from the 1900s to 1950s, much of which has never been released to the public.


Buying collections, getting them digitized and finally put online takes thousands of dollars, but I do it because people are in awe of the material I am working on (Think Netherlands in 1928, amateur 9,5mm recordings of nazi soldiers, german family building a house in Berlin in 1930s to 1940s.)


Obviously if a documentary or company wants to use part of the footage I upload, I want some money or a share of it so I can buy more amateur films and keep the cycle going.

Under what license should I upload that kind of footage? Public Domain, Copyright, Creative Commons, Fair Use?



Anyways, here is an other interpolation:

[lingyi]

As I suggested above, you really, really need to consult a lawyer. Money spent today may save you in the future. It may take years for the real owners to make a claim, but once they do: "Copyright law allows statutory damages as high as $150,000 per work for willful infringement." This is from this article about the Internet Archive being sued for lending digital copies of a greater number of books than are covered by Fair Use. Copyright law allows statutory damages as high as $150,000 per work for willful infringement.

It took them years to go after anyone, but in 2019 Nintendo sued a site sharing their ROMS https://www.polygon.com/2019/9/11/20860039/nintendo-copyright-trademark-infringement-rom-lawsuit

Your thousands of dollars spent on obtaining your videos and pics will get a few hours at best of lawyer time in the future when you're sued for millions.

[Dutchsteammachine]

Yeah... you are right.

I do not think it is a problem if I put any amateur material online of which the author is unknown.

But when material is going to be licenced and sold I am entering a really gray area.

In the Netherlands, orphaned work can be published and sold after diligent search for the author has been committed. This includes for example, searching at online databases, organizations, media archives.

For a lot of material this is going to be rather difficult, because sometimes there is no writing at all on the film canister or slides. Only broad searchwords can be extracted from the visuals or audio. It would help if archives can search by image, like Google Image Search.

If no author can be identified after reasonable, documented diligent search it should be Ok, by Dutch copyright law. Of course i'm not a lawyer...


But the chance that any of my material is ever going to be licenced is rare... last year National Geographic was interested in some Mercury Atlas 8 photos and film but I havent heard anything from them.

Regardless, some more processing of public domain footage!