new method for file distribution

[deadrats]

as the title suggests, i have devised a new method of file distribution, one that i'm sure will give the mpaa and riaa fits but that will be loved by broadband providers and end users. in a nutshell, rather than distributing the actual file, what end users do is share a description of the file and then use a program to build the file on their computers using said description.

here's how it works: basically all files are stored as a sequence of electrical states, either on or off, and said states are represented as 1's or 0's. basically my plan involves creating either a binary or hex dump of the file and sharing that.

there are still some kinks to work out, namely that the binary/hex dumps are significantly larger than the original file but the resulting text file can be compressed using an archiver back to almost the same size as the original file.

furthermore, i am currently trying to see if it's possible to represent a series of 1's and 0's as a mathematical formula, if it is, then the above problems don't surface as then the file is stored as a math formula.

as it stands now this method of file sharing makes one immune from lawsuits pertaining to copyright infringement as there is no sharing of the actual content, just a representation of the bits that make up the content and instructions on how to recreate it, so someone not only has to download the description but also run the program that rebuilds it, furthermore since the text file can be encrypted, it becomes impossible for anyone sniffing packets to actually know what is being shared.

edit: for the sake of clarity, i did not invent reversible hex/binary dumps, nor did i write the code for the application, the former has existed since the beginning of computers and the latter was done for me, all i did was come up with the idea for using reversible dumps in this manner.

as such, the source code will most likely be gpl'd, and the implimentation will probably be in the public domain as well.

if anyone has a solid math background and can help with the algorithm for representing a series of 1'a and 0's as a function, please help, i will be including your name in the credits section of the final software as soon as it's released (which i'm hoping is real soon).

note: here is a sample series, taken from an actual binary dump:

11111111 11011000 11111111 11100000 00000000

[Krispy Kritter]

Just my 2 cents, but the mpaa/riaa will still care because your are using copyrighted material. If anything, it makes their case against users stronger because they will then have to manually recreate the files. ISP's don't care about the content, they care about the amount of data. Which your plan seems to increase. P4P should take care of the ISP concerns.

[TJohns]

What's being distributed now isn't the original RIAA/MPAA content anyway...someone's already converted it into a different "representation" of 0 and 1 using a technique called ENCODING

[johns0]

Doesnt matter if you change the attributes,you will still be using copyright material no matter how you encode it.

[jagabo]

there is no sharing of the actual content, just a representation of the bits that make up the content and instructions on how to recreate it

That's how all digital video and audio encoding works now.

Are you also going to argue that selling photocopies of books isn't copyright infringement because you're not selling the actual content, just pictures of it?

[deadrats]

Just my 2 cents, but the mpaa/riaa will still care because your are using copyrighted material. If anything, it makes their case against users stronger because they will then have to manually recreate the files. ISP's don't care about the content, they care about the amount of data. Which your plan seems to increase. P4P should take care of the ISP concerns.

at the moment, yes the amount of data transmitted does increase, however if it's mathematically possible to represent a series of 1's and 0's as a math formula then the amount of data transmitted becomes minuscule, you could transfer a hi def movie using a dial up modem in less than a minute.

[jagabo]

if it's mathematically possible to represent a series of 1's and 0's as a math formula

That's how all digital video and audio compression works now.

There have been a lot of people who know a lot more thanyou about information theory, and video and audio encoding working on these things for many years.

[deadrats]

What's being distributed now isn't the original RIAA/MPAA content anyway...someone's already converted it into a different "representation" of 0 and 1 using a technique called ENCODING

thank you, i wasn't aware of that <---please note the sarcasm.

perhaps i wasn't clear, what i want to do is perform a binary dump, represent said dump as a math formula and transmit just that formula.

what i propose is different from "encoding", "bit packing", and encryption, in the manner with which the data is stored and transmitted.

to make this as clear as possible, the actual data isn't shared, just instructions on how to recreate the data.

[jagabo]

to make this as clear as possible, the actual data isn't shared, just instructions on how to recreate the data.

Please read up on Fourier transforms, discrete cosine transformations, and Wavelets. These are the basis for audio/video compression and are exactly what you are describing -- reducing data to mathematical equations which can be used to reconstruct the original data. You are about 50 years and several PhDs too late.

While you're at it you might as well read up on huffman encoding, Liv, Zempel and Welch encoding, and binhex.

[JohnnyMalaria]

Transmitting a simple hex dump has been done for years in the form of Base64. It allows 8-bit binary data to be transmitted in a 7-bit ASCII format. This is obviously less efficient.

As far as a mathematical formula is concerned, for something like a 1GB file, you'd need to be able to uniquely define an 8 billion bit binary number with perfect accuracy. The complexity of such a function would still require a large amount of information. Probably the same amount since you'd effectively be using a transfer function. There would certainly be no ability to create a predictive function. The problem posed is very similar to chaos. Any functions that approximate the original data will fail.

[deadrats]

there is no sharing of the actual content, just a representation of the bits that make up the content and instructions on how to recreate it

That's how all digital video and audio encoding works now.

Are you also going to argue that selling photocopies of books isn't copyright infringement because you're not selling the actual content, just pictures of it?

the difference is a photocopy is an exact facsimile of the original, think of my plan as simple distributing plans on how to make your own gun, but not actually selling guns.

to make this as clear as possible, unlike a traditional encoding scheme where you might have the original file with hex values of:

1h 1h 1h 1h 1h 1h 1h 1h

and you losslessly encode it as

8h

what i want to do is take a series like:


10010101 11000111 00001111 11110000

do the following to it:
10010101
11000111
00001111
11110000

get:

32121323

take that and get:

17

take that and get

8

and combine that with other sums until i arrive at something like this:

S (n=0, i=87) = sqr ((1-n) ^ i) - (1n + i)

and all the would be shared is a text file with the above formula.

the question is whether it's possible to "mechanically" represent a finite, quasi-repeating series of only 2 values with a math formula.

note, that what i wish to share would be protected under the first amendment as free speech, the courts have repeatedly held that the sharing of instructions is protected speech.

to make this as clear as possible, the riaa and mpaa (as well as copyright proponents) would equate the sharing of movies and music the same as shoplifting or stealing a car, i.e. you stole a copy of their intellectual property (interestingly, the courts, at least in the usa seem to have started leaning towards the side that says this is ridiculous) and broadband providers are having shit fits because all the data is supposedly clogging up the networks, what my plan entails is the sharing of the plans on how you could recreate the content.

the difference between what is done now and what i want to do, other than the obvious reduction in needed bandwidth, is that currently i have to have a playable copy of the movie or song on my hard drive and when you are finished downloading you also have a playable copy of said movie or song, under my plan i would have to run the program to create a mathematical representation of the movie or song, something that on it's own can't be viewed or played, said formula would be shared and again once you had said formula you couldn't view the movie or song and you couldn't edit it because it wouldn't exist yet on your hard drive, only when you ran the program and it was done building the file would you have the content.

this so everyone can understand the difference:

original file (warning, hot chick):

http://www.mediafire.com/imageview.php?quickkey=91yjjihdcdn&thumb=5

binary dump:

http://www.mediafire.com/?m2hn3yjy30m

hex dump:

http://www.mediafire.com/?nujmm3tty23

[JohnnyMalaria]

You can't uniquely reduce 32-bits to 3 and expect to get back the 32. There would be more than one 32-bit sequence that would yield 3.

"the question is whether it's possible to "mechanically" represent a finite, quasi-repeating series of only 2 values with a math formula."

No, because the series is not deterministically repeating. The very phrase "quasi-repeating" means it doesn't repeat. There are over 8 billion permutations of 1s and 0s for a 1GB file. You cannot determine from the billionth bit if even the billionth plus ten is 1 or 0. It's just like trying to write an equation for a chaotic process.

[jman98]

OK, I'll bite.

deadrats - I have a Bachelor of Science degree in Computer Science from an American university. So I'm not a PhD, but I say that to point out that I have a degree in the field. I'm not, for example, an accountant who likes to do stuff with videos and barely knows how to use Windows. There is real money to be made with data compression and finding efficient ways to quickly deliver large amounts of data. I would like to know exactly what makes you think that you have had some kind of keen insight that nobody else in the field has ever had. So basically you think that all the PhDs and geniuses who came before you have totally missed the boat. On what basis do you think you've figured out something that nobody else has? Could it be that maybe you haven't? Hmm...

Which of these sounds more rational to you - that you figured out something that has escaped everybody who came before you or that you are hypothesizing something that isn't really workable? Because if it was really workable I kind of doubt you'd be asking for help from us. By the way, if it was easy to do what you propose, compression algorithms would already do it.

Let's say for sake of argument that your method is somehow doable and nobody else ever though of it before. My friend you are 100% delusional if you honestly think that such a method would be immune to MPAA/RIAA action.

[deadrats]

No, because the series is not deterministically repeating. The very phrase "quasi-repeating" means it doesn't repeat. There are over 8 billion permutations of 1s and 0s for a 1GB file. You cannot determine from the billionth bit if even the billionth plus ten is 1 or 0. It's just like trying to write an equation for a chaotic process.

i was afraid of that would be the case.

You can't uniquely reduce 32-bits to 3 and expect to get back the 32. There would be more than one 32-bit sequence that would yield 3.

not necessarily true. while it may not be possible to do it via a math formula, there is more than one way to skin a cat. my backup idea for a method to implement what i want to do was to use nested look up tables, something similar to pascal's triangle. a ponzi scheme of sorts, if you will:

assume the following 2 bytes:

11111111 00000000

using nested lookup tables they would be reduced to:

81 80

and then that would be reduced to

a b

and they would be reduced to:

z

thus with 3 nested lookup tables (built into the program) 2 bytes can be represented by 1, and as the file gets bigger and bigger and the number of nested lookup tables increases, the "compression" becomes greater and greater.

i'm going to start coding it tomorrow afternoon, i hope to have a beta available soon...

[JohnnyMalaria]

If 81 and 80 are hex, you have the same number of bits as the source...

Even with look ups, you are really on a hiding to nothing trying to reduce a system with 16 degrees of freedom to perhaps 4 or 5. How can you represent every number from 0 to 65535 with only 256 numbers? If it were possible, wouldn't you be able to take the resulting bit sequence and repeat the process multiple times until you end up with a single byte?

Ponzi seems apt.

[deadrats]

OK, I'll bite.

deadrats - I have a Bachelor of Science degree in Computer Science from an American university. So I'm not a PhD, but I say that to point out that I have a degree in the field. I'm not, for example, an accountant who likes to do stuff with videos and barely knows how to use Windows. There is real money to be made with data compression and finding efficient ways to quickly deliver large amounts of data. I would like to know exactly what makes you think that you have had some kind of keen insight that nobody else in the field has ever had. So basically you think that all the PhDs and geniuses who came before you have totally missed the boat. On what basis do you think you've figured out something that nobody else has? Could it be that maybe you haven't? Hmm...

Which of these sounds more rational to you - that you figured out something that has escaped everybody who came before you or that you are hypothesizing something that isn't really workable? Because if it was really workable I kind of doubt you'd be asking for help from us. By the way, if it was easy to do what you propose, compression algorithms would already do it.

even as i read the above over and over again, i have trouble believing that i am actually reading it.

using your argument you could just as easily dismiss any idea ever conceived, you seem like the type of guy that would have said to Archimedes"i told you not to be stupid, you moron" because he was proposing his method of exhaustion (which basically was the first implementation of the limit process ever) and by your reasoning if Euclid hadn't already thought of it then obviously it wasn't doable.

Let's say for sake of argument that your method is somehow doable and nobody else ever though of it before. My friend you are 100% delusional if you honestly think that such a method would be immune to MPAA/RIAA action.

you absolutely would be immune to mpaa/riaa action, sure they would try and sue, but i guarantee you they would lose every single time, at least in the usa.

[deadrats]

If 81 and 80 are hex, you have the same number of bits as the source...

you are absolutely right, after the binary dump you end up with 16 times the number of bits as the source, after the first pass you end up with twice the number of bits, after the second pass you end up with the same number of bits and after the third pass you end up with half the number of bits.

Even with look ups, you are really on a hiding to nothing trying to reduce a system with 16 degrees of freedom to perhaps 4 or 5. How can you represent every number from 0 to 65535 with only 256 numbers? If it were possible, wouldn't you be able to take the resulting bit sequence and repeat the process multiple times until you end up with a single byte?

once again you are correct, think of a reversible pascal's triangle, i know you write code so consider the following:


open bin_dump.txt
do until (eof)
read string$
if string$ = "11111111"
then
write "a"
if string$ = "00000000"
then
write "b"

... (until you reach the letter z)
loop

read string$

for i = 1 to 9
if string$ = "aaaaaaaa"
then write "1"
if string$ = "bbbbbbbb'
then write "0"
next i

at the start of the file you put in a simple marker to denote how many times the above process was done.

to reverse it, you would do something like this:

open some_name.txt

read marker$
for i = 1 to marker$

read string$

if string$ = "0"

then write "bbbbbbbb"

and so on.

can you see any reason why you can't hard code the lookup tables into a simple looping mechanism and keep the process going until any file no matter how big is represented by a single byte and still have the process reversible?

[JohnnyMalaria]

Your process is similar to Huffman run length encoding which is lossless. i.e., patterns of bits are encoded by substitution with other sequences. The most common are shorter than the original and, by necessity, the less common are longer. This is used in most DCT-based video compression. Somewhat related is LZW compression.

[jagabo]

Deadrats, there are thousands and thousands of very smart and knowledgeable people who have worked on this problem for many years. All of them have already thought of what you're proposing. Every now and then someone claims they have come up with a revolutionary compression algorithm that can be used recursively like you suggest. None of these algorithms has panned out. That's not to say that you're not the person who will finally crack the nut -- but I won't be holding my breath.

Once again, I suggest you read up on information theory. Here's one place to start:

http://en.wikipedia.org/wiki/Information_theory

Pay particular attention to the concept of information entropy.

[ahhaa]

However... think of it!
Instead of chanting Om padme oom for bowls of rice, Pixar strings trillions of 10010101010101001010101s together and gets billions of dollars (which are just as numerically chimerical). In fact several major industries are based on nothing 'real' at all, except 'string$' theory!

[Legendsk]

A popular myth is that great discoveries come from people who don't know anything about the subject and therefore don't know that what they are trying to do is regarded as impossible. Deadrats is clearly well qualified in this regard, so we should all wish him well.

But that is only a myth. Great discoveries come from people who spend a lot of time and effort learning a topic and gain enough understanding to discover things that others with a more shallow knowledge missed. Unfortunately, only a small minority of those people manage to surpass others who also made the investment and actually make a great discovery.

The version of this problem I remember from school is:
Encode all the knowledge of man (starting with the library of congress) into a string of numbers (such as ASCII). Then put a decimal point in front of the vast number that you get, so it is a fraction. Get a bar, like a yard stick, of very stable material and cut it so that a piece is exactly that fraction. Then just ship the two pieces to someone and all they have to do is measure the bars and they have the number back and can recreate all the information.