Art’s post below got me thinking about BitCasa and the security of the data…and it seems BitCasa’s CEO mentioned something about how they plan to protect the data in a recent interview (http://techcrunch.com/2011/09/18/bitcasa-explains-encryption/). The obvious answer is encryption, but the question is how? Note, I’m not stating this is HOW BitCasa works, simply presenting an option for how this may work.
One issue with successfully de-duplicating data is data encryption itself. So for example, if I have a file and you have a file but our encryption keys are both different than the file appears completely different to the de-duplication system. It fails to identify two exact files because they no longer match. However, there is another way in which we can secure the data using the same key if we derive the encryption key from the data itself. So in a new example, let’s take the file mentioned above and split it into chunks of data. Now, if I hash a chunk and use the hash as the encryption key for the chunk I have a “secure” chunk. If I transmit the chunk across the wire and it is intercepted by an adversary it is still secure as the adversary doesn’t know the plaintext which generated the key for encryption. Sure, depending on the size of the chunk we could be subject to brute-force attacks…so care needs to be taken to make brute-force possible only after the data has “expired” or lost all value (you choose: years, decades, millennia, etc.). Next, I upload the chunk to the server for assessment. Thinking about de-duplication for a second, since the hash and algorithms are all the same (SHA-256 and AES-256 in BitCasa’s case) and the key, which is derived from two identical chunks of data is also the same, the resulting cipher text will also be identical. And if I see two identical chunks on the server-side I know I have a duplicate chunk and only need to store one of the two.
Given that I’m talking about chunks there is another layer to this system which I’m still trying to understand…the metadata. Something has to map all of those chunks to a single file if we are indeed breaking it up into smaller pieces. But that’s for another post…hopefully after BitCasa tells us more on how the system works. Also, the secret-sauce that stores “something” on the local drive needs some explanation as well.
Comments
why not just hash on the client-side before anything is transferred, perform a check on the server-side to see if the computed hash exists and only transmit what isn’t already stored?
Yea, that would save a step…and maybe that is how it works. You would think that it would perform the comparison prior to sending any data across the wire. Since upload speeds are generally not symmetric for home users that would be optimal.
Update: Not sure what I was thinking when I wrote the comment above …you’d be transmitting the keys to the chunks if you sent the hashes only. You could get around this by using public key crypto where you encrpyt the transfer with the public key of the service so it would be protected in transit.