Someone recently sent me an email about troubles when opening in Origami encrypted PDF documents produced by Acrobat Pro X. At first I thought it was a bug, but while looking in the data of the document I noticed two unusual things: the specified PDF version was Extension Level 8 and the revision level of the cryptographic handler was 6. However at this time, the latest published specification by Adobe is the Extension Level 5, and it makes no mention about a revision 6 of the security handler.
After some quick researchs, it appears that the specifications for this PDF version have not been released by Adobe, but are yet implemented in Adobe Reader X. This undocumented version makes use of a new password validation algorithm when opening encrypted documents.
Apparently the Extension Level 8 is an intermediate PDF version to prepare for the future arrival of the new ISO specifications (32000-2, aka PDF 2.0). Those are still in development, and the current drafts seem to be only available for the members of the ISO committee. As I could not find any publicly available documentation about the new password validation scheme, I decided to directly take a look inside Adobe Reader X.
Before detailing the algorithm, just a short history about Adobe encryption schemes used in PDF.
The first implementations of PDF used about 50 rounds of MD5 to validate the user password, then used RC4 or AES-128 to encrypt the document contents. With the arrival of Adobe Reader 9 and the PDF Extension Level 3, Adobe switched to a single round of SHA-256 combined with AES-256. While SHA-256 is considered as cryptographically more secure than MD5, this scheme was less resistant against a bruteforce attack. Adobe tried to remove this weakness by replacing the single SHA-256 pass with a custom key stretching algorithm to validate the user password. AES-256 is still the cipher being used for the encryption part.
The new algorithm is based on SHA-256, SHA-384, SHA-512 and AES-128-CBC. It takes a user password encoded in UTF-8 up to 127 bytes, combined with an 8 or 56 bytes salt, and produces a 256-bit hash. I am personally not aware of any known algorithm similar to this one, so I assume this is Adobe's personal design.
Here is the algorithm in pseudocode:
revision6_hash(password,salt,vector=''){block_size=32;input=SHA-256(password+salt+vector);key=input[0..15];iv=input[0..15];i=0;while(i<64||i<x[block_size-1]+32){block=input[0..block_size-1];aes=AES-128-CBC.init(key,iv);for(j=0;j<64;j++);{x="";if(len(password)>0)x=x+aes.update(password);x=x+aes.update(block);if(len(vector)>0)x=x+aes.update(vector);if(j==0)switch(sum_of_first_16_bytes_of_x%3){case0:block_size=32;digest=SHA-256;case1:block_size=48;digest=SHA-384;case2:block_size=64;digest=SHA-512;}digest.update(x);}h=digest.final();input=h;key=h[0..15];iv=h[16..31];i++;}returnh[0..31];}The parameter salt is the 8-bytes (user or owner) key validation salt.The parameter vector is actually only present when hashing the owner password (it then contains the 48-bytes /U key).
I upgraded Origami to version 1,1 and included this algorithm. Beware that documents encrypted with this method cannot be opened in Adobe Reader 9 or earlier. By the way, other undocumented PDF features might also potentially be present in Adobe Reader X.
You can use the helper script pdfencrypt if you want to test it (--hardened switch). For example, using this method with a null password:
$ pdfencrypt -c aes -s 256 --hardened clear.pdf -o protected.pdf
The previous implementation (single SHA-256 pass) is not available anymore in the latest version of Acrobat Pro X, so Adobe manifestly wants to get rid of it.
Update: I missed a few lines in the assembly, so the previous algorithm was only matching Adobe's implementation when using null passwords. This is fixed now.
