After much searching, studying, testing various PDF encryption tools
out in the wild FOSS jungle, these are my findings:
*Password-based cryptography*
encryption with signing: use JSignPDF
encryption w/out signing: use qpdf (not pdftk)
decryption: use qpdf (not pdftk)
viewing RC-4, AES-128: evince, okular, or xpdf
viewing AES-256: impossible
(JSignPDF rationale) It's the only FOSS PDF tool that can both sign
and encrypt. But note that the signature
obviously uses key-based cryptography.
(qpdf rationale) qpdf handles both RC4 and AES, and it supports key
sizes of 40, 128, and 256. It does all the crypto
that pdftk does, plus AES, so there is no benefit to
using pdftk for this.
(viewer rationale) evince, okular, and Xpdf can all view RC4 and
AES-128 encrypted documents. Note that Xpdf is
aesthetically favorable, as it's the only viewer
capable of replacing white backgrounds with a
less harsh color.
(viewer limitations) No FOSS PDF viewer can open AES-256-encrypted
documents at this time. The only option for
non-COTS-using recipients is to downgrade or
remove the encryption (using qpdf) before
viewing. Also note that Xpdf has a -opw option
that's broken (it does not parse the
commandline as documented), but the -upw option
works.
*Key-based cryptography*
key generation: use openssl
encryption with signing: use JSignPDF
encryption w/out signing: impossible
decryption: impossible
viewing: impossible
(rationale) JSignPDF is in fact the only FOSS tool to be capable of
encrypting a PDF using a pkcs12 key and producing output
that is still within the PDF spec (unlike gpg, for
example, which imposes pgp-capability on recipients).
(JSignPDF limitations) JSignPDF is primarily designed as a signing
tool, and it forces signing. It is incapable
of encrypting without signing (as a
workaround, one could do a non-visible sig
with a one-time dummy key and then destroy
the key).
(viewing limitations) The payload cannot be decrypted or viewed
using any FOSS tools. So key-based encrypted
PDFs are useless unless the document recipient
uses something commercial and proprietary to
open the document.