Develop Reference

changing the hash digest to big-endian in Python

I am using the following code to create hash for a message:
from Crypto.Hash import SHA256
from Crypto.Signature import PKCS1_PSS
msg = b'\x15\x00\xea'
hash = SHA256.new(msg)
...
signer = PKCS1_PSS.new(privKey)
signature = signer.sign(hash)
However, I need that the hash digest value i.e hash.digest() to be in big-endian format.
Please note that I need to pass the whole hash to the signer. I can get the hash.digest() and convert it to big-endian, but then I need to construct a variable of type hash to pass to the signer.sig() function.
How can I do that?

As mentioned in my comment, I'm not sure if it is really required to invert the hash.
But if so, you need an implementation that does not process the raw data, but the already hashed data (and does not hash anymore).
PyCryptodome does not support this, but the Cryptography library with the Prehashed class does, s. here, 2nd snippet.
The following code performs a signing with PKCS#1 v1.5 and:
with PyCryptodome (implicit hashing)
with Cryptography (implicit hashing)
with Cryptography (using the Prehashed class and explicit hashing)
with Cryptography (using the Prehashed class and explicit hashing) and using the inverted hash
# Short key only for testing purposes!
pkcs8 = '''-----BEGIN PRIVATE KEY-----
MIIBVQIBADANBgkqhkiG9w0BAQEFAASCAT8wggE7AgEAAkEA2gdsVIRmg5IH0rG3
u3w+gHCZq5o4OMQIeomC1NTeHgxbkrfznv7TgWVzrHpr3HHK8IpLlG04/aBo6U5W
2umHQQIDAQABAkEAu7wulGvZFat1Xv+19BMcgl3yhCdsB70Mi+7CH98XTwjACk4T
+IYv4N53j16gce7U5fJxmGkdq83+xAyeyw8U0QIhAPIMhbtXlRS7XpkB66l5DvN1
XrKRWeB3RtvcUSf30RyFAiEA5ph7eWXbXWpIhdWMoe50yffF7pW+C5z07tzAIH6D
Ko0CIQCyveSTr917bdIxk2V/xNHxnx7LJuMEC5DcExorNanKMQIgUxHRQU1hNgjI
sXXZoKgfaHaa1jUZbmOPlNDvYYVRyS0CIB9ZZee2zubyRla4qN8PQxCJb7DiICmH
7nWP7CIvcQwB
-----END PRIVATE KEY-----'''
message = b'The quick brown fox jumps over the lazy dog'
# 1. PyCryptodome (implicit hashing)
from Crypto.Signature import pkcs1_15
from Crypto.Hash import SHA256
from Crypto.PublicKey import RSA
privateKey = RSA.import_key(pkcs8)
digest = SHA256.new(message)
signature = pkcs1_15.new(privateKey).sign(digest)
print(signature.hex())
# 2. Cryptography (implicit hashing)
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.asymmetric import padding
from cryptography.hazmat.primitives import serialization
privateKey = serialization.load_pem_private_key(
pkcs8.encode('utf8'),
password=None,
)
digest = hashes.SHA256()
signature = privateKey.sign(
message,
padding.PKCS1v15(),
digest
)
print(signature.hex())
# 3. Cryptography (explicit hashing)
from cryptography.hazmat.primitives.asymmetric import utils
digest = hashes.SHA256()
hasher = hashes.Hash(digest)
hasher.update(message)
hash = hasher.finalize()
signature = privateKey.sign(
hash,
padding.PKCS1v15(),
utils.Prehashed(digest) # The digest must still be specified because the digest ID is included in the signature.
)
print(signature.hex())
# 4. Cryptography (explicit hashing), inverse hash
hashBA = bytearray(hash)
hashBA.reverse()
hashReversed = bytes(hashBA)
signature = privateKey.sign(
hashReversed,
padding.PKCS1v15(),
utils.Prehashed(digest) # The digest must still be specified because the digest ID is included in the signature.
)
print(signature.hex())
with the output:
8c83cad897eda249fec9eba231061d585dafc99177267e3e71bb8a3fce07cc6663bf4df7af2e1c1945d2a6bb42eb25f042228b591fc18cda82d92caae844670c
8c83cad897eda249fec9eba231061d585dafc99177267e3e71bb8a3fce07cc6663bf4df7af2e1c1945d2a6bb42eb25f042228b591fc18cda82d92caae844670c
8c83cad897eda249fec9eba231061d585dafc99177267e3e71bb8a3fce07cc6663bf4df7af2e1c1945d2a6bb42eb25f042228b591fc18cda82d92caae844670c
550ba1cd3c968fa1e24b79a939edb6740b63d2ab021fe87f0639ce978d5127792661e83f4e8fdff8124a12fe208bd70bdca9db2b9c82306f2ed018ab06363c9e
Since the PKCS#1 v1.5 signature is deterministic, cases 1 to 3 produce the same signature.

The SHA-256 standard uses 32 bit words, and those are already specified to be big endian.
If you get 32 bit little endian words then you may need to reorder the bytes within them, but this is unlikely.

How to migrate old python code from m2crypto to cryptography (RSA signing of a digest)

Hi I'm having old legacy code (python2) , that creates a signature file for a digest. (in my case this digest is always a 40 character string with only 0-9a-f characters).
It uses m2crypto
I'm also having a function, that verifies whether the digest and a signature file do match.
Now I have to reimplement the signing code using cryptography instead of m2crypto (and switch to python3 lateron)
However the resulting signature must be verifiable with the old m2crypto code. (The code doing the verification runs on machines.
that I cannot update now, the signing however has to be migrated to cryptography (and python3)
Attached I have a full self explaining example.
I guess the issue lies somewhere in the padding, which is not identical with the m2crypto and cryptography solution. but I don't know how to proceed.
The old signing code is:
def old_sign(digest):
import M2Crypto
rsa = M2Crypto.RSA.load_key("k.key")
return rsa.sign(digest, "sha1")
The code to verify the signature is:
def old_check_signature(digest, signature):
import M2Crypto
rsa = M2Crypto.RSA.load_pub_key("k.pub")
try:
rsa.verify(digest, signature, algo="sha1")
return True
except M2Crypto.RSA.RSAError as exc:
args = exc.args
if len(args) < 1:
raise
return False
I don't know what kind of padding has been used by m2crypto and I don't know all possible variations of padding.
I tried two different paddings with cryptography
First attempt:
def new_sign1(digest):
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import padding
from cryptography.hazmat.primitives.asymmetric import rsa
key_data = open("k.key").read()
key = serialization.load_pem_private_key(
key_data, password=None, backend=default_backend())
return key.sign(
digest,
padding.PSS(
mgf=padding.MGF1(hashes.SHA1()),
salt_length=padding.PSS.MAX_LENGTH,
),
hashes.SHA1())
second attempt
def new_sign2(digest):
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import padding
from cryptography.hazmat.primitives.asymmetric import rsa
key_data = open("k.key").read()
key = serialization.load_pem_private_key(
key_data, password=None, backend=default_backend())
return key.sign(digest, padding.PKCS1v15(), hashes.SHA1())
After Paul's feedback I also tried
def new_sign3(digest):
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import padding
from cryptography.hazmat.primitives.asymmetric import rsa
from cryptography.hazmat.primitives.asymmetric import utils
key_data = open("k.key", "rb").read()
key = serialization.load_pem_private_key(
key_data, password=None, backend=default_backend())
return key.sign(
digest,
padding.PKCS1v15(),
utils.Prehashed(hashes.SHA1()),
)
which fails with
ValueError: The provided data must be the same length as the hash algorithm's digest size.
Following code can reproduce the error if all above functions are declared:
dig = "f7ff9e8b7bb2e09b70935a5d785e0cc5d9d0abf0"
sigs = [old_sign(dig), new_sign1(dig), new_sign2(dig)]
print([old_check_signature(dig, sig) for sig in sigs])
The output is [True, False, False], which means, that only the signature created with m2crypto is correct and the signatures created with cryptography cannot be verified with m2crypto.
If you want to test the code and don't have a key file and a public key file you can use following snippet to create one:
KEY = """
-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEAusZYCLS8RPUJ3fhod3cr2foK++t3R30Eiqstq+B5vIIrcJN1
2HnKfU1o3m9cwEJOezHxuI7Ks8/YVt/bGy6HyY4i3bk5AS9BK4gEptmUlldsvV+l
hOLel884dwhUk2ZrEpCq7YvSpMBb7ORKcUwcTu9tyuSHTlVpabYADTStebDWwp0+
heeRWN/zG5CN0zO4BScRNTpnw/TAqBTytuWgo/YFhBX6a4U3fKKASqynn5ZJYAg3
CDqdAe4BM2OZUv73UHWQsltC3xyKqsN6+gH7O7WoetxdMZsvYtEp4hvhUjgCrDSU
gSvtZD8BMxcPPO59w670DdVa3EPLWZzhGel18wIDAQABAoIBAB+DrAL8C/BOsDWF
3oqZzwpeiE/tcRjc3VFQhMpFfAT0qcO6/d1i32m5EALII4xFI9zhlnmfjlA8t7Ig
32V8umil1Pg4cofio0pnDvHgMJQVeEGTy+faJ9jRnCNpgmvEkjh1tIGUYBxwYJJe
CrmHMBeZipr7aGEtRDYUAXo48zRe+mO41ICkPeVRRSK0b5F9P7SOm+QnkN8rHmtm
MsCJ0jPJ1fG5Xi5BVVdjhQ6+S42tbm/A68Yx/uWSC3pt+ScIklgYF2owawVWDbWJ
Eekl0sNUTm1j1OlnyAEbb8R/VkYsLYfEwpgULTzkokBRq4NZxjCKY7VJRZ05neev
4JbdcOECgYEA46e+L/uvHUUzyhWgrNgHAdT7Eff4Imu7WdG+KQ7d7kNazMJFyMrk
1Ln66i/oPyiMwMXcACmEoA5Jl/wBk3sjKjLno9dvMM3amfwkBZav4FirQS+9Xk/l
EQAoc/WxsxlVywM00c20CgGqNu0hRBlM5C8zCoIFlt6G2Mwv58yd8hkCgYEA0geU
7FXh/xVnq4wU9bnTF0laPd4HlVYYNpRMjTdRwwIycrCynAlfEF+keUyOXhgYw5vm
loyICCqsNi5pw8tRPFyzBiESkLUL+ZTcfxswfgmhlTs71Zk72b8LCM3sRPeuUg2a
6GrNvycWRLfc+toq5NvSKnfVQS2tgMKJ0CtPIesCgYB+Za0H8SKaCsklY3qxXMQP
NVQs9tOTMON1jCmbnECGQGlSlG6wfE4u+g+hJPY60uXLRk/O2z5iq2wa8XVikBTH
IjpQUpXOsAy2QDMz0yVVV4XGDJ6ElbFmDgNn1rtR6DglHmOeNSrH/4KlOmWk7LMv
YjFhnS1DRcvy5POYLJhpSQKBgEILIEkwuF/92xuWcQDT7gzkg/vwVXIgIH0JJQlC
2/L2PebSqVdnmv0LFi0OZbYw3Zik7V1p01y+Dmj7L0biKClS/PhwbeYTCDDzHmLZ
qeX4IVdLyQThqnBOIqoiFqmZOLeUj6GF9Cynndj99/7pm5NbjDrOc8CLHIPgqHVN
KRUBAoGBAKEr1mg+Hjf+M/5ru1tQT6xTvlKW1rS4ioGB35XQf+n/OAO98SPQkwPQ
XfX5nKHif/TmDZygGeHYm96qCceOmzCL7LqaYOb1qTZLEk6L18eEtHwG+hXyMKuA
t2XLK89blAoPT+9x10KKp27IWe/W+QCD6LnrdjoN9BQ7fCnV5v3Q
-----END RSA PRIVATE KEY-----
"""
open("k.key", "w").write(KEY)
PUB_KEY = """
-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAusZYCLS8RPUJ3fhod3cr
2foK++t3R30Eiqstq+B5vIIrcJN12HnKfU1o3m9cwEJOezHxuI7Ks8/YVt/bGy6H
yY4i3bk5AS9BK4gEptmUlldsvV+lhOLel884dwhUk2ZrEpCq7YvSpMBb7ORKcUwc
Tu9tyuSHTlVpabYADTStebDWwp0+heeRWN/zG5CN0zO4BScRNTpnw/TAqBTytuWg
o/YFhBX6a4U3fKKASqynn5ZJYAg3CDqdAe4BM2OZUv73UHWQsltC3xyKqsN6+gH7
O7WoetxdMZsvYtEp4hvhUjgCrDSUgSvtZD8BMxcPPO59w670DdVa3EPLWZzhGel1
8wIDAQAB
-----END PUBLIC KEY-----
"""
open("k.pub", "w").write(PUB_KEY)
Thanks in advance for any suggestions.
I don't use SO that often,
I have also a full blown python file (102 lines), that is completely self contained. How to share such files on SO in a question
Addendum 2019-06-13 23:15:00 UTC
It seems. that M2Crypto.RSA.sign()
calls finally following C code. (from the m2crypto git repository in the file SWIG/_m2crypto_wrap.c.
I'm not really good with libssl, but perhaps somebody else understands this
and can tell me how to 'reproduce the calculation' (perhaps with openssl commandline or with ctypes) without using M2Crypto.
PyObject *rsa_sign(RSA *rsa, PyObject *py_digest_string, int method_type) {
int digest_len = 0;
int buf_len = 0;
int ret = 0;
unsigned int real_buf_len = 0;
char *digest_string = NULL;
unsigned char * sign_buf = NULL;
PyObject *signature;
ret = m2_PyString_AsStringAndSizeInt(py_digest_string, &digest_string,
&digest_len);
if (ret == -1) {
/* PyString_AsStringAndSize raises the correct exceptions. */
return NULL;
}
buf_len = RSA_size(rsa);
sign_buf = (unsigned char *)PyMem_Malloc(buf_len);
ret = RSA_sign(method_type, (const unsigned char *)digest_string, digest_len,
sign_buf, &real_buf_len, rsa);
if (!ret) {
m2_PyErr_Msg(_rsa_err);
PyMem_Free(sign_buf);
return NULL;
}
signature = PyBytes_FromStringAndSize((const char*) sign_buf, buf_len);
PyMem_Free(sign_buf);
return signature;
}
A workaround using subprocess.Popen or ctypes (directly attacking libssl) is (in my context) acceptable.

M2Crypto is probably using PKCS1 padding when calling sign so the PKCS1v15 example is close to what you want. Assuming it defaults to PKCS1 (and not no padding, which is wildly insecure and cryptography does not support) then the digest is likely the problem.
Is digest already the hash you want signed or is it the data to be hashed? Cryptography, by default, hashes the provided data for you. If the data has already been hashed then you need to use Prehashed to avoid that step.

Fixing Invalid signature when decrypting fernet token

i am relatively new to pyhton and the cryptography module, so i'm trying to learn the basics of encrypting and decrypting.
It all works fine when i encrypt a file and decrypt it on the same program, but if i try to just run a decrypt code on a pre-encrypted file (i used the same key, of course) i get an InvalidSignature error, followed by an InvalidToken.
Now, i assumed that for some reason the key didn't match, but they are indeed the same.
Then i thought that for some reason i was passing a string instead of a byte to the functions, or that there are some sort of conversion errors that might alter the encrypted message. But the encrypt-decrypt code works, so i can't figure why the decrypt-only should face errors.
At last, i had a look at the source code for the decrypt function, and tried to figure out if the time stamp had something to do with the error i get, but i couldn't get anything relevant since i'm not too experienced.
This is the encrypt-decrypt code: given a password by the user it encrypts and prints a file, that can decrypt right away.
import base64
import os
from cryptography.fernet import Fernet
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.kdf.pbkdf2 import PBKDF2HMAC
print("Insert password: ")
password_str = input()
password = password_str.encode()
salt = os.urandom(16)
kdf = PBKDF2HMAC(algorithm=hashes.SHA256(), length=32, salt=salt,
iterations=100000, backend=default_backend())
key = base64.urlsafe_b64encode(kdf.derive(password))
f = Fernet(key)
message = "A really secret message. Not for prying eyes.".encode()
token = f.encrypt(message)
file = open("text_encrypted.txt", "wb")
file.write(token)
file.close()
file = open("text_encrypted.txt", "rb")
data = file.read()
file.close()
token = f.decrypt(data)
file = open("text_decrypted.txt", "wb")
file.write(token)
file.close()
Now, that works just fine and i get the two files that contain the encrypted and decrypted message.
If i delete the:
message = "A really secret message. Not for prying eyes.".encode()
token = f.encrypt(message)
file = open("text_encrypted.txt", "wb")
file.write(token)
file.close()
part, i should be left with just a decryption code, that should work on a previously generated encrypted file, that should be decrypted by the same password.
I'm obviously missing something maybe trivial, since it raises both invalid signature and invalid token.
Thanks for your help

The encryption key you’re using is a result of a PBKDF2. In order for PBKDF2 to return the same encryption key it must get the exact same parameters. That includes salt, which in your example is generated randomly every time.
You need to store the generated salt together with the encrypted file in order to be able to decrypt it later.

How to decrypt RSA encrypted file (via PHP and OpenSSL) with pyopenssl?

Simple n00b question: I am trying to replicate the behavior of the openssl_private_decrypt function in PHP to decrypt a file that a vendor is sending me which was encrypted via the openssl_public_encrypt function. I am using python 3.4 and thus the only library I can see available is pyopenssl, but it's sufficiently low-level that I cannot easily find out how to do what I want to do. It's probably very simple, but does anyone have an exemple of what I want to do?

With the Cryptography module, which you can install with:
$ pip install cryptography
Assuming you have the private key stored in a file called "path/to/key.pem", first you load the private key:
from cryptography.hazmat.primitives import serialization
with open("path/to/key.pem", "rb") as key_file:
private_key = serialization.load_pem_private_key(
key_file.read(),
password=None,
backend=default_backend()
)
And then you decrypt with:
plaintext = private_key.decrypt(
ciphertext,
padding.OAEP(
mgf=padding.MGF1(algorithm=hashes.SHA1()),
algorithm=hashes.SHA1(),
label=None
)
)

Thanks to #mnistic it got to work, with a couple of modifications though. Here is the final working code (you have to keep in mind the defaults of openssl_private_decrypt):
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives.asymmetric import padding
# It's critical that the file be opened in mode "rb"!
with open("private.key", 'rb') as key_file:
private_key = serialization.load_pem_private_key(key_file.read(), password=None, backend=default_backend())
with open('encrypted_file', 'rb') as encrypted_file:
ciphertext = encrypted_file.read()
plaintext = private_key.decrypt(ciphertext, padding.PKCS1v15())
Please note that ciphertext needs to be shorter than the maximum chunk size of the key (which is the number of bits in the key divided by 8 for RSA). Hope that helps future Googlers!

openssl_seal() in Python

To connect a server, I've found that, using PHP, I need to use openssl_seal(). That's OK, but I want to use Python. I'm not able to convert openssl_seal() in an equivalent function.
Can you help me?
This is what openssl_seal() does:
Description
int openssl_seal ( string $data , string &$sealed_data , array &$env_keys ,
array $pub_key_ids )
openssl_seal() seals (encrypts) data by using RC4 with a randomly generated
secret key. The key is encrypted with each of the public keys associated
with the identifiers in pub_key_ids and each encrypted key is returned in
env_keys. This means that one can send sealed data to multiple recipients
(provided one has obtained their public keys). Each recipient must receive
both the sealed data and the envelope key that was encrypted with the
recipient's public key.

this blogpost has a very detailed description of what's going on inside openssl_seal(). It also has an implementation in java.
From this, I would think it should be relatively straightforward ("the proof left as an exercise to the reader" kind of straightforward) to do an equivalent implementation in python using pyopenssl, which includes RC4, or the newer, but for these purposes more focused tlslite.

What openssl_seal does is:
Extract the public_key from the certificate
Generate a 128 bits (16 bytes) long random_key (this will be used to encrypt the message using a symmetrical algorithm, since it's faster)
Encrypt the random_key using PKCS #1
Encrypt the message using ARC4 a secure cipher method and the random_key (Note that ARC4 is no longer considered secure and that PHP strongly recommends to explicitly specify a secure cipher method using the cipher_algo param)
Output the encrypted_random_key and the encrypted_message
The receiving party can then decrypt the encrypted_random_key using their private_key and then decrypt the encrypted_message using the random_key.
Since there's no way of doing this in Python via the standard library, I'm just gonna' throw out the 3 approaches that I've tried out:
# pyca/cryptography (cryptography.io) version
# pip install cryptography
import os
import cryptography
from cryptography import x509
message = 'Super secret secret message'
message = message.encode('utf-8')
certificate_data = open('/path/to/certificate.cer', 'r').read()
certificate_data = certificate_data.encode('utf-8')
certificate = cryptography.x509.load_pem_x509_certificate(data=certificate_data, backend=cryptography.hazmat.backends.default_backend())
public_key = certificate.public_key()
random_key = os.urandom(16)
encrypted_random_key = public_key.encrypt(plaintext=random_key, padding=cryptography.hazmat.primitives.asymmetric.padding.PKCS1v15())
print(encrypted_random_key)
algorithm = cryptography.hazmat.primitives.ciphers.algorithms.AES(random_key)
cipher = cryptography.hazmat.primitives.ciphers.Cipher(algorithm=algorithm, mode=None, backend=cryptography.hazmat.backends.default_backend())
encryptor = cipher.encryptor()
encrypted_message = encryptor.update(message)
print(encrypted_message)
.
# M2Crypto version
# pip install pip install git+https://gitlab.com/m2crypto/m2crypto#python3
import M2Crypto
message = 'Super secret secret message'
message = message.encode('utf-8')
certificate = M2Crypto.X509.load_cert('/path/to/certificate.cer')
public_key = certificate.get_pubkey()
rsa_pub = public_key.get_rsa()
random_key = M2Crypto.Rand.rand_bytes(16)
encrypted_random_key = rsa_pub.public_encrypt(random_key, M2Crypto.RSA.pkcs1_padding)
print(encrypted_random_key)
cipher = M2Crypto.EVP.Cipher(alg='aes_128_cbc', key=random_key, iv=b'', op=M2Crypto.encrypt)
encrypted_message = cipher.update(message)
encrypted_message += cipher.final()
print(encrypted_message)
.
# PyCrypto version
# Update: PyCrypto 2.x is unmaintained, obsolete, and contains security vulnerabilities!!!
# pip install pycrypto
# Please bear in mind that PyCrypto cannot handle x509 certificates.
# You will have to extract the public_key to a pem file:
# openssl x509 -inform pem -in certificate.cer -pubkey -noout > public_key.pem
from Crypto import Random
from Crypto.Cipher import ARC4
from Crypto.Cipher import PKCS1_OAEP
from Crypto.Cipher import PKCS1_v1_5
from Crypto.PublicKey import RSA
message = 'Super secret secret message'
message = message.encode('utf-8')
public_key_data = open('/path/to/public_key.pem', 'r').read()
public_key = RSA.importKey(public_key_data)
random_key = Random.new().read(16)
cipher = PKCS1_v1_5.new(public_key)
encrypted_random_key = cipher.encrypt(random_key)
print(encrypted_random_key)
cipher = ARC4.new(random_key)
encrypted_message = cipher.encrypt(message)
print(encrypted_message)
You can check out my post at => http://helpfulsheep.com/2017-09-01-openssl-seal-in-python/

Since I can't post comments yet, I need to add to Gabi Nagy's answer, that while their answer describes a correct algorithm, it is not the same as using openssl_seal() function.
OpenSSL doesn't let unencrypted key to get outside of OpenSSL structures. It generates key somewhere inside and keeps it there, giving you only encrypted key. The crucial difference is that when OpenSSL cleans it's structures, it should dispose of unencrypted key in a safe way.