Develop Reference

Python AES CTR Mode only encrypts first two bytes

I want to achieve the same result in Dart and in Python in encryption using AES.
My problem is that my Python version only outputs the first two bytes successfully and nothing else.
In doing so, I (must) start in Dart.
As seen in the code, I generate an IV and a key, which is passed as a parameter.
List<String> aesEncryption(List<int> key, String message) {
final iv =
enc.IV.fromLength(16); // IV generation
final aesKey = enc.Key(Uint8List.fromList(
key)); // transform the key into the right format
enc.Encrypter encrypter =
enc.Encrypter(enc.AES(aesKey, mode: enc.AESMode.ctr));
final encryptedMessage = encrypter.encrypt(message, iv: iv);
// Here I send my encrypted message and IV to my Flask app, to use the same IV and to
// compare the results with eacht other
return [encryptedMessage.base64, iv.base64];
}
Then, eventually, I just encrypt the message "test" with the following key:
The sha256 Hash of the string "low;high"
I then send this tuple to my Flask app and receive the data as follows:
#app.route("/test", methods=["POST"])
def test():
values = request.get_json()
iv = base64.b64decode(values["iv"]) # my IV which was sent of the Dart App
sig = values["signature"] # just the AES encryption value of the Dart App
key = values["key"] # the string "low;high"
aes_key = hashlib.sha256(key.encode('utf-8')).digest() # as mentioned before, I have
# to hash the string "low;high" first and I will use this as my aes key
# I tested if the computed hash values are the same in python and dart, and yes it is so
# I create a Counter with my IV, I do not completely understand this part, maybe here is my mistake somewhere.
counter = Counter.new(128, initial_value=int.from_bytes(iv, byteorder='big'))
cipher = AES.new(aes_key, AES.MODE_CTR, counter=counter)
test = "test".encode('utf-8')
# The output of this print is: b'\xd52\xf8m\x0b\xcb"\xa7\x0e\xf7\xab\x96\x0f\xfc\x9f\n'
print(base64.b64decode(sig))
encrypted = cipher.encrypt(test)
# The output of this print is: b'\xd52\xf8m'
print(encrypted)
return "", 200
As you can see only the first two bytes are output of the python AES encryption and unfortunately, I don't understand why. I hope someone can help me with this because this implementation is essential for me.
Thank you in advance for every helpful answer!
Kind regards,
Bagheera

I was able to find the error again pretty quickly.
The problem is that automatic PKCS7 padding is used in Dart and not in Python.
So the solution (just for the matter that the encryption values are the same) is the following in Dart:
enc.Encrypter encrypter =
enc.Encrypter(enc.AES(aesKey, mode: enc.AESMode.ctr, padding: null));

PHP bcrypt to PYTHON bycrypt not giving same values

Hello I am building an API on python to create a user and insert password in database. The problem is that the application is on Laravel PHP and using bcrypt. For example encrypting "test1234$%" in PYTHON gives "$2b$12$rsGZPtjctbI6bSGzS4P3mOSdrABnJuHfnKxEQwvm4KFu72BN3XNKK" and encrypting same in PHP gives "$2y$10$cO2nvRURLRdlW8j6CbWu8OeVlv7dyeozpBZcxVB2nd8hbyILyg7Xa"
and when trying to login with users created by the api on the app it does not work.
Even if i test with this it does not work the output is invalid:
$hash = '$2b$12$rsGZPtjctbI6bSGzS4P3mOSdrABnJuHfnKxEQwvm4KFu72BN3XNKK';
//validade hash in php
if(password_verify ( "test1234$%", $hash )){
echo "valid";
} else {
echo "invalid";
}
echo("\n".phpversion());
on python side used the following code:
pip install bcrypt
import bcrypt
password = item.password
bpassword = b"password"
hashed = bcrypt.hashpw(bpassword, bcrypt.gensalt())
on PHP side:
if (! function_exists('bcrypt')) {
/**
* Hash the given value against the bcrypt algorithm.
*
* #param string $value
* #param array $options
* #return string
*/
function bcrypt($value, $options = [])
{
return app('hash')->driver('bcrypt')->make($value, $options);
}
}

bcrypt use different salt each runtime that is why its perfect for storing password on database... unless you force it to use the same salt each time it will keep generating different resulting hash

I found a solution in the Python api i call bcrypt in PHP using subprocess
code = """echo password_hash("""'"'+item.password+'"'""",PASSWORD_BCRYPT);"""
hashed_password = await myClass.php(code)
async def php(self, code):
p = subprocess.Popen(["php", "-r", code],
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out = p.communicate()
if out[1] != b'': raise Exception(out[1].decode('UTF-8'))
return out[0].decode('UTF-8')

Cannot find a way to decrypt a JWE token in python (but created in ASP.Net) using jwcrypto

I am having difficulties decrypting my JWE token in python once it has been encrypted using ASP.Net.
Here is my C# code (fake passwords):
var signingKey = new SymmetricSecurityKey(Encoding.UTF8.GetBytes("ae743683f78d498a9026d0c87020b9d3"));
var secret = new SymmetricSecurityKey(Encoding.UTF8.GetBytes("MbPeShVmYq3t6w9z"));
var signingCreds = new SigningCredentials(signingKey, SecurityAlgorithms.HmacSha256);
var encryptingCreds = new EncryptingCredentials(secret, SecurityAlgorithms.Aes128KW, SecurityAlgorithms.Aes128CbcHmacSha256);
var handler = new JwtSecurityTokenHandler();
var jwtSecurityToken = handler.CreateJwtSecurityToken(
_issuer,
_audience,
new ClaimsIdentity(claimsList),
DateTime.Now,
_expires,
DateTime.Now,
signingCreds,
encryptingCreds);
var token = handler.WriteToken(jwtSecurityToken);
The token looks like this when using the encrypting credentials (should typ not be JWE?):
{
{
"alg": "A128KW",
"enc": "A128CBC-HS256",
"typ": "JWT"
}.{
"userId": "151aedd5-76c3-4eb2-8b73-a16004315731",
"prop1": "test1",
"prop2": "test2",
"nbf": 1549894420,
"exp": 1550240017,
"iat": 1549894420,
"iss": "https://localhost:56880/",
"aud": "https://localhost:56880/"
}
}
This is the token when encrypted:
eyJhbGciOiJBMTI4S1ciLCJlbmMiOiJBMTI4Q0JDLUhTMjU2IiwidHlwIjoiSldUIn0.4w4MZv5WFALGbXhmaYqtTv1VGrUpQpDJ0jN8VmLpQwDRU0j16RbPyg.hGt_z5j8THCiNEhpVvlJmw.WehLBKdyB_eYtDRvHxJHgYwa4GA7f8oKYf3GgIqAAih1eVqU084kHu1lIhC8ibxxziwmFZ4IhBFT-nWgWQrH9thhgqndF4ojtGRBgdHtW3GDAgYV6fgI11h6meyBBuLBs7mkQC5PX8EYsMTDiNE9iNTH4pWtDElc07CGGXlHsm6ntuq7G3sinagZtZMchy1shTY73NTS40FqW37C9HTIPDbrTdsm-USHcGaBMLSmjF5eOZ9Po3p4fhRT42l_gwJc9tlurttYBucvIiO1r_3NB8xGeORizYW1P_P9XGusAFy4L8h8XU9P0FctsMjUFy64LOIK8Qv8YZVq4q82vv-r9uGH6bApUdpCIcYFfGu86w63t1QLQcDT_OYMCqwo9ZmZP5Gd07lB1ypNZbP6hQTgkosp3js3i4K4bFQY7CiSXB_pSTH623TMLHNfUXWMRMIBHmXGr-zTZiKj5vkVUZLjNg.sdBUYvadnwMhkCXP8sABgA
I have tried a couple of different python packages, including 2 versions of jose (python-jose and jose), but could not get jose to work with encryption (did not seem to support the A128KW algorithm).
I am now trying jwcrypto but it seems to expect me to generate a new key rather than use my existing one (the one used to encrypt the JWT in ASP.Net):
from jwcrypto import jwk, jwe
encrypted_token = request.cookies.get(cookie_name)
private_key = "MbPeShVmYq3t6w9z"
jwk_key = jwk.JWK()
# not sure how to use my existing "private_key" value,
# and no support for "A128KW" with jwcrypto despite
# the documentation saying there is support
key = jwk_key.import_key(alg='A128KW', kty="A256CBC-HS512")
jwe_token = jwe.JWE()
jwe_token.deserialize(encrypted_token)
jwe_token.decrypt(key) # decrypt requires an instance of JWK
decrypted_payload = jwe_token.payload
How can I get this to work? Thanks for any advice you can give.

It looks like python-jose DO NOT support JWE. In their online documentation or in the source code I cannot find any line of code related to the JWE parsing or encryption/decryption.
Hopefully in the library list from jwt.io, I've found jwcrypto that should support such encrypted token (see this example dealing with A256KW) and in the srouce code we can see A128KW is listed.
You can give it a try.
from jwcrypto import jwk, jwe
encrypted_token = request.cookies.get(cookie_name)
key = jwk.JWK.from_json('{"kty":"oct","k":"TWJQZVNoVm1ZcTN0Nnc5eg"}')
jwe_token = jwe.JWE()
jwe_token.deserialize(encrypted_token)
jwe_token.decrypt(key)
decrypted_payload = jwe_token.payload

implementing USER_SRP_AUTH with python boto3 for AWS Cognito

Amazon provides iOS, Android, and Javascript Cognito SDKs that offer a high-level authenticate-user operation.
For example, see Use Case 4 here:
https://github.com/aws/amazon-cognito-identity-js
However, if you are using python/boto3, all you get are a pair of primitives: cognito.initiate_auth and cognito.respond_to_auth_challenge.
I am trying to use these primitives along with the pysrp lib authenticate with the USER_SRP_AUTH flow, but what I have is not working.
It always fails with "An error occurred (NotAuthorizedException) when calling the RespondToAuthChallenge operation: Incorrect username or password." (The username/password pair work find with the JS SDK.)
My suspicion is I'm constructing the challenge response wrong (step 3), and/or passing Congito hex strings when it wants base64 or vice versa.
Has anyone gotten this working? Anyone see what I'm doing wrong?
I am trying to copy the behavior of the authenticateUser call found in the Javascript SDK:
https://github.com/aws/amazon-cognito-identity-js/blob/master/src/CognitoUser.js#L138
but I'm doing something wrong and can't figure out what.
#!/usr/bin/env python
import base64
import binascii
import boto3
import datetime as dt
import hashlib
import hmac
# http://pythonhosted.org/srp/
# https://github.com/cocagne/pysrp
import srp
bytes_to_hex = lambda x: "".join("{:02x}".format(ord(c)) for c in x)
cognito = boto3.client('cognito-idp', region_name="us-east-1")
username = "foobar#foobar.com"
password = "123456"
user_pool_id = u"us-east-1_XXXXXXXXX"
client_id = u"XXXXXXXXXXXXXXXXXXXXXXXXXX"
# Step 1:
# Use SRP lib to construct a SRP_A value.
srp_user = srp.User(username, password)
_, srp_a_bytes = srp_user.start_authentication()
srp_a_hex = bytes_to_hex(srp_a_bytes)
# Step 2:
# Submit USERNAME & SRP_A to Cognito, get challenge.
response = cognito.initiate_auth(
AuthFlow='USER_SRP_AUTH',
AuthParameters={ 'USERNAME': username, 'SRP_A': srp_a_hex },
ClientId=client_id,
ClientMetadata={ 'UserPoolId': user_pool_id })
# Step 3:
# Use challenge parameters from Cognito to construct
# challenge response.
salt_hex = response['ChallengeParameters']['SALT']
srp_b_hex = response['ChallengeParameters']['SRP_B']
secret_block_b64 = response['ChallengeParameters']['SECRET_BLOCK']
secret_block_bytes = base64.standard_b64decode(secret_block_b64)
secret_block_hex = bytes_to_hex(secret_block_bytes)
salt_bytes = binascii.unhexlify(salt_hex)
srp_b_bytes = binascii.unhexlify(srp_b_hex)
process_challenge_bytes = srp_user.process_challenge(salt_bytes,
srp_b_bytes)
timestamp = unicode(dt.datetime.utcnow().strftime("%a %b %d %H:%m:%S +0000 %Y"))
hmac_obj = hmac.new(process_challenge_bytes, digestmod=hashlib.sha256)
hmac_obj.update(user_pool_id.split('_')[1].encode('utf-8'))
hmac_obj.update(username.encode('utf-8'))
hmac_obj.update(secret_block_bytes)
hmac_obj.update(timestamp.encode('utf-8'))
challenge_responses = {
"TIMESTAMP": timestamp.encode('utf-8'),
"USERNAME": username.encode('utf-8'),
"PASSWORD_CLAIM_SECRET_BLOCK": secret_block_hex,
"PASSWORD_CLAIM_SIGNATURE": hmac_obj.hexdigest()
}
# Step 4:
# Submit challenge response to Cognito.
response = cognito.respond_to_auth_challenge(
ClientId=client_id,
ChallengeName='PASSWORD_VERIFIER',
ChallengeResponses=challenge_responses)

There are many errors in your implementation. For example:
pysrp uses SHA1 algorithm by default. It should be set to SHA256.
_ng_const length should be 3072 bits and it should be copied from amazon-cognito-identity-js
There is no hkdf function in pysrp.
The response should contain secret_block_b64, not secret_block_hex.
Wrong timestamp format. %H:%m:%S means "hour:month:second" and +0000 should be replaced by UTC.
Has anyone gotten this working?
Yes. It's implemented in the warrant.aws_srp module.
https://github.com/capless/warrant/blob/master/warrant/aws_srp.py
from warrant.aws_srp import AWSSRP
USERNAME='xxx'
PASSWORD='yyy'
POOL_ID='us-east-1_zzzzz'
CLIENT_ID = '12xxxxxxxxxxxxxxxxxxxxxxx'
aws = AWSSRP(username=USERNAME, password=PASSWORD, pool_id=POOL_ID,
client_id=CLIENT_ID)
tokens = aws.authenticate_user()
id_token = tokens['AuthenticationResult']['IdToken']
refresh_token = tokens['AuthenticationResult']['RefreshToken']
access_token = tokens['AuthenticationResult']['AccessToken']
token_type = tokens['AuthenticationResult']['TokenType']
authenticate_user method supports only PASSWORD_VERIFIER challenge. If you want to respond to other challenges, just look into the authenticate_user and boto3 documentation.

Unfortunately it's a hard problem since you don't get any hints from the service with regards to the computations (it mainly says not authorized as you mentioned).
We are working on improving the developer experience when users are trying to implement SRP on their own in languages where we don't have an SDK. Also, we are trying to add more SDKs.
As daunting as it sounds, what I would suggest is to take the Javascript or the Android SDK, fix the inputs (SRP_A, SRP_B, TIMESTAMP) and add console.log statements at various points in the implementation to make sure your computations are similar. Then you would run these computations in your implementation and make sure you are getting the same. As you have suggested, the password claim signature needs to be passed as a base64 encoded string to the service so that might be one of the issues.
Some of the issues I encountered while implementing this was related to BigInteger library differences (the way they do byte padding and transform negative numbers to byte arrays and inversely).

AES: how to detect that a bad password has been entered?

A text s has been encrypted with:
s2 = iv + Crypto.Cipher.AES.new(Crypto.Hash.SHA256.new(pwd).digest(),
Crypto.Cipher.AES.MODE_CFB,
iv).encrypt(s.encode())
Then, later, a user inputs the password pwd2 and we decrypt it with:
iv, cipher = s2[:Crypto.Cipher.AES.block_size], s2[Crypto.Cipher.AES.block_size:]
s3 = Crypto.Cipher.AES.new(Crypto.Hash.SHA256.new(pwd2).digest(),
Crypto.Cipher.AES.MODE_CFB,
iv).decrypt(cipher)
Problem: the last line works even if the entered password pw2 is wrong. Of course the decrypted text will be random chars, but no error is triggered.
Question: how to make Crypto.Cipher.AES.new(...).decrypt(cipher) fail if the password pw2 is incorrect? Or at least how to detect a wrong password?
Here is a linked question: Making AES decryption fail if invalid password
and here a discussion about the cryptographic part (less programming) of the question: AES, is this method to say “The password you entered is wrong” secure?
.

AES provides confidentiality but not integrity out of the box - to get integrity too, you have a few options. The easiest and arguably least prone to "shooting yourself in the foot" is to just use AES-GCM - see this Python example or this one.
You could also use an HMAC, but this generally requires managing two distinct keys and has a few more moving parts. I would recommend the first option if it is available to you.
A side note, SHA-256 isn't a very good KDF to use when converting a user created password to an encryption key. Popular password hashing algorithms are better at this - have a look at Argon2, bcrypt or PBKDF2.
Edit: The reason SHA-256 is a bad KDF is the same reason it makes a bad password hash function - it's just too fast. A user created password of, say, 128 bits will usually contain far less entropy than a random sequence of 128 bits - people like to pick words, meaningful sequences etc. Hashing this once with SHA-256 doesn't really alleviate this issue. But hashing it with a construct like Argon2 that is designed to be slow makes a brute-force attack far less viable.

The best way is to use authenticated encryption, and a modern memory-hard entropy-stretching key derivation function such a scrypt to turn the password into a key. The cipher's nounce can be used as salt for the key derivation. With PyCryptodome that could be:
from Crypto.Random import get_random_bytes
from Crypto.Cipher import AES
from Crypto.Protocol.KDF import scrypt
# initialize an AES-128-GCM cipher from password (derived using scrypt) and nonce
def cipherAES(pwd, nonce):
# note: the p parameter should allow use of several processors, but did not for me
# note: changing 16 to 24 or 32 should select AES-192 or AES-256 (not tested)
return AES.new(scrypt(pwd, nonce, 16, N=2**21, r=8, p=1), AES.MODE_GCM, nonce=nonce)
# encryption
nonce = get_random_bytes(16)
print("deriving key from password and nonce, then encrypting..")
ciphertext, tag = cipherAES(b'pwdHklot2',nonce).encrypt_and_digest(b'bonjour')
print("done")
# decryption of nonce, ciphertext, tag
print("deriving key from password and nonce, then decrypting..")
try:
plaintext = cipherAES(b'pwdHklot2', nonce).decrypt_and_verify(ciphertext, tag)
print("The message was: " + plaintext.decode())
except ValueError:
print("Wrong password or altered nonce, ciphertext, tag")
print("done")
Note: Code is here to illustrate the principle. In particular, the scrypt parameters should not be fixed, but rather be included in a header before nonce, ciphertext, and tag; and that must be somewhat grouped for sending, and parsed for decryption.
Caveat: nothing in this post should be construed as an endorsement of PyCryptodome's security.
Addition (per request):
We need scrypt or some other form of entropy stretching only because we use a password. We could use a random 128-bit key directly.
PBKDF2-HMAC-SHAn with 100000 iterations (as in the OP's second code fragment there) is only barely passable to resist Hashcat with a few GPUs. It would would be almost negligible compared to other hurdles for an ASIC-assisted attack: a state of the art Bitcoin mining ASIC does more than 2*1010 SHA-256 per Joule, 1 kWh of electricity costing less than $0.15 is 36*105 J. Crunching these numbers, testing the (62(8+1)-1)/(62-1) = 221919451578091 passwords of up to 8 characters restricted to letters and digits cost less than $47 for energy dedicated to the hashing part.
scrypt is much more secure for equal time spent by legitimate users because it requires a lot of memory and accesses thereof, slowing down the attacker, and most importantly making the investment cost for massively parallel attack skyrocket.

Doesn't use the Crypto package, but this should suit your needs:
import base64
import os
from cryptography.fernet import Fernet
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives.kdf.scrypt import Scrypt
def derive_password(password: bytes, salt: bytes):
"""
Adjust the N parameter depending on how long you want the derivation to take.
The scrypt paper suggests a minimum value of n=2**14 for interactive logins (t < 100ms),
or n=2**20 for more sensitive files (t < 5s).
"""
kdf = Scrypt(salt=salt, length=32, n=2**16, r=8, p=1, backend=default_backend())
key = kdf.derive(password)
return base64.urlsafe_b64encode(key)
salt = os.urandom(16)
password = b'legorooj'
bad_password = b'legorooj2'
# Derive the password
key = derive_password(password, salt)
key2 = derive_password(bad_password, salt) # Shouldn't re-use salt but this is only for example purposes
# Create the Fernet Object
f = Fernet(key)
msg = b'This is a test message'
ciphertext = f.encrypt(msg)
print(msg, flush=True) # Flushing pushes it strait to stdout, so the error that will come
print(ciphertext, flush=True)
# Fernet can only be used once, so we need to reinitialize
f = Fernet(key)
plaintext = f.decrypt(ciphertext)
print(plaintext, flush=True)
# Bad Key
f = Fernet(key2)
f.decrypt(ciphertext)
"""
This will raise InvalidToken and InvalidSignature, which means it wasn't decrypted properly.
"""
See my comment for links to the documentation.

For future reference, here is a working solution following the AES GCM mode (recommended by #LukeJoshuaPark in his answer):
from Crypto.Cipher import AES
from Crypto.Random import get_random_bytes
# Encryption
data = b"secret"
key = get_random_bytes(16)
cipher = AES.new(key, AES.MODE_GCM)
ciphertext, tag = cipher.encrypt_and_digest(data)
nonce = cipher.nonce
# Decryption
key2 = get_random_bytes(16) # wrong key
#key2 = key # correct key
try:
cipher = AES.new(key2, AES.MODE_GCM, nonce=nonce)
plaintext = cipher.decrypt_and_verify(ciphertext, tag)
print("The message was: " + plaintext.decode())
except ValueError:
print("Wrong key")
It does fail with an exception when the password is wrong indeed, as desired.
The following code uses a real password derivation function:
import Crypto.Random, Crypto.Protocol.KDF, Crypto.Cipher.AES
def cipherAES(pwd, nonce):
return Crypto.Cipher.AES.new(Crypto.Protocol.KDF.PBKDF2(pwd, nonce, count=100000), Crypto.Cipher.AES.MODE_GCM, nonce=nonce)
# encryption
nonce = Crypto.Random.new().read(16)
cipher = cipherAES(b'pwd1', nonce)
ciphertext, tag = cipher.encrypt_and_digest(b'bonjour')
# decryption
try:
cipher = cipherAES(b'pwd1', nonce=nonce)
plaintext = cipher.decrypt_and_verify(ciphertext, tag)
print("The message was: " + plaintext.decode())
except ValueError:
print("Wrong password")
#fgrieu's answer is probably better because it uses scrypt as KDF.