Develop Reference

ZIP's CRC-32 for encryption isn't quite zlib's crc32... why?

I'm writing my own unzip code, and (from trial and error, no understanding) it looks like the CRC-32 algorithm on the one byte that decryption requires doesn't quite match up with zlib's. To convert from one to the other:
def crc32(ch, crc):
crc = zlib.crc32(bytes([~ch & 0xFF]), crc)
return (~crc & 0xFF000000) + (crc & 0x00FFFFFF)
Why is this? (/ Am I wrong?)
Edit: the reason why I think there is at least the possibility of me being right, at https://github.com/uktrade/stream-unzip/blob/d23400028abbe3b0d7e1951cb562cd0541bfc960/stream_unzip.py#L89 I use the above successfully to decrypt encrypted ZIP files
def decrypt(chunks):
key_0 = 305419896
key_1 = 591751049
key_2 = 878082192
def crc32(ch, crc):
crc = zlib.crc32(bytes([~ch & 0xFF]), crc)
return (~crc & 0xFF000000) + (crc & 0x00FFFFFF)
def update_keys(byte):
nonlocal key_0, key_1, key_2
key_0 = crc32(byte, key_0)
key_1 = (key_1 + (key_0 & 0xFF)) & 0xFFFFFFFF
key_1 = ((key_1 * 134775813) + 1) & 0xFFFFFFFF
key_2 = crc32(key_1 >> 24, key_2)
def decrypt(chunk):
chunk = bytearray(chunk)
for i, byte in enumerate(chunk):
temp = key_2 | 2
byte ^= ((temp * (temp ^ 1)) >> 8) & 0xFF
update_keys(byte)
chunk[i] = byte
return chunk
yield_all, _, get_num, _ = get_byte_readers(chunks)
for byte in password:
update_keys(byte)
if decrypt(get_num(12))[11] != mod_time >> 8:
raise ValueError('Incorrect password')
for chunk in yield_all():
yield decrypt(chunk)
However, if I replace the crc32 function above with just calling zlib's, it doesn't (e.g. it will complain about an incorrect password)

Ok, you're not completely wrong. It is indeed the same CRC-32 algorithm, but without the pre and post-processing (inverting the CRC coming in and going out). It is truly odd code that is trying to replicate that with the zlib.crc32 function. All you need is this:
def crc32(ch, crc):
return ~zlib.crc32(bytes([ch]), ~crc) & 0xffffffff

getting the error "Invalid DES key size. Key must be exactly 8 bytes long" while encrypting message in python

I have the below java code to encode a string with a pass key
public static String encrypt(String message, String passkey) throws Exception {
final MessageDigest md = MessageDigest.getInstance("SHA-1");
final byte[] digestOfPassword = md.digest(passkey.getBytes("utf-8"));
final byte[] keyBytes = ( byte[])resizeArray(digestOfPassword, 24);
for (int j = 0, k = 16; j < 8;) {
keyBytes[k++] = keyBytes[j++];
}
final SecretKey key = new SecretKeySpec(keyBytes, "DESede");
final IvParameterSpec iv = new IvParameterSpec(new byte[8]);
final Cipher cipher = Cipher.getInstance("DESede/CBC/PKCS5Padding");
cipher.init(Cipher.ENCRYPT_MODE, key, iv);
final byte[] plainTextBytes = message.getBytes("utf-8");
final byte[] cipherText = cipher.doFinal(plainTextBytes);
String encryptedString = Base64.encodeBase64String(cipherText);
return encryptedString;
}
Now I converted the same code into python(Python 2.7), and I tried as below.
def encrypt(message, passkey):
hash_object = hashlib.sha1(passkey.encode("utf-8"))
digested_passkey = hash_object.digest() //hashing
key24 = "{: <24}".format(digested_passkey) // for resizing the byte array to size 24
import pyDes
des = pyDes.des(key24);(at this line I m getting the error "Invalid DES key size. Key must be exactly 8 bytes long".
message = message.encode('utf-8')
message = message + (16 - len(message) % 16) * chr(16 - len(message) % 16) // this is for padding
iv = Random.new().read(AES.block_size)
cipher = AES.new(des, AES.MODE_CBC, iv)
return base64.b64encode(iv + cipher.encrypt(message))
At the line des = pyDes.des(key24), I am getting the error "Invalid DES key size. Key must be exactly 8 bytes long."
The passkey that I sent as parameter is "f!16*hw$sda66"
Can anyone please let me know if there is anything wrong with the line
des = pyDes.des(key24)

I think the reason you are getting this error is because the Class initialisation method is expecting the key to be exactly 8, if it's anything else it raises the error you are seeing, this is the init of the class you are calling from pyDes:
# Initialisation
def __init__(self, key, mode=ECB, IV=None, pad=None, padmode=PAD_NORMAL):
# Sanity checking of arguments.
if len(key) != 8:
raise ValueError("Invalid DES key size. Key must be exactly 8 bytes long.")
If you do this for debugging:
def encrypt(message, passkey):
hash_object = hashlib.sha1(passkey.encode("utf-8"))
digested_passkey = hash_object.digest() //hashing
key24 = "{: <24}".format(digested_passkey)
print len(key24)
You will see the length of the key is 24, which is why I think it is not being accepted.
I might be wrong but at a quick glance that looks like the issue.

python mmap skipping every second byte when writing

I have a strange problem with using mmap in python when writing to memory (/dev/mem).
To be clear, reading is done in the same manner and it works OK.
But when it comes to writing, it seems that every second byte is unwritable.
ex.
when I read i get
addr 0x200 val 0x1234
but when I try to write
addr 0x200 val 0x4321
what really is written is
addr 0x200 val 0x0021
When I try to write byte by byte, the same happens.
ex.
write:
addr 0x200 0x43
addr 0x201 0x21
I get
addr 0x200 0x00
addr 0x201 0x21
Code:
class Pydump:
def __init__(self, addr, length = 1, word_size = 4, filename = '/dev/mem'):
if addr < 0 or length < 0: raise ValueError('Address or length invalid')
self._verbose = verbose
self.word_size = word_size
self.mask = ~(self.word_size - 1)
self.base_addr = addr & ~(mmap.PAGESIZE - 1)
self.addr_offset = addr - self.base_addr
self.word_length = length
self.no_of_bytes = self.word_length * self.word_size
# align length to full words
end_addr = addr + self.no_of_bytes
if end_addr % self.mask:
end_addr = (end_addr + self.word_size) & self.mask
self.map_length = end_addr - self.base_addr
self.fname = filename
self.fd = os.open(self.fname, os.O_RDWR | os.O_SYNC)
self.mem = mmap.mmap(self.fd, self.map_length, mmap.MAP_SHARED,
mmap.PROT_READ | mmap.PROT_WRITE,
offset=self.base_addr)
def read(self):
mem = self.mem
virt_base_addr = self.addr_offset & self.mask
mem.seek(virt_base_addr)
data = []
for i in range(self.no_of_bytes):
data.append(struct.unpack('B', mem.read_byte())[0])
abs_addr = self.base_addr + virt_base_addr
return PydumpBuffer(abs_addr, data, self.word_size)
def write(self, data):
mem = self.mem
virt_base_addr = self.addr_offset & self.mask
mem.seek(virt_base_addr)
if self.word_size == 1:
mem.write(struct.pack('B', data))
elif self.word_size == 2:
mem.write(struct.pack('H', data))
elif self.word_size == 4:
mem.write(struct.pack('I', data))
else:
raise ValueError('Invalid word size')
def write_bytes(self, bytes):
if len(bytes) != self.no_of_bytes: raise ValueError('Wrong number of bytes given')
mem = self.mem
virt_base_addr = self.addr_offset & self.mask
mem.seek(virt_base_addr)
for byte in bytes:
mem.write_byte(byte)
Example run (I prepared the memory with other memdump tool [bin] to be 0xEEEEEEEE):
>>> from pydump import Pydump as memdump
>>> memdump(0x18007C20, 1, 4).read()
0xEEEEEEEE
>>> memdump(0x18007C20, 1, 4).write(0x12345678)
>>> memdump(0x18007C20, 1, 4).read()
0x00340078
>>> memdump(0x18007C20, 1, 4).write(0x87654321)
>>> memdump(0x18007C20, 1, 4).read()
0x00650021
Example no 2 (I could not write even 2 bytes at 'first' byte place):
>>> memdump(0x18007C20, 1, 2).write(0xABCD)
>>> memdump(0x18007C20, 1, 4).read()
0x00650021
>>> memdump(0x18007C21, 1, 1).write(0xCD)
>>> memdump(0x18007C20, 1, 4).read()
0x00650021
>>> memdump(0x18007C22, 1, 1).write(0xCD)
>>> memdump(0x18007C20, 1, 4).read()
0x00CDCD00
Any thoughts on what could be the problem ?

I know what is wrong now.
The answer is that it's fault as well of python as my specific hardware/driver.
I looked through the implementation of mmap and it uses memcpy, which as we know copies byte by byte. In C implemention this did not happen, when we needed to write 4 bytes, 4 bytes were written.
So here comes the limitation of my hardware/driver which is that every register must be written in full (I did not know this at the time of stating the question), thus when writing byte by byte I will get weird behaviour.
Some registers are 16bit and some are 32bit.
When I was writing 0xFFFF to 16bit register I ended up having 0x00FF. So memcpy was copying 0xFF twice. My driver got write(0xFF) then write(0xFF), so what I was actually doing (because of the python) was two writes to register of byte 0xFF :)
Probably the same happened with 32bit registers, although it looked different (but also the driver may behave different).
For it to work I would have to change the implementation of python mmap :(

Encrypt data using Objective-C and Decrypt in Python

I have the same issue as this question but unfortunately there was no answer on it.
I have the following objective-c code to encrypt using CCCrypt:
(NSData *)doCrypt:(NSData *)data usingKey:(NSData *)key withInitialVector:(NSData *)iv mode:(int)mode error: (NSError *)error
{
int buffersize = 0;
if(data.length % 16 == 0) { buffersize = data.length + 16; }
else { buffersize = (data.length / 16 + 1) * 16 + 16; }
// int buffersize = (data.length <= 16) ? 16 : data.length;
size_t numBytesEncrypted = 0;
void *buffer = malloc(buffersize * sizeof(uint8_t));
CCCryptorStatus result = CCCrypt(mode, 0x0, 0x1, [key bytes], [key length], [iv bytes], [data bytes], [data length], buffer, buffersize, &numBytesEncrypted);
return [NSData dataWithBytesNoCopy:buffer length:numBytesEncrypted freeWhenDone:YES];
}
I use kCCAlgorithmAES128 with kCCOptionPKCS7Padding as options and call the function with [Cryptor doCrypt:data usingKey:key withInitialVector:nil mode:0x0 error:nil];
Now I would like to decrypt it using python and to do so I have the following code:
def decrypt(self, data, key):
iv = '\x00' * 16
encoder = PKCS7Encoder()
padded_text = encoder.encode(data)
mode = AES.MODE_CBC
cipher = AES.new(key, mode, iv)
decoded = cipher.decrypt(padded_text)
return decoded
The PKCS7Encoder looks like this:
class PKCS7Encoder():
"""
Technique for padding a string as defined in RFC 2315, section 10.3,
note #2
"""
class InvalidBlockSizeError(Exception):
"""Raised for invalid block sizes"""
pass
def __init__(self, block_size=16):
if block_size < 2 or block_size > 255:
raise PKCS7Encoder.InvalidBlockSizeError('The block size must be ' \
'between 2 and 255, inclusive')
self.block_size = block_size
def encode(self, text):
text_length = len(text)
amount_to_pad = self.block_size - (text_length % self.block_size)
if amount_to_pad == 0:
amount_to_pad = self.block_size
pad = chr(amount_to_pad)
return text + pad * amount_to_pad
def decode(self, text):
pad = ord(text[-1])
return text[:-pad]
Yet whenever I call the decrypt() function, it returns garbage. Am I missing something or having a wrong option enabled somewhere?
Example in and output:
NSData *keyData = [[NSData alloc] initWithRandomData:16];
NSLog(#"key: %#", [keyData hex]);
NSString *str = #"abcdefghijklmno";
NSLog(#"str: %#", str);
NSData *encrypted = [Cryptor encrypt:[str dataUsingEncoding:NSUTF8StringEncoding] usingKey:keyData];
NSLog(#"encrypted str: %#", [encrypted hex]);
Gives:
key: 08b6cb24aaec7d0229312195e43ed829
str: a
encrypted str: 52d61265d22a05efee2c8c0c6cd49e9a
And python:
cryptor = Cryptor()
encrypted_hex_string = "52d61265d22a05efee2c8c0c6cd49e9a"
hex_key = "08b6cb24aaec7d0229312195e43ed829"
print cryptor.decrypt(encrypted_hex_string.decode("hex"), hex_key.decode("hex"))
Result:
láz
Which is weird, but if dump the hex I get 610f0f0f0f0f0f0f0f0f0f0f0f0f0f0fb02b09fd58cccf04f042e2c90d6ce17a and 61 = a so I think it just shows wrong.
A bigger input:
key: 08b6cb24aaec7d0229312195e43ed829
str: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
encrypted str: 783fce3eca7ebe60d58b01da3d90105a93bf2d659cfcffc1c2b7f7be7cc0af4016b310551965526ac211f4d6168e3cc5
Result:
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaôNÍ“ƒ˜�Üšw6C%
Here you see that the a's are printed with garbage... so I assume this is a padding error or something like that
The IV is nill at the iOs side and 16x 0's at the Python side (see the code)

Your decryption: aes_decrypt(pkcs7_pad(ciphertext))
Correct decryption: pkcs7_unpad(aes_decrypt(ciphertext))
It has to be done this way, because AES in CBC mode expects plaintexts of a multiple of the block size, but you generally want to encrypt arbitrary plaintexts. Therefore, you need to apply the padding before encryption and remove the padding after decryption.
Keep in mind that a - (b % a) cannot be 0 for any (positive) value of a or b. This means that
if amount_to_pad == 0:
amount_to_pad = self.block_size
is unreachable code and can be removed. Good thing is that a - (b % a) already does what you wanted to do with the if block.
You also should extend the unpad (decode) function to actually check whether every padding byte is the same byte. You should also check that the every padding byte is not zero or larger than the block size.

Python HyBi10 websocket server

I've struggled the past 2 hours with the new Websocket version. I've managed to get the handshake and receiving these new frames, but I'm having problems sending them now.
I'm encoding my text like this:
def encode_hybi(buf, opcode, base64=False):
""" Encode a HyBi style WebSocket frame.
Optional opcode:
0x0 - continuation
0x1 - text frame (base64 encode buf)
0x2 - binary frame (use raw buf)
0x8 - connection close
0x9 - ping
0xA - pong
"""
if base64:
buf = b64encode(buf)
b1 = 0x80 | (opcode & 0x0f) # FIN + opcode
payload_len = len(buf)
if payload_len <= 125:
header = struct.pack('>BB', b1, payload_len)
elif payload_len > 125 and payload_len < 65536:
header = struct.pack('>BBH', b1, 126, payload_len)
elif payload_len >= 65536:
header = struct.pack('>BBQ', b1, 127, payload_len)
#print("Encoded: %s" % repr(header + buf))
#return header + buf, len(header), 0
return header+buf
But I don't know in what form I have to pour it to send it over the socket.
By the way: isn't there some easy python websocket module somewhere? My code has now seen 3 websocket versions and it's an utter mess.