I want to create a .tsq file like openssl with command:
openssl ts -query -data <file> -no_nonce -sha512 -out <out.tsq>
I want to implement this with python, Anyone know how to do this, any module or something like that?
Right now I can think of three different approaches:
Use some premade python module of unknown quality like python-rfc3161 mentioned by #J.F.Sebastian in his comment.
Use hashlib module to compute SHA512 hash of the data you want to timestamp and then use pyasn1 module to construct and encode TimeStampReq request structure defined in RFC3161.
Use hashlib module to compute SHA512 hash of the data you want to timestamp and pre-pend these bytes 0x30 0x56 0x02 0x01 0x01 0x30 0x51 0x30 0x0D 0x06 0x09 0x60 0x86 0x48 0x01 0x65 0x03 0x04 0x02 0x03 0x05 0x00 0x04 0x40 to the hash value. This should work for you because OpenSSL command you have provided is creating TS request which does not contain any variable part (such as nonce or policy OID) so the first part of the request structure will not change no matter what input data you will use.
Here's a Python 3 implementation of the 3rd idea from #jariq's answer:
#!/usr/bin/env python3
"""Emulate `openssl ts -query -data <file> -no_nonce -sha512 -out <out.tsq>`
Usage: %(prog)s <file> [<out.tsq>]
If <out.tsq> is not given; use <file> name and append '.tsq' suffix
"""
import hashlib
import sys
from functools import partial
def hash_file(filename, hashtype, chunksize=2**15, bufsize=-1):
h = hashtype()
with open(filename, 'rb', bufsize) as file:
for chunk in iter(partial(file.read, chunksize), b''):
h.update(chunk)
return h
try: # parse command-line arguments
filename, *out_filename = sys.argv[1:]
out_filename.append(filename + '.tsq')
except ValueError:
sys.exit(__doc__ % dict(prog=sys.argv[0]))
h = hash_file(filename, hashlib.sha512) # find hash of the input file
with open(out_filename[0], 'wb') as file: # write timestamp query
file.write(b'0V\x02\x01\x010Q0\r\x06\t`\x86H\x01'
b'e\x03\x04\x02\x03\x05\x00\x04#')
file.write(h.digest())
To expand upon #j-f-sebastian's answer, if you want to hash using sha-256 (or any 256 bit hash function), use the following constant:
b'06\x02\x01\x01010\r\x06\t`\x86H\x01e\x03\x04\x02\x01\x05\x00\x04 '
(and yes, the last character is an empty space)
Related
I am trying to run the following openssl command in python:
cmd = "openssl x509 -sha1 -in esx.crt -noout -fingerprint"
tmp = os.popen(cmd)
tmp_sha1 = tmp.readline()
This command is supposed to generate a fingerprint of the certificate. I am trying to capture the output through the file object. But when I read this file object, there's nothing in it. I have executed this command on the command line and it runs fine, generates the fingerprint. Could you tell me how can I get the fingerprint?
You achieve this natively within Python using the OpenSSL module.
from OpenSSL.crypto import load_certificate, FILETYPE_PEM
cert_file_string = open("esx.crt", "rb").read()
cert = load_certificate(FILETYPE_PEM, cert_file_string)
sha1_fingerprint = cert.digest("sha1")
print sha1_fingerprint
You can use two modules to establish what you want: subprocess and os.
Using subprocess you can check for the output for the process using communicate(), which reads data from stdout and stderr until EOF.
>>> import subprocess
>>> p = subprocess.Popen("openssl x509 -sha1 -in 17.cert -noout -fingerprint", stdout=subprocess.PIPE)
>>> out, _ = p.communicate() #return a tuple (stdout, stderr)
>>> out
b'SHA1 Fingerprint=87:68:8B:B0:6A:E2:DF:A3:E2:63:76:97:A9:2B:B4:F4:82:4E:0B:D1\n'
Using os module works fine as well, using both read() and readline() methods: (please note that os.popen() is deprecated)
>>> import os
>>> p = os.popen("openssl x509 -sha1 -in 17.cert -noout -fingerprint")
>>> p.read()
'SHA1 Fingerprint=87:68:8B:B0:6A:E2:DF:A3:E2:63:76:97:A9:2B:B4:F4:82:4E:0B:D1\n'
>>> p = os.popen("openssl x509 -sha1 -in 17.cert -noout -fingerprint")
>>> out = p.readline()
'SHA1 Fingerprint=87:68:8B:B0:6A:E2:DF:A3:E2:63:76:97:A9:2B:B4:F4:82:4E:0B:D1\n'
If you want to write the value to a file, that works as well, which you can verify by opening a file output.txt in your current working directory:
>>> with open('./output.txt', 'w') as f:
... f.write(out)
...
77
The 77 notifies us that 77 bytes were written to the file, which you can verify by opening it in your favourite text editor.
I've got problem trying open .pcap file. In scapy.utils there is RawPcapReader
try:
self.f = gzip.open(filename,"rb")
magic = self.f.read(4)
except IOError:
self.f = open(filename,"rb")
magic = self.f.read(4)
if magic == "\xa1\xb2\xc3\xd4": #big endian
self.endian = ">"
elif magic == "\xd4\xc3\xb2\xa1": #little endian
self.endian = "<"
else:
raise Scapy_Exception("Not a pcap capture file (bad magic)")
hdr = self.f.read(20)
if len(hdr)<20:
raise Scapy_Exception("Invalid pcap file (too short)")
My magic has value "\n\r\r\n" but RawPcapReader is expecting magic == "\xa1\xb2\xc3\xd4" or magic == "\xd4\xc3\xb2\xa1".
Could you tell me what can be the problem? With .pcap file? I'm using python version 2.7
The magic value of "\n\r\r\n" (\x0A\x0D\x0D\x0A) indicates that your file is actually in .pcapng format, rather than libpcap
The solution is simple
In Wireshark 'Save As': Wireshark/tcpdump - pcap
Or use tshark:
$tshark -r old.pcapng -w new.pcap -F libpcap
As an alternative to saving the file in pcap format, scapy now has PcapNgReader so you could do:
mypcap = PcapNgReader(filename)
I have been pulling my hair out trying to get a proxy working. I need to decrypt the packets from a server and client ((this may be out of order..)), then decompress everything but the packet header.
The first 2 packets ((10101 and 20104)) are not compressed, and decrypt, destruct, and decompile properly.
Alas, but to no avail; FAIL!; zlib.error: Error -5 while decompressing data: incomplete or truncated stream
Same error while I am attempting to decompress the encrypted version of the packet.
When I include the packet header, I get a randomly chosen -3 error.
I have also tried changing -zlib.MAX_WBITS to zlib.MAX_WBITS, as well as a few others, but still get the same error.
Here's the code;
import socket, sys, os, struct, zlib
from Crypto.Cipher import ARC4 as rc4
cwd = os.getcwd()
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
ss = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('192.168.2.12',9339))
s.listen(1)
client, addr = s.accept()
key = "fhsd6f86f67rt8fw78fw789we78r9789wer6renonce"
cts = rc4.new(key)
stc = rc4.new(key)
skip = 'a'*len(key)
cts.encrypt(skip)
stc.encrypt(skip)
ss.connect(('game.boombeachgame.com',9339))
ss.settimeout(0.25)
s.settimeout(0.25)
def io():
while True:
try:
pack = client.recv(65536)
decpack = cts.decrypt(pack[7:])
msgid, paylen = dechead(pack)
if msgid != 10101:
decopack = zlib.decompress(decpack, -zlib.MAX_WBITS)
print "ID:",msgid
print "Payload Length",paylen
print "Payload:\n",decpack
ss.send(pack)
dump(msgid, decpack)
except socket.timeout:
pass
try:
pack = ss.recv(65536)
msgid, paylen = dechead(pack)
decpack = stc.decrypt(pack[7:])
if msgid != 20104:
decopack = zlib.decompress(decpack, -zlib.MAX_WBITS)
print "ID:",msgid
print "Payload Length",paylen
print "Payload:\n",decpack
client.send(pack)
dump(msgid, decpack)
except socket.timeout:
pass
def dump(msgid, decpack):
global cwd
pdf = open(cwd+"/"+str(msgid)+".bin",'wb')
pdf.write(decpack)
pdf.close()
def dechead(pack):
msgid = struct.unpack('>H', pack[0:2])[0]
print int(struct.unpack('>H', pack[5:7])[0])
payload_bytes = struct.unpack('BBB', pack[2:5])
payload_len = ((payload_bytes[0] & 255) << 16) | ((payload_bytes[1] & 255) << 8) | (payload_bytes[2] & 255)
return msgid, payload_len
io()
I realize it's messy, disorganized and very bad, but it all works as intended minus the decompression.
Yes, I am sure the packets are zlib compressed.
What is going wrong here and why?
Full Traceback:
Traceback (most recent call last):
File "bbproxy.py", line 68, in <module>
io()
File "bbproxy.py", line 33, in io
decopack = zlib.decompress(decpack, zlib.MAX_WBITS)
zlib.error: Error -5 while decompressing data: incomplete or truncated stream
I ran into the same problem while trying to decompress a file using zlib with Python 2.7. The issue had to do with the size of the stream (or file input) exceeding the size that could be stored in memory. (My PC has 16 GB of memory, so it was not exceeding the physical memory size, but the buffer default size is 16384.)
The easiest fix was to change the code from:
import zlib
f_in = open('my_data.zz', 'rb')
comp_data = f_in.read()
data = zlib.decompress(comp_data)
To:
import zlib
f_in = open('my_data.zz', 'rb')
comp_data = f_in.read()
zobj = zlib.decompressobj() # obj for decompressing data streams that won’t fit into memory at once.
data = zobj.decompress(comp_data)
It handles the stream by buffering it and feeding in into the decompressor in manageable chunks.
I hope this helps to save you time trying to figure out the problem. I had help from my friend Jordan! I was trying all kinds of different window sizes (wbits).
Edit: Even with the below working on partial gz files for some files when I decompressed I got empty byte array and everything I tried would always return empty though the function was successful. Eventually I resorted to running gunzip process which always works:
def gunzip_string(the_string):
proc = subprocess.Popen('gunzip',stdout=subprocess.PIPE,
stdin=subprocess.PIPE, stderr=subprocess.DEVNULL)
proc.stdin.write(the_body)
proc.stdin.close()
body = proc.stdout.read()
proc.wait()
return body
Note that the above can return a non-zero error code indicating that the input string is incomplete but it still performs the decompression and hence the stderr being swallowed. You may wish to check errors to allow for this case.
/edit
I think the zlib decompression library is throwing an exception because you are not passing in a complete file just a 65536 chunk ss.recv(65536). If you change from this:
decopack = zlib.decompress(decpack, -zlib.MAX_WBITS)
to
decompressor = zlib.decompressobj(-zlib.MAX_WBITS)
decopack = decompressor(decpack)
it should work as that way can handle streaming.
A the docs say
zlib.decompressobj - Returns a decompression object, to be used for decompressing data streams that won’t fit into memory at once.
or even if it does fit into memory you might just want to do the beginning of the file
Try this:
decopack = zlib.decompressobj().decompress(decpack, zlib.MAX_WBITS)
I'm trying to validate a signature from OpenSSL's rsautl command with PyCryptodome. The Bash command I use to sign the data (data is "38b..da1" string) is:
echo 38b2be8850bbf6e52bee4d4c5889df436285479f5472efe856787d7068a50da1 | openssl rsautl -inkey another_guys_priv_key.pem -sign | base64
I have converted this command into a Python script so I can compare things in Python:
from subprocess import Popen, PIPE
from shlex import split
# Use openssl to sign the data ("38b2..0da1") with another_guys private key
p1 = Popen(split('echo 38b2be8850bbf6e52bee4d4c5889df436285479f5472efe856787d7068a50da1'), stdout=PIPE)
p2 = Popen(split('openssl rsautl -inkey another_guys_priv_key.pem -sign'), stdin=p1.stdout, stdout=PIPE)
output, _ = p2.communicate()
Then I use PyCryptodome to sign the same data and compare the signatures:
from Crypto.Signature import PKCS1_v1_5
from Crypto.Hash import SHA256
from Crypto.PublicKey import RSA
# Use PyCrypto to sign the data with another_guys private key
key = RSA.importKey(open('another_guys_priv_key.pem').read())
signer = PKCS1_v1_5.new(key)
h = SHA256.new()
h.update('38b2be8850bbf6e52bee4d4c5889df436285479f5472efe856787d7068a50da1'.encode('utf-8'))
signature = signer.sign(h)
# Compare signature with OpenSSL signature
output == signature # False
and the results aren't the same.
So I was thinking that maybe OpenSSL automatically converts hexadecimal data into bytes and signs that, so I tried it with Python:
from binascii import unhexlify
# Use PyCryptodome to sign the hex-decoded data with another_guys private key
hh = SHA256.new()
hh.update(unhexlify('38b2be8850bbf6e52bee4d4c5889df436285479f5472efe856787d7068a50da1'))
hex_signature = signer.sign(hh)
# Compare signature with OpenSSL signature
output == hex_signature # False
but those results aren't the same either.
I'm obviously doing something wrong, but I'm not familiar enough with the Python crypto libraries to figure out what it is. What am I doing wrong?
The echo command (or shell built-in) adds a new line to the output by default, so signatures never match. Try disabling it.
$ echo -n foo | wc -c # hint - this counts number of bytes
3
$ echo foo | wc -c
4
But personally, I would just pass the string as standard input to the second process directly.
I am new to using the OpenSSL library and I cannot get its output to match the output from python's crypto libraries. Below is an example.
Python Code:
import hmac
secret = "asdf1234"
string = '{"request"}'
hmac.new(secret, string, hashlib.sha384).hexdigest()
'92904f1b3c210a4fb19f476c19f24452717e85329aa9ffaba4a5fbe1111c2e3fa7f5a5fb35fdc58b3d158f5f886c3d02'
OpenSSl:
echo -n {"request"} | openssl dgst -sha384 -hmac asdf1234 -hex
(stdin)= 4c3d525b8a7095b9063a3bd974e56f0a5872399365912297d6ee18e400d2b55d0142395ba5fb4f33655ceca209ba9570
What am I doing wrong? Is either implementation correct?
To match the openssl output, the Python string should be '{request}' rather than '{"request"}'.
import hmac
import hashlib
secret = "asdf1234"
string = '{request}'
hmac.new(secret, string, hashlib.sha384).hexdigest()
yields
'4c3d525b8a7095b9063a3bd974e56f0a5872399365912297d6ee18e400d2b55d0142395ba5fb4f33655ceca209ba9570'
Or, if you want the openssl command to match the Python output, use
echo -n '{"request"}' | openssl dgst -sha384 -hmac asdf1234 -hex
which yields
(stdin)= 92904f1b3c210a4fb19f476c19f24452717e85329aa9ffaba4a5fbe1111c2e3fa7f5a5fb35fdc58b3d158f5f886c3d02
After all, the inputs have to match for the outputs to have a chance at matching:
% echo -n {"request"}
{request}
and
>>> print('{"request"}')
{"request"}