I am creating a Python function to perform counter mode encryption using the PyCrypto module. I am aware of the builtin, but want to implement it myself.
I'm trying Test Vector #1 from RFC 3686, and have the correct Counter Block and the correct Key in ASCII form. But when I encrypt the Counter Block using the Key, I don't get the expected Key Stream.
The relevant parts of my code:
cipher = AES.new(key)
ctr_block = iv + nonce + ctr
key_stream = base64.b64decode(cipher.encrypt(ctr_block))
I can provide more code if needed, but I'm not sure how because ctr_block and key have many question mark characters when I print them.
Why am I not getting the expected answer? It seems like everything should go right. Perhaps I made some mistake with the encoding of the string.
Edit
Self-contained code:
from Crypto.Cipher import AES
import base64
def hex_to_str(hex_str):
return str(bytearray([int(n, 16) for n in hex_str.split()]))
key = hex_to_str("AE 68 52 F8 12 10 67 CC 4B F7 A5 76 55 77 F3 9E")
iv = hex_to_str("00 00 00 00 00 00 00 00")
nonce = hex_to_str("00 00 00 30")
ctr = hex_to_str("00 00 00 01")
cipher = AES.new(key)
ctr_block = iv + nonce + ctr
key_stream = base64.b64decode(cipher.encrypt(ctr_block))
print "".join([hex(ord(char)) for char in key_stream])
# 0xd90xda0x72
First, the correct CTR block order is nonce + iv + ctr. Second, that base64.b64decode call is wrong: cipher.encrypt produces a decoded string. After these two fixes your code prints 0xb70x600x330x280xdb0xc20x930x1b0x410xe0x160xc80x60x7e0x620xdf which seems to be a correct key stream.
First, use byte strings:
In [14]: keystring = "AE 68 52 F8 12 10 67 CC 4B F7 A5 76 55 77 F3 9E"
In [15]: keystring.replace(' ', '').decode('hex')
Out[15]: '\xaehR\xf8\x12\x10g\xccK\xf7\xa5vUw\xf3\x9e'
Second, you shouldn't use base64.
Related
I have a private key with the below format. The private key is just for testing.
xprvA3bdZ5Dz3QFmyC6Y7tKeJahknnUZPgpw2Zhf7LNmZ1uLfJo2b557DpPBeBVW6Etbggpnd6VRUEWvKUj3NnBuU1MeWH8CY7eVTQ2yvZUXYSq
That is very likely in a Ethereum format. I need to convert it to a base64 format, in order to use it in signatures. (filecoin transaction need to base64 encoded format of privatekey)
For sending Filecoin transaction I need to use this method:
transactionSignLotus(unSignedMessage, privatekey)
## transactionSignLotus (support Lotus schema)
Sign a transaction and return a JSON string of the signed transaction which can then be sent to a lotus node.
Arguments:
* **transaction**: a filecoin transaction;
* **privatekey**: a private key (hexstring or buffer);
My privatekey doesn't work for this method. I think I need to convert to base64, but I used most of the ways for converting the private key to base64 format, cannot got results.
Note: When I say base64 I mean something like this:
px2g1zwEd1+EMfj4nX1oh0roouBGHhPo7QUNkPBHk1Q=
$ cat 68879302.js
let bip32 = require('bitcoinjs-lib').bip32
let x = bip32.fromBase58('xprvA3bdZ5Dz3QFmyC6Y7tKeJahknnUZPgpw2Zhf7LNmZ1uLfJo2b557DpPBeBVW6Etbggpnd6VRUEWvKUj3NnBuU1MeWH8CY7eVTQ2yvZUXYSq')
console.log(x.__D)
$ node 68879302.js
<Buffer cf 8c f7 64 cb d5 6b 5c 40 fa d6 7c 40 9d 52 2e c1 17 e4 93 16 dd 03 ec ec c4 1f 6c 80 6c ee d4>
import base64
key = b'xprvA3bdZ5Dz3QFmyC6Y7tKeJahknnUZPgpw2Zhf7LNmZ1uLfJo2b557DpPBeBVW6Etbggpnd6VRUEWvKUj3NnBuU1MeWH8CY7eVTQ2yvZUXYSq'
b64 = base64.b64encode(key)
print(b64)
# got b'eHBydkEzYmRaNUR6M1FGbXlDNlk3dEtlSmFoa25uVVpQZ3B3MlpoZjdMTm1aMXVMZkpvMmI1NTdEcFBCZUJWVzZFdGJnZ3BuZDZWUlVFV3ZLVWozTm5CdVUxTWVXSDhDWTdlVlRRMnl2WlVYWVNx'
Did it help?
Assuming you use *nix with bash tools:
$> echo -n “* privatekey: PUT-YOUR-HEX-HERE” | base64 -w 0
PYTHON
I would like to open a text file, search for a HEX pattern that starts with 45 3F, grab the following hex 6 HEX values, for example 45 4F 5D and put all this in a new file. I also know that it always ends with 00 00.
So the file can look like: bla bla sdsfsdf 45 3F 08 DF 5D 00 00 dsafasdfsadf 45 3F 07 D3 5F 00 00 xztert
And should be put in a new file like this:
08 DF 5D
07 D3 5F
How can I do that?
I have tried:
output = open('file.txt','r')
my_data = output.read()
print re.findall(r"45 3F[0-9a-fA-F]", my_data)
but it only prints:
[]
Any suggestions?
Thank you :-)
Here's a complete answer (Python 3):
with open('file.txt') as reader:
my_data = reader.read()
matches = re.findall(r"45 3F ([0-9A-F]{2} [0-9A-F]{2} [0-9A-F]{2})", my_data)
data_to_write = " ".join(matches)
with open('out.txt', 'w') as writer:
writer.write(data_to_write)
re.findall returns the capturing group, so there's no need to clear out '45 3F'.
When dealing with files, use with open so that the file descriptor will be released when indentation ends.
if you want to accept the first two octets dynamically:
search_string = raw_input()
regex_pattern = search_string.upper() + r" ([0-9A-F]{2} [0-9A-F]{2} [0-9A-F]{2})"
matches = re.findall(regex_pattern, my_data)
I'm writing python script and I need to obtain exif information from raw photo file (.CR2 for example).
I found Python Rawkit offer the ability to do that.
with Raw(filename=image_path) as raw:
print raw.metadata
Metadata(aperture=-1.2095638073643314e+38, timestamp=4273602232L,
shutter=-1.1962713245823862e+38, flash=True,
focal_length=-1.2228562901462766e+38, height=3753,
iso=-1.182978841800441e+38,
make='Canon', model='EOS 5D Mark II',
orientation=0, width=5634)
But I'm a little bit confused, how read this values ?. For example I'm expecting iso value like 100/200/400 but what is -1.182978841800441e+38 ?
My question is not specific for iso, it's also for shutter, aperture, ...
I ckecked libraw and rawkit doc but was not able to find how read / convert this kind of values.
This part in the doc is not very detailed :
float iso_speed;
ISO sensitivity.
float shutter;
Shutter speed.
Can someone help me understand how to read these values?
Thanks
[Update]
As neo suggest, I will use ExifRead. In fact it's a better choice, I'm writting a python script. With ExifRead no need of extra C library dependency.
I was able to open Canon raw file and parse Exif but unfortunately facing a wrong value for the aperture :
EXIF ApertureValue (Ratio): 3
# My photo was taken in 2.8 (maybe a rounded value on this flag ?)
Quick answer : use Fnumber flag
EXIF FNumber (Ratio): 14/5
14/5 is in fact 2.8 (do the math)
Long answer (how I found / debug that) :
Reading this exelent link Understanding What is stored in a Canon RAW .CR2 file, How and Why ( http://lclevy.free.fr/cr2/ ) I decided to decode myself and know what is going on.
This link send me on the graal to decode a raw file cr2_poster.pdf
From that I thought the best value seems to be in my canon specific MakerNote section on the FNumber value. (All values description is here canon_tags)
Tag Id : 3 (In fact 0x0003 that you write 0x3)
Name : FNumber
I opened my file with an Hexa editor (hexedit) and ... I was totally lost.
Key things :
An offset is a address in the file that will contain your value.
Read : C8 05 in the file should be read 05C8. Example for an offset, the address is 0x5C8
With that found the MakeNote section is easy.
Quick way is to search directly the 0x927c MarkerNote (so in the file 7C 92) flag that contain the address of the MakerNote section.
If you are not able to found that, go throught the IFD section to find the EXIF subsection. And then in that subsection you will find the MakerNote section
Tag Type Count Value
7C 92 07 00 B8 A0 00 00 84 03 00 00
Offset : 84 03 00 00 -> 00 00 03 84 (0x384 address)
Go to this address and search in the MakerNote section the FNumber 0x3
Tag Type Count Value
03 00 03 00 04 00 00 00 C8 05 00 00
Go to the offset 0x5C8 to find our value (count 4 x type 3 ushort, 16 bits)
0x0x5C8 : 00 00 00 00 00 00 00 00
And ... fail, in fact my canon does not filled this section.
Reading http://www.exiv2.org/tags.html The FNumber can be found in EXIF subsection.
Do the same process to find the EXIF subsection and the tag "0x829d Exif.Image.FNumber type 5 Rational"
Rational type is composed of 64 bits (numerator and denominator ulongs) Rational_data_type
Tag Type Count Value
9D 82 05 00 01 00 00 00 34 03 00 00
And then read the 0x334 offset
1C 00 00 00 0A 00 00 00
As we can read in Hexa : 0x1C / 0XA
In decimal, do the math : 28/10 = 14/5 = 2.8
Verify I have this value in ExifRead
EXIF.py 100EOS5D/IMG_8813.CR2 -vv | grep -i 14/5
EXIF FNumber (Ratio): 14/5
And voila !
I was looking for 2.8 float and this value is stored in fraction format. So the library don't do the math and just simplify the fraction.
This is why we have 14/5 and not 2.8 as expected.
I suggest you use a library that is focused on EXIF reading. The stuff available in libraw/rawkit is really just a nice extra. I can recommend the ExifRead library. It's pure Python and also damn fast. And it gives you better to understand values.
If compatibility with many formats is more of an issue to you than performance you could call exiftool as a subprocess with -j option to give you a json string which you can turn into a dictionary.
That should set you up for most raw formats and even stuff that isn't images at all. And it is going to squeeze every last bit of exif info out of the file. However in comparison with other options it is rather sluggish (like 200x slower):
from PIL import Image
import PIL.ExifTags
import subprocess
import json
import datetime
import exifread
filePath = "someImage.jpg"
filePath = "someRawImage.CR2"
filePath = "someMovie.mov"
filePath = "somePhotoshopImage.psd"
try:
start = datetime.datetime.now()
img = Image.open(filePath)
exif_0 = {
PIL.ExifTags.TAGS[k]: v
for k, v in img.getexif().items()
if k in PIL.ExifTags.TAGS
}
end = datetime.datetime.now()
print("Pillow time:")
print(end-start)
print(str(len(exif_0)), "tags retrieved")
print (exif_0, "\n")
except:
pass
try:
start = datetime.datetime.now()
exif_1 = json.loads(subprocess.run(["/usr/local/bin/exiftool", "-j", filePath], stdout=subprocess.PIPE).stdout.decode("utf-8"))
end = datetime.datetime.now()
print("subprocess time:")
print(end-start)
print(str(len(exif_1[0])), "tags retrieved")
print(exif_1, "\n")
except:
pass
try:
start = datetime.datetime.now()
f = open(filePath, "rb")
exif_2 = exifread.process_file(f)
end = datetime.datetime.now()
print("Exifread time:")
print(end-start)
print(str(len(exif_2)), "tags retrieved")
print(exif_2, "\n")
except:
pass
I'm trying to translate the following python script to R, but I'm having difficulty that reflects that fact that I am not well versed with Python or R.
Here is what I have for Python:
import hashlib, hmac
print hmac.new('123456', 'hello'.encode('utf-8'),hashlib.sha256).digest()
When I run this in Python I'm getting a message that says standard output is empty.
Question: What am I doing wrong?
Here's what I'm using for R
library('digest')
hmac('123456','hello', algo='sha256', serialize=FALSE)
My questions with the R code are:
How do I encode to utf-8 in R. I couldn't find a package.
What are the correct parameter settings for serialize and raw for R given I want to match the output of the Python function above (once its working).
If you want to get the bytes of the hash in R, set raw=TRUE. Then you can write it out as a binary fine
library('digest')
x <- hmac('123456', enc2utf8('hello'), algo='sha256', serialize=FALSE, raw=TRUE)
writeBin(x, "Rout.txt")
If you're not outputting text, the encoding doesn't matter. These are raw bytes. The only different in the output is that the python print seems to be adding a new line character. If I hexdump on the R file i see
0000000 ac 28 d6 02 c7 67 42 4d 0c 80 9e de bf 73 82 8b
0000010 ed 5c e9 9c e1 55 6f 4d f8 e2 23 fa ee c6 0e dd
I'm working on Windows with Python 2.6.1.
I have a Unicode UTF-16 text file containing the single string Hello, if I look at it in a binary editor I see:
FF FE 48 00 65 00 6C 00 6C 00 6F 00 0D 00 0A 00
BOM H e l l o CR LF
What I want to do is read in this file, run it through Google Translate API, and write both it and the result to a new Unicode UTF-16 text file.
I wrote the following Python script (actually I wrote something more complex than this with more error checking, but this is stripped down as a minimal test case):
#!/usr/bin/python
import urllib
import urllib2
import sys
import codecs
def translate(key, line, lang):
ret = ""
print "translating " + line.strip() + " into " + lang
url = "https://www.googleapis.com/language/translate/v2?key=" + key + "&source=en&target=" + lang + "&q=" + urllib.quote(line.strip())
f = urllib2.urlopen(url)
for l in f.readlines():
if l.find("translatedText") > 0 and l.find('""') == -1:
a,b = l.split(":")
ret = unicode(b.strip('"'), encoding='utf-16', errors='ignore')
break
return ret
rd_file_name = sys.argv[1]
rd_file = codecs.open(rd_file_name, encoding='utf-16', mode="r")
rd_file_new = codecs.open(rd_file_name+".new", encoding='utf-16', mode="w")
key_file = open("api.key","r")
key = key_file.readline().strip()
for line in rd_file.readlines():
new_line = translate(key, line, "ja")
rd_file_new.write(unicode(line) + "\n")
rd_file_new.write(new_line)
rd_file_new.write("\n")
This gives me an almost-Unicode file with some extra bytes in it:
FF FE 48 00 65 00 6C 00 6C 00 6F 00 0D 00 0A 00 0A 00
20 22 E3 81 93 E3 82 93 E3 81 AB E3 81 A1 E3 81 AF 22 0A 00
I can see that 20 is a space, 22 is a quote, I assume that "E3" is an escape character that urllib2 is using to indicate that the next character is UTF-16 encoded??
If I run the same script but with "cs" (Czech) instead of "ja" (Japanese) as the target language, the response is all ASCII and I get the Unicode file with my "Hello" first as UTF-16 chars and then "Ahoj" as single byte ASCII chars.
I'm sure I'm missing something obvious but I can't see what. I tried urllib.unquote() on the result from the query but that didn't help. I also tried printing the string as it comes back in f.readlines() and it all looks pretty plausible, but it's hard to tell because my terminal window doesn't support Unicode properly.
Any other suggestions for things to try? I've looked at the suggested dupes but none of them seem to quite match my scenario.
I believe the output from Google is UTF-8, not UTF-16. Try this fix:
ret = unicode(b.strip('"'), encoding='utf-8', errors='ignore')
Those E3 bytes are not "escape characters". If one had no access to documentation, and was forced to make a guess, the most likely suspect for the response encoding would be UTF-8. Expectation (based on a one-week holiday in Japan): something like "konnichiwa".
>>> response = "\xE3\x81\x93\xE3\x82\x93\xE3\x81\xAB\xE3\x81\xA1\xE3\x81\xAF"
>>> ucode = response.decode('utf8')
>>> print repr(ucode)
u'\u3053\u3093\u306b\u3061\u306f'
>>> import unicodedata
>>> for c in ucode:
... print unicodedata.name(c)
...
HIRAGANA LETTER KO
HIRAGANA LETTER N
HIRAGANA LETTER NI
HIRAGANA LETTER TI
HIRAGANA LETTER HA
>>>
Looks close enough to me ...