I have a unity3d application that request a json string of image name including its hash in my django webserver. Then my unity app will check my existing image hash if its the same as the json requested. My problem is that unity hash result is different from my python hash result value. I also tried to hash string on both and it returns the same hash value.
Python Hash:
>>> image_file = open('C:/image.png').read()
>>> hashlib.md5(image_file).hexdigest()
'658e8dc0bf8b9a09b36994abf9242099'
Unity3d Hash:
public static string ComputeHash()
{
// Form hash
System.Security.Cryptography.MD5 h =System.Security.Cryptography.MD5.Create();
var myImage = File.OpenRead(PathByPlatform("image.png"));
var data = h.ComputeHash(myImage );
System.Text.StringBuilder sb = new System.Text.StringBuilder();
for (int i = 0; i < data.Length; ++i)
{
sb.Append(data[i].ToString("x2"));
}
return sb.ToString();
//This fucntion returns
//fac7f19792a696d81be77aca7dd499d0
}
Did you try open('C:/image.png', "rb").read() in order to read the file in binary mode?
Reading files without the "b" will change line ending characters on Windows from CR/LF to LF which has an impact on the hash. (at least for python2)
Related
I'm trying to utilize a database from another program in a php based website tool, and apparently the original was built in python and puts some of it's data into a python tuple and serializes it to store it as a blob in the sql table.
I'm not a python programmer so I'm not sure how to even see what is in this blob, but I do know that some of the 'type' indicators for the data field are stored in there and I want to extract them and anything else useful.
Is there any way to 'unserialize' a python tuple in php?
The blob data turned out to be a pickled tuple (part of the reason I despise python - both data types that only python can read! Python programmers: 'standardized conventions? Who needs standardized conventions?!?!')
I came up with a cludgy way to 'unpickle' the data and json serialize it using a command line. To get the binary blob data into the command line, I base64 encode it. It's janky but it works for what I need:
/**
* use a python exec call to 'unpickle' the blob_data
* to get the binary blob into a command line argument, base64 encode it
* to get the data back out of python, json serialize it
* #param string $blob binary blob data
* #return mixed
*/
public static function unpickle($blob) {
$cmd = sprintf("import pickle; import base64; import json; print(json.dumps(pickle.loads(base64.b64decode('%s'))))", base64_encode($blob));
$pcmd = sprintf("python -c \"%s\"", $cmd);
$result = exec($pcmd);
$resdec = json_decode($result);
return $resdec;
}
With a little more playing on this concept, I gave myself a few more alternatives. First is, I took the command line version above and made it into a little more functional python script:
unpickle.py:
#!/usr/bin/env python3
import pickle
import json
import sys
import base64
import select
def isBase64(s):
try:
return s == base64.b64encode(base64.b64decode(s)).decode('ascii')
except Exception:
return False
bblob = None
if (len(sys.argv) > 1) and isBase64(sys.argv[1]):
bblob = base64.b64decode(sys.argv[1])
elif select.select([sys.stdin, ], [], [], 0.0)[0]:
try:
with open(0, 'rb') as f:
bblob = f.read()
except Exception as e:
err_unknown(e)
if bblob != None:
unpik = pickle.loads(bblob)
jsout = json.dumps(unpik)
print(jsout)
This script allows you to either specify the blob data from the pickled tuple 'byte' as a base64 encoded string on the command line, or you can pipe raw blob data into the script. Both variations will output json if the data is valid and formatted properly. (null if not)
You can convert this to a self-contained binary to plop on systems without python using pyinstaller -F if need be. To play with it in the event I am running it on systems with the pyinstaller binary vs one with the python script vs one with just python, I created the following static methods in my laravel model. (I'll eventually move it into a service module)
/**
* call either a pyinstaller binary or python script with raw blob data to be unpickled
*
* #param string $b binary data of blob
* #return false|mixed
*/
public static function unpickle($b)
{
$cmd = base_path(env('UNPICKLE_BINARY', 'bin/unpickle'));
if(!(is_file($cmd) && is_executable($cmd))) { // make sure unpickle cmd exists
// check for UNPICKLE_BINARY with .py after and python binary
$pyExe = env('PYTHON_EXE', '/usr/bin/python');
if (is_file($cmd.".py") && (is_file($pyExe) && is_executable($pyExe))) {
$cmd = sprintf("%s %s.py", $pyExe, $cmd);
} else
return static::unpyckle($b); // try direct python call
}
$descriptorspec = [
["pipe", "r"],
["pipe", "w"],
["pipe", "w"]
];
$cwd = dirname($cmd);
$env = [];
$process = proc_open($cmd, $descriptorspec, $pipes, $cwd, $env);
if (is_resource($process)) {
fwrite($pipes[0], $b);
fclose($pipes[0]);
$output = stream_get_contents($pipes[1]);
fclose($pipes[1]);
$return_value = proc_close($process);
if(static::isJson($output))
return json_decode($output);
else
return false;
}
return false;
}
/**
* use a python exec call to 'unpickle' the blob_data
* to get the binary blob into a command line argument, base64 encode it
* to get the data back out of python, json serialize it
* #param string $blob binary blob data
* #return mixed
*/
public static function unpyckle($blob) {
$pyExe = env('PYTHON_EXE', '/usr/bin/python');
if (!(is_file($pyExe) && is_executable($pyExe)))
throw new Exception('python executable not found!');
$bblob = base64_encode($blob);
$cmd = sprintf("import pickle; import base64; import json; print(json.dumps(pickle.loads(base64.b64decode('%s'))))", $bblob);
$pcmd = sprintf("%s -c \"%s\"", $pyExe, $cmd);
$result = exec($pcmd);
$resdec = json_decode($result);
return $resdec;
}
/**
* try to detect if a string is a json string
*
* #param $str
* #return bool
*/
public static function isJson($str) {
if(is_string($str) && !empty($str)) {
json_decode($str);
return (json_last_error() == JSON_ERROR_NONE);
}
return false;
}
example .env values:
UNPICKLE_BINARY=bin/unpickle
PYTHON_EXE=/usr/bin/python3
basically showing three different ways to call python to do essentially the same thing...
Guacamole provides a default username and password (guacadmin and guacadmin) initialized in a postgres database like this:
INSERT INTO guacamole_user (entity_id, password_hash, password_salt, password_date)
SELECT
entity_id,
decode('CA458A7D494E3BE824F5E1E175A1556C0F8EEF2C2D7DF3633BEC4A29C4411960', 'hex'), -- 'guacadmin'
decode('FE24ADC5E11E2B25288D1704ABE67A79E342ECC26064CE69C5B3177795A82264', 'hex'),
CURRENT_TIMESTAMP
FROM guacamole_entity WHERE name = 'guacadmin' AND guacamole_entity.type = 'USER';
I'm trying to to understand how that password hash was generated. From the documentation:
Every user has a corresponding entry in the guacamole_user and guacamole_entity tables. Each user has a corresponding unique username, specified via guacamole_entity, and salted password. The salted password is split into two columns: one containing the salt, and the other containing the password hashed with SHA-256.
[...]
password_hash
The result of hashing the user’s password concatenated with the contents of password_salt using SHA-256. The salt is appended to the password prior to hashing.
password_salt
A 32-byte random value. When a new user is created from the web interface, this value is randomly generated using a cryptographically-secure random number generator.
And I think the corresponding Java code is here:
StringBuilder builder = new StringBuilder();
builder.append(password);
if (salt != null)
builder.append(BaseEncoding.base16().encode(salt));
// Hash UTF-8 bytes of possibly-salted password
MessageDigest md = MessageDigest.getInstance("SHA-256");
md.update(builder.toString().getBytes("UTF-8"));
return md.digest();
I'm trying to reproduce this in Python. It looks like they're taking
the password, appending the hex-encoded salt, and then calculating the
sha256 checksum of the resulting byte string. That should be this:
>>> from hashlib import sha256
>>> password_salt = bytes.fromhex('FE24ADC5E11E2B25288D1704ABE67A79E342ECC26064CE69C5B3177795A82264')
>>> password_hash = sha256('guacadmin'.encode() + password_salt.hex().encode())
>>> password_hash.hexdigest()
'523912c05f1557e2da15350fae7217c04ee326edacfaa116248c1ee4e680bd57'
...but I'm not getting the same result. Am I misreading (or
misunderstanding) the Java code?
...and of course I figured it out right after posting the question. The difference is that BaseEncoding.base16().encode(...) produces a hex encoding using upper-case characters, while Python's hex() method uses lower case. That means the equivalent code is in fact:
>>> from hashlib import sha256
>>> password_salt = bytes.fromhex('FE24ADC5E11E2B25288D1704ABE67A79E342ECC26064CE69C5B3177795A82264')
>>> password_hash = sha256("guacadmin".encode() + password_salt.hex().upper().encode())
>>> password_hash.hexdigest()
'ca458a7d494e3be824f5e1e175a1556c0f8eef2c2d7df3633bec4a29c4411960'
In case anyone stumbles across the same issue, I was able to extract the Java code into a simple test case:
import com.google.common.io.BaseEncoding;
import java.io.UnsupportedEncodingException;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.HexFormat;
class Main {
public static void main(String args[]) {
String password = "guacadmin";
byte[] salt = HexFormat.of().parseHex("FE24ADC5E11E2B25288D1704ABE67A79E342ECC26064CE69C5B3177795A82264");
try {
StringBuilder builder = new StringBuilder();
builder.append(password);
if (salt != null)
builder.append(BaseEncoding.base16().encode(salt));
System.out.println("builder is: " + builder.toString());
// Hash UTF-8 bytes of possibly-salted password
MessageDigest md = MessageDigest.getInstance("SHA-256");
md.update(builder.toString().getBytes("UTF-8"));
System.out.println(BaseEncoding.base16().encode(md.digest()));
}
catch (UnsupportedEncodingException e) {
System.out.println("no such encoding");
}
catch (NoSuchAlgorithmException e) {
System.out.println("no such algorithm");
}
}
}
This gave me something to run interactively and check the output. This requires the guava library, and can be compiled like this:
$ javac -classpath .:guava-31.1-jre.jar -d . Main.java
And run like this:
$ java -classpath .:guava-31.1-jre.jar Main
I am using RabbitMQ with Docker. I would like to update the configurations directly in the definitions.json file. The users should have their password stored there with rabbit_password_hashing_sha256 hashing algorithm. I have found a useful Python script for hashing the password but I was not able to reproduce it's logic in NodeJS with Crypto library.
Python script:
#!/usr/bin/env python3
# RabbitMQ password hashing algorith as laid out in:
# http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2011-May/012765.html
from __future__ import print_function
import base64
import os
import hashlib
import struct
import sys
# The plain password to encode
password = sys.argv[1]
# Generate a random 32 bit salt
salt = os.urandom(4)
# Concatenate with the UTF-8 representation of plaintext password
tmp0 = salt + password.encode('utf-8')
# Take the SHA256 hash and get the bytes back
tmp1 = hashlib.sha256(tmp0).digest()
# Concatenate the salt again
salted_hash = salt + tmp1
# Convert to base64 encoding
pass_hash = base64.b64encode(salted_hash)
# Print to the console (stdout)
print(pass_hash.decode("utf-8"))
Output: python hash-password.py test >> t7+JG/ovWbTd9lfrYrPXdFhNZLcO+y56x4z0d8S2OutE6XTE
First implementation failure:
const crypto = require('crypto');
this.password = process.argv[2];
this.salt = crypto.randomBytes(16).toString('hex');
this.password_hash = crypto.pbkdf2Sync(this.password.trim(), this.salt, 1000, 24, `sha256`).toString(`hex`);
console.log(this.password_hash);
Output: node password.js test >> 7611058fb147f5e7a0faab8a806f56f047c1a091d8355544
I was not able to reproduce it in NodeJS, so I collected the stdout result of the executed child process, which is not too elegant.
Second implementation failure:
const crypto = require('crypto');
const utf8 = require('utf8');
this.password = process.argv[2];
this.salt = crypto.randomBytes(4);
this.tmp0 = this.salt + utf8.encode(this.password);
this.tmp1 = crypto.createHash(`sha256`).digest();
this.salted_hash = this.salt + this.tmp1;
this.pass_hash = Buffer.from(this.salted_hash).toString('base64');
console.log(utf8.decode(this.pass_hash));
Output: node password.js test >> Mu+/ve+/vWnvv73vv71C77+977+9HBTvv73vv73vv73ImW/vv70kJ++/vUHvv71k77+977+9TO+/ve+/ve+/vRt4Uu+/vVU=
Can anyone help with the right implementation?
You can do the port to NodeJS more or less 1:1:
var crypto = require('crypto')
// The plain password to encode
var password = Buffer.from('my passphrase', 'utf8') // sample password
// Generate a random 32 bit salt
var salt = crypto.randomBytes(4);
//var salt = Buffer.from('1234', 'utf8'); // for testing, gives pass_hash = MTIzNNcAIpZVAOz2It9VMePU/k4wequLpsQVl+aYDdJa6y9r
// Concatenate with the UTF-8 representation of plaintext password
var tmp0 = Buffer.concat([salt, password])
// Take the SHA256 hash and get the bytes back
var tmp1 = crypto.createHash('sha256').update(tmp0).digest()
// Concatenate the salt again
var salted_hash = Buffer.concat([salt, tmp1])
// Convert to base64 encoding
pass_hash = salted_hash.toString('base64')
// Print to the console (stdout)
console.log(pass_hash)
The code above uses as example password my passphrase. You need to replace the password with yours.
Note that even if the passwords are identical, you cannot directly compare the results of Python and NodeJS code because of the random salt.
Therefore, the commented out line with the UTF-8 encoded salt 1234 can be used to produce a result for comparison with the Python code: MTIzNNcAIpZVAOz2It9VMePU/k4wequLpsQVl+aYDdJa6y9r
The issue in your first implementation is, among other things, the use of PBKDF2, which is not applied in the Python code.
The second implementation is closer to the actual solution. One problem is that the hashing does not take into account the data to be hashed.
Another defect is the use of strings, where some operations implicitly apply UTF-8 encoding, which corrupts the (arbitrary binary) data. To prevent this, binary must be used as encoding instead of UTF-8. Just because of the possible encoding issues, the implementation with strings is less robust here than with buffers.
I got a task to add a digital signature and the signer's public key to an XML file. The XML file in question would look like this:
<data>
<head>
<field1>foo</field2>
<fieldn>bar</fieldn>
</head>
<headSigner>John Doe</headSigner>
<signerPublicKey>dfgdgd...sdfgdgdsg</signerPublicKey>
<headSignature>sdafa...sfsafsasdfsafasd</headSignature>
</data>
Is this common or even feasible? I can write something in Python or Powershell that would:
1) Write the head XML and dump it to a file.
2) Run gpg to sign the file with the --clear-sign flag.
3) Parse the signed file that gpg makes for the signature string.
4) Add that string to the corresponding element in the XML.
Is there are an easier-or standard-way to do this? Maybe a Python or Powershell module that's already set up for that?
Use C# console app to sign the XML file using a certificate. Like this:
using System;
using System.Xml;
using System.Security.Cryptography.X509Certificates;
using System.Security.Cryptography;
using System.Security.Cryptography.Xml ;
var certStore = new X509Store(StoreLocation.CurrentUser);
certStore.Open(OpenFlags.ReadOnly);
// get cert rerquired
var certificateThumbPrint = "33eeededldodoijdlnkddzippy2e";
var certCollection = certStore.Certificates.Find(X509FindType.FindByThumbprint , certificateThumbPrint, true);
var myCert = certCollection[0];
// I can get the correct certificate but the following line throws "Invalid provider type specified." error
var SigningKey = myCert.GetRSAPrivateKey();
XmlDocument xmlDoc = new XmlDocument();
xmlDoc.Load(args[0] );
xmlDoc.PreserveWhitespace = true;
Sign file using this:
private static void SignXml2(XmlDocument xmlDoc, X509Certificate2 cert , string file1)
{
// Create a SignedXml object.
SignedXml signedXml = new SignedXml(xmlDoc);
// Add the key to the SignedXml document.
signedXml.SigningKey = cert.PrivateKey;
// Create a reference to be signed.
Reference reference = new Reference();
reference.Uri = "";
// Add an enveloped transformation to the reference.
var env = new XmlDsigEnvelopedSignatureTransform();
reference.AddTransform(env);
// Include the public key of the certificate in the assertion.
signedXml.KeyInfo = new KeyInfo();
signedXml.KeyInfo.AddClause(new KeyInfoX509Data(cert, X509IncludeOption.WholeChain));
// Add the reference to the SignedXml object.
signedXml.AddReference(reference);
// Compute the signature.
signedXml.ComputeSignature();
// Get the XML representation of the signature and save
// it to an XmlElement object.
XmlElement xmlDigitalSignature = signedXml.GetXml();
// Append the element to the XML document.
xmlDoc.DocumentElement.AppendChild(xmlDoc.ImportNode(xmlDigitalSignature, true));
xmlDoc.PreserveWhitespace = true;
xmlDoc.Save(file1);
}
There is function creating file with data in C++ Builder:
int HandleFile;
if (!FileExists(fnm))
{HandleFile = FileCreate(fnm);FileClose(HandleFile);}
HandleFile = FileOpen(fnm,fmOpenWrite);
if(! HandleFile) {return 0;}
AnsiString str = IntToStr(num)+"#" +IntToStr( GetLastError() )+": "+ AnsiLastError();
FileSeek(HandleFile,0,2);
FileWrite(HandleFile, &str, sizeof(str));
FileClose(HandleFile);
return 1;
Is there any way to read it in python?
When I open file by Notepad I see only unrecognized symbols
FileWrite(HandleFile, &str, sizeof(str));
isn't correct.
FileWrite expects a pointer to a raw buffer and writes x bytes of the buffer to the file given by HandleFile.
An AnsiString object contains a pointer to the heap where all data is stored (and some other variables). So sizeof(str) != str.Length() and &str != str.c_str().
You should write something like:
FileWrite(HandleFile, str.c_str(), str.Length());
Anyway take a look at TStringList, it could be what you need.