Develop Reference

Generate a Java compatible Diffie-Hellman using Python

I am trying to rewrite the following code in Python. The original code is written in Javascript using the sjcl library.
// Inputs
var serverPubX = "WIUBDotrk02Rk/apL11jQPbmX0quyaYz2EIkGUlVf7s=";
var serverPubY = "diZ2CbfSUy5Kr82OIfd4Ajusq2K+/kjGZ7ymcqVwn2k=";
// The code
var serverPubXBits = sjcl.codec.base64.toBits(serverPubX);
var serverPubYBits = sjcl.codec.base64.toBits(serverPubY);
var serverPubKeyPointBits = serverPubXBits.concat(serverPubYBits);
var serverPubKey = new sjcl.ecc.elGamal.publicKey(
sjcl.ecc.curves.c256, serverPubKeyPointBits);
var clientKeys = sjcl.ecc.elGamal.generateKeys(256, 1);
// What I need:
var sharedKey = clientKeys.sec.dhJavaEc(serverPubKey);
My main problem is the dhJavaEc function. According to the sjcl documentation, it's a Java compatible Diffie-Hellman function. But I couldn't find anything equivalent in the pycryptodome library.
I checked the dhJavaEc code this is what it does:
// Looks like it converts the server key to Jacobian and then multiply it by client key
serverPubKey.J.toJac().mult(clientKeys.sec.I, serverPubKey.J).toAffine().x.toBits()
// serverPubKey.J is the X and Y keys concatenated:
sjcl.codec.base64.fromBits(serverPubKey.J.toBits())
"WIUBDotrk02Rk/apL11jQPbmX0quyaYz2EIkGUlVf7t2JnYJt9JTLkqvzY4h93gCO6yrYr7+SMZnvKZypXCfaQ=="
// In my example, clientKeys.sec.I is this:
sjcl.codec.base64.fromBits(clientKeys.sec.I.toBits())
"zIhDVlFUpWQiRP+bjyEIhSLq8rcB8+XInXGhm6JGcVI="
// And the calculated key is:
sjcl.codec.base64.fromBits(sharedKey)
"ZBin/RV1qnfKoIuel+5fzv1y8rn3UZkMPO3pXva3VzQ="
How can I generate a "sharedKey" equivalent using Python?

PyCryptodome does not seem to support ECDH at the moment, see Future plans. An alternative is the Cryptography library, see Elliptic Curve Key Exchange algorithm.
The library expects the private key as int and the public key in uncompressed format as bytes object. The uncompressed format consists of the concatenated x and y coordinates preceded by a 0x04 byte.
sjcl.ecc.curves.c256 defines secp256r1 (aka prime256v1 aka NIST P-256).
Then a possible implementation in Python with Cryptography is:
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives.asymmetric import ec
import base64
curve = ec.SECP256R1()
clientKey = 'zIhDVlFUpWQiRP+bjyEIhSLq8rcB8+XInXGhm6JGcVI='
privateKeyInt = int.from_bytes(base64.b64decode(clientKey), byteorder='big', signed=False)
privateKey = ec.derive_private_key(privateKeyInt, curve, default_backend())
serverPubKey = 'WIUBDotrk02Rk/apL11jQPbmX0quyaYz2EIkGUlVf7t2JnYJt9JTLkqvzY4h93gCO6yrYr7+SMZnvKZypXCfaQ=='
publicKeyUncomp = b'\x04' + base64.b64decode(serverPubKey)
publicKey = ec.EllipticCurvePublicKey.from_encoded_point(curve, publicKeyUncomp)
sharedSecret = privateKey.exchange(ec.ECDH(), publicKey)
print(base64.b64encode(sharedSecret).decode('utf8')) # ZBin/RV1qnfKoIuel+5fzv1y8rn3UZkMPO3pXva3VzQ=
which produces the same shared secret as the JavaScript code.

How to rebuild the libmem_crc32_direct CRC function in python?

I like to rebuild the libmem_crc32_direct function in python.
I used the crcmod python package before. So I like to setup the crc generator by using it.
the c-code looks like:
uint32_t crc_process_chunk(uint8_t* data, uint32_t len) {
return ~libmem_crc32_direct(data, len, 0xFFFFFFFF);
}
my python code looks so far:
def bit_not(n, numbits=8):
return (1 << numbits) - 1 - n
def getCRC(imageBA):
crcGen = crcmod.mkCrcFun(0x104C11DB7, initCrc=0xFFFFFFFF)
val = crcGen(imageBA)
val = bit_not(val, 32)
return val
The returned value of the python code is not equal of the one in c. So I guess I mad some error.
Any ideas?

Doesn't (1 << numbits) == 0? If this is two's complement math it should work as bit_not could be return 0-1-n. However, this isn't needed, since there is an optional xorOut parameter for crcmod. I'm thinking that since the optional rev parameter for reversed (reflected) input and output defaults to true, it needs to be set to false. I think the call to create the crc generator should be:
crcGen = crcmod.mkCrcFun(0x104C11DB7, initCrc=0xFFFFFFF, rev=False, xorOut=0xFFFFFFFF)

B bit tricky because 64Bit arithmetic on PC vs 32Bit arithmetic on ARM STM32F4, but finally this solution works:
def libmem_crc32_direct_with_xor(im, startAddr, l):
fw = im[startAddr:startAddr+l]
crcGen = crcmod.Crc(0x104C11DB7, initCrc=0xFFFFFFFF, rev = False)
crcGen.update(fw)
return (~crcGen.crcValue ) & 0xFFFFFFFF # 32bit xor

Decode C const char* in Python with ctypes

I am using ctypes (imported as c) in Python 3 to execute a C++ shared library. The library is loaded into python using:
smpLib = c.cdll.LoadLibrary(os.getcwd()+os.sep+'libsmpDyn.so')
One of the functions has the extern 'C' declaration const char* runSmpModel(...). The python function prototype is coded and run as:
proto_SMP = c.CFUNCTYPE(c.c_char_p,...)
runSmpModel = proto_SMP(('runSmpModel',smpLib))
res = runSmpModel(...)
This all works beautifully, but I'm unable to decode the res variable and obtain the string passed out by the C runSmpModel function. The value of res is displayed (I'm using ipython3) as b'\xd0'. The best solution I've found online - res.decode('utf-8') gives me the error:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 0: unexpected end of data
The const char* return value from the runSmpModel function comes from
std::string scenID = SMPLib::SMPModel::runModel(...);
return scenID.c_str();
inside runModel, it is ultimately defined as shown here, where scenName is an input string:
auto utcBuffId = newChars(500);
sprintf(utcBuffId, "%s_%u", scenName.c_str(), microSeconds); // catenate scenario name & time
uint64_t scenIdhash = (std::hash < std::string>() (utcBuffId)); // hash it
auto hshCode = newChars(100);
sprintf(hshCode, "%032llX", scenIdhash);
scenId = hshCode;
The value of this specific res should be 0000000000000000BBB00C6CA8B8872E. How can I decode this string?
After a lot of further testing, I've identified the problem as the length of the string passed from the C function. No problems if the string is up to 15 characters in length, but if it's 16 or longer - no dice. For a minimum-working example, the C-code is:
extern "C" {
const char* testMeSO()
{
string scenarioID = "abcdefghijklmnop";
return scenarioID.c_str();
}
}
and python code is (same definition of smpLib as shown above):
proto_TST = c.CFUNCTYPE(c.c_char_p)
testMeSO = proto_TST(('testMeSO',smpLib))
res = testMeSO()
print("Scenario ID: %s"%res.decode('utf-8'))
This gives the decode error, unless any character is removed from the scenarioID variable in the C function. So it seems the question is "how can Python read a C char* longer than 15 characters, using ctypes.

After several days of debugging and testing, I've finally gotten this working, using the second solution posted by #Petesh on this SO post. I don't understand why ctypes is apparently limiting the char * value passed from C to 15 characters (+termination = 256 bits?).
Essentially, the solution is to pass into the C function an extra char * buff buffer that has already been created using ctypes.create_string_buffer(32*16), as well as an unsigned int buffsize of value 32*16. Then, in the C function execute scenarioID.copy(buff,buffsize). The python prototype function is modified in an obvious way.

Python File Generator in Perl [duplicate]

I am learning Perl at my work and enjoying it. I usually do my work in Python but boss wants Perl.
Most of the concepts in Python and Perl match nicely: Python dictionary=Perl hash; Python tuple=Perl list; Python list=Perl array; etc.
Question: Is there a Perl version of the Python form of an Iterator / Generator?
An example: A Classic Python way to generate the Fibonacci numbers is:
#!/usr/bin/python
def fibonacci(mag):
a, b = 0, 1
while a<=10**mag:
yield a
a, b = b, a+b
for number in fibonacci(15):
print "%17d" % number
Iterators are also useful if you want to generate a subsection of a much larger list as needed. Perl 'lists' seem more static - more like a Python tuple. In Perl, can foreach be dynamic or is only based on a static list?
The Python form of Iterator is a form that I have gotten used to, and I do not find it documented in Perl... Other than writing this in loops or recursively or generating a huge static list, how do I (for ex) write the Fibonacci subroutine it in Perl? Is there a Perl yield that I am missing?
Specifically -- how do I write this:
#!/usr/bin/perl
use warnings; use strict; # yes -- i use those!
sub fibonacci {
# What goes here other than returning an array or list?
}
foreach my $number (fibonacci(15)) { print $number . "\n"; }
Thanks in advance to being kind to the newbie...

The concept of an iterator is a little different in Perl. You basically want to return a one-use subroutine "closed" over the persistent variables.
use bigint;
use strict;
use warnings;
sub fibonacci {
my $limit = 10**( shift || 0 );
my ( $a, $b ) = ( 0, 1 );
return sub {
return if $a > $limit;
( my $r, $a, $b ) = ( $a, $b, $a + $b );
return $r;
};
}
my $fit = fibonacci( 15 );
my $n = 0;
while ( defined( my $f = $fit->())) {
print "F($n): $f\n";
$n++;
}
And if you don't like the while loop, then here is two shots at some syntactic sugar, which basically accomplish an each-item loop.:
sub iterate ($$) {
my $iter = shift;
my $action = shift;
while ( defined( my $nextval = $iter->())) {
local *_ = \$nextval;
$action->( $_ );
}
return;
}
iterate fibonacci( 15 ) => sub { print "$_\n"; };
sub iter (&$) {
my $action = shift;
my $iter = shift;
while ( defined( my $nextval = $iter->())) {
local *_ = \$nextval;
$action->( $_ );
}
return;
}
iter { print "$_\n" } fibonacci( 15 );

For an even more flexible solution than Python's generators, I have written the module List::Gen on CPAN which provides random access lazy generator arrays:
use List::Gen;
my $fib; $fib = cache gen {$_ < 2 ? $_ : $$fib[$_ - 1] + $$fib[$_ - 2]};
say "#$fib[0 .. 15]"; # 0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610
Since generators pretend to be arrays, they can mix seamlessly with normal perl code. There is also an object oriented approach:
my $fib; $fib = cache gen {$_ < 2 ? $_ : $fib->get($_ - 1) + $fib->get($_ - 2)};
say join ' ' => $fib->slice(0 .. 15);
In each case, the generator is lazy, calculating nothing upon creation, and then calculating only those values required to satisfy the slices. The recursive definition of the Fibonacci sequence calls itself many times, so the cache function is used to make sure each value is only calculated once.
You can also use generators as iterators:
while (my $num = $fib->next) {
last if $num > 10**15;
print "$_\n";
}
$fib->next can also be written $fib->(). Since the generator is still random access, you can $fib->reset() or $fib->index = 10;
Let me know if you have any questions.
Update:
I have released a new version of the module (0.80) that makes it easier to use iterative algorithms in generators. Here is an example that closely mirrors the OP's example:
use List::Gen '*';
sub fibonacci {
my $limit = 10**shift;
my ($x, $y) = (0, 1);
While {$_ < $limit} gather {
($x, $y) = ($y, take($x) + $y)
}
}
say for #{fibonacci 15};
if you use bigint; before or at the top of the sub, you can of course:
say for #{fibonacci 400}; # or more

The excellent Higher-Order Perl book (available for free at the specified link) contains a lot of information on related topics, and in particular has a whole chapter on iterators. By "higher order" the author implies using Perl's abilities as a functional language with first-class functions to implement all kinds of cool stuff. It really is a very good book - I read most of it, and the chapters on iterators and streams are terrific. I highly recommend to at least skim through it if you plan to write Perl code.

There is a similar method to produce a Iterator / Generator, but it is not a "first class citizen" as it is on Python.
In Perl, if you do not see what you want (after a MANDATORY trip to CPAN FIRST!), you can roll your own that is similar to a Python iterator based on Perl closures and an anonymous subroutine.
Consider:
use strict; use warnings;
sub fibo {
my ($an, $bn)=(1,0);
my $mag=(shift || 1);
my $limit=10**$mag;
my $i=0;
return sub {
($an, $bn)=($bn, $an+$bn);
return undef if ($an >=$limit || wantarray );
return $an;
}
}
my $num;
my $iter=fibo(15);
while (defined($num=$iter->()) ) { printf "%17d\n", $num; }
The sub fibo maintains a Perl closure that allows persistent variables to be maintained. You can do the same by having a module, similar to C / C++. Inside fibo an anonymous subroutine does the work of returning the next data item.
To quote from the Perl Bible "You will be miserable until you learn the difference between scalar and list context" -- p 69 (A highly recommended book btw...)
In this case, the annon sub only returns a single value. The only looping mechanism that I know of in Perl that can work in scalar context is while; The others try to fill the list before proceeding I think. Therefor, if you called the anon sub in list context, it will dutifully return the next fibonacci number, unlike Python's for iterators, and the loop would terminate. That is why I put the return undef if .... wantarray because it does not work in list context as written.
There are ways to fix that. Indeed, you can write subroutines that act like map foreach etc but it is not as straightforward as Python's yield. You will need an additional function to use inside a foreach loop. The tradeoff is the Perl approach has tremendous power and flexibility.
You can read more about Perl iterators in Mark Jason Dominus' excellent book "Higher Order Perl" Chapter 4 is all about Interators brian d foy also has an excellent article on Interators in the Perl Review.

There's a good practical example here and a PDF article here... but I'm too rusty in Perl to try to implement your challenge directly (as you'll see, both the example and the approach in the PDF use a less direct approach).

In this case, memoization can be used.
use strict;
use warnings;
use Memoize;
memoize('fib');
foreach my $i (1..15) {
print "$i -> ",fib($i),"\n";
}
sub fib {
my $n = shift;
return $n if $n < 2;
fib($n-1) + fib($n-2);
}

Here is a response tailored to conform closely to the question as originally posed.
Any perl module that implements lazy lists (e.g. List::Gen, Memoize, etc.) and also lets you supply your own generator subroutine (I don't mean 'generator' as in Python) will allow you to do as shown in this example. Here the module that lazily produces the list is called Alef.
#!/usr/bin/perl -w
use strict; use warnings;
use Alef;
my $fibo;
BEGIN {
my ($a, $b) = (0, 1);
$fibo = sub {
($a, $b) = ($b, $a+$b);
$a;
}
}
my $fibonacci = new Alef($fibo);
foreach my $number ($fibonacci->take(15)){ print $number . "\n"; }
Here is the output:
[spl#briareus ~]$ ./fibo.pl
0
1
1
2
3
5
8
13
21
34
55
89
144
233
377
There is nothing magical happening behind the scenes with the lazy list module used here. This is what Alef's take subroutine looks like.
sub take {
my ($self,$n) = (#_);
my #these = ();
my $generator = $self->{'generator'};
for (1..$n){
push(#these,$self->{'this'});
$self->{'this'} = &$generator($self->{'this'});
}
#these;
}

There are a few iterator/generator modules on CPAN that would help here. Here is your example directly translated to the Coro::Generator module:
use 5.016;
use warnings;
use Coro::Generator;
sub gen_fibonacci {
my $mag = shift;
generator {
my ($a, $b) = (0, 1);
while ($a <= 10 ** $mag) {
yield $a;
($a, $b) = ($b, $a + $b);
}
yield undef; # stop it!
};
}
my $fibonacci = gen_fibonacci(15);
while (defined (my $number = $fibonacci->())) {
printf "%17d\n", $number;
}

Different results in Python and C++ when computing SHA-1 digest

I get different results when trying to compute SHA-1 digest in Python and C++.
Python code:
import hashlib
salt = 0xF0C020D239062F875C7BD8FB218D8102C9B37656F653E8DF0C655EF2D4A0CB61
password = 'pass1'
m = hashlib.sha1()
m.update( bytearray.fromhex(hex(salt)[2:-1]) )
m.update( password )
print m.hexdigest()
# output: e92f9504b2d46db0af7732c6e89e0260e63ae9b8
I extracted from the C++ code a snippet:
BigNumber salt, x;
Sha1Hash xhash;
uint8 password[] = "pass1";
// salt is received from a network packet (32 bytes)
// 2014-08-16 16:06:37 --> salt=F0C020D239062F875C7BD8FB218D8102C9B37656F653E8DF0C655EF2D4A0CB61
salt.SetBinary(lc.salt, 32);
xhash.UpdateData(salt.AsByteArray(), salt.GetNumBytes());
xhash.UpdateData(password, sizeof(password) - 1);
xhash.Finalize();
x.SetBinary(xhash.GetDigest(), xhash.GetLength());
logdebug("--> x=%s", x.AsHexStr());
// output: E5B463090B335BBC734BD3F4683F310E87ED6E4A
How must I modify my Python code to have the same results as in C++?

You use different endiness in C++ and python. So you have to reverse the bytes of your salt in python.
import hashlib
salt = 'F0C020D239062F875C7BD8FB218D8102C9B37656F653E8DF0C655EF2D4A0CB61'.decode('hex')
password = 'pass1'
m = hashlib.sha1()
m.update( salt[::-1] )
m.update( password )
print m.hexdigest()
# output: 4a6eed870e313f68f4d34b73bc5b330b0963b4e5 <- the reversed of the C++ result

SHAs are specifications, hence some initial state can be implementation dependent.
I suggest to use the very same implementation if you need to compare Python and C++ codes.