Develop Reference

'550 The system cannot find the file specified' with using method request.urlopen for file on FTP

Problem with encoding when i call request.urlopen() method.
Instance of ftplib.FTP() in urllib.request.ftpwrapper init() and retrfile() methods work with default latin-1 and i need to chose between utf-8 and cp1251
I see 3 ways:
Way i want, but don't know how.
Call request.urlopen() with param that contains encoding. And that encoding must be written to the self.ftp.encoding (ftplib.FTP())
Way I don't like.
Get file name encoding from ftp (ftp lib) and use it in request.urlopen(url.encode(file_name_encoding).decode('latin-1')).
Problem description.
I have a file with Cyrillic (rus) characters in its name.
Steps:
Connecting to FTP
con = ftplib.FTP()
con.connect(host, port)
con.login(username, password)
Getting files list
list_files = [_v for _v in self.con.nlst(_path)]
['Message.xml', 'Message_ÁÏ_TT.xml']
(For files Message.xml, Message_БП_TT.xml)
Fix it with using on the first step
con.encoding = 'utf-8'
con.sendcmd('OPTS UTF8 ON')
Then I need to use:
from urllib import request
url = 'ftp://login:password#ftpaddr:21/folder//Message_БП_TT.xml'
request.urlopen(url.encode().decode('latin-1'))
And then getting Exception:
{URLError}<urlopen error ftp error: URLError("ftp error: error_perm('550 The system cannot find the file specified. ')")>
In request lib there are init() and retrfile() where ftp connection initializing.
And i don't see the way how to change ftp default encoding "latin-1".
Use this method because with urllib.response.addinfourl parse heavy xml files.
P.S.
With some FTP this method works well and the file can be successfully read. And with some of them getting that exception. The reasons are not clear yet. And there is no way to get and analyze the FTP settings.

Solution I don't like.
As i understand file name on FTP can be in utf-8 or in cp1251 (win-1251) encoding.
When ftplib initing with standard encode (latin-1) its will look like:
Message_ÐÐ_TT.xml - utf-8
Message_ÁÏ_TT.xml - cp1251
I don't know what encoding uses on ftp while making request, and always use utf-8 (encode()). So i don't like it, but it works:
try:
return request.urlopen(url.encode('utf-8').decode('latin-1'))
except URLError:
return request.urlopen(url.encode('cp1251').decode('latin-1'))
P.S. utf-8 under try for clarity

What's the best way to verify XML signature in python?

I'm trying to validate XML message signature with given public key in Python which is validated fine on a PHP code with openssl.
Here's PHP code that's working fine.
$pubKey = openssl_pkey_get_public(file_get_contents("public_key.pem"));
$xmlDoc = new DOMDocument();
$xmlDoc->load("message.xml");
$signedInfo=$xmlDoc->getElementsByTagName("SignedInfo")->item(0)->C14N(true, true);
$signature = base64_decode($xmlDoc->documentElement->getElementsByTagName("SignatureValue")->item(0)->nodeValue);
$ok = openssl_verify($signedInfo, $signature, $pubKey, OPENSSL_ALGO_SHA1);
I've found different libraries in Python to achieve this but none of them are verifying fine. I've listed the libraries and the problems I've faced on. Is there any other preferred ways to achieve this?
1. pyOpenSSL
It fails with following message: [('rsa routines', 'INT_RSA_VERIFY', 'wrong signature length')]
import OpenSSL.crypto as c
from StringIO import StringIO
import xml.etree.ElementTree as xml_et
from myapp import settings
namespace = "{http://www.w3.org/2000/09/xmldsig#}"
xml_bytes = open(settings.STATIC_ROOT + '/file/test.xml', 'rt').read()
response_xml = xml_et.fromstring(xml_bytes.encode('utf-8'))
signature_elem = response_xml.find(namespace + 'Signature')
signature_value = signature_elem.find(namespace + 'SignatureValue').text
signed_info_output = StringIO()
signed_info_tree = xml_et.ElementTree(signature_elem.find(namespace + 'SignedInfo'))
signed_info_tree.write_c14n(signed_info_output)
signed_info = signed_info_output.getvalue()
# load certificate
cert = c.load_certificate(c.FILETYPE_PEM, open(settings.STATIC_ROOT + '/file/public.cert', 'rt').read())
# verify signature
try:
c.verify(cert, signature_value, signed_info, 'sha1')
print 'success'
except Exception, e:
print 'fail'
2. M2Crypto
Tried to install M2Crypto but it fails with cannot find openssl/err.h header file. So I've installed openssl 1.1.0e and copied lib and include directories to C:/pkg directory and it throws different error like:
SWIG/_m2crypto_wrap.c(3754) : error C2065: 'CRYPTO_NUM_LOCKS' : undeclared ident ifier
And found precompiled M2Crypto msi installer but during runtime it throws following error:
ImportError: DLL load failed: The specified module could not be found.
This library seems outdated and not enough documentation available.
3. signxml
So far it's the only library that works partially for me.
Xml verification works fine but it throws error on sign: ValueError: Could not unserialize key data.
from xml.etree import ElementTree
from signxml import XMLSigner, XMLVerifier
from myapp import settings
cert = open(settings.STATIC_ROOT + '/file/public.cert', 'rt').read()
key = open(settings.STATIC_ROOT + '/file/public.key', 'rt').read()
root = ElementTree.fromstring('<xml1>12</xml1>')
signed_root = XMLSigner().sign(root, key=key, cert=cert)
verified_data = XMLVerifier().verify(signed_root).signed_xml
print verified_data

The answer to the question in the title is signxml.
It is the library designed for the stated purpose. PyOpenSSL and M2Crypto operate on lower-level objects than XML signature; verifying the latter would involve canonicalizing XML, digesting proper parts of it, and comparing the provided signature over the digest. While possible, this is not trivial and provides much space for errors. For example in your code for PyOpenSSL you do not base64-decode the signature value.
With signxml, your example is mostly correct. For verification of the signature, you do not need the private key, so the error you get is not relevant to the question. You should in general read the certificate and the key in binary, not text mode (open(filename, "rb")) - even if the files are PEM-encoded.
The following is a working example:
from xml.etree import ElementTree
from signxml import XMLSigner, XMLVerifier
cert = open("cert.pem", "rb").read()
key = open("key.pem", "rb").read()
xml_obj = ElementTree.fromstring("<Example/>")
signed_xml_obj = XMLSigner().sign(xml_obj, key=key)
XMLVerifier().verify(signed_xml_obj, x509_cert=cert)
If your XML object has been serialized and de-serialized after signing, this simplest code might be not enough; the (complicated) details are described here.

ValueError: Not a valid PEM pre boundary

this piece of code
key=RSA.importKey(open("receiver.pem","rb").read())
returns this error
ValueError: Not a valid PEM pre boundary
and
key=RSA.importKey(open("receiver.pem","r").read())
returns
ValueError: Not a valid PEM post boundary
The code worked fine when we used Pycrypto and Python 2.7, now i have shifted to Pycryptodome and Python 3.4.3(using 2to3). But now this code won't work. I can't even try anything because i can't even understand what it means.
To generate the .PEM file this code was used.
random_generator = Random.new().read
rsakey = RSA.generate(1024, random_generator)
f=open(email+'.pem','wb')
f.write(rsakey.exportKey("PEM"))
f.write(rsakey.publickey().exportKey("PEM"))
f.close()
This is what the contents of a .PEM file is like.
-----BEGIN RSA PRIVATE KEY-----
MIICWwIBAAKBgQCIQhU/+nPVFgw+T0Tf7NEpHYB12I/qywo5xBdp5kaLxEHD9zOx
2FTOX2OMPiL7fv/PW/AXuSrvD3pZAFzGmkigWdQP6TES5ZM65LUzeUUy8noHkZ00
D4mz+4a4YtBGaFyNL2CCxOAczi9rx5UB6qbY6+5kkBNd7k75XDp28g2bjwIDAQAB
AoGAaFRQ+P/HmSyci0ker2YgcJ7KMXF0II7+cWSXmNpcwb+vq2CoEFBX/vxCuKsL
Fg4TyK3YlBGPsiPjxink35xaZm7eI5sqbmD8Bnw4JZsQ1FN/Si6pbNLZkmOxyZgl
CoQvuvLavKH5GSWQ5wqvLD6OHBGd7w0YyGVOQHNQvOKhLgECQQC6EgYqOOz8ddQ2
qaLHxJl1LwpwvA4nWUWqeP69yl4QrhOmfTyLxLmw8HJFuz8XYiAxKq9fxnrU0j8H
W+QKwxRBAkEAu3eVGHZF5AA+K/Co+y2MTh1uzaSqbPZY/D4+zs1eLxoVM/e0MLYI
SqPciDTHl3HjZqivpJ5SbU3DcfvGSlV7zwJAJUxRogsRLjYsWNy+PY8iN8Q7Mofv
ymFxvo9MeRzkqDFMzRXTmizQEDDSpzm2luhbjZ+B0hAGNT0D12TLHIEoQQI/N6dI
m/qAxS9NRb4sbGUZQhd6zZIVBkQcJsZT3xEY5OLZaJQg6lUgIQiEb+s7Vbp5yABM
JJLb5ZcwbqZQN8EpAkEAt716AEn2qyxONCfLan1tuZVF+3V0KVphdhu6gdXpyHBv
9hLm2Ezb5VXMoU+IoeYGQ3SaSr6Gb1ein/sXGyaZuQ==
-----END RSA PRIVATE KEY----------BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCIQhU/+nPVFgw+T0Tf7NEpHYB1
2I/qywo5xBdp5kaLxEHD9zOx2FTOX2OMPiL7fv/PW/AXuSrvD3pZAFzGmkigWdQP
6TES5ZM65LUzeUUy8noHkZ00D4mz+4a4YtBGaFyNL2CCxOAczi9rx5UB6qbY6+5k
kBNd7k75XDp28g2bjwIDAQAB
-----END PUBLIC KEY-----

You are getting that error because of this function:
def decode(pem_data, passphrase=None):
...
# Verify Pre-Encapsulation Boundary
r = re.compile("\s*-----BEGIN (.*)-----\n")
m = r.match(pem_data)
if not m:
raise ValueError("Not a valid PEM pre boundary")
marker = m.group(1)
# Verify Post-Encapsulation Boundary
r = re.compile("-----END (.*)-----\s*$")
m = r.search(pem_data)
if not m or m.group(1) != marker:
raise ValueError("Not a valid PEM post boundary")
Unfortunately, in non-multiline regular expressions, $ means "end of the string". This implies that PyCryptoDome expects the END boundary at the end of the string, and there are no ways to work around this problem.
You have three options:
split the two keys and import them separately;
fix PyCryptoDome;
switch back to PyCrypto.

As Peter Wood commented previously, the issue is the end-of-line format.
I'm working on a project using python 2.7 in CentOS, and I've got a publickey from a website, which gave me the above error.
The problem was that it came with \r\n end of line sequence, which made the python-jose library jwt.decode call to pyCripto library fail.
My fix was to just remove '\r' character before returning the response this way:
<pre>
response = get(url).content
return response.replace('\r', '')
And this made the thing work at last. Hope this help!

How can I use Python to output a certificates-only PKCS#7

I'm implementing a SCEP server in Python. The SCEP spec calls for me to respond to a PKIOperation with a "a certificate-only PKCS#7". Apple has a reference implementation in Ruby that does the following.
require 'openssl'
##root_cert = OpenSSL::X509::Certificate.new(File.read("ca_cert.pem"))
##ra_cert = OpenSSL::X509::Certificate.new(File.read("ra_cert.pem"))
scep_certs = OpenSSL::PKCS7.new()
scep_certs.type="signed"
scep_certs.certificates=[##root_cert, ##ra_cert]
File.open('from_ruby.der', 'w') { |file| file.write(scep_certs.to_der)}
That code correctly outputs a PCKS7 DER file that contains both the CA and RA certificates. I'm trying to port this code to Python. I'm using the M2Crypto library for access to OpenSSL. I am struggling with the fact that M2Crypto.SMIME.PKCS7 does not have a certificates method. So far I have come up with the following.
from M2Crypto import X509
ca_cert = X509.load_cert('ca_cert.pem')
ra_cert = X509.load_cert('ra_cert.pem')
stack = X509.X509_Stack()
stack.push(ca_cert)
stack.push(ra_cert)
derFile = open('from_python.der', 'w')
derFile.write(stack.as_der())
This Python code does output a DER encoded file that does look like it contains both certs. However OpenSSL is unable to read this file.
openssl pkcs7 -in from_ruby.der -inform DER -print_certs
Prints out the certs from the Ruby script just fine, while
openssl pkcs7 -in from_python.der -inform DER -print_certs
throws this error
unable to load PKCS7 object
89377:error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:/SourceCache/OpenSSL098/OpenSSL098-47.1/src/crypto/asn1/tasn_dec.c:1315:
89377:error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:/SourceCache/OpenSSL098/OpenSSL098-47.1/src/crypto/asn1/tasn_dec.c:827:
89377:error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:/SourceCache/OpenSSL098/OpenSSL098-47.1/src/crypto/asn1/tasn_dec.c:747:Field=type, Type=PKCS7
How can I get Python to output both the CA and RA certs in the same format that Ruby is?
I have posted the test certs I'm using as a gist.
Update:
I figured out the openssl command that will produce the same file.
openssl crl2pkcs7 -nocrl -certfile ca_cert.pem -certfile ra_cert.pem -out crl.der -outform DER
So now, how do I do that in Python. Which is the same as this question

Had the same requirement, and ended up forking M2Crypto to add a new function that would create a degenerate PKCS7 object. https://github.com/HanSooloo/M2Crypto-martinpaljak
The steps involved were the following:
Fork M2Crypto from Martin Paljak's repo to a new one.
Modify _pkcs7.i SWIG interface file to add the function below.
_pkcs7.i Modifications
// Adding X.509 related header files to be able to use their data types.
#include <openssl/x509.h>
#include <openssl/x509v3.h>
// Adding PKCS7_SIGNED data type to help create the degenerate data structure.
%apply Pointer NONNULL { PKCS7_SIGNED * };
// Additional interface definitions for degenerate PKCS#7 object creation.
// Inspired by the crl2p7.c file from OpenSSL. Will need to clean up a bit for function returns.
%threadallow pkcs7_create_degenerate;
%inline %{
int pkcs7_create_degenerate(STACK_OF(X509) *cert_stack, BIO *bio) {
int ret=1;
PKCS7 *p7=NULL;
PKCS7_SIGNED *p7s=NULL;
X509_CRL *crl=NULL;
STACK_OF(X509_CRL) *crl_stack=NULL;
if ((p7=PKCS7_new()) == NULL) goto end;
if ((p7s=PKCS7_SIGNED_new()) == NULL) goto end;
p7->type=OBJ_nid2obj(NID_pkcs7_signed);
p7->d.sign=p7s;
p7s->contents->type=OBJ_nid2obj(NID_pkcs7_data);
if (!ASN1_INTEGER_set(p7s->version,1)) goto end;
if ((crl_stack=sk_X509_CRL_new_null()) == NULL) goto end;
p7s->crl=crl_stack;
p7s->cert=cert_stack;
ret=i2d_PKCS7_bio(bio, p7);
end:
p7s->cert=NULL;
if (p7 != NULL) {
// printf("about to free p7: ");
PKCS7_free(p7);
// printf("freed.\n");
}
return ret;
}
%}
Function Details
The function takes an X509 stack pointer and BIO pointer as inputs and returns an integer indicating success.
The X509 stack pointer needs to point to a stack that contains the certificates one wishes to place in the degenerate PKCS#7 object.
The BIO pointer needs to point to an empty BIO structure that will later be populated with the PKCS#7 object.
Python code example that uses the above function:
from M2Crypto import X509, BIO, m2
sk = X509.X509_Stack()
cert = X509.load_cert('ra.crt')
num = sk.push(cert)
cert = X509.load_cert('ca.crt')
num = sk.push(cert)
# At this point, the X509 stack contains 2 certificates.
print('num: %d' %num)
# Create the BIO that will hold the PKCS#7 object.
bio = BIO.MemoryBuffer()
# Request to create the degenerate PCKS#7 object.
ret = m2.pkcs7_create_degenerate(sk._ptr(), bio._ptr())
# Open the file for writing.
f = open('deg.p7s', 'w')
# Read from BIO and write to file.
b = bio.read()
f.write(b)
# Close the file.
f.close()

You can use Python ctypes to call the OpenSSL shared library functions directly to accomplish this. Here's an example.

I have a feeling (totally untested) that it's something like this:
an_smime = M2Crypto.SMIME.SMIME()
an_smime.set_x509_stack(stack)
an_smime.write(M2Crypto.BIO.File('from_python.der'), pkcs7=True)

python function for retrieving key and encryption

M2Crypto package is not showing the 'recipient_public_key.pem' file at linux terminal.
How do I get/connect with recipient public key.
Exactly, I need to check how can I open this file through linux commands.
import M2Crypto
def encrypt():
recip = M2Crypto.RSA.load_pub_key(open('recipient_public_key.pem','rb').read())
print recip;
plaintext = whatever i need to encrypt
msg = recip.public_encrypt(plaintext,RSA.pkcs1_padding)
print msg;
after calling the function its not giving any output and even any error
i also tried as 'Will' said
pk = open('public_key.pem','rb').read()
print pk;
rsa = M2Crypto.RSA.load_pub_key(pk)
what is the mistake I am not getting?

I have never used M2Crypto, but according to the API documentation, load_pub_key expects the file name as the argument, not the key itself. Try
recip = M2Crypto.RSA.load_pub_key('recipient_public_key.pem')