Develop Reference

MQTT PAHO [CERTIFICATE_VERIFY_FAILED]

I have a problem with Python (I'm a Python noob and learning it).
I used the version 2.7.9 on a Debian 9 System. I installed paho and tinkerforge package in python.
I developed a script using the Paho MQTT client to connected my mosquitto broker. I want to use a crypted connection. My connection work fine when not encrypted but fails when encrypted. The connection work fine encrypted on openHAB (MQTT-Subscriber) and MQTTFX (MQTT-Subscriber and Producer)
I'm using these parameters for my script:
self.client = mqtt.Client()
self.client.tls_set("/home/pi/ca-cert.pem","/home/pi/IWILR1-1-cert.pem","/home/pi/IWILR1-1.pem",tls_version=ssl.PROTOCOL_TLSv1)
# disables peer verification
self.client.tls_insecure_set(False)
self.client.on_connect = self.mqtt_on_connect
self.client.on_disconnect = self.mqtt_on_disconnect
self.client.on_message = self.mqtt_on_message
self.device_proxies = {}
self.device_proxy_classes = {}
for subclass in DeviceProxy.subclasses():
self.device_proxy_classes[subclass.DEVICE_CLASS.DEVICE_IDENTIFIER] = subclass
def connect(self):
if self.broker_username is not None:
self.client.username_pw_set(self.broker_username, self.broker_password)
self.client.connect(self.broker_host, self.broker_port)
self.client.loop_start()
But now the problem is the error on Python.
sudo python /home/pi/brick-mqtt-proxy.py
Traceback (most recent call last):
File "/home/pi/brick-mqtt-proxy.py", line 1250, in <module>
proxy.connect()
File "/home/pi/brick-mqtt-proxy.py", line 1109, in connect
self.client.connect(self.broker_host, self.broker_port)
File "/usr/local/lib/python2.7/dist-packages/paho/mqtt/client.py", line 760, in connect
return self.reconnect()
File "/usr/local/lib/python2.7/dist-packages/paho/mqtt/client.py", line 919, in reconnect
sock.do_handshake()
File "/usr/lib/python2.7/ssl.py", line 840, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)
and on mosquitto these errors arrived.
1504896114: New connection from 143.93.197.20 on port 8883.
1504896114: OpenSSL Error: error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca
1504896114: OpenSSL Error: error:140940E5:SSL routines:SSL3_READ_BYTES:ssl handshake failure
1504896114: Socket error on client <unknown>, disconnecting.
Mosquitto conf
# Place your local configuration in /etc/mosquitto/conf.d/
#
# A full description of the configuration file is at
# /usr/share/doc/mosquitto/examples/mosquitto.conf.example
pid_file /var/run/mosquitto.pid
persistence true
persistence_location /var/lib/mosquitto/
log_type all
log_facility 5
log_timestamp true
log_dest file /var/log/mosquitto/mosquitto.log
include_dir /etc/mosquitto/conf.d
port 8883
cafile /etc/mosquitto/ca_certificates/ca-cert.pem
certfile /etc/mosquitto/certs/server-cert.pem
keyfile /etc/mosquitto/certs/server-key.pem
Only Server and Ca matched the broker hostname. Client use there own hostname for CN. I hope thats right?
I hope you can help me to fix my problem.
PS: I used a self-signed certificate! TLS Version 1.2

If you are using TLS v1.2 you need to modify the expression (2nd line: self.client.tls_set()) 'tls_version=ssl.PROTOCOL_TLSv1' to 'tls_version=ssl.PROTOCOL_TLSv1_2', not as expected to ...TLSv1.2. This worked for me.

try providing something like below. Default port for ssl is 8883. We can start multiple listeners. In this case non-ssl on 1883 and ssl on 8883.
port 1883
listener 8883

Using python to send email through TLS

I am trying to automate sending email through a python script. I am presently using an expect script to use openssl s_client to connect to the server. Presently we only use a certificate file along with the username password and it allows me to send the email. I found another question in which it was mentioned that in python you either need a hack to or a wrapper around the smtp class to use only the CA cert file and not the key file(which i don't have).
>>> smtpobj = smtplib.SMTP("mymailserver.com",465)
Traceback (innermost last):
File "<stdin>", line 1, in <module>
File "C:\Program Files (x86)\Python35-32\lib\smtplib.py", line 251, in __init__
(code, msg) = self.connect(host, port)
File "C:\Program Files (x86)\Python35-32\lib\smtplib.py", line 337, in connect
(code, msg) = self.getreply()
File "C:\Program Files (x86)\Python35-32\lib\smtplib.py", line 390, in getreply
+ str(e))
smtplib.SMTPServerDisconnected: Connection unexpectedly closed: [WinError 10054] An existing connection was forcibly closed by the remote host
The problem I am facing right now is that I am unable to connect to the server through python.
If I use the certificate file to connect through
smtplib.SMTP_SSL(myserver, port, certfile="mycert.cert")
then it throws the following error.
ssl.SSLError: [Errno 336265225] _ssl.c:339: error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib
Please note, I am able to connect to the server using thunderbird, without a cert file. Any ideas, on how I can use python smtp(tls) to send the emails?

smtpobj = smtplib.SMTP("mymailserver.com",465)
smtplib.SMTPServerDisconnected: Connection unexpectedly closed: [WinError 10054] An existing connection was forcibly closed by the remote host
This error has nothing to do with validation of the certificate. It is simply that you are using explicit TLS (i.e. STARTTLS command on plain connection) on a port which requires implicit TLS (TLS from start). Try this instead:
smtpobj = smtplib.SMTP_SSL("mymailserver.com",465)
Apart from that:
... you either need a hack to or a wrapper around the smtp class to use only the CA cert file and not the key file(which i don't have).
I think you mixing up some concepts:
CA cert: this contains the trusted root which is needed to verify the certificate of the server. You don't have a key for this certificate and you don't need one.
local cert, local key: these are used if the server requires authentication with a client certificate. In this case both cert and key are needed
What you probably want to specify is a CA cert in order to verify the servers certificate. Unfortunately smtplib does not give you a way to specify this CA certificate. You've tried certfile but this is used for specifying the local cert for client certificate authentication and it requires a key file.
The good news is: it works without specifying a CA cert because smtplib simply does not verify the servers certificate at all. The bad news is: because there is no verification of the server certificate a man in the middle attack against the encrypted connection is easy.

You need to remove the key pass phrase first using -
openssl rsa -in key.pem -out tempkey.pem
And then type passphrase once more -
openssl rsa -in mycert.pem -out tempkey.pem
openssl x509 -in mycert.pem >>tempkey.pem
Refer this for more info.

Trusting self signed certificate with Python requests

My apache ssl conf has the following configs
# Server Certificate:
SSLCertificateFile /etc/pki/tls/certs/localhost.crt
# Server Private Key:
SSLCertificateKeyFile /etc/pki/tls/private/localhost.key
I do not have the CA certificates for this server. Can I still install the localhost.crt into my clients to successfully verify my server?
On the client:
I am using Python requests library (2.2.1). The default CA BUNDLE path is used. Even when I add the localhost.crt to the cacert.pem in the default path, I am unable to see the verification go through. I see the exception:
File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 385, in send
raise SSLError(e)
SSLError: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
Am I doing anything wrong? Should I only add the CA who signed the localhost.crt in the server?
Thanks,
Vijay

If you provided code and be more clear on what you're doing then you'd get a good answer.
If you want don't want to get the error even if you use an invalid certificate then try the verify=False attribute.
>>> requests.get('https://kennethreitz.com', verify=False)
If you want to use a custom certificate, then place the certificate in the script folder and use the cert=('/path/client.cert', '/path/client.key') argument.
>>> requests.get('https://kennethreitz.com', cert=('/path/client.cert', '/path/client.key')).
For more info read the docs.python-requests.org/en/master/user/advanced/ site

troubleshooting python code

I am in my first steps in learning python so excuse my questions please. I want to run the code below (taken from: http://docs.python.org/library/ssl.html) :
import socket, ssl, pprint
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# require a certificate from the server
ssl_sock = ssl.wrap_socket(s,
ca_certs="F:/cert",
cert_reqs=ssl.CERT_REQUIRED)
ssl_sock.connect(('www.versign.com', 443))
print repr(ssl_sock.getpeername())
print ssl_sock.cipher()
print pprint.pformat(ssl_sock.getpeercert())
# Set a simple HTTP request -- use httplib in actual code.
ssl_sock.write("""GET / HTTP/1.0\r
Host: www.verisign.com\r\n\r\n""")
# Read a chunk of data. Will not necessarily
# read all the data returned by the server.
data = ssl_sock.read()
# note that closing the SSLSocket will also close the underlying socket
ssl_sock.close()
I got the following errors:
Traceback (most recent call last):
File "C:\Users\e\workspace\PythonTesting\source\HelloWorld.py", line 38, in
ssl_sock.connect(('www.versign.com', 443))
File "C:\Python27\lib\ssl.py", line 331, in connect
self._real_connect(addr, False)
File "C:\Python27\lib\ssl.py", line 314, in _real_connect
self.ca_certs, self.ciphers)
ssl.SSLError: [Errno 185090050] _ssl.c:340: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib
The error reporting in python does not look guiding to find the source of the problem. i might be mistaken. Can anybody help in telling me what is the problem in the code ?

This is one area where the Python standard library is known to be difficult to use. Instead you may want to use the requests library. Documentation on sending certificates is available at: http://docs.python-requests.org/en/latest/user/advanced/#ssl-cert-verification

Your code is referring to a certificate file on drive 'F:' (using the ca_certs parameter), which is not found during execution -- is there one?
See the relevant documentation:
The ca_certs file contains a set of concatenated “certification
authority” certificates, which are used to validate certificates
passed from the other end of the connection.

Does the certificate referenced exist on your filesystem? I think that error is in response to invalid cert from this code:
ssl_sock = ssl.wrap_socket(s,ca_certs="F:/cert",cert_reqs=ssl.CERT_REQUIRED)

I/O error(socket error): [Errno 111] Connection refused

I have a program that uses urllib to periodically fetch a url, and I see intermittent
errors like :
I/O error(socket error): [Errno 111] Connection refused.
It works 90% of the time, but the othe r10% it fails. If retry the fetch immediately after it fails, it succeeds. I'm unable to figure out why this is so. I tried to see if any ports are available, and they are. Any debugging ideas?
For additional info, the stack trace is:
File "/usr/lib/python2.6/urllib.py", line 203, in open
return getattr(self, name)(url)
File "/usr/lib/python2.6/urllib.py", line 342, in open_http
h.endheaders()
File "/usr/lib/python2.6/httplib.py", line 868, in endheaders
self._send_output()
File "/usr/lib/python2.6/httplib.py", line 740, in _send_output
self.send(msg)
File "/usr/lib/python2.6/httplib.py", line 699, in send
self.connect()
File "/usr/lib/python2.6/httplib.py", line 683, in connect
self.timeout)
File "/usr/lib/python2.6/socket.py", line 512, in create_connection
raise error, msg
Edit - A google search isn't very helpful, what I got out of it is that the server
I'm fetching from sometimes refuses connections, how can I verify its not a bug in my code
and this is indeed the case?

Use a packet sniffer like Wireshark to look at what happens. You need to see a SYN-flagged packet outgoing, a SYN+ACK-flagged incoming and then a ACK-flagged outgoing. After that, the port is considered open on the local side.
If you only see the first packet and the error message comes after several seconds of waiting, the other side is not answering at all (like in: unplugged cable, overloaded server, misguided packet was discarded) and your local network stack aborts the connection attempt. If you see RST packets, the host actually denies the connection. If you see "ICMP Port unreachable" or host unreachable packets, a firewall or the target host inform you of the port actually being closed.
Of course you cannot expect the service to be available at all times (consider all the points of failure in between you and the data), so you should try again later.

Getting an ECONNREFUSED errno means that your kernel was refused a connection at the other end, so if it's a bug, it's either in your kernel or in the other end.
What you can do is to trap the error in a very specific way and try again in a little while, since this seems to work:
# This is Python > 2.5 code
import errno, time
for attempt in range(MAXIMUM_NUMBER_OF_ATTEMPTS):
try:
# your urllib call here
except EnvironmentError as exc: # replace " as " with ", " for Python<2.6
if exc.errno == errno.ECONNREFUSED:
time.sleep(A_COUPLE_OF_SECONDS)
else:
raise # re-raise otherwise
else: # we tried, and we had no failure, so
break
else: # we never broke out of the for loop
raise RuntimeError("maximum number of unsuccessful attempts reached")
Replace the two all-caps constants with your favourite numbers.

I previously had this problem with my EC2 instance (I was serving couchdb to serve resources -- am considering Amazon's S3 for the future).
One thing to check (assuming Ec2) is that the couchdb port is added to your open ports within your security policy.
I specifically encountered
"[Errno 111] Connection refused"
over EC2 when the instance was stopped and started. The problem seems to be a pidfile race. The solution for me was killing couchdb (entirely and properly) via:
pkill -f couchdb
and then restarting with:
/etc/init.d/couchdb restart

I'm not exactly sure what's causing this. You can try looking in your socket.py (mine is a different version, so line numbers from the trace don't match, and I'm afraid some other details might not match as well).
Anyway, it seems like a good practice to put your url fetching code in a try: ... except: ... block, and handle this with a short pause and a retry. The URL you're trying to fetch may be down, or too loaded, and that's stuff you'll only be able to handle in with a retry anyway.

Its seems that server is not running properly so ensure that with terminal by
telnet ip port
example
telnet localhost 8069
It will return connected to localhost so it indicates that there is no problem with the connection
Else it will return Connection refused it indicates that there is problem with the connection