Develop Reference

Get a server certificate despite handshake failure in Python

I am writing a tool to monitor server certificate expiration. I'm using python3 ssl and socket modules to get the server cert using a pretty basic method of creating a default context, disabling hostname validation and certificate verification, calling SSLSocket.connect(), then SSLSocket.getpeercert(), with the sole purpose of grabbing the server certificate, and that is all.
This is all within a private network and I am not concerned with validation.
I have some devices that require client certs signed by a private CA (which my tool doesn't have), so the handshake fails on SSLSocket.connect(), making SSLSocket.getpeercert() impossible.
I know that the server certificate is indeed being provided to my client (along with that pesky Certificate Request) during the handshake. I can see it in a packet capture, as well as just using the openssl s_client command line.
Here is my code.
def get_cert(self, host, port):
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
with ctx.wrap_socket(socket.socket(), server_hostname=host) as s:
s.settimeout(10)
s.connect((host, port))
binary_cert = s.getpeercert(True)
cert = OpenSSL.crypto.load_certificate(OpenSSL.crypto.FILETYPE_ASN1, binary_cert)
pem_cert = OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYPE_PEM, cert).decode()
return pem_cert
Is there any way to get a little lower into the handshake messages to get the server cert, even though the handshake ultimately fails?
My current solution is to just run openssl s_client -connect host:port using subprocess.run() in the event of a ssl.SSLError.

You may catch exception that do_handshake() produced and then continue to process server certificate.
import OpenSSL
import socket
dst = ('10.10.10.10', 443)
sock = socket.create_connection(dst)
context = OpenSSL.SSL.Context(OpenSSL.SSL.SSLv23_METHOD)
connection = OpenSSL.SSL.Connection(context, sock)
connection.set_connect_state()
try:
connection.do_handshake()
except:
print(connection.get_peer_cert_chain())
Tested on python 2.7.17 and 3.8.5

It looks like there's unfortunately no way to do it with python's ssl module in versions < 3.10. In those versions, the only way to get the peer certificate that I can see is through the low-level _ssl.SSLSocket.getpeercert() method and that immediately throws exception if the handshake is not complete.
Since python 3.10, there's a new _ssl.SSLSocket.get_unverified_chain() method that does not do the handshake check, so perhaps something like this abomination could work?
ssock = context.wrap_socket(sock, do_handshake_on_connect=False)
try:
ssock.do_handshake()
except ssl.SSLError as e:
pass
certs = ssock._sslobj._sslobj.get_unverified_chain()
... but I have not tested it.

Connect to FTP TLS 1.2 Server with ftplib

I try to connect to a FTP Server which only supports TLS 1.2
Using Python 3.4.1
My Code:
import ftplib
import ssl
ftps = ftplib.FTP_TLS()
ftps.ssl_version = ssl.PROTOCOL_TLSv1_2
print (ftps.connect('108.61.166.122',31000))
print(ftps.login('test','test123'))
ftps.prot_p()
print (ftps.retrlines('LIST'))
Error on client side:
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:598)
Error on server side:
Failed TLS negotiation on control channel, disconnected. (SSL_accept():
error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol)
The credentials in the example are working for testing.

See the end of this post for the final solution. The rest are the steps needed to debug the problem.
I try to connect to a FTP Server which only supports TLS 1.2 Using Python 3.4.1
How do you know?
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:598)
I would suggest one of the many SSL problems between client and server, like the server not supporting TLS 1.2, no common ciphers etc. These problems are hard to debug because you either get only some SSL alert or the server will simply close the connection without any obvious reason. If you have access to the server look for error messages on the server side.
You may also try to not to enforce an SSL version but use the default instead, so that client and server will agree to the best SSL version both support. If this will still not work try with a client which is known to work with this server and make a packet capture of the good and bad connections and compare. If you need help with that post the packet captures to cloudshark.org.
Edit#1: just tried it with python 3.4.0 and 3.4.2 against a test server:
python 3.4.0 does a TLS 1.0 handshake, i.e. ignores the setting
python 3.4.2 does a successful TLS 1.2 handshake
In both versions ftplib has the minor bug, that it sends AUTH SSL instead of AUTH TLS if ftps.ssl_version is something else then TLS 1.0, i.e. SSLv3 or TLS1.1.+. While I doubt that this is the origin of the problem it might actually be if the FTP server handles AUTH TLS and AUTH SSL differently.
Edit#2 and Solution:
A packet capture shows that setting ftps.ssl_version has no effect and the SSL handshake will still be done with TLS 1.0 only. Looking at the source code of ftplib in 3.4.0 gives:
ssl_version = ssl.PROTOCOL_TLSv1
def __init__(self, host='', user='', passwd='', acct='', keyfile=None,
certfile=None, context=None,
timeout=_GLOBAL_DEFAULT_TIMEOUT, source_address=None):
....
if context is None:
context = ssl._create_stdlib_context(self.ssl_version,
certfile=certfile,
keyfile=keyfile)
self.context = context
Since __init__ is called when ftplib.FTP_TLS() is called the SSL context will be created with the default ssl_version used by ftplib (ssl.PROTOCOL_TLSv1) and not with your own version. To enforce another SSL version you must to provide your own context with the needed SSL version. The following works for me:
import ftplib
import ssl
ctx = ssl._create_stdlib_context(ssl.PROTOCOL_TLSv1_2)
ftps = ftplib.FTP_TLS(context=ctx)
print (ftps.connect('108.61.166.122',31000))
print(ftps.login('test','test123'))
ftps.prot_p()
print (ftps.retrlines('LIST'))
Alternatively you could set the protocol version globally instead of only for this FTP_TLS object:
ftplib.FTP_TLS.ssl_version = ssl.PROTOCOL_TLSv1_2
ftps = ftplib.FTP_TLS()
And just a small but important observation: it looks like that ftplib does not do any kind of certificate validation, since it accepts this self-signed certificate which does not match the name without complaining. This makes a active man-in-the-middle attack possible. Hopefully they will fix this insecure behavior in the future, in which case the code here will fail because of an invalid certificate.

Firstly AFAIK no ftp supports SSL directly, for which ftps is introduced. Also sftp and ftps are two different concepts: http://en.wikipedia.org/wiki/FTPS .Now, your problem is regarding the programming and not related to SSL or FTPs or any such client-server communication
import ftplib
import ssl
ftps = ftplib.FTP_TLS()
#ftps.ssl_version = ssl.PROTOCOL_TLSv1_2
print (ftps.connect('108.61.166.122',31000))
print(ftps.login('test','test123'))
ftps.prot_p()
print (ftps.retrlines('LIST'))
as ftplib has no attribute PROTOCOL_TLSv1_2 besides which it works fine. and well, your host is not responding !
Hopefully it helps !

How to validate server's ssl certificate in python?

I have configured my server to serve only https creating a self-signed certificate. I have a client that I has to validate the server's certificate and after that will download a file from the server.
How do I implement the validation in client? Is there any code example?
My question is similar with this one: How can the SSL client validate the server's certificate?
but although the fine explanation, I didn't find any help.
So far, in my code I create a directory and then I download the file with urllib2:
[...] #imports
def dir_creation(path):
try:
os.makedirs(path)
except OSError as exception:
if exception.errno != errno.EEXIST:
raise
def file_download(url):
ver_file = urllib2.urlopen(url)
data = ver_file.read()
with open(local_filename, "wb") as code:
code.write(data)
dir_creation(path)
file_download(url)

Rather than configuring your server to present a self-signed certificate, you should use a self-signed certificate as a certificate authority to sign the server certificate. (How to do this is beyond the scope of your question, but I'm sure you can find help on Stack Overflow or elsewhere.)
Now you must configure your client to trust your certificate authority. In python (2.7.9 or later), you can do this using the ssl module:
import ssl
... # create socket
ctx = ssl.create_default_context(cafile=path_to_ca_certificate)
sslsock = ctx.wrap_socket(sock)
You can then transmit and read data on the secure socket. See the ssl module documentation for more explanation.
The urllib2 API is simpler:
import urllib2
resp = urllib2.urlopen(url, cafile=path_to_ca_certificate)
resp_body = resp.read()
If you wish to use Requests, according to the documentation you can supply a path to the CA certificate as the argument to the verify parameter:
resp = requests.get(url, verify=path_to_ca_certificate)

tor via python - connection ok, but no showing up as on tor

i am using the stem example of connection to the tor network, this should connect a client to the tor network, it seems to be doing this but when i check the ip address it is incorrect and not of a tor ip, any ideas as to why this and more importantly how can i fix this issue :)
import StringIO
import socket
import urllib
import socks # SocksiPy module
import stem.process
from stem.util import term
SOCKS_PORT = 7000
# Set socks proxy and wrap the urllib module
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, '127.0.0.1', SOCKS_PORT)
socket.socket = socks.socksocket
# Perform DNS resolution through the socket
def getaddrinfo(*args):
return [(socket.AF_INET, socket.SOCK_STREAM, 6, '', (args[0], args[1]))]
socket.getaddrinfo = getaddrinfo
def query(url):
"""
Uses urllib to fetch a site using SocksiPy for Tor over the SOCKS_PORT.
"""
try:
return urllib.urlopen(url).read()
except:
return "Unable to reach %s" % url
# Start an instance of Tor configured to only exit through Russia. This prints
# Tor's bootstrap information as it starts. Note that this likely will not
# work if you have another Tor instance running.
def print_bootstrap_lines(line):
if "Bootstrapped " in line:
print term.format(line, term.Color.BLUE)
print term.format("Starting Tor:\n", term.Attr.BOLD)
tor_process = stem.process.launch_tor_with_config(
config = {
'SocksPort': str(SOCKS_PORT),
'ExitNodes': '{ru}',
},
init_msg_handler = print_bootstrap_lines,
)
I get the output :
richard#Tornado:~/Documents/Masters Project$ python russiaExample.py
Starting Tor:
May 26 21:56:49.000 [notice] Bootstrapped 80%: Connecting to the Tor network.
May 26 21:56:50.000 [notice] Bootstrapped 85%: Finishing handshake with first hop.
May 26 21:56:50.000 [notice] Bootstrapped 90%: Establishing a Tor circuit.
May 26 21:56:50.000 [notice] Bootstrapped 100%: Done.
however when i visit https://check.torproject.org/ to check i am using tor, it says i am now, and my normal ip is shown,
what is causing this issue, as the output shown above seems to suggest it has established a circuit all ok to Tor, but seems as although it is not using it ?
i am on the right lines here ?
Thanks guys

You have to set up your browser to use Tor as a proxy.
If you are using Firefox:
Go to Edit, preference, advanced and choose "configure how firefox connects to the internet" settings.
In socks host enter 127.0.0.1 and under port enter 7000.
Go to whatismyip.com and you will see a new ip.
Or check tor,project to see you are using Tor successfully.

Sending SSL data over a TCP proxy connection in Python

I am facing the following scenario:
I am forced to use an HTTP proxy to connect to an HTTPS server. For several reasons I need access to the raw data (before encryption) so I am using the socket library instead of one of the HTTP specific libraries.
I thus first connect a TCP socket to the HTTP proxy and issue the connect command.
At this point, the HTTP proxy accepts the connection and seemingly forwards all further data to the target server.
However, if I now try to switch to SSL, I receive
error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
indicating that the socket attempted the handshake with the HTTP proxy and not with the HTTPS target.
Here's the code I have so far:
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('proxy',9502))
s.send("""CONNECT en.wikipedia.org:443 HTTP/1.1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:15.0) Gecko/20100101 Firefox/15.0.1
Proxy-Connection: keep-alive
Host: en.wikipedia.org
""")
print s.recv(1000)
ssl = socket.ssl(s, None, None)
ssl.connect(("en.wikipedia.org",443))
What would be the correct way to open an SSL socket to the target server after connecting to the HTTP proxy?

(Note that in generally, it would be easier to use an existing HTTPS library such as PyCurl, instead of implementing it all by yourself.)
Firstly, don't call your variable ssl. This name is already used by the ssl module, so you don't want to hide it.
Secondly, don't use connect a second time. You're already connected, what you need is to wrap the socket. Since Python doesn't do any certificate verification by default, you'll need to verify the remote certificate and verify the host name too.
Here are the steps involved:
Establish your plain-text connection and use CONNECT like you're doing in the first few lines.
Read the HTTP response you get, and make sure you get a 200 status code. (You'll need to read the header line by line).
Use ssl_s = ssl.wrap_socket(s, cert_reqs=ssl.CERT_REQUIRED, ssl_version=ssl.PROTOCOL_TLS1, ca_certs='/path/to/cabundle.pem') to wrap the socket. Then, verify the host name. It's worth reading this answer: the connect method and what it does after wrapping the socket.
Then, use ssl_s as if it was your normal socket. Don't call connect again.

works with python 3
< proxy > is an ip or domain name
< port > 443 or 80 or whatever your proxy is listening to
< endpoint > your final server you want to connect to via the proxy
< cn > is an optional sni field your final server could be expecting
import socket,ssl
def getcert_sni_proxy(cn,endpoint,PROXY_ADDR=("<proxy>", <port>)):
#prepare the connect phrase
CONNECT = "CONNECT %s:%s HTTP/1.0\r\nConnection: close\r\n\r\n" % (endpoint, 443)
#connect to the actual proxy
conn = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
conn.connect(PROXY_ADDR)
conn.send(str.encode(CONNECT))
conn.recv(4096)
#set the cipher for the ssl layer
context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
#connect to the final endpoint via the proxy, sending an optional servername information [cn here]
sock = context.wrap_socket(conn, server_hostname=cn)
#retreive certificate from the server
certificate = ssl.DER_cert_to_PEM_cert(sock.getpeercert(True))
return certificate