Develop Reference

How do I connect to a website port using python sockets

I'm coding in Python and I'm looking for a way to connect to a website port using sockets so that I can send commands to the server. My code is:
import socket
HOST = 'www.google.com'
PORT = 80
server=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind((HOST, PORT))
server.listen(5)
This code is giving me an error "The requested address is not valid in its context". How do I do this?

You're trying to bind on Google's IP, which doesn't make sense because there isn't a network adapter connected to your computer with that IP (thus the error). You're mixing up creating a server and being a client connecting to a remote server. You want to connect to the Google server:
import socket
HOST = 'www.google.com'
PORT = 80
socket = socket.socket()
socket.connect((HOST, PORT))
# Send an HTTP GET request to request the page
socket.send(b"""
GET / HTTP/1.1
Host: www.google.com
""")
msg = socket.recv(8192)
print(msg)

What details about the host machine does python send to the web server while establishing a connection through sockets?

I am building a passive reconnaissance tool and one of it functionality is to get certificate info about a domain.
What details of my machine are being to sent to the webserver?
The below code is used to get the certificate info.
from OpenSSL import SSL
from cryptography import x509
from cryptography.x509.oid import NameOID
import idna
from socket import socket
from collections import namedtuple
HostInfo = namedtuple(field_names='cert hostname peername', typename='HostInfo')
HOSTS = [
('google.com', 443),
('yahoo.com', 443),
('yahoo.com', 443),
]
def get_certificate(hostname, port):
hostname_idna = idna.encode(hostname)
sock = socket()
sock.connect((hostname, port))
peername = sock.getpeername()
ctx = SSL.Context(SSL.SSLv23_METHOD) # most compatible
ctx.check_hostname = False
ctx.verify_mode = SSL.VERIFY_NONE
sock_ssl = SSL.Connection(ctx, sock)
sock_ssl.set_connect_state()
sock_ssl.set_tlsext_host_name(hostname_idna)
sock_ssl.do_handshake()
cert = sock_ssl.get_peer_certificate()
crypto_cert = cert.to_cryptography()
sock_ssl.close()
sock.close()
return HostInfo(cert=crypto_cert, peername=peername, hostname=hostname)
import concurrent.futures
if __name__ == '__main__':
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as e:
for hostinfo in e.map(lambda x: get_certificate(x[0], x[1]), HOSTS):
print_basic_info(hostinfo)
Is my IP address being sent?
Is python sending any user-agent and what other details are being sent while establishing a connection?

While Python doesn't explicitly send your IP, it is always sent with any TCP request. If it isn't sent, you cannot receive the response from the server.
According to http://evanhahn.com/python-requests-library-useragent, the useragent is
python-requests/{package version} {runtime}/{runtime version} {uname}/{uname -r} when using python-requests, but you can override it by setting the request header.
However, you are not using python-requests but raw sockets, so no data is transferred except what you tell it to (and your IP and some metadata for the TCP connection)

How can a Python proxy server (using SSL socket) pretend to be an HTTPS server and specify my own keys to get decrypted data?

I'm trying to handle HTTPS connections from web browsers in decrypted data using Python SSL socket. I thought I can do this like a man-in-the-middle attack via my Python proxy server program, by pretending to be an HTTPS server, and I've already generated my keys using OpenSSL. However, browser treats my program as a proxy server, not as an HTTPS server. How can my program pretend to be an HTTPS server and specify my own keys to communicate with the browser?
I'm using Chrome and a Chrome extension SwitchySharp to forward requests to another local port (HTTP to 127.0.0.1:8080, and HTTPS to 127.0.0.1:8081). It works well for HTTP requests, but doesn't work for HTTPS.
Background Knowledge
When handling HTTPS with a proxy server, the browser first sends a CONNECT request to proxy:
CONNECT xxx.xxx:443 HTTP/1.1
Host: xxx.xxx:443
Proxy-Connection: keep-alive
(some other headers)
and proxy creates a tunnel to target server, and then browser sends encrypted data via this tunnel. Therefore, the proxy server never gets decrypted data.
To get decrypted data, I've tried two ways.
First, I tried ssl.wrap_socket():
import socket
import ssl
import thread
def handle(conn):
request = conn.recv(4096)
print request
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(('127.0.0.1', 8081))
sock.listen(10)
sock = ssl.wrap_socket(sock, 'pkey.pem', 'cert.pem', True)
while True:
conn, addr = sock.accept()
thread.start_new_thread(handle, (conn,))
But I got the following error:
Traceback (most recent call last):
File "https.py", line 12, in <module>
conn, addr = sock.accept()
File "C:\Python27\lib\ssl.py", line 898, in accept
server_side=True)
File "C:\Python27\lib\ssl.py", line 369, in wrap_socket
_context=self)
File "C:\Python27\lib\ssl.py", line 617, in __init__
self.do_handshake()
File "C:\Python27\lib\ssl.py", line 846, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: HTTPS_PROXY_REQUEST] https proxy request (_ssl.c:726)
The cause of this error is that the CONNECT request is not encrypted so ssl can't handle it. I have no idea how to get rid of this error.
Then, I tried the following code:
import socket
import thread
def handler(conn, tunn):
while True:
data = conn.recv(40960)
if data:
tunn.sendall(data)
else:
break
def handle(conn):
request = conn.recv(40960)
print request
i = request.find(' ') + 1
j = request.find(' ', i)
host, port = request[i : j].split(':')
port = int(port)
tunn = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
tunn.connect((host, port))
conn.sendall('HTTP/1.1 200 Connection Established\r\n\r\n')
thread.start_new_thread(handler, (conn, tunn))
thread.start_new_thread(handler, (tunn, conn))
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(('127.0.0.1', 8081))
sock.listen(10)
while True:
conn, addr = sock.accept()
thread.start_new_thread(handle, (conn,))
It worked (without ssl module), but I can't specify my own keys and therefore can't get decrypted data.
It seems that my program has to pretend to be an HTTPS server to "directly" communicate with browser, so I can specify my own key and get decrypted data. But I really have no idea how to pretend to be an HTTPS server and specify my own keys.
Can anyone help me? Thanks in advance.
UPD: I finally solved this problem using ssl module.
I was wrong --- I thought that after connection established the browser sends encrypted data. But it doesn't. In fact, those unreadable data are just SSL handshake. So what I need to do is just to use ssl.wrap_socket() after connection established. Besides, I have to additionally call do_handshake() of the ssl-wrapped socket.
import socket
import ssl
import thread
def handle(tid, conn):
request = conn.recv(40960)
i = request.find(' ') + 1
j = request.find(' ', i)
if request[i : j].find(':') != -1:
host, port = request[i : j].split(':')
else:
host, port = request[i : j], 80
port = int(port)
tunn = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
tunn.connect((host, port))
conn.sendall('HTTP/1.1 200 Connection Established\r\n\r\n')
conn_s = ssl.wrap_socket(conn, keyfile = 'cert/userkey.pem', certfile = 'cert/usercert.pem', server_side = True, do_handshake_on_connect = False)
conn_s.do_handshake()
data = conn_s.recv(40960)
print data
conn_s.sendall('HTTP/1.1 200 OK\r\n\r\n<h1>Hello, world!</h1>')
conn_s.close()
tunn.close()
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(('127.0.0.1', 8081))
sock.listen(10)
buf = {}
while True:
conn, addr = sock.accept()
thread.start_new_thread(handle, (conn,))
Anyway, it doesn't pretend to be an HTTPS server -- it still behaves as a proxy server. I just misunderstood those unreadable data.
Thanks to those who helped me in the comments.

Proxy not working over SSL connection

I'm trying to use tor, socksipy and ssl to proxy a ssl connection. My client looks like this:
import socks, ssl
s = socks.socksocket()
s.setproxy(socks.PROXY_TYPE_SOCKS5,"127.0.0.1", 9050)
ssl_sock = ssl.wrap_socket(s, ssl_version=ssl.PROTOCOL_TLSv1)
ssl_sock.connect(('127.0.0.1', 443))
The server just accepts connections and prints getpeername.
The peer name is always 127.0.0.1. It doesn't even matter if I give it a non-valid proxy. The client won't complain, it will connect anyway.
How do I make it connect through the proxy?

I managed to figure it out so I will leave the answer here for future reference.
The first problem was that I tried to connect to 127.0.0.1. As the request was proxied, the proxy would try to connect to 127.0.0.1, so it would try to connect to itself, not to me.
I had to configure my router to forward requests on port 443 to my laptop and then I replaced 127.0.0.1 with my routers IP.
After that was out of the way, I found out that socksipy doesn't play very well with ssl.
I had to call connect on the socket before wrapping it, otherwise I'd get a handshake failure. The code became:
import socks, ssl
s = socks.socksocket()
s.setproxy(socks.PROXY_TYPE_SOCKS5,"127.0.0.1", 9050)
s.connect(('127.0.0.1', 443))
ssl_sock = ssl.wrap_socket(s, ssl_version=ssl.PROTOCOL_TLSv1)
After that, everything was working fine.

HTTPS proxy tunneling with the ssl module

I'd like to manually (using the socket and ssl modules) make an HTTPS request through a proxy which itself uses HTTPS.
I can perform the initial CONNECT exchange just fine:
import ssl, socket
PROXY_ADDR = ("proxy-addr", 443)
CONNECT = "CONNECT example.com:443 HTTP/1.1\r\n\r\n"
sock = socket.create_connection(PROXY_ADDR)
sock = ssl.wrap_socket(sock)
sock.sendall(CONNECT)
s = ""
while s[-4:] != "\r\n\r\n":
s += sock.recv(1)
print repr(s)
The above code prints HTTP/1.1 200 Connection established plus some headers, which is what I expect. So now I should be ready to make the request, e.g.
sock.sendall("GET / HTTP/1.1\r\n\r\n")
but the above code returns
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
Reason: You're speaking plain HTTP to an SSL-enabled server port.<br />
Instead use the HTTPS scheme to access this URL, please.<br />
</body></html>
This makes sense too, since I still need to do an SSL handshake with the example.com server to which I'm tunneling. However, if instead of immediately sending the GET request I say
sock = ssl.wrap_socket(sock)
to do the handshake with the remote server, then I get an exception:
Traceback (most recent call last):
File "so_test.py", line 18, in <module>
ssl.wrap_socket(sock)
File "/usr/lib/python2.6/ssl.py", line 350, in wrap_socket
suppress_ragged_eofs=suppress_ragged_eofs)
File "/usr/lib/python2.6/ssl.py", line 118, in __init__
self.do_handshake()
File "/usr/lib/python2.6/ssl.py", line 293, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [Errno 1] _ssl.c:480: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
So how can I do the SSL handshake with the remote example.com server?
EDIT: I'm pretty sure that no additional data is available before my second call to wrap_socket because calling sock.recv(1) blocks indefinitely.

This should work if the CONNECT string is rewritten as follows:
CONNECT = "CONNECT %s:%s HTTP/1.0\r\nConnection: close\r\n\r\n" % (server, port)
Not sure why this works, but maybe it has something to do with the proxy I'm using. Here's an example code:
from OpenSSL import SSL
import socket
def verify_cb(conn, cert, errun, depth, ok):
return True
server = 'mail.google.com'
port = 443
PROXY_ADDR = ("proxy.example.com", 3128)
CONNECT = "CONNECT %s:%s HTTP/1.0\r\nConnection: close\r\n\r\n" % (server, port)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(PROXY_ADDR)
s.send(CONNECT)
print s.recv(4096)
ctx = SSL.Context(SSL.SSLv23_METHOD)
ctx.set_verify(SSL.VERIFY_PEER, verify_cb)
ss = SSL.Connection(ctx, s)
ss.set_connect_state()
ss.do_handshake()
cert = ss.get_peer_certificate()
print cert.get_subject()
ss.shutdown()
ss.close()
Note how the socket is first opened and then open socket placed in SSL context. Then I manually initialize SSL handshake. And output:
HTTP/1.1 200 Connection established
<X509Name object '/C=US/ST=California/L=Mountain View/O=Google Inc/CN=mail.google.com'>
It's based on pyOpenSSL because I needed to fetch invalid certificates too and Python built-in ssl module will always try to verify the certificate if it's received.

Judging from the API of the OpenSSL and GnuTLS library, stacking a SSLSocket onto a SSLSocket is actually not straightforwardly possible as they provide special read/write functions to implement the encryption, which they are not able to use themselves when wrapping a pre-existing SSLSocket.
The error is therefore caused by the inner SSLSocket directly reading from the system socket and not from the outer SSLSocket. This ends in sending data not belonging to the outer SSL session, which ends badly and for sure never returns a valid ServerHello.
Concluding from that, I would say there is no simple way to implement what you (and actually myself) would like to accomplish.

Finally I got somewhere expanding on #kravietz and #02strich answers.
Here's the code
import threading
import select
import socket
import ssl
server = 'mail.google.com'
port = 443
PROXY = ("localhost", 4433)
CONNECT = "CONNECT %s:%s HTTP/1.0\r\nConnection: close\r\n\r\n" % (server, port)
class ForwardedSocket(threading.Thread):
def __init__(self, s, **kwargs):
threading.Thread.__init__(self)
self.dest = s
self.oursraw, self.theirsraw = socket.socketpair(socket.AF_UNIX, socket.SOCK_STREAM)
self.theirs = socket.socket(_sock=self.theirsraw)
self.start()
self.ours = ssl.wrap_socket(socket.socket(_sock=self.oursraw), **kwargs)
def run(self):
rl, wl, xl = select.select([self.dest, self.theirs], [], [], 1)
print rl, wl, xl
# FIXME write may block
if self.theirs in rl:
self.dest.send(self.theirs.recv(4096))
if self.dest in rl:
self.theirs.send(self.dest.recv(4096))
def recv(self, *args):
return self.ours.recv(*args)
def send(self, *args):
return self.outs.recv(*args)
def test():
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(PROXY)
s = ssl.wrap_socket(s, ciphers="ALL:aNULL:eNULL")
s.send(CONNECT)
resp = s.read(4096)
print (resp, )
fs = ForwardedSocket(s, ciphers="ALL:aNULL:eNULL")
fs.send("foobar")
Don't mind custom cihpers=, that only because I didn't want to deal with certificates.
And there's depth-1 ssl output, showing CONNECT, my response to it ssagd and depth-2 ssl negotiation and binary rubbish:
[dima#bmg ~]$ openssl s_server -nocert -cipher "ALL:aNULL:eNULL"
Using default temp DH parameters
Using default temp ECDH parameters
ACCEPT
-----BEGIN SSL SESSION PARAMETERS-----
MHUCAQECAgMDBALAGQQgmn6XfJt8ru+edj6BXljltJf43Sz6AmacYM/dSmrhgl4E
MOztEauhPoixCwS84DL29MD/OxuxuvG5tnkN59ikoqtfrnCKsk8Y9JtUU9zuaDFV
ZaEGAgRSnJ81ogQCAgEspAYEBAEAAAA=
-----END SSL SESSION PARAMETERS-----
Shared ciphers: [snipped]
CIPHER is AECDH-AES256-SHA
Secure Renegotiation IS supported
CONNECT mail.google.com:443 HTTP/1.0
Connection: close
sagq
�u\�0�,�(�$��
�"�!��kj98���� �m:��2�.�*�&���=5�����
��/�+�'�#�� ����g#32��ED���l4�F�1�-�)�%���</�A������
�� ������
�;��A��q�J&O��y�l

It doesn't sound like there's anything wrong with what you're doing; it's certainly possible to call wrap_socket() on an existing SSLSocket.
The 'unknown protocol' error can occur (amongst other reasons) if there's extra data waiting to be read on the socket at the point you call wrap_socket(), for instance an extra \r\n or an HTTP error (due to a missing cert on the server end, for instance). Are you certain you've read everything available at that point?
If you can force the first SSL channel to use a "plain" RSA cipher (i.e. non-Diffie-Hellman) then you may be able to use Wireshark to decrypt the stream to see what's going on.

Building on #kravietz answer. Here is a version that works in Python3 through a Squid proxy:
from OpenSSL import SSL
import socket
def verify_cb(conn, cert, errun, depth, ok):
return True
server = 'mail.google.com'
port = 443
PROXY_ADDR = ("<proxy_server>", 3128)
CONNECT = "CONNECT %s:%s HTTP/1.0\r\nConnection: close\r\n\r\n" % (server, port)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(PROXY_ADDR)
s.send(str.encode(CONNECT))
s.recv(4096)
ctx = SSL.Context(SSL.SSLv23_METHOD)
ctx.set_verify(SSL.VERIFY_PEER, verify_cb)
ss = SSL.Connection(ctx, s)
ss.set_connect_state()
ss.do_handshake()
cert = ss.get_peer_certificate()
print(cert.get_subject())
ss.shutdown()
ss.close()
This works in Python 2 also.