Develop Reference

How to receive data from a raw socket in Python?

I am trying to create a port scanner (using SYN packets) with the sockets library (yes I know scapy would make this much easier, but I'm mostly doing this for a learning exercise.) I have crafted the packet and successfully sent it, however I'm having troubled receiving and parsing the subsequent response.
So far I've tried the s.recv(1024) and 4096, as well as recvfrom().
s = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_RAW)
s.sendto(packet, (dstip, 80))
r = s.recv(1024)
print(r)
However, I am having trouble receiving the response, I can see that the packet is being sent correctly via Wireshark, and the SYN-ACK is sent to my machine, however I am unable to properly receive and print it. Is there a better way I can use the s.recv() function for this sort of input? Or am I using the wrong function?
Any help is appreciated, I'm new to the sockets library. Thanks.

The book Black Hat Python has en example using the socket library to create a scanner, unfortunately not a port scanner. They check if a host is up, and they use a raw socket to receive data. The code is available here.
They are sending SYN-packets with one socket object in a new thread, and sniffing the replies using another socket object.
In the example they use socket.IPPROTO_IP or socket.IPPROTO_ICMP instead of socket.IPPROTO_RAW depending on if it is Windows or not.
For the sniffer they use the function setsockopt(socket.IPPROTO_IP, socket.IP_HDRINCL, 1) for sniffing, where IPPROTO_IP is a dummy-protocol for TCP, IP_HDRINCL is to include headers in the IP packets, and 1 is mapped to the ICMP-protocol in the code.
Good luck!

Below is a recent module I wrote with the help from various sources for socket IO, take what you would like from it.
import socket
import threading
import time
import pygogo as gogo
from icentralsimulator.bridgeio.read_packets import PacketFactory
from icentralsimulator.bridgeio.write_packets import WritePacket
from icentralsimulator.configurations.interfaces import IServerInfoProvider
logger = gogo.Gogo(__name__).logger
send_lock = threading.Lock()
class BridgeConnection:
def __init__(self, bridge_info_provider: IServerInfoProvider):
info = bridge_info_provider.get_bridge_server_info()
self.callback = None
self.bridge_ip = info.IpAddress
self.bridge_port = info.Port
self._connection = None
self._terminate_wait_for_incoming = False
#property
def is_connected(self):
return self._connection is not None
def connect(self, callback):
"""
The purpose of this method is to create (and hold) a connection to the server. At the same time,
it creates a new thread for the purpose of waiting on incoming packets.
"""
if self._connection is not None: return
self._connection = socket.create_connection((self.bridge_ip, self.bridge_port))
self._connection.settimeout(0.5)
self.callback = callback
t = threading.Thread(target=self._wait_for_incoming)
t.start()
time.sleep(5)
def disconnect(self):
"""
Breaks existing connection to the server if one is currently made and cancels the thread that is waiting
for incoming packets. If the connection is not currently open, simply returns silently -- thus it is safe
to call this method repeatedly.
"""
self._terminate_wait_for_incoming = True
while self._terminate_wait_for_incoming:
time.sleep(0.1)
self._connection.close()
self._connection = None
def send_packet(self, packet: WritePacket):
"""
Sends an arbitrary packet to the server.
"""
with send_lock:
logger.debug(f"Sending packet: {packet.payload_plain_text}")
payload = packet.payload
self._connection.sendall(payload)
def _wait_for_incoming(self):
"""
Continually runs a loop to wait for incoming data on the open socket. If data is received, it is converted
to a receive packet and forwarded to the consumer as part of a callback.
"""
self._terminate_wait_for_incoming = False
buf_len = 4096
try:
while not self._terminate_wait_for_incoming:
data = None
try:
_cnx = self._connection
if _cnx is None: break
data = _cnx.recv(buf_len)
if data is not None and len(data) > 0:
while True:
new_data = _cnx.recv(buf_len)
if new_data is None or len(new_data) == 0:
break
data = data + new_data
except socket.timeout:
if data is not None and self.callback is not None:
packet = PacketFactory.get_packet(data)
self.callback(packet)
logger.debug(f"Received packet: {data}")
time.sleep(0.5)
except OSError: # Happens when stopping the application
logger.info("Application aborted")
return
finally:
self._terminate_wait_for_incoming = False
Note that I don't include IServerInfoProvider, or the PacketFactory here. Those are pretty custom to my application. You will need to interpret the packet according to the packet data that arrives in your specific use case.

How to handle ssl connections in raw Python socket?

I'm writing a program to download a given webpage. I need to only use raw python sockets for all the connection due to some restriction. So I make a socket connection to a given domain (the Host field in the response header of an object) and then send the GET request on this. Now when the url is a https url, I think I need to first do the SSL handshake (because otherwise I'm getting non-200 OK responses from the server and other error responses mentioning P3P policies). I inspected curl's response to check how it's able to successfully download while I'm not, turns out curl first does the SSL handshake (that's all the difference). curl is always able to successfully download a given object, the only difference always being the SSL handshake it does.
So I'm wondering how to do the SSL handshake in raw python sockets? Basically I want as easy a solution which allows me to do the minimum besides using raw sockets.

Here is an example of a TCP client with SLL.
Not sure if it's the best way to download a web page but it should answer your question "SSL handshake in raw python socket".
You will probably have to adapt the struct.pack/unpack but you get the general idea:
import socket
import ssl
import struct
import binascii
import sys
class NotConnectedException(Exception):
def __init__(self, message=None, node=None):
self.message = message
self.node = node
class DisconnectedException(Exception):
def __init__(self, message=None, node=None):
self.message = message
self.node = node
class Connector:
def __init__(self):
pass
def is_connected(self):
return (self.sock and self.ssl_sock)
def open(self, hostname, port, cacert):
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.ssl_sock = ssl.wrap_socket(self.sock, ca_certs=cacert, cert_reqs=ssl.CERT_REQUIRED)
if hostname == socket.gethostname():
ipaddress = socket.gethostbyname_ex(hostname)[2][0]
self.ssl_sock.connect((ipaddress, port))
else:
self.ssl_sock.connect((hostname, port))
self.sock.setsockopt(socket.SOL_TCP, socket.TCP_NODELAY, 1)
def close(self):
if self.sock: self.sock.close()
self.sock = None
self.ssl_sock = None
def send(self, buffer):
if not self.ssl_sock: raise NotConnectedException("Not connected (SSL Socket is null)")
self.ssl_sock.sendall(struct.pack('L', len(buffer)))
self.ssl_sock.sendall(buffer)
def receive(self):
if not self.ssl_sock: raise NotConnectedException("Not connected (SSL Socket is null)")
data_size_buffer = self.ssl_sock.recv(4)
if len(data_size_buffer) <= 0:
raise DisconnectedException()
data_size = struct.unpack('L', data_size_buffer)[0]
received_size = 0
data_buffer = ""
while received_size < data_size:
chunk = self.ssl_sock.recv(1024)
data_buffer += chunk
received_size += len(chunk)
return data_buffer
Then you use the class like this:
connector = Connector.Connector()
connector.open(server_ip, server_port, path_to_the_CA_cert.pem)
connector.send(your_data)
response = connector.receive()
connector.close()

You can use the wrap_socket method of the python ssl module to turn your socket into one that talks SSL. Once you've done this you can use it like normal, but internally the data will be encrypted and decrypted for you. These are the docs for the method:
https://docs.python.org/2/library/ssl.html#ssl.wrap_socket

I think the easier way to do that would be using SSL contexts and wraping the TCP socket.
Python SSL module's documentation give a very thoroughful explanation with examples. I recommend you to read the relevant sections of Python 2 or Python 3 ssl module documentation. It should be very easy to achieve what you want.
Hope this helps!

got stuck programming https proxy - what to do next?

I'm currently making a proxy which sits between the browser and the web. Everything works except https. I'm having troubles understanding some passages of it and haven't found many resources on the web. And so I'm stuck.
The code I'm using is:
conn, addr = server.accept()
request = conn.recv(9999) #get a CONNECT request
conn.send(b'HTTP/1.1 200 Connection estabilished\n\n')
enc_req = conn.recv(9999) #this gets an encrypted request
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM) #plaintext client
client.connect((host, 443)) #connect to chosen host
client.send(enc_req)
resp1 = client.recv(9999) #this gets something unreadable (encrypted?)
#could it be the certificate?
#now what?
Is the resp1 I'm getting the certificate? And what do I need to do after that? (Or, which is the same, what does usually happens next with https?)
P.S. I know the question is somewhat generic, but please don't judge me too harshly. I've tried researching on the web but all I keep finding is the encryption method used for ssl. I really don't know how to proceed.

I haven't tested this code (and it's mainly pseudo code), but this should give you an idea of what you need to do.
conn, addr = server.accept()
request = conn.recv(9999) #get a CONNECT request
# Here, parse the CONNECT string and get the host and port (not sure if you were doing that already.
# Then, try to connect *before* you tell the client the connection was established (in case it fails)
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM) #plaintext client
client.connect((host, 443)) #connect to chosen host
conn.send(b'HTTP/1.1 200 Connection estabilished\n\n')
# Then loop until the connections are closed.
while True:
# Read from the client, send the data to the server.
enc_req = conn.recv(9999) #this gets an encrypted request
client.send(enc_req)
# Read from the server, send the data to the client.
resp1 = client.recv(9999) #this gets something unreadable (encrypted?)
#could it be the certificate?
#now what?
# The first time it's certainly the Client Hello message, not encrypted, but in a binary format indeed.
# Just send everything you've just read to the server.
conn.send(resp1)
This is just a quick overview of the idea of the loop you need to write. In reality, you may be able to process both in parallel. You'd also want to be a bit more careful when closing the connection (allowing it to happen in any order while still relaying the last data sent by either party).

As mentioned in the comments, a proxy handling encrypted end-to-end traffic can only pass it on.
Here is a fully working proxy written using circuits that has been fully tested with passing and proxying SSH traffic so it should work equally as well as a pass-through TCP proxy even if SSL is involved:
#!/usr/bin/env python
from uuid import uuid4 as uuid
from circuits import Component
from circuits.net.events import close, connect, write
from circuits.net.sockets import TCPClient, TCPServer
class Client(Component):
channel = "client"
def init(self, sock, host, port, channel=channel):
self.sock = sock
self.host = host
self.port = port
TCPClient(channel=self.channel).register(self)
def ready(self, *args):
self.fire(connect(self.host, self.port))
def disconnect(self, *args):
self.fire(close(self.sock), self.parent.channel)
def read(self, data):
self.fire(write(self.sock, data), self.parent.channel)
class Proxy(Component):
channel = "server"
def init(self, bind, host, port):
self.bind = bind
self.host = host
self.port = port
self.clients = dict()
TCPServer(self.bind).register(self)
def connect(self, sock, host, port):
channel = uuid()
client = Client(
sock, self.host, self.port, channel=channel
).register(self)
self.clients[sock] = client
def disconnect(self, sock):
client = self.clients.get(sock)
if client is not None:
client.unregister()
del self.clients[sock]
def read(self, sock, data):
client = self.clients[sock]
self.fire(write(data), client.channel)
app = Proxy(("0.0.0.0", 3333), "127.0.0.1", 22)
from circuits import Debugger
Debugger().register(app)
app.run()

python asynchat state

I've just started working with the basics of python socket networking. As an exercise in understanding, I've been trying to hash out a basic server that will ask it's client for a file type, and upon receiving a string of the extension, ask for the actual file. I've found numerous tutorials online that use the asyncore library, specifically asynchat to setup this kind of call and response functionality.
The most basic one I've been following can be found here (I've copied it)
http://effbot.org/librarybook/asynchat.htm
import asyncore, asynchat
import os, socket, string
PORT = 8000
class HTTPChannel(asynchat.async_chat):
def __init__(self, server, sock, addr):
asynchat.async_chat.__init__(self, sock)
self.set_terminator("\r\n")
self.request = None
self.data = ""
self.shutdown = 0
def collect_incoming_data(self, data):
self.data = self.data + data
def found_terminator(self):
if not self.request:
# got the request line
self.request = string.split(self.data, None, 2)
if len(self.request) != 3:
self.shutdown = 1
else:
self.push("HTTP/1.0 200 OK\r\n")
self.push("Content-type: text/html\r\n")
self.push("\r\n")
self.data = self.data + "\r\n"
self.set_terminator("\r\n\r\n") # look for end of headers
else:
# return payload.
self.push("<html><body><pre>\r\n")
self.push(self.data)
self.push("</pre></body></html>\r\n")
self.close_when_done()
class HTTPServer(asyncore.dispatcher):
def __init__(self, port):
asyncore.dispatcher.__init__(self)
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
self.bind(("", port))
self.listen(5)
def handle_accept(self):
conn, addr = self.accept()
HTTPChannel(self, conn, addr)
#
# try it out
s = HTTPServer(PORT)
print "serving at port", PORT, "..."
My question has to do with the handle_accept method of the HTTPServer class. If every time a request comes in, the HTTPChannel object is initialized, wouldn't it be impossible in this kind of setup to create a call and response? I was thinking one could set flags for _hastype and _hasfile in the channel object, but since the accept inits it for each individual connection, the object's state is forgotten with every inidividual request. I realize this setup is supposed to be a basic HTTPServer, but my question is, how could I edit it to setup something like what I've described? Would the server object need to inherit asynchat itself and forego dispatcher completely? The channel object would have to have some state to know that the filetype has already been sent, and then ask for the binary of the file instead. I'm very curious to know what the cleanest possible implementation of this might look like.
Thanks a ton - I'm very new to sockets. Please let me know if I haven't been clear.

Normally the connection would be kept open after it's initially created, so all the parts of the communication from the same client go to the same HTTPChannel object - accept is only called when a new connection is created.

How do I use TLS with asyncore?

An asyncore-based XMPP client opens a normal TCP connection to an XMPP server. The server indicates it requires an encrypted connection. The client is now expected to start a TLS handshake so that subsequent requests can be encrypted.
tlslite integrates with asyncore, but the sample code is for a server (?) and I don't understand what it's doing.
I'm on Python 2.5. How can I get the TLS magic working?
Here's what ended up working for me:
from tlslite.api import *
def handshakeTls(self):
"""
Encrypt the socket using the tlslite module
"""
self.logger.info("activating TLS encrpytion")
self.socket = TLSConnection(self.socket)
self.socket.handshakeClientCert()

Definitely check out twisted and wokkel. I've been building tons of xmpp bots and components with it and it's a dream.

I've followed what I believe are all the steps tlslite documents to make an asyncore client work -- I can't actually get it to work since the only asyncore client I have at hand to tweak for the purpose is the example in the Python docs, which is an HTTP 1.0 client, and I believe that because of this I'm trying to set up an HTTPS connection in a very half-baked way. And I have no asyncore XMPP client, nor any XMPP server requesting TLS, to get anywhere close to your situation. Nevertheless I decided to share the fruits of my work anyway because (even though some step may be missing) it does seem to be a bit better than what you previously had -- I think I'm showing all the needed steps in the __init__. BTW, I copied the pem files from the tlslite/test directory.
import asyncore, socket
from tlslite.api import *
s = open("./clientX509Cert.pem").read()
x509 = X509()
x509.parse(s)
certChain = X509CertChain([x509])
s = open("./clientX509Key.pem").read()
privateKey = parsePEMKey(s, private=True)
class http_client(TLSAsyncDispatcherMixIn, asyncore.dispatcher):
ac_in_buffer_size = 16384
def __init__(self, host, path):
asyncore.dispatcher.__init__(self)
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
self.connect( (host, 80) )
TLSAsyncDispatcherMixIn.__init__(self, self.socket)
self.tlsConnection.ignoreAbruptClose = True
handshaker = self.tlsConnection.handshakeClientCert(
certChain=certChain,
privateKey=privateKey,
async=True)
self.setHandshakeOp(handshaker)
self.buffer = 'GET %s HTTP/1.0\r\n\r\n' % path
def handle_connect(self):
pass
def handle_close(self):
self.close()
def handle_read(self):
print self.recv(8192)
def writable(self):
return (len(self.buffer) > 0)
def handle_write(self):
sent = self.send(self.buffer)
self.buffer = self.buffer[sent:]
c = http_client('www.readyhosting.com', '/')
asyncore.loop()
This is a mix of the asyncore example http client in the Python docs, plus what I've gleaned from the tlslite docs and have been able to reverse engineer from their sources. Hope this (even though incomplete/not working) can at least advance you in your quest...
Personally, in your shoes, I'd consider switching from asyncore to twisted -- asyncore is old and rusty, Twisted already integrates a lot of juicy, useful bits (the URL I gave is to a bit in the docs that already does integrate TLS and XMPP for you...).