Develop Reference

How to receive data from a raw socket in Python?

I am trying to create a port scanner (using SYN packets) with the sockets library (yes I know scapy would make this much easier, but I'm mostly doing this for a learning exercise.) I have crafted the packet and successfully sent it, however I'm having troubled receiving and parsing the subsequent response.
So far I've tried the s.recv(1024) and 4096, as well as recvfrom().
s = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_RAW)
s.sendto(packet, (dstip, 80))
r = s.recv(1024)
print(r)
However, I am having trouble receiving the response, I can see that the packet is being sent correctly via Wireshark, and the SYN-ACK is sent to my machine, however I am unable to properly receive and print it. Is there a better way I can use the s.recv() function for this sort of input? Or am I using the wrong function?
Any help is appreciated, I'm new to the sockets library. Thanks.

The book Black Hat Python has en example using the socket library to create a scanner, unfortunately not a port scanner. They check if a host is up, and they use a raw socket to receive data. The code is available here.
They are sending SYN-packets with one socket object in a new thread, and sniffing the replies using another socket object.
In the example they use socket.IPPROTO_IP or socket.IPPROTO_ICMP instead of socket.IPPROTO_RAW depending on if it is Windows or not.
For the sniffer they use the function setsockopt(socket.IPPROTO_IP, socket.IP_HDRINCL, 1) for sniffing, where IPPROTO_IP is a dummy-protocol for TCP, IP_HDRINCL is to include headers in the IP packets, and 1 is mapped to the ICMP-protocol in the code.
Good luck!

Below is a recent module I wrote with the help from various sources for socket IO, take what you would like from it.
import socket
import threading
import time
import pygogo as gogo
from icentralsimulator.bridgeio.read_packets import PacketFactory
from icentralsimulator.bridgeio.write_packets import WritePacket
from icentralsimulator.configurations.interfaces import IServerInfoProvider
logger = gogo.Gogo(__name__).logger
send_lock = threading.Lock()
class BridgeConnection:
def __init__(self, bridge_info_provider: IServerInfoProvider):
info = bridge_info_provider.get_bridge_server_info()
self.callback = None
self.bridge_ip = info.IpAddress
self.bridge_port = info.Port
self._connection = None
self._terminate_wait_for_incoming = False
#property
def is_connected(self):
return self._connection is not None
def connect(self, callback):
"""
The purpose of this method is to create (and hold) a connection to the server. At the same time,
it creates a new thread for the purpose of waiting on incoming packets.
"""
if self._connection is not None: return
self._connection = socket.create_connection((self.bridge_ip, self.bridge_port))
self._connection.settimeout(0.5)
self.callback = callback
t = threading.Thread(target=self._wait_for_incoming)
t.start()
time.sleep(5)
def disconnect(self):
"""
Breaks existing connection to the server if one is currently made and cancels the thread that is waiting
for incoming packets. If the connection is not currently open, simply returns silently -- thus it is safe
to call this method repeatedly.
"""
self._terminate_wait_for_incoming = True
while self._terminate_wait_for_incoming:
time.sleep(0.1)
self._connection.close()
self._connection = None
def send_packet(self, packet: WritePacket):
"""
Sends an arbitrary packet to the server.
"""
with send_lock:
logger.debug(f"Sending packet: {packet.payload_plain_text}")
payload = packet.payload
self._connection.sendall(payload)
def _wait_for_incoming(self):
"""
Continually runs a loop to wait for incoming data on the open socket. If data is received, it is converted
to a receive packet and forwarded to the consumer as part of a callback.
"""
self._terminate_wait_for_incoming = False
buf_len = 4096
try:
while not self._terminate_wait_for_incoming:
data = None
try:
_cnx = self._connection
if _cnx is None: break
data = _cnx.recv(buf_len)
if data is not None and len(data) > 0:
while True:
new_data = _cnx.recv(buf_len)
if new_data is None or len(new_data) == 0:
break
data = data + new_data
except socket.timeout:
if data is not None and self.callback is not None:
packet = PacketFactory.get_packet(data)
self.callback(packet)
logger.debug(f"Received packet: {data}")
time.sleep(0.5)
except OSError: # Happens when stopping the application
logger.info("Application aborted")
return
finally:
self._terminate_wait_for_incoming = False
Note that I don't include IServerInfoProvider, or the PacketFactory here. Those are pretty custom to my application. You will need to interpret the packet according to the packet data that arrives in your specific use case.

How to handle ssl connections in raw Python socket?

I'm writing a program to download a given webpage. I need to only use raw python sockets for all the connection due to some restriction. So I make a socket connection to a given domain (the Host field in the response header of an object) and then send the GET request on this. Now when the url is a https url, I think I need to first do the SSL handshake (because otherwise I'm getting non-200 OK responses from the server and other error responses mentioning P3P policies). I inspected curl's response to check how it's able to successfully download while I'm not, turns out curl first does the SSL handshake (that's all the difference). curl is always able to successfully download a given object, the only difference always being the SSL handshake it does.
So I'm wondering how to do the SSL handshake in raw python sockets? Basically I want as easy a solution which allows me to do the minimum besides using raw sockets.

Here is an example of a TCP client with SLL.
Not sure if it's the best way to download a web page but it should answer your question "SSL handshake in raw python socket".
You will probably have to adapt the struct.pack/unpack but you get the general idea:
import socket
import ssl
import struct
import binascii
import sys
class NotConnectedException(Exception):
def __init__(self, message=None, node=None):
self.message = message
self.node = node
class DisconnectedException(Exception):
def __init__(self, message=None, node=None):
self.message = message
self.node = node
class Connector:
def __init__(self):
pass
def is_connected(self):
return (self.sock and self.ssl_sock)
def open(self, hostname, port, cacert):
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.ssl_sock = ssl.wrap_socket(self.sock, ca_certs=cacert, cert_reqs=ssl.CERT_REQUIRED)
if hostname == socket.gethostname():
ipaddress = socket.gethostbyname_ex(hostname)[2][0]
self.ssl_sock.connect((ipaddress, port))
else:
self.ssl_sock.connect((hostname, port))
self.sock.setsockopt(socket.SOL_TCP, socket.TCP_NODELAY, 1)
def close(self):
if self.sock: self.sock.close()
self.sock = None
self.ssl_sock = None
def send(self, buffer):
if not self.ssl_sock: raise NotConnectedException("Not connected (SSL Socket is null)")
self.ssl_sock.sendall(struct.pack('L', len(buffer)))
self.ssl_sock.sendall(buffer)
def receive(self):
if not self.ssl_sock: raise NotConnectedException("Not connected (SSL Socket is null)")
data_size_buffer = self.ssl_sock.recv(4)
if len(data_size_buffer) <= 0:
raise DisconnectedException()
data_size = struct.unpack('L', data_size_buffer)[0]
received_size = 0
data_buffer = ""
while received_size < data_size:
chunk = self.ssl_sock.recv(1024)
data_buffer += chunk
received_size += len(chunk)
return data_buffer
Then you use the class like this:
connector = Connector.Connector()
connector.open(server_ip, server_port, path_to_the_CA_cert.pem)
connector.send(your_data)
response = connector.receive()
connector.close()

You can use the wrap_socket method of the python ssl module to turn your socket into one that talks SSL. Once you've done this you can use it like normal, but internally the data will be encrypted and decrypted for you. These are the docs for the method:
https://docs.python.org/2/library/ssl.html#ssl.wrap_socket

I think the easier way to do that would be using SSL contexts and wraping the TCP socket.
Python SSL module's documentation give a very thoroughful explanation with examples. I recommend you to read the relevant sections of Python 2 or Python 3 ssl module documentation. It should be very easy to achieve what you want.
Hope this helps!

got stuck programming https proxy - what to do next?

I'm currently making a proxy which sits between the browser and the web. Everything works except https. I'm having troubles understanding some passages of it and haven't found many resources on the web. And so I'm stuck.
The code I'm using is:
conn, addr = server.accept()
request = conn.recv(9999) #get a CONNECT request
conn.send(b'HTTP/1.1 200 Connection estabilished\n\n')
enc_req = conn.recv(9999) #this gets an encrypted request
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM) #plaintext client
client.connect((host, 443)) #connect to chosen host
client.send(enc_req)
resp1 = client.recv(9999) #this gets something unreadable (encrypted?)
#could it be the certificate?
#now what?
Is the resp1 I'm getting the certificate? And what do I need to do after that? (Or, which is the same, what does usually happens next with https?)
P.S. I know the question is somewhat generic, but please don't judge me too harshly. I've tried researching on the web but all I keep finding is the encryption method used for ssl. I really don't know how to proceed.

I haven't tested this code (and it's mainly pseudo code), but this should give you an idea of what you need to do.
conn, addr = server.accept()
request = conn.recv(9999) #get a CONNECT request
# Here, parse the CONNECT string and get the host and port (not sure if you were doing that already.
# Then, try to connect *before* you tell the client the connection was established (in case it fails)
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM) #plaintext client
client.connect((host, 443)) #connect to chosen host
conn.send(b'HTTP/1.1 200 Connection estabilished\n\n')
# Then loop until the connections are closed.
while True:
# Read from the client, send the data to the server.
enc_req = conn.recv(9999) #this gets an encrypted request
client.send(enc_req)
# Read from the server, send the data to the client.
resp1 = client.recv(9999) #this gets something unreadable (encrypted?)
#could it be the certificate?
#now what?
# The first time it's certainly the Client Hello message, not encrypted, but in a binary format indeed.
# Just send everything you've just read to the server.
conn.send(resp1)
This is just a quick overview of the idea of the loop you need to write. In reality, you may be able to process both in parallel. You'd also want to be a bit more careful when closing the connection (allowing it to happen in any order while still relaying the last data sent by either party).

As mentioned in the comments, a proxy handling encrypted end-to-end traffic can only pass it on.
Here is a fully working proxy written using circuits that has been fully tested with passing and proxying SSH traffic so it should work equally as well as a pass-through TCP proxy even if SSL is involved:
#!/usr/bin/env python
from uuid import uuid4 as uuid
from circuits import Component
from circuits.net.events import close, connect, write
from circuits.net.sockets import TCPClient, TCPServer
class Client(Component):
channel = "client"
def init(self, sock, host, port, channel=channel):
self.sock = sock
self.host = host
self.port = port
TCPClient(channel=self.channel).register(self)
def ready(self, *args):
self.fire(connect(self.host, self.port))
def disconnect(self, *args):
self.fire(close(self.sock), self.parent.channel)
def read(self, data):
self.fire(write(self.sock, data), self.parent.channel)
class Proxy(Component):
channel = "server"
def init(self, bind, host, port):
self.bind = bind
self.host = host
self.port = port
self.clients = dict()
TCPServer(self.bind).register(self)
def connect(self, sock, host, port):
channel = uuid()
client = Client(
sock, self.host, self.port, channel=channel
).register(self)
self.clients[sock] = client
def disconnect(self, sock):
client = self.clients.get(sock)
if client is not None:
client.unregister()
del self.clients[sock]
def read(self, sock, data):
client = self.clients[sock]
self.fire(write(data), client.channel)
app = Proxy(("0.0.0.0", 3333), "127.0.0.1", 22)
from circuits import Debugger
Debugger().register(app)
app.run()

python asynchat: how to store information on individual connections, and how to know when a client disconnects

For fun, I'm writing a minimal IRC server with asynchat. I'm trying to clear up a few fundamentals (my specific questions follow the code). I've decided not to use anything in Twisted just so I can implement a little more myself. First, the code I have:
import asyncore,asynchat
import socket
class Connection(asynchat.async_chat):
def __init__(self, server, sock, addr):
asynchat.async_chat.__init__(self, sock)
self.set_terminator('\n')
self.data = ""
print "client connecting:",addr
# do some IRC protocol initialization stuff here
def collect_incoming_data(self, data):
self.data = self.data + data
def found_terminator(self):
print self.data
self.data = ''
class Server(asyncore.dispatcher):
def __init__(self, host, port):
asyncore.dispatcher.__init__(self)
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
self.bind((host, port))
self.listen(5)
def handle_accept(self):
conn, addr = self.accept()
Connection(self, conn, addr)
def handle_close(self):
self.close()
s = Server('127.0.0.1',5006)
asyncore.loop()
So, in my mind, this code structure is similar to a Twisted client factory: the Server class is initialized once and basically instantiates Connection every time a client connects. First question: is the best way to keep track of all connected clients by storing all of the Connections in a list within Server?
Also, I don't understand how I am to know when a specific client closes their connection to my socket? Connection implements asynchat (and by extension asyncore) but adding the handle_close() callback to the Connection class doesn't fire when a client disconnects. It seems to be only for when the bound socket on the server is destroyed. I don't see any methods for this purpose. This socket always stays open, whether or not clients connect, right?

to handle client side closed connections check the handle_error method, does your client issue a clean close connection?
handle_error() :Called when an exception is raised and not otherwise handled. The default version prints a condensed traceback.
hope it helps.

How to deliver instance of object to instance of SocketServer.BaseRequestHandler?

This is problem.
My primary work is : deliver "s" object to "handle" method in TestRequestHandler class.
My first step was : deliver "s" object through "point" method to TestServer class, but here im stuck. How to deliver "s" object to TestRequestHandler? Some suggestions?
import threading
import SocketServer
from socket import *
class TestRequestHandler(SocketServer.BaseRequestHandler):
def __init__(self, request, client_address, server):
SocketServer.BaseRequestHandler.__init__(self, request, client_address, server)
return
def setup(self):
return SocketServer.BaseRequestHandler.setup(self)
def handle(self):
data = self.request.recv(1024)
if (data):
self.request.send(data)
print data
def finish(self):
return SocketServer.BaseRequestHandler.finish(self)
class TestServer(SocketServer.TCPServer):
def __init__(self, server_address, handler_class=TestRequestHandler):
print "__init__"
SocketServer.TCPServer.__init__(self, server_address, handler_class)
return
def point(self,obj):
self.obj = obj
print "point"
def server_activate(self):
SocketServer.TCPServer.server_activate(self)
return
def serve_forever(self):
print "serve_forever"
while True:
self.handle_request()
return
def handle_request(self):
return SocketServer.TCPServer.handle_request(self)
if __name__ == '__main__':
s = socket(AF_INET, SOCK_STREAM)
address = ('localhost', 6666)
server = TestServer(address, TestRequestHandler)
server.point(s)
t = threading.Thread(target=server.serve_forever())
t.setDaemon(True)
t.start()

If I understand correctly, I think you perhaps are misunderstanding how the module works. You are already specifying an address of 'localhost:6666' for the server to bind on.
When you start the server via your call to serve_forever(), this is going to cause the server to start listening to a socket on localhost:6666.
According to the documentation, that socket is passed to your RequestHandler as the 'request' object. When data is received on the socket, your 'handle' method should be able to recv/send from/to that object using the documented socket API.
If you want a further abstraction, it looks like your RequestHandler can extend from StreamRequestHandler and read/write to the socket using file-like objects instead.
The point is, there is no need for you to create an additional socket and then try to force your server to use the new one instead. Part of the value of the SocketServer module is that it manages the lifecycle of the socket for you.
On the flip side, if you want to test your server from a client's perspective, then you would want to create a socket that you can read/write your client requests on. But you would never pass this socket to your server, per se. You would probably do this in a completely separate process and test your server via IPC over the socket.
Edit based on new information
To get server A to open a socket to server B when server A receives data one solution is to simply open a socket from inside your RequestHandler. That said, there are likely some other design concerns that you will need to address based on the requirements of your service.
For example, you may want to use a simple connection pool that say opens a few sockets to server B that server A can use like a resource. There may already be some libraries in Python that help with this.
Given your current design, your RequestHandler has access to the server as a member variable so you could do something like this:
class TestServer(SocketServer.TCPServer):
def point (self, socketB):
self.socketB = socketB # hold serverB socket
class TestRequestHandler(SocketServer.BaseRequestHandler):
def handle(self):
data = self.request.recv(1024)
if (data):
self.request.send(data)
print data
self.server.socketB ... # Do whatever with the socketB
But like I said, it may be better for you to have some sort of connection pool or other object that manages your server B socket such that your server A handler can just acquire/release the socket as incoming requests are handled.
This way you can better deal with conditions where server B breaks the socket. Your current design wouldn't be able to handle broken sockets very easily. Just some thoughts...

If the value of s is set once, and not reinitialized - you could make it a class variable as opposed to an instance variable of TestServer, and then have the handler retrieve it via a class method of TestServer in the handler's constructor.
eg: TestServer._mySocket = s

Ok, my main task is this. Construction of the listening server (A-server - localhost, 6666) which during start will open "hard" connection to the different server (B-server - localhost, 7777).
When the customer send data to the A-server this (A-server) sends data (having that hard connection to the B-server) to B-server, the answer receives from the B-server to A-server and answer sends to the customer.
Then again : the customer sends data, A-server receives them, then sends to the B-server, the answer receives data from the B-server and A-server send data to the customer.
And so round and round. The connection to the B-server is closes just when the server A will stop.
All above is the test of making this.