Develop Reference

sel.unregister(clt_socket) not removing socket from selector

New to community here, trying to learn python as part of a career pivot.
Attempting to create a basic multi-client socket server using selectors. Everything seems okay until a client submits a request to close the chat. At this point server calls sel.unregister(clt_socket) and then closes the clt_socket on server side, all without error. However, the associated key stays registered, and thus sel.select() still tries to pass it on to the service_client() method, which results in an OSError.
Despite banging my head for a few hours, and despite reading python docs, google search, stackoverflow search, I can't figure out what I'm doing wrong.
Any help you could provide would be most appreciated.
Thank you!
Error:
Failed while working with key: SelectorKey(fileobj=<socket.socket [closed] fd=-1, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0>, fd=64, events=3, data=namespace(clt_address=('127.0.0.1', 50377), inc_msg=b'q', out_msg=b''))
Server:
from socket import *
import selectors
import types
def accept_client(srv_socket):
# accept new client if selector finds srv_socket gets a NIL data connection
clt_socket, clt_address = srv_socket.accept()
print ('New client connected: '+str(clt_address))
clt_socket.setblocking(False)
# create a place to put our data
data = types.SimpleNamespace(clt_address=clt_address,inc_msg=b'',out_msg=b'')
# registors socket in selector for both reading and writing
events = selectors.EVENT_READ | selectors.EVENT_WRITE
sel.register(clt_socket,events,data=data)
# process ready events (mask) for clt_socket(key.data)
def service_client(key,mask):
clt_socket = key.fileobj
# socket ready to read when both are True
if mask & selectors.EVENT_READ:
key.data.inc_msg = key.fileobj.recv(1024)
print (str(key.data.clt_address)+' said: '+key.data.inc_msg.decode('utf-8'))
if key.data.inc_msg == 'q'.encode('utf-8'):
sel.unregister(key.fileobj)
key.fileobj.close()
print ('Chat closed BY: '+str(key.data.clt_address))
else:
key.data.out_msg = input('Reply to '+str(key.data.clt_address)+' :').encode('utf-8')
# sockets should always be ready for write
if mask & selectors.EVENT_WRITE:
sent = key.fileobj.send(key.data.out_msg)
key.data.out_msg = key.data.out_msg[sent:]
if key.data.out_msg == 'q'.encode('utf-8'):
sel.unregister(key.fileobj)
key.fileobj.close()
print ('Chat closed FOR : '+str(key.data.clt_address))
# define server parameters (using localhost IPv4 address)
print ('Defining server parameters...')
srv_ip = '127.0.0.1'
srv_port = 6789
srv_address = (srv_ip,srv_port)
# create & bind TCP srv_socket, with NON-BLOCKING listen for connections
print ('Starting server. Listening for connections...')
srv_socket = socket(AF_INET,SOCK_STREAM)
srv_socket.bind((srv_address))
srv_socket.listen() # default queue 5
srv_socket.setblocking(False)
# register srv_socket with selector to monitor for read evts (clt connections?)
sel = selectors.DefaultSelector()
sel.register (srv_socket,selectors.EVENT_READ,data=None)
# create event loop
while True:
events = sel.select(timeout = None) # get (key,evts) per reg socket
for key, mask in events:
try:
if key.data is None: # no data = new clt (from listen)
accept_client(key.fileobj)# accept it! (key.fileobj = srv_socket)
else: # yes data = old clt
service_client(key,mask) # service it!
except:
print('Failed while working with key: ',str(key))
continue
# NOT being called at the moment
srv_socket.close()
print ('Exiting program')```
Client(s):
from datetime import datetime
HOST = '127.0.0.1'
PORT = 6789
max_size = 1024
print ('Starting the client at : ',datetime.now())
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.connect((HOST,PORT))
print ('Connection established with server.')
print ('Initating protocols to send and receive messages...')
while True:
message_to_send = input('Enter msg: ').encode('utf-8')
s.sendall(message_to_send)
if message_to_send == 'q'.encode('utf-8'):
break
received_message = s.recv(max_size).decode('utf-8')
print('Server said: ',received_message)
if received_message == 'q':
break
print ('Closing connection...')
s.close()```

I returned the selector and that seemed to do the trick.
Not sure if that's the correct way, but it worked.

Trying to serve HTML5 <audio> element from Python+SSL to Android Chrome

I have a Python-based web application server; everything was great
with plain HTTP. With HTTPS, everything works perfectly with
all browsers--except Android Chrome. With Chrome, HTML and icons are fine
over HTTPS, but the <audio> element fails to start. I see
Chrome pull one initial byte of the mp3 (presumably its way of testing file
presence), which is served back to it. And then Chrome greys out
the <audio> element.
I've pulled that same single byte using curl, and it works fine:
curl -r 0-1 -k https://localhost:8083/foo.mp3 > foo1.mp3
I've added various delays and flushes, without fixing the problem.
It would seem to not be a content-range issue, as I've also
changed the code to return the full mp3 (with 200 code), with
the same Chrome behavior.
Firefox is happy with it (both Android and Linux), as is Midori (a
Webkit based browser). On Linux, Chrome/Chromium are both happy with
it. Android Chrome--no luck.
I've extracted just the relevant bits of code; I'm assuming there's
some HTTP nicety I'm missing (and, believe me, I've been looking).
To exercise it, plug in a self-signed certificate pair at the
hard-coded files "server.key" and "server.crt". Then put an mp3
of your choice at "foo.mp3" and point your browser at:
https://localhost:8083
TIA for any suggestions! I'm sorry the code runs a little long;
I extracted just the bits to reproduce this. I almost left off
the range support, but didn't want to chase legacy code paths
of Chrome.
#
# debug.py
# Sample code snippet to debug Chrome HTML5 <audio> SSL problem
#
import pdb
import sys, socket, ssl, threading, os, time
from BaseHTTPServer import BaseHTTPRequestHandler
OURMP3 = "foo.mp3"
# Decode a "range:" header option, return
# (offset,length) or None if we don't like
# the region (TBD, multiple ranges and
# multipart)
# We're passed the file's os.stat as well as
# the range: field value.
def decode_range(st, range):
# Byte units, please
if not range.startswith("bytes="):
return None
range = range[6:]
# Single range
if ',' in range:
return None
# Start to offset
if range[0] == '-':
range = range[1:]
if not range.isdigit():
return None
val1 = int(range)
if val1 >= st.st_size:
return None
return (0, val1)
# Offset to end...
elif range[-1] == '-':
range = range[:-1]
if not range.isdigit():
return None
val2 = int(range)
if val2 >= st.st_size:
return None
return (val2, st.st_size - val2)
# Offset1 to offset2
else:
parts = range.split('-')
if len(parts) != 2:
return None
if not all(p.isdigit() for p in parts):
return None
val1 = int(parts[0])
val2 = int(parts[1])
if val1 >= val2:
return None
return (val1, val2-val1)
class Client(BaseHTTPRequestHandler):
# Send the mp3 file OURMP3
def send_mp3(self):
# For simplicity, just a big buffer
f = open(OURMP3, "rb")
st = os.fstat(f.fileno())
buf = f.read()
f.close()
# Partial
ranged = 'range' in self.headers
if ranged:
tup = decode_range(st, self.headers['range'])
assert tup
startoff,nbyte = tup
assert (nbyte + startoff) <= len(buf)
self.send_response(206)
else:
startoff = 0
nbyte = len(buf)
self.send_response(200)
# Headers
self.send_response(200)
self.send_header("Content-type", "audio/mpeg")
self.send_header("Content-Length", nbyte)
if ranged:
self.send_header("Content-Range",
"bytes %d-%d/%d" % (startoff, startoff+nbyte-1, st.st_size))
self.send_header("Last-Modified",
time.asctime(time.localtime(st.st_mtime)))
self.end_headers()
# Let our upper layers write it back (or discard
# it, for HEAD)
return buf[startoff:(startoff+nbyte)]
# Send an HTML5 <audio> player for our mp3
def send_root(self):
buf = \
'''
<html><head><title>Test MP3</title></head>\r
Audio player:<br>\r
<body><audio src="foo.mp3" controls autoplay></audio></body>\r
</html>\r
'''
self.send_response(200)
self.send_header("Content-type", "text/html")
self.send_header("Content-Length", len(buf))
self.end_headers()
return buf
# Process HTTP GET's
def do_GET(self):
sys.stderr.write("GET %r\n" % (self.path,))
path = self.path
# Root; our HTML to play the mp3
if (not path) or (path == '/'):
buf = self.send_root()
# Our mp3
elif path.endswith(OURMP3):
buf = self.send_mp3()
# That's all we serve
else:
self.send_error(404)
return None
# Body?
if buf:
self.wfile.write(buf)
# Dispatch this web client
def serve_client(conn, tup):
h = Client(conn, tup, None)
conn.close()
sys.exit(0)
# Endless server loop on port 8083
def server():
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s = ssl.wrap_socket(s,
"server.key", "server.crt",
server_side=True, suppress_ragged_eofs=False,
do_handshake_on_connect=False)
s.bind( ("", 8083) )
s.listen(10)
while True:
conn,tup = s.accept()
sys.stderr.write("client %r\n" % (tup,))
conn.do_handshake()
t = threading.Thread(target=serve_client, args=(conn,tup))
t.start()
if __name__ == "__main__":
server()

Python - Scapy and nfqueue change outgoing GET request, set_payload not effecting payload

I'm new to nfqueue and am trying to change a packet's payload using it.
however, when I poll the payload from the 'set-ed' packet(simply named 'packet' in the following snippet), i get the original payload of the packet before the change.
even-though when i read scapkt[TCP].payload it has the right payload.
where did i go wrong? (change_image is the callback)
def change_image(packet):
scapkt = IP(packet.get_payload())
#is a TCP packet
if scapkt.proto == 6:
data = str(scapkt[TCP].payload)
getImage = re.search('GET [a-zA-Z0-9/]{1,}\.(jpg|JPG|jpeg|JPEG|png|PNG) HTTP/1.1\r\nHost: [a-zA-Z0-9\.]{1,}', data)
if getImage != None:
original_len = len(scapkt[TCP].payload)
data = data.replace(getImage.group(), 'GET ' + image + ' HTTP/1.1\r\nHost: ' + host)
scapkt[TCP].payload = data
postMod_len = len(scapkt[TCP].payload)
scapkt[IP].len = original_len + (postMod_len - original_len)
del scapkt[IP].chksum
del scapkt[TCP].chksum
packet.set_payload(str(scapkt))
print packet.get_payload()
packet.accept()
I am using the NetfilterQueue fork by the awesome fqrouter
FYI: i know this code might not work as intended, but I'm learning to work with nfqueue and am trying to write a simple PoC

The raw sockets can't create an ARP request packet with source (MAC and IP )taken as some other machine .. Any suggestions?

I'm using the below script for injecting an ARP packet request. When I keep the source (MAC and IP) as my machine, I can happily see the packets in the wire and receive ARP replies however on changing the source to a different machine in the LAN, the ARP requests don't get back the ARP replies.
I am dicey if the RAW sockets can only frame up an ARP request for the base machine or am I going wrong somewhere ?
Below is the code ...
#!/usr/bin/python
import sys
import socket
import binascii
import struct
from itertools import chain
try:
iFace = raw_input("Enter the interface using which the Injection needs to be done ...\n")
rawSocket = socket.socket(socket.PF_PACKET, socket.SOCK_RAW,socket.htons(0x0800))
rawSocket.bind((iFace, socket.htons(0x0800)))
print "Raw Socket got created .... with the Ethernet Protocol Id : 0x0806 at interface %s"%str(iFace)
except:
print "Something unexpected happened during the Program execution."
else:
def checkMac(mac):
if len(mac.split(":")) != 6:
print "The MAC is in correct. It should be in Hexadecimal Format with each byte separated with colon...\n"
sys.exit(0)
else:
macList = mac.split(":")
macLen = len(macList)
return tuple ([int(macList[index],16) for index in range(macLen)])
def checkIp(ip):
ipList = ip.split(".")
ipLen = len(ipList)
return int( "".join( [ "{:02X}".format(int(ele)) for ele in ipList ] ), 16 )
dMac = raw_input("Enter the Destination MAC .. hexadecimal charaters separated with ':' \n")
# dMac = "0X:XX:XX:XX:XX:4X"
dMacTup = checkMac(dMac)
# sMac = raw_input("Enter the Source MAC .. hexadecimal charaters separated with ':' \n")
sMac = "XX:XX:XX:XX:XX:XX"
sMacTup = checkMac(sMac)
type = 0x0806
# Creating an Ethernet Packet .... using dMac, sMac, type
etherPack = struct.pack ("!6B6BH",*tuple(chain(dMacTup,sMacTup,[type])))
# Creating an ARP Packet .... now
hardwareType = 0x0001
protocolType = 0x0800
hln = 0x06
pln = 0x04
op = 0x0001
# srcIp = raw_input("Enter the Source IP ':' \n")
srcIp = "10.0.2.216"
intSrcIp = checkIp(srcIp)
destIp = raw_input("Enter the Destination IP .. \n")
# destIp = "10.0.2.1"
intDestIp = checkIp(destIp)
arpPack = struct.pack("!HHBBH6BI6BI", *tuple(chain( [hardwareType,protocolType,hln,pln,op], sMacTup,[intSrcIp], dMacTup,[intDestIp] )))
# Framing the final Packet
finalPack = etherPack + arpPack
for i in range(50):
rawSocket.send(finalPack + "Hacker in the wires ...")
print "Sending Packet %d"%i
finally:
print "Closing the created Raw Socket ..."
rawSocket.close()

How to find mtu value of network through code(in python)?

I have to write a code where I need to send data using udp protocol in python. I need to set the packet size to the MTU value of the network. Is there any way that I can decide the MTU value of the network writing some code in python?

This answer was taken from
http://books.google.co.il/books?id=9HGUc8AO2xQC&pg=PA31&lpg=PA31&dq#v=onepage&q&f=false
(page 31)
s = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
hostName = #ip here
Port = 9999
s.connect((hostName, Port))
s.setsockopt(socket.IPPROTO_IP, IN.IP_MTU_DISCOVER, IN.IP_PMTUDISC_DO)
try:
s.send('#' * 1473)
except socket.error:
print 'The message did not make it'
option = getattr(IN, 'IP_MTU', 14)
print 'MTU:', s.getsockopt(socket.IPPROTO_IP, option)
else:
print 'The big message was sent! Your network supports really big packets!'

There is a github-gist providing this functionality:
import re
import socket
import struct
import logging
import subprocess
from fcntl import ioctl
SIOCGIFMTU = 0x8921
SIOCSIFMTU = 0x8922
log = logging.getLogger(__name__)
def get_mtu_for_address(ip):
routeinfo = subprocess.check_output(['ip', 'route', 'get', ip])
dev = re.search('.*dev (\w+) .*', routeinfo).groups()[0]
mtuinfo = subprocess.check_output(['ip', 'link', 'show', dev])
mtu = re.search('.*mtu ([0-9]+) .*', mtuinfo).groups()[0]
return int(mtu)
class Iface:
def __init__(self, ifname):
self.ifname = ifname
def get_mtu(self):
'''Use socket ioctl call to get MTU size'''
s = socket.socket(type=socket.SOCK_DGRAM)
ifr = self.ifname + '\x00'*(32-len(self.ifname))
try:
ifs = ioctl(s, SIOCGIFMTU, ifr)
mtu = struct.unpack('<H',ifs[16:18])[0]
except Exception, s:
log.critical('socket ioctl call failed: {0}'.format(s))
raise
log.debug('get_mtu: mtu of {0} = {1}'.format(self.ifname, mtu))
self.mtu = mtu
return mtu
def set_mtu(self, mtu):
'''Use socket ioctl call to set MTU size'''
s = socket.socket(type=socket.SOCK_DGRAM)
ifr = struct.pack('<16sH', self.ifname, mtu) + '\x00'*14
try:
ifs = ioctl(s, SIOCSIFMTU, ifr)
self.mtu = struct.unpack('<H',ifs[16:18])[0]
except Exception, s:
log.critical('socket ioctl call failed: {0}'.format(s))
raise
log.debug('set_mtu: mtu of {0} = {1}'.format(self.ifname, self.mtu))
return self.mtu
if __name__ == "__main__":
import sys
logging.basicConfig()
mtu = None
if len(sys.argv) > 2:
dev,mtu = sys.argv[1:]
elif len(sys.argv) > 1:
dev = sys.argv[1]
else:
dev = 'eth0'
iface = Iface(dev)
if mtu is not None:
iface.set_mtu(int(mtu))
print dev,'mtu =',iface.get_mtu()
Source: https://gist.github.com/nzjrs/8934855

The accepted answer did not work for me in Python 3.7. I get: OSError: [Errno 6] Device not configured
But, psutil now has this built in.
import psutil
print(psutil.net_if_stats())
Results in:
{
'lo0': snicstats(isup=True, duplex=<NicDuplex.NIC_DUPLEX_UNKNOWN: 0>, speed=0, mtu=16384),
'en0': snicstats(isup=True, duplex=<NicDuplex.NIC_DUPLEX_UNKNOWN: 0>, speed=0, mtu=1500),
...
}

You can simply do a binary search over ping with DF (Don't Fragment) flag. Here is a working coding to find MTU through the above-mentioned technique. It gives you `minimum MTU of the full packet routing path AKA the max payload you can send.
Tested only on Windows (won't work on Linux/Mac as ping flags are different in different OS)
# tested on Windows 10 Home and python 3.6 [at Great Istanbul, Turkey]
import subprocess
from time import perf_counter
class FindMinMtu:
"""
- Find Minimum "Maximum Transmission Unit" of a packet routing path via Binary Search
- Suppose you want to find how much data you can send in each packet
from London to Turkey?
- Now we need to remember MTU and MSS (Max. Segment size) isn't not the same.
MSS is the actual data (not headers) you can send. A typical formula for MSS is
MSS = MTU - (IP header_size + TCP/UDP/Any Transport Layer Protocol header_size)
whereas MTU = Everything in packet - Ethernet headers
MTU typical refers to Ethernet MTU, AKA how much payload can an ethernet cable push through next hop.
"""
def __init__(self, url: str):
self.url = url
self._low_mtu = 500
# typically ethernet cables can carry 1500 bytes (but Jumbo fiber can carry upto 9K bytes AFAIK)
# so increase it as per your requirements
self._high_mtu = 1500
self._last_accepted = self._low_mtu
#staticmethod
def yield_console_output(command):
p = subprocess.Popen(command,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
return iter(p.stdout.readline, b'')
def does_accept_mtu_size(self, size) -> bool:
command = 'ping {domain_name} -t -f -l {size}'.format(domain_name=self.url,
size=size).split()
for line in self.yield_console_output(command):
line = line.decode(encoding='utf-8')
if line.startswith('Packet') and 'DF' in line:
return False
elif line.startswith('Reply'):
return True
def find_min_mtu(self):
while self._low_mtu <= self._high_mtu:
if not (self.does_accept_mtu_size(self._low_mtu), self.does_accept_mtu_size(self._high_mtu)):
return self._last_accepted
else:
middle = (self._high_mtu + self._low_mtu) // 2
print("Low: {} High: {} Middle: {}".format(self._low_mtu, self._high_mtu, middle))
if self.does_accept_mtu_size(middle):
self._last_accepted = middle
self._low_mtu = middle + 1
else:
self._high_mtu = middle - 1
return self._last_accepted
if __name__ == '__main__':
start = perf_counter()
# please provide protocol less domain name (without http://, https:// and also without www or any subdomain)
# provide the naked url (without www/subdomain)
f = FindMinMtu("libwired.com")
print("\nMTU: {} bytes (Found in {} seconds)".format(f.find_min_mtu(), perf_counter() - start))