I have an app, software defined radio, that broadcasts UDP packets on a port that tell listeners what frequency and demodulation mode have been set (among other things.)
I've written a demo python client (code below) that listens to the port, and dumps out the information in the appropriate packets to the console.
These are both running under OSX 10.6, Snow Leopard. They work there.
The question/issue I have is: the Python app has to be started before the radio app or it claims the port is already in use (ERRNO 47) during bind, and I don't understand why. The radio app is broadcasting UDP; certainly I want to accommodate multiple listeners -- that's the idea of broadcasting, or at least, so I thought.
So here's the Python code (the indent is a little messed up due to stack overflow's really dumb "make-it-code" indent, but I assure you it's ok):
#!/usr/bin/python
import select, socket
# AA7AS - for SdrDx UDP broadcast
# this is a sample python script that captures the UDP messages
# that are coming from SdrDx. SdrDx tells you what frequency and
# mode it has been set to. This, in turn, would be used to tell
# another radio to tune to that frequency and mode.
# UDP packet from SdrDx is zero terminated, but receiving the
# packet makes it seem like the string contains zeros all the
# way out to the 1024th character. This function extracts a
# python string up to the point where it hits the first zero,
# then returns that string.
# -----------------------------------------------------------
def zeroterm(msg):
counter = 0;
for c in msg:
if ord(c) != 0:
counter += 1
strn = msg[:counter]
return strn
port = 58083 # port where we expect to get a msg
bufferSize = 1024 # room for message
# Create port to listen upon
# --------------------------
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
try:
s.bind(('', port))
except:
print 'failure to bind'
s.close()
raise
s.setblocking(0)
# Listen for messages
# -------------------
looping = True
while looping:
try:
result = select.select([s],[],[])
except: # you can kill the client here with control-c in its shell
s.close() # must close socket
print 'Closing, exception encountered during select' # warn
raise SystemExit # and quit
msg = result[0][0].recv(bufferSize) # actually fetch the UDP data
msg = zeroterm(msg) # convert content into python string
# in next line, [] contain optional repeats
# message format is keyword:data[|keyword:data]
# where from 1...n keyword:data pairs may appear, up to 1024 bytes
# ----------------------------------------------------------------
try:
msgs = msg.split('|') # can be more than one message in packet
except: # failed to split
msgs = [] # on the other hand, can just be one. :)
msgs.append(msg) # so build array with that one.
for m in msgs: # now, for every message we have
keyw,data = m.split(':') # break into keyword and data
print keyw + "-->" + data # you'd do something with this
if keyw == "closing": # Our client terminates when SdrDx does
looping = False # loop stops
s.close() # must close socket
print 'Normal termination'
For reference, here's the Qt code that is sending the UDP message:
Setup:
bcast = new QHostAddress("192.168.1.255");
if (bcast)
{
udpSocketSend = new QUdpSocket(0);
if (udpSocketSend)
{
udpSocketSend->bind(*bcast, txudp);
}
}
Broadcast:
if (udpSocketSend)
{
QByteArray *datagram = new QByteArray(1024,0); // datagram is full of zeroes
strcpy(datagram->data(),msg); // except where text message is in it at beginning
udpSocketSend->writeDatagram(*datagram, QHostAddress::Broadcast,txudp); // send
}
You are trying to bind the same port, twice.
You bind it once in the sender:
if (udpSocketSend)
{
udpSocketSend->bind(*bcast, txudp);
}
and again at the receiver
s.bind(('', port))
And since these are running on the same machine, you are getting an error.
Unless you care what the source port is, you don't need to bind() on the sender, just send it and the stack will pick an appropriate outgoing source port number. In the case of a sender, when you transmit a UDP datagram you specify the destination (udpSocketSend->writeDatagram(...)), and the bind actually determines the source of the outgoing datagram. If you don't bind, thats fine, the stack will assign you a port.
If you do care what the source port is, then I suggest you use a different port number for outgoing source port and incoming destination port. Then you would be able to bind both sender and receiver without issue.
Lastly, there is the option to set the SO_REUSEADDR socket option (in whatever language you're using). This would be necessary if you want to run multiple clients on the same machine, as all the clients would have to bind to the same address. But, I'm not sure whether this socket option is cross platform (*nix works fine) and I think the above suggestions are better.
Related
I have the following problem: I want a sever to send the contents of a textfile
when requested to do so. I have writen a server script which sends the contents to the client and the client script which receives all the contents with a revcall loop. The recvall works fine when
I run the server and client from the same device for testing.
But when I run the server from a different device in the same wifi network to receive the textfile contents from the server device, the recvall doesn't work and I only receive the first 1460 bytes of the text.
server script
import socket
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(("", 5000))
server.listen(5)
def send_file(client):
read_string = open("textfile", "rb").read() #6 kilobyte large textfile
client.send(read_string)
while True:
client, data = server.accept()
connect_data = client.recv(1024)
if connect_data == b"send_string":
send_file(client)
else:
pass
client script
import socket
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(("192.168.1.10", 5000))
connect_message = client.send(b"send_string")
receive_data = ""
while True: # the recvall loop
receive_data_part = client.recv(1024).decode()
receive_data += receive_data_part
if len(receive_data_part) < 1024:
break
print(receive_data)
recv(1024) means to receive at least 1 and at most 1024 bytes. If the connection has closed, you receive 0 bytes, and if something goes wrong, you get an exception.
TCP is a stream of bytes. It doesn't try to keep the bytes from any given send together for the recv. When you make the call, if the TCP endpoint has some data, you get that data.
In client, you assume that anything less than 1024 bytes must be the last bit of data. Not so. You can receive partial buffers at any time. Its a bit subtle on the server side, but you make the same mistake there by assuming that you'll receive exactly the command b"send_string" in a single call.
You need some sort of a protocol that tells receivers when they've gotten the right amount of data for an action. There are many ways to do this, so I can't really give you the answer. But this is why there are protocols out there like zeromq, xmlrpc, http, etc...
I need to communicate between a package written in Python (PsychoPy, for psychological / behavioral experiments) and a legacy piece of C++ software that can use UDP or TCP. In particular, on the Python / PsychoPy side I need an asynchronous process like Matlab's pnet() that will poll a socket to see if it has any data to read, process the data if yes, or just move on if no.
ZMQ was recommended to me; but all the example code I see using zmq_polling assumes that both the sending and receiving occur with ZMQ protocols. Is there some simple Python ZMQ code that connects to a non-zmq TCP or UDP source, and does polling to check for the presence of data without getting hung up if there is no data to read?
Thanks
Aniruddha
import zmq
import time
# Prepare our context and sockets
context = zmq.Context()
# Bind socket to local host
receiver = context.socket(zmq.PULL)
receiver.bind("tcp://129.236.162.112:55513")
#print( "Connected to server with port %s" % port_push)
# Initialize poll set
poller = zmq.Poller()
poller.register(receiver, zmq.POLLIN)
# Process messages from socket
while True:
print('Entered the queue')
try:
socks = dict(poller.poll())
except KeyboardInterrupt:
break
if receiver in socks:
message = receiver.recv()
# process task
print(repr(message))
else:
print('Nothing to show')
time.sleep(0.01)
I can send small TCP packets from the legacy C++ machine; they get sent out without any error messages, implying no problem. But nothing happens with this Python code
The above code enters the 'try' and just stays there.
How do I access error / status messages to debug?
Thanks Aniruddha
Welcome to the Zen-of-Zero : Yes, this is possible
Your post asks many questions at once. Let's go from one to another.
Q1 : How do I access error / status messages to debug?
ZeroMQ documentation presents tools for this. C-side bindings have in common to receive an explicit return code, that may get inspected via assert() plus some more details could be retrieved from errno:
void *ctx = zmq_ctx_new(); assert( ctx && "EXC: Context failed to instantiate" );
void *socket = zmq_socket( ctx, ZMQ_STREAM ); assert( socket && "EXC: Socket failed to instantiate" );
int rc = zmq_bind( socket, "tcp://*:8080" );assert( rc == 0 && "EXC: Bind failed to setup a Transport-Class" );
Q2+3 : Is there some simple Python ZMQ code that connects to a non-zmq TCP or UDP source (2), and does polling (3) to check for the presence of data without getting hung up if there is no data to read?
For very this purpose (2), ZeroMQ framework has been equipped somewhere about version 3.2+ with a STREAM Scalable Formal Communication Pattern Archetype. If not sure, how ZeroMQ architecture uses Context, Context's Socket(s)'s Archetypes, Socket's Transport-Class AccessPoint(s), you may like a short read into "ZeroMQ Principles in less than Five Seconds" before diving into even further details about ZeroMQ
A socket of type ZMQ_STREAM is used to send and receive TCP data from a non-ØMQ peer, when using the tcp:// transport. A ZMQ_STREAM socket can act as client and/or server, sending and/or receiving TCP data asynchronously.
When receiving TCP data, a ZMQ_STREAM socket shall prepend a message part containing the identity of the originating peer to the message before passing it to the application. Messages received are fair-queued from among all connected peers.
When sending TCP data, a ZMQ_STREAM socket shall remove the first part of the message and use it to determine the identity of the peer the message shall be routed to, and unroutable messages shall cause an EHOSTUNREACH or EAGAIN error.
To open a connection to a server, use the zmq_connect call, and then fetch the socket identity using the ZMQ_IDENTITY zmq_getsockopt call.
To close a specific connection, send the identity frame followed by a zero-length message (see EXAMPLE section).
When a connection is made, a zero-length message will be received by the application. Similarly, when the peer disconnects (or the connection is lost), a zero-length message will be received by the application.
You must send one identity frame followed by one data frame. The ZMQ_SNDMORE flag is required for identity frames but is ignored on data frames.
The use of polling (3) has two premises: never use a blocking-mode of any .recv()-methods. ZeroMQ has flags to tell the method not to block: zmq.NOBLOCK on python side. Plus, design the python code around a non-blocking form of .poll() or use a .Poller()-instance.
Example:
import zmq; print( zmq.zmq_version() ) # self-identify
aContext = zmq.Context(); print( "Context()", " instantiated." if zmq.zmq_errno() == 0 else " failed [#{}]".format( zmq.strerror( zmq.zmq_errno() ) ) )
aXmitSOCKET = aContext.socket( zmq.PUSH ); aXmitSOCKET.setsockopt( zmq.LINGER, 0 ); ...
aCtrlSOCKET = aContext.socket( zmq.STREAM ); aCtrlSOCKET.setsockopt( zmq.LINGER, 0 ); ...
while True:
if ( 0 == aXmitSOCKET.poll( 200, zmq.POLLIN ) ): # ~ 200 msec WAIT
# ---------------------------------------------[aXmitPORT].hasNoIncomingMSG
aCountDownREG -= 1 #.DEC CDOWN as XmitPORT has no incoming DataToPREDICT_MSG
aCountUpREG += 1 #.INC CNTUP
if ( 0 == aCtrlSOCKET.poll( 1, zmq.POLLIN ) ): # ~ 1 msec WAIT
# ---------------------------------------------[aCtrlPORT].hasNoIncomingMSG
...
else: #
# ---------------------------------------------[aCtrlPORT].hasAnIncomingMSG
idF,aCMD = aCtrlSOCKET.recv_multipar( zmq.NOBLOCK ) # .recv()<-MSG as CtrlPORT has an incoming COMMAND_MSG
...
#--------------
# finally:
_ = [ aCtrlSOCKET.send_multipart( [ anIdentityFRAME, "" ], zmq.NOBLOCK ) for anIdentityFRAME in aListOfIdFRAMEs ]
aCtrlSOCKET.close()
aXmitSOCKET.close()
#--------------
# always:
aContext.term()
Feel free to also inspect the live-documentation of the methods:
>>> print( aCtrlSOCKET.recv_multipart.__doc__ )
Receive a multipart message as a list of bytes or Frame objects
Parameters
----------
flags : int, optional
Any valid flags for :func:`Socket.recv`.
copy : bool, optional
Should the message frame(s) be received in a copying or non-copying manner?
If False a Frame object is returned for each part, if True a copy of
the bytes is made for each frame.
track : bool, optional
Should the message frame(s) be tracked for notification that ZMQ has
finished with it? (ignored if copy=True)
Returns
-------
msg_parts : list
A list of frames in the multipart message; either Frames or bytes,
depending on `copy`.
Raises
------
ZMQError
for any of the reasons :func:`~Socket.recv` might fail
>>> print( aCtrlSOCKET.send_multipart.__doc__ )
Send a sequence of buffers as a multipart message.
The zmq.SNDMORE flag is added to all msg parts before the last.
Parameters
----------
msg_parts : iterable
A sequence of objects to send as a multipart message. Each element
can be any sendable object (Frame, bytes, buffer-providers)
flags : int, optional
Any valid flags for :func:`Socket.send`.
SNDMORE is added automatically for frames before the last.
copy : bool, optional
Should the frame(s) be sent in a copying or non-copying manner.
If copy=False, frames smaller than self.copy_threshold bytes
will be copied anyway.
track : bool, optional
Should the frame(s) be tracked for notification that ZMQ has
finished with it (ignored if copy=True).
Returns
-------
None : if copy or not track
MessageTracker : if track and not copy
a MessageTracker object, whose `pending` property will
be True until the last send is completed.
This is probably very simple socket message exchange programming, but with my flimsy idea on socket and all, I could not really make it work properly so I'm reaching out for help.
Scenario : Two clients send messages. First one sends Halo and Seeya. After the first sends those messages, the second sends Hello and Bye (This client will just time sleep 6 secs to keep this order). To all messages, server replies with (original msg), client (number)!and a reply message is broadcasted to both clients.
So ideally, the result on both clients would look like this :
Halo, client 1!
Seeya, client 1!
Hello, client 2!
Bye, client 2 !
I couldn't make it to numbering each client, but here is my code that works weird.
server
import socket
clients = []
# send msgs to every client
def broadcast(message):
for client in clients :
client.send(message)
# connection part
s = socket.socket()
s.bind(('127.0.0.1', 7070))
s.listen(2)
while True:
c, addr = s.accept()
if c:
clients.append(c)
msg = c.recv(1024).decode()
msg += ', client!'
broadcast(msg.encode())
client1
### here goes connection part ###
s.send(("Halo").encode())
print(f"server = {(s.recv(1024)).decode()}")
# I've tried adding socket closing/connection part here
s.send(("Seeya").encode())
print((s.recv(1024)).decode())
time.sleep(3)
s.close()
client2 - first connected, but waits 6 secs for message order
### here goes connection part ###
time.sleep(6) # waits for message order
s.send(("Hello").encode())
print(f"server = {(s.recv(1024)).decode()}")
# I've tried adding socket closing/connection part here
s.send(("Bye").encode())
print((s.recv(1024)).decode())
time.sleep(3)
s.close()
The result I get is...
# On client 1 side
Halo, client! # no Seeya, Hello, Bye
# connection isn't closed
# On client 2 side
Hello, client! # no Seeya, Bye
Halo, client! # connection is closed
You have several issues going on here. The first and primary one is that your server's main loop is messed up. Each time through the loop, your server wants to accept a connection. So, the first client to connect gets accepted and immediately its first message is received. But the other client hasn't yet been accepted and so will not receive this first message. Then the second client connection is accepted and its first message is then sent to both clients, but then the loop iterates again and no more messages will be sent from the server until a third client connects. Etc.
So you need to separate accepting connections and receiving messages. This can be done in several ways. The most straight-forward way is to use the select function to wait on a number of sockets at once. That is, if you have a list of sockets including the listening socket and previously accepted ones, you'd do something like this:
# List starts with only listening socket
list_of_sockets = [lsock]
...
while True:
# Wait until some socket becomes "readable"
rfds, _wfds, _xfds = select.select(list_of_socks, [], [])
for sock in rfds:
if sock is lsock:
# Listening socket ready. Accept new connection
c, addr = lsock.accept()
print(f"Accepted {c}")
list_of_socks.append(c)
else:
msg = sock.recv(1024)
if msg:
# Received data from connected socket. Send to all
print(f"Got {msg.decode()} from {sock}")
broadcast(msg)
else:
# Got end of file (this client closed). Remove client from list
print(f"Closed {sock}")
list_of_socks.remove(sock)
Another issue with your code that will not be addressed by the server code above: You cannot assume that each message you send will be received as a distinct unit. That is, if the server sends 'Halo' and then it sends 'Hello' before you (a client) have done a recv, then in all likelihood, all the data will be returned in one fell swoop; that is 'HaloHello'.
Generally therefore you will want to put some kind of separator in the data (like a newline [\n] -- but then you'll need to parse the received data) or, better yet, place a fixed-length field in front of each message, giving the length of the subsequent variable-length part, so that you can receive and process exactly one message at a time. (In python, this typically involves using the struct module's pack and unpack functions.) As a result, your current client code will probably not properly sequence messages as you wish.
Also -- though it is less likely to cause a problem -- the same goes for send: you should not assume that send(N) sends exactly N bytes. It might send 1, 2, 3 or N-1 bytes. You can use sendall to ensure that all bytes are sent.
I have two functions. One sends a UDP packet to a port number and returns the port number if it gets a response. The second cycles through addresses calling the first function repeatedly incrementing the port number. It stops when a port number is returned. I'm basically spamming an IP to find which port number is being used.
All works well with very long time outs but I'm trying to speed up the rate at which I send my test packets. For example I might get a packet back from port 27018 even though the spam port function is sending to 27022. It then incorrectly reports that 27022 is the port to use. This happens even when I return the returned packet info from the first function since you can't tell the arguments which were used originally.
def check_port(s, serverip, serverport):
payload = "ffffffff54536f7572636520456e67696e6520517565727900".decode('hex')
s.sendto(payload, (serverip, serverport))
while 1:
try:
s.settimeout(0.1) # time to wait for response
d = s.recvfrom(1400)
data = d[0]
addr = d[1]
if len(data) > 1:
break
except socket.error:
return False
return addr[1]
def spam_ports(serverip):
s = serverComms.port_config()
start_highport = 27015
end_highport = 28015
start_lowport = 2299
end_lowport = 4000
port = start_highport
while check_port(s,serverip, port) == False:
port += 1
if port == end_highport:
port = start_lowport
if port == end_lowport:
return 'no query ports found in range'
else:
return check_port(s,serverip, port)
I really appreciate any help with this.
I think I know what happens.
It takes some time for the server to reply. If the delay is shorter than that, your application becomes confused. Let me explain:
You send packages to port 1000, 1001, 1002, ...
Say port 1010 produces a reply. But lets assume the server needs a full second to reply. Your application has progressed well bejond 1010 since the timeout is less then a second. By the time the reply arrives your application is already at, say, 1020. Now it looks like the received package is the result of sending something to 1020.
Possible approch
What you need is a way to know to which port a received reply belongs. Here it gets tricky:
Each package has a source and a destination port. With the packages you send the destination port is incremented. The source port is probably arbitrarly assigned. When the server answers it will send a package with an arbitrary source port and the destination port equal the source port of your package.
What you could do is check with the documentation and see how you can control the source port of the packages you're sending. With that you make sure that each sent package has a different source port. This identifies each package uniquely. Now you can use the destination port of the reply to know were it belongs to.
Then you can use, for example, two threads. One sending out packages and one receiving the answers. You keep a dict that maps my_source_port to server_port. The sender fills the dict and the receiver reads it. You can let the sender send as fast as it can, now timeout required. Once the sender is done you give the receiver thread a second or so to catch up.
port_map = {}
active_ports = []
def send(min, max):
for dest_port in range(min,max):
src_port = get_free_port()
port_map[src_port] = dest_port
send_package(src_port, dest_port, 'somedata')
def recv():
while True:
src_port, dest_port, data = receive_package()
port = port_map[dest_port]
active_ports.append(port)
This is not a working example, just the basic idea. The methods don't exist in that form, thread synchronization is missing and recv() would run forever, but it shows the basic idea.
You probably have to create a socket for each package you send.
Am newbie to python and stuck at a point. I want to create port scanner with using only python 3 inbuilt libraries (means avoiding scapy etc) I have following code :
import socket
for i in range(1,26):
s = socket.socket()
s.settimeout(0.5)
ip = "74.207.244.221" #scanme.nmap.org
response = s.connect_ex((ip, i))
if response:
print ("%d\tclose" %i)
else:
print ("%d\topen" %i)
s.close()
Now I want to add 2 functionalities to this : that is
Distinguish between close and filtered ports . In both cases am receiving same errno in return so how can I check if I have received back a rst packet or nothing ? As far as I have tried s.recv() isn't working for this.
I want to control the number of tries (attempts), i.e I want to send only one or two syn packets. I don't want this program to send more than 2 syn packets for probes. How can this thing be achieved ?
Distinguish between close and filtered ports . In both cases am
receiving same errno in return so how can I check if I have received
back a rst packet or nothing
You've probably only checked with servers that send back a RST. Here's what I tried:
First case, normal config:
>>> os.strerror(s.connect_ex((ip, 81)))
'Connection refused'
Second, with manual iptables:
iptables -A OUTPUT -p tcp --dport 81 -j DROP
>>> os.strerror(s.connect_ex((ip, 81)))
'Resource temporarily unavailable'
I want to control the number of tries (attempts), i.e I want to send
only one or two syn packets.
I don't think there's a setsockopt TCP option exposed, but on linux there's:
net.ipv4.tcp_syn_retries
However, since you limited the timeout for the socket, all operations that don't finish within 0.5 seconds will time out. So it's likely only 1 or 2 SYNs will leave the station.
#!/usr/bin/python
import socket
s = socket.socket(socket.AF_INET, socekt.SOCK_STREAM)
host = 74.207.244.221
def portscan(port):
try:
s.connect((host,port))
return True
else:
return False
for x in range(1,255):
if portscan(x):
print('Port',x,'Is Open')