I have a udp server in python that i'm testing out by sending packets with netcat -u server_ip server_port
on the udp server, I can receive the packets with
data,addrport = socket.recvfrom(some_number) — I can read the data received and see the other socket's address port with addrport.
But if I try to use socket.getpeername() on the same variable instead it gives the OSError: [Errno 107] Transport endpoint is not connected error.
What causes this? I'm confused as my netcat terminal doesn't close after sending, which I assume means its already connected to my UDP socket.
I can receive the packets with data,addrport = socket.recvfrom(some_number)
recvfrom means that you are working with an unconnected UDP socket, i.e. the case where a single socket could receive packets from various sources and also send data to various sources using sendto. getpeername instead expects a connected socket, i.e. one which will only receive data from a single source (using recv not recvfrom) and only send to a single source (using send not sendto). This is the case with TCP established sockets (the ones returned by accept) but also with UDP socket which are explicitly connected by calling connect.
I've written server and client programs with Python.
Server.py
import socket
sock = socket.socket (socket.AF_INET, socket.SOCK_STREAM)
host = socket.gethostname()
port = 5555
sock.bind((host, port))
sock.listen(1)
conn, addr = sock.accept()
data = "Hello!"
data = bytes(data, 'utf-8')
conn.send(data)
sock.close()
Client.py on Linux
import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
host = socket.gethostname()
port = 5555
sock.connect((host, port))
data = sock.recv(2048)
data = str(data, "utf-8")
print(data)
sock.close()
When I run the server and then the client on my local machine (a Linux Mint), it works correctly. I got "Hello!" in bash, and everything is fine. BUT when I ran my client program on another machine (a Windows 8) and ran it (previously I ran server on Linux, of course, and change IP address in client to my static Linux mint's IP) it says:
ConnectionRefusedError: [WinError 10061] No connection could be made
because the target machine actively refused it
client.py on Windows
import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
host = "here is my static ip"
port = 5555
sock.connect((host, port))
data = sock.recv(2048)
data = str(data, "utf-8")
print(data)
sock.close()
I must say that I had done port forwarding in my router settings on port 5555. Earlier, I had done same thing to port 80 and my own site worked correctly, but now it doesn't work to 5555 with Python sockets! Why? I can't get it! And one more thing: I tried to change the port to 80 in my server and client files, but it didn't work too. PLease, help.
You have to change the socket.gethostname() in the server script to the empty string (or just directly call socket.bind(('', port))).
Your problem is not in Python but in the usage of sockets generally. When you create a socket, you just prepare your process to receive/send some data from/to another process.
Server
The first step for creating a socket is to specify what kind of protocol will be used for communication between those processes. In your case it is the socket.AF_INET which is constant for use of IP protocol and the socket.SOCK_STREAM is specify reliable stream-oriented service. The reliable stream-oriented service means that you want to be sure that every single sent byte will be delivered to the other side and nothing can be lost during the communication (the underlying OS will use the TCP protocol for that). From this point we are using the IPv4 protocol (because we set the socket.AF_INET).
The second step is bind it to an address. The bind process assigns an address where you expect a client will join (with your socket's settings it's a IP address and the TCP port). Your PC has multiple IP address (well, at least two). It always has 127.0.0.1, which is called "callback" and works only when your applications communicate on the same PC (that is you Linux - Linux scenario in the question), and then you have your external IP address, for communication with other computers (let's pretend it is 10.0.0.1).
When you call socket.bind(('127.0.0.1', 5555)), you're setting the socket to listen only for communication from the same PC. If you call socket.bind(('10.0.0.1', 5555)), then the socket setting is ready to receive data targeted to the 10.0.0.1 address.
But what if you have 10 IPs or more and you want to receive everything (with the right TCP port)? For those scenarios you can leave the IP address in bind() empty, and it does exactly what you want.
With Python's version of bind(), you can also enter a "computer name" instead of the concrete IP. The socket.gethostname() call returns your computer's name. The problem is in the translation of "computer name" to the IP which Python performs behind your back. The translation has some rules but generally your "computer name" can be translated into any IP address which you have set on your computer. In your case, the your computer's name is converted into 127.0.0.1, and that's why communication works only between processes on the same computer.
After socket.bind(), you have the socket ready to use but it is still "inactive". The call to socket.listen() activates the socket and causes it to wait until it receives an attempted connection. When a socket receives a new connection request, it will put it into a queue and wait for processing.
That's what socket.accept() does. It pulls the connection request from the queue, accepts it, and establishes the stream (remember the socket.SOCK_STREAM when you set up the socket) between the server and the client. The new stream is actually a new socket, but is ready to communicate with other side.
What happened with the old socket? Well, it's still alive, and you can call socket.listen() again to get another stream (connection).
How is it possible to have multiple sockets on the same port?
Every connection within computer's network is defined by flow which is 5-item tuple of:
L4 protocol (usually TCP or UDP)
Source IP address
Source L4 port
Destination IP address
Destination L4 port
When you create a new connection with a client, the flow can look like this: (TCP, 192.168.0.1, 12345, 10.0.0.1, 55555). Just for clarification, the server's response flow is (TCP, 10.0.0.1, 55555, 192.168.0.1, 12345), but it isn't important for us. If you create another connection with a client, that it will differ at source TCP port (if you do it from another computer that it will differ also at the source IP). Only from this information you can distinguish every connection created to your computer.
When you create a server socket in your code and call socket.listen(), it listens for any flow with this pattern (TCP, *, *, *, 55555) (the * means "match everything"). So when you get a connection with (TCP, 192.168.0.1, 12345, 10.0.0.1, 55555), then socket.accept() creates another socket which works only with this one concrete flow while the old socket carries on accepting new connections which haven't yet been established.
When the operating system receives a packet, it looks in the packet and checks the flow. At this point, several scenarios can take place:
The packet's flow matches all 5 items exactly (without usage of *). Then the packet's content is delivered to the queue associated with that socket (you're reading the queue when you call socket.recv()).
The packet's flow matched socket with associated flow contains * then it is considered as new connection and you can call scoket.accept().
The operating system doesn't contain open socket which would match the flow. In that case the OS refuse connection (or just ignore the packet it depends on firewall settings).
Probably an example can clarify these scenarios. The operating system has something like a table where it maps flows to sockets. When you call socket.bind(), it will assign a flow to the socket. After the call, the table can look like this:
+=====================================+========+
| Flow | Socket |
+=====================================+========+
| (TCP, *, *, *, 55555) | 1 |
+-------------------------------------+--------+
When it receive a packet with flow (TCP, 1.1.1.1, 10, 10.0.0.1, 10) then it won't match any flow (last port won't match). So, the connection is refused. If it receives a packet with flow (TCP, 1.1.1.1, 10, 10.0.0.1, 55555), the packet is delivered to the socket 1 (because there is a match). The socket.accept() call creates a new socket and record in the table.
+=====================================+========+
| Flow | Socket |
+=====================================+========+
| (TCP, 1.1.1.1, 10, 10.0.0.1, 55555) | 2 |
+-------------------------------------+--------+
| (TCP, *, *, *, 55555) | 1 |
+-------------------------------------+--------+
Now you have 2 sockets for 1 port. Every received packet which matches the flow associated with the socket 2 also matches the flow associated with socket 1 (on the contrary, it does not apply). It's not a problem because the socket 2 has a preciser match (is doesn't use the *), so any data with that flow will be delivered to socket 2.
How to serve multiple connections
If you want to implement a "real" server, your application should be able to process multiple connections without restarting. There are 2 basic approaches:
Sequential processing
try:
l = prepare_socket()
while True:
l.listen()
s, a = socket.accept()
process_connection(s) # before return you should call s.close()
except KeyboardInterrupt:
l.close()
In this case, you can process only one client while others clients have to wait for accept. If the process_connection() takes too long, then others clients will timeout.
Parallel processing
import threading
threads = []
try:
l = prepare_socket()
while True:
l.listen()
s, a = socket.accept()
t = threading.Thread(target=process_connection, s)
threads.append(t)
t.start()
except KeyboardInterrupt:
for t in threads:
t.join()
l.close()
Now when you receive a new connection, it will create a new thread so that every connection is processed in parallel. The main disadvantage of this solution is that you have to solve common troubles with threading (like access to shared memory, deadlocks etc.).
Beware that the above snippets are only examples, and are not complete! For example, they don't contain code for graceful exit on unexpected exceptions.
Servers in Python
Python also contains a module called socketserver, which contains shortcuts for creating servers in Python. You can find examples of how to use it here.
Client
With the client, it's much more simpler than with the server. You just have to create a socket with some settings (same as server side), and then tell it where the server is (what its IP and TCP port are). This is accomplished through the socket.connect() call. As a bonus, it also establishes the stream between your client and server, so from this point you can communicate.
You can find more information about sockets at the Beej's Guide to Network Programming. It's written for usage with C, but the concepts are the same.
I was stuck with the same problem months ago and also wasn't able to do port forwarding. I found a way out of port forwarding Ngrock
For your information what Ngrock does is it is a useful utility to create secure tunnels to locally hosted applications using a reverse proxy. It is a utility to expose any locally hosted application over the web
For How To use it, please see the steps shown below :
If you are in Mac write this command in your terminal to download ngrock
brew install ngrok
For windows
choco install ngrok
After installing You need need to sign up on the Ngrok website
You will get your Ngrock authentication token then paste this command in the terminal
For Mac and Windows
ngrok config add-authtoken <token>
Now that Ngrock is all setup you can start a tunnel using
ngrok tcp <Your Port Number Used In Server.py>
ngrok tcp 5321
Note : Please Give the command inside the directory in which the Python Socket Server File in Located
That's it Your Socket can connect you any computer over the internet anywhere in the world
If You are still struggling to see the detailed explanation in this video
You can also refer ngrock documentation here
IP multicast gives you the ability to send a single packet, which is picked up by multiple interfaces if they are subscribed to that multicast. If I understand it correctly.
Now if I want to use UDP, in combination with IP multicast, I am obligated to assign a port to listen to. But now I understand that I only receive UDP packets on a multicast specifically sent to that port. But I would like to intercept all UDP packets send to a certain multicast IP address, regardless of the port and receive them at my single socket.
Is something like this possible?
Preferably accompanied by a python example, if this is possible.
This isn't possible using the BSD socket API (which is roughly the API Python exposes in its socket module) - except by creating 2 ** 16 - 1 sockets and using them to bind to all ports.
It's possible using lower-level interfaces such as the TUN/TAP system offered by Linux.
This is the following code example from Twisted for dealing with receiving of multicasts. I currently am listening to many groups with the same client and I want to be able to print out which group a certain datagram packet came from. I would think that this can be received from the address parameter of datagramReceived; however, this only gives me a tuple containing the local ip and port that a group is bound to, but not the address of the group itself.
Question: How can I print the multicast address from where a datagram originated from within the Twisted protocol/API?
from twisted.internet.protocol import DatagramProtocol
from twisted.internet import reactor
class MulticastPingClient(DatagramProtocol):
def startProtocol(self):
# Join the multicast address, so we can receive replies:
self.transport.joinGroup("228.0.0.5")
self.transport.joinGroup("229.0.2.11")
self.transport.joinGroup("221.3.3.3")
# Send to 228.0.0.5:8005 - all listeners on the multicast address
# (including us) will receive this message.
self.transport.write('Client: Ping', ("228.0.0.5", 8005))
def datagramReceived(self, datagram, address):
print "Datagram %s received from %s" % (repr(datagram), repr(address))
reactor.listenMulticast(8005, MulticastPingClient(), listenMultiple=True)
reactor.run()
Unfortunately the socket APIs Twisted wraps (via Python's standard library) do not provide this information. I would recommend having a separate DatagramProtocol for each multicast group, and listening on different ports for each. Although someone could still send a UDP datagram directly to to that port and you wouldn't be able to distinguish it from multicast.
I have a vague memory indicating that the recvmsg() API does provide the necessary information, though I lack the time to verify this. Twisted 12.1 has the beginning of a recvmsg() wrapper, so this may be a possible way to add this functionality to Twisted (or your code) with a bit more work.
I have a Python process on one Linux machine server1 that receives and processes raw UDP packets. I want to have another Linux machine server2 capable of listening to the same UDP packets server1 is receiving.
Is there any Python solution capable of sniffing UDP packets addressed to the another (Linux) machine?
If you want more than one machine to process the same data, you'd be better off going to mulitcast (if you can control the sender and the infrastructure)
Else, http://sourceforge.net/projects/pylibpcap/ will enable packet capture via python. You will still have to configure the infrastructure to get the packets to the machine you want to sniff them. Either by iptables (if is a Linux machine) or a mirror port on the switch etc.
Edit:
If you want the processes on different machines (you think one machine can't do it all) I would have a Linux machine receive the data, and using iptables, send it to multiple other machines. Maybe to a different socket on the same machine. This is possible because its UDP. If you want it all on the same machine, I would have a single process that spawns subprocesses with connected PIPEs, binds the UDP socket and copies the data to each subprocess' pipe; maybe after some input validation
This not depends on Python but on your network architecture. If server1 and server2 are connected (probably they are) through a switch then you can't do it, because the packet passing through the router will be sent only to the requested IP.
So first of all, tell us how is composed your network architecture. Where are server1 and server2? How the reach each other?
Your problem solution won't depend neither on your OS nor in the language used.
Anyway, you tagged your question "linux", so I think you are familiar with that OS.
If this is the case, and server1 and server2 access the LAN through the same router, you can evaluate the possibility of installing linux on your router (have a look at openwrt), and perform the sniffing and whatever from the router itself.
I have had a similar problem, and wrote a small python script to forward incoming udp packets to multiply hosts. A drawback here is ofcourse that you loose the source IP of the originating udp packets.
import socket
import sys, time, string
def sendUDP(remotehost,remoteport,UDPSock,data):
UDPSock.sendto( data, (remotehost,remoteport))
def serverLoop(listenport,remotes):
# Set up socket
UDPSock = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
UDPSock.bind( ("0.0.0.0",listenport) )
while 1:
data, addr = UDPSock.recvfrom(1024)
if not data: pass
else:
sys.stdout.write(".") ; sys.stdout.flush()
# Send udp packet to remotes...
for remote in remotes:
if remote[0] == addr: pass
else: sendUDP(remote[0],remote[1],UDPSock,data)
time.sleep(0.001)
if __name__ == "__main__":
if len(sys.argv) < 3:
print "%s listenport remotehost1:port1 remotehostN:portN ..." % sys.argv[0]
sys.exit(-1)
listenport = int(sys.argv[1])
print "Local foward port %d" % listenport
remotes = []
for pair in sys.argv[2:]:
host,port = string.split(pair,":")
remotes.append( (host,int(port)) )
print "Adding remote forward %s:%s" % (host,port)
print "Starting serverloop"
serverLoop(listenport,remotes)