Related
So far I have made a VERY basic client/server application that creates a TCP connection. I have a lot of programming experience, just never did this low-level stuff and especially nothing with networks. Note that all the prints are just to help me figuring out what is going on. One of the known issues is that jsonip sometimes gives me an IPv4 and sometimes v6, I don't know why but that doesn't matter for now, just to warn anyone who wants to recreate my code.
Server:
import socket
import requests
port = int(input("Enter port you want to open:\n"))
#todo: add errorhandling
print("Adding socket...")
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
hostname = socket.gethostname()
print(f"Hostname: {hostname}")
ip_address = socket.gethostbyname(hostname)
print(f"Host address: {ip_address}")
r = requests.get(r'http://jsonip.com')
public_ip_address = r.json()['ip']
s.bind((ip_address, port))
print("Is open for connections on IP: "+public_ip_address+" and Port: "+str(port))
s.listen(5)
print("Done initialisation, listening for incoming connections...")
while True:
clientsocket, address = s.accept()
print(f"Connection from {address} has been established")
clientsocket.send(bytes(f"You have connected to server: {hostname}", "utf-8"))
Client:
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
ip = input("Enter IP to connect to:\n")
port = int(input("Enter Port to connect to:\n"))
print(f"Connecting to server {ip} ...")
s.connect((ip, port))
msg = s.recv(1024)
print(msg.decode("utf-8"))
On my local machine: Open 20000 in my server.py, it tells me the host is 127.0.1.1, I then enter 127.0.1.1 into my client script and 20000, and they connect. So the Socket has been bound with the 127.0.1.1. (Side question: What is this IP address, is it like the internal IP address of processes in my PC or something? If running ip a on my other machine it is the first one shown of 2)
Using Virtmanager on my machine and running one Linux Server (command line only) and one normal Ubuntu, the server tells me the host is, again, 127.0.1.1 which I don't need to enter into the other VM to know it won't work, what does work however, is getting the IP-address of the Server via ip a, which in this case is 192.168.122.37, and when I enter this IP address into the client, it connects. But in the socket here I bind, again, the 127.0.1.1, so is it arbitrary what I put here? What SHOULD I bind here, the public, the weird or the 192. address?
The first thing I could not get to work was using 2 physical devices. When opening a server on my Linux machine, I cannot connect with my windows machine at all, no matter if I use my public, my 127. or my 192. IP-address. Now my end goal is doing this over the internet so I am walking myself up, describing here the steps I took to try and get where I want to be but here I hit a brick wall where I don't know what is wrong. Am I binding the wrong address on the server, is my router being a problem, is there something else wrong?
I also tried leaving my network using my friends pc a few countries over, but this also just results in a timeout (my theory is that the Router port he is trying to open is closed and I have now idea how I can make the router send data to his PC, which should be not impossible as firefox and every application using internet does it without me having to manually forward every port, I just don't know how). This is my end goal, creating a connection between my friends PC and mine, and this is how far I got (I wouldn't mind skipping the local network if it is not relevant for fixing the global connection problem), so, tl;dr: what did i do wrong, what do i need to bind and what do i need to do for the final result to work?
There are many questions to answer.
Addresses 127.X.X.X are reserved for the loopback interface, most common one is 127.0.0.1. The loopback is a virtual, but important interface and as you have probably guessed, it is usable on the local machine only. You cannot use 127.X.X.X address to make two hosts to communicate with each other.
Addresses 192.168.X.X (and also 10.X.X.X and 172.16-31.X.X.) are reserved for local LANs. They are not valid on the Internet.
You cannot use these addresses to make two hosts to communicate with each other over the public Internet (unless you create a tunnel, an advanced networking topic)
Almost everybody uses them, because we ran out of IPv4 addresses long time ago, they were difficult to get, expensive, etc. Also such hosts are isolated from the Internet, they can be reached only via a router that translates addresses. Such router feature is called NAT. A typical router has one valid Internet address and all connections to the Internet appear as coming from the router. If you contant a service like jsonip.com from a PC, you get your router's address, not your PC's address.
See also: Finding local IP addresses using Python's stdlib
To make your program working, make it to accept connections on all interfaces. See the first example in the socket docs. On Linux, use port numbers >= 1024. Ports < 1024 are reserved, not available to regular users.
Final point is that a firewall may prevent connections to your server. It depends on your system and setup.
I'm trying to retrieve data from a PLC (AutomationDirect P2000). I have set up the PLC as the server with their software program (I can also connect to it with their software via Ethernet and use Wireshark to see it is in fact sending UDP packets to my machine at roughly every 200ms). I am trying to set up a very simple Python script to retrieve said data, without bothering to encode it or do anything with it, but my program hangs at the socket.recv(). Whenever I try to run it "Got here" will be printed, but "Now here" will not. From what I've read the fact that it hangs means there's no data to be received, but from my (limited) understanding of what I see on Wireshark this is not the case. I am pretty new to all of this and would appreciate any help.
I have tried using socket.recvfrom(), which produces the same result. I've also tried using socket.bind() instead of socket.connect() but I get a "The requested address is not valid in its context" exception. Additionally, I've tried playing around with various IPs and ports. For example, I've tried using IP = '' instead of the actual IP, and I've tried the source/destination information from Wireshark as what I try to bind or connect to, but nothing thus far has worked.
import socket
IP = '192.168.3.1'
PORT = 9999
BUFFER_SIZE = 4096
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect((IP, PORT))
while True:
print("Got here")
data = s.recv(BUFFER_SIZE)
print("Now here")
print(f"Received {data}")
I am expecting to get a print out of the packet in byte format, but instead the program is hanging. If I try socket.bind() instead of socket.connect() I get an error message reading "...line 8, in
s.bind((IP, PORT))
OSError: [WinError 10049] The requested address is not valid in its context"
you can't use bind like this, because the ip address does not belong to your PC.
when you connect to the server, it (the server) doesn't send anything, but you try to get data from the server, so the socket awaits until it gets data, and only then it will continue the execution (this is called a blocking function, since it blocks the execution until it finishes).
The issue was with how I set up the PLC as the server. The UDP data I was seeing on port 9999 wasn't the communications I was thinking it was, and was only the inherent communication between the PLC and the network via its proprietary program. For anyone curious, I am using a P2000 PLC from AutomationDirect and initially I set it up as an EtherNet/IP Adapter following one of their videos, but I had to use the Custom Protocol over Ethernet functionality provided in the "Communications" section.
I've written server and client programs with Python.
Server.py
import socket
sock = socket.socket (socket.AF_INET, socket.SOCK_STREAM)
host = socket.gethostname()
port = 5555
sock.bind((host, port))
sock.listen(1)
conn, addr = sock.accept()
data = "Hello!"
data = bytes(data, 'utf-8')
conn.send(data)
sock.close()
Client.py on Linux
import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
host = socket.gethostname()
port = 5555
sock.connect((host, port))
data = sock.recv(2048)
data = str(data, "utf-8")
print(data)
sock.close()
When I run the server and then the client on my local machine (a Linux Mint), it works correctly. I got "Hello!" in bash, and everything is fine. BUT when I ran my client program on another machine (a Windows 8) and ran it (previously I ran server on Linux, of course, and change IP address in client to my static Linux mint's IP) it says:
ConnectionRefusedError: [WinError 10061] No connection could be made
because the target machine actively refused it
client.py on Windows
import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
host = "here is my static ip"
port = 5555
sock.connect((host, port))
data = sock.recv(2048)
data = str(data, "utf-8")
print(data)
sock.close()
I must say that I had done port forwarding in my router settings on port 5555. Earlier, I had done same thing to port 80 and my own site worked correctly, but now it doesn't work to 5555 with Python sockets! Why? I can't get it! And one more thing: I tried to change the port to 80 in my server and client files, but it didn't work too. PLease, help.
You have to change the socket.gethostname() in the server script to the empty string (or just directly call socket.bind(('', port))).
Your problem is not in Python but in the usage of sockets generally. When you create a socket, you just prepare your process to receive/send some data from/to another process.
Server
The first step for creating a socket is to specify what kind of protocol will be used for communication between those processes. In your case it is the socket.AF_INET which is constant for use of IP protocol and the socket.SOCK_STREAM is specify reliable stream-oriented service. The reliable stream-oriented service means that you want to be sure that every single sent byte will be delivered to the other side and nothing can be lost during the communication (the underlying OS will use the TCP protocol for that). From this point we are using the IPv4 protocol (because we set the socket.AF_INET).
The second step is bind it to an address. The bind process assigns an address where you expect a client will join (with your socket's settings it's a IP address and the TCP port). Your PC has multiple IP address (well, at least two). It always has 127.0.0.1, which is called "callback" and works only when your applications communicate on the same PC (that is you Linux - Linux scenario in the question), and then you have your external IP address, for communication with other computers (let's pretend it is 10.0.0.1).
When you call socket.bind(('127.0.0.1', 5555)), you're setting the socket to listen only for communication from the same PC. If you call socket.bind(('10.0.0.1', 5555)), then the socket setting is ready to receive data targeted to the 10.0.0.1 address.
But what if you have 10 IPs or more and you want to receive everything (with the right TCP port)? For those scenarios you can leave the IP address in bind() empty, and it does exactly what you want.
With Python's version of bind(), you can also enter a "computer name" instead of the concrete IP. The socket.gethostname() call returns your computer's name. The problem is in the translation of "computer name" to the IP which Python performs behind your back. The translation has some rules but generally your "computer name" can be translated into any IP address which you have set on your computer. In your case, the your computer's name is converted into 127.0.0.1, and that's why communication works only between processes on the same computer.
After socket.bind(), you have the socket ready to use but it is still "inactive". The call to socket.listen() activates the socket and causes it to wait until it receives an attempted connection. When a socket receives a new connection request, it will put it into a queue and wait for processing.
That's what socket.accept() does. It pulls the connection request from the queue, accepts it, and establishes the stream (remember the socket.SOCK_STREAM when you set up the socket) between the server and the client. The new stream is actually a new socket, but is ready to communicate with other side.
What happened with the old socket? Well, it's still alive, and you can call socket.listen() again to get another stream (connection).
How is it possible to have multiple sockets on the same port?
Every connection within computer's network is defined by flow which is 5-item tuple of:
L4 protocol (usually TCP or UDP)
Source IP address
Source L4 port
Destination IP address
Destination L4 port
When you create a new connection with a client, the flow can look like this: (TCP, 192.168.0.1, 12345, 10.0.0.1, 55555). Just for clarification, the server's response flow is (TCP, 10.0.0.1, 55555, 192.168.0.1, 12345), but it isn't important for us. If you create another connection with a client, that it will differ at source TCP port (if you do it from another computer that it will differ also at the source IP). Only from this information you can distinguish every connection created to your computer.
When you create a server socket in your code and call socket.listen(), it listens for any flow with this pattern (TCP, *, *, *, 55555) (the * means "match everything"). So when you get a connection with (TCP, 192.168.0.1, 12345, 10.0.0.1, 55555), then socket.accept() creates another socket which works only with this one concrete flow while the old socket carries on accepting new connections which haven't yet been established.
When the operating system receives a packet, it looks in the packet and checks the flow. At this point, several scenarios can take place:
The packet's flow matches all 5 items exactly (without usage of *). Then the packet's content is delivered to the queue associated with that socket (you're reading the queue when you call socket.recv()).
The packet's flow matched socket with associated flow contains * then it is considered as new connection and you can call scoket.accept().
The operating system doesn't contain open socket which would match the flow. In that case the OS refuse connection (or just ignore the packet it depends on firewall settings).
Probably an example can clarify these scenarios. The operating system has something like a table where it maps flows to sockets. When you call socket.bind(), it will assign a flow to the socket. After the call, the table can look like this:
+=====================================+========+
| Flow | Socket |
+=====================================+========+
| (TCP, *, *, *, 55555) | 1 |
+-------------------------------------+--------+
When it receive a packet with flow (TCP, 1.1.1.1, 10, 10.0.0.1, 10) then it won't match any flow (last port won't match). So, the connection is refused. If it receives a packet with flow (TCP, 1.1.1.1, 10, 10.0.0.1, 55555), the packet is delivered to the socket 1 (because there is a match). The socket.accept() call creates a new socket and record in the table.
+=====================================+========+
| Flow | Socket |
+=====================================+========+
| (TCP, 1.1.1.1, 10, 10.0.0.1, 55555) | 2 |
+-------------------------------------+--------+
| (TCP, *, *, *, 55555) | 1 |
+-------------------------------------+--------+
Now you have 2 sockets for 1 port. Every received packet which matches the flow associated with the socket 2 also matches the flow associated with socket 1 (on the contrary, it does not apply). It's not a problem because the socket 2 has a preciser match (is doesn't use the *), so any data with that flow will be delivered to socket 2.
How to serve multiple connections
If you want to implement a "real" server, your application should be able to process multiple connections without restarting. There are 2 basic approaches:
Sequential processing
try:
l = prepare_socket()
while True:
l.listen()
s, a = socket.accept()
process_connection(s) # before return you should call s.close()
except KeyboardInterrupt:
l.close()
In this case, you can process only one client while others clients have to wait for accept. If the process_connection() takes too long, then others clients will timeout.
Parallel processing
import threading
threads = []
try:
l = prepare_socket()
while True:
l.listen()
s, a = socket.accept()
t = threading.Thread(target=process_connection, s)
threads.append(t)
t.start()
except KeyboardInterrupt:
for t in threads:
t.join()
l.close()
Now when you receive a new connection, it will create a new thread so that every connection is processed in parallel. The main disadvantage of this solution is that you have to solve common troubles with threading (like access to shared memory, deadlocks etc.).
Beware that the above snippets are only examples, and are not complete! For example, they don't contain code for graceful exit on unexpected exceptions.
Servers in Python
Python also contains a module called socketserver, which contains shortcuts for creating servers in Python. You can find examples of how to use it here.
Client
With the client, it's much more simpler than with the server. You just have to create a socket with some settings (same as server side), and then tell it where the server is (what its IP and TCP port are). This is accomplished through the socket.connect() call. As a bonus, it also establishes the stream between your client and server, so from this point you can communicate.
You can find more information about sockets at the Beej's Guide to Network Programming. It's written for usage with C, but the concepts are the same.
I was stuck with the same problem months ago and also wasn't able to do port forwarding. I found a way out of port forwarding Ngrock
For your information what Ngrock does is it is a useful utility to create secure tunnels to locally hosted applications using a reverse proxy. It is a utility to expose any locally hosted application over the web
For How To use it, please see the steps shown below :
If you are in Mac write this command in your terminal to download ngrock
brew install ngrok
For windows
choco install ngrok
After installing You need need to sign up on the Ngrok website
You will get your Ngrock authentication token then paste this command in the terminal
For Mac and Windows
ngrok config add-authtoken <token>
Now that Ngrock is all setup you can start a tunnel using
ngrok tcp <Your Port Number Used In Server.py>
ngrok tcp 5321
Note : Please Give the command inside the directory in which the Python Socket Server File in Located
That's it Your Socket can connect you any computer over the internet anywhere in the world
If You are still struggling to see the detailed explanation in this video
You can also refer ngrock documentation here
I am playing with my python server, but I'm through with using localhost and I want to go over the internet. My code thus-far is:
import socket
import threading
import socketserver
class ThreadedTCPRequestHandler(socketserver.BaseRequestHandler):
def handle(self):
data = self.request.recv(1024)
cur_thread = threading.current_thread()
response = "{}: {}".format(cur_thread.name, data)
self.request.sendall(b'worked')
class ThreadedTCPServer(socketserver.ThreadingMixIn, socketserver.TCPServer):
pass
def client(ip, port, message):
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((ip, port))
try:
sock.sendall(message)
response = sock.recv(1024)
print("Received: {}".format(response))
finally:
sock.close()
if __name__ == "__main__":
# Port 0 means to select an arbitrary unused port
HOST, PORT = "0.0.0.0", 9001
server = ThreadedTCPServer((HOST, PORT), ThreadedTCPRequestHandler)
ip, port = server.server_address
# Start a thread with the server -- that thread will then start one
# more thread for each request
server_thread = threading.Thread(target=server.serve_forever)
# Exit the server thread when the main thread terminates
server_thread.daemon = True
server_thread.start()
print("Server loop running in thread:", server_thread.name)
ip = '12.34.56.789' #Not my real ip address This is just to hide my ip
print(ip, PORT)
client(ip, PORT, b'Hello World 1')
#client(ip, port, b'Hello World 2')
#client(ip, port, b'Hello World 3')
server.shutdown()
When I run this i get the error:
Server loop running in thread: Thread-1
12.34.56.789 9001
Traceback (most recent call last):
File "C:/Python32/serverTesty.py", line 43, in <module>
client(ip, PORT, b'Hello World 1')
File "C:/Python32/serverTesty.py", line 18, in client
sock.connect((ip, port))
socket.error: [Errno 10061] No connection could be made because the target machine actively refused it
I know the port works because when I use canyouseeme.org on port 9001 when my program is running it says its active and working. So I think I just have my connection wrong somewhere.
ip = '12.34.56.789' #Not my real ip address, its the one i got from whatismyip.org
The first problem is that '12.34.56.789' isn't a valid IP address at all. Each component has to fit in 8 bits (0-255); 789 is impossible. But I assume that isn't the actual code you're running, because the output shows 12.45.29.122.
The second problem is that you're using an address that isn't your real address.
Your machine presumably has an internal IP address, that can only be accessed from your LAN. Then, your router has an external IP address. The router uses a technique called Network Address Translation to let each machine on your LAN pretend that external address belongs to them, when they're acting as clients (which is why whatismyip.org shows you that address). But that doesn't work when they're acting as servers.
If you think about it, there's really no way it could work. If you make an outbound connection, and someone replies, the router knows that the reply should go to your machine. But if someone just comes along and talks to the router out of the blue, how could it know which machine to send the connection to?
If you're trying to connect from inside the same LAN, there's a very easy solution: use the server's real internal address, not the router's external address.
If you need to connect from outside, you can't, without some extra work. There are four ways around this:
Give your machine a real publicly-addressable IP address (e.g., by putting it on the router's DMZ). This is generally not even an option for home users, and it's a bad option for people who don't know what they're doing (unless you want your machine to be part of someone's botnet by lunchtime).
Set up static port forwarding in your router's configuration. This is different for each router, but the idea is that you tell it "if someone comes looking for port 9001, always send them to machine 192.168.1.64".
Use UPnP to set up port forwarding dynamically.
Set up a NAT hole punching.
Options 3 and 4 are more complex, and I think option 2 is the one you want, so I won't explain them.
On top of all that:
HOST, PORT = "192.168.1.64", 9001
server = ThreadedTCPServer((HOST, PORT), ThreadedTCPRequestHandler)
You've told the server explicitly "listen on 192.168.1.64". Even if you put your server machine on the DMZ, so it had addresses 192.168.1.64 and 12.45.29.122, your program is only listening for connections on the first one, so nobody would be able to reach it using the second. If you want to listen on all addresses, use 0.0.0.0.
In the edited version, you're now listening on 0.0.0.0, and connecting to the router's public IP, and you claim to have set up port forwarding on the router, and you're still getting the exact same error.
If that's all correct, there are three obvious things that could be going wrong:
You're not actually port forwarding; something is wrong with the setup.
You're not actually listening on 0.0.0.0:9001.
You've got a firewall blocking the connection.
There are a few tests you can do to narrow things down.
Open two terminals. In one, type nc -kl 9001. In the other, type nc 12.34.56.78 9001. They should connect up, so anything you type into one window appears in the other (maybe only after you hit Return). If that works, the port forwarding is working, and there's no firewall problem, so it's a problem in your code.
If that didn't work, please post exactly what you saw in each window. Then Ctrl-C the second nc, and type nc 192.168.1.64 9001. If that now works, either the port forwarding isn't set up right, unless you have a clever firewall that allows same-host (or same-interface) connections but not remote connections.
If neither one worked, it's probably a firewall problem. (Unless you're wrong about your IP addresses or something.) You can probably find logs somewhere, but without knowing what platform you're on and what firewall you're using it's hard to offer much help. (Also, that's probably a problem for a different site than SO.)
If you're on Windows, or some linux distros, you need to get a copy of nc (netcat) from somewhere; on most linux distros, and Mac, it should be built in. Also, GNU, BSD, and Hobbit nc are slightly different, so if nc -kl 6000 gives you an error, you might have to read the man page or --help. (If I remember right, Hobbit nc requires -l -p6000, BSD requires -l 6000, GNU allows either.)
Or you may want ncat, a re-implementation of netcat that I know can handle the syntax I used above, and has a single-file static executable for Windows.
If you can't get started with nc, at least try changing your code to connect to 192.168.1.64 instead of 12.34.56.78. If that fixes the problem, at least you'll know it's either port forwarding or a firewall that allows same-host/interface connections but not remote.
I am writing 2 small programs (a server and a client) and whenever I run both, and have the client connect to the server, the server output says that I am connected on a port of which I didn't bind in the code. I binded both the server and the client socket to the localhost and port 8000, but every time the server is connected to by the client, it says that the client is connected on port 52304 or some other number larger than 50000, shouldn't it at least be a constant port number even if it isn't the one I bound it to? Also, I know, that if I run the server program more than once in the same terminal, even if I exited the program, the port is still taken, so I usually run the server, quit, then exit the terminal, which usually solves that problem. That is another note I should make, when I do run the server program the second time in the same terminal, it recognizes I am trying to bind to port 8000 and the program wont run, then when it does it chooses some random port.
Here is my server code:
import socket
import os
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind('', 8000)
s.listen(5)
while 1:
client,addr = s.accept()
print "Accepted a connection from: ", addr
data = client.recv(1024)
client.send("You said: " + data)
The port number it's reporting is the one the client connected from. And it will be a random port number. If (as your question seems to imply) you have a bind call in the client that looks just like the one in the server, then I'm surprised it's succeeding since the server has already bound itself to that port and only one thing can be bound to a given port at a time.
Please post your client code. Contrary to what your question implies, I don't think that you are binding to a port on the client side. I'm betting you're just connecting. Now, that, generally speaking, is what you're supposed to be doing. So the fact you're confused by the results just means that you don't really understand what's happening exactly. The results you're seeing are perfectly expected and normal.
Here is an explanation of what's going on:
A TCP connection is uniquely identified (globally unique, as in no other TCP connections in the entire world will have the same identifier (though this isn't really exactly true with NAT and private IP ranges)) by these 4 pieces of information:
client ip
client port #
server ip
server port #
When your server is reporting a connection, it's printing out the first two values because they are what is returned by the accept call. When you are doing a bind call in the server, you are specifying values 3 and 4. The OS generally picks values 1 and 2 for the client automatically when it does a connect call.
A client normally does not bind to a port (though it can). It normally lets the OS pick a port for it. The client's OS will pick a port number from a list of unused port numbers. In your connect call on the client side, you are giving values 3 and 4 (the values specified in the bind call on the server side). The OS should automatically assign your client values 1 and 2 for you.
Think about it like the sender and recipient address on an envelope. The accept call on the server side reports the sender address because presumably the server already knows its own address. The client is most concerned with the recipient address (the address of the server) and lets a clerk (the OS) just paste on a return address,
The port and socket that the server listens on is not the same socket that is used once the connection is established. The accept call creates a new socket when a client connects for sending and receiving data. Otherwise if it used the same socket...then no other clients would be able to connect.
You also need to properly close your socket so it does not hang around after your program terminates.
You can never bind the same port to more then one program, the port the server gives you is the port the client wants you to send the data over. I think its to avoid connection collisions.
So you don't have to worry about the ports if the connection is establish.
But if you want the server to be able to receive more the one connection take a look at this:
Multiple simultaneous network connections - Telnet server, Python