Develop Reference

How can I create a TCP connection in Python between 2 PCs

So far I have made a VERY basic client/server application that creates a TCP connection. I have a lot of programming experience, just never did this low-level stuff and especially nothing with networks. Note that all the prints are just to help me figuring out what is going on. One of the known issues is that jsonip sometimes gives me an IPv4 and sometimes v6, I don't know why but that doesn't matter for now, just to warn anyone who wants to recreate my code.
Server:
import socket
import requests
port = int(input("Enter port you want to open:\n"))
#todo: add errorhandling
print("Adding socket...")
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
hostname = socket.gethostname()
print(f"Hostname: {hostname}")
ip_address = socket.gethostbyname(hostname)
print(f"Host address: {ip_address}")
r = requests.get(r'http://jsonip.com')
public_ip_address = r.json()['ip']
s.bind((ip_address, port))
print("Is open for connections on IP: "+public_ip_address+" and Port: "+str(port))
s.listen(5)
print("Done initialisation, listening for incoming connections...")
while True:
clientsocket, address = s.accept()
print(f"Connection from {address} has been established")
clientsocket.send(bytes(f"You have connected to server: {hostname}", "utf-8"))
Client:
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
ip = input("Enter IP to connect to:\n")
port = int(input("Enter Port to connect to:\n"))
print(f"Connecting to server {ip} ...")
s.connect((ip, port))
msg = s.recv(1024)
print(msg.decode("utf-8"))
On my local machine: Open 20000 in my server.py, it tells me the host is 127.0.1.1, I then enter 127.0.1.1 into my client script and 20000, and they connect. So the Socket has been bound with the 127.0.1.1. (Side question: What is this IP address, is it like the internal IP address of processes in my PC or something? If running ip a on my other machine it is the first one shown of 2)
Using Virtmanager on my machine and running one Linux Server (command line only) and one normal Ubuntu, the server tells me the host is, again, 127.0.1.1 which I don't need to enter into the other VM to know it won't work, what does work however, is getting the IP-address of the Server via ip a, which in this case is 192.168.122.37, and when I enter this IP address into the client, it connects. But in the socket here I bind, again, the 127.0.1.1, so is it arbitrary what I put here? What SHOULD I bind here, the public, the weird or the 192. address?
The first thing I could not get to work was using 2 physical devices. When opening a server on my Linux machine, I cannot connect with my windows machine at all, no matter if I use my public, my 127. or my 192. IP-address. Now my end goal is doing this over the internet so I am walking myself up, describing here the steps I took to try and get where I want to be but here I hit a brick wall where I don't know what is wrong. Am I binding the wrong address on the server, is my router being a problem, is there something else wrong?
I also tried leaving my network using my friends pc a few countries over, but this also just results in a timeout (my theory is that the Router port he is trying to open is closed and I have now idea how I can make the router send data to his PC, which should be not impossible as firefox and every application using internet does it without me having to manually forward every port, I just don't know how). This is my end goal, creating a connection between my friends PC and mine, and this is how far I got (I wouldn't mind skipping the local network if it is not relevant for fixing the global connection problem), so, tl;dr: what did i do wrong, what do i need to bind and what do i need to do for the final result to work?

There are many questions to answer.
Addresses 127.X.X.X are reserved for the loopback interface, most common one is 127.0.0.1. The loopback is a virtual, but important interface and as you have probably guessed, it is usable on the local machine only. You cannot use 127.X.X.X address to make two hosts to communicate with each other.
Addresses 192.168.X.X (and also 10.X.X.X and 172.16-31.X.X.) are reserved for local LANs. They are not valid on the Internet.
You cannot use these addresses to make two hosts to communicate with each other over the public Internet (unless you create a tunnel, an advanced networking topic)
Almost everybody uses them, because we ran out of IPv4 addresses long time ago, they were difficult to get, expensive, etc. Also such hosts are isolated from the Internet, they can be reached only via a router that translates addresses. Such router feature is called NAT. A typical router has one valid Internet address and all connections to the Internet appear as coming from the router. If you contant a service like jsonip.com from a PC, you get your router's address, not your PC's address.
See also: Finding local IP addresses using Python's stdlib
To make your program working, make it to accept connections on all interfaces. See the first example in the socket docs. On Linux, use port numbers >= 1024. Ports < 1024 are reserved, not available to regular users.
Final point is that a firewall may prevent connections to your server. It depends on your system and setup.

Python socket listening to specific interface

I currently have a raspberry pi that is set up as a wifi to ethernet bridge. The raspberry pi acts as an access point to the entire ethernet subnet that I have. The subnet and the network bridge work perfectly but when I try to get my python program on the raspberry pi to listen to requests on the ethernet interface/subnet it doesn't seem to do that. It is set to bind the socket to ('',10000) but it never receives any messages. However, it is able to send messages via sockets to the subnet just fine, just not receive. I think it is listening to the wifi interface rather than the ethernet one but I'm not sure how to specify which interface the socket is suppose to listen to.
here is my receiving code
receive_group = ('',10000)
receive = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
receive.bind(receive_group)
while(True):
data, address = receive.recv(65536)
print(data)

The bind part should be correct. The receive part is wrong because recv only return the (bytes) data, you should use recvfrom to also get the sender address. But this should work:
import socket
receive = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
receive.bind(('', 10000))
while True:
data, address = receive.recvfrom(64)
print(data, address)
I used this code for the send part:
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.sendto(b'foo', (addr, 10000))
where addr is one of the (reachable) addresses of the receiver, and the receiver could successfully receive data from any of its interfaces.

To get something from socket.recv something must connect to this socket and send something. Are you sure that some program on the network is doing this?
For listening/sniffing to packets and network traffic, better use pyshark.

Turns out it wasn't anything with python. When I created the access point on the pi it created a firewall rule that blocked that port even though I never configured the firewall that way. Adding an exception to that port fixed my problem

how to connect two different computers in two different networks

this code below is working fine between two computers on the same network but it doesn't work between two computers on two different networks and i have tried to use the public IP address
clint
import socket
s = socket.socket()
port = 10000
ip=input("what is ip : ")
s.connect((ip, port))
print (s.recv(1024))
s.close
server
import socket
s = socket.socket()
ip=socket.gethostbyname(socket.gethostname())
print (ip)
port = 10000
s.bind((ip, port))
s.listen(1)
while True:
c, addr = s.accept()
print ('Got connection from', addr)
c.send(bytes([int(1)]))
c.close()
this question have already been asked many times but all the answers i could found is that i have to do " port forwarding " with out any code examples how to do it in some of the answers they say i should download application that will do the port forwarding to me but i don't know how to use it then in my python code ?

Port forwarding is something you do at the router level. For example if the two computers you are trying to connect are behind two different WiFi routers at two different homes then you need to set both WiFi routers in both locations to forward port 10000 (the one you're using) to forward to the internal IP of the computer where your code is running.
Your WiFi router has a public IP (what you're probably using now) and your computers running your python code have local IPs internal to the network that are assigned by the WiFi routers. Port forwarding takes traffic going to the public IP of a router and forwards it internally in the local network to the computer at some local IP that runs your program.
In other words this is not something you will do in your code — this is something you will do in your router's software.

If you are feeling ambitious you can follow this tutorial that removes the need to configure your devices for port forwarding. Assuming you have uPNP capable routers.
https://www.electricmonk.nl/log/2016/07/05/exploring-upnp-with-python/
And there may even be a module that takes all the leg work out.(not checked how it works)
https://pypi.python.org/pypi/UPnP/1.3

Python sockets/port forwarding

I've written server and client programs with Python.
Server.py
import socket
sock = socket.socket (socket.AF_INET, socket.SOCK_STREAM)
host = socket.gethostname()
port = 5555
sock.bind((host, port))
sock.listen(1)
conn, addr = sock.accept()
data = "Hello!"
data = bytes(data, 'utf-8')
conn.send(data)
sock.close()
Client.py on Linux
import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
host = socket.gethostname()
port = 5555
sock.connect((host, port))
data = sock.recv(2048)
data = str(data, "utf-8")
print(data)
sock.close()
When I run the server and then the client on my local machine (a Linux Mint), it works correctly. I got "Hello!" in bash, and everything is fine. BUT when I ran my client program on another machine (a Windows 8) and ran it (previously I ran server on Linux, of course, and change IP address in client to my static Linux mint's IP) it says:
ConnectionRefusedError: [WinError 10061] No connection could be made
because the target machine actively refused it
client.py on Windows
import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
host = "here is my static ip"
port = 5555
sock.connect((host, port))
data = sock.recv(2048)
data = str(data, "utf-8")
print(data)
sock.close()
I must say that I had done port forwarding in my router settings on port 5555. Earlier, I had done same thing to port 80 and my own site worked correctly, but now it doesn't work to 5555 with Python sockets! Why? I can't get it! And one more thing: I tried to change the port to 80 in my server and client files, but it didn't work too. PLease, help.

You have to change the socket.gethostname() in the server script to the empty string (or just directly call socket.bind(('', port))).
Your problem is not in Python but in the usage of sockets generally. When you create a socket, you just prepare your process to receive/send some data from/to another process.
Server
The first step for creating a socket is to specify what kind of protocol will be used for communication between those processes. In your case it is the socket.AF_INET which is constant for use of IP protocol and the socket.SOCK_STREAM is specify reliable stream-oriented service. The reliable stream-oriented service means that you want to be sure that every single sent byte will be delivered to the other side and nothing can be lost during the communication (the underlying OS will use the TCP protocol for that). From this point we are using the IPv4 protocol (because we set the socket.AF_INET).
The second step is bind it to an address. The bind process assigns an address where you expect a client will join (with your socket's settings it's a IP address and the TCP port). Your PC has multiple IP address (well, at least two). It always has 127.0.0.1, which is called "callback" and works only when your applications communicate on the same PC (that is you Linux - Linux scenario in the question), and then you have your external IP address, for communication with other computers (let's pretend it is 10.0.0.1).
When you call socket.bind(('127.0.0.1', 5555)), you're setting the socket to listen only for communication from the same PC. If you call socket.bind(('10.0.0.1', 5555)), then the socket setting is ready to receive data targeted to the 10.0.0.1 address.
But what if you have 10 IPs or more and you want to receive everything (with the right TCP port)? For those scenarios you can leave the IP address in bind() empty, and it does exactly what you want.
With Python's version of bind(), you can also enter a "computer name" instead of the concrete IP. The socket.gethostname() call returns your computer's name. The problem is in the translation of "computer name" to the IP which Python performs behind your back. The translation has some rules but generally your "computer name" can be translated into any IP address which you have set on your computer. In your case, the your computer's name is converted into 127.0.0.1, and that's why communication works only between processes on the same computer.
After socket.bind(), you have the socket ready to use but it is still "inactive". The call to socket.listen() activates the socket and causes it to wait until it receives an attempted connection. When a socket receives a new connection request, it will put it into a queue and wait for processing.
That's what socket.accept() does. It pulls the connection request from the queue, accepts it, and establishes the stream (remember the socket.SOCK_STREAM when you set up the socket) between the server and the client. The new stream is actually a new socket, but is ready to communicate with other side.
What happened with the old socket? Well, it's still alive, and you can call socket.listen() again to get another stream (connection).
How is it possible to have multiple sockets on the same port?
Every connection within computer's network is defined by flow which is 5-item tuple of:
L4 protocol (usually TCP or UDP)
Source IP address
Source L4 port
Destination IP address
Destination L4 port
When you create a new connection with a client, the flow can look like this: (TCP, 192.168.0.1, 12345, 10.0.0.1, 55555). Just for clarification, the server's response flow is (TCP, 10.0.0.1, 55555, 192.168.0.1, 12345), but it isn't important for us. If you create another connection with a client, that it will differ at source TCP port (if you do it from another computer that it will differ also at the source IP). Only from this information you can distinguish every connection created to your computer.
When you create a server socket in your code and call socket.listen(), it listens for any flow with this pattern (TCP, *, *, *, 55555) (the * means "match everything"). So when you get a connection with (TCP, 192.168.0.1, 12345, 10.0.0.1, 55555), then socket.accept() creates another socket which works only with this one concrete flow while the old socket carries on accepting new connections which haven't yet been established.
When the operating system receives a packet, it looks in the packet and checks the flow. At this point, several scenarios can take place:
The packet's flow matches all 5 items exactly (without usage of *). Then the packet's content is delivered to the queue associated with that socket (you're reading the queue when you call socket.recv()).
The packet's flow matched socket with associated flow contains * then it is considered as new connection and you can call scoket.accept().
The operating system doesn't contain open socket which would match the flow. In that case the OS refuse connection (or just ignore the packet it depends on firewall settings).
Probably an example can clarify these scenarios. The operating system has something like a table where it maps flows to sockets. When you call socket.bind(), it will assign a flow to the socket. After the call, the table can look like this:
+=====================================+========+
| Flow | Socket |
+=====================================+========+
| (TCP, *, *, *, 55555) | 1 |
+-------------------------------------+--------+
When it receive a packet with flow (TCP, 1.1.1.1, 10, 10.0.0.1, 10) then it won't match any flow (last port won't match). So, the connection is refused. If it receives a packet with flow (TCP, 1.1.1.1, 10, 10.0.0.1, 55555), the packet is delivered to the socket 1 (because there is a match). The socket.accept() call creates a new socket and record in the table.
+=====================================+========+
| Flow | Socket |
+=====================================+========+
| (TCP, 1.1.1.1, 10, 10.0.0.1, 55555) | 2 |
+-------------------------------------+--------+
| (TCP, *, *, *, 55555) | 1 |
+-------------------------------------+--------+
Now you have 2 sockets for 1 port. Every received packet which matches the flow associated with the socket 2 also matches the flow associated with socket 1 (on the contrary, it does not apply). It's not a problem because the socket 2 has a preciser match (is doesn't use the *), so any data with that flow will be delivered to socket 2.
How to serve multiple connections
If you want to implement a "real" server, your application should be able to process multiple connections without restarting. There are 2 basic approaches:
Sequential processing
try:
l = prepare_socket()
while True:
l.listen()
s, a = socket.accept()
process_connection(s) # before return you should call s.close()
except KeyboardInterrupt:
l.close()
In this case, you can process only one client while others clients have to wait for accept. If the process_connection() takes too long, then others clients will timeout.
Parallel processing
import threading
threads = []
try:
l = prepare_socket()
while True:
l.listen()
s, a = socket.accept()
t = threading.Thread(target=process_connection, s)
threads.append(t)
t.start()
except KeyboardInterrupt:
for t in threads:
t.join()
l.close()
Now when you receive a new connection, it will create a new thread so that every connection is processed in parallel. The main disadvantage of this solution is that you have to solve common troubles with threading (like access to shared memory, deadlocks etc.).
Beware that the above snippets are only examples, and are not complete! For example, they don't contain code for graceful exit on unexpected exceptions.
Servers in Python
Python also contains a module called socketserver, which contains shortcuts for creating servers in Python. You can find examples of how to use it here.
Client
With the client, it's much more simpler than with the server. You just have to create a socket with some settings (same as server side), and then tell it where the server is (what its IP and TCP port are). This is accomplished through the socket.connect() call. As a bonus, it also establishes the stream between your client and server, so from this point you can communicate.
You can find more information about sockets at the Beej's Guide to Network Programming. It's written for usage with C, but the concepts are the same.

I was stuck with the same problem months ago and also wasn't able to do port forwarding. I found a way out of port forwarding Ngrock
For your information what Ngrock does is it is a useful utility to create secure tunnels to locally hosted applications using a reverse proxy. It is a utility to expose any locally hosted application over the web
For How To use it, please see the steps shown below :
If you are in Mac write this command in your terminal to download ngrock
brew install ngrok
For windows
choco install ngrok
After installing You need need to sign up on the Ngrok website
You will get your Ngrock authentication token then paste this command in the terminal
For Mac and Windows
ngrok config add-authtoken <token>
Now that Ngrock is all setup you can start a tunnel using
ngrok tcp <Your Port Number Used In Server.py>
ngrok tcp 5321
Note : Please Give the command inside the directory in which the Python Socket Server File in Located
That's it Your Socket can connect you any computer over the internet anywhere in the world
If You are still struggling to see the detailed explanation in this video
You can also refer ngrock documentation here

I can't connect to socket from the outside

I am trying to make a simple server/client program pair.
On LAN they work fine, but when i try to connect from the "outside" it says connection refused. I shut down firewalls on both machines but i am still unable to connect, and i double checked the ip.
What am i doing wrong?
Thanks
Jake
Code:
import socket
host = ''
port = 9888
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((host,port))
s.listen(1)
conn, adrr = s.accept()
conn.send("Hello, world!")
s.close()
Client:
import socket
host = '68.x.x.x'
port = 9888
s = socket.socket(socket.AF_INET, socket_SOCK_STREAM)
s.connect((host,port))
print s.recv(200)
s.close()

You have one of two possible issues.
Erroneous network configuration
Bug(s) in code
The way to debug this is to try and rule one out. If we can get rid of the Code issue then we know it is a network issue.
Get a Socket Server and client that you know works and then try them as standalone programs. inside and outside of the firewall.
Go to this site and download the examples. Change the ports in both the client and the server, compile and run them. First on same machine within network, second from two machines on same network and then server from within and client from outside of network.

How's the argument you're passing to the .bind call for your server socket? That's the single likeliest cause -- e.g. if you're using 192.168.x.y for whatever values of x and y, or 10.x.y.z likewise, that's a local-network address only, not routed by inter-network routers by internet conventions (most routers can be programmed to forward some incoming packets to a specific local-network address, typically depending on ports, but that's very specific to router's brands and models).