Basically what I'm trying to achieve is a program which allow users to connect to a each other over a network in, essentially, a chat room. What I'm currently struggling with is writing the code so that the users can connect to each other without knowing the IP-address of the computer that the other users are using or knowing the IP-address of a server.
Does anyone know of a way in which I could simply have all of the users scan the IP range of my network in order to find any active 'room' and then give the user a chance to connect to it?
Also, the hope is that there will be no need for a central server to run this from, rather every user will simply be connected to all other user, essentially being the server and client at the same time.
I can give you two suggestions. First of all, UDP packets to the broadcast address of your network will be received by everybody. Secondly, there is a protocol for programs offering certain services to find each other on a local network. That protocol is called mDNS, ZeroConf, or Bonjour.
Using broadcast UDP is likely going to be the faster route. But if I were you, I'd learn how to use ZeroConf instead. It's supported well under IPv6 and already used by several interesting programs such as SubEthaEdit and Gobby.
Here is a link to a nice tutorial for implementing something that speaks ZeroConf in Python.
Another recommendation... If you want to hand roll your own broadcast/multicast UDP code and you can be sure that all of the systems you're on are running a Linux that's newer than 2003 or so, and all the Windows systems are XP or better, you can probably get away with using IPv6. The IPv6 link-local (think same LAN) all hosts multicast address is ff02::1. That's really simple and easy, and it will reach all the other systems on the same LAN. It's much better than having to figure out what your network's broadcast address is with IPv4.
Related
I am attempting to create a basic chatroom in python, and I would like to know how I could transmit data from one script to another, preferably without using google drive. If needed to, I could create a webserver on Replit, but I don't do well with HTML or PHP.
Side note: I can't port forward, as my google wifi doesn't accept any level of port forwarding.
I would send messages of about 50 characters every couple seconds
Since you mention port forwarding, I assume you want two chat clients that run on different local networks to talk to each other, for example your own and the chat client of a friend in a remote location, over the internet.
If you (or your counterpart) cannot set up port forwarding, then direct communication between the script on your computer and theirs is hard, if not impossible. The solution is to set up a third computer or service on the internet that can be reached by both clients and use it for relaying messages between them.
A network is typically protected by a firewall of sorts and will typically be behind a router that performs network address translation (NAT) to help multiple devices on a network to simultaneously access services on the internet, whilst all using the same IP address on the internet. Port forwarding fits into that by connecting a specific port from the outside directly to a port on a machine on the inside - without that, an outside computer might be able to reach your IP address, but they could never connect to a computer or program on the inside of the network, as the router wouldn't know what computer to contact, also the firewall might disallow the connection to begin with.
But if your computer on the inside establishes a connection with an accessible server on the internet, expecting a response, that creates a temporary conduit through the router and firewall that can be used by the server to send messages (look up 'hole punching' for more information). And if both computers do this, the server can relay message between both clients. Only the server then needs to run in an environment that doesn't have firewall restrictions or NAT that prevent this.
You could write a simple Python server, that accepts incoming connections and can send several responses and a simple client that connects to it, identifying itself and joining a chatroom, or having a direct conversation with another connected client. There are many techniques that would allow you to do this, but I think web sockets might be a good starting point, as long as you don't plan to do advanced fast or high volume stuff that would require something like a UDP connection.
A library like websockets could be a good starting point, but you may want to start out by figuring out where you would have this service hosted first, since there may be limitations on what you're able and allowed to do.
Also, if all you're looking to do is send simple messages, you may want to stay away from writing your own server an protocols at all - have a look around for open source message servers written in a language you are comfortable with, or that just work out of the box without any development, in which case the language doesn't even really matter, as long as you can connect to it and exchange messages from Python.
I'm developing a client-server game in python and I want to know more about port forwarding.
What I'm doing for instance is going to my router (192.168.0.1) and configure it to allow request for my real IP-adress to be redirected to my local adress 192.168.0.X. It works really well. But I'm wondering if I can do it by coding something automatically ?
I think skype works like a kind of p2p and I can see in my router that skype is automatically port forwarded to my pc adress. Can I do it in Python too?
There are different solutions here, but most are not trivial, and you'll have to do some reading, and you'll need some kind of fallback.
UPnP/IGD is the simplest. If your router supports it, and is configured to allow it, and you know how to write either low-level networking code or old-fashioned SOAP web service code, you can ask the router to assign you a port. If it responds with success, start using that port and you're basically done.
If you can run a (very low-bandwidth) server with a public address for all of your users, Hole punching may solve the problem.
Think about how a behind-the-NAT client talks to a public server. You make a request to some IP and port, but the server is seeing your router's IP, not yours (which is a good thing, because yours isn't accessible). When it replies, your router has to know to forward it to you—which it does just by remembering that you're the behind-the-NAT client that just sent a request to that server.
What if, instead of talking to a public server, you talk to some other peer behind his own separate NAT? Well, Your router doesn't know the difference; as long as you get a response from the same place, it'll get through. But how do you get a response, when your message isn't going to get through his NAT? He does the same thing, of course. One of the messages may get lost, but the other one will get through, and then you're both set and can communicate. You will need to send keep-alives regularly so the router doesn't forget you were communicating, but other than that, there's really nothing tricky.
The only problem is that you need to know the public IP address for the other peer, and the port that he's expecting you to come from, and he needs to know the same about you. That's why you need a server—to act as an introducer between peers.
Hole punching will work with UDP from most home networks. It won't work with TCP from many home networks, or with either UDP or TCP from many corporate networks. (Also, in corporate networks, you may have multiple layers of NATs, which means you need introducers at every interface rather than just one on the internet, or symmetric NATs, which can't be punched.)
You can use STUN (or similar) services like ICE or TURN. This only works if there is an ICE, TURN, etc. service to use—which is generally not the case for two peers on different home NATs, unless you deploy your own server and build an introducer to help out, and if you're going to do that, you can just use hole punching. But in a corporate environment, this can be the best way to provide connectivity for P2P apps.
Finally, you can make the user configure port-forwarding manually and enter the forwarded port number into your problem. This is not ideal, but you should always provide it as a fallback (except maybe for apps meant only for corporate deployment), because nothing else is going to work for all of your users.
I believe Skype uses all of these. If you enable UPnP, it tries to IGD your router. Or you can configure it to use a TURN server. Or you can just enter a specific port that you've forwarded manually. If you do none of the above, it tries to use UDP hole punching, with an introducer that Skype runs.
So your application needs to do TCP/UDP networking if I understand correctly. That means that at least one of the connecting clients needs a properly open port, and if both of them is behind NAT (a router) and have no configured open ports, your clients cannot connect.
There are several possible solutions for this, but not all are reliable: UPnP, as suggested here, can open ports on demand but isn't supported (or enabled) on all routers (and it is a security threat), and P2P solutions are complex and still require open ports on some clients.
The only reliable solution is to have a dedicated server that all clients can connect to that will negotiate the connections, and possibly proxy between them.
You could look at something like this (assuming your router supports it): http://en.wikipedia.org/wiki/Universal_Plug_and_Play#NAT_traversal
For implementing port forwarding using python, there's a fantastic ActriveState recipe that does asynchronous port forwarding server using only Python standard library (socket, syncope). Look at
http://code.activestate.com/recipes/483732-asynchronous-port-forwarding/
I made some python script and ran it both on my computer and some distant shell (some website that provides shell access).
I'm using threads, pipes and UDP sockets to transfer data in a P2P fashion, so each script can both receive and send through the same socket. To test if this works, I open one terminal on my computer and one other terminal with ssh, connected to my shell. I make sure the script is the same on both machines, and feed it with an ip address.
Here is the script: http://codepad.org/V9Q1KcDT
(I don't know if I should paste it directly here or not)
My problem is this: strings I send seems to land something 20% of the time, sometimes often, sometimes not, and it seems to be random...
What am I doing wrong ?
Are UDP so unreliable ?
Are python thread+pipe+socket too slow ?
Could it be some kind of network problem with my shell provider ?
Is my program flawed ?
Are pipes a good solution to communicate with threads ?
I have no problem not using a shell and I have not tried, but it's useful for testing purposes.
BTW if i'm behing a router, how does the router knows where to send the packet if I'm not the only computer connected ? (I tried when I was the only one, it behaved identically).
User Datagram Protocol(UDP) could easily stand for Unreliable Datagram Protocol. UDP provides no guarantees regarding delivery so if you need reliability you have to implement it yourself by resending messages.
That said, threading and multiprocessing and remote machines and networks are a lot of variables to diagnose at once. I'd advise you to step back and try to boil the problem down to something simpler.
For instance:
first try TCP instead of UDP
instead of threads, run 2 processes (one doing nit_send, one doing nit_recv)
run them both on your local machine (use localhost: 127.0.0.1)
Once you have this basic test working then add features back in. For instance, switch to UDP. Once you've got your UDP issues sorted out locally try introducing remote machines or threads, etc.
Regarding the router question all machines connected to your router are most likely NAT'd so they all appear to have the same IP address (the IP address of your router) to machines on the internet. Look into NAT and port-forwarding if you want to route traffic to a specific machine on your local subnet.
I want to write a Python script that will check the users local network for other instances of the script currently running.
For the purposes of this question, let's say that I'm writing an application that runs solely via the command line, and will just update the screen when another instance of the application is "found" on the local network. Sample output below:
$ python question.py
Thanks for running ThisApp! You are 192.168.1.101.
Found 192.168.1.102 running this application.
Found 192.168.1.104 running this application.
What libraries/projects exist to help facilitate something like this?
One of the ways to do this would be the Application under question is broadcasting UDP packets and your application is receiving that from different nodes and then displaying it. Twisted Networking Framework provides facilities for doing such a job. The documentation provides some simple examples too.
Well, you could write something using the socket module. You would have to have two programs though, a server on the users local computer, and then a client program that would interface with the server. The server would also use the select module to listen for multiple connections. You would then have a client program that sends something to the server when it is run, or whenever you want it to. The server could then print out which connections it is maintaining, including the details such as IP address.
This is documented extremely well at this link, more so than you need but it will explain it to you as it did to me. http://ilab.cs.byu.edu/python/
You can try broadcast UDP, I found some example here: http://vizible.wordpress.com/2009/01/31/python-broadcast-udp/
You can have a server-based solution: a central server where clients register themselves, and query for other clients being registered. A server framework like Twisted can help here.
In a peer-to-peer setting, push technologies like UDP broadcasts can be used, where each client is putting out a heartbeat packet ever so often on the network, for others to receive. Basic modules like socket would help with that.
Alternatively, you could go for a pull approach, where the interesting peer would need to discover the others actively. This is probably the least straight-forward. For one, you need to scan the network, i.e. find out which IPs belong to the local network and go through them. Then you would need to contact each IP in turn. If your program opens a TCP port, you could try to connect to this and find out your program is running there. If you want your program to be completely ignorant of these queries, you might need to open an ssh connection to the remote IP and scan the process list for your program. All this might involve various modules and libraries. One you might want to look at is execnet.
My app opens a TCP socket and waits for data from other users on the network using the same application. At the same time, it can broadcast data to a specified host on the network.
Currently, I need to manually enter the IP of the destination host to be able to send data. I want to be able to find a list of all hosts running the application and have the user pick which host to broadcast data to.
Is Bonjour/ZeroConf the right route to go to accomplish this? (I'd like it to cross-platform OSX/Win/*Nix)
it can broadcast data to a specified host on the network
This is a non-sequitur.
I'm presuming that you don't actually mean broadcast, you mean Unicast or just "send"?
Is Bonjour/ZeroConf the right route to go to accomplish this?
This really depends on your target environment and what your application is intended to do.
As Ignacio points out, you need to install the Apple software on Windows for Zeroconf/mDNS to work at the moment.
This might be suitable for small office / home use.
However larger networks may have Layer 2 Multicast disabled for a variety of reasons, at which point your app might be in trouble.
If you want it to work in the enterprise environment, then some configuration is required, but that doesn't have to be done at the edge (in the app client instances).
Could be via a DHCP option, or by DNS service records.. in these cases you'd possibly be writing a queryable server to track active clients.. much like a BitTorrent Tracker.
Two things to consider while designing your networked app:
Would there ever be reason to run more than one "installation" of your application on a network?
Always consider the implications of versioning: One client is more up to date than another, can they still talk to each other or at least fail gracefully?
Zeroconf/DNS-SD is an excellent idea in this case. It's provided by Bonjour on OS X and Windows (but must be installed separately or as part of an Apple product on Windows), and by Avahi on FOSS *nix.
I think that ZeroConf is a very good start. You may find this document useful.
I have a list on a webpage, nice if you need internet communications.
<dl_service updated="2010-12-03 11:55:40+01:00">
<client name="internal" ip="10.0.23.234" external_ip="1.1.1.1"/>
<client name="bigone" ip="2.2.2.2" external_ip="2.2.2.2">
<messsage type="connect" from="Bigone" to="internal" />
</client>
</dl_service>
My initial idea was to add firewall punching and all that, but I just couldn't be bothered too many of the hosts where using external IPs for it to be a problem..
But I really recommend Zeroconf, at least if you use Linux+MacOSX, don't know about Windows at all.