Proper way to publish and find services on a LAN using Python - python

My app opens a TCP socket and waits for data from other users on the network using the same application. At the same time, it can broadcast data to a specified host on the network.
Currently, I need to manually enter the IP of the destination host to be able to send data. I want to be able to find a list of all hosts running the application and have the user pick which host to broadcast data to.
Is Bonjour/ZeroConf the right route to go to accomplish this? (I'd like it to cross-platform OSX/Win/*Nix)

it can broadcast data to a specified host on the network
This is a non-sequitur.
I'm presuming that you don't actually mean broadcast, you mean Unicast or just "send"?
Is Bonjour/ZeroConf the right route to go to accomplish this?
This really depends on your target environment and what your application is intended to do.
As Ignacio points out, you need to install the Apple software on Windows for Zeroconf/mDNS to work at the moment.
This might be suitable for small office / home use.
However larger networks may have Layer 2 Multicast disabled for a variety of reasons, at which point your app might be in trouble.
If you want it to work in the enterprise environment, then some configuration is required, but that doesn't have to be done at the edge (in the app client instances).
Could be via a DHCP option, or by DNS service records.. in these cases you'd possibly be writing a queryable server to track active clients.. much like a BitTorrent Tracker.
Two things to consider while designing your networked app:
Would there ever be reason to run more than one "installation" of your application on a network?
Always consider the implications of versioning: One client is more up to date than another, can they still talk to each other or at least fail gracefully?

Zeroconf/DNS-SD is an excellent idea in this case. It's provided by Bonjour on OS X and Windows (but must be installed separately or as part of an Apple product on Windows), and by Avahi on FOSS *nix.

I think that ZeroConf is a very good start. You may find this document useful.

I have a list on a webpage, nice if you need internet communications.
<dl_service updated="2010-12-03 11:55:40+01:00">
<client name="internal" ip="10.0.23.234" external_ip="1.1.1.1"/>
<client name="bigone" ip="2.2.2.2" external_ip="2.2.2.2">
<messsage type="connect" from="Bigone" to="internal" />
</client>
</dl_service>
My initial idea was to add firewall punching and all that, but I just couldn't be bothered too many of the hosts where using external IPs for it to be a problem..
But I really recommend Zeroconf, at least if you use Linux+MacOSX, don't know about Windows at all.

Related

How do bigger applications deal with potential (web)socket port clashes?

I'm writing a websocket connection between a chrome plugin and a desktop application that exclusively talk to each other locally. I know that you can't have two sockets listening on the same port (is this true for all ports?)
Port clashes won't happen for the majority of my users, but the application will run on a few thousand computers; I think I can reasonably expect there to be clashes in rare situations, where another application already occupies the port.
How do bigger applications deal with this potential issue? I'm fairly unfamiliar with networking and (web)socket programming in general.
I guess there are a few non-technical workarounds like these:
Log the issue well, and let support deal with it
Let the user configure a port
But I'd rather have a sound technical way to deal with it, if possible. I could come up with some algorithm that tries a few preconfigured ports, but I expect that this is a fairly well known problem in the industry, and am hoping to learn what the consensus is for this problem.
The question is primarily technology agnostic, but if it matters, my stack is python with this websockets library on the desktop side, and of course JavaScript on the client side.
I don't think there's a reason to overthink it. A list of seemingly unoccupied fallback ports and finally a prompt to specify a port should be fine. If both parties are capable of HDD IO (I don't know how much can a chrome plugin can do), then it's even easier for the plugin and the app to meet.
For two apps on a same host, you may also add an additional loopback interface. Example.
If to speak about large networks, there's a class of so-called service discovery protocols that may help dealing with this kind of issue. Basically they are IP- or UDP-based multicast protocols that advertise different services across the network. Usually these protocols have a standardized port for communications, so it is unlikely to get occupied by another app. Examples are DHCP, Bonjour.
However, dealing with port conflicts are not their primary function. In large networks machines should be under some sort of control and should not have lots of apps grabbing random ports. If a conflict happens with third party apps, the ideal solution is to look for a setting that allows to set ports manually.
Well I don't know how bigger applications actually do it but a way could be to try till you find a free port?
found_port = False
ports = [port1, port2, port3 ....]
for port in ports
try:
mySocket.bind ( ( ip_address, port ) )
found_port = True
except:
pass
if found_port:
break

Client/Server role reversal with SimpleXMLRPCServer in Python

I'm working on a project to expose a set of methods from various client machines to a server for the purpose of information gathering and automation. I'm using Python at the moment, and SimpleXMLRPCServer seems to work great on a local network, where I know the addresses of the client machines, and there's no NAT or firewall.
The problem is that the client/server model is backwards for what I want to do. Rather than have an RPC server running on the client machine, exposing a service to the software client, I'd like to have a server listening for connections from clients, which connect and expose the service to the server.
I'd thought about tunneling, remote port forwarding with SSH, or a VPN, but those options don't scale well, and introduce more overhead and complexity than I'd like.
I'm thinking I could write a server and client to reverse the model, but I don't want to reinvent the wheel if it already exists. It seems to me that this would be a common enough problem that there would be a solution for it already.
I'm also just cutting my teeth on Python and networked services, so it's possible I'm asking the wrong question entirely.
What you want is probably WAMP routed RPC.
It seems to address your issue and it's very convenient once you get used to it.
The idea is to put the WAMP router (let's say) in the cloud, and both RPC caller and RPC callee are clients with outbound connections to the router.
I was also using VPN for connecting IoT devices together through the internet, but switching to this router model really simplified things up and it scales pretty well.
By the way WAMP is implemented in different languages, including Python.
Maybe Pyro can be of use? It allows for many forms of distributed computing in Python. You are not very clear in your requirements so it is hard to say if this might work for you, but I advise you to have a look at the documentation or the many examples of Pyro and see if there's something that matches what you want to do.
Pyro abstracts most of the networking intricacy away, you simply invoke a method on a (remote) python object.

ZeroRPC auto-assign free port number

I am using ZeroRPC for a project, where there may be multiple instances running on the same machine. For this reason, I need to abe able to auto-assign unused port numbers. I know how to accomplish this with regular sockets using socket.bind(('', 0)) or with ZeroMQ using the bind_to_random_port method, but I cannot figure out how to do this with ZeroRPC.
Since ZeroRPC is based on ZeroMQ, it must be possible.
Any ideas?
Having read details about ZeroRPC-python current state, the safest option to solve the task would be to create a central LotterySINGLETON, that would receive <-REQ/REP-> send a next free port# upon an instance's request.
This approach is isolated from ZeroRPC-dev/mod(s) modifications of use of the otherwise stable ZeroMQ API and gives you the full control over the port#-s pre-configured/included-in/excluded-from LotterySINGLETON's draws.
The other way aroung would be to try to by-pass the ZeroRPC layer and ask ZeroMQ directly about the next random port, but the ZeroRPC documentation discourages from bypassing their own controls imposed on (otherwise pure) ZeroMQ framework elements ( which is quite a reasonable to be emphasised, as it erodes the ZeroRPC-layer consistency of it's add-on operations & services, so it shall rather be "obeyed" than "challenged" in trial/error ... )
The following will let ZMQ choose a free port:
s = zerorpc.Server(SomeClass())
s.bind('tcp://127.0.0.1:0')
The problem with this is that now you don't know which port it bound to. I managed to find the port with netstat and successfully connected to it, but that's probably not what you want to do. I made a seperate question out of this: Find out bound ports of zerorpc server

Decentralized networking in Python - How?

I want to write a Python script that will check the users local network for other instances of the script currently running.
For the purposes of this question, let's say that I'm writing an application that runs solely via the command line, and will just update the screen when another instance of the application is "found" on the local network. Sample output below:
$ python question.py
Thanks for running ThisApp! You are 192.168.1.101.
Found 192.168.1.102 running this application.
Found 192.168.1.104 running this application.
What libraries/projects exist to help facilitate something like this?
One of the ways to do this would be the Application under question is broadcasting UDP packets and your application is receiving that from different nodes and then displaying it. Twisted Networking Framework provides facilities for doing such a job. The documentation provides some simple examples too.
Well, you could write something using the socket module. You would have to have two programs though, a server on the users local computer, and then a client program that would interface with the server. The server would also use the select module to listen for multiple connections. You would then have a client program that sends something to the server when it is run, or whenever you want it to. The server could then print out which connections it is maintaining, including the details such as IP address.
This is documented extremely well at this link, more so than you need but it will explain it to you as it did to me. http://ilab.cs.byu.edu/python/
You can try broadcast UDP, I found some example here: http://vizible.wordpress.com/2009/01/31/python-broadcast-udp/
You can have a server-based solution: a central server where clients register themselves, and query for other clients being registered. A server framework like Twisted can help here.
In a peer-to-peer setting, push technologies like UDP broadcasts can be used, where each client is putting out a heartbeat packet ever so often on the network, for others to receive. Basic modules like socket would help with that.
Alternatively, you could go for a pull approach, where the interesting peer would need to discover the others actively. This is probably the least straight-forward. For one, you need to scan the network, i.e. find out which IPs belong to the local network and go through them. Then you would need to contact each IP in turn. If your program opens a TCP port, you could try to connect to this and find out your program is running there. If you want your program to be completely ignorant of these queries, you might need to open an ssh connection to the remote IP and scan the process list for your program. All this might involve various modules and libraries. One you might want to look at is execnet.

Network Communication program in python

Basically what I'm trying to achieve is a program which allow users to connect to a each other over a network in, essentially, a chat room. What I'm currently struggling with is writing the code so that the users can connect to each other without knowing the IP-address of the computer that the other users are using or knowing the IP-address of a server.
Does anyone know of a way in which I could simply have all of the users scan the IP range of my network in order to find any active 'room' and then give the user a chance to connect to it?
Also, the hope is that there will be no need for a central server to run this from, rather every user will simply be connected to all other user, essentially being the server and client at the same time.
I can give you two suggestions. First of all, UDP packets to the broadcast address of your network will be received by everybody. Secondly, there is a protocol for programs offering certain services to find each other on a local network. That protocol is called mDNS, ZeroConf, or Bonjour.
Using broadcast UDP is likely going to be the faster route. But if I were you, I'd learn how to use ZeroConf instead. It's supported well under IPv6 and already used by several interesting programs such as SubEthaEdit and Gobby.
Here is a link to a nice tutorial for implementing something that speaks ZeroConf in Python.
Another recommendation... If you want to hand roll your own broadcast/multicast UDP code and you can be sure that all of the systems you're on are running a Linux that's newer than 2003 or so, and all the Windows systems are XP or better, you can probably get away with using IPv6. The IPv6 link-local (think same LAN) all hosts multicast address is ff02::1. That's really simple and easy, and it will reach all the other systems on the same LAN. It's much better than having to figure out what your network's broadcast address is with IPv4.

Categories

Resources