I'm trying to add IPv6 support to an application written with Twisted, which unfortunately does not support IPv6. There's a three-year-old ticket for this in Twisted's trac, but it seems to be stuck in debate with no indication of progress any time soon.
Ideally I'd like existing reactor methods like listenTCP and connectTCP to simply work, i.e. connectTCP would accept either IPv4 or IPv6 addresses and deal with the socket differences behind the scenes. The ticket does include a patch to do this, but it's rather old, and completely untested, so not really suitable for a production app, except as a reference point.
Before I spend a lot of time writing my own patch, I'm wondering whether anyone else has dealt with this, and if so, how.
Yes. Use HostnameEndpoint to connect to a remote host regardless of its address family.
Related
I'm writing a websocket connection between a chrome plugin and a desktop application that exclusively talk to each other locally. I know that you can't have two sockets listening on the same port (is this true for all ports?)
Port clashes won't happen for the majority of my users, but the application will run on a few thousand computers; I think I can reasonably expect there to be clashes in rare situations, where another application already occupies the port.
How do bigger applications deal with this potential issue? I'm fairly unfamiliar with networking and (web)socket programming in general.
I guess there are a few non-technical workarounds like these:
Log the issue well, and let support deal with it
Let the user configure a port
But I'd rather have a sound technical way to deal with it, if possible. I could come up with some algorithm that tries a few preconfigured ports, but I expect that this is a fairly well known problem in the industry, and am hoping to learn what the consensus is for this problem.
The question is primarily technology agnostic, but if it matters, my stack is python with this websockets library on the desktop side, and of course JavaScript on the client side.
I don't think there's a reason to overthink it. A list of seemingly unoccupied fallback ports and finally a prompt to specify a port should be fine. If both parties are capable of HDD IO (I don't know how much can a chrome plugin can do), then it's even easier for the plugin and the app to meet.
For two apps on a same host, you may also add an additional loopback interface. Example.
If to speak about large networks, there's a class of so-called service discovery protocols that may help dealing with this kind of issue. Basically they are IP- or UDP-based multicast protocols that advertise different services across the network. Usually these protocols have a standardized port for communications, so it is unlikely to get occupied by another app. Examples are DHCP, Bonjour.
However, dealing with port conflicts are not their primary function. In large networks machines should be under some sort of control and should not have lots of apps grabbing random ports. If a conflict happens with third party apps, the ideal solution is to look for a setting that allows to set ports manually.
Well I don't know how bigger applications actually do it but a way could be to try till you find a free port?
found_port = False
ports = [port1, port2, port3 ....]
for port in ports
try:
mySocket.bind ( ( ip_address, port ) )
found_port = True
except:
pass
if found_port:
break
I'm working on a project to expose a set of methods from various client machines to a server for the purpose of information gathering and automation. I'm using Python at the moment, and SimpleXMLRPCServer seems to work great on a local network, where I know the addresses of the client machines, and there's no NAT or firewall.
The problem is that the client/server model is backwards for what I want to do. Rather than have an RPC server running on the client machine, exposing a service to the software client, I'd like to have a server listening for connections from clients, which connect and expose the service to the server.
I'd thought about tunneling, remote port forwarding with SSH, or a VPN, but those options don't scale well, and introduce more overhead and complexity than I'd like.
I'm thinking I could write a server and client to reverse the model, but I don't want to reinvent the wheel if it already exists. It seems to me that this would be a common enough problem that there would be a solution for it already.
I'm also just cutting my teeth on Python and networked services, so it's possible I'm asking the wrong question entirely.
What you want is probably WAMP routed RPC.
It seems to address your issue and it's very convenient once you get used to it.
The idea is to put the WAMP router (let's say) in the cloud, and both RPC caller and RPC callee are clients with outbound connections to the router.
I was also using VPN for connecting IoT devices together through the internet, but switching to this router model really simplified things up and it scales pretty well.
By the way WAMP is implemented in different languages, including Python.
Maybe Pyro can be of use? It allows for many forms of distributed computing in Python. You are not very clear in your requirements so it is hard to say if this might work for you, but I advise you to have a look at the documentation or the many examples of Pyro and see if there's something that matches what you want to do.
Pyro abstracts most of the networking intricacy away, you simply invoke a method on a (remote) python object.
I'm using webpy to make a small site. When I want to use OAuth, i find that the firewall will stop the http request to any site, I even can't use IE to browse the Internet.
So i asked the aministrator to open some ports for me, but i don't know which ports will be used by python or IE to send http request.
Thanks!
I assume you're talking about the remote ports. In that case, just tell the admin to open the standard web ports. Really, if your admin doesn't know how to make IE work through the firewall, he's hopeless. I suggest walking up to random people on the street and say "80 and 443" until someone looks up, then fire your admin and hire that guy; he can't be any worse.
If your admin does know what he's doing, and wants you to use an HTTP proxy instead of connecting directly, ask him to set up the proxy for you in IE, look at the settings he uses, then come back here and search for how to use HTTP proxies in Python (there are lots of answers on that), and ask for help if you get stuck.
If you're talking about the local ports, because you're got an insane firewall, they'll be picked at random from some large range. If you want to cover every common OS, you need all of 1024-65535 to be opened up, although if you only need to deal with a single platform, most use a smaller range than that, and if the machine won't be doing much but running your program, most have a way to restrict it to an even smaller range (e.g., as small as 255 ports on Windows). Google "ephemeral port" for details.
If you need to restrict your local port, the key is to call bind on your socket before calling connect. If you think you're talking about the local ports, you're probably wrong. Go ask your admin (or the new one you just hired) and make sure. But if you are…
If you're using urllib/urllib2, it does not have any way to do what you want, so you can't do that anymore. You can drop down to httplib, which lets you pass a source_address, a (host, port) tuple that it will use to bind the socket before connecting. It's not as simple as what you're using, but it's a lot easier than implementing HTTP from scratch.
You might also want to look at requests, which I know has its own native OAuth support, and probably has a way to specify a source address. (I haven't looked, but I usually find that whenever I want to know if requests can do X, it can, and in the most obvious way I think of…) The API for requests is generally similar to urllib2 when urllib2 is sane, simpler and cleaner when urllib2 is messy, so it's usually pretty easy to port things.
At any rate, however you do this, you will have to consider the fact that only one socket can be bound to the same local port at a time. So, what happens if two programs are running at once, and they both need to make an outbound connections, and your admin has only given you one port? One of them will fail. Is that acceptable?
If not, what you really need to do is open a range of ports, and write code that does a random.shuffle on the range, then tries to bind them until one succeeds. Which means you'll need an HTTP library that lets you feed in a socket factory or a pre-opened socket instead of just specifying a port, which most of them do not, which probably means you'll be hacking up a copy of the httplib source.
If all else fails, you can always set up a local proxy that binds to whatever source port (or port range) you want when proxying outward. Then you can just use your favorite high-level library, as-is, and connect to the local proxy, and there's no way the firewall can tell what's going on behind the proxy.
As you can see, this is not easy. That's mainly because you very actually rarely this.
Generally when a program wants to use a port but doesn't care which number it has, it uses an "ephemeral port." This is typical for client applications, where the remote port is fixed (by the server), but the local port doesn't make any difference.
Often a firewall will allow outgoing connections from any port, simply blocking incoming connections to unknown ports, on the theory that internal machines making outgoing requests should be allowed to decide what is proper, and that bad actors are all on the "public" side.
You may find that your administrator requires you to use an "HTTP proxy." If so, here are the instructions for Ruby which I imagine you can port to Python: Ruby and Rails - oauth and http proxy
thanks for the interesting responses thus far. In light of said responses I have changed my question a bit.
guess what I really need to know is, is socketserver as opposed to the straight-up socket library designed to handle both periods of latency and stress, i.e. does it have additional mechanisms or features that justify its implicitly advertised status as a "server," or is it just slightly easier to use?
everyone seems to be recommending socketserver but I'm still not entirely clear why, as opposed to socket.
thanks!!!
I've built some server programs in
python based on the standard socket
library
http://docs.python.org/library/socket.html
I've noticed that they seem to work
just fine except that without load
they have a tendency to go to sleep
after a while. I guess this may not
be an issue in production (no doubt
there will be plenty of other issues)
but I would like to know if I am
using the right code for the job here.
Looking around I saw that python also
provides a socketserver library -
http://docs.python.org/library/socketserver.html
The socket library provides the
ability to listen for multiple
connections, typically up to 5.
According to the socketserver page,
its services are synchronous, i.e.
blocking, but one may support
asynchronous behavior via threading.
I did notice it has the ability to
maintain a request queue, with a
default value of up to 5 requests...so
maybe not much difference there.
I have also read that Twisted runs
socketserver under the hood. Though I
would rather not get into a beast the
size of Twisted unless it's going to
be worthwhile.
so my question is, is socketserver
more robust than socket? If so, why?
(And how do you know?)
incidentally, is socketserver built on
top of python's socket or is it
entirely separate?
finally, as a bonus if anyone knows
what one could do wrong such that
standard sockets 'fall asleep' please
feel free to chime in on that too.
Oh, and I'm talking python 2.x rather
than 3.x here if that makes a
difference.
thanks folks!
jsh
Well, I don't have a technical answer but I've implemented SocketServer per folks' recommendations and it IS definitely more reliable. If anyone ever comes up with the low-level explanation please let me know...thanks!
The socket module is a very low-level module for sending and receiving packets. As said in the documentation, it "provides access to the BSD socket interface".
If you want something more elaborate, there is "socketserver" that takes care of the gory details for you, but it is still relatively low level.
On top of that you can find an HTTP server, with or without CGI, an XML-RPC server, and so on. These are frameworks, which usually means that their code calls your code. It makes things simpler because you just have to fill some "gaps" to have a fully working server, but it also means you have a little bit less control over what it does.
If you only need features of socketserver, I would probably go with it, unless you want to reinvent the wheel for some reason (and there are always good reasons to design new wheels, for example to understand how it works).
Is it possible to write a peer-to-peer chat application in Python?
I am thinking of this from a hobbyist project point-of-view. Can two machines connect to each other directly without involving a server? I have always wondered this, but never actually seen it implemented anywhere so I am thinking there must be a catch somewhere.
PS: I intend to learn Twisted, so if that is involved, it would be an added advantage!
Yes. You can do this pretty easily with Twisted. Just have one of the peers act like a server and the other one act like a client. In fact, the twisted tutorial will get you most of the way there.
The only problem you're likely to run into is firewalls. Most people run their home machines behind SNAT routers, which make it tougher to connect directly to them from outside. You can get around it with port forwarding though.
Yes, each computer (as long as their on the same network) can establish a server instance with inbound and outbound POST/GET.
I think i am way too late in putting my two bits here, i accidentally stumbled upon here as i was also searching on similar lines. I think you can do this fairly easily using just sockets only, however as mentioned above one of the machines would have to act like a server, to whome the other will connect.
I am not familiar with twisted, but i did achieved this using just sockets. But yes even i am curious to know how would you achieve peer2peer chat communication if there are multiple clients connected to a server. Creating a chat room kind of app is easy but i am having hard time in thinking how to handle peer to peer connections.