Serverless communication between network nodes in python - python

I am not sure if this question belongs here as it may be a little to broad. If so, I apologize. Anyway, I am planning to start a project in python and I am trying to figure out how best to implement it, or if it is even possible in any practical way. The system will consist of several "nodes" that are essentially python scripts that translate other protocols for talking to different kinds of hardware related to i/o, relays to control stuff, inputs to measure things, rfid-readers etc, to a common protocol for my system. I am no programming or network expert, but this part I can handle, I have a module from an old alarm system that uses rs-485 that I can sucessfully control and read. I want to get the nodes talking to eachother over the network so I can distribute them to different locations (on the same subnet for now). The obvious way would be to use a server that they all connect to so they can be polled and get orders to flip outputs or do something else. This should not be too hard using twisted or something like it.
The problem with this is that if this server for some reason stops working, everything else does too. I guess what I would like is some kind of serverless communication, that has no single point of failure besides the network itself. Message brokers all seem to require some kind of server, and I can not really find anything else that seems suitable for this. All nodes must know the status of all other nodes as I will need to be able to make functions based on the status of things connected to other nodes, such as, do not open this door if that door is already open. Maybe this could be done by multicast or broadcast, but that seems a bit insecure and just not right. One way I thought of could be to somehow appoint one of the nodes to accept connections from the other nodes and act as a message router and arrange for some kind of backup so that if this node crashes or goes away, another predetermined node takes over and the other nodes connect to it instead. This seems complicated and I am not sure this is any better than just using a message broker.
As I said, I am not sure this is an appropriate question here but if anyone could give me a hint to how this could be done or if there is something that does something similar to this that I can study. If I am beeing stupid, please let me know that too :)

There are messaging systems that don't require a central message broker. You might start by looking at ZeroMQ.

I ended up making my own serverless messaging system in python. It is not ready and the code is a mess, but it works. It has autodiscovery of nodes, sharing of channels and topics, and the features that I needed. If anyone is interested in using it or wants to help, it is here:
https://bitbucket.org/ssspeq/connection-manager

Related

First client cant connect to server [duplicate]

I need to do some basic networking for a Pygame project.
Basically, it's a 2D single player or cooperative game. The networking only needs to support two players, with one as a host.
The only information that needs to be sent is the positions of players, creeps and bullets.
I've been reading around and Twisted keeps coming up, but I haven't done networking before, and I'm not sure if that might be an overkill.
So, is it possible for a relative newbie to implement networking in Pygame?
This was asked recently on Reddit, so I'll more or less just copy my answer over from there. I apologize for not being able to provide more links, I have <10 rep so I can only post two at a time.
Twisted might work, but I don't have a whole lot of experience with it. I'd recommend going with sockets, as that's what Twisted uses in the background anyway. Beej's guide (google it) is pretty much the Holy Bible of sockets if you want to learn how they work (in C++, but the concepts extend everywhere). Python does abstract some of the complexity away, but it's still a good idea to know what's going on in the background.
For Python specific sockets, you can go ahead and just use the howto (user745294 posted a link above). Here's a nice article titled "What every programmer needs to know about Game Networking". It goes into the different types of major networking styles (client-server, p2p, udp v. tcp, etc.) and the history behind what some major games used for their networking.
Below is a link to a demo I did on making a networked "game" in Python 2.6/Pygame. It's not actually a game, but each client you create connects to the server and controls a character. You can move your character with the arrow keys and the character will move on all connected clients. I tried commenting the source code with some indication of what I'm sending back and forth, but you may need a little knowledge about sockets to understand it.
The source code is provided in the codepad links in the comment below this post. You will need to provide two images in the same directory as the scripts:
bg.png is the background sprite. It should be an image 400px wide and 300px tall (this can be changed in the GameClient class if needed)
sprite.png is the player character. It should be smaller than the background so that you can see it moving around.
You can use Twisted for networking with Pygame. The "game" project on Launchpad has some examples of how one might integrate the main loops together; basically, use twisted.internet.task.LoopingCall to draw Pygame frames and handle input, while letting the Twisted reactor of your choice run normally.
Since you are already using Pygame, I think this light networking library made for Pygame will do what you need and teach you, but not overwhelm you.
"Mastermind Networking Lib" via pygame.org
There is Pyro (Python remote objects) as another solution for networking in Python.
http://irmen.home.xs4all.nl/pyro/
Using raw sockets is low-level and full of danger. As said before, Twisted is complex and takes to time get up and running. To save yourself some headaches I'd try something like zerorpc.
You need the following solutions:
discovering other player(s) on the (local) network, you don't want player to enter some IP address
handle network errors
serialize messages containing your data (positions, player name, etc.)
handle threading as networking is asynchronous I/O
Above should still be called 'basic', you should really use some fancy networking library with idiomatic API.
Essentially you need to expose the network service (in its own thread) that will push messages to Python's Queue, and then access this same queue from your Pygame code, and if there is a message then you update whatever structures you use to store player's position and draw it on screen.
You shouldn't send stuff like bullet positions over the network as they can be easily (and faster) calculated locally. You just send an event like bullet_shot over the network with a source position and velocity vector.

Pickle won't save object state, uses only init values [duplicate]

I need to do some basic networking for a Pygame project.
Basically, it's a 2D single player or cooperative game. The networking only needs to support two players, with one as a host.
The only information that needs to be sent is the positions of players, creeps and bullets.
I've been reading around and Twisted keeps coming up, but I haven't done networking before, and I'm not sure if that might be an overkill.
So, is it possible for a relative newbie to implement networking in Pygame?
This was asked recently on Reddit, so I'll more or less just copy my answer over from there. I apologize for not being able to provide more links, I have <10 rep so I can only post two at a time.
Twisted might work, but I don't have a whole lot of experience with it. I'd recommend going with sockets, as that's what Twisted uses in the background anyway. Beej's guide (google it) is pretty much the Holy Bible of sockets if you want to learn how they work (in C++, but the concepts extend everywhere). Python does abstract some of the complexity away, but it's still a good idea to know what's going on in the background.
For Python specific sockets, you can go ahead and just use the howto (user745294 posted a link above). Here's a nice article titled "What every programmer needs to know about Game Networking". It goes into the different types of major networking styles (client-server, p2p, udp v. tcp, etc.) and the history behind what some major games used for their networking.
Below is a link to a demo I did on making a networked "game" in Python 2.6/Pygame. It's not actually a game, but each client you create connects to the server and controls a character. You can move your character with the arrow keys and the character will move on all connected clients. I tried commenting the source code with some indication of what I'm sending back and forth, but you may need a little knowledge about sockets to understand it.
The source code is provided in the codepad links in the comment below this post. You will need to provide two images in the same directory as the scripts:
bg.png is the background sprite. It should be an image 400px wide and 300px tall (this can be changed in the GameClient class if needed)
sprite.png is the player character. It should be smaller than the background so that you can see it moving around.
You can use Twisted for networking with Pygame. The "game" project on Launchpad has some examples of how one might integrate the main loops together; basically, use twisted.internet.task.LoopingCall to draw Pygame frames and handle input, while letting the Twisted reactor of your choice run normally.
Since you are already using Pygame, I think this light networking library made for Pygame will do what you need and teach you, but not overwhelm you.
"Mastermind Networking Lib" via pygame.org
There is Pyro (Python remote objects) as another solution for networking in Python.
http://irmen.home.xs4all.nl/pyro/
Using raw sockets is low-level and full of danger. As said before, Twisted is complex and takes to time get up and running. To save yourself some headaches I'd try something like zerorpc.
You need the following solutions:
discovering other player(s) on the (local) network, you don't want player to enter some IP address
handle network errors
serialize messages containing your data (positions, player name, etc.)
handle threading as networking is asynchronous I/O
Above should still be called 'basic', you should really use some fancy networking library with idiomatic API.
Essentially you need to expose the network service (in its own thread) that will push messages to Python's Queue, and then access this same queue from your Pygame code, and if there is a message then you update whatever structures you use to store player's position and draw it on screen.
You shouldn't send stuff like bullet positions over the network as they can be easily (and faster) calculated locally. You just send an event like bullet_shot over the network with a source position and velocity vector.

Python too slow with network [duplicate]

I need to do some basic networking for a Pygame project.
Basically, it's a 2D single player or cooperative game. The networking only needs to support two players, with one as a host.
The only information that needs to be sent is the positions of players, creeps and bullets.
I've been reading around and Twisted keeps coming up, but I haven't done networking before, and I'm not sure if that might be an overkill.
So, is it possible for a relative newbie to implement networking in Pygame?
This was asked recently on Reddit, so I'll more or less just copy my answer over from there. I apologize for not being able to provide more links, I have <10 rep so I can only post two at a time.
Twisted might work, but I don't have a whole lot of experience with it. I'd recommend going with sockets, as that's what Twisted uses in the background anyway. Beej's guide (google it) is pretty much the Holy Bible of sockets if you want to learn how they work (in C++, but the concepts extend everywhere). Python does abstract some of the complexity away, but it's still a good idea to know what's going on in the background.
For Python specific sockets, you can go ahead and just use the howto (user745294 posted a link above). Here's a nice article titled "What every programmer needs to know about Game Networking". It goes into the different types of major networking styles (client-server, p2p, udp v. tcp, etc.) and the history behind what some major games used for their networking.
Below is a link to a demo I did on making a networked "game" in Python 2.6/Pygame. It's not actually a game, but each client you create connects to the server and controls a character. You can move your character with the arrow keys and the character will move on all connected clients. I tried commenting the source code with some indication of what I'm sending back and forth, but you may need a little knowledge about sockets to understand it.
The source code is provided in the codepad links in the comment below this post. You will need to provide two images in the same directory as the scripts:
bg.png is the background sprite. It should be an image 400px wide and 300px tall (this can be changed in the GameClient class if needed)
sprite.png is the player character. It should be smaller than the background so that you can see it moving around.
You can use Twisted for networking with Pygame. The "game" project on Launchpad has some examples of how one might integrate the main loops together; basically, use twisted.internet.task.LoopingCall to draw Pygame frames and handle input, while letting the Twisted reactor of your choice run normally.
Since you are already using Pygame, I think this light networking library made for Pygame will do what you need and teach you, but not overwhelm you.
"Mastermind Networking Lib" via pygame.org
There is Pyro (Python remote objects) as another solution for networking in Python.
http://irmen.home.xs4all.nl/pyro/
Using raw sockets is low-level and full of danger. As said before, Twisted is complex and takes to time get up and running. To save yourself some headaches I'd try something like zerorpc.
You need the following solutions:
discovering other player(s) on the (local) network, you don't want player to enter some IP address
handle network errors
serialize messages containing your data (positions, player name, etc.)
handle threading as networking is asynchronous I/O
Above should still be called 'basic', you should really use some fancy networking library with idiomatic API.
Essentially you need to expose the network service (in its own thread) that will push messages to Python's Queue, and then access this same queue from your Pygame code, and if there is a message then you update whatever structures you use to store player's position and draw it on screen.
You shouldn't send stuff like bullet positions over the network as they can be easily (and faster) calculated locally. You just send an event like bullet_shot over the network with a source position and velocity vector.

Python Twisted or Concurrence Programming for Learning Sockets/Networking

Well this is my last ditch effort of finding help/guidance. We've only just started learning Python at university and i really want to get a jump on everyone else and i've really been focusing on networking with Python. I've had a look at the twisted framework and it's massive ! even the krondo.com site is.
I've seen Concurrence for python on LinuxQuestions a couple of times, which seems easier (however it looks like Concurrence has been dis-continued?), and was wondering if im better off knuckling down and really getting into twisted or maybe trying something else.... Its not that im afraid of hard work, but i dont want to go down a path that when im finished, i find out that know one use's it out in the real world, or knows of it.
Im sorry its not a technical question, but this is really the only place where i know i can get proper guidance. If you think putting my head down and learning twisted will be worth my while, then ill do that. Thanks in advance for your time.
Honestly, I would take a step back on frameworks and work with the Python standard library on a more basic level. The python docs are a very good and authoritative source of information on Python. If you're looking into networking, here's a path I'd recommend looking over:
Sockets
Learn how to work with basic socket connections, making clients, and making servers.
SSL
A lot of web services use this to communicate data securely for APIs. It may be difficult to wrap your head around, but it's a highly valuable piece of knowledge to have.
Select
One way of handling multiple clients connecting to a service. This gets into the concurrency realm which is a fairly sought after skill.
Threads
Another way of handling multiple connections, networking tasks.
Protocols
Now that you've wrapped your head around basic networking, it's time to get into protocols. These are sending specifically formatted data over the network to a server that can handle such data. Yes, I'd recommend going through all the different protocols.
Data Parsing
Data that is sent and received during different protocol communication can be encoded in various ways to make it easier to deal with data.
Pack and Unpack
Sometimes network communication isn't done with a standard format. These functions are helpful in dealing with binary protocols if there's no library available.
Pickle
This is some advanced stuff and python specific. You can use it to send python objects over the network. Just be careful to do this on trusted networks only for security purposes.
With that in mind these are general recommendations. Check the docs out, write some code, and ask questions if you get stuck.

Dataflow computing in python

I have n (typically n < 10 but it should scale) processes running on different machines and communicating through amqp using RabbitMQ. Processes are typically long running and may be implemented in any language (though most are java/python).
Each process requires a number of inputs (numbers/strings) and produces a number of outputs (also just numbers or strings). Executing a process happens asynchronously: sending a message on its input queue and waiting for a callback to be triggered by the output queue.
Ideally the user specifies some inputs and desired outputs and the system should:
detect which processes are needed and generate the dependency graph
topologically sort the graph and execute it, node transitions will need to be event driven
A node should fire if its input is ready, allowing parallelism per branch. I can assume no cycles for now, but eventually there will be cycles (e.g., two processes may need to iterate until the output no longer changes).
This should be a known problem from (data)flow programming (discussed here before) and I want to avoid re-inventing the wheel. I would prefer a python solution and a search leads to Trellis and Pypes. Trellis is no longer developed but seems to support cycles, while pypes does not. Also not sure how actively developed pypes is.
Further searches reveal a whole list of event based programming frameworks, none of which I am particularly knowledgeable about. There are of course workflow environments like Taverna and KNIME, but that seems overkill.
Does anybody have any experience tackling this type of problem or with the libraries mentioned?
Edit: Other libraries I found are:
Stream
zflow
pyf
javafbp (Java)
python.org has a Wiki page on "Flow Based Programming" -- http://wiki.python.org/moin/FlowBasedProgramming
The bottom line is that if you can reinvent the wheel in a small number of lines of code ( a few hundred) which you completely understand and can document, then do it.
This is an area where the abstractions used are not that hard to implement given some basic foundation tools. RabbitMQ is such a tool. Node.js is another. There are lots of libraries around that implement useful ways to manages dataflows, workflows, finite state machines, etc., but they have a lot of overlap and they tend to be incomplete. Probably the original developer just built enough to get over his initial problem, and since this type of programming was not that popular, there was not the critical mass to keep development going.
There is a lot to be said for ranking all the possible solutions by popularity, picking the most popular one, and putting your effort into making it work (while sharing your work, of course).

Categories

Resources