A python program needs to accept a string every second from a serial port. I plan on using a RS-232 to USB converter. The application is going to work under Ubuntu 10.04.
How do I approach this? Do I use pySerial or libusb?
There needs to be done some processing in the meantime, so synchronous communication is not viable. Do I use some kind of interrupts or do I need to open separate threads? Or do I use blocking reads, believing that 1s is enough for my computations (it is plenty ... for now)?
I know, RTFM, but heading in the correct direction from the start will save me a lot of time! Thanks for bearing with me.
If your RS232-USB converter has a driver in Ubuntu that makes it look like a COM port then you will want to use pySerial (the interface is the same as any other COM port). If there is no driver for your device then you might have to use libusb and find the protocol for your specific device. Most major RS232-USB converters these days have usbserial based drivers committed and maintained in the Linux kernel. Simply check with your vendor for this.
There are many ways to do parallel processing but typically I write my applications in two ways:
Have a read thread that does nothing but read and fill a local thread safe buffer so data is ready for other threads when needed.
Have a read thread that reads data, determines where it goes and delivers it via messaging/event processing to the component that needs it.
The decisions here will depend on what your goals are, and how much processing is needed outside the reading.
Related
If this is a stupid question, please don't mind me. But I spent some time trying to find the answer but I couldn't get anything solid. Maybe this is a hardware question, but I figured I'd try here first.
Does Serial Communication only work one to one? The reason this came up is because I had an arduino board listening for communication on its serial port. I had a python script feed bytes to the port as well. However, whenever I opened up the arduino's serial monitor, the connection with the python script failed. The serial monitor also connects to the serial port for communication for its little text input field.
So what's the deal? Does serial communication only work between a single client and a single server? Is there a way to get multiple clients writing to the server? I appreciate your suggestions.
Multiple clients (e.g. Arduinos) communicating with one server (e.g. a desktop computer) is commonly done with the serial variant:
RS-485
This is a simple method widely used in industrial settings where you want to have many devices connected to one computer via one serial port. This type of arrangement is also called multi-drop, because one cable strings around a building with Tees that tap in and drop lines to each device.
The hardware for this is widely available. You can buy USB serial adapters that provide the hardware interface for a computer. Programmatically the port looks just like an RS232 port. For the Arduino you would just add a transceiver chip. A sea of serial transceivers exists, e.g.
Example computer USB adapter with 485 interface
Sample RS485 transceiver chip from Element14
All the devices hang on the same bus listening at the same time. A simple communication protocol used is just add a device address before every command. For example:
001SETLIGHT1 <- tells Arduino "001" to turn on the light
013SETLIGHT0 <- tells "013" to turn off the light
Any device hanging on the cable ignores commands that do not start with their address. When a device responds, it prepends its address.
001SETLIGHT1DONE <- response from device "001" that the command has been received and executed
The address in the response lets the receiving party know which device was talking.
Well, your question can be quite wide, so I'm going to layer my answer:
On the hardware side, the same pair of wires can work be shared with many devices. It is mostly a question of electronics (maintaining the signal in the good voltage range), and not having all devices writing to the serial port at the same time (or you'll get wreckage).
On the software side, on the host, yes you can share the same serial connection to a device with multiple processes. But that's not straight forward. I'll assume you're using an unix (macos or linux):
in unix, everything is a file, your serial connection is one too: /dev/ttyACM0 on linux, for example.
When you have a process opening that file, it will block it (using ioctl, iirc) so no other process can mess with that file too.
Then, you can input and output to that file using the process that opened it, that's all.
But hopefully, it is still possible to share the connection between processes. One of them would simply be to use the tee command, that will be able to get input from one process, and give it back output, and copy the output to another process. You can also do it from within python, by duplicating the file descriptor.
To easily output stuff that can be redirected the unix way (using pipes), you can use socat: http://www.dest-unreach.org/socat/
here's an usage example:
socat -,raw,echo=0,escape=0x0f /dev/ttyACM0,raw,echo=0,crnl
you may want to tweak it for your needs.
Edit:
I forgot about RS-485, which 'jdr5ca' was smart enough to recommend. My explanation below is restricted to RS-232, the more "garden variety" serial port. As 'jdr5ca' points out, RS-485 is a much better alternative for the described problem.
Original:
To expand on zmo's answer a bit, it is possible to share serial at the hardware level, and it has been done before, but it is rarely done in practice.
Likewise, at the software driver level, it is again theoretically possible to share, but you run into similar problems as the hardware level, i.e. how to "share" the link to prevent collisions, etc.
A "typical" setup would be two serial (hardware) devices attached to each other 1:1. Each would run a single software process that would manage sending/receiving data on the link.
If it is desired to share the serial link amongst multiple processes (on either side), the software process that manages the link would also need to manage passing the received data to each reading process (keeping track of which data each process had read) and also arbitrate which sending process gets access to the link during "writes".
If there are multiple read/write processes on each end of the link, the handshaking/coordination of all this gets deep as some sort of meta-signaling arrangement may be needed to coordinate the comms between the process on each end.
Either a real mess or a fun challenge, depending on your needs and how you view such things.
I have a GPS module connected through serial port(USB->Virtual COM port). A measurement software is using this port, so with other software I can't access to the data. I would like to create two virtual COM port and share this data through that. Is it possible using Python? Is there any opensource example written in Python?
I don't think you can do that if you cannot modify the sources of the measurement software.
Serial port protocols are written as "point to point" protocols, so there's no general way to multiplex them. You can write a program that shares the access to the GPS module (handling it exclusively and exposing an API to multiple programs), but every program that wanted to use the GPS module should be written to talk to your API and not directly to the serial port - and in this case it can be done only if you can change the measurement software.
Notice that probably it's not impossible to implement your "virtual port" solution, but it would be an ad-hoc hack (it would work just with that specific protocol) and it may be quite complicated: you would need to emulate two GPS modules and multiplex the requests to the real GPS module; depending on how does it work (e.g. if it has a "complicated" persistent state) it may be simple or very complicated. But surely Python wouldn't be enough, to emulate serial ports you have to go in kernel mode.
Do you need two-way communication, or just reading? You could build or buy hardware to physically split the Rx data line so you could use two COM ports, each of which would read the same data. You could do this with Tx data as well, but you would have to be careful about trashing the data if both ports tried to write at the same time.
I have a program which will be running on multiple devices on a network. These programs will need to send data between each other - to specified devices (not all devices).
server = server.Server('192.168.1.10')
server.identify('device1')
server.send('device2', 'this will be pickled and sent to device2')
That's some basic example code for what I need to do. Of course, it will also need to receive.
I was looking at building my own simple message passing server using Twisted when someone pointed me in the direction of MPI. I've never looked into the MPI protocol before and that website gives rather vague examples.
Is MPI a good approach? Are there better alternatives?
MPI is really good at doing the communications for running a tightly-coupled program accross several or many machines in a cluster. If you're running very loosely coupled programs - only interacting occasionally - or the machines are more distributed than within a cluster, like scattered around a LAN - then MPI is probably not what you're looking for.
There are several Open Source message brokers that already handle this kind of stuff for you, and come with a full API ready to use.
You should take a look at:
ActiveMQ which has a Python Stomp client.
RabbitMQ has a Python client too - see Building RabbitMQ apps using Python.
You could build it yourself, but that would be like reinventing the wheel (and on a side-note: I actually only realised I was half-way building a message broker before I started looking at existing solutions - building one takes a lot of work).
Consider using something like ZeroMQ. It supports the most useful messaging idioms - push/pull, publish/subscribe and so on, and although it's not 100% clear from your question which one you need, I'm pretty sure you will find the answer there.
They have a great user guide here, and the Python bindings are well-developed and supported. Some code samples are here.
You can implement MPI functions in order to create a communication between different codes. In this case the server program should public "MPI ports" with differents IDs. Clients should look for this ports and try to connect to them. Only server can accept each communication. Once the communication is stablished, codes can exchange data between them.
Another posibility is to run different programs in Multiple Instruction MPI option. In this case all programs are executed at the same time, and there is not necessity to create port communicators. After they are executed, you can create particular communicators between groups of programms you select.
Please tell me what kind of method you need and I can provide c code to implement the functions.
I have very little idea what I'm doing here, I've never done anything like this before, but a friend and I are writing competing chess programs and they need to be able to communicate to each other.
He'll be writing mainly in C, the bulk of mine will be in Python, and I can see a few options:
Alternately write to a temp file, or successive temp files. As the communication won't be in any way bulky this could work, but seems like an ugly work-around to me, the programs will have to keep checking for change/new files, it just seems ugly.
Find some way of manipulating pipes i.e. mine.py| ./his . This seems like a bit of a dead end.
Use sockets. But I don't know what I'd be doing, so could someone give me a pointer to some reading material? I'm not sure if there are OS-independent, language independent methods. Would there have to be some kind of supervisor server program to administrate?
Use some kind of HTML protocol, which seems like overkill. I don't mind the programs having to run on the same machine.
What do people recommend, and where can I start reading?
If you want and need truly OS independent, language independent inter process communication, sockets are probably the best option.
This will allow the two programs to communicate across machines, as well (without code changes).
For reading material, here's a Python Socket Programming How To.
Two possibilities:
Use IP sockets. There are some examples in the Python docs. (Really not that hard if you just use the basic read/write stuff.) On the other hand, sockets in C are generally not that simple to use.
Create a third application. It launches both applications using subprocess and communicates with both applications through pipes. The chess applications must only be able to read/write to stdin/stdout.
This has the additional benefit that this application could check if a move is legal. This helps you finding bugs and keeping the games fair.
You can use Protobuf as the inter-program protocol and read/write from a file each one turns.
You may read the intermediate file every n seconds.
Once you have this working, you may move to use sockets, where each program would start a server and wait for connections.
The change should be small, because the protocol would be protobuf already. So, the only place you have to change is where you either read from a socket or from a file.
In either case you'll need an interchange protocol.
edit
Ooops I misread and I thought it was C++.
Anyway, here's the C support for protobuf but is still work in progress work
http://code.google.com/p/protobuf-c/
I would say just write an xml file that contains moves for black and white. Mark in a separate file who's turn it is and make sure only the program who's turn it is will write to that file to commit their turn.
Here is a link to a proposed xml format for storing your moves that another group came up with
http://www.xml.com/pub/a/2004/08/25/tourist.html
Sockets with a client/server model...
Basically, you and your friend are creating different implementations of the client.
The local client shows a visual representation of the game and stores the state of the pieces (position, killed/not-killed) and the rules about what the pieces can/can't do (which moves can be made with which pieces and whether the board's state is in check).
The remote server stores state about the players (whose turn it is, points earned, whether the game is won or not), and a listing of moves that have occurred.
When you make a move, your client validates the move against the rules of the game, then sends a message to the server that says, I've made this move, your turn.
The other client sees that a turn has been made, pulls the last move from the server, calculates whether where the movement took place, validates the move against the game rules, and replays the action locally. After that's all done, it's now allows the user to make the next move (or not if the game is over).
The most important part of client/server gaming communication is, send as little data to and store as little state as possible on the server. That way you can play it locally, or across the world with little or no latency. As long as your client is running under the same set of rules as your opponent's client everything should work.
If you want to ensure that no one can cheat by hacking their version of the client, you can make the position and rule calculations all be done on the server and just make the clients nothing but simple playback mechanisms.
The reason why sockets are the best communication medium are:
the limitations on cross process communication are almost as difficult as cross node communication
networking is widely supported on all systems
there's little or no barrier-of-entry to using this remotely if you choose
the networking is robust, flexible, and proven
That's part of the reason why many major systems like Databases uses sockets as a networking as-well-as local communication medium.
if both applications running on same computer, use socket and serialize your objects to jsun. otherwise, use web service and jsun or xml. You can find jsun and xml parser in both languages.
How can I detect when a flash drive is plugged in? I'm using a bare Debian installation, without any GUI and want to be notified in my Python script when a new flash drive appears... I know that D-BUS distributes such information, but i dont want to use D-BUS. Is there a more bare access to that information? Shouldn't that be available under /proc or /sys? How can I "connect" to that source?
Bye
falstaff
All mayor Linux distros include udev, that allows you to write custom scripts on hardware events.
You can read uevents from kernel via a Netlink socket and look for events where "ACTION" is "add" (you can also watch if a partition from a device was mounted or unmounted, or if a device is being removed). That's as close to the source of events as one can get in user space. As far as I know, this is how udev detects inserted removable media.
But probably using D-Bus/HAL API via Python bingings will be much easier (no uevents data parsing, etc). Not sure why you are opposed to this. Since you are using Python, I suspect that resources are not really the issue.
If you are targetting an embedded device, then you can run mdev instead of udev.
Then you can write mdev rules that are quite simple and triggers a script.
Of course you are not directly monitoring from your script, mdev is doing it, but you can launch any command. You can probably do the same thing with udev, but it always looked quite complicated to me.
When an USB device is plugged in syslog writes messages concerning this to /var/log/messages. The "dmesg" command shows this log. You can check near the end of the log to see which channel the device was attached to, it is usually /dev/sd(letter)(number) depending on the partitions and number of serial disks plugged into the system.
/proc/partitions shows all the partitions known to the kernel.
I did this using zenity in a script and udev with rule on rhel6 with:
KERNEL=="sd[b-d]", DRIVERS=="usb", ACTION=="add", RUN+="/path/to/script"