If this is a stupid question, please don't mind me. But I spent some time trying to find the answer but I couldn't get anything solid. Maybe this is a hardware question, but I figured I'd try here first.
Does Serial Communication only work one to one? The reason this came up is because I had an arduino board listening for communication on its serial port. I had a python script feed bytes to the port as well. However, whenever I opened up the arduino's serial monitor, the connection with the python script failed. The serial monitor also connects to the serial port for communication for its little text input field.
So what's the deal? Does serial communication only work between a single client and a single server? Is there a way to get multiple clients writing to the server? I appreciate your suggestions.
Multiple clients (e.g. Arduinos) communicating with one server (e.g. a desktop computer) is commonly done with the serial variant:
RS-485
This is a simple method widely used in industrial settings where you want to have many devices connected to one computer via one serial port. This type of arrangement is also called multi-drop, because one cable strings around a building with Tees that tap in and drop lines to each device.
The hardware for this is widely available. You can buy USB serial adapters that provide the hardware interface for a computer. Programmatically the port looks just like an RS232 port. For the Arduino you would just add a transceiver chip. A sea of serial transceivers exists, e.g.
Example computer USB adapter with 485 interface
Sample RS485 transceiver chip from Element14
All the devices hang on the same bus listening at the same time. A simple communication protocol used is just add a device address before every command. For example:
001SETLIGHT1 <- tells Arduino "001" to turn on the light
013SETLIGHT0 <- tells "013" to turn off the light
Any device hanging on the cable ignores commands that do not start with their address. When a device responds, it prepends its address.
001SETLIGHT1DONE <- response from device "001" that the command has been received and executed
The address in the response lets the receiving party know which device was talking.
Well, your question can be quite wide, so I'm going to layer my answer:
On the hardware side, the same pair of wires can work be shared with many devices. It is mostly a question of electronics (maintaining the signal in the good voltage range), and not having all devices writing to the serial port at the same time (or you'll get wreckage).
On the software side, on the host, yes you can share the same serial connection to a device with multiple processes. But that's not straight forward. I'll assume you're using an unix (macos or linux):
in unix, everything is a file, your serial connection is one too: /dev/ttyACM0 on linux, for example.
When you have a process opening that file, it will block it (using ioctl, iirc) so no other process can mess with that file too.
Then, you can input and output to that file using the process that opened it, that's all.
But hopefully, it is still possible to share the connection between processes. One of them would simply be to use the tee command, that will be able to get input from one process, and give it back output, and copy the output to another process. You can also do it from within python, by duplicating the file descriptor.
To easily output stuff that can be redirected the unix way (using pipes), you can use socat: http://www.dest-unreach.org/socat/
here's an usage example:
socat -,raw,echo=0,escape=0x0f /dev/ttyACM0,raw,echo=0,crnl
you may want to tweak it for your needs.
Edit:
I forgot about RS-485, which 'jdr5ca' was smart enough to recommend. My explanation below is restricted to RS-232, the more "garden variety" serial port. As 'jdr5ca' points out, RS-485 is a much better alternative for the described problem.
Original:
To expand on zmo's answer a bit, it is possible to share serial at the hardware level, and it has been done before, but it is rarely done in practice.
Likewise, at the software driver level, it is again theoretically possible to share, but you run into similar problems as the hardware level, i.e. how to "share" the link to prevent collisions, etc.
A "typical" setup would be two serial (hardware) devices attached to each other 1:1. Each would run a single software process that would manage sending/receiving data on the link.
If it is desired to share the serial link amongst multiple processes (on either side), the software process that manages the link would also need to manage passing the received data to each reading process (keeping track of which data each process had read) and also arbitrate which sending process gets access to the link during "writes".
If there are multiple read/write processes on each end of the link, the handshaking/coordination of all this gets deep as some sort of meta-signaling arrangement may be needed to coordinate the comms between the process on each end.
Either a real mess or a fun challenge, depending on your needs and how you view such things.
Related
I have two python scripts which need to get data from the same modbus server.
However they cannot be simultaneously connected to the same modbus server, so I am looking for a "proxy" software.
This proxy will then sequentially transfer the modbus queries to the modbus server.
Solution should run on debian 11 lite.
Thanks!
I have the same challenges before and ended up with Modbus-to-MQTT bridge solution, because each protocol has its limit, Modbus is designed for master-slave poll in nature and MQTT is good for multiple clients/subscribers.
Modbus + MQTT, such a combination is just nice and could be very powerful in modern IoT applications. You can try modpoll, which is a python-based Modbus tool and can run as a Modbus-to-MQTT bridge in your case.
Assuming you are dealing with Modbus TCP you can try modbus-proxy:
Many modbus devices support only one or very few clients. This proxy
acts as a bridge between the client and the modbus device. It can be
seen as a layer 7 reverse proxy. This allows multiple clients to
communicate with the same modbus device.
When multiple clients are connected, cross messages are avoided by
serializing communication on a first come first served REQ/REP basis.
EDIT: According to your comment below:
...it is RTU over a linux file descriptor /dev/ttyS1
That seems to me a not-so-straightforward way of saying that you are dealing with a serial link.
If that's the case I'm afraid you need to go back all the way to the physical layer.
If you have a point-to-point serial connection (no matter the voltage levels you have, they can be TTL 3.3VDC or RS-232) there is no way you can have more than one Modbus client connecting to the same server.
For a serial link to have more than two devices (two or multiple clients sending queries to a Modbus server, for instance) you need a way for the devices to take control of the bus when they communicate and let go of it when they are idle. An RS-485 multipoint serial link is the most frequent solution to this problem.
If you follow the link you will find many resources and documentation. That might be a bit overwhelming but it is actually very easy: instead of connecting RX to TX as you'd do for a normal serial port you will have two cables for each device (they are called A and B or D+ and D-) that you can put together. And you only need to add a TTL to RS-485 transceiver in between each device and the bus (or RS-232 to RS-485, depending on your devices).
There are a couple of nuances:
You might need terminating resistors, but that should only be the case if your devices are far away (more than several meters) from each other.
If you go for the cheapest transceivers you might end up being forced to toggle the bus in your software, which is not a good solution. See my answer here for more details. And better aim for something with automatic TX enable.
The most complete reference on all these topics I know of is Jan Axelson's book Serial Port Complete. It's a bit old (the last edition was released in 2007) but still very relevant. And you can find older editions secondhand for next to nothing. If you need to work with serial ports, this book will be a very valuable companion.
Final note: if you are not able or willing to mess with the phy layer, you might, of course, find other solutions that are more palatable. You might, for instance, implement a software forwarder to take queries from your clients and convert them to Modbus RTU to be sent to your server. Pymodbus includes an example, but since you did not elaborate much I'm not sure it will be useful for your situation.
A python program needs to accept a string every second from a serial port. I plan on using a RS-232 to USB converter. The application is going to work under Ubuntu 10.04.
How do I approach this? Do I use pySerial or libusb?
There needs to be done some processing in the meantime, so synchronous communication is not viable. Do I use some kind of interrupts or do I need to open separate threads? Or do I use blocking reads, believing that 1s is enough for my computations (it is plenty ... for now)?
I know, RTFM, but heading in the correct direction from the start will save me a lot of time! Thanks for bearing with me.
If your RS232-USB converter has a driver in Ubuntu that makes it look like a COM port then you will want to use pySerial (the interface is the same as any other COM port). If there is no driver for your device then you might have to use libusb and find the protocol for your specific device. Most major RS232-USB converters these days have usbserial based drivers committed and maintained in the Linux kernel. Simply check with your vendor for this.
There are many ways to do parallel processing but typically I write my applications in two ways:
Have a read thread that does nothing but read and fill a local thread safe buffer so data is ready for other threads when needed.
Have a read thread that reads data, determines where it goes and delivers it via messaging/event processing to the component that needs it.
The decisions here will depend on what your goals are, and how much processing is needed outside the reading.
I have a Python test program for testing features of another software component, let's call the latter the component under test (COT).
The Python test program is connected to the COT via a persistent TCP connection.
The Python program is using the Python socket API for this.
Now in order to simulate a failure of the physical link, I'd like to have the Python program shut the socket down, but without disconnecting appropriately.
I.e. I don't want anything to be sent on the TCP channel any more, including any TCP SYN/ACK/FIN. I just want the socket to go silent. It must not respond to the remote packets any more.
This is not as easy as it seems, since calling close on a socket will send TCP FIN packets to the remote end. (graceful disconnection).
So how can I kill the socket without sending any packets out?
I cannot shut down the Python program itself, because it needs to maintain other connections to other components.
For information, the socket runs in a separate thread. So I thought of abruptly killing the thread, but this is also not so easy. (Is there any way to kill a Thread?)
Any ideas?
You can't do that from a userland process since in-kernel network stack still holds resources and state related to given TCP connection. Event if you kill your whole process the kernel is going to send a FIN to the other side since it knows what file descriptors your process had and will try to clean them up properly.
One way to get around this is to engage firewall software (on local or intermediate machine). Call a script that tells the firewall to drop all packets from/to given IP and port (that of course would need appropriate administrative privileges).
Contrary to Nikolai's answer, there is indeed a way to reset the connection from userland such that an RST is sent and pending data discarded, rather than a FIN after all the pending data. However as it is more abused than used, I won't publish it here. And I don't know whether it can be done from Python. Setting one of the three possible SO_LINGER configurations and closing will do it. I won't say more than that, and I will say that this technique should only be used for the purpose outlined in the question.
I guess it's socket programming. But I have never done socket programming expect for running the tutorial examples while learning Python. I need some more ideas to implement this.
What I specifically need is to run a monitoring program of a server which will poll or listen to traffic being exchange from different IPs across different popular ports. For example, how do I get data received and sent through port 80 of 192.168.1.10 and 192.168.1.1 ( which is the gateway).
I checked out a number of ready made tools like MRTG, Bwmon, Ntop etc but since we are looking at doing some specific pattern studies, we need to do data capturing within the program.
Idea is to monitor some popular ports and do a study of network traffic across some periods and compare them with some other data.
We would like to figure a way to do all this with Python....
You probably want to use scapy for that. Just sniff all ethernet traffic on a particular interface, drop everything that is not TCP and doesn't match the port.
Not sure if scapy can already track TCP connections (stuff like recognizing duplicate sequence numbers, extracting just the payload stream) but I would guess it probably can, and if not it's not too hard to hack together a good-enough TCP connection tracker that works for 95% of the traffic.
Alternatives would be to use sockets directly (look for raw sockets) or libpcap, which can both be done from Python. You may also want to check out the filter experssion syntax of the 'tcpdump' commandline tool, maybe it can do what you want already.
I bet there are more specialized high-level tools for this, but I don't know them.
PS: if you don't know wireshark yet, go check it out and play around with it first. It can follow TCP streams and will teach you what TCP connection tracking means. Maybe its commandline binary, tshark, can be used to extract TCP streams for what you want.
IPTraf is an ncurses based IP LAN monitoring tool. Has a capability to generate network statistics including TCP,UDP,ICMP and some more.
Since you're thinking to execute it from python, you may consider to use screen (screen manager with VT100/ANSI terminal emulation) to overcome ncurses issues and you may want to pass logging and interval parameters to IPTraf which forces iptraf to log to a file in a given interval. Little bit tricky but eventually you can have what you are looking for by basically parsing the log file.
I have a GPS module connected through serial port(USB->Virtual COM port). A measurement software is using this port, so with other software I can't access to the data. I would like to create two virtual COM port and share this data through that. Is it possible using Python? Is there any opensource example written in Python?
I don't think you can do that if you cannot modify the sources of the measurement software.
Serial port protocols are written as "point to point" protocols, so there's no general way to multiplex them. You can write a program that shares the access to the GPS module (handling it exclusively and exposing an API to multiple programs), but every program that wanted to use the GPS module should be written to talk to your API and not directly to the serial port - and in this case it can be done only if you can change the measurement software.
Notice that probably it's not impossible to implement your "virtual port" solution, but it would be an ad-hoc hack (it would work just with that specific protocol) and it may be quite complicated: you would need to emulate two GPS modules and multiplex the requests to the real GPS module; depending on how does it work (e.g. if it has a "complicated" persistent state) it may be simple or very complicated. But surely Python wouldn't be enough, to emulate serial ports you have to go in kernel mode.
Do you need two-way communication, or just reading? You could build or buy hardware to physically split the Rx data line so you could use two COM ports, each of which would read the same data. You could do this with Tx data as well, but you would have to be careful about trashing the data if both ports tried to write at the same time.