I have a GPS module connected through serial port(USB->Virtual COM port). A measurement software is using this port, so with other software I can't access to the data. I would like to create two virtual COM port and share this data through that. Is it possible using Python? Is there any opensource example written in Python?
I don't think you can do that if you cannot modify the sources of the measurement software.
Serial port protocols are written as "point to point" protocols, so there's no general way to multiplex them. You can write a program that shares the access to the GPS module (handling it exclusively and exposing an API to multiple programs), but every program that wanted to use the GPS module should be written to talk to your API and not directly to the serial port - and in this case it can be done only if you can change the measurement software.
Notice that probably it's not impossible to implement your "virtual port" solution, but it would be an ad-hoc hack (it would work just with that specific protocol) and it may be quite complicated: you would need to emulate two GPS modules and multiplex the requests to the real GPS module; depending on how does it work (e.g. if it has a "complicated" persistent state) it may be simple or very complicated. But surely Python wouldn't be enough, to emulate serial ports you have to go in kernel mode.
Do you need two-way communication, or just reading? You could build or buy hardware to physically split the Rx data line so you could use two COM ports, each of which would read the same data. You could do this with Tx data as well, but you would have to be careful about trashing the data if both ports tried to write at the same time.
Related
I have two python scripts which need to get data from the same modbus server.
However they cannot be simultaneously connected to the same modbus server, so I am looking for a "proxy" software.
This proxy will then sequentially transfer the modbus queries to the modbus server.
Solution should run on debian 11 lite.
Thanks!
I have the same challenges before and ended up with Modbus-to-MQTT bridge solution, because each protocol has its limit, Modbus is designed for master-slave poll in nature and MQTT is good for multiple clients/subscribers.
Modbus + MQTT, such a combination is just nice and could be very powerful in modern IoT applications. You can try modpoll, which is a python-based Modbus tool and can run as a Modbus-to-MQTT bridge in your case.
Assuming you are dealing with Modbus TCP you can try modbus-proxy:
Many modbus devices support only one or very few clients. This proxy
acts as a bridge between the client and the modbus device. It can be
seen as a layer 7 reverse proxy. This allows multiple clients to
communicate with the same modbus device.
When multiple clients are connected, cross messages are avoided by
serializing communication on a first come first served REQ/REP basis.
EDIT: According to your comment below:
...it is RTU over a linux file descriptor /dev/ttyS1
That seems to me a not-so-straightforward way of saying that you are dealing with a serial link.
If that's the case I'm afraid you need to go back all the way to the physical layer.
If you have a point-to-point serial connection (no matter the voltage levels you have, they can be TTL 3.3VDC or RS-232) there is no way you can have more than one Modbus client connecting to the same server.
For a serial link to have more than two devices (two or multiple clients sending queries to a Modbus server, for instance) you need a way for the devices to take control of the bus when they communicate and let go of it when they are idle. An RS-485 multipoint serial link is the most frequent solution to this problem.
If you follow the link you will find many resources and documentation. That might be a bit overwhelming but it is actually very easy: instead of connecting RX to TX as you'd do for a normal serial port you will have two cables for each device (they are called A and B or D+ and D-) that you can put together. And you only need to add a TTL to RS-485 transceiver in between each device and the bus (or RS-232 to RS-485, depending on your devices).
There are a couple of nuances:
You might need terminating resistors, but that should only be the case if your devices are far away (more than several meters) from each other.
If you go for the cheapest transceivers you might end up being forced to toggle the bus in your software, which is not a good solution. See my answer here for more details. And better aim for something with automatic TX enable.
The most complete reference on all these topics I know of is Jan Axelson's book Serial Port Complete. It's a bit old (the last edition was released in 2007) but still very relevant. And you can find older editions secondhand for next to nothing. If you need to work with serial ports, this book will be a very valuable companion.
Final note: if you are not able or willing to mess with the phy layer, you might, of course, find other solutions that are more palatable. You might, for instance, implement a software forwarder to take queries from your clients and convert them to Modbus RTU to be sent to your server. Pymodbus includes an example, but since you did not elaborate much I'm not sure it will be useful for your situation.
Is it possible, with Python, to communicate directly on the data link layer, prior-to or outside of, an IP address? Similarly to communicating with USB?
I have a client interested in trying this. As far I can know, there's no way. But I never want to underestimate the power of Python.
There's nothing intrinsic about Python which prevents you from writing your own user-level networking stack. However, if you want to say, access the raw ethernet driver to send raw ethernet packets, that has to be supported by the operating system.
I'll try to paint a vague picture of what's going on. Some of this you may know already (or not). A conventional operating system provides an abstraction called the system call layer to allow programs to interact with hardware. This abstraction is typically somewhat "high level" in that it abstracts away some of the details of the hardware. In an operating system which implements Unix abstractions, one of the network abstraction system calls is socket(int domain, int type, int proto), which creates a new socket endpoint. What's getting abstracted away here? Well, for most protocols, dealing with data-link layer details becomes unnecessary. Obviously you lose some flexibility here so that you can gain safety (you don't have to worry about clobbering other OS data structures if you have raw hardware access) and convenience (most people don't need to implement a user-level networking stack).
So, whether it "can" be done without modifying your kernel depends on what abstractions are provided by the OS. Linux provides the packet(7) interface which allows you to use AF_PACKET as your socket domain. According to the man page, "Packet sockets are used to receive or send raw packets at the device driver (OSI Layer 2) level."
So can this be accessed in Python? You bet!
import socket
s = socket(socket.AF_PACKET, socket.SOCK_RAW)
s.bind(("eth1", 0))
s should now be a socket which you can use to send raw packets. See the other Stack Overflow post for more information about how to do this -- they do a better job than I can. It looks like this technique should work on Windows as well, as I suspect they provide a similar abstraction.
A python program needs to accept a string every second from a serial port. I plan on using a RS-232 to USB converter. The application is going to work under Ubuntu 10.04.
How do I approach this? Do I use pySerial or libusb?
There needs to be done some processing in the meantime, so synchronous communication is not viable. Do I use some kind of interrupts or do I need to open separate threads? Or do I use blocking reads, believing that 1s is enough for my computations (it is plenty ... for now)?
I know, RTFM, but heading in the correct direction from the start will save me a lot of time! Thanks for bearing with me.
If your RS232-USB converter has a driver in Ubuntu that makes it look like a COM port then you will want to use pySerial (the interface is the same as any other COM port). If there is no driver for your device then you might have to use libusb and find the protocol for your specific device. Most major RS232-USB converters these days have usbserial based drivers committed and maintained in the Linux kernel. Simply check with your vendor for this.
There are many ways to do parallel processing but typically I write my applications in two ways:
Have a read thread that does nothing but read and fill a local thread safe buffer so data is ready for other threads when needed.
Have a read thread that reads data, determines where it goes and delivers it via messaging/event processing to the component that needs it.
The decisions here will depend on what your goals are, and how much processing is needed outside the reading.
If this is a stupid question, please don't mind me. But I spent some time trying to find the answer but I couldn't get anything solid. Maybe this is a hardware question, but I figured I'd try here first.
Does Serial Communication only work one to one? The reason this came up is because I had an arduino board listening for communication on its serial port. I had a python script feed bytes to the port as well. However, whenever I opened up the arduino's serial monitor, the connection with the python script failed. The serial monitor also connects to the serial port for communication for its little text input field.
So what's the deal? Does serial communication only work between a single client and a single server? Is there a way to get multiple clients writing to the server? I appreciate your suggestions.
Multiple clients (e.g. Arduinos) communicating with one server (e.g. a desktop computer) is commonly done with the serial variant:
RS-485
This is a simple method widely used in industrial settings where you want to have many devices connected to one computer via one serial port. This type of arrangement is also called multi-drop, because one cable strings around a building with Tees that tap in and drop lines to each device.
The hardware for this is widely available. You can buy USB serial adapters that provide the hardware interface for a computer. Programmatically the port looks just like an RS232 port. For the Arduino you would just add a transceiver chip. A sea of serial transceivers exists, e.g.
Example computer USB adapter with 485 interface
Sample RS485 transceiver chip from Element14
All the devices hang on the same bus listening at the same time. A simple communication protocol used is just add a device address before every command. For example:
001SETLIGHT1 <- tells Arduino "001" to turn on the light
013SETLIGHT0 <- tells "013" to turn off the light
Any device hanging on the cable ignores commands that do not start with their address. When a device responds, it prepends its address.
001SETLIGHT1DONE <- response from device "001" that the command has been received and executed
The address in the response lets the receiving party know which device was talking.
Well, your question can be quite wide, so I'm going to layer my answer:
On the hardware side, the same pair of wires can work be shared with many devices. It is mostly a question of electronics (maintaining the signal in the good voltage range), and not having all devices writing to the serial port at the same time (or you'll get wreckage).
On the software side, on the host, yes you can share the same serial connection to a device with multiple processes. But that's not straight forward. I'll assume you're using an unix (macos or linux):
in unix, everything is a file, your serial connection is one too: /dev/ttyACM0 on linux, for example.
When you have a process opening that file, it will block it (using ioctl, iirc) so no other process can mess with that file too.
Then, you can input and output to that file using the process that opened it, that's all.
But hopefully, it is still possible to share the connection between processes. One of them would simply be to use the tee command, that will be able to get input from one process, and give it back output, and copy the output to another process. You can also do it from within python, by duplicating the file descriptor.
To easily output stuff that can be redirected the unix way (using pipes), you can use socat: http://www.dest-unreach.org/socat/
here's an usage example:
socat -,raw,echo=0,escape=0x0f /dev/ttyACM0,raw,echo=0,crnl
you may want to tweak it for your needs.
Edit:
I forgot about RS-485, which 'jdr5ca' was smart enough to recommend. My explanation below is restricted to RS-232, the more "garden variety" serial port. As 'jdr5ca' points out, RS-485 is a much better alternative for the described problem.
Original:
To expand on zmo's answer a bit, it is possible to share serial at the hardware level, and it has been done before, but it is rarely done in practice.
Likewise, at the software driver level, it is again theoretically possible to share, but you run into similar problems as the hardware level, i.e. how to "share" the link to prevent collisions, etc.
A "typical" setup would be two serial (hardware) devices attached to each other 1:1. Each would run a single software process that would manage sending/receiving data on the link.
If it is desired to share the serial link amongst multiple processes (on either side), the software process that manages the link would also need to manage passing the received data to each reading process (keeping track of which data each process had read) and also arbitrate which sending process gets access to the link during "writes".
If there are multiple read/write processes on each end of the link, the handshaking/coordination of all this gets deep as some sort of meta-signaling arrangement may be needed to coordinate the comms between the process on each end.
Either a real mess or a fun challenge, depending on your needs and how you view such things.
I guess it's socket programming. But I have never done socket programming expect for running the tutorial examples while learning Python. I need some more ideas to implement this.
What I specifically need is to run a monitoring program of a server which will poll or listen to traffic being exchange from different IPs across different popular ports. For example, how do I get data received and sent through port 80 of 192.168.1.10 and 192.168.1.1 ( which is the gateway).
I checked out a number of ready made tools like MRTG, Bwmon, Ntop etc but since we are looking at doing some specific pattern studies, we need to do data capturing within the program.
Idea is to monitor some popular ports and do a study of network traffic across some periods and compare them with some other data.
We would like to figure a way to do all this with Python....
You probably want to use scapy for that. Just sniff all ethernet traffic on a particular interface, drop everything that is not TCP and doesn't match the port.
Not sure if scapy can already track TCP connections (stuff like recognizing duplicate sequence numbers, extracting just the payload stream) but I would guess it probably can, and if not it's not too hard to hack together a good-enough TCP connection tracker that works for 95% of the traffic.
Alternatives would be to use sockets directly (look for raw sockets) or libpcap, which can both be done from Python. You may also want to check out the filter experssion syntax of the 'tcpdump' commandline tool, maybe it can do what you want already.
I bet there are more specialized high-level tools for this, but I don't know them.
PS: if you don't know wireshark yet, go check it out and play around with it first. It can follow TCP streams and will teach you what TCP connection tracking means. Maybe its commandline binary, tshark, can be used to extract TCP streams for what you want.
IPTraf is an ncurses based IP LAN monitoring tool. Has a capability to generate network statistics including TCP,UDP,ICMP and some more.
Since you're thinking to execute it from python, you may consider to use screen (screen manager with VT100/ANSI terminal emulation) to overcome ncurses issues and you may want to pass logging and interval parameters to IPTraf which forces iptraf to log to a file in a given interval. Little bit tricky but eventually you can have what you are looking for by basically parsing the log file.