Develop Reference

Why doesn't my Python socket correctly send data but my serial terminal does?

I have to send some serial commands from a PC to an equipment as part of a bigger application and no matter what i try, i cannot seem to get python to send the data correctly. When i send the same commands through Termite, everything works as expected. I've been trying to solve this for days and i'm at a loss as to what i could try next.
The things i did and could rule out as not the cause for this issue:
Improper socket configuration: I've made sure to use the correct baud rate, bitlength, parity and stopbits as Termite (which happen to be the default settings for the socket anyways)
Check the command's correctness and termination: Within Termite i have to append CRLF as termination, so in Python i just add \r\n at the end of the command. This seems to be identical, as i've checked both commands on the oscilloscope and they are identical, termination characters included. I've went further and also used a serial port monitor and compared the two and they're identical - no missing bits and correct termination.
I've tried adding a heading \r\n to each command (this is what the equipment expects as command termination) thinking there might be some garbage or noise when first sending some data, but to no avail.
Clearing the input buffer, maybe there could be some issue when not reading from a socket that sends some response. I dont need what the equipment sends via serial, it offers no useful feedback but i did this anyways and have gotten no results. Before each transmission, i make sure to read all the bytes available.
Making sure Windows does not close my port. This i'm not that sure of, maybe it still does it after some long time, but so far i've gotten no errors and could always write and receive data, as confirmed by the serial monitor.
Below are some excerpts of socket configuration and command sending:
try:
self.SPC = serial.Serial(port=connectionData.get("spc_com"), baudrate=115200, timeout=1)
except serial.SerialException as err:
print(f'SPC Serial Error: {err.strerror}')
return False
def SPCCommand(self, command: str):
if not command:
return
try:
self.readAndClearBuffer()
self.SPC.write(command.encode())
except serial.SerialException as err:
print(f'SPC Serial Error: {err.strerror}')
return
def readAndClearBuffer(self):
data = ''
while True:
try:
data = self.SPC.read(1024)
if not data:
return
print(data)
except serial.SerialException as err:
return
self.SPCCommand("\r\n8000011200000000")
I don't have any idea what else to try. The fact that the commands are identical on the oscilloscope, as well as the serial monitor, leaves me at a loss. Could there be a different issue? Is there anything else i could try?

Python: File Can't be Found (pySerial)

Having a small problem with opening a serial/console port via pySerial.
My program is meant to get the active com port, open a console connection and then send data. When the program is running and I plug in my RS232 USB I receive a SerialException error. (More specifically, "Could not open port: FileNotFoundError")
In the event where the program is run, it will keep printing "No RS232 Connected", but when the RS232 USB is connected the program breaks and runs into the SerialException error.
If I plug in the RS232 USB before running the program and then run it, it reads and performs normal operation without issue.
ports = serial.tools.list_ports.comports(include_links=False)
if not ports:
print("No RS232 Connected")
if ports:
for port in ports:
print('Found port ' + port.device)
ser = serial.Serial(port.device)
if ser.isOpen():
ser.close()
break
console = serial.Serial(port.device, baudrate=9600, parity="N", stopbits=1, bytesize=8, timeout=0.4)
I am quite new to Python and programming in general, but I feel that the problem may be around the 'ports' list already being populated twice due to the while True loop. Then when we go to create the console by opening the port we are expecting one entry in the list, but there are two.
Since we can't have 2 open console connections on the same COM port, we receive the error.
If I print the 'ports' list I get this.
"[<serial.tools.list_ports_common.ListPortInfo object at 0x000002B5D77F0D68>]
[<serial.tools.list_ports_common.ListPortInfo object at 0x000002B5D77F0D68>]"
Any help would be greatly appreciated! Please let me know if you require any more details.
Thanks,

After further research into it, I have realised that when a RS232 USB is plugged into the PC, we need to give it a bit of time to open a stream. It sounds like it is opened when the temp file is created for it.
Although it was identifying that a COM port was available almost immediately, it was not ready by the time I was trying to create the Serial instance, which was why I was getting the FileNotFound error.
A simple sleep function of half a second has solved the issue!

how can i connect and send commands to a serial port while another program is using it?

I am using a telit he910g card. it is connected to my PC directly using a miniPCI slot.
I am using it for 3G internet connection and A-GPS/GPS services.
My system is running linux mint 17.1, the 3G connection is handled using the network manager APP and works great. The 3G connection is started and handled using a module that is part of a program I am writing.
The code I am using in order to connect to the serial port is this:
def _connect_to_device(self):
""" Connect to a serial port """
try:
self._device = serial.Serial(self._filename, baudrate=self._baud_rate)
except StandardError, e:
raise StandardError("Couldn't connect to GPS device. Error: %s" % str(e))
When I use the python program alone it works great. But when I try and use it while the 3G is on i cant connect to the serial device. The wierd thing is that if I try to connect to it using a program like "minicom" while 3G is turned on it DOES work.
So my question is: how can I make both run and work together? since now they are mutually exclusive.
thanks to all who help. :)

Glad you found a way round your problem. Just for completeness:
Normally, serial ports can be opened by multiple processes.
If one of them does ioctl(,TIOCEXCL) on the open file then further opens will return EBUSY until everyone closes the device. Only root can get past this and open the device at all times.
If root opens the device and does an ioctl(,TIOCNXCL), then other processes can open the device too.
In python, TIOCNXCL isnt defined anywhere, but you can do the ioctl (eg on stdin) with:
import fcntl
TIOCEXCL = 0x540c # from /usr/lib64/perl5/asm-generic/ioctls.ph
TIOCNXCL = 0x540d
print fcntl.ioctl(0, TIOCNXCL)

Ok, so it is solved.
the issue was that the telit module has 2 ports /dev/ttyACM0 (high speed) and /dev/ttyACM3 (lower speed).
I tried to connect to the high speed one, but apparently the 3G uses that one and it causes contentions.
So moving to use the lower speed port in my script solved the issue.

Avoiding TCP/IP connection hanging

I am communicating with an instrument via TCP/IP using the Python socket package.
The program sends a command to the instrument to perform an action, and then repetitively sends another "check" command until it receives a "done" reply. However, after many loops, the program hangs while waiting for a "done" reply.
I have circumvented this problem by using the recv_timeout() function below, which returns no data if the socket is hanging, then I close the connection with socket.close() and reconnect.
Is there a more elegant solution without having to reboot anything?
import socket
import time
def recv_timeout(self,timeout=0.5):
'''
code from http://code.activestate.com/recipes/408859/
'''
self.s.setblocking(0)
total_data=[];data='';begin=time.time()
while 1:There must be a way I can reboot to carry on communicating with the instrument, without having to restart.
#if you got some data, then break after wait sec
if total_data and time.time()-begin>timeout:
break
#if you got no data at all, wait a little longer
elif time.time()-begin>timeout*2:
break
try:
data=self.s.recv(8192)
if data:
total_data.append(data)
begin=time.time()
else:
time.sleep(0.1)
except:
pass
return ''.join(total_data)
sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
sock.connect(('555.555.55.555',23))
for action_num in range(0,1000):
socket.sendall(('performaction %s \r'%action_num).encode())
while True:
time.sleep(0.2)
socket.sendall(('checkdone \r').encode())
done = socket.recv_timeout()
if not done:
print 'communication broken...what should I do?'
socket.close()
time.sleep(60)
sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
sock.connect(('555.555.55.555',23))
elif done == '1':
print 'done performing action'
break
socket.close()

I have circumvented this problem by using the recv_timeout() function
below, which returns no data if the socket is hanging
Are you certain that the socket will hang forever? What about the possibility that the instrument just sometimes takes more than half a second to respond? (Note that even if the instrument's software is good at responding in a timely manner, that is no guarantee that the response data will actually get to your Python program in a timely manner. For example, if the TCP packets containing the response get dropped by the network and have to be resent, that could cause them to take more than .5 seconds to return to your program. You can force that scenario to occur by pulling the Ethernet cable out of your PC for a second or two, and then plugging it back in... you'll see that the response bytes still make it through, just a second or two later on (after the dropped packets get resent); that is, if your Python program hasn't given up on them and closed the socket already.
Is there a more elegant solution without having to reboot anything?
The elegant solution is to figure out what is happening to the reply bytes in the fault scenario, and fixing the underlying bug so that the reply bytes no longer get lost. WireShark can be very helpful in diagnosing where the fault is; for example if WireShark shows that the response bytes did enter your computer's Ethernet port, then that is a pretty good clue that the bug is in your Python program's handling of the incoming bytes(*). On the other hand if the response bytes never show up in WireShark, then there might be a bug in the instrument itself that causes it to fail to respond sometimes. Wireshark would also show you if the problem is that your Python script failed to send out the "check" command for some reason.
That said, if you really can't fix the underlying bug (e.g. because it's a bug in the instrument and you don't have the ability to upgrade the source code of the software running on the instrument) then the only thing you can do is what you are doing -- close the socket connection and reconnect. If the instrument doesn't want to respond for some reason, you can't force it to respond.
(*) One thing to do is print out the contents of the string returned by recv_timeout(). You may find that you did get a reply, but it just wasn't the '1' string you were expecting.

Launch script on any network connection

On linux (ubuntu), is it possible to execute a script upon any incoming network connection?
During long periods inactivity on my home server, I plan on stopping the software raid, parking the data disks and restarting the raid array upon encountering an incoming network connection.
I've been researching this problem and haven't come across an appropriate solution yet, any help is appreciated.
Neil
Note: I'm aware it's unnessecary to stop/start the array and there are downsides (disk wear & tear...). However I'm set in my ways, please do not post advising me not to stop/start the disks.
Edit (1):
Thought I'd share my opinion and hopefully get some feedback on it.
I think I'd need to use xinetd binding a catcher script that catches incoming connection data on all ports, launches my launch script and passes the incoming connection data to the relevant service for that port. Only issue is I don't know what data I would need to catch or how to in python.
Edit (2):
I've hacked together a solution using the python library pypcap http://code.google.com/p/pypcap/ . For future reference, here is the code.
import pcap
...
pc = pcap.pcap(name)
pc.setfilter(' '.join(args))
pc.setnonblock()
try:
for ts, pkt in pc:
# subprocess.call([filename, args])
print 'readying disks...'
break
except OSError, ValueError:
print 'Error: invalid arguements passed to subprocess'
except KeyboardInterrupt:
print 'Error: keyboard interupt'

Take a look at netfilter/iptables. You may be able to write code to use the userspace API libraries to detect incoming packet events.