I have a python script which will parse xml file for serial numbers and will write them to a text file. The problem with the below code is, It is going on infinite loop. If I am adding a break statement some where after logging to a file, It is writing only one serial number. How do I increase the counter, so that the program will exit after writing all the serial numbers.
try:
while True:
data, addr = s.recvfrom(65507)
mylist=data.split('\r')
url = re.findall('http?://(?:[a-zA-Z]|[0-9]|[$-_#.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+', data)
print url[0]
response = urllib2.urlopen(url[0])
the_page = response.read()
tree = ET.XML(the_page)
with open("temp.xml", "w") as f:
f.write(ET.tostring(tree))
document = parse('temp.xml')
actors = document.getElementsByTagName("ns0:serialNumber")
for act in actors:
for node in act.childNodes:
if node.nodeType == node.TEXT_NODE:
r = "{}".format(node.data)
print r
logToFile(str(r))
time.sleep(10)
s.sendto(msg, ('239.255.255.250', 1900) )
except socket.timeout:
pass
I would normally create a flag so that the while would be
while working == True:
Then reset the flag at the appropriate time.
This allows you to use the else statement to close the text file and output the final results after the while loop is complete. Else clause on Python while statement.
Note that it is always better to explicitly close open files when finished rather than relying on garbage collection. You should also close the file and output a timeout message in the except logic.
For debugging, you can output a statement at each write to the text file.
If your s.recvfrom(65507) is working correctly it should be an easy fix. Write this code just below your data, addr = s.recvfrom(65507)
if not data:
break
You open a UDP socket and you use recvfrom to get data from the socket.
You set a high timeout which makes this function a blocking function. It means when you start listening on the socket, if no data have been sent from the sender your program will be blocked on that line until either the sender sends something or the timeout reaches. In case of timeout and no data the function will raise an Exception.
I see two options:
Send something from the sender that indicates the end of stream (the serial numbers in your case).
Set a small timeout then catch the Exception and use it to break the loop.
Also, take a look at this question: socket python : recvfrom
Hope it helps.
Related
I have created a client/server setup for transferring PGP signature information from the client to the server. The code below shows part of the server code, where adds signatures received from the client to an output text file
However, my code isn't able to move on after the second while loop, after breaking.
It receives 2 signatures from the client, and successfully prints the "test" string only twice and adds both received strings to the output file, but the program will not continue after breaking, and doesn't print the other "test2" and "test3" strings.
while True:
# Accepts incoming connection, creating socket used for data transfer to client
conn, addr = server.accept()
print("Connected successfully")
directory = "(Hidden for question)"
outputFile = open((directory + "\\signatures.txt"), "w")
while True:
data = conn.recv(2048)
if not data: break
print("test")
outputFile.write(data.decode()+"\n")
outputFile.flush()
print("test2")
conn.close()
print("test3")
I feel like I am missing something very obvious but cannot figure out what the issue is.
Your loop will never break as the recv function on a socket is a blocking call.
This means the function will not return until it receives some data, there for not data will always be false.
Try sending more information (after the first 2 signatures) into the socket and see that your script will continue to write it into the file.
If you want to receive a specific amount of data/times, track it using a variable and break your loop using that.
Alternatively to #Nadav's answer, remove the inner while loop. Since recv() is synchronous, you don't need to loop.
I write the following code to receive data, then write the data to a file.
My question is:
I found the if branch (" if not data: break ") never be executed,
1). Why does the if branch never be reached?
2). How my code can exit the while loop?
s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
s.connect("/var/run/output.socket")
while True:
data = s.recv(1024)
if not data:
break
else:
f = open("/home/ematt/test.log",'a')
f.write(data)
socket.recv always has data. It will wait until some data arrives.
See https://docs.python.org/2/library/socket.html
socket.recv is a blocking call, it returns when data has been received (or the peer has closed the connection)
In case you want to avoid waiting on data to arrive, you can set the socket to non-blocking mode using
s.setblocking(False)
In such mode, recv throws an exception when no data is available.
Please see this SO QA for more info and a code example on how to use it
In Python 3.4/Asyncio I'm using StreamReader/Writer.
To detect a client disconnect the common method seems to be to read from the client and if there's nothing there then the client disconnected.
while True:
data = (yield from asyncio.wait_for(client_reader.readline(),
timeout=1.0))
if not data: #client disconnected
break
However quickly you run out of lines to read from the client header (it moves to the next line on each loop) and if there are no additional lines sent from the client (in my case the client is not sending just listening) you hit the timeout.
What I would like to do is to only read the first line of the header over and over.. or possibly even just the first character of the first line, or if that's not possible when it gets to the last line loop back around to the first.
What's the best/most elegant way to accomplish this task? 3.4/Asyncio/StreamReader/Writer. (detecting client disconnects)
I had a similar problem. The way that worked for me was to check for EOF first and then raise a ConnectionError exception if true. So for your code I would add the following:
while True:
try:
if client_reader.at_eof():
raise ConnectionError
data = (yield from asyncio.wait_for(client_reader.readline(),
timeout=1.0))
if not data: #client disconnected
break
except ConnectionError:
break
except:
break # This is here to catch things like the asyncio futures timeout exception
Hope that helps. If anyone has a better way I'd be interested.
I am writing python code that reads and writes to a serial device. The device is basically an Arduino Mega running the Marlin 3D printer firmware.
My python code is sending a series of GCode commands (ASCII strings terminated by newlines, including checksums and line numbers). Marlin responds to each successfully received line with an "ok\n". Marlin only has a limited line buffer size so if it is full Marlin will hold off on sending the "ok\n" response until space is freed up.
If the checksum fails then Marlin requests the line to be sent again with a "Resend: 143\n" response. Another possible response is "ok T:{temp value}\n" if the current temperature is requested.
My code uses three threads. The main thread, a read thread and a write thread. Here is a stripped down version of the code:
class Printer:
def connect(self):
self.s = serial.Serial(self.port, self.baudrate, timeout=3)
self.ok_received.set()
def _start_print_thread(self):
self.print_thread = Thread(target=self._empty_buffer, name='Print')
self.print_thread.setDaemon(True)
self.print_thread.start()
def _start_read_thread(self):
self.read_thread = Thread(target=self._continous_read, name='Read')
self.read_thread.setDaemon(True)
self.read_thread.start()
def _empty_buffer(self):
while not self.stop_printing:
if self.current_line_idx < len(self.buffer):
while not self.ok_received.is_set() and not self.stop_printing:
logger.debug('waiting on ok_received')
self.ok_received.wait(2)
line = self._next_line()
self.s.write(line)
self.current_line_idx += 1
self.ok_received.clear()
else:
break
def _continous_read(self):
while not self.stop_reading:
if self.s is not None:
line = self.s.readline()
if line == 'ok\n':
self.ok_received.set()
continue # if we got an OK then we need to do nothing else.
if 'Resend:' in line: # example line: "Resend: 143"
self.current_line_idx = int(line.split()[1]) - 1
if line: # if we received _anything_ then set the flag
self.ok_received.set()
else: # if no printer is attached, wait 10ms to check again.
sleep(0.01)
In the above code, self.ok_received is a threading.Event. This mostly works ok. Once every couple of hours however it gets stuck in the while not self.ok_received.is_set() and not self.stop_printing: loop inside of _empty_buffer(). This kills the print by locking up the machine.
When stuck inside the loop, I can get the print to continue by sending any command manually. This allows the read thread to set the ok_recieved flag.
Since Marlin does not respond with checksums, I guess it is possible the "ok\n" gets garbled. The third if statement in the read thread is supposed to handle this by setting the flag if anything is received from Marlin.
So my question is: Do I have a possible race condition somewhere? Before I add locks all over the place or combine the two threads into one I would really like to understand how this is failing. Any advice would be greatly appreciated.
It looks like the read thread could get some data in the window where the write thread has broken out of the is_set loop, but has not yet called self.ok_received.clear(). So, the read thread ends up calling self.ok_received.set() while the write thread is still processing the previous line, and then the write thread unknowingly calls clear() once its done processing the previous message, and never knows that another line should be written.
def _empty_buffer(self):
while not self.stop_printing:
if self.current_line_idx < len(self.buffer):
while not self.ok_received.is_set() and not self.stop_printing:
logger.debug('waiting on ok_received')
self.ok_received.wait(2)
# START OF RACE WINDOW
line = self._next_line()
self.s.write(line)
self.current_line_idx += 1
# END OF RACE WINDOW
self.ok_received.clear()
else:
break
A Queue might be a good way to handle this - you want to write one line in the write thread every time the read thread receives a line. If you replaced self.ok_received.set() with self.recv_queue.put("line"), then the write thread could just write one line every time it pulls something from the Queue:
def _empty_buffer(self):
while not self.stop_printing:
if self.current_line_idx < len(self.buffer):
while not self.stop_printing:
logger.debug('waiting on ok_received')
try:
val = self.recv_queue.get(timeout=2)
except Queue.Empty:
pass
else:
break
line = self._next_line()
self.s.write(line)
self.current_line_idx += 1
else:
break
You could also shrink the window to the point you probably won't hit it in practice by moving the call to self.ok_received.clear() up immediately after exiting the inner while loop, but technically there will still be a race.
I've got a Python program which is reading data from a serial port via the PySerial module. The two conditions I need to keep in mind are: I don't know how much data will arrive, and I don't know when to expect data.
Based on this I have came up with the follow code snippets:
#Code from main loop, spawning thread and waiting for data
s = serial.Serial(5, timeout=5) # Open COM5, 5 second timeout
s.baudrate = 19200
#Code from thread reading serial data
while 1:
tdata = s.read(500) # Read 500 characters or 5 seconds
if(tdata.__len__() > 0): #If we got data
if(self.flag_got_data is 0): #If it's the first data we recieved, store it
self.data = tdata
else: #if it's not the first, append the data
self.data += tdata
self.flag_got_data = 1
So this code will loop forever getting data off the serial port. We'll get up to 500 characters store the data, then alert the main loop by setting a flag. If no data is present we'll just go back to sleep and wait.
The code is working, but I don't like the 5s timeout. I need it because I don't know how much data to expect, but I don't like that it's waking up every 5 seconds even when no data is present.
Is there any way to check when data becomes available before doing the read? I'm thinking something like the select command in Linux.
Note: I found the inWaiting() method, but really that seems it just change my "sleep" to a poll, so that's not what I want here. I just want to sleep until data comes in, then go get it.
Ok, I actually got something together that I like for this. Using a combination of read() with no timeout and the inWaiting() method:
#Modified code from main loop:
s = serial.Serial(5)
#Modified code from thread reading the serial port
while 1:
tdata = s.read() # Wait forever for anything
time.sleep(1) # Sleep (or inWaiting() doesn't give the correct value)
data_left = s.inWaiting() # Get the number of characters ready to be read
tdata += s.read(data_left) # Do the read and combine it with the first character
... #Rest of the code
This seems to give the results I wanted, I guess this type of functionality doesn't exist as a single method in Python
You can set timeout = None, then the read call will block until the requested number of bytes are there. If you want to wait until data arrives, just do a read(1) with timeout None. If you want to check data without blocking, do a read(1) with timeout zero, and check if it returns any data.
(see documentation https://pyserial.readthedocs.io/en/latest/)
def cmd(cmd,serial):
out='';prev='101001011'
serial.flushInput();serial.flushOutput()
serial.write(cmd+'\r');
while True:
out+= str(serial.read(1))
if prev == out: return out
prev=out
return out
call it like this:
cmd('ATZ',serial.Serial('/dev/ttyUSB0', timeout=1, baudrate=115000))