I'm having a problem with a block of Python code reading in a string from an Arduino connected over USB. I understand that serial doesn't know what a string is or care. I'm using serial.readline, which from the documentation sounds like the perfect match, but my string isn't always complete. The weird problem is, the string doesn't always have the front of the string, but it always has the end of the string. I'm really lost on this and I'm sure it's just my lack of understanding about the nuances of reading serial data or how Python handles it.
In the code below, I loop through the serial interfaces until I find the one I'm looking for. I flush the input and give it a sleep for a couple seconds to make sure it has time to get a new read.
arduinoTemp = serial.Serial(iface, 9600, timeout=1)
arduinoTemp.flushInput()
arduinoTemp.flushOutput()
arduinoTemp.write("status\r\n".encode())
time.sleep(2)
read = arduinoTemp.readline().strip()
if read != "":
#check the string to make sure it's what I'm expecting.
I'm sending the string in JSON.
I'm expecting something in line with this:
{"id": "env monitor","distance": {"forward": {"num":"0","unit": "inches"}},"humidity": {"num":"0.00","unit": "%"},"temp": {"num":"0.00","unit": "fahrenheit"},"heatIndex": {"num":"0.00","unit": "fahrenheit"}}
I might get something back like this:
": t": "%"},"temp": {"num":"69.80","unit": "fahrenheit"},"heatIndex": {"num":"68.13","unit": "fahrenheit"}}
or this:
atIndex": {"num":"0.00","unit": "fahrenheit"}}
At first I thought it was the length of the string that might be causing some issues, but the cut off isn't always consistent, and since it has the end of the string, it stands to reason that it should have gotten everything before that.
I've verified that my Arduino is broadcasting correctly by interfacing with it directly and the Arduino IDE and serial monitor. This is definitely an issue with my Python code.
In (serial) communications you should always expect to receive partial answers.
A usual solution in this case is to add whatever you read from the serial to a string/buffer until you can parse it successfully with json.loads.
import serial
import json
import time
ser = serial.Serial('/dev/ttyACM0', 9600)
buffer = ''
while True:
buffer += ser.read()
try:
data = json.loads(buffer)
print(data)
buffer = ''
except json.JSONDecodeError:
time.sleep(1)
(From this answer).
Note that if you flush, you will lose data!
Also note that this is a somewhat simplified solution. Ideally the buffer should be reset to whatever remains after the successful parse. But as far as I know, the json module doesn't offer that functionality.
Related
i am new to python. I have some experience with Pascal and a little bit with C++.
At the moment i have to program some code for a research project demonstrator.
The setup is as follows:
We have a 868MHz radio master device. i can communicate with this device via a USB port (COM4 at the moment but may change in the future).
The 868MHz master communicates with a slave unit. The slave unit replies with a message that i can read from the USB port.
Until this point everything works good. I request data packages and also receive them.
From the moment of receiving the data packages i have a propblem i seem not
able to solve on myself.
I use Anaconda 32 bit with the Spyder editor
# -*- coding: utf-8 -*-
"""
Created on Thu May 7 13:35:59 2015
#author: roland
"""
import serial
portnr = 3 #Serial Port Number 0=Com1, 3=Com4
portbaud = 38400 #Baud rate
tiout = 0.1 #Timout in seconds
i = 1
wrword = ([0x02,0x04,0x00,0x00,0x00,0x02,0x71,0xF8])
try:
ser = serial.Serial(portnr, portbaud, timeout=tiout) # open port
except:
ser.close() # close port
ser = serial.Serial(portnr, portbaud, timeout=tiout) # open port
print(ser.name) # check which port was really used
while (i < 100):
ser.write(wrword)
seread = ser.readline()
print(seread)
i = i+1
sere = seread.split()
try:
readdat = str(sere[0])
except:
print("Index Error")
retlen = len(readdat)
print(retlen)
readdat = readdat[2:retlen-1]
print(readdat)
ser.close() # close port
The variable wrword is my request to the 868MHz radio master.
The Format is as follows:
0x02 Address of the unit
0x04 Command to send information from a certain register range
0x00 0x00 Address of first Register (Start address 0 is valid!)
0x00 0x02 Information how much registers are to be sent (in this case Registers 0 and 1 shall be transmitted to the Radio master)
0x71 0xF8 Checksum of the command sentence.
The program sends the command sequence successful to the master unit and the slave unit answers. Each time the command is send an answer is expected. Nevertheless it may happen that now correct answer is given thats why the
try command is in use.
I know i use ser.readline() but this is sufficient for the application.
I receive a list as answer from the USB Port.
The data look as follows:
b'\x02\x04\x04\x12\xb6\x12\xa5\xe0\xc1' (This is the Output from print(seread) )
For clarification this answer is correct and must be read as follows:
\x02 Address of the answering unit
\x04 Function that was executed (Read from certain register area)
\x04 Number of Bytes of the answer
\x12 \xb6 Value of first register (2 Byte)
\x12 \xa5 Value of second register (2 Byte)
\xe0 \xc1 Checksum of answer
If the data from the com port had all this Format i might be able to get the data values from the both Registers. But unfortunately the data format is not always the same.
Sometimes i receive answers in the following style:
b'\x02\x04\x04\x12\x8e\x12{\xe1T'
The answer is similar to the example above (different values in the Registers and different checksum) but the Format i receive has changed.
If i use the hex ASCII codes for the symbols obviously not hex values i find a valid answer telegram.
b'\x02\x04\x04\x12\x8e\x12{\xe1T'
becomes
b'\x02\x04\x04\x12\x8e\x12\x7b\xe1\x54'
when i Exchange the ASCII symbols by their hex code by Hand.
So my questions are:
Is it possible to force Python to give me the answer in a defined Format?
If not is it possible to handle the list or the string i can derive from the list in such a way that i get my values in the required format?
Does somebody can give me a hint how to extract my register values from the list and convert the two hex numbers of each register into one integer value for each register (the first value is the high byte, the second the low byte)?
Thanks in advance for your answer(s)
sincerely
Roland
I found a solution.
During a small testpiece of program i stumbled upon the fact that the variable seread contains already the data in a suitable and usable format for me.
I assume that the Spyder Editor causes the format change when displaying byte type objects.
If i Access the single Bytes using seread[i] while i is in range 0 to len(seread)-1 i receive the correct values for the single bytes.
So i can acess my data and calculate my measurement values as required.
Nevertheless thanks to keety for reading my question.
As you know sometimes we can't know what the size of the data(if there is no Content-Length in http response header).
What is the best way to receive http response data(use socket)?
The follow code can get all the data but it will blocking at buf = sock.recv(1024).
from socket import *
import sys
sock = socket(AF_INET, SOCK_STREAM)
sock.connect(('www.google.com', 80))
index = "GET / HTTP/1.1\r\nHOST:www.google.com\r\nConnection:keep-alive\r\n\r\n"
bdsock.send(index)
data = ""
while True:
buf = bdsock.recv(1024)
if not len(buf):
break
data += buf
I'm assuming you are writing the sender as well.
A classic approach is to prefix any data sent over the wire with the length of the data. On the receive side, you just greedily append all data received to a buffer, then iterate over the buffer each time new data is received.
So if I send 100 bytes of data, I would prefix an int 100 to the beginning of the packet, and then transmit. Then, the receiver knows exactly what it is looking for. IF you want to get fancy, you can use a special endline sequence like \x00\x01\x02 to indicate the proper end of packet. This is an easily implemented form of error checking.
Use a bigger size first, do a couple of tests, then see what is the lenght of those buffers, you will then have an idea about what would the maximum size be. Then just use that number +100 or so just to be sure.
Testing different scenarios will be your best bet on finding your ideal buf size.
It would also help to know what protocol you are using the sockets for, then we would have a better idea and response for you.
Today I got the same question again.
And I found the simple way is use httplib.
r = HTTPResponse(sock)
r.begin()
# now you can use HTTPResponse method to get what you want.
print r.read()
I am having some strange behavior while using urllib2 to open a URL and download a video.
I am trying to open a video resource and here is an example link:
https://zencoder-temp-storage-us-east-1.s3.amazonaws.com/o/20130723/b3ed92cc582885e27cb5c8d8b51b9956/b740dc57c2a44ea2dc2d940d93d772e2.mp4?AWSAccessKeyId=AKIAI456JQ76GBU7FECA&Signature=S3lvi9n9kHbarCw%2FUKOknfpkkkY%3D&Expires=1374639361
I have the following code:
mp4_url = ''
#response_body is a json response that I get the mp4_url from
if response_body['outputs'][0]['label'] == 'mp4':
mp4_url = response_body['outputs'][0]['url']
if mp4_url:
logging.info('this is the mp4_url')
logging.info(mp4_url)
#if I add the line directly below this then it works just fine
mp4_url = 'https://zencoder-temp-storage-us-east-1.s3.amazonaws.com/o/20130723/b3ed92cc582885e27cb5c8d8b51b9956/b740dc57c2a44ea2dc2d940d93d772e2.mp4?AWSAccessKeyId=AKIAI456JQ76GBU7FECA&Signature=S3lvi9n9kHbarCw%2FUKOknfpkkkY%3D&Expires=1374639361'
mp4_video = urllib2.urlopen(mp4_url)
logging.info('succesfully opened the url')
The code works when I add the designated line but it gives me a HTTP Error 403: Forbidden message when I don't which makes me think it is messing up the mp4_url somehow. But the confusing part is that when I check the logging line for mp4_url it is exactly what I hardcoded in there. What could the difference be? Are there some characters in there that may be disrupting it? I have tried converting it to a string by doing:
mp4_video = urllib2.urlopen(str(mp4_url))
But that didn't do anything. Any ideas?
UPDATE:
With the suggestion to use print repr(mp4_url) it is giving me:
u'https://zencoder-temp-storage-us-east-1.s3.amazonaws.com/o/20130723/b3ed92cc582885e27cb5c8d8b51b9956/b740dc57c2a44ea2dc2d940d93d772e2.mp4?AWSAccessKeyId=AKIAI456JQ76GBU7FECA&Signature=S3lvi9n9kHbarCw%2FUKOknfpkkkY%3D&Expires=1374639361'
And I suppose the difference is what is causing the error but what would be the best way to parse this?
UPDATE II:
It ended up that I did need to cast it to a string but also the source that I was getting the link (an encoded video) needed nearly a 60 second delay before it could serve that URL so that is why it kept working when I hardcoded it because it had that delay. Thanks for the help!
It would be better to simply dump the response obtained. This way you would be able to check what response_body['outputs'][0]['label'] evaluates to. In you case, you are initializing mp4_url to ''. This is not the same as None and hence the condition if mp4_url: will always be true.
You may want to check that the initial if statement where you check that response_body['outputs'][0]['label'] is correct.
After a long time learning python I finally managed to make some breakthroughs:
I'm using the following code to connect to a personal communications terminal:
from ctypes import *
import sys
PCSHLL32 = windll.PCSHLL32
hllapi = PCSHLL32.hllapi
def connect_pcomm(presentation_space):
function_number = c_int(1)
data_string = c_char_p(presentation_space)
lenght = c_int(4)
ps_position = c_int(0)
hllapi(byref(function_number), data_string, byref(lenght), byref(ps_position))
And so far so good. It does connect to the terminal and I can use other functions to send keys to the screen, disconnect, etc etc etc.
My problem is with function 5, as defined by the IBM documentation:
http://publib.boulder.ibm.com/infocenter/pcomhelp/v5r9/index.jsp?topic=/com.ibm.pcomm.doc/books/html/emulator_programming08.htm
''The Copy Presentation Space function copies the contents of the host-connected presentation space into a data string that you define in your EHLLAPI application program.''
The code I wrote to do this (which is not that special):
def copy_presentation_space():
function_number = c_int(5)
data_string = c_char_p("")
lenght = c_int(0)
ps_position = c_int(0)
hllapi(byref(function_number), data_string, byref(lenght), byref(ps_position))
The main problem is the data_string var is supposed to be: "Preallocated target string the size of your host presentation space."
Since I wasn't exactly aware of what this means I simply tried to run the code. And pythonw.exe crashed. Epically. The terminal window proceeded to crash too. It didn't give any type of error, it simply said that it stopped working.
Now, my main question is, how can I preallocate the string like it's mentioned on the IBM ref. material?
Can I simply add a 'print data_string' after copying the screen to see the information, or do I need to use some ctypes to be able to view the copied information?
EDIT:
I forgot to mention that I don't need to use that function, I could just use this one:
Copy Presentation Space to String (8)
I tried to use it, but the data_string variable never changes value.
EDIT2:
Following kwatford suggestion, I changed the line
data_string = c_char_p("")
To
data_string = create_string_buffer(8000)
Now the function will not crash and returns a value of 0, meaning that: "'The host presentation space contents were copied to the application program. The target presentation space was active, and the keyboard was unlocked.' But when I try to print the variable data_string I still get an empty result.
You can create a preallocated string buffer using ctypes.create_string_buffer.
However, you'll still need to know how large the buffer is going to be. I'm not familiar with the software you're trying to run, but it sounds like you'll need:
Space for at least 25x80 Unicode characters
Possibly space for extended attributes for those characters
So as a rough guess, I'd say the string should have at least 25*80*2*2 = 8000 bytes.
I recommend reading the documentation in greater depth to determine the correct value if that doesn't work.
I have data coming in to a python server via a socket. Within this data is the string '<port>80</port>' or which ever port is being used.
I wish to extract the port number into a variable. The data coming in is not XML, I just used the tag approach to identifying data for future XML use if needed. I do not wish to use an XML python library, but simply use something like regexp and strings.
What would you recommend is the best way to match and strip this data?
I am currently using this code with no luck:
p = re.compile('<port>\w</port>')
m = p.search(data)
print m
Thank you :)
Regex can't parse XML and shouldn't be used to parse fake XML. You should do one of
Use a serialization method that is nicer to work with to start with, such as JSON or an ini file with the ConfigParser module.
Really use XML and not something that just sort of looks like XML and really parse it with something like lxml.etree.
Just store the number in a file if this is the entirety of your configuration. This solution isn't really easier than just using JSON or something, but it's better than the current one.
Implementing a bad solution now for future needs that you have no way of defining or accurately predicting is always a bad approach. You will be kept busy enough trying to write and maintain software now that there is no good reason to try to satisfy unknown future needs. I have never seen a case where "I'll put this in for later" has led to less headache later on, especially when I put it in by doing something completely wrong. YAGNI!
As to what's wrong with your snippet other than using an entirely wrong approach, angled brackets have a meaning in regex.
Though Mike Graham is correct, using regex for xml is not 'recommended', the following will work:
(I have defined searchType as 'd' for numerals)
searchStr = 'port'
if searchType == 'd':
retPattern = '(<%s>)(\d+)(</%s>)'
else:
retPattern = '(<%s>)(.+?)(</%s>)'
searchPattern = re.compile(retPattern % (searchStr, searchStr))
found = searchPattern.search(searchStr)
retVal = found.group(2)
(note the complete lack of error checking, that is left as an exercise for the user)