I wanted to fetch some OBD data and tried it like described by this article
The article (using python 2.7) said:
The elm327 device returns values in HEX.
To read the value you just requested in Python type speed_hex = ser.readline().split(' ')
Convert the HEX to decimal by using: speed = float(int('0x'+speed_hex[3], 0 ))
But the answers I get from my OBD do not contain any whitespace furthermore they didn't look like they made any sense at all.
['\xd0MA\r?\r\r>\xd0\x15\r?\r\r>\x981\xf0\n']
so I gave up on this approach and oved to Python 3 instead.
I then wrote a little script inspired by this SO post
However I changed it to pull a lot of data periodically and added a timestamp, all together is saved to a csv file - as the path implies I am working on Windows (10 to be precise), COM is correctly configured and the car is OBD compliant.
Now when I run my script it prints out what it read:
b'h\xf4\rSTOPPED\r\r>'
and then tells me
Traceback (most recent call last):
File "python3test.py", line 36, in <module> r.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf4 in position 1:
invalid continuation byte
I do understand that encoding certainly is an issue, but this string does not make much sense at all. even if I take a closer look at 0xf4 - it would only be 244 (dec) which is no where near sensible RPM data.
Why does it work in the other script then? It can't be a Windows issue, can it?
Can anyone please help me understand whats going on in here?
Here is my code.
import serial
import time
import string
import io
import os
import sys
import datetime as dt
ser = serial.Serial("COM1")
ser.baudrate = 38400
ser.write(bytes('ATSP0\r\n', encoding = 'utf-8'))
ser.timeout = 1
ser.write(bytes('ATE0\r\n', encoding = 'utf-8'))
ser.timeout = 1
def millis_interval(start, end):
'''start and end are datetime instances'''
diff = end - start
millis = diff.days * 24 * 60 * 60 * 1000
millis += diff.seconds * 1000
millis += diff.microseconds / 1000
return millis
Testtime = str(dt.datetime.now()).split(' ')
PIDs = ["0C","0D","43","04","11","5C","05","10","46"]
response = []
''' open file and create header row'''
with open("C:/test/test_"+Testtime[0]+"_"+Testtime[1].replace(":","-")+".csv", "a") as myfile:
myfile.write("Time,Timestep,RPM,Speed, AbsLoad, CalcLoad, ThrottlePos, CoolTemp, OilTemp, MAF, AmbAirTemp")
start = dt.datetime.now() # need initial value
while 1:
end = dt.datetime.now()
myfile.write("\n"+str(dt.datetime.now())+','+str(millis_interval(start,end))+)
count = 0
for s in PIDs:
start = dt.datetime.now()
ser.write(bytes(("01" + s) + '\r\n', encoding = 'utf-8'))
ser.timeout = 1
''' if answer not finished but new information sent OBD will stop writing and listen to UART *'''
r = ser.read(999)
while ">" not in r
r = r + ser.read(999)
print(r) # debug
r.decode('utf-8')
response[count] = r
print(response[count]) #see what it looks like in utf-8
myfile.write(str(','+response[count]))
count +=1
ser.close()
If it is of any interest: I am using an ELM327 compatible device via USB which seems to work flawlesly with the Scantool Software, my car is built 2000 and the ECU is a VW one.
new information on the STOPPED info:
https://www.scantool.net/forum/index.php?topic=10164.0
The answer was the keyword protocol. So just in case anyone wonders about the same issues:
Before you waste 10 minutes of your life: No, I do not provide a code solution. I merely figured out, why it did not work. So here you will only find a description of the protocol. The interested reader is referred to the Springer Bosch Mechatronic Series by Konrad Reif.
Among the early protocols (1990s) there is the K-Line Protocol as described by ISO 9141 and ISO 14230. VW uses the K-Line which was implemented in cars around the millennium in two ways. The test vehicle was a 2000 built VAG Model, by the data given the car is expected to feature OBD II - yet no CAN - and ISO9141 with the key word protocol KWP1282 or ISO14230 KWP2000. After some more research it became clear that the car implements the ISO9140 KWP1281. One major problem with KWP1281 is the initialization as the K-Line and L-Line use 5 baud signals for wakeup. The wakeup request looks as follows:
The tester issues the initialization request at address 0x33 with 5 bits per second. Once the vehicle’s ECU has validated the address (after time W1) a confirmation is sent to the tester at 0x55, the so called synchronization byte. This synchronization byte tells the tester the baud rate at which communication shall take place, usually 10400 baud. The tester then reconfigures the baud rate, while the vehicle waits (W2). After time W2 has passed the vehicle sends two key bytes (either 08,08 or 94,94) to the tester with a delay of W3. These key bytes describe the collision prevention time P2MIN after which an ECU is checking the K-Line for a falling edge. [Reif, Automotive Mechatronics, BOSCH]
If the tester acknowledges P2Min the second key byte is inverted and returned to the vehicle. The vehicle then sends the complement of 0x33 to the tester as confirmation, signaling ready for interaction.
The ELM327 chips do not support KWP1281 at all, it does support KWP2000 but in this case it simply serves as a matter of transport for a few OBD2 parameters, the rest is vendor specific. As far as emission legislation is concerned for VW vehicles the ELM327 KWP2000 delivers emission specific fault codes via OBD2 but no more.
Due to the limited time at hand, I did not bother to implement a solution as there already is software to do it for VW cars and my thesis did not allow me to spend the time. I might, one day.
With the VW specific KKL adapter in combination with VW software VCDS the OBD of the test vehicle finally delivers sufficient results.
VCDS (VAG-COM) features slow, yet sufficient recording capabilities and includes the VAG-Scope software for result evaluation, the export format is CSV which is simple and easy to process with a majority of programs. The resolution of the measurement is approximately 3Hz, reading three measurement groups with four parameters each per second.
The interested reader is referred to the Springer Bosch Mechatronic Series by Konrad Reif for further information.
Sorry I could not provide an answer on how to implement it. But at least a guide on why it might not work for your (VW, Skoda, Audi, Seat, ...) car.
Related
I am new to programming and started with Python about 2 weeks ago using a course on FCC, I am currently in the networking chapter.
The exercise was about creating a program which counts the maximum number of characters in a document of a website and only display the first 3000 characters of that document using the socket library in Python. The next exercise was to do the same with the urllib library. I have noticed that, when using socket, I was sometimes missing some letters in the file when the bufsize parameter of the sock.recv(bufsize,[flag]) method wasn't set to the total length of received bytes from the document. For example when I used 1024 as the value for bufsize, there were some letters missing here and there from the retrieved document, but when I put the bufsize to 95000 (exact number of bytelength of that document), I got all the letters and everything worked fine.
Please don't be too harsh on me with the code, I am just starting to write something, but here is my example:
import socket
import re
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
while True:
userinp = input("Enter a URL: ")
try:
if userinp.startswith("http"):
url = userinp.split("/")[2]
#print(url)
sock.connect((url, 80))
#print("http start connected")
break
elif userinp.startswith("www"):
url = userinp.split("/")[0]
#print(url)
sock.connect((url, 80))
#print("www start connected")
break
else:
url = userinp.split("/")[0]
#print(url)
sock.connect((url, 80))
#print("else start connected")
break
except:
print("Please enter a valid URL")
continue
if userinp.startswith("http:"):
cmd0 = "GET " + userinp + " HTTP/1.0\r\n\r\n"
cmd = cmd0.encode()
#print("http bytes: ", cmd)
elif userinp.startswith("https:"):
cmd0 = "GET " + userinp + " HTTP/1.1\r\nHost: " + url + "\r\n\r\n"
cmd = cmd0.encode()
#print("https bytes: ", cmd)
else:
cmd0 = "GET http://" + userinp + " HTTP/1.0\r\n\r\n"
cmd = cmd0.encode()
sock.send(cmd)
#print("cmd request sent")
count = 0
str = ""
while True:
data = sock.recv(95000) ##536 magic number in romeo.txt, 95000 in mbox-short.txt
if len(data) < 1: ##http://data.pr4e.org/mbox-short.txt
break
#print("Byte length:", len(data))
data = data.decode()
pos = data.find("\r\n\r\n") + 4
for each in data[pos:]:
count += 1
if count <= 3000:
str += each
print(str, "Total characters:", count, len(str))
sock.shutdown(socket.SHUT_RDWR)
sock.close()
The first if statements are meant for the first exercise in the chapter, which was handling userinput URLs using the socket library. On many websites I have some problems with that too, since it often says
301: Moved Permanently
But the location specified in the document says it moved to the exact same location.\
So my questions are:
Why do I have to set the bufsize parameter to the exact bytelength of the retrieved document in order to get all letters out of it? Is there a way around this using the socket library?
Why do some websites specify that they are moved permanently, but show the exact same location of the website?
With the urllib library it is much easier, since it does "all the stuff" for me, but I would like to know how I need to write the program with the socket library too, just to get a better understanding of it.
I'm sorry for the noob questions, but I've read that beginner questions are welcome aswell! I hope you can help me with my problem, thank you in advance! :)
Actually, before HTTP/1.1, Content-Length header is SHOULD on RFC1945, of course this means the header was not required. How did application distinguish end of file, closing of TCP connection was regarded as it. Therefore, there are files of which we can't know
size before downloading even now. This story is about HTTP, layer 7, application layer in OSI model.
Sockets which you use belong TCP, layer 5 and 4. TCP doesn't have how to know size of files. It just manages connections and sends bytes only. It doesn't think any other thing. If TCPs work correctly each other, other layers are guaranteed to work. This is same as HTTP too.
How network works? is itself able theme to be written a thick book. If you are interested in, I recommend to read some books about network.
If anyone is interested in the answer to this question (probably not, because it is a complete beginner question):
I played around with the program a little and added print statements basically everywhere to be able to see what it is doing at what point exactly. The received data every time in the sock.recv is set to 512, so it sends 512 bytes worth of information per iteration of that loop. Then those 512 bytes of information will be used by the for loop after decoding it to a string, iterating through every character of the string but only up to the end of those 512 bytes of information, which in this case (romeo.txt) ends with the "s" in the last line of the poem. Then the sock.recv starts receiving the rest of the information of the document and the for loop starts iterating through the rest again, but this time because of the "data[pos:]" (initially used to remove the header) it starts at the beginning position of those remaining bytes + 4. So with this I would have 3 letters less for each iteration of the loops.
I'm having a problem with a block of Python code reading in a string from an Arduino connected over USB. I understand that serial doesn't know what a string is or care. I'm using serial.readline, which from the documentation sounds like the perfect match, but my string isn't always complete. The weird problem is, the string doesn't always have the front of the string, but it always has the end of the string. I'm really lost on this and I'm sure it's just my lack of understanding about the nuances of reading serial data or how Python handles it.
In the code below, I loop through the serial interfaces until I find the one I'm looking for. I flush the input and give it a sleep for a couple seconds to make sure it has time to get a new read.
arduinoTemp = serial.Serial(iface, 9600, timeout=1)
arduinoTemp.flushInput()
arduinoTemp.flushOutput()
arduinoTemp.write("status\r\n".encode())
time.sleep(2)
read = arduinoTemp.readline().strip()
if read != "":
#check the string to make sure it's what I'm expecting.
I'm sending the string in JSON.
I'm expecting something in line with this:
{"id": "env monitor","distance": {"forward": {"num":"0","unit": "inches"}},"humidity": {"num":"0.00","unit": "%"},"temp": {"num":"0.00","unit": "fahrenheit"},"heatIndex": {"num":"0.00","unit": "fahrenheit"}}
I might get something back like this:
": t": "%"},"temp": {"num":"69.80","unit": "fahrenheit"},"heatIndex": {"num":"68.13","unit": "fahrenheit"}}
or this:
atIndex": {"num":"0.00","unit": "fahrenheit"}}
At first I thought it was the length of the string that might be causing some issues, but the cut off isn't always consistent, and since it has the end of the string, it stands to reason that it should have gotten everything before that.
I've verified that my Arduino is broadcasting correctly by interfacing with it directly and the Arduino IDE and serial monitor. This is definitely an issue with my Python code.
In (serial) communications you should always expect to receive partial answers.
A usual solution in this case is to add whatever you read from the serial to a string/buffer until you can parse it successfully with json.loads.
import serial
import json
import time
ser = serial.Serial('/dev/ttyACM0', 9600)
buffer = ''
while True:
buffer += ser.read()
try:
data = json.loads(buffer)
print(data)
buffer = ''
except json.JSONDecodeError:
time.sleep(1)
(From this answer).
Note that if you flush, you will lose data!
Also note that this is a somewhat simplified solution. Ideally the buffer should be reset to whatever remains after the successful parse. But as far as I know, the json module doesn't offer that functionality.
UPDATE: I've put together the following script to use the url for the XML without the time-code-like suffix as recommended in the answer below, and report the downlink powers which clearly fluctuate on the website. I'm getting three hour old, unvarying data.
So it looks like I need to properly construct that (time code? authorization? secret password?) in order to do this successfully. Like I say in the comment below, "I don't want to do anything that's not allowed and welcome - NASA has enough challenges already trying to talk to a forty year old spacecraft 20 billion kilometers away!"
def dictify(r,root=True):
"""from: https://stackoverflow.com/a/30923963/3904031"""
if root:
return {r.tag : dictify(r, False)}
d=copy(r.attrib)
if r.text:
d["_text"]=r.text
for x in r.findall("./*"):
if x.tag not in d:
d[x.tag]=[]
d[x.tag].append(dictify(x,False))
return d
import xml.etree.ElementTree as ET
from copy import copy
import urllib2
url = 'https://eyes.nasa.gov/dsn/data/dsn.xml'
contents = urllib2.urlopen(url).read()
root = ET.fromstring(contents)
DSNdict = dictify(root)
dishes = DSNdict['dsn']['dish']
dp_dict = dict()
for dish in dishes:
powers = [float(sig['power']) for sig in dish['downSignal'] if sig['power']]
dp_dict[dish['name']] = powers
print dp_dict['DSS26']
I'd like to keep track of which spacecraft that the NASA Deep Space Network (DSN) is communicating with, say once per minute.
I learned how to do something similar from Flight Radar 24 from the answer to my previous question, which also still represents my current skills in getting data from web sites.
With FR24 I had explanations in this blog as a great place to start. I have opened the page with the Developer Tools function in the Chrome browser, and I can see that data for items such as dishes, spacecraft and associated numerical data are requested as an XML with urls such as
https://eyes.nasa.gov/dsn/data/dsn.xml?r=293849023
so it looks like I need to construct the integer (time code? authorization? secret password?) after the r= once a minute.
My Question: Using python, how could I best find out what that integer represents, and how to generate it in order to correctly request data once per minute?
above: screen shot montage from NASA's DSN Now page https://eyes.nasa.gov/dsn/dsn.html see also this question
Using a random number (or a timestamp...) in a get parameter tricks the browser into really making the request (instead of using the browser cache).
This method is some kind of "hack" the webdevs use so that they are sure the request actually happens.
Since you aren't using a web browser, I'm pretty sure you could totally ignore this parameter, and still get the refreshed data.
--- Edit ---
Actually r seems to be required, and has to be updated.
#!/bin/bash
wget https://eyes.nasa.gov/dsn/data/dsn.xml?r=$(date +%s) -O a.xml -nv
while true; do
sleep 1
wget https://eyes.nasa.gov/dsn/data/dsn.xml?r=$(date +%s) -O b.xml -nv
diff a.xml b.xml
cp b.xml a.xml -f
done
You don't need to emulate a browser. Simply set r to anything and increment it. (Or use a timestamp)
Regarding your updated question, why avoid sending the r query string parameter when it is very easy to generate it? Also, with the requests module, it's easy to send the parameter with the request too:
import time
import requests
import xml.etree.ElementTree as ET
url = 'https://eyes.nasa.gov/dsn/data/dsn.xml'
r = int(time.time() / 5)
response = requests.get(url, params={'r': r})
root = ET.fromstring(response.content)
# etc....
i am new to python. I have some experience with Pascal and a little bit with C++.
At the moment i have to program some code for a research project demonstrator.
The setup is as follows:
We have a 868MHz radio master device. i can communicate with this device via a USB port (COM4 at the moment but may change in the future).
The 868MHz master communicates with a slave unit. The slave unit replies with a message that i can read from the USB port.
Until this point everything works good. I request data packages and also receive them.
From the moment of receiving the data packages i have a propblem i seem not
able to solve on myself.
I use Anaconda 32 bit with the Spyder editor
# -*- coding: utf-8 -*-
"""
Created on Thu May 7 13:35:59 2015
#author: roland
"""
import serial
portnr = 3 #Serial Port Number 0=Com1, 3=Com4
portbaud = 38400 #Baud rate
tiout = 0.1 #Timout in seconds
i = 1
wrword = ([0x02,0x04,0x00,0x00,0x00,0x02,0x71,0xF8])
try:
ser = serial.Serial(portnr, portbaud, timeout=tiout) # open port
except:
ser.close() # close port
ser = serial.Serial(portnr, portbaud, timeout=tiout) # open port
print(ser.name) # check which port was really used
while (i < 100):
ser.write(wrword)
seread = ser.readline()
print(seread)
i = i+1
sere = seread.split()
try:
readdat = str(sere[0])
except:
print("Index Error")
retlen = len(readdat)
print(retlen)
readdat = readdat[2:retlen-1]
print(readdat)
ser.close() # close port
The variable wrword is my request to the 868MHz radio master.
The Format is as follows:
0x02 Address of the unit
0x04 Command to send information from a certain register range
0x00 0x00 Address of first Register (Start address 0 is valid!)
0x00 0x02 Information how much registers are to be sent (in this case Registers 0 and 1 shall be transmitted to the Radio master)
0x71 0xF8 Checksum of the command sentence.
The program sends the command sequence successful to the master unit and the slave unit answers. Each time the command is send an answer is expected. Nevertheless it may happen that now correct answer is given thats why the
try command is in use.
I know i use ser.readline() but this is sufficient for the application.
I receive a list as answer from the USB Port.
The data look as follows:
b'\x02\x04\x04\x12\xb6\x12\xa5\xe0\xc1' (This is the Output from print(seread) )
For clarification this answer is correct and must be read as follows:
\x02 Address of the answering unit
\x04 Function that was executed (Read from certain register area)
\x04 Number of Bytes of the answer
\x12 \xb6 Value of first register (2 Byte)
\x12 \xa5 Value of second register (2 Byte)
\xe0 \xc1 Checksum of answer
If the data from the com port had all this Format i might be able to get the data values from the both Registers. But unfortunately the data format is not always the same.
Sometimes i receive answers in the following style:
b'\x02\x04\x04\x12\x8e\x12{\xe1T'
The answer is similar to the example above (different values in the Registers and different checksum) but the Format i receive has changed.
If i use the hex ASCII codes for the symbols obviously not hex values i find a valid answer telegram.
b'\x02\x04\x04\x12\x8e\x12{\xe1T'
becomes
b'\x02\x04\x04\x12\x8e\x12\x7b\xe1\x54'
when i Exchange the ASCII symbols by their hex code by Hand.
So my questions are:
Is it possible to force Python to give me the answer in a defined Format?
If not is it possible to handle the list or the string i can derive from the list in such a way that i get my values in the required format?
Does somebody can give me a hint how to extract my register values from the list and convert the two hex numbers of each register into one integer value for each register (the first value is the high byte, the second the low byte)?
Thanks in advance for your answer(s)
sincerely
Roland
I found a solution.
During a small testpiece of program i stumbled upon the fact that the variable seread contains already the data in a suitable and usable format for me.
I assume that the Spyder Editor causes the format change when displaying byte type objects.
If i Access the single Bytes using seread[i] while i is in range 0 to len(seread)-1 i receive the correct values for the single bytes.
So i can acess my data and calculate my measurement values as required.
Nevertheless thanks to keety for reading my question.
I found this code to detect the length of encrypted data in the frame :
header = self.request.recv(5)
if header == '':
#print 'client disconnected'
running = False
break
(content_type, version, length) = struct.unpack('>BHH', header)
data = self.request.recv(length)
Souce :
https://github.com/EiNSTeiN-/poodle/blob/master/samples/poodle-sample-1.py
https://gist.github.com/takeshixx/10107280
https://gist.github.com/ixs/10116537
This code, listen the connection between a client and a server. When the client talk to the server, self.request.recv(5) can get you the length of the header in the frame. Then we use that length to take the data.
If we print the exchange between the client and the server :
Client --> [proxy] -----> Server
length : 24 #why 24 ?
Client --> [proxy] -----> Server
length: 80 #length of the data
Client <-- [proxy] <----- Server
We can see that the client will send two packet to the server.
If i change
data = self.request.recv(length)
to
data = self.request.recv(4096)
Only one exchange is made.
Client --> [proxy] -----> Server
length: 109 #length of the data + the header
Client <-- [proxy] <----- Server
My question is why we only need to take a size of 5 to get the lenght, content_type informations ? Is there an understandable doc about this ?
Why there is two request: one with 24 and another with the lenght of our data ?
why we only need to take a size of 5 to get the lenght, content_type
informations ?
Because obviously that's the way the protocol was designed.
Binary streams only guarantee that when some bytes are put into one end of the stream, they arrive in the same order on the other end of the stream. For message transmission through binary streams the obvious problem is: where are the message boundaries? The classical solution to this problem is to add a prefix to messages, a so-called header. This header has a fixed size, known to both communication partners. That way, the recipient can safely read header, message, header, message (I guess you grasp the concept, it is an alternating fashion). As you see, the header does not contain message data -- it is just communication "overhead". This overhead should be kept small. The most efficient (space-wise) way to store such information is in binary form, using some kind of code that must, again, be known to both sides of the communication. Indeed, 5 bytes of information is quite a lot.
The '>BHH' format string indicates that this 5 byte header is built up like this:
unsigned char (1 Byte)
unsigned short (2 Bytes)
unsigned short (2 Bytes)
Plenty of room for storing information such as length and content type, don't you think? This header can encode 256 different content types, 65536 different versions, and a message length between 0 and 65535 bytes.
Why there is two request: one with 24 and another with the lenght of
our data ?
If your network forensics / traffic analysis does not correspond to what you have inferred from code, one of both types of analyses is wrong/incomplete. In this case, I guess that your traffic analysis is correct, but that you have not understood all relevant code for this kind of communication. Note that I did not look at the source code you linked to.