i am new to python. I have some experience with Pascal and a little bit with C++.
At the moment i have to program some code for a research project demonstrator.
The setup is as follows:
We have a 868MHz radio master device. i can communicate with this device via a USB port (COM4 at the moment but may change in the future).
The 868MHz master communicates with a slave unit. The slave unit replies with a message that i can read from the USB port.
Until this point everything works good. I request data packages and also receive them.
From the moment of receiving the data packages i have a propblem i seem not
able to solve on myself.
I use Anaconda 32 bit with the Spyder editor
# -*- coding: utf-8 -*-
"""
Created on Thu May 7 13:35:59 2015
#author: roland
"""
import serial
portnr = 3 #Serial Port Number 0=Com1, 3=Com4
portbaud = 38400 #Baud rate
tiout = 0.1 #Timout in seconds
i = 1
wrword = ([0x02,0x04,0x00,0x00,0x00,0x02,0x71,0xF8])
try:
ser = serial.Serial(portnr, portbaud, timeout=tiout) # open port
except:
ser.close() # close port
ser = serial.Serial(portnr, portbaud, timeout=tiout) # open port
print(ser.name) # check which port was really used
while (i < 100):
ser.write(wrword)
seread = ser.readline()
print(seread)
i = i+1
sere = seread.split()
try:
readdat = str(sere[0])
except:
print("Index Error")
retlen = len(readdat)
print(retlen)
readdat = readdat[2:retlen-1]
print(readdat)
ser.close() # close port
The variable wrword is my request to the 868MHz radio master.
The Format is as follows:
0x02 Address of the unit
0x04 Command to send information from a certain register range
0x00 0x00 Address of first Register (Start address 0 is valid!)
0x00 0x02 Information how much registers are to be sent (in this case Registers 0 and 1 shall be transmitted to the Radio master)
0x71 0xF8 Checksum of the command sentence.
The program sends the command sequence successful to the master unit and the slave unit answers. Each time the command is send an answer is expected. Nevertheless it may happen that now correct answer is given thats why the
try command is in use.
I know i use ser.readline() but this is sufficient for the application.
I receive a list as answer from the USB Port.
The data look as follows:
b'\x02\x04\x04\x12\xb6\x12\xa5\xe0\xc1' (This is the Output from print(seread) )
For clarification this answer is correct and must be read as follows:
\x02 Address of the answering unit
\x04 Function that was executed (Read from certain register area)
\x04 Number of Bytes of the answer
\x12 \xb6 Value of first register (2 Byte)
\x12 \xa5 Value of second register (2 Byte)
\xe0 \xc1 Checksum of answer
If the data from the com port had all this Format i might be able to get the data values from the both Registers. But unfortunately the data format is not always the same.
Sometimes i receive answers in the following style:
b'\x02\x04\x04\x12\x8e\x12{\xe1T'
The answer is similar to the example above (different values in the Registers and different checksum) but the Format i receive has changed.
If i use the hex ASCII codes for the symbols obviously not hex values i find a valid answer telegram.
b'\x02\x04\x04\x12\x8e\x12{\xe1T'
becomes
b'\x02\x04\x04\x12\x8e\x12\x7b\xe1\x54'
when i Exchange the ASCII symbols by their hex code by Hand.
So my questions are:
Is it possible to force Python to give me the answer in a defined Format?
If not is it possible to handle the list or the string i can derive from the list in such a way that i get my values in the required format?
Does somebody can give me a hint how to extract my register values from the list and convert the two hex numbers of each register into one integer value for each register (the first value is the high byte, the second the low byte)?
Thanks in advance for your answer(s)
sincerely
Roland
I found a solution.
During a small testpiece of program i stumbled upon the fact that the variable seread contains already the data in a suitable and usable format for me.
I assume that the Spyder Editor causes the format change when displaying byte type objects.
If i Access the single Bytes using seread[i] while i is in range 0 to len(seread)-1 i receive the correct values for the single bytes.
So i can acess my data and calculate my measurement values as required.
Nevertheless thanks to keety for reading my question.
Related
I'm tring to understand the meaning of the python socket address info output.
import socket
rawSocket = socket.socket(socket.PF_PACKET, socket.SOCK_RAW, socket.htons(0x0800))
pkt = rawSocket.recvfrom(2048)
print pkt[1]
('ens33', 2048, 1, 1, 'HE \xfd\x12h')
ens33 is the interface sending the data.
I guess that 2048 is the buffer size.
I have no idea what the first "1" is. Sometimes it's "0".
I noticed the second "1" relates to the interface (i.e. "772" for "lo")
'HE \xfd\x12h' : Reverting the converted hex values, we get '\x48\x45\x20\xfd\x12\x68', it gives the mac address of host machine in a VM bridged connection.
So, the main question is for #3. What 1 or 0 means here ?
In short, the third 1 means it's a broadcast packet. 0 would mean it was a packet addressed to the machine running the Python code. Details follow.
This is based on Python 3.6, but the answer should be similar for other Py2 or Py3 versions. The answer is split between the source for the socket module, the packet(7) man page, and the Linux source.
The Python library includes function makesockaddr(). For PF_PACKET sockets (same as AF_PACKET), the relevant portion gives you the following order of fields. Explanations from the man page are italicized.
ifname (e.g., ens33) (the interface name, as you noted)
sll_protocol (e.g., 2048) Physical-layer protocol
sll_pkttype (e.g., 1) Packet type
sll_hatype (e.g., 1) ARP hardware type
sll_addr (e.g., the MAC address, as you noted above) Physical-layer address
The Linux source gives the various values for packet type. In that list, 1 is PACKET_BROADCAST. 0 is PACKET_HOST, explained as "To us".
I'm coding to tcp client/server using python twisted
in order to replace for Java or C#.
I have to parse length prefixed string messages based on ANS(alpha numeric string) in connected permanent session.
like this :
message format : [alpha numeric string:4byte][message data]
example-1 : 0004ABCD ==> ABCD
example-2 : 0002AB0005HELLO ==> AB, HELLO
it can't be solved by IntNProtocol, NetStringProtocol.
And if a client send a 2kb message from application layer, the kernel split message data by MSS(maximum segment size) and send packets are splitted.
in TCP send/receive environment, it often raise like this :
example : 1000HELLO {not yet arrived 995 byte data}
so it has to wait for receiving spare data using array, queue...
in the twisted, I don't know how to parse multiple large-message.
Anybody help me to give some information or URL?
class ClientProtocol(protocol.Protocol):
def dataReceived(self, data):
# how can I code to parse multiple large message?
# is there solution to read specific size for data ?
It looks like you can implement this protocol using StatefulProtocol as a base. Your protocol basically has two states. In the first state, you're waiting for 4 bytes which you will interpret as a zero-padded base 10 integer. In the second state, you're waiting for a number of bytes equal to the integer read in the first state.
from twisted.protocols.stateful import StatefulProtocol
class ANSProtocol(StatefulProtocol):
def getInitialState(self):
return (self._state_length, 4)
def _state_length(self, length_bytes):
length = int(length_bytes)
return self._state_content, length
def _state_content(self, content):
self.application_logic(content)
return self.getInitialState()
def application_logic(self, content):
# Application logic operating on `content`
# ...
I'm having a problem with a block of Python code reading in a string from an Arduino connected over USB. I understand that serial doesn't know what a string is or care. I'm using serial.readline, which from the documentation sounds like the perfect match, but my string isn't always complete. The weird problem is, the string doesn't always have the front of the string, but it always has the end of the string. I'm really lost on this and I'm sure it's just my lack of understanding about the nuances of reading serial data or how Python handles it.
In the code below, I loop through the serial interfaces until I find the one I'm looking for. I flush the input and give it a sleep for a couple seconds to make sure it has time to get a new read.
arduinoTemp = serial.Serial(iface, 9600, timeout=1)
arduinoTemp.flushInput()
arduinoTemp.flushOutput()
arduinoTemp.write("status\r\n".encode())
time.sleep(2)
read = arduinoTemp.readline().strip()
if read != "":
#check the string to make sure it's what I'm expecting.
I'm sending the string in JSON.
I'm expecting something in line with this:
{"id": "env monitor","distance": {"forward": {"num":"0","unit": "inches"}},"humidity": {"num":"0.00","unit": "%"},"temp": {"num":"0.00","unit": "fahrenheit"},"heatIndex": {"num":"0.00","unit": "fahrenheit"}}
I might get something back like this:
": t": "%"},"temp": {"num":"69.80","unit": "fahrenheit"},"heatIndex": {"num":"68.13","unit": "fahrenheit"}}
or this:
atIndex": {"num":"0.00","unit": "fahrenheit"}}
At first I thought it was the length of the string that might be causing some issues, but the cut off isn't always consistent, and since it has the end of the string, it stands to reason that it should have gotten everything before that.
I've verified that my Arduino is broadcasting correctly by interfacing with it directly and the Arduino IDE and serial monitor. This is definitely an issue with my Python code.
In (serial) communications you should always expect to receive partial answers.
A usual solution in this case is to add whatever you read from the serial to a string/buffer until you can parse it successfully with json.loads.
import serial
import json
import time
ser = serial.Serial('/dev/ttyACM0', 9600)
buffer = ''
while True:
buffer += ser.read()
try:
data = json.loads(buffer)
print(data)
buffer = ''
except json.JSONDecodeError:
time.sleep(1)
(From this answer).
Note that if you flush, you will lose data!
Also note that this is a somewhat simplified solution. Ideally the buffer should be reset to whatever remains after the successful parse. But as far as I know, the json module doesn't offer that functionality.
I wanted to fetch some OBD data and tried it like described by this article
The article (using python 2.7) said:
The elm327 device returns values in HEX.
To read the value you just requested in Python type speed_hex = ser.readline().split(' ')
Convert the HEX to decimal by using: speed = float(int('0x'+speed_hex[3], 0 ))
But the answers I get from my OBD do not contain any whitespace furthermore they didn't look like they made any sense at all.
['\xd0MA\r?\r\r>\xd0\x15\r?\r\r>\x981\xf0\n']
so I gave up on this approach and oved to Python 3 instead.
I then wrote a little script inspired by this SO post
However I changed it to pull a lot of data periodically and added a timestamp, all together is saved to a csv file - as the path implies I am working on Windows (10 to be precise), COM is correctly configured and the car is OBD compliant.
Now when I run my script it prints out what it read:
b'h\xf4\rSTOPPED\r\r>'
and then tells me
Traceback (most recent call last):
File "python3test.py", line 36, in <module> r.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf4 in position 1:
invalid continuation byte
I do understand that encoding certainly is an issue, but this string does not make much sense at all. even if I take a closer look at 0xf4 - it would only be 244 (dec) which is no where near sensible RPM data.
Why does it work in the other script then? It can't be a Windows issue, can it?
Can anyone please help me understand whats going on in here?
Here is my code.
import serial
import time
import string
import io
import os
import sys
import datetime as dt
ser = serial.Serial("COM1")
ser.baudrate = 38400
ser.write(bytes('ATSP0\r\n', encoding = 'utf-8'))
ser.timeout = 1
ser.write(bytes('ATE0\r\n', encoding = 'utf-8'))
ser.timeout = 1
def millis_interval(start, end):
'''start and end are datetime instances'''
diff = end - start
millis = diff.days * 24 * 60 * 60 * 1000
millis += diff.seconds * 1000
millis += diff.microseconds / 1000
return millis
Testtime = str(dt.datetime.now()).split(' ')
PIDs = ["0C","0D","43","04","11","5C","05","10","46"]
response = []
''' open file and create header row'''
with open("C:/test/test_"+Testtime[0]+"_"+Testtime[1].replace(":","-")+".csv", "a") as myfile:
myfile.write("Time,Timestep,RPM,Speed, AbsLoad, CalcLoad, ThrottlePos, CoolTemp, OilTemp, MAF, AmbAirTemp")
start = dt.datetime.now() # need initial value
while 1:
end = dt.datetime.now()
myfile.write("\n"+str(dt.datetime.now())+','+str(millis_interval(start,end))+)
count = 0
for s in PIDs:
start = dt.datetime.now()
ser.write(bytes(("01" + s) + '\r\n', encoding = 'utf-8'))
ser.timeout = 1
''' if answer not finished but new information sent OBD will stop writing and listen to UART *'''
r = ser.read(999)
while ">" not in r
r = r + ser.read(999)
print(r) # debug
r.decode('utf-8')
response[count] = r
print(response[count]) #see what it looks like in utf-8
myfile.write(str(','+response[count]))
count +=1
ser.close()
If it is of any interest: I am using an ELM327 compatible device via USB which seems to work flawlesly with the Scantool Software, my car is built 2000 and the ECU is a VW one.
new information on the STOPPED info:
https://www.scantool.net/forum/index.php?topic=10164.0
The answer was the keyword protocol. So just in case anyone wonders about the same issues:
Before you waste 10 minutes of your life: No, I do not provide a code solution. I merely figured out, why it did not work. So here you will only find a description of the protocol. The interested reader is referred to the Springer Bosch Mechatronic Series by Konrad Reif.
Among the early protocols (1990s) there is the K-Line Protocol as described by ISO 9141 and ISO 14230. VW uses the K-Line which was implemented in cars around the millennium in two ways. The test vehicle was a 2000 built VAG Model, by the data given the car is expected to feature OBD II - yet no CAN - and ISO9141 with the key word protocol KWP1282 or ISO14230 KWP2000. After some more research it became clear that the car implements the ISO9140 KWP1281. One major problem with KWP1281 is the initialization as the K-Line and L-Line use 5 baud signals for wakeup. The wakeup request looks as follows:
The tester issues the initialization request at address 0x33 with 5 bits per second. Once the vehicle’s ECU has validated the address (after time W1) a confirmation is sent to the tester at 0x55, the so called synchronization byte. This synchronization byte tells the tester the baud rate at which communication shall take place, usually 10400 baud. The tester then reconfigures the baud rate, while the vehicle waits (W2). After time W2 has passed the vehicle sends two key bytes (either 08,08 or 94,94) to the tester with a delay of W3. These key bytes describe the collision prevention time P2MIN after which an ECU is checking the K-Line for a falling edge. [Reif, Automotive Mechatronics, BOSCH]
If the tester acknowledges P2Min the second key byte is inverted and returned to the vehicle. The vehicle then sends the complement of 0x33 to the tester as confirmation, signaling ready for interaction.
The ELM327 chips do not support KWP1281 at all, it does support KWP2000 but in this case it simply serves as a matter of transport for a few OBD2 parameters, the rest is vendor specific. As far as emission legislation is concerned for VW vehicles the ELM327 KWP2000 delivers emission specific fault codes via OBD2 but no more.
Due to the limited time at hand, I did not bother to implement a solution as there already is software to do it for VW cars and my thesis did not allow me to spend the time. I might, one day.
With the VW specific KKL adapter in combination with VW software VCDS the OBD of the test vehicle finally delivers sufficient results.
VCDS (VAG-COM) features slow, yet sufficient recording capabilities and includes the VAG-Scope software for result evaluation, the export format is CSV which is simple and easy to process with a majority of programs. The resolution of the measurement is approximately 3Hz, reading three measurement groups with four parameters each per second.
The interested reader is referred to the Springer Bosch Mechatronic Series by Konrad Reif for further information.
Sorry I could not provide an answer on how to implement it. But at least a guide on why it might not work for your (VW, Skoda, Audi, Seat, ...) car.
I found this code to detect the length of encrypted data in the frame :
header = self.request.recv(5)
if header == '':
#print 'client disconnected'
running = False
break
(content_type, version, length) = struct.unpack('>BHH', header)
data = self.request.recv(length)
Souce :
https://github.com/EiNSTeiN-/poodle/blob/master/samples/poodle-sample-1.py
https://gist.github.com/takeshixx/10107280
https://gist.github.com/ixs/10116537
This code, listen the connection between a client and a server. When the client talk to the server, self.request.recv(5) can get you the length of the header in the frame. Then we use that length to take the data.
If we print the exchange between the client and the server :
Client --> [proxy] -----> Server
length : 24 #why 24 ?
Client --> [proxy] -----> Server
length: 80 #length of the data
Client <-- [proxy] <----- Server
We can see that the client will send two packet to the server.
If i change
data = self.request.recv(length)
to
data = self.request.recv(4096)
Only one exchange is made.
Client --> [proxy] -----> Server
length: 109 #length of the data + the header
Client <-- [proxy] <----- Server
My question is why we only need to take a size of 5 to get the lenght, content_type informations ? Is there an understandable doc about this ?
Why there is two request: one with 24 and another with the lenght of our data ?
why we only need to take a size of 5 to get the lenght, content_type
informations ?
Because obviously that's the way the protocol was designed.
Binary streams only guarantee that when some bytes are put into one end of the stream, they arrive in the same order on the other end of the stream. For message transmission through binary streams the obvious problem is: where are the message boundaries? The classical solution to this problem is to add a prefix to messages, a so-called header. This header has a fixed size, known to both communication partners. That way, the recipient can safely read header, message, header, message (I guess you grasp the concept, it is an alternating fashion). As you see, the header does not contain message data -- it is just communication "overhead". This overhead should be kept small. The most efficient (space-wise) way to store such information is in binary form, using some kind of code that must, again, be known to both sides of the communication. Indeed, 5 bytes of information is quite a lot.
The '>BHH' format string indicates that this 5 byte header is built up like this:
unsigned char (1 Byte)
unsigned short (2 Bytes)
unsigned short (2 Bytes)
Plenty of room for storing information such as length and content type, don't you think? This header can encode 256 different content types, 65536 different versions, and a message length between 0 and 65535 bytes.
Why there is two request: one with 24 and another with the lenght of
our data ?
If your network forensics / traffic analysis does not correspond to what you have inferred from code, one of both types of analyses is wrong/incomplete. In this case, I guess that your traffic analysis is correct, but that you have not understood all relevant code for this kind of communication. Note that I did not look at the source code you linked to.