I'm trying to send floating point data from arduino to python.The data is sent as 8 successive bytes of data (size of double) followed by newline character ('\n').How to collect these successive bytes and convert it to proper format at python end (system end)
void USART_transmitdouble(double* d)
{
union Sharedblock
{
char part[sizeof(double)];
double data;
}my_block;
my_block.data = *d;
for(int i=0;i<sizeof(double);++i)
{
USART_send(my_block.part[i]);
}
USART_send('\n');
}
int main()
{
USART_init();
double dble=5.5;
while(1)
{
USART_transmitdouble(&dble);
}
return 0;
}
python code.Sure this wouldn't print the data in proper format but just want to show what i have tried.
import serial,time
my_port = serial.Serial('/dev/tty.usbmodemfa131',19200)
while 1:
print my_port.readline(),
time.sleep(0.15)
Update:
my_ser = serial.Serial('/dev/tty.usbmodemfa131',19200)
while 1:
#a = raw_input('enter a value:')
#my_ser.write(a)
data = my_ser.read(5)
f_data, = struct.unpack('<fx',data)
print f_data
#time.sleep(0.5)
Using struct module as shown in the above code is able to print float values. But,
50% of the time,the data is printed correctly.But if I mess with time.sleep() or stop the transmission and restart it,incorrect values are printed out.I guess the wrong set of 4 bytes are being unpacked in this case.Any idea on what we can do here??
On Arduino, a double is the same as float, i.e. a little-endian single-precision floating-point number that occupies 4 bytes of memory. This means that you should read exactly 5 bytes, use the little-endian variant of the f format to unpack it, and ignore the trailing newline with x:
import struct
...
data = my_port.read(5)
num, = struct.unpack('<fx', data)
Note that you don't want to use readline because any byte of the representation of the floating-point number can be '\n'.
As Nikklas B. pointed out, you don't even need to bother with the newline at all, just send the 4 bytes and read as many from Python. In that case the format string will be '<f'.
Related
It seems that the output of zlib.compress uses all possible byte values. Is this possible to use 255 of 256 byte values (for example avoid using \n)?
Note that I just use the python manual as a reference, but the question is not specific to python (i.e. any other languages that has a zlib library).
No, this is not possible. Apart from the compressed data itself, there is standardized control structures which contain integers. Those integers may accidentially lead to any 8-bit character ending up in the bytestream.
Your only chance would be to encode the zlib bytestream into another format, e.g. base64.
The whole point of compression is to reduce the size as much as possible. If zlib or any compressor only used 255 of the 256 byte values, the size of the output would be increased by at least 0.07%.
That may be perfectly fine for you, so you can simply post-process the compressed output, or any data at all, to remove one particular byte value at the expense of some expansion. The simplest approach would be to replace that byte when it occurs with a two-byte escape sequence. You would also then need to replace the escape prefix with a different two-byte escape sequence. That would expand the data on average by 0.8%. That is exactly what Hans provided in another answer here.
If that cost is too high, you can do something more sophisticated, which is to decode a fixed Huffman code that encodes 255 symbols of equal probability. To decode you then encode that Huffman code. The input is a sequence of bits, not bytes, and most of the time you will need to pad the input with some zero bits to encode the last symbol. The Huffman code turns one symbol into seven bits and the other 254 symbols into eight bits. So going the other way, it will expand the input by a little less than 0.1%. For short messages it will be a little more, since often less than seven bits at the very end will be encoded into a symbol.
Implementation in C:
// Placed in the public domain by Mark Adler, 26 June 2020.
// Encode an arbitrary stream of bytes into a stream of symbols limited to 255
// values. In particular, avoid the \n (10) byte value. With -d, decode back to
// the original byte stream. Take input from stdin, and write output to stdout.
#include <stdio.h>
#include <string.h>
// Encode arbitrary bytes to a sequence of 255 symbols, which are written out
// as bytes that exclude the value '\n' (10). This encoding is actually a
// decoding of a fixed Huffman code of 255 symbols of equal probability. The
// output will be on average a little less than 0.1% larger than the input,
// plus one byte, assuming random input. This is intended to be used on
// compressed data, which will appear random. An input of all zero bits will
// have the maximum possible expansion, which is 14.3%, plus one byte.
int nolf_encode(FILE *in, FILE *out) {
unsigned buf = 0;
int bits = 0, ch;
do {
if (bits < 8) {
ch = getc(in);
if (ch != EOF) {
buf |= (unsigned)ch << bits;
bits += 8;
}
else if (bits == 0)
break;
}
if ((buf & 0x7f) == 0) {
buf >>= 7;
bits -= 7;
putc(0, out);
continue;
}
int sym = buf & 0xff;
buf >>= 8;
bits -= 8;
if (sym >= '\n' && sym < 128)
sym++;
putc(sym, out);
} while (ch != EOF);
return 0;
}
// Decode a sequence of symbols from a set of 255 that was encoded by
// nolf_encode(). The input is read as bytes that exclude the value '\n' (10).
// Any such values in the input are ignored and flagged in an error message.
// The sequence is decoded to the original sequence of arbitrary bytes. The
// decoding is actually an encoding of a fixed Huffman code of 255 symbols of
// equal probability.
int nolf_decode(FILE *in, FILE *out) {
unsigned long lfs = 0;
unsigned buf = 0;
int bits = 0, ch;
while ((ch = getc(in)) != EOF) {
if (ch == '\n') {
lfs++;
continue;
}
if (ch == 0) {
if (bits == 0) {
bits = 7;
continue;
}
bits--;
}
else {
if (ch > '\n' && ch <= 128)
ch--;
buf |= (unsigned)ch << bits;
}
putc(buf, out);
buf >>= 8;
}
if (lfs)
fprintf(stderr, "nolf: %lu unexpected line feeds ignored\n", lfs);
return lfs != 0;
}
// Encode (no arguments) or decode (-d) from stdin to stdout.
int main(int argc, char **argv) {
if (argc == 1)
return nolf_encode(stdin, stdout);
else if (argc == 2 && strcmp(argv[1], "-d") == 0)
return nolf_decode(stdin, stdout);
fputs("nolf: unknown options (use -d to decode)\n", stderr);
return 1;
}
As #ypnos says, this isn't possible within zlib itself. You mentioned that base64 encoding is too inefficient, but it's pretty easy to use an escape character to encode a character you want to avoid (like newlines).
This isn't the most efficient code in the world (and you might want to do something like finding the least used bytes to save a tiny bit more space), but it's readable enough and demonstrates the idea. You can losslessly encode/decode, and the encoded stream won't have any newlines.
def encode(data):
# order matters
return data.replace(b'a', b'aa').replace(b'\n', b'ab')
def decode(data):
def _foo():
pair = False
for b in data:
if pair:
# yield b'a' if b==b'a' else b'\n'
yield 97 if b==97 else 10
pair = False
elif b==97: # b'a'
pair = True
else:
yield b
return bytes(_foo())
As some measure of confidence you can check this exhaustively on small bytestrings:
from itertools import *
all(
bytes(p) == decode(encode(bytes(p)))
for c in combinations_with_replacement(b'ab\nc', r=6)
for p in permutations(c)
)
I am receiving data via socket interface from an application (server) written in C. The data being posted has the following structure. I am receiving data with a client written in Python.
struct hdr
{
int Id;
char PktType;
int SeqNo;
int Pktlength;
};
struct trl
{
char Message[16];
long long info;
};
struct data
{
char value[10];
double result;
long long count;
short int valueid;
};
typedef struct
{
struct hdr hdr_buf;
struct data data_buf[100];
struct trl trl_buf;
} trx_unit;
How do I unpack the received data to access my inner data buffer?
Using the struct library is the way to go. However, you will have to know a bit more about the C program that is serializing the data. Consider the hdr structure. If the C program is sending it using the naive approach:
struct hdr header;
send(sd, &hdr, sizeof(header), 0);
Then your client cannot safely interpret the bytes that are sent to it because there is an indeterminate amount of padding inserted between the struct members. In particular, I would expect three bytes of padding following the PktType member.
The safest way to approach sending around binary data is to have the server and client serialize the bytes directly to ensure that there is no additional padding and to make the byte ordering of multibyte integers explicit. For example:
/*
* Send a header over a socket.
*
* The header is sent as a stream of packed bytes with
* integers in "network" byte order. For example, a
* header value of:
* Id: 0x11223344
* PktType: 0xff
* SeqNo: 0x55667788
* PktLength: 0x99aabbcc
*
* is sent as the following byte stream:
* 11 22 33 44 ff 55 66 77 88 99 aa bb cc
*/
void
send_header(int sd, struct hdr const* header)
{ /* NO ERROR HANDLING */
uint32_t num = htonl((uint32_t)header->Id);
send(sd, &num, sizeof(num), 0);
send(sd, &header->PktType, sizeof(header->PktType), 0);
num = htonl((uint32_t)header->SeqNo);
send(sd, &num, sizeof(num), 0);
num = htonl((uint32_t)header->PktLength);
send(sd, &num, sizeof(num), 0);
}
This will ensure that your client can safely decode it using the struct module:
buf = s.recv(13) # packed data is 13 bytes long
id_, pkt_type, seq_no, pkt_length = struct.unpack('>IBII', buf)
If you cannot modify the C code to fix the serialization indeterminacy, then you will have to read the data from the stream and figure out where the C compiler is inserting padding and manually build struct format strings to match using the padding byte format character to ignore padding values.
I usually write a decoder class in Python that reads a complete value from the socket. In your case it would look something like:
class PacketReader(object):
def __init__(self, sd):
self._socket = sd
def read_packet(self):
id_, pkt_type, seq_no, pkt_length = self._read_header()
data_bufs = [self._read_data_buf() for _ in range(0, 100)]
message, info = self._read_trl()
return {'id': id_, 'pkt_type': pkt_type, 'seq_no': seq_no,
'data_bufs': data_bufs, 'message': message,
'info': info}
def _read_header(self):
"""
Read and unpack a ``hdr`` structure.
:returns: a :class:`tuple` of the header data values
in order - *Id*, *PktType*, *SeqNo*, and *PktLength*
The header is assumed to be packed as 13 bytes with
integers in network byte order.
"""
buf = self._socket.read(13)
# > Multibyte values in network order
# I Id as 32-bit unsigned integer value
# B PktType as 8-bit unsigned integer value
# I SeqNo as 32-bit unsigned integer value
# I PktLength as 32-bit unsigned integer value
return struct.unpack('>IBII', buf)
def _read_data_buf(self):
"""
Read and unpack a single ``data`` structure.
:returns: a :class:`tuple` of data values in order -
*value*, *result*, *count*, and *value*
The data structure is assumed to be packed as 28 bytes
with integers in network byte order and doubles encoded
as IEEE 754 binary64 in network byte order.
"""
buf = self._socket.read(28) # assumes double is binary64
# > Multibyte values in network order
# 10s value bytes
# d result encoded as IEEE 754 binary64 value
# q count encoded as a 64-bit signed integer
# H valueid as a 16-bit unsigned integer value
return struct.unpack('>10sdqH', buf)
def _read_trl(self):
"""
Read and unpack a ``trl`` structure.
:returns: a :class:`tuple` of trl values in order -
*Message* as byte string, *info*
The structure is assumed to be packed as 24 bytes with
integers in network byte order.
"""
buf = self.socket.read(24)
# > Multibyte values in network order
# 16s message bytes
# q info encoded as a 64-bit signed value
return struct.unpack('>16sq', buf)
Mind you that this is untested and probably contains syntax errors but that is how I would approach the problem.
The struct library has all you need to do this.
I have a little problem concerning string generation in C.
The following code snippet is part of a C Extension for a Python/Tkinter app which generates images (mandelbrot, gradients and such). Before anyone asks: I don't want to power up Photoshop for such a simple task - overkill...
The problem I'm having is at the end of the snippet in the last for-loop.
This function generates a PPM image file for further processing. The main goal is to generate a string containing the raster data in binary format and pass that string back to Python and then to Tkinter image data to have a preview of the result.
At the moment I write a file to disk which is pretty slow.
The iterator-function returns a pointer to a RGB-array.
If I now write every single color-value to the file using
fputc(col[0], outfile)
it works (the section which is commeted out).
To get closer to my main goal I tried to merge the three color values into a string and write that into the file.
When I run that code from my Python app, I end up with a file containing just the header.
Could anyone please point me in the right direction? Tha whole C-thing is pretty new to me - so I'm pretty much stuck here...
static PyObject* py_mandelbrotppm(PyObject* self, PyObject* args)
{
//get filename from argument
char *filename;
PyArg_ParseTuple(args, "s", &filename);
//---------- open file for writing and create header
FILE *outfile = NULL;
outfile = fopen(filename, "w");
//---------- create ppm header
char header[17];
sprintf(header,"P6\n%d %d\n255\n", dim_x, dim_y);
fputs(header, outfile);
//---------- end of header generation
for(int y = 0;y<dim_y;y++)
{
for(int x = 0;x<dim_x;x++)
{
int *col = iterator(x,y);
char pixel[3] = {col[0], col[1], col[2]};
fputs(pixel, outfile);
/*
for(int i = 0;i<3;i++)
{
fputc(pixel[i], outfile);
}
*/
}
}
fclose(outfile);
Py_RETURN_NONE;
}
You have a couple of problems with your new code.
pixel is missing a null terminator (and space for it). Fix it like this:
char pixel[4] = {col[0], col[1], col[2], '\0'};
But I'll let you in on a little secret. Putting a bunch of ints into an array of chars is going to truncate them and do all sorts of weird, squirrly things. Maybe not for char-length numbers, but in terms of general style I wouldn't recommend it. Consider this:
...
for(int x = 0;x<dim_x;x++){
int *col = iterator(x,y);
fprintf(outfile, "%d, %d, %d", col[0], col[1], col[2]);
}
...
On the other hand, I'm a little confused as to why iterator returns ints when RGB values are from 0-255, which is precisely the range an unsigned char has:
unsigned char *col = iterator(x,y);
fprintf(outfile, "%u, %u, %u", col[0], col[1], col[2]);
I'm debugging with python audio, having a hard time with the audio coding.
Here I have a string full of audio data, say, [10, 20, 100].
However the data is stored in a string variable,
data = "����������������"
I want to inspect the values of this string.
Below is the things I tried
Print as int
I tried to use print "%i" % data[0]
ended up with
Traceback (most recent call last):
File "wire.py", line 28, in <module>
print "%i" % data[i]
TypeError: %d format: a number is required, not str
Convert to int
int(data[0]) ended up with
Traceback (most recent call last):
File "wire.py", line 27, in <module>
print int(data[0])
ValueError: invalid literal for int() with base 10: '\xd1'
Any idea on this? I want to print the string in a numerical way since the string is actually an array of sound wave.
EDIT
All your answers turned out to be really helpful.
The string is actually generated from the microphone so I believe it to be raw wave form, or vibration data. Further this should be referred to the audio API document, PortAudio.
After looking into PortAudio, I find this helpful example.
** This routine will be called by the PortAudio engine when audio is needed.
** It may called at interrupt level on some machines so don't do anything
** that could mess up the system like calling malloc() or free().
static int patestCallback( const void *inputBuffer, void *outputBuffer,
unsigned long framesPerBuffer,
const PaStreamCallbackTimeInfo* timeInfo,
PaStreamCallbackFlags statusFlags,
void *userData )
{
paTestData *data = (paTestData*)userData;
float *out = (float*)outputBuffer;
unsigned long i;
(void) timeInfo; /* Prevent unused variable warnings. */
(void) statusFlags;
(void) inputBuffer;
for( i=0; i<framesPerBuffer; i++ )
{
*out++ = data->sine[data->left_phase]; /* left */
*out++ = data->sine[data->right_phase]; /* right */
data->left_phase += 1;
if( data->left_phase >= TABLE_SIZE ) data->left_phase -= TABLE_SIZE;
data->right_phase += 3; /* higher pitch so we can distinguish left and right. */
if( data->right_phase >= TABLE_SIZE ) data->right_phase -= TABLE_SIZE;
}
return paContinue;
}
This indicates that there is some way that I can interpret the data as float
To be clear, your audio data is a byte string. The byte string is a representation of the bytes stored in the audio file. You are not going to simply be able to convert those bytes into meaningful values without knowing what is in the binary first.
As an example, the mp3 specification says that each mp3 contains header frames (described here: http://en.wikipedia.org/wiki/MP3). To read the header you would either need to use something like bitstring, or if you feel comfortable doing the bitwise manipulation yourself then you would just need to unpack an integer (4 bytes) and do some math to figure out the values of the 32 individual bits.
It really all depends on what you are trying to read, and how the data was generated. If you have whole byte numbers, then struct will serve you well.
If you're ok with the \xd1 mentioned above:
for item in data: print repr(item),
Note that for x in data will iterate over each value in the list rather than its location. If you want the location you can use for i in range(len(data)): ...
If you want them in numerical form, replace repr(item) with ord(item).
It is better if you use the new {}.format method:
data = "����������������"
print '{0}'.format(data[3])
You could use ord to map each byte to its numeric value between 0-255:
print map(ord, data)
Or, for Python 3 compatibility, do:
print([ord(c) for c in data])
It will also work with Unicode glyphs, which might not be what you want, so make sure you have a bytearray or an actual str or bytes object in Python 2.
I need to properly format the string in order to send it to the arduino connected through a serial port. For example I have this python2.7.5 code:
x = int(7)
y = int(7000.523)
self.ser.write("%s%s" % (x, y))
but I want x in a byte and y in different bytes from x so I can assign a variable for each recieved byte in the arduino code similar to this:
for (i=0; i<3; i++)
{
bufferArray[i] = Serial.read();
}
d1 = bufferArray[0];
d2 = bufferArray[1];
d3 = bufferArray[2];
x = d1;
y = (d2 << 8) + d3;
In other words, I don't want that a piece of y is in the x byte.
What is the proper string format to do this?
Following the advice of #Mattias Nilsson there is a sample code if you want to send two consecutive 16 bit unsigned integers:
import struct
x = int(7)
y = int(7000.523)
buf = struct.pack("<HH", x, y)
# read it back
for i in buf:
print "%02x" % (ord(i))
You can see that they are send each in 2 bytes and the LSB byte is always first. (Tested on intel x64 machine python 2.7.5)
Edit: You should be able to explicitly set the endiannes using the < character for little endian order at the beginning of the format string.
Then you could just send both buffer and the string using Serial.write:
self.ser.write(buf+yourstring+'\0')
You can nottice the zero charater that will terminate your string. If you send the string like this you should not send any zero byte character in your string.
On the arduino side you should read and decode those two integers first and then to read characters in a loop that will end reading if you read a zero byte. You should definitely check if your reading buffer won't overflow too.