Develop Reference

How to read a file in Python written out by C++

I have one program written in C++ that outputs the data from several different types of arrays. For simplicity, I'm using ints and just writing them out one at a time to figure this out.
I need to be able to read the file in on Python, but clearly am missing something. I'm having trouble translating the concepts from C++ over to Python.
This is the C++ I have that's working - it writes out two numbers to a file and then reads that file back in (yes, I have to use the ostream.write() and istream.read() functions - that's how at the base level the library I'm using does it and I can't change it).
int main(int argc, char **argv) {
std::ofstream fout;
std::ifstream fin;
int outval1 = 1234;
int outval2 = 5678;
fout.open("out.txt");
fout.write(reinterpret_cast<const char*>(&outval1), sizeof(int));
fout.write(reinterpret_cast<const char*>(&outval2), sizeof(int));
fout.close();
int inval;
fin.open("out.txt");
while (fin.read(reinterpret_cast<char*>(&inval), sizeof(int))) {
std::cout << inval << std::endl;
}
fin.close();
return 0;
}
This is what I have on the Python side, but I know it's not correct. I don't think I should need to read in as binary but that's the only way it's working so far
with open("out.txt", "rb") as f:
while (byte := f.read(1)):
print(byte)

In the simple case you have provided, it is easy to write the Python code to read out 1234 and 5678 (assuming sizeof(int) is 4 bytes) by using int.from_bytes.
And you should open the file in binary mode.
import sys
with open("out.txt", "rb") as f:
while (byte := f.read(4)):
print(int.from_bytes(byte, sys.byteorder))
To deal with floats, you may want to try struct.unpack:
import struct
byte = f.read(4)
print(struct.unpack("f", byte)[0])

How can I read unsigned shorts using Python?

Main question
I would like to understand how to read a C++ unsigned short in Python. I was trying to use np.fromfile('file.bin',np.uint16) but it seems it doesn't work. Refer to this as the main question.
Case study:
For giving some more contest
I have an array of unsigned shorts exported as a binary file using C++ and QDataStream method of QT.
Header:
QVector<unsigned short> rawData;
main.cpp
QFile rawFile(QString("file.bin"));
rawFile.open(QIODevice::Truncate | QIODevice::ReadWrite);
QDataStream rawOut(&rawFile);
rawOut.writeRawData((char *) &rawData, 2*rawData.size());
rawFile.close();
I'm trying to read it using Python and numpy but I can't find how to read unsigned shorts. From literature unsigned shorts should be 2 bytes so I have tried to read it using:
import numpy as np
np.readfromfile('file.bin',np.uint16)
However if I compare a single unsigned_value reading it with python and prining as a string using in C++:
Qstring single_value = QString::number(unsigned_value)
They are different.

I'd experiment with endedness. Try '<u2' or '>u2'
https://docs.scipy.org/doc/numpy/reference/arrays.dtypes.html
'>' reverses the order of the 2 bytes
In [674]: np.array(123, np.dtype('>u2')).tostring()
Out[674]: b'\x00{'
In [675]: np.array(123, np.dtype('<u2')).tostring()
Out[675]: b'{\x00'
In [678]: np.array(123, np.uint16).tostring()
Out[678]: b'{\x00'

rawOut.writeRawData((char *) &rawData, 2*rawData.size()); is writing loads of rubbish in your file. QVector is not directly castable to an array of short as you are trying to do.
Use the code below to write your data
for(const auto& singleVal : rawData)
rawOut << singleVal;

Take a look at struct module
import struct
with open('file.bin', 'rb') as f:
unsigned_shorts = struct.iter_unpack('H', f.read())
print(list(unsigned_shorts))
Example output:
>>>[(1,), (2,), (3,)]

Disable Cache/Buffer on Specific File (Linux)

I am currently working in a Yocto Linux build and am trying to interface with a hardware block on an FPGA. This block is imitating an SD card with a FAT16 file system on it; containing a single file (cam.raw). This file represents the shared memory space between the FPGA and the linux system. As such, I want to be able to write data from the linux system to this memory and get back any changes the FPGA might make (Currently, the FPGA simply takes part of the data from the memory space and adds 6 to the LSB of a 32-bit word, like I write 0x40302010 and should get back 0x40302016 if I read back the data). However, due to some caching somewhere, while I can write the data to the FPGA, I cannot immediately get back the result.
I am currently doing something like this (using python because its easy):
% mount /dev/mmcblk1 /memstick
% python
>> import mmap
>> import os
>> f = os.open("/memstick/cam.raw", os.O_RDWR | os.O_DIRECT)
>> m = mmap.mmap(f, 0)
>> for i in xrange(1024):
... m[i] = chr(i % 256)
...
>> m.flush() # Make sure data goes from linux to FPGA
>> hex(ord(m[0])) # Should be 0x6
'0x0'
I can confirm with dd that the data is changed (though I frequently run into buffering issues with that too) and using the tools for the FPGA (SignalTap/ChipScope) that I am indeed getting correct answer (ie, the first 32-bit word in this case is 0x03020106). However, someone, whether its python or linux or both are buffering the file and not reading from the "SD card" (FPGA) again and storing the file data in memory. I need to shut this completely off so all reads result in reads from the FPGA; but Im not sure where the buffering is taking place or how to do that.
Any insight would be appreciated! (Note, I can use mmap.flush() to take any data I write from python to dump it to the FPGA, but I need like a reverse flush or something to have it reread the file data into the mmap!)
Update:
As suggested in the comments, the mmap approach might not be the best one to implement what I need. However, I have now tried both in python and C, but using basic I/O functions (os.read/write in python, read/write in C) using the O_DIRECT flag. For most of these operations, I end up getting errno 22. Still looking into this....

After doing digging, I found out what I was doing wrong with the O_DIRECT flag. In my C and Python versions, I wasnt using memalign to create the buffer and wasn't doing block reads/writes. This post has a good explanation:
How can I read a file with read() and O_DIRECT in C++ on Linux?
So, in order to achieve what I am doing, this C program works as a basic example:
#include <stdio.h>
#include <fcntl.h>
#include <errno.h>
#define BLKSIZE 512
int main() {
int fd;
int x;
char* buf;
fd = open("/home/root/sd/fpga/cam.raw", O_RDWR | O_SYNC | O_DIRECT);
if (!fd) {
printf("Oh noes, no file!\n");
return -1;
}
printf("%d %d\n", fd, errno);
buf = (char*) memalign(BLKSIZE, BLKSIZE*2);
if (!buf) {
printf("Oh noes, no buf!\n");
return -1;
}
x = read(fd, buf, BLKSIZE);
printf("%d %d %x %x %x %x\n", x, errno, buf[0], buf[1], buf[2], buf[3]);
lseek(fd, 0, 0);
buf[0] = '1';
buf[1] = '2';
buf[2] = '3';
buf[3] = '4';
x = write(fd, buf, BLKSIZE);
printf("%d %d\n", fd, errno);
lseek(fd, 0, 0);
x = read(fd, buf, BLKSIZE);
printf("%d %d %x %x %x %x\n", x,errno, buf[0], buf[1], buf[2], buf[3]);
return 0;
}
This will work for my purposes, I didnt look how to do proper memory alignment to use Python's os.read/os.write functions in a similar way.

How to read packet saved in a file with Python?

I have a C++ code that generates an IP Packet Header. The code use a struct representing each field in the packet:
struct cip {
uint8_t ip_hl:4, /* both fields are 4 bytes */
ip_v:4;
uint8_t ip_tos;
uint16_t ip_len;
uint16_t ip_id;
uint16_t ip_off;
uint8_t ip_ttl;
uint8_t ip_p;
uint16_t ip_sum;
struct in_addr ip_src;
struct in_addr ip_dst;
char head[100];
};
The user is prompt an input message to enter the values for each variable in the struct:
Enter the filename to save the packet: packet
Enter IP version(0-15): 4
Enter Header Length(5-15): 5
Enter type of service(0-255): 55
Enter packet total size(bytes, 20, 200): 25
The packet is created and saved in a file:
FILE* f = fopen(file, "w");
int success = fwrite(&packet, sizeof(char), ((unsigned int)packet.ip_hl)*4,f);
if(success <= 0) {
printf("Error writing packet header");
}
success = fwrite(&data, sizeof(char),ntohs(packet.ip_len)-(4*packet.ip_hl),f);
if(success < 0) {
printf("Error writing packet data");
}
fflush(f);
fclose(f);
printf("\nPacket Written.\n");
I didn't create this code, someone gave me the code so I can create other program in Python that will validate the packet created by the program above. The validation includes verifying the checksum generated for the packet, the version of the Ip Packet, protocol, length of header and so on.
So I will like to know if someone can help me figuring out how can I read the file and parse the frame. I tried to read the line in the file as a string, but the problem I'm having is that the file looks like this after the creation: (it is unreadable)
O È ,# šÀ¨À¨
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxDATA_______________________DATA____________ ô·
I don't understand why: (I'm guessing that this is because the variables bigger than 1 byte are converted to big endian by the function "htons":
printf("\nEnter ip ID number(0-65535):\n");
scanf("%d", &input);
packet.ip_id = htons(input);
I tried to search for another option as dealing this with socket.makefile(), but this will help me the socket in my program as a file, but what I need to do is parse the frame gave to me in this file.
Any ideas?
Thanks.
P.S.: Also can someone give me a link where I can find how to convert integer from big endian to small endian and vicerversa in Python. Thanks!

You should read file as usual (specifying "binary" mode for Windows):
with open("test.txt", 'br') as f:
for line in f.readlines():
# process lines
To unpack binary data you should use struct package, which can also handle big and little endian and so on. Example for your struct:
print struct.unpack('BBHHHBBH100s', line)
I omitted ip_src and ip_dst unpacking since you didn't specify the contents of their struct. The least possible value to read is one byte, so to split first field into two parts you can use:
(ip_hl, ip_v) = (value >> 4, value & 15)
Of course, the order of 8-bit component depends on your struct endianess.

Read pcap header length field with python

I have captured some packets using pcap library in c. Now i am using python program to read that saved packet file. but i have a problem here. I have a file which first have pkthdr(provided by lybrary) and then actual packet.
format of pkthdr is-
struct pcap_pkthdr {
struct timeval ts; /* time stamp 32bit */ 32bit
bpf_u_int32 caplen; /* length of portion present */
bpf_u_int32 len; /* length this packet (off wire) */
};
now i want to read len field, so i have skipped timeval and cap len, and printed len field using python in binary form.. the binary code which i got is-
01001010 00000000 00000000 00000000
Now how to read it in u_int32, i dont think it is correct value(too large), actual len field value should be 74 byte(check in wireshark).. so please tell me what i am doing wrong..
thanks in advance

Or have a look at the pylibpcap module, the pypcap module, or the pcapy module, which let you just call pcap APIs with relative ease. That way you don't have to care about the details of pcap files, and your code will, with libpcap 1.1 or later, also be able to read at least some of the pcap-ng files that Wireshark can produce and that it will produce by default in the 1.8 release.
Writing your own code to read pcap files, rather than relying on libpcap/WinPcap to do so, is rarely worth doing. (Wireshark does so, as part of its library that reads a number of capture file formats and supports pcap-ng format in ways that the current pcap API can't, but the library in question also supports pcap-ng....)

Have a look at the struct module, which lets you unpack such binary data with relative ease, for example:
struct.unpack('LLL', yourbuffer)
This will give you a tuple of the three (L = unsigned long) values. If the len value doesn't seem right, the byte order of the file is different from your native one. In that case prefix the format string with either > (big-endian) or < (little-endian):
struct.unpack('>LLL', yourbuffer)