Develop Reference

Deserializing a Streamed Protocol Buffer Message With Header and Repeated fields

I am working on deserializing a log file that has been serialized in C using protocol buffers (and NanoPB).
The log file has a short header composed of: entity, version, and identifier. After the header, the stream of data should be continuous and it should log the fields from the sensors but not the header values (this should only occur once and at the beginning).The same .proto file was used to serialize the file. I do not have separate .proto files for the header and for the streamed data.
After my implementation, I assume it should look like this:
firmware "1.0.0"
GUID "1231214211321" (example)
Timestamp 123123
Sens1 2343
Sens2 13123
Sens3 13443
Sens4 1231
Sens5 190
Timestamp 123124
Sens1 2345
Sens2 2312
...
I posted this question to figure out how to structure the .proto file initially, when I was implementing the serialization in C. And in the end I used a similar approach but did no include the: [(nanopb).max_count = 1];
Finally I opted with the following .proto in Python (There can be more sensors than 5):
syntax = "proto3";
import "timestamp.proto";
message SessionLogs {
int32 Entity = 1;
string Version = 2;
string GUID = 3;
repeated SessionLogsDetail LogDetail = 4;
}
message SessionLogsDetail
{
int32 DataTimestamp = 1; // internal counter to identify the order of session logs
// Sensor data, there can be X amount of sensors.
int32 sens1 = 2;
int32 sens2= 3;
int32 sens3= 4;
int32 sens4= 5;
}
At this point, I can serialize a message as I log with my device and according to the file size, the log seems to work, but I have not been able to deserialize it on Python offline to check if my implementation has been correct. And I can't do it in C since its an embedded application and I want to do the post-processing offline with Python.
Also, I have checked this online protobuf deserializer where I can pass the serialized file and get it deserialized without the need of the .proto file. In it I can see the header values (field 3 is empty so its not seen) and the logged information. So this makes me think that the serialization is correct but I am deserializing it wrongly on Python.
This is my current code used to deserialize the message in Python:
import PSessionLogs_pb2
with open('$PROTOBUF_LOG_FILENAME$', 'rb') as f:
read_metric = PSessionLogs_pb2.PSessionLogs()
read_metric.ParseFromString(f.read())
Besides this, I've used protoc to generate the .py equivalent of the .proto file to deserialize offline.

It looks like you've serialized a header, then serialized some other data immediately afterwards, meaning: instead of serializing a SessionLogs that has some SessionLogsDetail records, you've serialized a SessionLogs, and then you've serialized (separately) a SessionLogsDetail - does that sound about right? if so: yes, that will not work correctly; there are ways to do what you're after, but it isn't quite as simple as just serializing one after the other, because the root protobuf object is never terminated; so what actually happens is that it overwrites the root object with later fields by number.
There's two ways of addressing this, depending on the data volume. If the size (including all of the detail rows) is small, you can just change the code so that it is a true parent / child relationship, i.e. so that the rows are all inside the parent. When writing the data, this does not mean that you need to have all the rows before you start writing - there are ways of making appending child rows so that you are sending data as it becomes available; however, when deserializing, it will want to load everything in one go, so this approach is only useful if you're OK with that, i.e. you don't have obscene open-ended numbers of rows.
If you have large numbers of rows, you'll need to add your own framing, essentially. This is often done by adding a length-prefix between each payload, so that you can essentially read a single message at a time. Some of the libraries include helper methods for this; for example, in the java API this is parseDelimitedFrom and writeDelimitedTo. However, my understand is that the python API does not currently support this utility, so you'd need to do the framing yourself :(
To summarize, you currently have:
{header - SessionLogs}
{row 0 - SessionLogsDetail}
{row 1 - SessionLogsDetail}
option 1 is:
{header - SessionLogs
{row 0 - SessionLogsDetail}
{row 1 - SessionLogsDetail}
}
option 2 is:
{length prefix of header}
{header - SessionLogs}
{length prefix of row0}
{row 0 - SessionLogsDetail}
{length prefix of row1}
{row 1 - SessionLogsDetail}
(where the length prefix is something simple like a raw varint, or just a 4-byte integer in some agreed endianness)

Leveraging Spell Checker on local machine?

I notice that common applications on a given machine (Mac, Linux, or Windows) have their respective spell checkers. Everything from various IDE, to MS Word/Office, to Note taking software.
I am trying to utilize the built in utility of our respective machines in order to analyze strings for syntactic correctness. It seems that I cant just use what is on the machine and would have to likely download a dictionary in which to compare against.
I was not sure if there was a better way to accomplish this. I was looking at trying to do things locally, but I was not opposed to doing api or curl requests to determine if the words in a string are spelled correctly.
I was looking at:
LanguageTool (hello wrold failed to return an error)
Google's tbproxy seems to not be functional
Dictionary / Meriam-Webster require api keys for automation.
I was looking at Node packages and noticed spell checker modules which encapsulate wordlists as well.
Is there a way to utilize the built in machine dictionaries at all, or would it be ideal if I download a dictionary / wordlist to compare against?
I am thinking a wordlist might be best bet, but i didnt want to reinvent the wheel. What have others done to accomplish similar?

The Credit is going to Lukas Knuth. I want to give an explicit how to for using dictionary and nspell.
Install The following 2 dependancies:
npm install nspell dictionary-en-us
Here is a Sample File I wrote in order to solve the problem.
// Node File
// node spellcheck.js [path]
// path: [optional] either absolute or local path from pwd/cwd
// if you run the file from within Seg.Ui.Frontend/ it works as well.
// node utility/spellcheck.js
// OR from the utility directory using a path:
// node spellcheck.js ../src/assets/i18n/en.json
var fs = require("fs");
var dictionary = require("dictionary-en-us");
var nspell = require("nspell");
var process = require("process");
// path to use if not defined.
var path = "src/assets/i18n/en.json"
let strings = [];
function getStrings(json){
let keys = Object.keys(json);
for (let idx of keys){
let val = json[idx];
if (isObject(val)) getStrings(val);
if (isString(val)) strings.push(val)
}
}
function sanitizeStrings(strArr){
let set = new Set();
for (let sentence of strArr){
sentence.split(" ").forEach(word => {
word = word.trim().toLowerCase();
if (word.endsWith(".") || word.endsWith(":") || word.endsWith(",")) word = word.slice(0, -1);
if (ignoreThisString(word)) return;
if (word == "") return;
if (isNumber(word)) return;
set.add(word)
});
}
return [ ...set ];
}
function ignoreThisString(word){
// we need to ignore special cased strings, such as items with
// Brackets, Mustaches, Question Marks, Single Quotes, Double Quotes
let regex = new RegExp(/[\{\}\[\]\'\"\?]/, "gi");
return regex.test(word);
}
function spellcheck(err, dict){
if (err) throw err;
var spell = nspell(dict);
let misspelled_words = strings.filter( word => {
return !spell.correct(word)
});
misspelled_words.forEach( word => console.log(`Plausible Misspelled Word: ${word}`))
return misspelled_words;
}
function isObject(obj) { return obj instanceof Object }
function isString(obj) { return typeof obj === "string" }
function isNumber(obj) { return !!parseInt(obj, 10)}
function main(args){
//node file.js path
if (args.length >= 3) path = args[2]
if (!fs.existsSync(path)) {
console.log(`The path does not exist: ${process.cwd()}/${path}`);
return;
}
var content = fs.readFileSync(path)
var json = JSON.parse(content);
getStrings(json);
// console.log(`String Array (length: ${strings.length}): ${strings}`)
strings = sanitizeStrings(strings);
console.log(`String Array (length: ${strings.length}): ${strings}\n\n`)
dictionary(spellcheck);
}
main(process.argv);
This will return a subset of strings to look at and they may be misspelled or false positives.
A false positive will be denoted as:
An acronym
non US English variants for words
Un recognized Proper Nouns, Days of the Week and Months for example.
Strings which contain parenthese. This can be augmented out by trimming them off the word.
Obviously, this isnt for all cases, but i added an ignore this string function you can leverage if say it contains a special word or phrase the developers want ignored.
This is meant to be run as a node script.

Your question is tagged as both NodeJS and Python. This is the NodeJS specific part, but I imagine it's very similar to python.
Windows (from Windows 8 onward) and Mac OS X do have built-in spellchecking engines.
Windows: The "Windows Spell Checking API" is a C/C++ API. To use it with NodeJS, you'll need to create a binding.
Mac OS X: "NSSpellChecker" is part of AppKit, used for GUI applications. This is an Objective-C API, so again you'll need to create a binding.
Linux: There is no "OS specific" API here. Most applications use Hunspell but there are alternatives. This again is a C/C++ library, so bindings are needed.
Fortunately, there is already a module called spellchecker which has bindings for all of the above. This will use the built-in system for the platform it's installed on, but there are multiple drawbacks:
1) Native extensions must be build. This one has finished binaries via node-pre-gyp, but these need to be installed for specific platforms. If you develop on Mac OS X, run npm install to get the package and then deploy your application on Linux (with the node_modules-directory), it won't work.
2) Using build-in spellchecking will use defaults dictated by the OS, which might not be what you want. For example, the used language might be dictated by the selected OS language. For a UI application (for example build with Electron) this might be fine, but if you want to do server-side spellchecking in languages other than the OS language, it might prove difficult.
At the basic level, spellchecking some text boils down to:
Tokenizing the string (e.g. by spaces)
Checking every token against a list of known correct words
(Bonus) Gather suggestions for wrong tokens and give the user options.
You can write part 1 yourself. Part 2 and 3 require a "list of known correct words" or a dictionary. Fortunately, there is a format and tools to work with it already:
simple-spellchecker can work with .dic-files.
nspell is a JS implementation of Hunspell, complete with its own dictionary packages.
Additional Dictionaries can be found for example in this repo
With this, you get to choose the language, you don't need to build/download any native code and your application will work the same on every platform. If you're spellchecking on the server, this might be your most flexible option.

GDB command for indexing the members of a structure

I have my structure with several members and I would like to see them all indexed in order.
Struct Ant
{
int type;
char name[100];
long long int food;
}
Now, when I execute the command in gdb
(gdb) ptype struct Ant
$1 = struct
{
int type;
char name[100];
long long int food;
}
I would like to see the output something like
{
0, int type;
1, char name[100];
2, long long int food;
}
Is there a way to get the index of each structure field in order in GDB ?

There is no built-in way to do this. If you need this you can write it yourself in a couple of ways.
One way would be to use the gdb CLI: use set logging and friends to dump the ptype output to a file, then use shell to run some other command on the file to format it the way you like.
Another way would be to use gdb's Python scripting capability to inspect the type and display it how you like. If you search for the pahole command (maybe already on your system, try locate pahole.py -- some Linux distros ship this) you can see an example of how this is done.

How can I parse this report tab-delimited response? Amazon MWS

I am trying to make a web app with Amazon MWS. User's can add, list, delete their products in this app and also list their orders.
For listing their products I am trying to use Reports API with "_GET_MERCHANT_LISTINGS_DATA_". But this method is returns with very bad tab-delimited response. And also when I make request with RequestReport method, It sends the listings report to the shop owner.
This is the dummy response example:
b'item-name\titem-description\tlisting-id\tseller-sku\tprice\tquantity\topen-date\timage-url\titem-is-marketplace\tproduct-id-type\tzshop-shipping-fee\titem-note\titem-condition\tzshop-category1\tzshop-browse-path\tzshop-storefront-feature\tasin1\tasin2\tasin3\twill-ship-internationally\texpedited-shipping\tzshop-boldface\tproduct-id\tbid-for-featured-placement\tadd-delete\tpending-quantity\tfulfillment-channel\nPropars deneme urunu 2 CD-ROM CD-ROM\tThis is a development test product\t0119QL9BRT8\tPARS12344321\t0.5\t9\t2016-01-19 05:26:44 PST\t\ty\t4\t\t\t11\t\t\t\tB01ATBY2NA\t\t\t1\t\t\t8680925084020\t\t\t0\tDEFAULT\n'
Is there anybody know another method for listing products in a shop, or what do you suggest for taking better results from "_GET_MERCHANT_LISTINGS_DATA_" report?
Or how can I parse this tab-delimited string?
Thanks.

You really have two options for obtaining bulk data from amazon, tab-delimited and xml. tab-delimited reads in Excel quite nicely actually, and the routines for splitting the values into usable format are pretty straight forward. Unfortunately Amazon doesn't give you the option of XML or Flat File on every report so you have to use a mix of both under most circumstances.
First off, your title indicates you need to list all listings active and inactive. That is going to be a combination of reports. If you want an 'all inclusive' including problem listings, active listings, hidden listings and cancelled listings you will need three reports:
_GET_MERCHANT_LISTINGS_DATA_
_GET_MERCHANT_CANCELLED_LISTINGS_DATA_
_GET_MERCHANT_LISTINGS_DEFECT_DATA_
All of these are flat file format so you will have a consistent method of reading the data in. In c# you simply read a line, split that line and read each array value. There will be a similar method of performing this in python, which is most likely well documented here on SO. The c# method would look something like this:
while ((line = file.ReadLine()) != null)
{
if (counter == 0)
{
string[] tempParts = line.Split(delimiters);
for (int i = 0; i < tempParts.Length; i++)
{
tempParts[i] = tempParts[i].Trim(); //Clean up remaining whitespace.
}
//Try and verify headers have not changed.
if (!isReportHeaderValid(tempParts))
{
reportStatus.IsError = true;
reportStatus.Exception = new Exception("Report Column headers were not validated!!!!");
return;
}
counter++;
continue;
}
counter++;
string[] parts = line.Split(delimiters);
for (int i = 0; i < parts.Length; i++)
{
parts[i] = parts[i].Trim(); //Clean up remaining whitespace.
}
//Do stuff with parts[1], parts[2] etc
}
This is an example from one of my pieces of code working with the Amazon Inventory report. Basically, I verify that headers are what I would expect them to be (indicating the report format has not changed) and then I split, clean up white space, and work with each element from split.
Python split method:
Python Split
Alternatively, you could probably take the entire stream and stick it straight into an excel spreadsheet since Excel understands how to delimit on tabs.
Edit
Just a note, in my code example i pass 'delimiters' in to the split routine, but I never define it. It is defined as char[] delimiters = new char[] { '\t' };

python can't make sense of c++ string sent over winsock

Goal:
I am writing a socket server/client program (c++ is the server, python is the client) to send xml strings that carry data. My goal is to be able to receive an xml message from c++ in Python via socket.
Method
VS2013 pro
Python 2.7.2 via Vizard 4.1
1) socket communication is created just fine, no problems. I can send/receive stuff
2) after communications are initialized, c++ begins creating xml objects using Cmarkup
3) c++ converts the xml object to std::string type
4) c++ sends the std::string over the stream to Python
Problem:
The "string" received in python from C++ is interpreted as garbage symbols (not trying to offend, someone may have strong feelings for them, I do not ;) that look like symbols you'd see in notepad if you opened a binary file. This is not surprising, since data sent over the stream is binary.
What I cannot figure out is how to get Python to make sense of the stream.
Failed Attempts to fix:
1) made sure that VS2013 project uses Unicode characters
2) tried converting stream to python string and decoding it string.decode()
3) tried using Unicode()
4) also tried using binascii() methods to get something useful, small improvement but still not the same characters I sent from c++
If anyone can lend some insight on why this is happening I'd be most grateful. I have read several forums about the way data is sent over sockets, but this aspect of encoding and decoding is still spam-mackerel-casserole to my mind.
Here's the server code that creates xml, converts to string, then sends
MCD_CSTR rootname("ROOT");//initializes name for root node
MCD_CSTR Framename("FRAME");//creates name for child node
CMarkup xml;//initializes xml object using Cmarkup method
xml.AddElem(rootname);//create the root node
xml.IntoElem();//move into it
xml.AddElem(Framename, MyClient.GetFrameNumber().FrameNumber);//create child node with data from elsewhere, FrameNumber is an int
CStringA strXML = xml.GetDoc();//convert the xml object to a string using Cmarkup method
std::string test(strXML);//convert the CstringA to a std::string type
std::cout << test << '\n';//verify that the xml as a string looks right
std::cout << typeid(test).name() << '\n';//make sure it is the right type
iSendResult = send(ClientSocket, (char *)&test, sizeof(test), 0);//send the string to the client
Here is the code to receive the xml string in Python:
while 1:
data = s.recv(1024)#receive the stream with larger than required buffer
print(data)#see what is in there
if not data: break#if no data then stop listening for more

Since test is a string, this cannot work:
iSendResult = send(ClientSocket, (char *)&test, sizeof(test), 0);//send the string
The std::string is not a character array. It is an object, and all that line does is send nonsensical bytes to the socket. You want to send the data, not the object.
iSendResult = send(ClientSocket, (char *)test.c_str(), test.length(), 0);//send the string

You can't just write the memory at the location of a std::string and think that's serialization. Depending on how the C++ library implemented it, std::string is likely to be a structure containing a pointer to the actual character data. If you transmit the pointer, not only will you fail to send the character data, but the pointer value is meaningless in any other context than the current instance of the program.
Instead, serialize the important contents of the string. Send the length, then send the character data itself. Something like this.
uint32_t len = test.length();
send(..., &len, sizeof(uint32_t), ...);
send(..., test.c_str(), len, ...);