Develop Reference

Leveraging Spell Checker on local machine?

I notice that common applications on a given machine (Mac, Linux, or Windows) have their respective spell checkers. Everything from various IDE, to MS Word/Office, to Note taking software.
I am trying to utilize the built in utility of our respective machines in order to analyze strings for syntactic correctness. It seems that I cant just use what is on the machine and would have to likely download a dictionary in which to compare against.
I was not sure if there was a better way to accomplish this. I was looking at trying to do things locally, but I was not opposed to doing api or curl requests to determine if the words in a string are spelled correctly.
I was looking at:
LanguageTool (hello wrold failed to return an error)
Google's tbproxy seems to not be functional
Dictionary / Meriam-Webster require api keys for automation.
I was looking at Node packages and noticed spell checker modules which encapsulate wordlists as well.
Is there a way to utilize the built in machine dictionaries at all, or would it be ideal if I download a dictionary / wordlist to compare against?
I am thinking a wordlist might be best bet, but i didnt want to reinvent the wheel. What have others done to accomplish similar?

The Credit is going to Lukas Knuth. I want to give an explicit how to for using dictionary and nspell.
Install The following 2 dependancies:
npm install nspell dictionary-en-us
Here is a Sample File I wrote in order to solve the problem.
// Node File
// node spellcheck.js [path]
// path: [optional] either absolute or local path from pwd/cwd
// if you run the file from within Seg.Ui.Frontend/ it works as well.
// node utility/spellcheck.js
// OR from the utility directory using a path:
// node spellcheck.js ../src/assets/i18n/en.json
var fs = require("fs");
var dictionary = require("dictionary-en-us");
var nspell = require("nspell");
var process = require("process");
// path to use if not defined.
var path = "src/assets/i18n/en.json"
let strings = [];
function getStrings(json){
let keys = Object.keys(json);
for (let idx of keys){
let val = json[idx];
if (isObject(val)) getStrings(val);
if (isString(val)) strings.push(val)
}
}
function sanitizeStrings(strArr){
let set = new Set();
for (let sentence of strArr){
sentence.split(" ").forEach(word => {
word = word.trim().toLowerCase();
if (word.endsWith(".") || word.endsWith(":") || word.endsWith(",")) word = word.slice(0, -1);
if (ignoreThisString(word)) return;
if (word == "") return;
if (isNumber(word)) return;
set.add(word)
});
}
return [ ...set ];
}
function ignoreThisString(word){
// we need to ignore special cased strings, such as items with
// Brackets, Mustaches, Question Marks, Single Quotes, Double Quotes
let regex = new RegExp(/[\{\}\[\]\'\"\?]/, "gi");
return regex.test(word);
}
function spellcheck(err, dict){
if (err) throw err;
var spell = nspell(dict);
let misspelled_words = strings.filter( word => {
return !spell.correct(word)
});
misspelled_words.forEach( word => console.log(`Plausible Misspelled Word: ${word}`))
return misspelled_words;
}
function isObject(obj) { return obj instanceof Object }
function isString(obj) { return typeof obj === "string" }
function isNumber(obj) { return !!parseInt(obj, 10)}
function main(args){
//node file.js path
if (args.length >= 3) path = args[2]
if (!fs.existsSync(path)) {
console.log(`The path does not exist: ${process.cwd()}/${path}`);
return;
}
var content = fs.readFileSync(path)
var json = JSON.parse(content);
getStrings(json);
// console.log(`String Array (length: ${strings.length}): ${strings}`)
strings = sanitizeStrings(strings);
console.log(`String Array (length: ${strings.length}): ${strings}\n\n`)
dictionary(spellcheck);
}
main(process.argv);
This will return a subset of strings to look at and they may be misspelled or false positives.
A false positive will be denoted as:
An acronym
non US English variants for words
Un recognized Proper Nouns, Days of the Week and Months for example.
Strings which contain parenthese. This can be augmented out by trimming them off the word.
Obviously, this isnt for all cases, but i added an ignore this string function you can leverage if say it contains a special word or phrase the developers want ignored.
This is meant to be run as a node script.

Your question is tagged as both NodeJS and Python. This is the NodeJS specific part, but I imagine it's very similar to python.
Windows (from Windows 8 onward) and Mac OS X do have built-in spellchecking engines.
Windows: The "Windows Spell Checking API" is a C/C++ API. To use it with NodeJS, you'll need to create a binding.
Mac OS X: "NSSpellChecker" is part of AppKit, used for GUI applications. This is an Objective-C API, so again you'll need to create a binding.
Linux: There is no "OS specific" API here. Most applications use Hunspell but there are alternatives. This again is a C/C++ library, so bindings are needed.
Fortunately, there is already a module called spellchecker which has bindings for all of the above. This will use the built-in system for the platform it's installed on, but there are multiple drawbacks:
1) Native extensions must be build. This one has finished binaries via node-pre-gyp, but these need to be installed for specific platforms. If you develop on Mac OS X, run npm install to get the package and then deploy your application on Linux (with the node_modules-directory), it won't work.
2) Using build-in spellchecking will use defaults dictated by the OS, which might not be what you want. For example, the used language might be dictated by the selected OS language. For a UI application (for example build with Electron) this might be fine, but if you want to do server-side spellchecking in languages other than the OS language, it might prove difficult.
At the basic level, spellchecking some text boils down to:
Tokenizing the string (e.g. by spaces)
Checking every token against a list of known correct words
(Bonus) Gather suggestions for wrong tokens and give the user options.
You can write part 1 yourself. Part 2 and 3 require a "list of known correct words" or a dictionary. Fortunately, there is a format and tools to work with it already:
simple-spellchecker can work with .dic-files.
nspell is a JS implementation of Hunspell, complete with its own dictionary packages.
Additional Dictionaries can be found for example in this repo
With this, you get to choose the language, you don't need to build/download any native code and your application will work the same on every platform. If you're spellchecking on the server, this might be your most flexible option.

Reading text file for specfic keyword included inside brackets '{'

I would like to read a text file which is like below.
It has Geometry names --> " hvac,OUTLET,INLET,Lamelle,duct and wall"
In this case only 6, but I may vary depending on the different simulation of CFD process.
I would like to extract only the Geometry names and its corresponding 'type'. In my case the geometry and types are " hvac,OUTLET,INLET,Lamelle,duct and wall" and "wall and patch" respectively.
Should I use Parse using XML or just search for the string after '{\n' and '}\n' Keyword .
geometry
{
hvac
{
type wall;
inGroups 1(wall);
nFaces 904403;
startFace 38432281;
}
OUTLET
{
type patch;
nFaces 8228;
startFace 39336684;
}
INLET
{
type patch;
nFaces 347;
startFace 39344912;
}
Lamelle
{
type wall;
inGroups 1(wall);
nFaces 204538;
startFace 39345259;
}
duct
{
type wall;
inGroups 1(wall);
nFaces 535136;
startFace 39549797;
}
wall
{
type wall;
inGroups 1(wall);
nFaces 118659;
startFace 40084933;
}
}

The answer depends on whether you want to support the whole general OpenFOAM's dictionary format, or not.
If you only need to support format similar to what you have shown in the question, then a simple regex like \b(\w+)\s+{\s+type\s+(\w+); would do: https://regex101.com/r/yV8tK2/1 . This can be your option if you fully control how this dictionary is created, though in that case it might be even simpler for you to obtain the needed information directly from the code that creates the dictionary.
However, OpenFOAM's format for dictionaries is much more rich than your example. It can allow for #include directives, can allow for regexs as keys, can allow for referencing of other keys using $ syntax, can allow for comments, C++ code snippets and probably many more (I do not pretend to know it well). A typical example can be two dictionaries:
---- File data.incl:
baseType wall;
---- File data
#inputMode merge;
#include "data.incl"
geometry {
/* foo {
type wrongType; // a commented entry
} */
foo {
type $baseType; // this will expand to wall
...
}
"(bar|buz)" { // this will match bar and buz
...
}
}
If you need to parse any such dictionary, then I strongly recommend you to code in C++ and use standard OpenFOAM classes, which will allow you to do this in just a couple of lines of code.

python can't make sense of c++ string sent over winsock

Goal:
I am writing a socket server/client program (c++ is the server, python is the client) to send xml strings that carry data. My goal is to be able to receive an xml message from c++ in Python via socket.
Method
VS2013 pro
Python 2.7.2 via Vizard 4.1
1) socket communication is created just fine, no problems. I can send/receive stuff
2) after communications are initialized, c++ begins creating xml objects using Cmarkup
3) c++ converts the xml object to std::string type
4) c++ sends the std::string over the stream to Python
Problem:
The "string" received in python from C++ is interpreted as garbage symbols (not trying to offend, someone may have strong feelings for them, I do not ;) that look like symbols you'd see in notepad if you opened a binary file. This is not surprising, since data sent over the stream is binary.
What I cannot figure out is how to get Python to make sense of the stream.
Failed Attempts to fix:
1) made sure that VS2013 project uses Unicode characters
2) tried converting stream to python string and decoding it string.decode()
3) tried using Unicode()
4) also tried using binascii() methods to get something useful, small improvement but still not the same characters I sent from c++
If anyone can lend some insight on why this is happening I'd be most grateful. I have read several forums about the way data is sent over sockets, but this aspect of encoding and decoding is still spam-mackerel-casserole to my mind.
Here's the server code that creates xml, converts to string, then sends
MCD_CSTR rootname("ROOT");//initializes name for root node
MCD_CSTR Framename("FRAME");//creates name for child node
CMarkup xml;//initializes xml object using Cmarkup method
xml.AddElem(rootname);//create the root node
xml.IntoElem();//move into it
xml.AddElem(Framename, MyClient.GetFrameNumber().FrameNumber);//create child node with data from elsewhere, FrameNumber is an int
CStringA strXML = xml.GetDoc();//convert the xml object to a string using Cmarkup method
std::string test(strXML);//convert the CstringA to a std::string type
std::cout << test << '\n';//verify that the xml as a string looks right
std::cout << typeid(test).name() << '\n';//make sure it is the right type
iSendResult = send(ClientSocket, (char *)&test, sizeof(test), 0);//send the string to the client
Here is the code to receive the xml string in Python:
while 1:
data = s.recv(1024)#receive the stream with larger than required buffer
print(data)#see what is in there
if not data: break#if no data then stop listening for more

Since test is a string, this cannot work:
iSendResult = send(ClientSocket, (char *)&test, sizeof(test), 0);//send the string
The std::string is not a character array. It is an object, and all that line does is send nonsensical bytes to the socket. You want to send the data, not the object.
iSendResult = send(ClientSocket, (char *)test.c_str(), test.length(), 0);//send the string

You can't just write the memory at the location of a std::string and think that's serialization. Depending on how the C++ library implemented it, std::string is likely to be a structure containing a pointer to the actual character data. If you transmit the pointer, not only will you fail to send the character data, but the pointer value is meaningless in any other context than the current instance of the program.
Instead, serialize the important contents of the string. Send the length, then send the character data itself. Something like this.
uint32_t len = test.length();
send(..., &len, sizeof(uint32_t), ...);
send(..., test.c_str(), len, ...);

Sending structured data over a network [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I'm a begginer in network programming, so, sorry if my questions may appear a little obvious.
I'm trying to send some data from Qt application to a Python server which will process them and send back some answer.
the methods that allows me to send data in the QTcpSocket class are:
// ...
write(const QByteArray &)
write(const char *)
// ...
my application will manage: authentification, sending and receiving some complexe data like struct, and files.
I've many questions about this situation:
Are the methods mentioned above sufficient to send complexe data, and how ?
how to deal with the data types in the server side (with Python) ?
do you think I should use an other protocole like HTTP (with the QNetworkAccessManager class) ?

Trying to answer your questions:
Are the methods mentioned above sufficient to send complexe data, and how ?
Well, yes, sending a raw byte array is the lowest level format. However, you need a something that can get you uniquely from your complex data to the byte array and from the byte array back to your complex data.
This process is called in different ways, encoding, serializing, marshalling.... But in general it just means creating a system for encoding complex structures into a sequence of bytes or characters
There are many you can choose from: ASN.1, JSON, XML, Google's protocol buffers or MIME....
You can even design your own (e.g. a simple schema is using TLV: (Tag-Length-Value), where Tag is an identifier of the Type and Value can be a basic type [and you have to define a representation for each type that you consider basic] or again one or more TLV), Length indicates how many bytes/characters are used to encode the Value.
What to choose depends a lot of where you encode (language/platform) and where you decode (language/platform) and your requirements for speed, bandwidth usage, transport, whether messages should be inspected... etc.
If you're dealing with heterogenous architectures you might need to think about endianness.
Finally, you should distinguish between the format (i.e. how the complex structure is expressed as a sequence of bytes in the line) and the library used for encoding (or the library used for decoding). Sometimes they will be linked, sometimes for the same format you will have a choice of libraries to use.
how to deal with the data types in the server side (with Python) ?
So, here you have a requirement... if you're going for a externally provided format, you must make sure it has a python library able to decode it.
if you're going for a home-grown solution, one of things you should define is the expression of your complex C++ structures as Python structures.
An additional possibility is to do everything in C++, and for the python server side use one of the systems for creating python extensions in C++ (e.g. boost-python or swig....)
do you think I should use an other protocol like HTTP (with the QNetworkAccessManager class) ?
It depends on what you try to do.
There are many HTTP libraries widely available that you can use on different languages and different architectures.
You still need to solve the problem of deciding the formatting of your information (although HTTP have some defined practices).
In addition HTTP is clearly biased towards the communication of a client with a server, with the action always initiated by the client.
Things get complex (or less widely supported) when is the server the one that needs to initiate the communication or the one that needs to send spontaneous information.

I do not think it is the language data structure type that should be differentiated, but it is more about the data you send over. Note that, different languages may have different language structures and so on. That is just really low-level details. What is more important is what you send.
You could look into the following example how the serialization/deserialization works with json format in QtCore. Json is also supported in python quite well by the json module, so you would have no issue on the server side to deserialize it:
JSON Save Game Example
This is basically the important part that would give you some hint on the client side. Do not get lost at saving into a file. It is basically writing the raw bytes to the file, which you would replace by sending over the network:
void Game::write(QJsonObject &json) const
{
QJsonObject playerObject;
mPlayer.write(playerObject);
json["player"] = playerObject;
QJsonArray levelArray;
foreach (const Level level, mLevels) {
QJsonObject levelObject;
level.write(levelObject);
levelArray.append(levelObject);
}
json["levels"] = levelArray;
}
... and then you would do something like this on the server side, again instead of reading from file, you would read from the network, but that is not a biggie as both are IO.
import json
json_data=open(file_directory).read()
data = json.loads(json_data)
pprint(data)
You could use raw protocol to design your own, or just use an extending. I would suggest to go with something standard, like http (tcp/udp). Then, you would only need to define the json format for your own data, and not deal with all the rest, like one-way or two-way communication, transaction identifier against reply attack, timestamp, data size and so on.
This would allow you to truly concentrate on the important stuff for you. Once, you have your own json format defined, you could look into the QtNetwork module to send post, get, put and delete requests as you wish.
You would probably work closely with the QNetworkManager, QNetworkReply classes, and so on. Here you can find a simple client implementation in Qt with QtCore's json for a simple pastebin functionality:
#include <QSslError>
#include <QNetworkAccessManager>
#include <QNetworkRequest>
#include <QNetworkReply>
#include <QTcpSocket>
#include <QJsonDocument>
#include <QJsonObject>
#include <QJsonParseError>
#include <QFile>
#include <QScopedPointer>
#include <QTextStream>
#include <QStringList>
#include <QCoreApplication>
#include <QDebug>
int main(int argc, char **argv)
{
QCoreApplication application{argc, argv};
application.setOrganizationName(R"("CutePaste")");
application.setApplicationName(R"("CutePaste Desktop Console Frontend")");
QTextStream standardOutputStream{stdout};
QFile dataFile;
QString firstArgument{QCoreApplication::arguments().size() < 2 ? QString() : QCoreApplication::arguments().at(1)};
if (!firstArgument.isEmpty()) {
dataFile.setFileName(firstArgument);
dataFile.open(QIODevice::ReadOnly);
} else {
dataFile.open(stdin, QIODevice::ReadOnly);
}
QByteArray pasteTextByteArray{dataFile.readAll()};
QJsonObject requestJsonObject;
requestJsonObject.insert(QStringLiteral("data"), QString::fromUtf8(pasteTextByteArray));
requestJsonObject.insert(QStringLiteral("language"), QStringLiteral("text"));
QJsonDocument requestJsonDocument{requestJsonObject};
QString baseUrlString{QStringLiteral(R"("http://pastebin.kde.org")")};
QNetworkRequest networkRequest;
networkRequest.setAttribute(QNetworkRequest::DoNotBufferUploadDataAttribute, true);
networkRequest.setHeader(QNetworkRequest::ContentTypeHeader, R"("application/json")");
networkRequest.setUrl(QUrl(baseUrlString + R"("/api/json/create")"));
QNetworkAccessManager networkAccessManager;
QScopedPointer<QNetworkReply> networkReplyScopedPointer(networkAccessManager.post(networkRequest, requestJsonDocument.toJson()));
QObject::connect(networkReplyScopedPointer.data(), &QNetworkReply::finished, [&] {
QJsonParseError jsonParseError;
QByteArray replyJsonByteArray{networkReplyScopedPointer->readAll()};
QJsonDocument replyJsonDocument{QJsonDocument::fromJson(replyJsonByteArray, &jsonParseError)};
if (jsonParseError.error != QJsonParseError::NoError) {
qDebug() << R"("The json network reply is not valid json:")" << jsonParseError.errorString();
QCoreApplication::quit();
}
if (!replyJsonDocument.isObject()) {
qDebug() << R"("The json network reply is not an object")";
QCoreApplication::quit();
}
QJsonObject replyJsonObject{replyJsonDocument.object()};
QJsonValue resultValue{replyJsonObject.value(QStringLiteral("result"))};
if (!resultValue.isObject()) {
qDebug() << R"("The json network reply does not contain an object for the "result" key")";
QCoreApplication::quit();
}
QJsonValue identifierValue{resultValue.toObject().value(QStringLiteral("id"))};
if (!identifierValue.isString()) {
qDebug() << R"("The json network reply does not contain a string for the "id" key")";
QCoreApplication::quit();
}
endl(standardOutputStream << baseUrlString << '/' << identifierValue.toString());
QCoreApplication::quit();
});
QObject::connect(networkReplyScopedPointer.data(), static_cast<void (QNetworkReply::*)(QNetworkReply::NetworkError)>(&QNetworkReply::error), [&](QNetworkReply::NetworkError networkReplyError) {
if (networkReplyError != QNetworkReply::NoError)
endl(standardOutputStream << networkReplyScopedPointer->errorString());
});
QObject::connect(networkReplyScopedPointer.data(), &QNetworkReply::sslErrors, [&](QList<QSslError> networkReplySslErrors) {
if (!networkReplySslErrors.isEmpty()) {
for (const auto &networkReplySslError : networkReplySslErrors)
endl(standardOutputStream << networkReplySslError.errorString());
}
});
int returnValue{application.exec()};
dataFile.close();
if (dataFile.error() != QFileDevice::NoError)
endl(standardOutputStream << dataFile.errorString());
return returnValue;
}
The JSON is defined in here:
http://sayakb.github.io/sticky-notes/pages/api/
For sure, it is not the only way of doing it, e.g. if you need efficiency, you may well look into a binary format like capnproto.

How can I mine an XML document with awk, Perl, or Python?

I have a XML file with the following data format:
<net NetName="abc" attr1="123" attr2="234" attr3="345".../>
<net NetName="cde" attr1="456" attr2="567" attr3="678".../>
....
Can anyone tell me how could I data mine the XML file using an awk one-liner? For example, I would like to know attr3 of abc. It will return 345 to me.

In general, you don't. XML/HTML parsing is hard enough without trying to do it concisely, and while you may be able to hack together a solution that succeeds with a limited subset of XML, eventually it will break.
Besides, there are many great languages with great XML parsers already written, so why not use one of them and make your life easier?
I don't know whether or not there's an XML parser built for awk, but I'm afraid that if you want to parse XML with awk you're going to get a lot of "hammers are for nails, screwdrivers are for screws" answers. I'm sure it can be done, but it's probably going to be easier for you to write something quick in Perl that uses XML::Simple (my personal favorite) or some other XML parsing module.
Just for completeness, I'd like to note that if your snippet is an example of the entire file, it is not valid XML. Valid XML should have start and end tags, like so:
<netlist>
<net NetName="abc" attr1="123" attr2="234" attr3="345".../>
<net NetName="cde" attr1="456" attr2="567" attr3="678".../>
....
</netlist>
I'm sure invalid XML has its uses, but some XML parsers may whine about it, so unless you're dead set on using an awk one-liner to try to half-ass "parse" your "XML," you may want to consider making your XML valid.
In response to your edits, I still won't do it as a one-liner, but here's a Perl script that you can use:
#!/usr/bin/perl
use strict;
use warnings;
use XML::Simple;
sub usage {
die "Usage: $0 [NetName] ([attr])\n";
}
my $file = XMLin("file.xml", KeyAttr => { net => 'NetName' });
usage() if #ARGV == 0;
exists $file->{net}{$ARGV[0]}
or die "$ARGV[0] does not exist.\n";
if(#ARGV == 2) {
exists $file->{net}{$ARGV[0]}{$ARGV[1]}
or die "NetName $ARGV[0] does not have attribute $ARGV[1].\n";
print "$file->{net}{$ARGV[0]}{$ARGV[1]}.\n";
} elsif(#ARGV == 1) {
print "$ARGV[0]:\n";
print " $_ = $file->{net}{$ARGV[0]}{$_}\n"
for keys %{ $file->{net}{$ARGV[0]} };
} else {
usage();
}
Run this script from the command line with 1 or 2 arguments. The first argument is the 'NetName' you want to look up, and the second is the attribute you want to look up. If no attribute is given, it should just list all the attributes for that 'NetName'.

I have written a tool called xml_grep2, based on XML::LibXML, the perl interface to libxml2.
You would find the value you're looking for by doing this:
xml_grep2 -t '//net[#NetName="abc"]/#attr3' to_grep.xml
The tool can be found at http://xmltwig.com/tool/

xmlgawk can use XML very easily.
$ xgawk -lxml 'XMLATTR["NetName"]=="abc"{print XMLATTR["attr3"]}' test.xml
This one liner can parse XML and print "345".

If you do not have xmlgawk and your XML format is fixed, normal awk can do.
$ nawk -F '[ ="]+' '/abc/{for(i=1;i<=NF;i++){if($i=="attr3"){print $(i+1)}}}' test.xml
This script can return "345".
But I think it is very dangerous because normal awk can not use XML.

You might try this nifty little script: http://awk.info/?doc/tools/xmlparse.html