Develop Reference

Get documents near a match in pyMongo

I'm trying to get the documents that surround a match in pyMongo. So I would search for a string and get the matches and the entries that are around this match (using the '_index' well, index), so the user has some context on the result.
I'm trying to do it using $setWindowFields to no success, as I'm getting no results. Probably I'm using the wrong syntax?. This is the aggregation that I'm trying:
show_near = ([{'$setWindowFields':{
'partitionBy':None,
'sortBy': {'_index':1},
'output':{
'nearIds':{
'$addToSet':'$_id',
'window':{'documents':[-2,2]}
}
}
}
},
{
'$match':
{field:{'$regex':f'({s})'}}
},
{'$lookup':
{'from':'collection',
'localField':'nearIds',
'foreignField':'_id',
'as':'nearDocs'}
},
{'$unwind':'$nearDocs'},
{'$replaceRoot':{
'newRoot':'$nearDocs'}}])
cursor = self.collection.aggregate(show_near)
Where 's' is the string I want to match and '_index' is the order of the entries.
Any idea? Maybe there is another method to do this? This feature looks perfect for what I want, but maybe I'm mistaken and there is another way. I've tried going back and forth with $gte and $lte, but is not feasible when results start to pile up.
Thanks!

embedding python in a json file

In my application I am using json as the main API. It stores a set of commands to process files. This works great except part of my requirements is that I need to embed python.
I have a clunky solution, but am looking for a better one.
Here is an example of my approach:
{ "file key" :
{ "instruction1" :
{ "commandList" : [
{
"conditionalPythonBlock" : [
"myval = int(dict['count'])",
"retval = myval > 0"
]
}
]}
}
}
I wrote a routine to collapse the "conditionalPythonBlock" into one multi line string which can then be executed. The multi line approach is necessary for correct indentation.
The 'dict' variable in 'dict['mycount']' is local context in the parser of this file.
The retval is also a local variable in the context of the parser.
Anyone have a more elegant solution?
Thanks!

I think one very good answer here is to use YAML instead of JSON.
PyYaml documentation
The above example becomes:
file key :
instruction1 :
- commands :
conditionalPythonBlock : |
myval = int(dict['count'])
retval = myval > 0
Much more readable, and the script section can just be copied in.

Reading text file for specfic keyword included inside brackets '{'

I would like to read a text file which is like below.
It has Geometry names --> " hvac,OUTLET,INLET,Lamelle,duct and wall"
In this case only 6, but I may vary depending on the different simulation of CFD process.
I would like to extract only the Geometry names and its corresponding 'type'. In my case the geometry and types are " hvac,OUTLET,INLET,Lamelle,duct and wall" and "wall and patch" respectively.
Should I use Parse using XML or just search for the string after '{\n' and '}\n' Keyword .
geometry
{
hvac
{
type wall;
inGroups 1(wall);
nFaces 904403;
startFace 38432281;
}
OUTLET
{
type patch;
nFaces 8228;
startFace 39336684;
}
INLET
{
type patch;
nFaces 347;
startFace 39344912;
}
Lamelle
{
type wall;
inGroups 1(wall);
nFaces 204538;
startFace 39345259;
}
duct
{
type wall;
inGroups 1(wall);
nFaces 535136;
startFace 39549797;
}
wall
{
type wall;
inGroups 1(wall);
nFaces 118659;
startFace 40084933;
}
}

The answer depends on whether you want to support the whole general OpenFOAM's dictionary format, or not.
If you only need to support format similar to what you have shown in the question, then a simple regex like \b(\w+)\s+{\s+type\s+(\w+); would do: https://regex101.com/r/yV8tK2/1 . This can be your option if you fully control how this dictionary is created, though in that case it might be even simpler for you to obtain the needed information directly from the code that creates the dictionary.
However, OpenFOAM's format for dictionaries is much more rich than your example. It can allow for #include directives, can allow for regexs as keys, can allow for referencing of other keys using $ syntax, can allow for comments, C++ code snippets and probably many more (I do not pretend to know it well). A typical example can be two dictionaries:
---- File data.incl:
baseType wall;
---- File data
#inputMode merge;
#include "data.incl"
geometry {
/* foo {
type wrongType; // a commented entry
} */
foo {
type $baseType; // this will expand to wall
...
}
"(bar|buz)" { // this will match bar and buz
...
}
}
If you need to parse any such dictionary, then I strongly recommend you to code in C++ and use standard OpenFOAM classes, which will allow you to do this in just a couple of lines of code.

Building an ElasticSearch search with exists using pyes

The goal of this example code is to figure out how to create a query consisting out of multiple filters and queries.
The below example is not working as expected.
I want to be able to execute my search only on document which contain a certain "key". That what I'm trying to reach with the ExistsFilter, but when enabling I don't get any results back.
Any pointers to clear up this question?
#!/usr/bin/python
import pyes
conn = pyes.ES('sandbox:9200')
conn.index('{"test":{"field1":"value1","field2":"value2"}}','2012.9.23','test')
filter = pyes.filters.BoolFilter()
filter.add_must(pyes.filters.LimitFilter(1))
filter.add_must(pyes.filters.ExistsFilter('test')) #uncommenting this line returns the documents
query = pyes.query.BoolQuery()
query.add_must(pyes.query.TextQuery('test.field1','value1'))
query.add_must(pyes.query.TextQuery('test.field2','value2'))
search = pyes.query.FilteredQuery(query, filter)
for reference in conn.search(query=search,indices=['2012.9.23']):
print reference

I don't use pyes (neither python). But, what I can see here is that some informations seems to miss in the ExistsFilter if I compare to the ExistsFilter documentation :
{
"constant_score" : {
"filter" : {
"exists" : { "field" : "user" }
}
}
}
Could it be your issue?

Parsing an unknown data structure in python

I have a file containing lots of data put in a form similar to this:
Group1 {
Entry1 {
Title1 [{Data1:Member1, Data2:Member2}]
Title2 [{Data3:Member3, Data4:Member4}]
}
Entry2 {
...
}
}
Group2 {
DifferentEntry1 {
DiffTitle1 {
...
}
}
}
Thing is, I don't know how many layers of parentheses there are, and how the data is structured. I need modify the data, and delete entire 'Entry's depending on conditions involving data members before writing everything to a new file. What's the best way of reading in a file like this? Thanks!

The data structure basically seems to be a dict where they keys are strings and the value is either a string or another dict of the same type, so I'd recommend maybe pulling it into that sort of python structure,
eg:
{'group1': {'Entry2': {}, 'Entry1': {'Title1':{'Data4': 'Member4',
'Data1': 'Member1','Data3': 'Member3', 'Data2': 'Member2'},
'Title2': {}}}
At the top level of the file you would create a blank dict, and then for each line you read, you use the identifier as a key, and then when you see a { you create the value for that key as a dict. When you see Key:Value, then instead of creating that key as a dict, you just insert the value normally. When you see a } you have to 'go back up' to the previous dict you were working on and go back to filling that in.
I'd think this whole parser to put the file into a python structure like this could be done in one fairly short recursive function that just called itself to fill in each sub-dict when it saw a { and then returned to its caller upon seeing }

Here is a grammar.
dict_content : NAME ':' NAME [ ',' dict_content ]?
| NAME '{' [ dict_content ]? '}' [ dict_content ]?
| NAME '[' [ list_content ]? ']' [ dict_content ]?
;
list_content : NAME [ ',' list_content ]?
| '{' [ dict_content ]? '}' [ ',' list_content ]?
| '[' [ list_content ]? ']' [ ',' list_content ]?
;
Top level is dict_content.
I'm a little unsure about the comma after dicts and lists embedded in a list, as you didn't provide any example of that.

If you have the grammar for the structure of your data file, or you can create it yourself, you could use a parser generator for Python, like YAPPS: link text.

I have something similar but written in java. It parses a file with the same basic structure with a little different syntax (no '{' and '}' only indentation like in python). It is a very simple script language.
Basically it works like this: It uses a stack to keep track of the inner most block of instructions (or in your case data) and appends every new instruction to the block on the top. If it parses an instruction which expects a new block it is pushed to the stack. If a block ends it pops one element from the stack.
I do not want to post the entire source because it is big and it is available on google code (lizzard-entertainment, revision 405). There is a few things you need to know.
Instruction is an abstract class and it has a block_expected method to indicate wether the concrete instruction needs a block (like loops, etc) In your case this is unnecessary you only need to check for '{'.
Block extends Instruction. It contains a list of instructions and has an add method to add more.
indent_level return how many spaces are preceding the instruction text. This is also unneccessary with '{}' singns.
placeholder
BufferedReader input = null;
try {
input = new BufferedReader(new FileReader(inputFileName));
// Stack of instruction blocks
Stack<Block> stack = new Stack<Block>();
// Push the root block
stack.push(this.topLevelBlock);
String line = null;
Instruction prev = new Noop();
while ((line = input.readLine()) != null) {
// Difference between the indentation of the previous and this line
// You do not need this you will be using {} to specify block boundaries
int level = indent_level(line) - stack.size();
// Parse the line (returns an instruction object)
Instruction inst = Instruction.parse(line.trim().split(" +"));
// If the previous instruction expects a block (for example repeat)
if (prev.block_expected()) {
if (level != 1) {
// TODO handle error
continue;
}
// Push the previous instruction and add the current instruction
stack.push((Block)(prev));
stack.peek().add(inst);
} else {
if (level > 0) {
// TODO handle error
continue;
} else if (level < 0) {
// Pop the stack at the end of blocks
for (int i = 0; i < -level; ++i)
stack.pop();
}
stack.peek().add(inst);
}
prev = inst;
}
} finally {
if (input != null)
input.close();
}

That depends on how the data is structured, and what kind of changes you need to do.
One option might be to parse that into a Python data structure, it seems similar, except that you don't have quotes around the strings. That makes complex manipulation easy.
On the other hand, if all you need to do is make changes that modify some entries to other entries, you can do it with search and replace.
So you need to understand the issue better before you can know what the best way is.

This is a pretty similar problem to XML processing, and there's a lot of Python code to do that. So if you could somehow convert the file to XML, you could just run it through a parser from the standard library. An XML version of your example would be something like this:
<group id="Group1">
<entry id="Entry1">
<title id="Title1"><data id="Data1">Member1</data> <data id="Data2">Member2</data></title>
<title id="Title2"><data id="Data3">Member3</data> <data id="Data4">Member4</data></title>
</entry>
<entry id="Entry2">
...
</entry>
</group>
Of course, converting to XML probably isn't the most straightforward thing to do. But your job is pretty similar to what's already been done with the XML parsers, you just have a different syntax to deal with. So you could take a look at some XML parsing code and write a little Python parser for your data file based on that. (Depending on how the XML parser is implemented, you might even be able to copy the code, just change a few regular expressions, and run it for your file)