Develop Reference

Dynamically update complex OrderedDict (based on yaml file)

I'm trying to build a piece of software that will rely on very dynamic configuration (or "ruleset", really). I've tried to capture it in the pseudocode:
"""
---
config:
item1:
thething: ${stages.current.variables.item1}
stages:
dev:
variables:
item1: stuff
prod:
variables:
item1: stuf2
"""
config_obj = yaml.load(config)
current_stage = 'dev'
#Insert artificial "current stage" to allow var matching
stages['current'] = stages[current_stage]
updated_config_obj = replace_vars(config_obj)
The goal is to have the updated_config_obj replace all variable-types with the actual value, so for this example it should replace ${stages.current.variables.item1} with stuff. The current part is easily solved by copying whatever's the current stage into a current item in the same OrderedDict, but I'm still stomped by how to actually perform the replace. The config yaml can be quite large and is totally depended on a plugin system so it must be dynamic.
Right now I'm looking at "walking" the entire object, checking for the existence of a $ on each "leaf" (indicating a variable) and performing a lookup backup to the current object to "resolve" the variable, but somehow that seems overly complex. Another alternative is (I guess) to ue Jinja2-templating on the "config string", with the parsed object as a lookup. Certainly doable but it somehow feels a little dirty.
I have the feeling that there should be a more elegant solution which can be done solely on the parsed object (without interacting with the string), but it escapes me.
Any pointers appreciated!

First, my two cents: try to avoid using any form of interpolation in your configuration file. This creates another layer of dependencies - one dependency for your program (the configuration file) and another dependency for your configuration file.
It's a slick solution at the moment, but consider that five years down the road some lowly developer might be staring down ${stages.current.variables.item1} for a month trying to figure out what this is, not understanding that this implicitly maps onto stages.dev. And then worse yet, some other developer comes along, and seeing that the floodgates of interpolation have been opened, they start using {{stages_dev}}, to mean that some value should interpolated from the system's environmental variables. And then some other developer starts using their own convention like {{!stagesdev!}}, which means that the value uses its own custom runtime interpolation, invoked in some obscure, downstream back-alley.
And then some consultant is hired to reverse-engineer the whole thing and now they are sailing the seas of spaghetti.
If you still want to do this, I'd recommend opening/parsing the configuration file into a dictionary (presumably using yaml.load()), then iterating through the whole thing, line-by-line, using regex to find instances of \$\{(.*)\}.
For each captured group, create an ordered list like:
# ["stages", "current", "variables", item1"]
yaml_references = ".".split("stages.current.variables.item1")
Then, you could do something like:
yaml_config_dict = "" # the parsed configuration file
interpolated_reference = None
for y in yaml_references:
interpolated_reference = yaml_config_dict[y]
i = interpolated_reference[0]
Now i should represent whatever ${stages.current.variables.item1} was pointing to in the context of the .yaml file and you should be able to do a string replace.

Define context variables in behave python

Sometimes, you need to define values dynamically, (like datetime now, random strings, random integers, file contents, etc.) and use them across different steps without being explicit or hard-coding the value.
So, my question is how could I define variables inside of steps (the correct way to do it) to use these variables in the following steps.
Some example
Given A random string of length "100" as "my_text"
And I log in to my platform
And I ask to add the following post:
| title | description |
| Some example of title | {{my_text}} |
When I submit the post form
Then The posts table shows these posts:
| title | description |
| Some example of title | {{my_text}} |
And I delete any post containing in the description "{{my_text}}"
This is a basic example trying to explain why I would like to define variables in steps and save them in the context to use it in the following steps.
My idea was to modify before_step and after_step methods... to set a variable in context to store my custom variables like this:
def before_step(context):
if not hasattr(context, 'vars'):
context.vars = {}
if hasattr(context, table) and context.table:
parse_table(context)
def parse_table(context):
# Here use a regex to check each cell and look for `"{{<identifier>}}"` and if match, replace the cell value by context.vars[identifier] so the step "the posts table shows these posts will never know what is `{{my_text}}` it will be abstract seeing the random string.
Scenarios Outline, use something like this defining variables like "<some_identifier>" and then for each example replace the value in the step.
It's basically to reproduce the behaviour but for any kind of step, simple or using tables.
Is it the right way to do something like this?

From Behave docs on the context:
When behave launches into a new feature or scenario it adds a new layer to the context, allowing the new activity level to add new values, or overwrite ones previously defined, for the duration of that activity. These can be thought of as scopes:
#given('I request a new widget for an account via SOAP')
def step_impl(context):
client = Client("http://127.0.0.1:8000/soap/")
// method client.Allocate(...) returns a dict
context.response = client.Allocate(customer_first='Firstname',
customer_last='Lastname', colour='red')
// context vars can be set more directly
context.new_var = "My new variable!"
#then('I should receive an OK SOAP response')
def step_impl(context):
eq_(context.response['ok'], 1)
cnv = str(context.new_var)
print (f"This is my new variable:'{cnv}'"
So, the value can be set using dot notation and retrieved the same.

To answer this question, one needs note:
Does the test data needs to be controlled externally? For example, test data can be inputed from command line so that the value can be chosen explicitly.
If the answer is no, then probably we should not bother hard coding anything in the feature file. And we can leave the data generated in one step, save it in context, and accessed again in any followed step.
The example I can think is exactly like what the question described. Do we care what the random text content we generated, posted and verified? Probably not. Then we should not expose such detail to user (i.e. feature file) since it is not important to the behaviour we are testing.
If the answer is yes, we do need a bit hack to make it happen. I am experiencing a case like this. What I want is to change the test data when I run the test so I don't have to hard code them in the feature files as in a table or scenario outline. How can I do this?
I can use -D option in command line to pass in as many user data as possible, which can then be accessed in context.config.userdata dictionary in any steps. If the number of test data is very limited. This approach is an easy way to go. But if the test data set contains many data that no one want type one by one in command line, it can be stored externally, for example, a ini file with section names like testdata_1...testdata_n, and thus a string can be passed in from command line to be used to address the section name in this config file. And the test data can be read out in either before_all, or before_scenario, etc., and get used in all steps.

In my experience , you cannot create a dynamic value in feature file.
for example, this step :
Given A random string of length "100" as "my_text"
I dont see any way to change {my_text} each time you run the scenario. (not consider to use behave -D to parse the value to context.config.userdata,I think it is also a wrong approach)
Even Scenario Outline, it actually splits to many scenarios. each scenario will have
different value but the value of {my_text} is already defined in Examples table for each scenario.
The way makes a step dynamic is using Step definition (Code layer).
You can generate a random number in step definition #given('A random string of length "100" as "{my_text}"')
And use context.my_text to store the created number and using it arround.
I also agree with Murphy Meng that you don't need to expose the generated random number
explicitly in feature file. You know which step will use that number, simply use context.my_text in that step to get the value. That's it.

Can Sphinx napoleon document function returning multiple arguments?

I am trying to use the Google code style to document a function that I then use sphinx with the napoleon extension to create documentation for. The function is unusual in that is returns two arguments. I don't think napoleon handles this. If so, could someone tell me how they handle it?
def foo(a):
'''one line summary
longer explanation
Args:
a (int): parameter description
Returns:
servers (list): list of servers to use
msg (str): logging message string
'''
pass
Maybe I'm getting a message that it is not great coding style to return multiple arguments, but can you do this? The html generated treats those two lines as part of a description for one argument. If I put a newline between the servers and msg line, it helps, but it is still documenting one arg.

Python only returns a single object. If you call
serv,msg = foo(myinput)
Then you are explicitly expanding the expression_list tuple which is generated when the function returns with this code
return servers,msg
You docstring should read some thing like this (with the Napoleon Google Style)
"""
one line summary
longer explanation
Args:
a (int): parameter description
Returns:
(tuple): tuple containing:
servers(list) servers to use
msg (str): logging message string
"""
Or with the Napoleon NumPy style:
"""
one line summary
longer explanation
Parameters
----------
a : int
parameter description
Returns
-------
servers : list
servers to use
msg : str
logging message string
"""
Have a look at the python docs for return and perhaps expression_list

Google style does not support multiple return values. As a workaround you can use:
Returns:
2-element tuple containing
- **rates** (*array*): the unnormalized rates (just the sum of the
exponential kernels). To obtain rates in Hz divide the
array by `2*tau` (or other conventional `x*tau` duration).
- **nph** (*array*): number of photons in -5*tau..5*tau window
for each timestamp. Proportional to the rate computed
with KDE and rectangular kernel.
This results in a nice output even with multi-line description for each returned item.

You can configure napoleon to interpret the Returns section of a Google-style docstring like the Args section using the napoleon_custom_sections setting.
napoleon_custom_sections = [('Returns', 'params_style')]
This way, multiple return values (as given in the question) are rendered nicely by Sphinx. However, I am not entirely sure if one is still strictly adhering to the Google-style docstring convention when using this option.

After trying several options, this format worked for me
def foo(a):
"""
Args:
a (int): parameter description
Returns:
- list:
parameter description
- str:
logging message string
"""
Note the two spaces after the linebreak.

Referencing long names with Python Sphinx

I'm working on documentation for my Python module (using Sphinx and reST), and I'm finding that when cross-referencing other Python objects (modules, classes, functions, etc) the full object name ends up being incredibly long. Often it is longer than 80 characters, which I would like to avoid at all costs.
Here is an example:
def exampleFunction():
'''Here is an example docstring referencing another
:class:`module1.module2.module3.module4.module5.ReallyLongExampleClassName`
'''
The issue is that when creating the documentation for the ReallyLongExampleClassName class, I generated it for the full path name module1.module2.module3.module4.module5.ReallyLongExampleClassaName.
I'm wondering if there is any way to solve this? I have tried the following methods, with no success:
1) Adding a line break in the middle of the module name. Example:
:class:`module1.module2.module3.module4.
module5.ReallyLongExampleClassName`
2) Referencing the class name in a different (but still Python importable) way. Example:
:class:`module1.module2.ReallyLongClassName`
I believe that since the documentation for ReallyLongClassName is tied to the full path names that Sphinx cannot correlate the shortened version with the fully named version.
Edit 04/05/2012:
As per the answer/suggestion of j13r (see below) I tried the following:
:class:`module1.module2.module3.module4.module5\
ReallyLongExampleClassName`
And this worked successfully. The only caveat to get this to work, is that the second line must not have spaces before it (which is quite frustrating when using this in a docstring). Thus to make my original example work it would look like:
def exampleFunction():
'''Here is an example docstring referencing another
:class:`module1.module2.module3.module4.module5.\
ReallyLongExampleClassName`
'''
Nice, and ugly. If you were to put spaces before ReallyLongExampleClassName to indent it to the same level as the line above it the output would include the spaces and thus Sphinx would try to reference something like module1.module2.module3.module4.module5.ReallyLongExampleClassName.
I should also note that I tried two other variations of this, which did NOT work:
# Note: Trying to put a space before the '\'
:class:`module1.module2.module3.module4.module5. \
ReallyLongExampleClassName`
# Note: Trying to leave out the '\'
:class:`module1.module2.module3.module4.module5.
ReallyLongExampleClassName`
I was looking for a solution that didn't involve destroying the formatting of the docstring, but I suppose it will do...I think I actually prefer a line that goes past 80 characters to this.
Thanks to j13r for the answer!

According to the sphinx documentation (https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#cross-referencing-python-objects) you could use a dot before your target class:
:class:`.ReallyLongExampleClassName`
or
:class:`.module5.ReallyLongExampleClassName`
and let sphinx search for the class:
... if the name is prefixed with a dot, and no exact match is found, the target is taken as a suffix and all object names with that suffix are searched. For example, :py:meth:.TarFile.close references the tarfile.TarFile.close() function, even if the current module is not tarfile. Since this can get ambiguous, if there is more than one possible match, you will get a warning from Sphinx.

You can use ~ as prefix, it does exactly what you want.
http://sphinx-doc.org/markup/inline.html#xref-syntax

Another strategy is to use reST Substitutions. This will let you save more space in the text by calling the :class: cross-reference later on:
def exampleFunction():
'''Here is an example docstring referencing another
|ReallyLongExampleClassName|
.. |ReallyLongExampleClassName| replace::
:class:`.ReallyLongExampleClassName`
'''
If you're referring to the same class in many of your files, you could instead put the substitution in your Sphinx conf.py file, using the rst_epilog setting. From the Sphinx documentation:
rst_epilog
A string of reStructuredText that will be included at the end of every source file that is read. This is the right place to add substitutions that should be available in every file. An example:
rst_epilog = """
.. |psf| replace:: Python Software Foundation
"""
New in version 0.6.
Then your docstring would just be:
def exampleFunction():
'''Here is an example docstring referencing another
|ReallyLongExampleClassName|
'''

Wild stab in the dark. Perhaps this works:
:class:`module1.module2.module3.module4.\
module5.ReallyLongExampleClassName`
It would be valid Python
import scipy.\
stats

How are arguments in a configuration file parsed regardless of position?

e.g. A configuration file can have
CFLAGS = "xyz"
CXXFLAGS = "xyz"
OR
CXXFLAGS = "xyz"
CFLAGS = "xyz"
Best implementation I could think of would be to just split the argument and feed the left side into a switch
for line in file
x = line.split("=")
switch(x[0])
case CFLAGS
do cflags
case CXXFLAGS
do cxxflags
But how do people who have way more experience than me do it? I know theres probably some open source programs who do this but I wouldn't even know where to look in their source for this.
I program mainly in python and C so implementations/pseudocode/whattolookup in both would be preferred although other languages are fine also.
Thanks in advance.
P.S. try to avoid saying any form of re, regex, regexp, regular expressions, or any derivative thereof in your answers unless its unavoidable :P.

In Python just use the ConfigParser module which will parse .ini-like configuration files for you.
Re implementing this yourself, I find it convenient to view configuration data as a kind of dictionary. This naturally translates to Python's dicts, so if I split the line to <key> = <value> I just go on and update:
confdict[key] = value
With this scheme, the order of the keys in the configuration file doesn't matter, just like it doesn't matter in the dictionary itself - as long as you can lookup values for keys, you're happy.
If you look under the hood of ConfigParser, for example (the relevant method is _read), you will find this is exactly what it does. The options are kept in a dictionary (one per section, because .ini configuration files have one level of hierarchy). Lines are parsed from the file using regular expressions and key, value pairs are added to the dictionary as I described above.
This is Python. In C, I imagine there are quite a few libraries for doing this, but implementing your own would follow exactly the same algorithm. You'd use some kind of associative array data structure for the dictionary (hash table, tree, or whatever, doesn't really matter) and do the same parsing & assigning.

As Eli Bendersky says, in Python you should just use the provided ConfigParser.
If you insist on doing it yourself, his method of storing the configuration as a dictionary is one I recommend. Another way is to map the options to functions which process the values and do something with them:
# Map the options to functions to handle them.
handlers = {
'CFLAGS': some_function,
'CXXFLAGS': other_function,
}
# Loop through each option.
for line in configuration:
# Get the option and value.
option, value = line.split('=')
option = option.strip().upper()
value = value.strip()
# Try to find a handler and process the option.
handler = handlers.get(option)
if handler:
handler(option, value)
else:
raise Exception('Unknown option.')
Obviously, the handler functions must be able to accept the option and value parameters you're passing it.