Develop Reference

Makefile - get arguments from .ini/.yml file

Is it possible to have a Makefile grabing arguments from either config.ini or config.yml file?
Let's consider this example, we have a python main.py file which is written as a CLI. Not we do not want users to be filling arguments to a python CLI in terminal so we have an example config.ini file with the arguments:
PYTHON FILE:
import typer
def say_name(name:str):
print('runnig the code')
print(f'Hello there {name}')
if __name__ == "__main__":
typer.run(say_name)
config.ini FILE:
[argument]
name = person
Makefile FILE:
run_code:
python main.py ${config.ini.argument.name}
Is it possible to have a project infrastructure like this?
I am aware that Spacy project does exactly this. However I would like to some something like those even outside NLP project without the need of using spacy.

You need to find, or write, a tool which will read in your .ini file and generate a set of makefile variables from it. I don't know where you would find such a thing but it's probably not hard to write one using a python module that parses .ini files.
Suppose you have a script ini2make that will do this, so that if you run:
ini2make config.ini
it will write to stdout makefile variable assignment lines like this:
config.ini.argument.name = person
config.ini.argument.email = person#somewhere.org
etc. Then you can integrate this into your makefile very easily (here I'm assuming you're using GNU make) through use of GNU make's automatic include file regeneration:
include config.ini.mk
config.ini.mk: config.ini
ini2make $< > $#
Done. Now whenever config.ini.mk doesn't exist or config.ini has been changed since config.ini.mk was last generated, make will run the ini2make script to update it then re-execute itself automatically to read the new values.
Then you can use variables that were generated, like $(config.ini.argument.name) etc.

What is the best way to read config values for a python program? Is referring to an Enum class defined in another file common?

For example, I might try the following config:
class Defaults(Enum):
a = 1
b = 2
Then from my main file, I can refer to it with:
import myconfig
windowSize = Defaults.a
This would allow me to change the enum values whenever I want to vary how my program runs. Is this a common way to use Enums in python configuration?

I think you're asking whether it's common to hold the configuration settings as members of an enumeration. As a more explicit example:
class Defaults(Enum):
window_width = 600
window_height = 480
font_size = 14
Technically, I think that would work, but what benefit is the enumeration providing? Enum is useful for providing choices to pick from. If you really want to do this, I think a plain class, a data class, or just module-level variables would be less confusing. Django's settings.py configuration file seems to be the closest thing to your idea that I've seen in common use.
Your broader question is how to read configuration values for a Python program. Personally, I like the style recommended by The Twelve-Factor App.
The twelve-factor app stores config in environment variables (often shortened to env vars or env). Env vars are easy to change between deploys without changing any code; unlike config files, there is little chance of them being checked into the code repo accidentally; and unlike custom config files, or other config mechanisms such as Java System Properties, they are a language- and OS-agnostic standard.
The most flexible way I've found is to use the argparse module, and use the environment variables as the defaults. That way, you can override the environment variables on the command line. Be careful about putting passwords on the command line, though, because other users can probably see your command line arguments in the process list.
Here's an example that uses argparse and environment variables:
def parse_args(argv=None):
parser = ArgumentParser(description='Watch the raw data folder for new runs.',
formatter_class=ArgumentDefaultsHelpFormatter)
parser.add_argument(
'--kive_server',
default=os.environ.get('MICALL_KIVE_SERVER', 'http://localhost:8000'),
help='server to send runs to')
parser.add_argument(
'--kive_user',
default=os.environ.get('MICALL_KIVE_USER', 'kive'),
help='user name for Kive server')
parser.add_argument(
'--kive_password',
default=SUPPRESS,
help='password for Kive server (default not shown)')
args = parser.parse_args(argv)
if not hasattr(args, 'kive_password'):
args.kive_password = os.environ.get('MICALL_KIVE_PASSWORD', 'kive')
return args
Setting those environment variables can be a bit confusing, particularly for system services. If you're using systemd, look at the service unit, and be careful to use EnvironmentFile instead of Environment for any secrets. Environment values can be viewed by any user with systemctl show.
I usually make the default values useful for a developer running on their workstation, so they can start development without changing any configuration.
If you do want to put the configuration settings in a settings.py file, just be careful not to commit that file to source control. I have often committed a settings_template.py file that users can copy.

How to get a list of warnings from sphinx compilation

I am developing a sphinx based collaborative writing tool. Users access the web application (developed in python/Flask) to write a book in sphinx and compile it to pdf.
I have learned that in order to compile a sphinx documentation from within python I should use
import sphinx
result = sphinx.build_main(['-c', 'path/to/conf',
'path/to/source/', 'path/to/out'])
So far so good.
Now my users want the app to show them their syntax mistakes. But the output (result in the example above) only gives me the exit code.
So, how do I get a list of warnings from the build process?
Perhaps I am being too ambitious, but since sphinx is a python tool, I was expecting to have a nice pythonic interface with the tool. For example, the output of sphinx.build_main could be a very rich object with warnings, line numbers...
On a related note, the argument to the method sphinx.build_main looks just like a wrapper to the command line interface.

sphinx.build_main() calls sphinx.cmdline.main(), which in turn creates a sphinx.application.Sphinx object. You could create such an object directly (instead of "making system calls within python"). Use something like this:
import os
from sphinx.application import Sphinx
# Main arguments
srcdir = "/path/to/source"
confdir = srcdir
builddir = os.path.join(srcdir, "_build")
doctreedir = os.path.join(builddir, "doctrees")
builder = "html"
# Write warning messages to a file (instead of stderr)
warning = open("/path/to/warnings.txt", "w")
# Create the Sphinx application object
app = Sphinx(srcdir, confdir, builddir, doctreedir, builder,
warning=warning)
# Run the build
app.build()

Assuming you used sphinx-quickstart to generate your initial Sphinx documentation set with a makefile, then you can use make to build docs, which in turn uses the Sphinx tool sphinx-build. You can pass the -w <file> option to sphinx-build to write warnings and errors to a file as well as stderr.
Note that options passed through the command line override any other options set in the makefile and conf.py.

Pythonic way to set module-wide settings from external file

Some background (not mandatory, but might be nice to know): I am writing a Python command-line module which is a wrapper around latexdiff. It basically replaces all \cite{ref1, ref2, ...} commands in LaTeX files with written-out and properly formatted references before passing the files to latexdiff, so that latexdiff will properly mark changes to references in the text (otherwise, it treats the whole \cite{...} command as a single "word"). All the code is currently in a single file which can be run with python -m latexdiff-cite, and I have not yet decided how to package or distribute it. To make the script useful for anybody else, the citation formatting needs to be configurable. I have implemented an optional command-line argument -c CONFIGFILE to allow the user to point to their own JSON config file (a default file resides in the module folder and is loaded if the argument is not used).
Current implementation: My single-file command-line Python module currently parses command-line arguments in if __name__ == '__main__', and loads the config file (specified by the user in -c CONFIGFILE) here before running the main function of the program. The config variable is thus available in the entire module and all is well. However, I'm considering publishing to PyPI by following this guide which seems to require me to put the command-line parsing in a main() function, which means the config variable will not be available to the other functions unless passed down as arguments to where it's needed. This "passing down by arguments" method seems a little cluttered to me.
Question: Is there a more pythonic way to set some configuration globals in a module or otherwise accomplish what I'm trying to? (I don't want to rely on 3rd party modules.) Am I perhaps completely off the tracks in some fundamental way?

One way to do it is to have the configurations defined in a class or a simple dict:
class Config(object):
setting1 = "default_value"
setting2 = "default_value"
#staticmethod
def load_config(json_file):
""" load settings from config file """
with open(json_file) as f:
config = json.load(f)
for k, v in config.iteritems():
setattr(Config, k, v)
Then your application can access the settings via this class: Config.setting1 ...

pylint ignores .pylintrc when run from TextMate

I'm trying to get pylint to give html output when I run Validate syntax on a python file in TextMate. I installed pycheckmate, pylint, and created a .pylintrc file in $HOME that sets the output format to html.
In TextMate's Advanced control panel, in the Shell Variables tab, I have TM_PYCHECKER set to /usr/local/share/python/pylint. If I trigger Validate Syntax, it runs pylint with all the default options, and gives me the output. If I change TM_PYCHECKER to /usr/local/share/python/pylint --rcfile "$HOME/.pylintrc" and Validate Syntax again, I get:
Please install PyChecker, PyFlakes or Pylint for more extensive code
checking.
If I run /usr/local/share/python/pylint from the commandline, without any arguments, the output is html, so I know in that case that it's reading the rcfile. What am I missing?

OK, I think I found the problem: pycheckmate sets --output-format=parseable' as a forced argument to pylint. I found this out by replacing /usr/local/share/python/pylint with a wrapper script that printed out its arguments:
#!/usr/bin/env python
import sys
from pylint import lint
print sys.argv[1:]
lint.Run(sys.argv[1:])
And when I ran it in TextMate I saw this:
['--output-format=parseable', '/Users/smithm5/test.py']
test.py:26 [C] Line too long (90/80)
…
So I dug into /Applications/TextMate.app/Contents/SharedSupport/Bundles/Python.tmbundle/Support/bin/pycheckmate.py itself. Sure enough, it adds that argument, as well as a whole lot of hard-coded html. So to fix it, I removed all the escape() wrappers, set opts = () on line 287 so I could set my own darn opts, and changed line 332 to print line.

A somewhat educated guess: try replacing $HOME by the absolute path to your home directory. Shell variables like $HOME are probably not available to use in TextMate's control panel.
UPDATE: Looking at the pycheckmate.py script included with the Python.tmbundle included with the version of TextMate I have, it appears that it is not possible to include arguments , like --rcfile /path/to/rcfile. The value of TM_PYCHECKER is expected to only be the path to the checker binary with no arguments. But, if you make your own copy of the Python.tmbundle, you should be able to edit pycheckmate.py to do as you wish.