Develop Reference

How to fake a module in unittest?

I have a code that is based on a configuration file called config.py which defines a class called Config and contains all the configuration options. As the config file can be located anywhere in the user's storage, so I use importlib.util to import it (as specified in this answer). I want to test this functionality with unittest for different configurations. How do I do it? A simple answer could be make a different file for every possible config I want to test and then pass its path to the config loader but this is not what I want. What I basically need is that I implement the Config class, and fake it as if it were the actual config file. How to achieve this?
EDIT Here is the code I want to test:
import os
import re
import traceback
import importlib.util
from typing import Any
from blessings import Terminal
term = Terminal()
class UnknownOption(Exception):
pass
class MissingOption(Exception):
pass
def perform_checks(config: Any):
checklist = {
"required": {
"root": [
"flask",
"react",
"mysql",
"MODE",
"RUN_REACT_IN_DEVELOPMENT",
"RUN_FLASK_IN_DEVELOPMENT",
],
"flask": ["HOST", "PORT", "config"],
# More options
},
"optional": {
"react": [
"HTTPS",
# More options
],
"mysql": ["AUTH_PLUGIN"],
},
}
# Check for missing required options
for kind in checklist["required"]:
prop = config if kind == "root" else getattr(config, kind)
for val in kind:
if not hasattr(prop, val):
raise MissingOption(
"Error while parsing config: "
+ f"{prop}.{val} is a required config "
+ "option but is not specified in the configuration file."
)
def unknown_option(option: str):
raise UnknownOption(
"Error while parsing config: Found an unknown option: " + option
)
# Check for unknown options
for val in vars(config):
if not re.match("__[a-zA-Z0-9_]*__", val) and not callable(val):
if val in checklist["optional"]:
for ch_val in vars(val):
if not re.match("__[a-zA-Z0-9_]*__", ch_val) and not callable(
ch_val
):
if ch_val not in checklist["optional"][val]:
unknown_option(f"Config.{val}.{ch_val}")
else:
unknown_option(f"Config.{val}")
# Check for illegal options
if config.react.HTTPS == "true":
# HTTPS was set to true but no cert file was specified
if not hasattr(config.react, "SSL_KEY_FILE") or not hasattr(
config.react, "SSL_CRT_FILE"
):
raise MissingOption(
"config.react.HTTPS was set to True without specifying a key file and a crt file, which is illegal"
)
else:
# Files were specified but are non-existent
if not os.path.exists(config.react.SSL_KEY_FILE):
raise FileNotFoundError(
f"The file at { config.react.SSL_KEY_FILE } was set as the key file"
+ "in configuration but was not found."
)
if not os.path.exists(config.react.SSL_CRT_FILE):
raise FileNotFoundError(
f"The file at { config.react.SSL_CRT_FILE } was set as the certificate file"
+ "in configuration but was not found."
)
def load_from_pyfile(root: str = None):
"""
This loads the configuration from a `config.py` file located in the project root
"""
PROJECT_ROOT = root or os.path.abspath(
".." if os.path.abspath(".").split("/")[-1] == "lib" else "."
)
config_file = os.path.join(PROJECT_ROOT, "config.py")
print(f"Loading config from {term.green(config_file)}")
# Load the config file
spec = importlib.util.spec_from_file_location("", config_file)
config = importlib.util.module_from_spec(spec)
# Execute the script
spec.loader.exec_module(config)
# Not needed anymore
del spec, config_file
# Load the mode from environment variable and
# if it is not specified use development mode
MODE = int(os.environ.get("PROJECT_MODE", -1))
conf: Any
try:
conf = config.Config()
conf.load(PROJECT_ROOT, MODE)
except Exception:
print(term.red("Fatal: There was an error while parsing the config.py file:"))
traceback.print_exc()
print("This error is non-recoverable. Aborting...")
exit(1)
print("Validating configuration...")
perform_checks(conf)
print(
"Configuration",
term.green("OK"),
)

Without seeing a bit more of your code, it's tough to give a terribly direct answer, but most likely, you want to use Mocks
In the unit test, you would use a mock to replace the Config class for the caller/consumer of that class. You then configure the mock to give the return values or side effects that are relevant to your test case.
Based on what you've posted, you may not need any mocks, just fixtures. That is, examples of Config that exercise a given case. In fact, it would probably be best to do exactly what you suggested originally--just make a few sample configs that exercise all the cases that matter.
It's not clear why that is undesirable--in my experience, it's much easier to read and understand a test with a coherent fixture than it is to deal with mocking and constructing objects in the test class. Also, you'd find this much easier to test if you broke the perform_checks function into parts, e.g., where you have comments.
However, you can construct the Config objects as you like and pass them to the check function in a unit test. It's a common pattern in Python development to use dict fixtures. Remembering that in python objects, including modules, have an interface much like a dictionary, suppose you had a unit test
from unittest import TestCase
from your_code import perform_checks
class TestConfig(TestCase):
def test_perform_checks(self):
dummy_callable = lambda x: x
config_fixture = {
'key1': 'string_val',
'key2': ['string_in_list', 'other_string_in_list'],
'key3': { 'sub_key': 'nested_val_string', 'callable_key': dummy_callable},
# this is your in-place fixture
# you make the keys and values that correspond to the feature of the Config file under test.
}
perform_checks(config_fixture)
self.assertTrue(True) # i would suggest returning True on the function instead, but this will cover the happy path case
def perform_checks_invalid(self):
config_fixture = {}
with self.assertRaises(MissingOption):
perform_checks(config_fixture)
# more tests of more cases
You can also override the setUp() method of the unittest class if you want to share fixtures among tests. One way to do this would be set up a valid fixture, then make the invalidating changes you want to test in each test method.

How to stop cherrypy auto-reload from changing the process name on Debian?

I am developing a project using cherrypy, on debian. At my work, the administrators want to see the name of the project instead of "python" displayed when using commands like ps -e. However, when cherrypy auto-reloads when modifying one source file, it automatically changes the process name.
For example, if I take the most basic cherrypy tutorial and save it under NameToSee.py:
#!/usr/bin/python
import cherrypy
class HelloWorld(object):
#cherrypy.expose
def index(self):
return "Hello world!"
if __name__ == '__main__':
cherrypy.quickstart(HelloWorld())
By adding the shebang at the start, when I launch it $ ./NameToSee.py &, I get a process (say 31051) whose name is "NameToSee.py":
$ head /proc/31051/status
Name: NameToSee.py
State: S (sleeping)
However, whenever I change the source code files (for example, by adding an empty line), the process name changes:
$ head /proc/31051/status
Name: python
State: S (sleeping)
So, my question is: Can I get both cherrypy auto-reload and custom process name ? If not, can I remove the cherrypy auto-reload?
I am running on debian wheezy, with python 2.7.3 and cherrypy 3.2.2

This sample covers both cases:
import cherrypy
from cherrypy.process.plugins import SimplePlugin
PROC_NAME = 'sample'
def set_proc_name(newname):
"""
Set the process name.
Source: http://stackoverflow.com/a/923034/298371
"""
from ctypes import cdll, byref, create_string_buffer
libc = cdll.LoadLibrary('libc.so.6')
buff = create_string_buffer(len(newname)+1)
buff.value = newname
libc.prctl(15, byref(buff), 0, 0, 0)
class NamedProcess(SimplePlugin):
"""
Set the name of the process everytime that the
engine starts.
"""
def start(self):
self.bus.log("Setting the name as '{}'".format(PROC_NAME))
set_proc_name(PROC_NAME)
class HelloWorld(object):
#cherrypy.expose
def index(self):
return "Hello world!"
def run_without_autoreload():
set_proc_name(PROC_NAME)
cherrypy.quickstart(HelloWorld(), config={
'global': {
'engine.autoreload.on': False
}
})
def run_with_autoreload():
# Work on any configuration but for the sake of the
# question this works with the autoreload.
NamedProcess(cherrypy.engine).subscribe()
cherrypy.quickstart(HelloWorld())
if __name__ == '__main__':
run_with_autoreload()
# run_without_autoreload()
You can test it with:
cat /proc/`pgrep sample`/status
(or maybe just use pgrep)
And as a final recommendation consider using the "production" environment (see the docs) which also included the disabling of the autoreload plugin.

Python CherryPy server, what does cherrypy.Application do?

I have here a Python program.
It uses CherryPy to create a server.
# coding:utf-8
import os.path
import cherrypy
from app import application
def main():
try:
currentDir_s = os.path.dirname(os.path.abspath(__file__))
except:
currentDir_s = os.path.dirname(os.path.abspath(sys.executable))
cherrypy.Application.currentDir_s = currentDir_s
configFileName_s = 'server.conf'
if os.path.exists(configFileName_s) == False:
configFileName_s = None
cherrypy.engine.autoreload.unsubscribe()
cherrypy.engine.timeout_monitor.unsubscribe()
cherrypy.quickstart(application.Application_cl(), config=configFileName_s)
if __name__ == '__main__':
main()
And in "server.conf" it configure the server:
[global]
tools.log_headers.on: True
tools.sessions.on: False
tools.encode.on: True
tools.encode.encoding:"utf-8"
server.socket_port: 8080
server.socket_timeout:60
server.thread_pool: 10
server.environment: "production"
log.screen: True
[/]
tools.staticdir.root: cherrypy.Application.currentDir_s
tools.staticdir.on = True
tools.staticdir.dir = '.'
There is one thing, I don't understand, this line (line 13 in the python code):
cherrypy.Application.currentDir_s = currentDir_s
I searched in the internet about this, but I couldn't find anything. What does "cherrypy.Application" do? Why I have to do this assignment (cherrypy.Application.currentDir_s = currentDir_s)?

I searched the cherrypy source code and here is what I found.
In _cptree.py module you will find the Application class. Below that, there is a Tree class which has mount method which we are used to binding applications with (e.g. cherrypy.tree.mount(Root(), "/", config=config))
def mount(self, root, script_name="", config=None):
...
When you look inside this method you will see the code below;
def mount(self, root, script_name="", config=None):
...
if isinstance(root, Application):
app = root
if script_name != "" and script_name != app.script_name:
raise ValueError(
"Cannot specify a different script name and pass an "
"Application instance to cherrypy.mount")
script_name = app.script_name
else:
app = Application(root, script_name)
# If mounted at "", add favicon.ico
if (script_name == "" and root is not None
and not hasattr(root, "favicon_ico")):
favicon = os.path.join(os.getcwd(), os.path.dirname(__file__),
"favicon.ico")
root.favicon_ico = tools.staticfile.handler(favicon)
if config:
app.merge(config)
self.apps[script_name] = app
So, the code says that every object (application) you pass to the mount method is either Application instance or wrapped in an Application instance. So why is that so? When you check Application class above Tree class, you will see a __call__ method like below;
def __call__(self, environ, start_response):
return self.wsgiapp(environ, start_response)
Yes you see it now, it is wsgi interface.
Therefore, Application is a wsgi wrapper for your cherrypy applications.
When you check the source code of cherrypy you may learn lots of things. I hope this answer help you.

pelican blog post not getting generated

I am trying to post to my pelican blog from another python app, so I am not doing pelican ./output -s settings.py from the commandline.
I have modified pelican to accept mocked argparse like object to pass what it needs, so I have moved the content of the main function in __init__.py
to function named runPelican(args) that accepts args, and from my app that has mocked Argparse like this,
class MockArgparse(object):
"""Mock for argparse's to pass to pelican
"""
def __init__(self, verbosity=True, theme=None, output=None, path=None, delete_outputdir=None,
settings=None, ignore_cache=None, cache_path=None, selected_paths=None, autoreload=None):
"""
Args:
path (str): content path
settings(str): settings python file path
"""
super(MockArgparse, self).__init__()
self.theme = theme
self.cache_path = cache_path
self.ignore_cache = ignore_cache
self.delete_outputdir = delete_outputdir
self.settings = settings
self.output = output
self.verbosity = verbosity
self.autoreload = autoreload
self.path = path
self.selected_paths = selected_paths
I am calling the runPelican from my python app like this:
if make_entry(args):
import pelican
arg = MockArgparse(path=CONTENT_PATH, theme=THEME_PATH, output=OUTPUT_PATH, settings=SETTINGS_PATH)
pelican.runPelican(arg)
Everything seems to go fine but the blog posts are not generated
and Only error I am getting is
CRITICAL: SimplerXMLGenerator instance has no attribute '_write'
any help would be greatly appreciated.

When is sys.stdin None in Python?

I'm trying to run Flask as a simple CGI app through IIS.
I have the following code:
from wsgiref.handlers import CGIHandler
from flask import Flask
app = Flask(__name__)
#app.route('/')
def main():
return 'Woo woo!'
CGIHandler().run(app)
I'm runing Python 3.3 on Windows. I get the following error:
File "C:\Python33\lib\wsgiref\handlers.py",
line 509, in __init__(self, sys.stdin.buffer, sys.stdout.buffer, sys.stderr, )
AttributeError: 'NoneType' object has no attribute 'buffer' ".
I added some logging code, and it turns out that sys.stdin is None.
Python is added to IIS as a CGI Handler as follows:
Request path: *.py
Executable: C:\Windows\py.exe -3 %s %s
So, why is sys.stdin None, and how can I fix it?
EDIT
It looks like sys.stdin is None because the file descriptor is invalid.

Interesting. You've answered half your own question. The other half ("how do I fix it") is easy enough, just open some suitable thing (os.devnull is the obvious one) and set sys.stdin to point there. You'll need to do sys.stdout and sys.stderr as well, presumably, so something like this:
import os, sys
for _name in ('stdin', 'stdout', 'stderr'):
if getattr(sys, _name) is None:
setattr(sys, _name, open(os.devnull, 'r' if _name == 'stdin' else 'w'))
del _name # clean up this module's name space a little (optional)
from wsgiref.handlers ...
should do the trick.