Develop Reference

Does pytest import modules twice?

I have a helper class in a test module that has a class-level member in which I cache already-created members of the class (sql dumps of database configurations composed in fixtures, so that I don't have to derive the data again for multiple tests).
It starts:
class SqlDump:
FIXUP = re.compile(r"^(\s*CREATE SEQUENCE[^;]*)AS INTEGER([^;]*);",
flags=re.MULTILINE | re.IGNORECASE | re.DOTALL)
PATH = os.path.join(os.path.dirname(__file__), 'test_data/sql_dumps/{script}.sql')
all = {}
def __init__(self, script):
self.__class__.all[script] = self
self.script = script
self.content = self.load()
If I place a breakpoint on the initialization of this member all, using it outside pytest, it is initialized only once.
But when I run pytest, the line that initializes the member is executed twice. This results in some values being lost.
Is there ever any reason a class-level member should be initialized twice? Why is pytest doing this?

This is a year old now, but in case it helps someone else:
There was a very similar issue in my case, where a module was getting rerun over and over from pytest. This particular module (SQLAlchemy) is highly sensitive to reinitializations, resulting in an opaque error of Multiple classes found for path in the registry of this declarative base. This did not occur during normal runs of the platform - only when pytest was run.
Here's how the project was structured:
ProjectName
│ __init__.py
│ script_outside_modules.py
│
└───actual_module
│ __init__.py
│ pytest.ini
│ some_code.py
│
└───some_subfolder
│ │ __init__.py
│ │ a_file_to_test.py
│
└───tests
│ __init__.py
│ test_suite.py
All imports were absolute from the actual_module root, e.g. actual_module.some_code.
If you want to triage exactly how sys sees your module imports, and whether the same module was imported such that it's seen as two different modules, try using the following code in a module which you believe could be getting double-imported, outside of any encapsulation (e.g. above class SqlDump in your example):
import sys
import traceback
print(f"{__file__} imported by module {__name__}, in sys.modules as {sys.modules[__name__]}")
try:
if hasattr(sys, 'double_import'):
raise AssertionError(f"{__file__} was double-imported!")
except Exception as e:
traceback.print_exc()
raise e
sys.double_import = 1
Reading what it's registered as in sys.modules should help you identify where the disconnect is happening and whether you have odd module import scenarios playing out.
After hours of investigating possible causes and logging, I found that the extra import was due to the __init__.py at the root of the project, inside ProjectName in this case. The code above helped to illustrate this, where sys.modules showed an module for actual_module.some_code during preparation phases, but then showed another module at ProjectName.actual_module.some_code within the tests themselves.
This seems to be because pytest will identify a root separately from whatever is defined in imports and prepend it when running tests (though that's just a working hypothesis). Deleting ProjectName/__init__.py resolved my issue.

Execute some code before Pytest runs

I'm using Pytest to test my code and I'm running into a little, but an infuriating issue.
One of the first things my program does is checking if there are any setting files available. If there aren't any it throws an error and calls exit(). This works well during normal runtime but messes with Pytest.
The solution I came up with is to simply create a temporary settings file for the duration of the tests, by copying the template settings file. I've already written and successfully tested the code to achieve that.
The problem I'm running into is that I can't find a Pytest hook that truly fires before everything else. This results in the program throwing the error before Pytest tries to create a temporary settings file. Which results in Pytest failing before it can perform any tests.
Does anyone know of a way to fire a function before Pytest does or loads anything? Preferably within Pytest itself.
Some code context:
Throwing the error and exit
This snippet runs on the import of the settings module.
if len(cycles) == 0:
log.error("No setting files found. Please create a file " +
"in the `settings` folder using the Template.py.")
exit()
Creating the temporary settings file
This code should be the very first thing Pytest runs.
def pytest_sessionstart(session):
""" Create a test settings file """
folderPath = os.path.dirname(os.path.abspath(__file__))
folderPath = os.path.split(folderPath)[0] + "/settings"
srcfile = folderPath + '/Template.py'
dstfile = folderPath + '/test.py'
shutil.copy(srcfile, dstfile)
Removing the temporary settings file
This code should be one of the last things Pytest runs.
def pytest_sessionfinish(session, exitstatus):
""" Delete the test settings file """
folderPath = os.path.dirname(os.path.abspath(__file__))
folderPath = os.path.split(folderPath)[0] + "/settings"
os.remove(folderPath + "/test.py")
Pytest output
With the call to exit() disabled, so you can see the execution order.
Lakitna$ pytest -v -s
No setting files found. Please create a file in the `settings` folder using the Template.py.
TEMPORARY SETTINGS FILE CREATED
========================= test session starts ==========================
platform darwin -- Python 3.6.4, pytest-3.3.1, py-1.5.2, pluggy-0.6.0 -- /usr/local/opt/python3/bin/python3.6
cachedir: .cache
rootdir: /Users/lakitna/Documents/Github/Onaeri-tradfri/Onaeri, inifile:
collected 24 items

Add conftest.py to your project with below code and add your code to method pytest_configure
def pytest_configure(config):
pass # your code goes here
# other config

First, pytest tries to make all imports as relative as possible: this means that if you keep your tests in package/tests, then even your conftest will be imported as package.test.conftest. To do that import, package.__init__ will be called (unfortunately for you).
In order to run run some configuration before, the tests need to be moved out of the package tree. I ran into just such a problem recently.
As shown in a pytest issue thread, pytest_configure is actually called before pytest_sessionstart, but both occur before test collection, so either should be fine.
# contents of tests/conftest
import os
from pathlib import Path
import shutil
import pytest # if necessary
# cannot import package here (and likely not in `tests.__init__` either)
folderPath = Path.cwd()
folderPath = folderpath / 'package' / 'settings'
srcfile = folderPath / 'Template.py'
dstfile = folderPath / 'test.py'
def pytest_configure(config):
""" Create a test settings file """
# can use `config` to use command line options
shutil.copy(srcfile, dstfile)
def pytest_unconfigure(config):
""" Delete the test settings file """
os.remove(dstfile)
This should work if you have your folders structured like this:
root_folder
├── package
│ ├── settings
│ │ └── Template.py
│ ├── __init__.py
│ └── module1.py
└── tests
├── __init__.py # Does not import package
├── conftest.py # Does not import package
└── test_module1.py # Can import package
pytest can then be invoked in the normal way: pytest -flags (or pytest tests -flags) when working from root_folder.
If you want to set up your settings file in different ways depending on custom command line flags to pytest, you can do that as well in conftest.py. They can then be accessed in pytest_configure as config.getoption(...).
Note: this syntax assumes Python 3.6+ with the use of Paths as arguments to os.remove and shutil.copy.

I've managed to find a solution.
After some additional research, I found out that Pytest preloads all modules. This means that you can never run code before a module import unless you can find a hook before the collection phase. There is no such hook as far as I know. I really wanted to make this work within Pytest, but it seems to be impossible.
Instead, I created a __main__.py file in my test folder with the following content:
import pytest
import os
import shutil
"""
Create a setting file for the test procedure
"""
folderPath = os.path.dirname(os.path.abspath(__file__))
folderPath = os.path.split(folderPath)[0] + "/settings"
srcfile = folderPath + '/Template.py'
dstfile = folderPath + '/test.py'
shutil.copy(srcfile, dstfile)
"""
Actually run pytest
"""
pytest.main()
"""
Remove the test settings file
"""
os.remove(dstfile)
This code creates a temporary settings file, starts Pytest, and then removes the temporary file again.
I can now run the test procedure as follows:
$ python3 test
Pytest flags still work as normal. For example, if you want more verbose output you can do the following:
$ python3 test -v

This deviates from what you are looking for, but I would question your strategy. I suspect your code is not test friendly. The available hooks should suffice for setting up test pre-conditions like that. That is if your tests need some file, somehow setup that file first in setup, delete them with teardown, or use fixtures as per the pytest docs.
If you don't actually need the file for the tests to run, except that the code tests its existence, you can fake its existence with mocks.
But you have code running on module import that is testing (without actually being test code) a pre-condition and with user interaction.
And I think you shouldn't do this.
Remove the execution of that code running with import (make it a function or class). And invoke it in some other initialising place of your code.
Setup the file (with pytest hooks). Test code that uses the settings file.
Teardown the file (with pytest hooks).
You can in alternative completely fake the file by overriding the data read with fake data, using some data struture and functions to read/write,
simulating the file.
Test the code that tests for the settings file existence (use mocks).
Test your other code methods.
Do not mix things. Try to write methods that do one thing. Test one thing at a time. Test your business code. User interaction is another thing. If you also want to test UI, don't do it at the same time (I mean in the same code).

To run some code before and after the actual code of your test, you can use pytest fixtures:
import pytest
#pytest.fixture
def context():
"""What to do before/after the test."""
print("Entering !")
yield
print("Exiting !")
def test_stuff(context):
"""The actual code of your test"""
print("Running the test")
This displays:
Entering !
Running the test
Exiting !
To use the fixture, just pass a parameter with the same name as input of the test function. The yield instruction acts like a separator between what's done before and after the test.

Can't import functions from different module in the same project

I have two scripts:
script1:
from test import script2
if __name__ == '__main__':
foo()
script2:
def foo():
print 'hello'
In the structure:
test:
│ A
│-----B:
│ script2.py
│script1.py
I am trying to call the function foo() from script2.py in script1.py.
I received the error:
This inspection detect names that should resolve but don't. Due to
dynamic dispatch and duck typing, this is possible in a limited but
useful number of cases. Top-level and class-level itesm are supported
better than instance items.
I read number of similar cases but they didn't help me:
Call a function from another file in Python
How to import a function from parent folder in python?
Import Script from a Parent Directory

For Python packages to work, you need __init__.py files in the folders. These can be empty.
Then, you are importing from a subfolder, so you need import script2 from A.B
At that point, you may use script2.foo()
Also worth mentioning that Python3 should be targeted for any new projects .

Python - why can I import modules without __init__.py at all?

I'm new to Python and I still can't get my head around why we need a __init__.py file to import modules. I have gone through other questions and answers, such as this.
What confuses me is that I can import my modules without __init__py, so why do I need it at all?
My example,
index.py
modules/
hello/
hello.py
HelloWorld.py
index.py,
import os
import sys
root = os.path.dirname(__file__)
sys.path.append(root + "/modules/hello")
# IMPORTS MODULES
from hello import hello
from HelloWorld import HelloWorld
def application(environ, start_response):
results = []
results.append(hello())
helloWorld = HelloWorld()
results.append(helloWorld.sayHello())
output = "<br/>".join(results)
response_body = output
status = '200 OK'
response_headers = [('Content-Type', 'text/html'),
('Content-Length', str(len(response_body)))]
start_response(status, response_headers)
return [response_body]
modules/hello/hello.py,
def hello():
return 'Hello World from hello.py!'
modules/hello/HelloWorld.py,
# define a class
class HelloWorld:
def __init__(self):
self.message = 'Hello World from HelloWorld.py!'
def sayHello(self):
return self.message
Result,
Hello World from hello.py!
Hello World from HelloWorld.py!
What it takes is just these two lines,
root = os.path.dirname(__file__)
sys.path.append(root + "/modules/hello")
Without any of __init__py. Can someone explain why it works in this way?
If __init__py is the proper way, what should I do/change in my code?

Based on this link: Since Python 3.3
Allowing implicit namespace packages means that the requirement to provide an __init__.py file can be dropped completely

__init__.py is for packages. A package contains a collection of related modules. If you just have a single module you want to use, you don't need to use __init__.py; just put the single .py file somewhere on the system path and you can import it.
The purpose of packages is not just to allow you to import the modules inside them. It's to group the modules together. The main benefit of this is that, if a module is inside a package, then that module can import other modules from the package using relative imports. If you have foo.py and bar.py in the same package, then foo can just do from . import bar. This makes intra-package imports more compact and easier to reorganize if you restructure the package or change its name.
Also, an obvious benefit is. . . if you make it a package, you don't have to do that sys.path stuff every time you want to import something from it.

I think that this might be due to Python version you are using. I did some experimentation and found out that having following structure:
jedrzej#jedrzej-UX303LB ~/temp $ tree .
.
├── main.py
└── packages
├── file.py
└── file.pyc
1 directory, 5 files
content of main.py:
import packages.file as p
p.fun()
and content of file.py:
import sys
def fun():
print(sys.path)
When I am executing main.py with Python 2.7.12 I get ImportError while execution of main.py with Python 3.5.2 simply works.
After adding __init__.py in packages directory, code works with both versions of Python.

Files named __init__.py are used to mark directories on disk as Python package directories. If you have the files
modules/spam/__init__.py
modules/spam/module.py
and modules is in your path, you can import the code in module.py as
import spam.module
or
from spam import module
If you remove the __init__.py file, Python will no longer look for submodules inside that directory, so attempts to import the module will fail.
The __init__.py file is usually empty, but can be used to export selected portions of the package under a more convenient name, hold convenience functions, etc. Given the example above, the contents of the init module can be accessed with
import spam
And finally here is what the official documentation has to say about this file:
The __init__.py files are required to make Python treat the
directories as containing packages; this is done to prevent
directories with a common name, such as string, from
unintentionally hiding valid modules that occur later on the
module search path. In the simplest case, __init__.py can just
be an empty file, but it can also execute initialization code
for the package or set the __all__ variable, described later.

I think this is a good 'answer' for what I didn't understand.
myMath/
__init__.py
adv/
__init__.py
sqrt.py
fib.py
add.py
subtract.py
multiply.py
divide.py
myMath/__init__.py
from add import add
from divide import division
from multiply import multiply
from subtract import subtract
from adv.fib import fibonacci
from adv.sqrt import squareroot
index.py
import sys
sys.path.append('C:\Users\mdriscoll\Documents')
import mymath
print mymath.add(4,5)
print mymath.division(4, 2)
print mymath.multiply(10, 5)
print mymath.fibonacci(8)
print mymath.squareroot(48)

Append different modules to the list by imp.load_module() will became a same modules

I write in Python 3.
I want to add plugins support to my program. I don't want to use heavy frameworks, so I deiced to write a minimal one by my self.
By the way, the plugins need to run by time. I don't always unload and load the plugin when re-run -- the plugins will lose all data.
My folder structs is here:
interfaces/
├── dummy
├── gmail
│   ├── __init__.py
│   └── __pycache__
│   └── __init__.cpython-33.pyc
└── hello
├── __init__.py
└── __pycache__
└── __init__.cpython-33.pyc
Then, I wrote a piece of code to load and execute the plugins:
#!/usr/bin/python3
import os
import imp
INTERFACES_FOLDER = './interfaces'
MAIN_MODULE = '__init__'
def search_plugins():
plugins = []
plugins_folders = os.listdir(INTERFACES_FOLDER)
for i in plugins_folders:
plugin_folder = os.path.join(INTERFACES_FOLDER, i)
if not os.path.isdir(plugin_folder):
continue
if not MAIN_MODULE + '.py' in os.listdir(plugin_folder):
continue
info = imp.find_module(MAIN_MODULE, [plugin_folder])
plugins.append({'name': i, 'info': info})
return plugins
def load_plugin(plugin):
return imp.load_module(MAIN_MODULE, *plugin["info"])
plugins_list = search_plugins()
plugins = []
for i in plugins_list:
module = load_plugin(i)
print(module)
plugins.append(module)
print(plugins)
Outputs:
# it works!
<module '__init__' from './interfaces/gmail/__init__.py'>
<module '__init__' from './interfaces/hello/__init__.py'>
# what's wrong?
[<module '__init__' from './interfaces/hello/__init__.py'>,
<module '__init__' from './interfaces/hello/__init__.py'>]
As you see, when I loaded a plugin, everything is working correctly. But when I append them to the a list, different modules will became a same one.
What's wrong?

You import different modules, but with the same name (they are all called "init"). Python will, when you import the second module notice that you have already imported a module called "init" and return that.
The module you should import is rather the "gmail" and "hello" modules. That might work.
However, I'd urge you to reconsider about writing your own plugin system. It really isn't that easy. A quick search finds several plugin systems, some slightweight, some not so much. Most have been abandoned, which is an indication that this isn't so easy as you might think.
The plugin system in Python that probably is the most widely used, also the most flexible and amongst the oldest, is the Zope Component Architecture. See also the docs. Note that in Python 3 you use class decorators instead of the class body statements.
Another popular plugin system is the "entry points" of Distribute. That's quite lightweight, although I don't know if you can load or unload plugins with that.
Others I have never looked at are Yapsy, one called "Plugins" that seem abandoned, another one called "PyPlugin" also abandoned, and one called "easy_plugins" which seem very new, and isn't abandoned yet.
And here is example code for a plugin manager.