How to integrate checking of readme in pytest - python

I use pytest in my .travis.yml to check my code.
I would like to check the README.rst, too.
I found readme_renderer via this StackO answer
Now I ask myself how to integrate this into my current tests.
The docs of readme_renderer suggest this, but I have not clue how to integrate this into my setup:
python setup.py check -r -s

I think the simplest and most robust option is to write a pytest plugin that replicates what the distutils command you mentioned in you answer does.
That could be as simple as a conftest.py in your test dir. Or if you want a standalone plugin that's distributable for all of us to benefit from there's a nice cookiecutter template.
Ofc there's inherently nothing wrong with calling the check manually in your script section after the call to pytest.

I check it like this now:
# -*- coding: utf-8 -*-
from __future__ import absolute_import, division, unicode_literals, print_function
import os
import subx
import unittest
class Test(unittest.TestCase):
def test_readme_rst_valid(self):
base_dir = os.path.dirname(os.path.dirname(os.path.dirname(__file__)))
subx.call(cmd=['python', os.path.join(base_dir, 'setup.py'), 'check', '--metadata', '--restructuredtext', '--strict'])
Source: https://github.com/guettli/reprec/blob/master/reprec/tests/test_setup.py

So I implemented something but it does require some modifications. You need to modify your setup.py as below
from distutils.core import setup
setup_info = dict(
name='so1',
version='',
packages=[''],
url='',
license='',
author='tarun.lalwani',
author_email='',
description=''
)
if __name__ == "__main__":
setup(**setup_info)
Then you need to create a symlink so we can import this package in the test
ln -s setup.py setup_mypackage.py
And then you can create a test like below
# -*- coding: utf-8 -*-
from __future__ import absolute_import, division, unicode_literals, print_function
import os
import unittest
from distutils.command.check import check
from distutils.dist import Distribution
import setup_mypackage
class Test(unittest.TestCase):
def test_readme_rst_valid(self):
dist = Distribution(setup_mypackage.setup_info)
test = check(dist)
test.ensure_finalized()
test.metadata = True
test.strict = True
test.restructuredtext = True
global issues
issues = []
def my_warn(msg):
global issues
issues += [msg]
test.warn = my_warn
test.check_metadata()
test.check_restructuredtext()
if len(issues) > 0:
assert len(issues) == 0, "\n".join(issues)
Running the test then I get
...
AssertionError: missing required meta-data: version, url
missing meta-data: if 'author' supplied, 'author_email' must be supplied too
Ran 1 test in 0.067s
FAILED (failures=1)
This is one possible workaround that I can think of

Upvoted because checking readme consistence is a nice thing I never integrated in my own projects. Will do from now on!
I think your approach with calling the check command is fine, although it will check more than readme's markup. check will validate the complete metadata of your package, including the readme if you have readme_renderer installed.
If you want to write a unit test that does only markup check and nothing else, I'd go with an explicit call of readme_renderer.rst.render:
import pathlib
from readme_renderer.rst import render
def test_markup_is_generated():
readme = pathlib.Path('README.rst')
assert render(readme.read_text()) is not None
The None check is the most basic test: if render returns None, it means that the readme contains errors preventing it from being translated to HTML. If you want more fine-grained tests, work with the HTML string returned. For example, I expect my readme to contain the word "extensions" to be emphasized:
import pathlib
import bs4
from readme_renderer.rst import render
def test_extensions_is_emphasized():
readme = pathlib.Path('README.rst')
html = render(readme.read_text())
soup = bs4.BeautifulSoup(html)
assert soup.find_all('em', string='extensions')
Edit: If you want to see the printed warnings, use the optional stream argument:
from io import StringIO
def test_markup_is_generated():
warnings = StringIO()
with open('README.rst') as f:
html = render(f.read(), stream=warnings)
warnings.seek(0)
assert html is not None, warnings.read()
Sample output:
tests/test_readme.py::test_markup_is_generated FAILED
================ FAILURES ================
________ test_markup_is_generated ________
def test_markup_is_generated():
warnings = StringIO()
with open('README.rst') as f:
html = render(f.read(), stream=warnings)
warnings.seek(0)
> assert html is not None, warnings.read()
E AssertionError: <string>:54: (WARNING/2) Title overline too short.
E
E ----
E fffffff
E ----
E
E assert None is not None
tests/test_readme.py:10: AssertionError
======== 1 failed in 0.26 seconds ========

Related

How to patch a module that hasn't been imported by parent package's __init__.py

I'm trying to test a tool I'm building which uses some jMetalPy functionality. I had/have a previous version working but I am now trying to refactor out some external dependencies (such as the aforementioned jMetalPy).
Project Code & Structure
Here is a minimalist structure of my project.
MyToolDirectory
¦--/MyTool
¦----/__init__.py
¦----/_jmetal
¦------/__init__.py
¦------/core
¦--------/quality_indicator.py
¦----/core
¦------/__init__.py
¦------/run_manager.py
¦----/tests
¦------/__init__.py
¦------/test_run_manager.py
The _jmetal directory is to remove external dependency on the jMetalPy package - and I have copied only the necessary packages/modules that I need.
Minimal contents of run_manager.py
# MyTool\core\run_manager.py
import jmetal
# from jmetal.core.quality_indicators import HyperVolume # old working version
class RunManager:
def __init__(self):
pass
#staticmethod
def calculate_hypervolume(front, ref_point):
if front is None or len(front) < 1:
return 0.
hv = jmetal.core.quality_indicator.HyperVolume(ref_point)
# hv = HyperVolume(ref_point)
hypervolume = hv.compute(front)
return hypervolume
Minimal contents of test_run_manager.py
# MyTool\tests\test_run_manager.py
import unittest
from unittest.mock import MagicMock, Mock, patch
from MyTool import core
class RunManagerTest(unittest.TestCase):
def setUp(self):
self.rm = core.RunManager()
def test_calculate_hypervolume(self):
ref_points = [0.0, 57.5]
front = [None, None]
# with patch('MyTool.core.run_manager.HyperVolume') as mock_HV: # old working version
with patch('MyTool.core.run_manager.jmetal.core.quality_indicator.HyperVolume') as mock_HV:
mock_HV.return_value = MagicMock()
res = self.rm.calculate_hypervolume(front, ref_points)
mock_HV.assert_called_with(ref_points)
mock_HV().compute.assert_called_with(front)
Main Question
When I run a test with the code as-is, I get this error message:
E ModuleNotFoundError: No module named 'MyTool.core.run_manager.jmetal'; 'MyTool.core.run_manager' is not a package
But when I change it to:
with patch('MyTool.core.run_manager.jmetal.core') as mock_core:
mock_HV = mock_core.quality_indicator.HyperVolume
mock_HV.return_value = MagicMock()
res = self.rm.calculate_hypervolume(front, ref_points)
mock_HV.assert_called_with(ref_points)
mock_HV().compute.assert_called_with(front)
... now the test passes. What gives?!
Why can't (or rather, how can) I surgically patch the exact class I want (i.e., HyperVolume) without patching out an entire sub-package as well? Is there a way around this? There may be code in jmetal.core that needs to run normally.
Is the reason this isn't working only because there is no from . import quality_indicator statement in jMetalPy's jmetal\core\__init__.py ?
Because even with patch('MyTool.core.run_manager.jmetal.core.quality_indicator) throws:
E AttributeError: <module 'jmetal.core' from 'path\\to\\venv\\lib\\site-packages\\jmetal\\core\\__init__.py'> does not have the attribute 'quality_indicator'
Or is there something I'm doing wrong?
In the case that it is just about adding those import statements, I could do that in my _jmetal sub-package, but I was hoping to let the user default to their own jMetalPy installation if they already had one by adding this to MyTool\__init__.py:
try:
import jmetal
except ModuleNotFoundError:
from . import _jmetal as jmetal
and then replacing all instances of import jmetal with from MyTool import jmetal. However, I'd run into the same problem all over again.
I feel that there is some core concept I am not grasping. Thanks for the help.

How do I structure a package so that its modules are callable?

I'm very new to Python and am learning how packages and modules work but have run into a snag. Initially I created a very simple package to be deployed as a Lambda function. I have a file in the root directory called lambda.py which contains a handler function, and I put most of my business logic in a separate file. I created a subdirectory - for this example let's say it's called testme - and more specifically put it in __init__.py underneath that subdirectory. Initially this worked great. Inside my lambda.py file I could use this import statement:
from testme import TestThing # TestThing is the name of the class
However, now that the code is growing, I've been splitting things into multiple files. As a result, my code no longer runs; I get the following error:
TypeError: 'module' object is not callable
Here's a simplified version of what my code looks like now, to illustrate the problem. What am I missing? What can I do to make these modules "callable"?
/lambda.py:
#!/usr/bin/env python2
# -*- coding: utf-8 -*-
from testme import TestThing
def handler(event, context):
abc = TestThing(event.get('value'))
abc.show_value()
if __name__ == '__main__':
handler({'value': 5}, None)
/testme/__init__.py:
#!/usr/bin/env python2
# -*- coding: utf-8 -*-
__project__ = "testme"
__version__ = "0.1.0"
__description__ = "Test MCVE"
__url__ = "https://stackoverflow.com"
__author__ = "soapergem"
__all__ = ["TestThing"]
/testme/TestThing.py:
#!/usr/bin/env python2
# -*- coding: utf-8 -*-
class TestThing:
def __init__(self, value):
self.value = value
def show_value(self):
print 'The value is %s' % self.value
Like I said, the reason I'm doing all this is because the real world example has enough code that I want to split it into multiple files inside the subdirectory. And so I left an __init__.py file there to just to serve as an index essentially. But I am very unsure of what the best practices are for package structure, or how to get this working.
You either have to import your class in your __init__ file:
testme/__init__.py:
from .TestThing import TestThing
or import it using the full path:
lambda.py:
from testme.TestThing import TestThing
When you use a __init__.py file, you create a package, and this package (named after the root directory, e.g. testme, may include submodules. Those are accessible via the package.module syntax, but the contents of the submodules are only visible in the root package if you explicitly import them there.

How do you set the results_directory in SST Python

I'm feeling a bit foolish asking this, as with basic selenium I have no problem saving screenshots, yet with SST I use the take_screenshot('screenshot_name.png') it tells me that the results_directory should be set. Question is how do you set the results_directory. All of the examples I find set it to "NONE", yet that doesn't satisfy my test's need.
Below is how my code is written:
import unittest
from sst.actions import *
from sst import cases, config
config.results_directory = None
class TestMyTest(cases.SSTTestCase):
def test_mytestcase_home_page(self):
go_to('http://www.mywebpage.com')
assert_title_contains('MyWebPage')
#Main page is displayed
take_screenshot(filename='C/Users/Brenda/test/SST Test Project/results/home_page.png',add_timestamp=True)
I had the following script working for me using Google. The trick was to add result directory to actual config file which is located #{dir}\Python27\Lib\site-packages\sst\config.py and add results_directory = "C:\Users\{me}\Desktop\Python-pip-SST\results"
import unittest
from sst.actions import *
from sst import cases, config
#config.results_directory = "C:\Users\{me}\Desktop\Python-pip-SST\results"
go_to('https://www.google.com/')
assert_title_contains('Google')
#Main page is displayed
take_screenshot(filename='home_page.png',add_timestamp=True)
And, You also should be able to overwrite the result path from your test. Your working code should look like something like the following
import unittest
from sst.actions import *
from sst import cases, config
#Just to be safe side try not to use any spaces in filename
config.results_directory = "C:/Users/Brenda/test/SSTTestProject/results"
class TestMyTest(cases.SSTTestCase):
def test_mytestcase_home_page(self):
go_to('http://www.mywebpage.com')
assert_title_contains('MyWebPage')
#Main page is displayed
take_screenshot(filename="home_page.png",add_timestamp=True)
I added the screenshot if that helps you somehow. Changing filepath innconfig file or from test works fine for me

How to run unittest discover from "python setup.py test"?

I'm trying to figure out how to get python setup.py test to run the equivalent of python -m unittest discover. I don't want to use a run_tests.py script and I don't want to use any external test tools (like nose or py.test). It's OK if the solution only works on python 2.7.
In setup.py, I think I need to add something to the test_suite and/or test_loader fields in config, but I can't seem to find a combination that works correctly:
config = {
'name': name,
'version': version,
'url': url,
'test_suite': '???',
'test_loader': '???',
}
Is this possible using only unittest built into python 2.7?
FYI, my project structure looks like this:
project/
package/
__init__.py
module.py
tests/
__init__.py
test_module.py
run_tests.py <- I want to delete this
setup.py
Update: This is possible with unittest2 but I want find something equivalent using only unittest
From https://pypi.python.org/pypi/unittest2
unittest2 includes a very basic setuptools compatible test collector. Specify test_suite = 'unittest2.collector' in your setup.py. This starts test discovery with the default parameters from the directory containing setup.py, so it is perhaps most useful as an example (see unittest2/collector.py).
For now, I'm just using a script called run_tests.py, but I'm hoping I can get rid of this by moving to a solution that only uses python setup.py test.
Here's the run_tests.py I'm hoping to remove:
import unittest
if __name__ == '__main__':
# use the default shared TestLoader instance
test_loader = unittest.defaultTestLoader
# use the basic test runner that outputs to sys.stderr
test_runner = unittest.TextTestRunner()
# automatically discover all tests in the current dir of the form test*.py
# NOTE: only works for python 2.7 and later
test_suite = test_loader.discover('.')
# run the test suite
test_runner.run(test_suite)
If you use py27+ or py32+, the solution is pretty simple:
test_suite="tests",
From Building and Distributing Packages with Setuptools (emphasis mine):
test_suite
A string naming a unittest.TestCase subclass (or a package or module
containing one or more of them, or a method of such a subclass), or naming
a function that can be called with no arguments and returns a unittest.TestSuite.
Hence, in setup.py you would add a function that returns a TestSuite:
import unittest
def my_test_suite():
test_loader = unittest.TestLoader()
test_suite = test_loader.discover('tests', pattern='test_*.py')
return test_suite
Then, you would specify the command setup as follows:
setup(
...
test_suite='setup.my_test_suite',
...
)
You don't need config to get this working. There are basically two main ways to do it:
The quick way
Rename your test_module.py to module_test.py (basically add _test as a suffix to tests for a particular module), and python will find it automatically. Just make sure to add this to setup.py:
from setuptools import setup, find_packages
setup(
...
test_suite = 'tests',
...
)
The long way
Here's how to do it with your current directory structure:
project/
package/
__init__.py
module.py
tests/
__init__.py
test_module.py
run_tests.py <- I want to delete this
setup.py
Under tests/__init__.py, you want to import the unittest and your unit test script test_module, and then create a function to run the tests. In tests/__init__.py, type in something like this:
import unittest
import test_module
def my_module_suite():
loader = unittest.TestLoader()
suite = loader.loadTestsFromModule(test_module)
return suite
The TestLoader class has other functions besides loadTestsFromModule. You can run dir(unittest.TestLoader) to see the other ones, but this one is the simplest to use.
Since your directory structure is such, you'll probably want the test_module to be able to import your module script. You might have already done this, but just in case you didn't, you could include the parent path so that you can import the package module and the module script. At the top of your test_module.py, type:
import os, sys
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
import unittest
import package.module
...
Then finally, in setup.py, include the tests module and run the command you created, my_module_suite:
from setuptools import setup, find_packages
setup(
...
test_suite = 'tests.my_module_suite',
...
)
Then you just run python setup.py test.
Here is a sample someone made as a reference.
One possible solution is to simply extend the test command for distutilsand setuptools/distribute. This seems like a total kluge and way more complicated than I would prefer, but seems to correctly discover and run all the tests in my package upon running python setup.py test. I'm holding off on selecting this as the answer to my question in hopes that someone will provide a more elegant solution :)
(Inspired by https://docs.pytest.org/en/latest/goodpractices.html#integrating-with-setuptools-python-setup-py-test-pytest-runner)
Example setup.py:
try:
from setuptools import setup
except ImportError:
from distutils.core import setup
def discover_and_run_tests():
import os
import sys
import unittest
# get setup.py directory
setup_file = sys.modules['__main__'].__file__
setup_dir = os.path.abspath(os.path.dirname(setup_file))
# use the default shared TestLoader instance
test_loader = unittest.defaultTestLoader
# use the basic test runner that outputs to sys.stderr
test_runner = unittest.TextTestRunner()
# automatically discover all tests
# NOTE: only works for python 2.7 and later
test_suite = test_loader.discover(setup_dir)
# run the test suite
test_runner.run(test_suite)
try:
from setuptools.command.test import test
class DiscoverTest(test):
def finalize_options(self):
test.finalize_options(self)
self.test_args = []
self.test_suite = True
def run_tests(self):
discover_and_run_tests()
except ImportError:
from distutils.core import Command
class DiscoverTest(Command):
user_options = []
def initialize_options(self):
pass
def finalize_options(self):
pass
def run(self):
discover_and_run_tests()
config = {
'name': 'name',
'version': 'version',
'url': 'http://example.com',
'cmdclass': {'test': DiscoverTest},
}
setup(**config)
Another less than ideal solution slightly inspired by http://hg.python.org/unittest2/file/2b6411b9a838/unittest2/collector.py
Add a module that returns a TestSuite of discovered tests. Then configure setup to call that module.
project/
package/
__init__.py
module.py
tests/
__init__.py
test_module.py
discover_tests.py
setup.py
Here's discover_tests.py:
import os
import sys
import unittest
def additional_tests():
setup_file = sys.modules['__main__'].__file__
setup_dir = os.path.abspath(os.path.dirname(setup_file))
return unittest.defaultTestLoader.discover(setup_dir)
And here's setup.py:
try:
from setuptools import setup
except ImportError:
from distutils.core import setup
config = {
'name': 'name',
'version': 'version',
'url': 'http://example.com',
'test_suite': 'discover_tests',
}
setup(**config)
Python's standard library unittest module supports discovery (in Python 2.7 and later, and Python 3.2 and later). If you can assume those minimum versions, then you can just add the discover command line argument to the unittest command.
Only a small tweak is needed to setup.py:
import setuptools.command.test
from setuptools import (find_packages, setup)
class TestCommand(setuptools.command.test.test):
""" Setuptools test command explicitly using test discovery. """
def _test_args(self):
yield 'discover'
for arg in super(TestCommand, self)._test_args():
yield arg
setup(
...
cmdclass={
'test': TestCommand,
},
)
This won't remove run_tests.py, but will make it work with setuptools. Add:
class Loader(unittest.TestLoader):
def loadTestsFromNames(self, names, _=None):
return self.discover(names[0])
Then in setup.py: (I assume you're doing something like setup(**config))
config = {
...
'test_loader': 'run_tests:Loader',
'test_suite': '.', # your start_dir for discover()
}
The only downside I see is it's bending the semantics of loadTestsFromNames, but the setuptools test command is the only consumer, and calls it in a specified way.

How can I get the version defined in setup.py (setuptools) in my package?

How could I get the version defined in setup.py from my package (for --version, or other purposes)?
Interrogate version string of already-installed distribution
To retrieve the version from inside your package at runtime (what your question appears to actually be asking), you can use:
import pkg_resources # part of setuptools
version = pkg_resources.require("MyProject")[0].version
Store version string for use during install
If you want to go the other way 'round (which appears to be what other answer authors here appear to have thought you were asking), put the version string in a separate file and read that file's contents in setup.py.
You could make a version.py in your package with a __version__ line, then read it from setup.py using execfile('mypackage/version.py'), so that it sets __version__ in the setup.py namespace.
Warning about race condition during install
By the way, DO NOT import your package from your setup.py as suggested in another answer here: it will seem to work for you (because you already have your package's dependencies installed), but it will wreak havoc upon new users of your package, as they will not be able to install your package without manually installing the dependencies first.
example study: mymodule
Imagine this configuration:
setup.py
mymodule/
/ __init__.py
/ version.py
/ myclasses.py
Then imagine some usual scenario where you have dependencies and setup.py looks like:
setup(...
install_requires=['dep1','dep2', ...]
...)
And an example __init__.py:
from mymodule.myclasses import *
from mymodule.version import __version__
And for example myclasses.py:
# these are not installed on your system.
# importing mymodule.myclasses would give ImportError
import dep1
import dep2
problem #1: importing mymodule during setup
If your setup.py imports mymodule then during setup you would most likely get an ImportError. This is a very common error when your package has dependencies. If your package does not have other dependencies than the builtins, you may be safe; however this isn't a good practice. The reason for that is that it is not future-proof; say tomorrow your code needs to consume some other dependency.
problem #2: where's my __version__ ?
If you hardcode __version__ in setup.py then it may not match the version that you would ship in your module. To be consistent, you would put it in one place and read it from the same place when you need it. Using import you may get the problem #1.
solution: à la setuptools
You would use a combination of open, exec and provide a dict for exec to add variables:
# setup.py
from setuptools import setup, find_packages
from distutils.util import convert_path
main_ns = {}
ver_path = convert_path('mymodule/version.py')
with open(ver_path) as ver_file:
exec(ver_file.read(), main_ns)
setup(...,
version=main_ns['__version__'],
...)
And in mymodule/version.py expose the version:
__version__ = 'some.semantic.version'
This way, the version is shipped with the module, and you do not have issues during setup trying to import a module that has missing dependencies (yet to be installed).
The best technique is to define __version__ in your product code, then import it into setup.py from there. This gives you a value you can read in your running module, and have only one place to define it.
The values in setup.py are not installed, and setup.py doesn't stick around after installation.
What I did (for example) in coverage.py:
# coverage/__init__.py
__version__ = "3.2"
# setup.py
from coverage import __version__
setup(
name = 'coverage',
version = __version__,
...
)
UPDATE (2017): coverage.py no longer imports itself to get the version. Importing your own code can make it uninstallable, because you product code will try to import dependencies, which aren't installed yet, because setup.py is what installs them.
Your question is a little vague, but I think what you are asking is how to specify it.
You need to define __version__ like so:
__version__ = '1.4.4'
And then you can confirm that setup.py knows about the version you just specified:
% ./setup.py --version
1.4.4
I wasn't happy with these answers... didn't want to require setuptools, nor make a whole separate module for a single variable, so I came up with these.
For when you are sure the main module is in pep8 style and will stay that way:
version = '0.30.unknown'
with file('mypkg/mymod.py') as f:
for line in f:
if line.startswith('__version__'):
_, _, version = line.replace("'", '').split()
break
If you'd like to be extra careful and use a real parser:
import ast
version = '0.30.unknown2'
with file('mypkg/mymod.py') as f:
for line in f:
if line.startswith('__version__'):
version = ast.parse(line).body[0].value.s
break
setup.py is somewhat of a throwaway module so not an issue if it is a bit ugly.
Update: funny enough I've moved away from this in recent years and started using a separate file in the package called meta.py. I put lots of meta data in there that I might want to change frequently. So, not just for one value.
With a structure like this:
setup.py
mymodule/
/ __init__.py
/ version.py
/ myclasses.py
where version.py contains:
__version__ = 'version_string'
You can do this in setup.py:
import sys
sys.path[0:0] = ['mymodule']
from version import __version__
This won't cause any problem with whatever dependencies you have in your mymodule/__init__.py
Create a file in your source tree, e.g. in yourbasedir/yourpackage/_version.py . Let that file contain only a single line of code, like this:
__version__ = "1.1.0-r4704"
Then in your setup.py, open that file and parse out the version number like this:
verstr = "unknown"
try:
verstrline = open('yourpackage/_version.py', "rt").read()
except EnvironmentError:
pass # Okay, there is no version file.
else:
VSRE = r"^__version__ = ['\"]([^'\"]*)['\"]"
mo = re.search(VSRE, verstrline, re.M)
if mo:
verstr = mo.group(1)
else:
raise RuntimeError("unable to find version in yourpackage/_version.py")
Finally, in yourbasedir/yourpackage/__init__.py import _version like this:
__version__ = "unknown"
try:
from _version import __version__
except ImportError:
# We're running in a tree that doesn't have a _version.py, so we don't know what our version is.
pass
An example of code that does this is the "pyutil" package that I maintain. (See PyPI or google search -- stackoverflow is disallowing me from including a hyperlink to it in this answer.)
#pjeby is right that you shouldn't import your package from its own setup.py. That will work when you test it by creating a new Python interpreter and executing setup.py in it first thing: python setup.py, but there are cases when it won't work. That's because import youpackage doesn't mean to read the current working directory for a directory named "yourpackage", it means to look in the current sys.modules for a key "yourpackage" and then to do various things if it isn't there. So it always works when you do python setup.py because you have a fresh, empty sys.modules, but this doesn't work in general.
For example, what if py2exe is executing your setup.py as part of the process of packaging up an application? I've seen a case like this where py2exe would put the wrong version number on a package because the package was getting its version number from import myownthing in its setup.py, but a different version of that package had previously been imported during the py2exe run. Likewise, what if setuptools, easy_install, distribute, or distutils2 is trying to build your package as part of a process of installing a different package that depends on yours? Then whether your package is importable at the time that its setup.py is being evaluated, or whether there is already a version of your package that has been imported during this Python interpreter's life, or whether importing your package requires other packages to be installed first, or has side-effects, can change the results. I've had several struggles with trying to re-use Python packages which caused problems for tools like py2exe and setuptools because their setup.py imports the package itself in order to find its version number.
By the way, this technique plays nicely with tools to automatically create the yourpackage/_version.py file for you, for example by reading your revision control history and writing out a version number based on the most recent tag in revision control history. Here is a tool that does that for darcs: http://tahoe-lafs.org/trac/darcsver/browser/trunk/README.rst and here is a code snippet which does the same thing for git: http://github.com/warner/python-ecdsa/blob/0ed702a9d4057ecf33eea969b8cf280eaccd89a1/setup.py#L34
We wanted to put the meta information about our package pypackagery in __init__.py, but could not since it has third-party dependencies as PJ Eby already pointed out (see his answer and the warning regarding the race condition).
We solved it by creating a separate module pypackagery_meta.py that contains only the meta information:
"""Define meta information about pypackagery package."""
__title__ = 'pypackagery'
__description__ = ('Package a subset of a monorepo and '
'determine the dependent packages.')
__url__ = 'https://github.com/Parquery/pypackagery'
__version__ = '1.0.0'
__author__ = 'Marko Ristin'
__author_email__ = 'marko.ristin#gmail.com'
__license__ = 'MIT'
__copyright__ = 'Copyright 2018 Parquery AG'
then imported the meta information in packagery/__init__.py:
# ...
from pypackagery_meta import __title__, __description__, __url__, \
__version__, __author__, __author_email__, \
__license__, __copyright__
# ...
and finally used it in setup.py:
import pypackagery_meta
setup(
name=pypackagery_meta.__title__,
version=pypackagery_meta.__version__,
description=pypackagery_meta.__description__,
long_description=long_description,
url=pypackagery_meta.__url__,
author=pypackagery_meta.__author__,
author_email=pypackagery_meta.__author_email__,
# ...
py_modules=['packagery', 'pypackagery_meta'],
)
You must include pypackagery_meta into your package with py_modules setup argument. Otherwise, you can not import it upon installation since the packaged distribution would lack it.
This should also work, using regular expressions and depending on the metadata fields to have a format like this:
__fieldname__ = 'value'
Use the following at the beginning of your setup.py:
import re
main_py = open('yourmodule.py').read()
metadata = dict(re.findall("__([a-z]+)__ = '([^']+)'", main_py))
After that, you can use the metadata in your script like this:
print 'Author is:', metadata['author']
print 'Version is:', metadata['version']
Simple and straight, create a file called source/package_name/version.py with the following contents:
#!/usr/bin/env python3
# -*- coding: UTF-8 -*-
__version__ = "2.6.9"
Then, on your file source/package_name/__init__.py, you import the version for other people to use:
#!/usr/bin/env python3
# -*- coding: UTF-8 -*-
from .version import __version__
Now, you can put this on setup.py
#!/usr/bin/env python3
# -*- coding: UTF-8 -*-
import re
import sys
try:
filepath = 'source/package_name/version.py'
version_file = open( filepath )
__version__ ,= re.findall( '__version__ = "(.*)"', version_file.read() )
except Exception as error:
__version__ = "0.0.1"
sys.stderr.write( "Warning: Could not open '%s' due %s\n" % ( filepath, error ) )
finally:
version_file.close()
Tested this with Python 2.7, 3.3, 3.4, 3.5, 3.6 and 3.7 on Linux, Windows and Mac OS. I used on my package which has Integration and Unit Tests for all theses platforms. You can see the results from .travis.yml and appveyor.yml here:
https://travis-ci.org/evandrocoan/debugtools/builds/527110800
https://ci.appveyor.com/project/evandrocoan/pythondebugtools/builds/24245446
An alternate version is using context manager:
#!/usr/bin/env python3
# -*- coding: UTF-8 -*-
import re
import sys
try:
filepath = 'source/package_name/version.py'
with open( filepath ) as file:
__version__ ,= re.findall( '__version__ = "(.*)"', file.read() )
except Exception as error:
__version__ = "0.0.1"
sys.stderr.write( "Warning: Could not open '%s' due %s\n" % ( filepath, error ) )
You can also be using the codecs module to handle unicode errors both on Python 2.7 and 3.6
#!/usr/bin/env python3
# -*- coding: UTF-8 -*-
import re
import sys
import codecs
try:
filepath = 'source/package_name/version.py'
with codecs.open( filepath, 'r', errors='ignore' ) as file:
__version__ ,= re.findall( '__version__ = "(.*)"', file.read() )
except Exception as error:
__version__ = "0.0.1"
sys.stderr.write( "Warning: Could not open '%s' due %s\n" % ( filepath, error ) )
If you are writing a Python module 100% in C/C++ using Python C Extensions, you can do the same thing, but using C/C++ instead of Python.
On this case, create the following setup.py:
#!/usr/bin/env python3
# -*- coding: UTF-8 -*-
import re
import sys
import codecs
from setuptools import setup, Extension
try:
filepath = 'source/version.h'
with codecs.open( filepath, 'r', errors='ignore' ) as file:
__version__ ,= re.findall( '__version__ = "(.*)"', file.read() )
except Exception as error:
__version__ = "0.0.1"
sys.stderr.write( "Warning: Could not open '%s' due %s\n" % ( filepath, error ) )
setup(
name = 'package_name',
version = __version__,
package_data = {
'': [ '**.txt', '**.md', '**.py', '**.h', '**.hpp', '**.c', '**.cpp' ],
},
ext_modules = [
Extension(
name = 'package_name',
sources = [
'source/file.cpp',
],
include_dirs = ['source'],
)
],
)
Which reads the version from the file version.h:
const char* __version__ = "1.0.12";
But, do not forget to create the MANIFEST.in to include the version.h file:
include README.md
include LICENSE.txt
recursive-include source *.h
And it is integrated into the main application with:
#include <Python.h>
#include "version.h"
// create the module
PyMODINIT_FUNC PyInit_package_name(void)
{
PyObject* thismodule;
...
// https://docs.python.org/3/c-api/arg.html#c.Py_BuildValue
PyObject_SetAttrString( thismodule, "__version__", Py_BuildValue( "s", __version__ ) );
...
}
References:
python open file error
Define a global in a Python module from a C API
How to include package data with setuptools/distribute?
https://github.com/lark-parser/lark/blob/master/setup.py#L4
How to use setuptools packages and ext_modules with the same name?
Is it possible to include subdirectories using dist utils (setup.py) as part of package data?
To avoid importing a file (and thus executing its code) one could parse it and recover the version attribute from the syntax tree:
# assuming 'path' holds the path to the file
import ast
with open(path, 'rU') as file:
t = compile(file.read(), path, 'exec', ast.PyCF_ONLY_AST)
for node in (n for n in t.body if isinstance(n, ast.Assign)):
if len(node.targets) == 1:
name = node.targets[0]
if isinstance(name, ast.Name) and \
name.id in ('__version__', '__version_info__', 'VERSION'):
v = node.value
if isinstance(v, ast.Str):
version = v.s
break
if isinstance(v, ast.Tuple):
r = []
for e in v.elts:
if isinstance(e, ast.Str):
r.append(e.s)
elif isinstance(e, ast.Num):
r.append(str(e.n))
version = '.'.join(r)
break
This code tries to find the __version__ or VERSION assignment at the top level of the module return is string value. The right side can be either a string or a tuple.
There's a thousand ways to skin a cat -- here's mine:
# Copied from (and hacked):
# https://github.com/pypa/virtualenv/blob/develop/setup.py#L42
def get_version(filename):
import os
import re
here = os.path.dirname(os.path.abspath(__file__))
f = open(os.path.join(here, filename))
version_file = f.read()
f.close()
version_match = re.search(r"^__version__ = ['\"]([^'\"]*)['\"]",
version_file, re.M)
if version_match:
return version_match.group(1)
raise RuntimeError("Unable to find version string.")
A lot of the other answers are outdated, I believe the standard way to get version information from an installed python 3.10 package is by using importlib.metadata as of PEP-0566
Official docs: https://docs.python.org/3.10/library/importlib.metadata.html
from importlib.metadata import version
VERSION_NUM = version("InstalledPackageName")
This is simple, clean, and no fuss.
This won't work if you are doing something weird in a script that runs during package installation, but if all you are doing is getting the version number for a version check to show the user in through a CLI --help command, about box, or anything else where your package is already installed and just needs the installed version number this seems like the best solution to me.
Cleaning up https://stackoverflow.com/a/12413800 from #gringo-suave:
from itertools import ifilter
from os import path
from ast import parse
with open(path.join('package_name', '__init__.py')) as f:
__version__ = parse(next(ifilter(lambda line: line.startswith('__version__'),
f))).body[0].value.s
Now this is gross and needs some refining (there may even be an uncovered member call in pkg_resources that I missed), but I simply do not see why this doesn't work, nor why no one has suggested it to date (Googling around has not turned this up)...note that this is Python 2.x, and would require requiring pkg_resources (sigh):
import pkg_resources
version_string = None
try:
if pkg_resources.working_set is not None:
disto_obj = pkg_resources.working_set.by_key.get('<my pkg name>', None)
# (I like adding ", None" to gets)
if disto_obj is not None:
version_string = disto_obj.version
except Exception:
# Do something
pass
deploy package to server and file naming convention for indices packages :
example for pip dynamic version conversion:
win:
test_pkg-1.0.0-cp36-cp36m-win_amd64.whl
test_pkg-1.0.0-py3.6-win-amd64.egg
mac:
test_pkg-1.0.0-py3.7-macosx-10.12-x86_64.egg
test_pkg-1.0.0-py3.7-macosx-10.12-x86_64.whl
linux:
test_pkg-1.0.0-cp36-cp36m-linux_x86_64.whl
from setuptools_scm import get_version
def _get_version():
dev_version = str(".".join(map(str, str(get_version()).split("+")[0]\
.split('.')[:-1])))
return dev_version
Find the sample setup.py calls the dynamic pip version matching from git commit
setup(
version=_get_version(),
name=NAME,
description=DESCRIPTION,
long_description=LONG_DESCRIPTION,
classifiers=CLASSIFIERS,
# add few more for wheel wheel package ...conversion
)
For what is worth, I wrote getversion to solve this issue for one of our projects' needs. It relies on a sequence of PEP-compliant strategies to return the version for a module, and adds some strategies for development mode (git scm).
Example:
from getversion import get_module_version
# Get the version of an imported module
from xml import dom
version, details = get_module_version(dom)
print(version)
Yields
3.7.3.final.0
Why was this version found ? You can understand it from the details:
> print(details)
Version '3.7.3.final.0' found for module 'xml.dom' by strategy 'get_builtin_module_version', after the following failed attempts:
- Attempts for module 'xml.dom':
- <get_module_version_attr>: module 'xml.dom' has no attribute '__version__'
- Attempts for module 'xml':
- <get_module_version_attr>: module 'xml' has no attribute '__version__'
- <get_version_using_pkgresources>: Invalid version number: None
- <get_builtin_module_version>: SUCCESS: 3.7.3.final.0
More can be found in the documentation.
I am using an environment variable as below
VERSION=0.0.0 python setup.py sdist bdist_wheel
In setup.py
import os
setup(
version=os.environ['VERSION'],
...
)
For consistency check with packer version, I am using below script.
PKG_VERSION=`python -c "import pkg; print(pkg.__version__)"`
if [ $PKG_VERSION == $VERSION ]; then
python setup.py sdist bdist_wheel
else
echo "Package version differs from set env variable"
fi
I created the regex pattern to find version number from setup.cfg ?:[\s]+|[\s])?[=](?:[\s]+|[\s])?(.*)
import re
with open("setup.cfg", "r") as _file:
data = _file.read()
print(re.findall(r"\nversion(?:[\s]+|[\s])?[=](?:[\s]+|[\s])?(.*)", data))
# -> ['1.1.0']
You can add this code to your __init__.py:
VERSION = (0, 3, 0)
def get_version():
"""Return the VERSION as a string.
For example, if `VERSION == (0, 10, 7)`, return '0.10.7'.
"""
return ".".join(map(str, VERSION))
__version__ = get_version()
And add this to the setup.py:
def get_version(version_tuple):
"""Return the version tuple as a string, e.g. for (0, 10, 7),
return '0.10.7'.
"""
return ".".join(map(str, version_tuple))
init = os.path.join(os.path.dirname(__file__), "your_library", "__init__.py")
version_line = list(filter(lambda line: line.startswith("VERSION"), open(init)))[0]
VERSION = get_version(eval(version_line.split("=")[-1]))
And finally, you can add the version=VERSION, line to the setup:
setup(
name="your_library",
version=VERSION,
)
I've solved this issue in the following way:
Created a version.py inside my module:
setup.py
mymodule/
/ __init__.py
/ version.py
/ myclasses.py
version.py
__version__ = '1.0.0'
setup.py
import sys
sys.path.insert(0, ("./mymodule"))
from version import __version__
I've avoided dependency this way. Of course, I'was inspired by many answers here.
Thank you!

Categories

Resources