My pyflakes portion of flake8 is not running for my global python instance (/usr/bin/python, not virtualenv).
flake8 --version
2.2.3 (pep8: 1.5.7, mccabe: 0.2.1) CPython 2.7.5 on Darwin
It doesn't seem like pyflakes is getting attached to flake8. pip freeze confirms that pyflakes==0.8.1 is installed. I installed on my global site-packages ($ sudo pip install flake8).
However, when running inside of a virtualenv, pyflakes is in the list and flake8 works as expected.
I had a similar problem with conda's flake8. Here are some debugging notes:
flake8 registers the pyflakes checker in its setup.py file:
setup(
...
entry_points={
'distutils.commands': ['flake8 = flake8.main:Flake8Command'],
'console_scripts': ['flake8 = flake8.main:main'],
'flake8.extension': [
'F = flake8._pyflakes:FlakesChecker',
],
},
...
When checking a file, flake8 loads the registered entry points for 'flake8.extension' and registers found checkers:
...
for entry in iter_entry_points('flake8.extension'):
checker = entry.load()
pep8.register_check(checker, codes=[entry.name])
...
conda's flake8 seems to have problems writing those entry points.
from pkg_resources import iter_entry_points
list(iter_entry_points('flake8.extension'))
returns an empty list for me, which is why pyflakes won't be registered and thus does not work, even though it is installed and importable.
Updating setuptools and installing via pip install flake8 fixes the problem for me.
Related
I am trying to run pre-commit hooks, but they fail when they come across the isort hook which throws the error below:
File "/home/el/.cache/pre-commit/repoffrjhcx0/py_env-python3/lib/python3.10/site-packages/_distutils_hack/__init__.py", line 92, in create_module
return importlib.import_module('setuptools._distutils')
File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named 'setuptools'
I am using docker, and I have checked that setuptools is installed on both my machine globally, and on my docker. I do not understand why this error occurs. I think isort sets up it's own environment, but then why would it be not installed since it is defined in their config file pyproject.toml.
Below are my pre-commit and isort configs:
.pre-commit-config.yaml
repos:
- repo: https://github.com/pycqa/isort
rev: 5.8.0
hooks:
- id: isort
args: ["--multi-line=5", "--line-length=120", "--use-parentheses", "--filter-files"]
exclude: "migrations"
stages: [commit]
tox.ini
[isort]
line_length=120
skip_glob=*migrations*
multi_line_output=5
sections=FUTURE,STDLIB,THIRDPARTY,FIRSTPARTY,LOCALFOLDER
use_parentheses=true
include_trailing_comma=true
lines_between_types=1
lines_after_imports=2
[testenv:isort]
deps =
isort
commands =
isort . --check-only --diff
Python version on system: 3.10.1
Python version on docker: 3.8
I appreciate any help!
this was a bug in setuptools which has already been fixed (see also the pinned issue on isort)
you can work around this bug by setting the following environment variable: SETUPTOOLS_USE_DISTUTILS=stdlib
the version of setuptools comes from the version of virtualenv you're using, so you may want to upgrade to get the correct version
here's a longer summary of what went wrong with setuptools:
here's the relevant things to follow up on:
https://github.com/pypa/setuptools/issues/2353
https://github.com/pypa/pip/issues/6264
https://github.com/pypa/setuptools/issues/2980
why we're suddenly seeing this:
the latest virtualenv release upgraded setuptools to 60.1.0 (despite what the changelog
says
it was upgraded to 60.1.0
here)
setuptools 60.* changes the default to using the setuptools-embedded distutils rather than the stdlib distutils
setuptools does this by way of a .pth file which redirects imports of distutils to setuptools._distutils
during pip's isolated builds (triggered by pyproject.toml, for example to install isort via poetry) pip attempts to clear the
current sys.path of the enclosing environment and isolates to the
pyproject.toml-installed build dependencies, but it's a bit too late
as the .pth files from the enclosing environment are applied. so in
this environment setuptools is not installed, but its import hooks
are
disclaimer: I created pre-commit
I'm trying to package a python library that has setup-time (and also run-time) dependencies: it imports the modules so that the modules can inform the setup process of the location of some provided C headers:
from distutils.extension import Extension
from pybedtools.helpers import get_includes as pybedtools_get_includes
from pysam import get_include as pysam_get_include
# [...]
extensions = [
Extension(
"bam25prime.libcollapsesam", ["bam25prime/libcollapsesam.pyx"],
include_dirs=pysam_get_include()),
Extension(
"bam25prime.libcollapsebed", ["bam25prime/libcollapsebed.pyx"],
include_dirs=pybedtools_get_includes(),
language="c++"),
]
# [...]
However, one of the dependencies (pybedtools) needs to be installed with a specific --global-option pip option (see at the end of the post what happens when the option is not provided).
If I understand correctly, the currently up-to-date way to automatically have some dependencies available before setup.py is used is to indicate them in the [build-system] section of a pyproject.toml file.
I tried the following pyproject.toml:
[build-system]
requires = [
"pysam",
"pybedtools # git+https://github.com/blaiseli/pybedtools.git#fix_missing_headers --global-option='cythonize'",
]
build-backend = "setuptools.build_meta"
(By the way, it took me quite some time to figure out how to specify the build-backend, the documentation is not easily discoverable.)
However, this generates the following error upon pip install:
ERROR: Invalid requirement: "pybedtools # git+https://github.com/blaiseli/pybedtools.git#fix_missing_headers --global-option='cythonize'"
Hint: It looks like a path. File 'pybedtools # git+https://github.com/blaiseli/pybedtools.git#fix_missing_headers --global-option='cythonize'' does not exist.
How can I correctly specify options for dependencies ?
If I simply specify the package and its URL ("pybedtools # git+https://github.com/blaiseli/pybedtools.git#fix_missing_headers), the install fails as follows:
Exception:
Cython-generated file 'pybedtools/cbedtools.cpp' not found.
Please install Cython and run
python setup.py cythonize
It was while trying to tackle the above error that I found out about the --global-option pip option.
I can manually run pip install --global-option="cythonize" git+https://github.com/blaiseli/pybedtools.git#fix_missing_headers, and the install works, provided the dependencies of that package are already installed, otherwise their install fails because of an unrecognized "cythonize" option (which is another issue...).
Note that this option is only needed when installing "from source" (for instance when installing from a fork on github, as is my case here).
Same thing as in your other question, I suspect cythonize is a setuptools command and not a global option.
If it's indeed the case, then you would be better off setting an alias in your setup.cfg. If you run python setup.py alias install cythonize install, this should add the following to your setup.cfg:
[aliases]
install = cythonize install
When running pip install later, pip will honor this alias and the cythonize command will be executed right before the install command.
I am facing a strange problem which I could track down to the python logging package. Let me shortly explain what I want to do: the goal is to create html reports with python. I am using the R rmarkdown package which runs python code trough reticulate using a local virtualenv.
The Problem:
As soon as I install the python logging package rmarkdown runs into a problem when loading matplotlib. I have written a small test script to reproduce the example.
My system:
Ubuntu 18.04 bionic
Python 2.7.15rc1
Test "1" script (without logging):
Create a new virtualenv (venv).
Use venv/bin/pip to install matplotlib.
Run reticulate::import (at the end via rmarkdown::render).
Test "2" script (with logging):
Create a new virtualenv (venv).
In addition to the first test: install logging via venv/bin/pip.
Use venv/bin/pip to install matplotlib.
Run reticulate::import (at the end via rmarkdown::render).
The modules installed (virtualenv):
backports.functools-lru-cache 1.5
cycler 0.10.0
kiwisolver 1.0.1
logging 0.4.9.6 <- only for "test 2"
matplotlib 2.2.3
numpy 1.15.1
pip 18.0
pkg-resources 0.0.0
pyparsing 2.2.0
python-dateutil 2.7.3
pytz 2018.5
setuptools 40.2.0
six 1.11.0
subprocess32 3.5.2
wheel 0.31.1
The system site packages do have the same module version.
Results:
All tests from test 1 (without logging) work nicely.
The tests from test 2 (with loging) fail when using the virtualenv. When calling rmarkdown::render (see below), when using the system python installation (not virtualenv) they work nice as well.
There seem to be something strange with reticulate when logging is installed in a virtualenenv.
The output of the test script (see below):
The full output including the error:
----------- no logging package installed ------------
Module(matplotlib)
Module(matplotlib)
--------- with logging package installed ------------
Error in py_module_import(module, convert = convert) :
AttributeError: 'module' object has no attribute 'NullHandler'
Detailed traceback:
File "/home/retos/Downloads/venvtest/venv/lib/python2.7/site-packages/matplotlib/__init__.py", line 168, in <module>
_log.addHandler(logging.NullHandler())
Calls: <Anonymous> -> py_module_import -> .Call
Execution halted
Module(matplotlib)
The Module(matplotlib) output is the success message of loading the module via reticulate::import. As one can see only the one test fails where the virtualenv is used with installed logging python module.
Anyone having an idea what could case these problems? I spent quite some time to identify the source of the error, but I am kind of lost now ...
Test script to reproduce the error:
Here is a small bash/shell script to reproduce my tests.
#!/bin/bash
# New virtual environment and install matplotlib
echo " ----------- no logging package installed ------------"
if [ -d venv ] ; then rm -rf venv ; fi
virtualenv venv &>/dev/null > /dev/null
venv/bin/pip install matplotlib > /dev/null
# Print installed packages
Rscript -e "reticulate::use_python('venv/bin/python'); reticulate::import('matplotlib')"
Rscript -e "reticulate::import('matplotlib')"
# New virtual environment and install logging and matplotlib
echo " --------- with logging package installed ------------"
if [ -d venv ] ; then rm -rf venv ; fi
virtualenv venv > /dev/null
venv/bin/pip install logging > /dev/null
venv/bin/pip install matplotlib > /dev/null
# Print installed packages
Rscript -e "reticulate::use_python('venv/bin/python'); reticulate::import('matplotlib')"
Rscript -e "reticulate::import('matplotlib')"
I first thought it is related to the problem "ImportError: cannot import name cbook" but the solution there did not work.
May thanks in advance!
R
Logging became a standard module included in Python library in version 2.3. You must not install it from PyPI. Remove it ASAP:
pip uninstall logging
I'm using python 2.7.6 on Ubuntu 14.04.2 LTS. I'm using mock to mock some unittests and noticing when I import mock it fails importing wraps.
Not sure if there's a different version of mock or six I should be using for it's import to work? Couldn't find any relevant answers and I'm not using virtual environments.
mock module says it's compatible with python 2.7.x: https://pypi.python.org/pypi/mock
mock==1.1.3
six==1.9.0
Python 2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from mock import Mock
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/mock/__init__.py", line 2, in <module>
import mock.mock as _mock
File "/usr/local/lib/python2.7/dist-packages/mock/mock.py", line 68, in <module>
from six import wraps
ImportError: cannot import name wraps
also tried with sudo with no luck.
$ sudo python -c 'from six import wraps'
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: cannot import name wraps
Installed mock==1.0.1 and that worked for some reason. (shrugs)
edit: The real fix for me was to updated setuptools to the latest and it allowed me to upgrade mock and six to the latest. I was on setuptools 3.3. In my case I also had to remove said modules by hand because they were owned by OS in '/usr/local/lib/python2.7/dist-packages/'
check versions of everything
pip freeze | grep -e six -e mock
easy_install --version
Update everything
wget https://bootstrap.pypa.io/ez_setup.py -O - | sudo python
pip install mock --upgrade
pip install six --upgrade
Thanks #lifeless
I encountered the same issue on my mac, which I was able to fix by realizing that my python's sys.path contained both
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/
and
/Library/Python/2.7/site-packages/
with the former earlier than the latter.
You can test if this is happening to you by running the following in the python console.
import six
six.__version__
my python was loading an outdated six.py from the former directory (which didn't have wrapper), even though pip had installed a newer version six in the second directory. (It seems mac's framework comes with a version of six by default.)
I was able to fix it by moving six.py and six.pyc out of the first directory (requires sudo access), so that python would find the newer version of six in the second directory. I'm sure you could also change the ordering of the paths in sys.path.
To find the older version of six that need to be deleted run this from the terminal console
find /System/Library/Frameworks/Python.framework/Versions -name six.py*
so mock 1.1.1 and above defines a versioned requirement on six 1.7 or above:
https://github.com/testing-cabal/mock/blob/master/requirements.txt#L6
This gets reflected into setuptools metadata by pbr, which there is a versioned setup_requires dependency on:
https://github.com/testing-cabal/mock/blob/master/setup.py#L17
So there are a couple of possibilities:
1) six 1.7 is not new enough
2) there's a distro six package claiming to be 1.9.0 that doesn't have wraps for some reason
3) the setuptools in use didn't integrate properly with pbr and deps are missing
4) the wheel metadata isn't being interrogated properly by your pip/setuptools combination.
We do have a hard requirement for setuptools 17.1, and that was only explicitly reported by setup.py more recently. I'd love it if you can figure which of these is the case and update https://github.com/testing-cabal/mock/issues/298 so that we can fix whatever interaction is leading to this silent failure of setup.py / wheels.
On Mac OSX, the previously installed version of six was blocking my upgraded version from being used. I verified this, as previously suggested by running the following in my interpreter:
import six
six.__version__
To fix this I moved the file:
mv/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six.py
/tmp/old_six.py
This is stated already in another answer on this site, but I wanted to provide a more streamlined response.
I originally had an issue with old "OS-owned" versions of and pip/setuptools. After I installed pip manually, like so:
wget https://bootstrap.pypa.io/get-pip.py
sudo python get-pip.py
sudo ln -s /usr/local/bin/pip /usr/bin/pip
And then installing the latest version of pip, mock and six, I still had the problem you've described above. Turns out that I had six installed twice in:
/usr/lib/python2.7/dist-packages/
and in
/usr/local/lib/python2.7/dist-packages/
After I removed the six from /usr/lib/ it worked fine:
rm /usr/lib/python2.7/dist-packages/*six*
I did a pip install of six==1.9.0 and it took the new version. It seems like mock==1.3.0 doesn't properly define the version of six that it needs to get wraps support.
Another solution is setting your PYTHONPATH environment variable to point to the installed packages.
Setting my environment variable in my bash config so that:
PYTHONPATH=/Library/Python/2.7/site-packages
Allowed me to run tests in terminal (without removing/renaming any libraries, etc).
However, when using PyCharm, it was less-than-helpfully not correctly importing this environment variable. Even though PyCharm was showing as including parent variables (with that listed in the ones it showed importing), it seems this import wasn't working correctly.
Manually setting the environment variable to the above in the PyCharm run configuration resolves this.
I am unsure if PyCharm overwrites the PYTHONPATH variable after importing it from system environment variables or some other trickery, but this did resolve the error for me.
Though you aren't using a virtual environment like virtualenv, it's certainly a great use case for it. By sandboxing your Python installation and all the dependencies for your project, you can avoid hacking away at the global/default python installation entirely, which is where a lot of the complexity/difficulty comes from.
This is what I used when I got the wraps error - requirements.txt contains mock==2.0.0 and six==1.10.0:
cd <my_project>
virtualenv venv
source venv/bin/activate
sudo pip install -r requirements.txt
Not only is this simpler to use in my opinion, it's also simpler to document for people who might want to run your code.
I found a interesting things!
There is a file named "functools.py" in my project root path, and while I run my project , pycharm will raise ImportError.
So I rename my file fix this problem~~
I'm trying to convert a project that I had installing fine with distribute over to a newer setuptools based installer. For some reason I can't get setuptools to run at all. When I run my setup.py I get errors from distutils about unsupported options which are all the extension options provided by setuptools. I can't figure out why setuptools isn't taking care of these correctly. This is on a Debian Wheezy system running Python 2.7.
I created a simple test case to demonstrate the problem. It is a standalone script that I want installed as a utility with an executable wrapper script:
foo.py
#!/usr/bin/python
def main():
print 'Foo main() ran'
if __name__ == '__main__':
main()
setup.py
from setuptools import setup
setup(name='foo',
version='1.0',
py_modules = ['foo'],
entry_points = {
'console_scripts': ['foo = foo:main'] # setuptools extension
},
include_package_data = True # another setuptools extension
)
The version of setuptools in the Debian package archive is 0.6.24 which predates the merger with distribute and I would prefer to use something that retains some of the distribute heritage. I've been installing various versions of setuptools with pip. The latest 4.0.1 won't install properly but most anything from 1.4 to 3.8 works:
$ sudo pip install setuptools==3.8
...
$ python
Python 2.7.3 (default, Mar 13 2014, 11:03:55)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from setuptools import version
>>> version.__version__
'3.8'
The setuptools package/egg is placed in /usr/local/lib/python2.7/dist-packages which is normal on a Debian system when using pip.
When I try to install with my setup.py I get the following errors:
$ sudo python setup.py install
/usr/lib/python2.7/distutils/dist.py:267:
UserWarning: Unknown distribution option: 'entry_points'
warnings.warn(msg)
/usr/lib/python2.7/distutils/dist.py:267:
UserWarning: Unknown distribution option: 'include_package_data'
warnings.warn(msg)
running install
running build
running build_py
running install_lib
running install_egg_info
Removing /usr/local/lib/python2.7/dist-packages/foo-1.0.egg-info
Writing /usr/local/lib/python2.7/dist-packages/foo-1.0.egg-info
Distutils installs foo.py to /usr/local/lib/python2.7/dist-packages/ but obviously the setuptools generated wrapper script is absent. I would have expected setuptools to do its subclassing/monkeypatching thing to take care of anything not supported by distutils.
I remember setuptools being cranky years ago which is why I settled on using distribute. It's disappointing that it still doesn't "just work". Is there anything obvious that I'm missing? I have a suspicion that this has something to do with Debian's use of the dist-packages directory in place of site-packages but that was never an issue with distribute.
Well it looks like the problem is that setuptools doesn't like being installed into /usr/local/lib/....
I forced pip to put it into /usr/lib/... with:
sudo pip install --install-option="--prefix=/usr/lib/python2.7/dist-packages" \
setuptools==3.8
setup.py installs cleanly after that.