I have made a bug fix in a popular python package fastparquet https://github.com/dask/fastparquet and attempting to try it out on my machine. The package comes with a setup.py and so I have attempted to use that to install:
fastparquet $ python setup.py install
This completed and appears to have done the requisite installation of files to /usr/local/lib/python2.7/site-packages:
..
Adding python-dateutil 2.6.0 to easy-install.pth file
Using /usr/local/lib/python2.7/site-packages
Finished processing dependencies for fastparquet==0.1.0
However when we examine that location - specifically for the api.py file that was updated - it is still the original github version:
11:35:38/fastparquet $ll /usr/local/lib/python2.7/site-packages/fastparquet/api.py
-rw-r--r-- 1 sboesch admin 27294 Jun 19 17:18 /usr/local/lib/python2.7/site-packages/fastparquet/api.py
After having looked a bit into setup.py it seems that were a standard approach used across most python packages: https://docs.python.org/2/distutils/setupscript.html.
So then
am I misunderstanding how the install works /what it does?
Related
I installed Ansible 2.8.2 using dnf on Fedora 30. I have an Ansible plug-in that requires a library. Using pip3 install I installed the required library.
When I run ansible-playbook directly, I see a ModuleNotFoundError for that module.
But if I run python3 /usr/bin/ansible-playbook, the module is found.
How can I get Ansible as installed by dnf to see this library?
Edit: further info: as installed from dnf, the main Ansible script has a shebang for /usr/bin/python3 -s. If I remove the -s, this problem is solved.
What's the benefit that the repo maintainers were seeking in adding this -s flag?
Is there a case to be made for asking the repo maintainers to omit the flag?
How can I get pip3 to install the library I need into a directory that will be seen when the -s flag is in effect?
Edit: Here's the output of ansible --version, and thanks for asking.
ansible 2.8.2
config file = /home/jdashton/proj/ansible-ccharacter/ansible.cfg
configured module search path = ['/home/jdashton/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python3.7/site-packages/ansible
executable location = /usr/bin/ansible
python version = 3.7.4 (default, Jul 9 2019, 16:32:37) [GCC 9.1.1 20190503 (Red Hat 9.1.1-1)]
This is different from the suggested duplicate question because that question describes a "remote" task being run on localhost, in which case Ansible uses the default "remote" python interpreter. In this question, the library is being called from a local plugin, under Ansible's own python process. The -s flag in the shebang line at the top of the /usr/bin/ansible script is preventing Ansible from seeing some local libraries.
Edit: Following the suggestion from #zigarn I tried removing the library and reinstalling it as root. This resulted in the library being reinstalled into the same directory /usr/local/lib/... as before. Is there a way to get pip3 to install into the system library?
Here are the commands I attempted:
# pip3 uninstall tenacity
Uninstalling tenacity-5.0.4:
Would remove:
/usr/local/lib/python3.7/site-packages/tenacity-5.0.4.dist-info/*
/usr/local/lib/python3.7/site-packages/tenacity/*
Proceed (y/n)? y
Successfully uninstalled tenacity-5.0.4
# pip3 install tenacity
Collecting tenacity
Using cached https://files.pythonhosted.org/packages/6a/93/dfcf5b1b46ab29196274b78dcba69fab5e54b6dc303a7eed90a79194d277/tenacity-5.0.4-py2.py3-none-any.whl
Requirement already satisfied: six>=1.9.0 in /usr/lib/python3.7/site-packages (from tenacity) (1.12.0)
Installing collected packages: tenacity
Successfully installed tenacity-5.0.4
# pip show tenacity
Name: tenacity
Version: 5.0.4
Summary: Retry code until it succeeeds
Home-page: https://github.com/jd/tenacity
Author: Julien Danjou
Author-email: julien#danjou.info
License: Apache 2.0
Location: /usr/local/lib/python3.7/site-packages
Requires: six
Required-by:
Also, for clarity, here is the source for the Ansible installation:
# dnf list ansible
Last metadata expiration check: 0:09:48 ago on Mon 12 Aug 2019 11:12:58 AM EDT.
Installed Packages
ansible.noarch 2.8.2-1.fc30 #updates
According to the documentation -s option of python is "Don’t add the user site-packages directory to sys.path."
So I would guess that you did pip3 install as your user instead as root, so the library was installed at your user side instead of system wide.
Try to reinstall the library with pip3 as root and it should be ok.
I'm trying to use Python build tools to run CI for my project.
However the recomended "Install Python" task fails with Unable to process command '##vso[task.setvariable :
2018-11-27T09:01:45.8077207Z ##[section]Starting: Install Python
2018-11-27T09:01:45.8084595Z ==============================================================================
2018-11-27T09:01:45.8084738Z Task : Install Python on Windows
2018-11-27T09:01:45.8084853Z Description : Installs one or more Python versions into your build environment
2018-11-27T09:01:45.8084937Z Version : 1.0.13
2018-11-27T09:01:45.8085016Z Author : Steve Dower
2018-11-27T09:01:45.8085188Z Help : The *Install Python* task will install Python using a package from [nuget.org](https://nuget.org) and add it to `PATH`. These packages are part of the official release of CPython.
Most other tasks assume you have already run this task to install the desired runtime. The folder where they are installed is customizable, but be aware that you may need to update subsequent task configuration if you change this.
The recommended configuration for using this task with multiple versions of Python is to enable multi-configuration on the phase and specify the version number as a Multiplier. See [the documentation](https://go.microsoft.com/fwlink/?linkid=835763) for information about parallel execution.
(Note that the first-party *Use Python Version* task is preferable on the Hosted queues and works on all platforms, though as of 02 May 2018 is not fully available.)
2018-11-27T09:01:45.8085489Z ==============================================================================
2018-11-27T09:02:00.5624146Z ##[command]"C:\vsts-agent\_work\_tasks\InstallPython_6095fbbe-853d-4f2e-bc15-a88c92a78b4d\1.0.13\nuget.exe" install -OutputDirectory "C:\vsts-agent\_work\_tool\PythonNuget" python
2018-11-27T09:02:02.3259235Z Feeds used:
2018-11-27T09:02:02.3260274Z https://api.nuget.org/v3/index.json
2018-11-27T09:02:02.3265409Z
2018-11-27T09:02:02.3356523Z Installing package 'python' to 'C:\vsts-agent\_work\_tool\PythonNuget'.
2018-11-27T09:02:03.3744998Z GET https://api.nuget.org/v3/registration3-gz-semver2/python/index.json
2018-11-27T09:02:04.2195046Z OK https://api.nuget.org/v3/registration3-gz-semver2/python/index.json 844ms
2018-11-27T09:02:04.6414413Z Package "python.3.7.1" is already installed.
2018-11-27T09:02:04.9698862Z ##[error]Unable to process command '##vso[task.setvariable variable=pythonLocation;issecret=False]C:\vsts-agent\_work\_tool\PythonNuget\python.3.7.1\tools' successfully. Please reference documentation (http://go.microsoft.com/fwlink/?LinkId=817296)
2018-11-27T09:02:04.9723742Z ##[error]Value cannot be null.
Parameter name: RefName
2018-11-27T09:02:05.0583055Z ##[section]Finishing: Install Python
The install itself seems to have worked, because I can disable that step and run the package task with the variable manually set.
However this task also fails with a similar error:
2018-11-27T09:19:58.1789219Z ##[section]Starting: Build wheels from setup.py
2018-11-27T09:19:58.1797911Z ==============================================================================
2018-11-27T09:19:58.1798116Z Task : Build Python wheels
2018-11-27T09:19:58.1798220Z Description : Creates a wheel using an installed copy of Python.
2018-11-27T09:19:58.1798313Z Version : 1.0.6
2018-11-27T09:19:58.1798408Z Author : Steve Dower
2018-11-27T09:19:58.1798530Z Help : The *Build Python wheels* task uses a `setup.py` file to compile wheels. It will ensure that `pip`, `wheel`, `setuptools` and `cython` are installed before running.
2018-11-27T09:19:58.1798632Z ==============================================================================
2018-11-27T09:20:00.1574022Z ##[command]"C:\vsts-agent\_work\_tool\PythonNuget\python.3.7.1\tools\python.exe" -m pip install pip setuptools wheel cython
2018-11-27T09:20:09.3277504Z Requirement already satisfied: pip in c:\vsts-agent\_work\_tool\pythonnuget\python.3.7.1\tools\lib\site-packages (10.0.1)
2018-11-27T09:20:09.3647762Z Requirement already satisfied: setuptools in c:\vsts-agent\_work\_tool\pythonnuget\python.3.7.1\tools\lib\site-packages (39.0.1)
2018-11-27T09:20:09.4030940Z Collecting wheel
2018-11-27T09:20:10.1048438Z Downloading https://files.pythonhosted.org/packages/ff/47/1dfa4795e24fd6f93d5d58602dd716c3f101cfd5a77cd9acbe519b44a0a9/wheel-0.32.3-py2.py3-none-any.whl
2018-11-27T09:20:10.2095319Z Collecting cython
2018-11-27T09:20:10.8171412Z Downloading https://files.pythonhosted.org/packages/de/c8/daeef2f83ae402fc3c3eef4295e07fc46b267e5f12c7a8ee05b685de5c59/Cython-0.29.1-cp37-cp37m-win_amd64.whl (1.7MB)
2018-11-27T09:20:11.6943676Z Installing collected packages: wheel, cython
2018-11-27T09:20:11.8283445Z The script wheel.exe is installed in 'C:\vsts-agent\_work\_tool\PythonNuget\python.3.7.1\tools\Scripts' which is not on PATH.
2018-11-27T09:20:11.8283671Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2018-11-27T09:20:15.3438913Z The scripts cygdb.exe, cython.exe and cythonize.exe are installed in 'C:\vsts-agent\_work\_tool\PythonNuget\python.3.7.1\tools\Scripts' which is not on PATH.
2018-11-27T09:20:15.3439571Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2018-11-27T09:20:15.3891222Z Successfully installed cython-0.29.1 wheel-0.32.3
2018-11-27T09:20:15.8153804Z You are using pip version 10.0.1, however version 18.1 is available.
2018-11-27T09:20:15.8154881Z You should consider upgrading via the 'python -m pip install --upgrade pip' command.
2018-11-27T09:20:15.9164164Z ##[command]"C:\vsts-agent\_work\_tool\PythonNuget\python.3.7.1\tools\python.exe" "C:\vsts-agent\_work\1\s\setup.py" build --build-base "C:\vsts-agent\_work\1\b" bdist_wheel -d "C:\vsts-agent\_work\1\a" --universal
2018-11-27T09:20:17.2811564Z running build
2018-11-27T09:20:17.2812503Z running build_py
2018-11-27T09:20:17.7476566Z creating C:\vsts-agent\_work\1\b\lib
2018-11-27T09:20:17.7478546Z creating C:\vsts-agent\_work\1\b\lib\pvsdockerlib
2018-11-27T09:20:17.7487714Z copying pvsdockerlib\functions.py -> C:\vsts-agent\_work\1\b\lib\pvsdockerlib
2018-11-27T09:20:17.7509731Z copying pvsdockerlib\__init__.py -> C:\vsts-agent\_work\1\b\lib\pvsdockerlib
2018-11-27T09:20:17.7528017Z running bdist_wheel
2018-11-27T09:20:17.7948184Z installing to C:\vsts-agent\_work\1\b\bdist.win-amd64\wheel
2018-11-27T09:20:17.7950366Z running install
2018-11-27T09:20:17.7960351Z running install_lib
2018-11-27T09:20:17.8287382Z creating C:\vsts-agent\_work\1\b\bdist.win-amd64
2018-11-27T09:20:17.8290061Z creating C:\vsts-agent\_work\1\b\bdist.win-amd64\wheel
2018-11-27T09:20:17.8300030Z creating C:\vsts-agent\_work\1\b\bdist.win-amd64\wheel\pvsdockerlib
2018-11-27T09:20:17.8310198Z copying C:\vsts-agent\_work\1\b\lib\pvsdockerlib\functions.py -> C:\vsts-agent\_work\1\b\bdist.win-amd64\wheel\.\pvsdockerlib
2018-11-27T09:20:17.8333787Z copying C:\vsts-agent\_work\1\b\lib\pvsdockerlib\__init__.py -> C:\vsts-agent\_work\1\b\bdist.win-amd64\wheel\.\pvsdockerlib
2018-11-27T09:20:17.8348679Z running install_egg_info
2018-11-27T09:20:18.0161526Z running egg_info
2018-11-27T09:20:18.0162158Z creating pvs_dockerlib.egg-info
2018-11-27T09:20:18.0169182Z writing pvs_dockerlib.egg-info\PKG-INFO
2018-11-27T09:20:18.0184157Z writing dependency_links to pvs_dockerlib.egg-info\dependency_links.txt
2018-11-27T09:20:18.0199803Z writing requirements to pvs_dockerlib.egg-info\requires.txt
2018-11-27T09:20:18.0208911Z writing top-level names to pvs_dockerlib.egg-info\top_level.txt
2018-11-27T09:20:18.0227310Z writing manifest file 'pvs_dockerlib.egg-info\SOURCES.txt'
2018-11-27T09:20:18.0262855Z reading manifest file 'pvs_dockerlib.egg-info\SOURCES.txt'
2018-11-27T09:20:18.0307698Z writing manifest file 'pvs_dockerlib.egg-info\SOURCES.txt'
2018-11-27T09:20:18.0320556Z Copying pvs_dockerlib.egg-info to C:\vsts-agent\_work\1\b\bdist.win-amd64\wheel\.\pvs_dockerlib-0.1-py3.7.egg-info
2018-11-27T09:20:18.0449785Z running install_scripts
2018-11-27T09:20:18.2548207Z adding license file "LICENSE" (matched pattern "LICEN[CS]E*")
2018-11-27T09:20:18.2597715Z creating C:\vsts-agent\_work\1\b\bdist.win-amd64\wheel\pvs_dockerlib-0.1.dist-info\WHEEL
2018-11-27T09:20:18.2616849Z creating 'C:\vsts-agent\_work\1\a\pvs_dockerlib-0.1-py2.py3-none-any.whl' and adding 'C:\vsts-agent\_work\1\b\bdist.win-amd64\wheel' to it
2018-11-27T09:20:18.2642603Z adding 'pvsdockerlib/__init__.py'
2018-11-27T09:20:18.2660569Z adding 'pvsdockerlib/functions.py'
2018-11-27T09:20:18.2668323Z adding 'pvs_dockerlib-0.1.dist-info/LICENSE'
2018-11-27T09:20:18.2674321Z adding 'pvs_dockerlib-0.1.dist-info/METADATA'
2018-11-27T09:20:18.2679798Z adding 'pvs_dockerlib-0.1.dist-info/WHEEL'
2018-11-27T09:20:18.2685103Z adding 'pvs_dockerlib-0.1.dist-info/top_level.txt'
2018-11-27T09:20:18.2688000Z adding 'pvs_dockerlib-0.1.dist-info/RECORD'
2018-11-27T09:20:18.2691808Z removing C:\vsts-agent\_work\1\b\bdist.win-amd64\wheel
2018-11-27T09:20:18.2719804Z C:\vsts-agent\_work\_tool\PythonNuget\python.3.7.1\tools\lib\distutils\dist.py:274: UserWarning: Unknown distribution option: 'long_description_content_type'
2018-11-27T09:20:18.2720555Z warnings.warn(msg)
2018-11-27T09:20:18.5079776Z ##[error]Unable to process command '##vso[task.setvariable variable=dist;issecret=False]C:\vsts-agent\_work\1\a' successfully. Please reference documentation (http://go.microsoft.com/fwlink/?LinkId=817296)
2018-11-27T09:20:18.5106520Z ##[error]Value cannot be null.
Parameter name: RefName
2018-11-27T09:20:18.5664259Z ##[section]Finishing: Build wheels from setup.py
How is this possible when Python build tools uses the built in powershell command Set-VstsTaskVariable?
Github Python build tools install set pythonLocation var
Github Python build tools package set dist var
Is this a bug in the azure-pipelines-agent? I'm using the latest pre-release v2.141.2, and on-prem TFS 2017 update 3.1
I am installing Twisted 12.1, which must be done by downloading from Twisted website. I am getting the following error after python setup.py install, which complains that a script file in setuptools is missing. Script file seems should be installed along with Python2.6. Can anyone point me to the direction of where to find such script file?
Processing Twisted-12.1.0-py2.6-solaris-2.12-i86pc.egg
creating /usr/lib/python2.6/site-packages/Twisted-12.1.0-py2.6-solaris-2.12-i86pc.egg
Extracting Twisted-12.1.0-py2.6-solaris-2.12-i86pc.egg to /usr/lib/python2.6/site-packages
Adding Twisted 12.1.0 to easy-install.pth file
error: /usr/lib/python2.6/vendor-packages/setuptools/script template.py: No such file or directory
I had a similar error installing Ansible on a Solaris 11.3 machine. System Python was 2.7.9 but I think it's comparable, i. e. 'script template.py' didn't exist in /usr/lib/python2.7/vendor-packages/setuptools.
I was successful installing a separate copy of setuptools und using them afterwards.
I. e. download setuptools and install them in a custom location (example):
setuptools-20.10.1$ python setup.py install --prefix=/export/dist/Solaris-x86/stw/setuptools-20.10.1
Then build your other package with (in this example) PYTHONPATH=/export/dist/Solaris-x86/stw/setuptools-20.10.1/lib/python2.7/site-packages.
I am trying to build a package using the "conda build fibtestpackage" command.
The "fibtestpackage" is built using Cython, hence in the .sh file I have the command:
python setup.py build_ext --inplace
Even though the Build ends successfully, and the output is as expected:
Fetching packages ...
fibtestpackage-1.0.0 100% |################################| Time: 0:00:00 3.03 MB/s
Extracting packages ... [ COMPLETE ] |##################################################| 100%
Linking packages ...
[ COMPLETE ] |##################################################| 100%
the one test I have in the YAML file fails. The only test I have looks like this:
test:
# Python imports
imports:
- fibtestpackage
#- numpy
#- samplers
Hence, its an import error,
ImportError: No module named fibtestpackage
My first guess for why there is an importerror is because the .SO, .C, and .PYX files generated by the build (and the other files of the library) are NOT being place into the anaconda/lib/python2.7/site-packages/ folder where the other modules are located. Do I need to specify this in the shell file or should this be covered by the "python setup.py build_ext --inplace" command?
My second guess is that there are two different versions of conda (after updating) in the distribution,
1) Python 2.7.8 |Anaconda 2.1.0 (64-bit)| (default, Aug 21 2014, 18:22:21)
2) Python 2.7.8 |Continuum Analytics, Inc.| (default, Aug 21 2014, 18:22:21)
Could this be causing the issue? Perhaps the build environment is installing libraries in the wrong one?
Thank you in advance for your help!
Conda build creates a build conda environment, called _build (generally in ~/anaconda/envs/_build), which is where you should install everything in. For Python, generally doing python setup.py install is good enough if you include Python as a build dependency in the meta.yaml because the python will be the one installed in the _build environment and it will install there.
At the end of the build, conda build wraps up all the new files in the build environment and creates a conda package out of them. If there is a test phase, it deletes the _build environment and creates a test environment, called _test, and installs and runs the package there.
To debug this problem, try running the Python in ~/anaconda/envs/_test.
In your case, the issue is likely that python setup.py build_ext --inplace does not actually install the files, it just builds them. Check at the end of the build to see how many files it says there are. If there are 0, then this is the issue.
I would like to try zeroRPC but couldn't install the package properly. I am using the latest python_xy distribution (python 2.7.3) under windows 7 and I must say I don't have much experience with installing new modules since the distribution is allready pretty complete.
I pulled the master zeroRPC-python from gitHub and tried to do "python setup.py install"
I had a first problem with something like "impossible to locate vcvarsall.bat". I solved it by installing mingw as explained here error: Unable to find vcvarsall.bat
Then I could run the install untill the end, but now, when I import zerorpc, I get the following ImportError (only the end of the stack):
C:\Python27\lib\site-packages\gevent-0.13.8-py2.7-win32.egg\gevent\greenlet.py in <module>()
4 import traceback
5 from gevent import core
----> 6 from gevent.hub import greenlet, getcurrent, get_hub, GreenletExit, Waiter
7 from gevent.timeout import Timeout
8
C:\Python27\lib\site-packages\gevent-0.13.8-py2.7-win32.egg\gevent\hub.py in <module>()
28
29 try:
---> 30 greenlet = __import__('greenlet').greenlet
31 except ImportError:
32 greenlet = __import_py_magic_greenlet()
ImportError: No module named greenlet
I wonder more generally if I am following the right procedure to install new packages (under windows) or if there is a simpler way (safer with dependancies) that I would be overlooking (easy_install)? I must say I am very new to this and any hints or link to the relevant documentation would be appreciated.
Thanks in advance,
Samuel
I was struggling with this question myself for a while now. The solution involves several components, and many answers out there seem to relate to different versions of those components that don't always play well together.
Here is the complete solution that worked for me, starting from an empty virtualenv:
mkvirtualenv myenv
python -m pip install --upgrade pip==6.0.8 wheel==0.24.0
pip install gevent-1.0.1-cp27-none-win32.whl pyzmq-13.1.0-cp27-none-win32.whl zerorpc==0.4.4
The first step installs wheel and upgrades pip itself to support wheel package installations. The next step installs binary wheels for gevent-1.0.1 (downloadable from this unofficial but extremely useful python windows binaries page) and pyzmq-13.1.0 (available here), and the zerorpc-0.4.4 package from source in the usual way.
Note that I hard-coded source package versions here (pip 6.0.8, wheel 0.24.0, zerorpc 0.4.4) because as I said other versions don't always follow the same build patterns. This may not be necessary and future versions may prove to work just as well together.
The final result for me:
(myenv) C:\work>pip freeze
gevent==1.0.1
greenlet==0.4.5
msgpack-python==0.4.5
pyzmq==13.1.0
wheel==0.24.0
zerorpc==0.4.4
I used a slightly different way, I am using Anaconda + Jupyter to run my python notebooks.
I used this link to zerorpc package, and installed using
conda install -c groakat zerorpc
which installed following -