I'm trying to package my local module with pex and I can't seem to manage to.
I've created a simple project:
→ python --version
Python 2.7.10
→ pex --version
pex 1.1.15
→ tree .
.
├── bla
│ ├── __init__.py
│ └── main.py
└── setup.py
bla/__init__.py
import main
bla/main.py
if __name__ == '__main__':
print 'yo'
Which to me seems like the simplest project possible.
→ pex -v . -o v.pex --disable-cache
Traceback (most recent call last):
File "/Users/Charly/repos/load_tester/venv/bin/pex", line 11, in <module>
sys.exit(main())
File "/Users/Charly/repos/load_tester/venv/lib/python2.7/site-packages/pex/bin/pex.py", line 540, in main
pex_builder = build_pex(reqs, options, resolver_options_builder)
File "/Users/Charly/repos/load_tester/venv/lib/python2.7/site-packages/pex/bin/pex.py", line 475, in build_pex
resolvables = [Resolvable.get(arg, resolver_option_builder) for arg in args]
File "/Users/Charly/repos/load_tester/venv/lib/python2.7/site-packages/pex/resolvable.py", line 61, in get
raise cls.InvalidRequirement('Unknown requirement type: %s' % resolvable_string)
pex.resolvable.InvalidRequirement: Unknown requirement type: .
Also tried to do python setup.py bdist_pex but this failed as well as it can't find the command.
It really seems like I'm misunderstanding something fundamental but I can't figure out what.
I recently had a bit of a fight with pex trying to make it include local modules. What I learned is:
You must provide a valid setup.py file for your module(s) in order for this to work, and:
You must specify the application's entry point
This was tricky to figure out for several reasons. From reading the documentation, I was able to infer that the correct command in my case should be something like this:
$ pex . -v -e usersnotifier:main -o usersnotifier.pex
However, when I tried this, I kept getting an error saying:
pex.resolvable.InvalidRequirement: Unknown requirement type: .
A web search for this error turns up—as its first hit—this Github issue, which is still open as I type this. A spent a long while thinking that the above command wasn't working because of this bug. I attempted to downgrade setuptools and made a half-dozen other fruitless attempts to 'fix' the problem before this SO answer hinted at the necessity of supplying a setup.py file. (That Github issue turned out to be a red herring. The setuptools bug it mentions has since been fixed, from what I can tell.)
So... I wrote a setup.py file. At first, I kept getting that error saying Unknown requirement type: . But then I realized that my setup.py simply contained a dead-obvious typographical error. The error message emitted by pex in this case was actually quite clear, but it was followed by a large-ish stack trace and the Unknown requirement type: . message. I just wasn't paying close attention and missed it for longer than I care to admit.
I finally noticed my typo and fixed it, but another flaw in my setup.py was failing to include my local modules. pex worked in this case, but the generated file didn't:
$ pex . -v -e usersnotifier:main -o usersnotifier.pex --disable-cache
usersnotifier 0.1: Resolving distributions :: Packaging paho-mqtt
pyinotify 0.9.6
paho-mqtt 1.3.1
pex: Building pex: 2704.3ms
pex: Resolving distributions: 2393.2ms
pex: Packaging usersnotifier: 319.3ms
pex: Packaging pyinotify: 347.4ms
pex: Packaging paho-mqtt: 361.1ms
Saving PEX file to usersnotifier.pex
$ ./usersnotifier.pex
Traceback (most recent call last):
File ".bootstrap/_pex/pex.py", line 367, in execute
File ".bootstrap/_pex/pex.py", line 293, in _wrap_coverage
File ".bootstrap/_pex/pex.py", line 325, in _wrap_profiling
File ".bootstrap/_pex/pex.py", line 410, in _execute
File ".bootstrap/_pex/pex.py", line 468, in execute_entry
File ".bootstrap/_pex/pex.py", line 482, in execute_pkg_resources
File ".bootstrap/pkg_resources/__init__.py", line 2297, in resolve
ImportError: No module named 'usersnotifier'
Here's the bare-bones setup.py that finally worked for me:
from setuptools import setup
setup(
name='usersnotifier',
version='0.1',
py_modules=['usersnotifier', 'userswatcher'],
install_requires=[
'paho-mqtt>=1.3.1',
'pyinotify>=0.9.6',
],
include_package_data=True,
zip_safe=False
)
The reason it hadn't worked before was that I was accidentally passing the parameter py_module to setup() rather than py_modules (plural). ¯\_(ツ)_/¯
The final hurdle I encountered was mentioned in #cmcginty's answer to this question, namely: unless your module version number changes, pex will cache/reuse artifacts from the last time you ran it. So, if you fix a problem in your setup.py and re-run pex, it won't actually incorporate your changes unless you: a) bump the version number, or b) pass --disable-cache when invoking pex.
At the end of the day, the whole thing turned into an exercise in writing a proper setup.py, and running:
$ pex . -v -e mymodule:main -o mymodule.pex --disable-cache
Here are a few tips I can offer (possibly to a future version of my self):
TIP 1
Use python setup.py sdist to test your setup.py file. It's surprisingly easy to screw this up, and there's no point involving pex until you're sure your package has the right contents. After running python setup.py sdist, try installing the source package it generates (located in the dist folder) into a fresh venv and see whether it contains all the files you expect. Only move on to invoking pex after this is working.
TIP 2
Always pass --disable-cache to pex unless you have a good reason not to.
TIP 3
While troubleshooting all of these issues, I discovered that I could run:
$ unzip mymodule.pex
to extract the contents of the PEX file. This can be helpful in resolving any discrepancies that remain between your sdist package contents and your pex-ified application.
one way to do is:
create create a source distribution (tarball, zip file, etc.) using python setup.py sdist
then run pex command with -f DIST_DIR switch
eg. pex $(pip freeze) -o aflaskapp.pex -e 'aflaskapp.app' -f dist -v
Some additional details I discovered. If you run:
pex .
It will attempt to use the setup.py build in the directory. If you do not have that, then pex will fail.
By default, pex will cache the local package output, so if you change your setup.py then use the command to make sure your the changes are applied on the next pex run.
pex . --disable-cache
Related
I am actively developing a Python module that I would like to deploy in SQL Server 2017 installed locally, so I deploy the module in c:\Program Files\Microsoft SQL Server\<Instance Name>\PYTHON_SERVICES\Lib\site-packagesusing setuptoolslike so:
"c:\Program Files\Microsoft SQL Server\<Instance_Name>\PYTHON_SERVICES\python" setup.py develop
This produces an .egg-info directory in my project root, and a .egg-link file in the site-packages directory mentioned above. The .egg-link file correctly points to the .egg-info directory in my project root, so it appears setuptools is working correctly.
Here's my setup.pyfor reference:
from setuptools import setup
setup(
setup_requires=['pbr'],
pbr=True,
)
And here's the corresponding setup.cfg file:
[metadata]
name = <module_name>
description = <Module Description>
description-file = README.md
description-content-type = text/markdown
[files]
package_root = py/src
Since I am just trying to make the plumbing work, I have a single python script called uploader.py in <project_root>/py/src:
#uploader.py
class Upload:
pass
With this deployment in place, I am hoping to simply import the module I just published through .egg-link into a sp_execute_external_script call like so:
execute sp_execute_external_script #language= N'Python', #script= N'from <module_name>.uploader import Upload';
However, executing this stored procedure from SSMS produces the following error message:
Msg 39004, Level 16, State 20, Line 10
A 'Python' script error occurred during execution of 'sp_execute_external_script' with HRESULT 0x80004004.
Msg 39019, Level 16, State 2, Line 10
An external script error occurred:
Error in execution. Check the output for more information.
Traceback (most recent call last):
File "<string>", line 5, in <module>
File "C:\SQL-MSSQLSERVER-ExtensibilityData-PY\MSSQLSERVER01\C08BB9A7-66B5-4B5E-AAFC-B0248EE64199\sqlindb.py", line 27, in transform
from <module_name>.uploader import Upload
ImportError: No module named '<module_name>'
SqlSatelliteCall error: Error in execution. Check the output for more information.
STDOUT message(s) from external script:
SqlSatelliteCall function failed. Please see the console output for more information.
Traceback (most recent call last):
File "C:\Program Files\Microsoft SQL Server\<Instance_Name>\PYTHON_SERVICES\lib\site-packages\revoscalepy\computecontext\RxInSqlServer.py", line 587, in rx_sql_satellite_call
rx_native_call("SqlSatelliteCall", params)
File "C:\Program Files\Microsoft SQL Server\<Instance_Name>\PYTHON_SERVICES\lib\site-packages\revoscalepy\RxSerializable.py", line 358, in rx_native_call
ret = px_call(functionname, params)
RuntimeError: revoscalepy function failed.
I have obviously redacted module_name and Instance_Name from the error message.
I tried using install command instead of develop just to make sure the .egg-link file is not a problem. install installs the .egg-info file in site-packages but I get the same error.
I also tried removing pbr from the mix, but got the same error.
Lastly, I tried adding my <project_root> to sys.path as suggested by How can I use an external python module with SQL 2017 sp_execute_external_script?, but that didn't help either.
So at this point, I don't have a clue what I might be doing wrong.
The python version is 3.5.2 and I don't think an __init__.py is needed in the project for it to qualify as a module. Inserting a blank __init__.py in py/src doesn't help either.
My pip version is 19.3.1 and setuptools version is 44.0.0 and pbr version is 5.4.4 and I have confirmed all modules are installed in the site-packages directory mentioned above.
Based on my extensive experimentation, it appears that sp_execute_external_script doesn't follow symlinks (i.e. through the.egg-link file). Therefore, development mode installations will not work, whether you use setuptools, pip, pbr or anything else.
I even tried symlinking <package_name> folder as an OS symlink. Since I am on Windows, I used mklink /D command on Command Prompt to symlink /py/src/<package_name> inside site-packages. While the command goes through correctly, and I can see the symlinked folder in File Explorer, sp_execute_external_script fails to detect the package. Which tells me that there is probably something in sp_execute_external_script code that avoids traversing symbolic links.
I wonder if there is a way to make it traverse symbolic links.
The only workable solution is to develop a package's code under its own directory, so, in my case /py/src/<package_name>. Then, before running exec sp_execute_external_script #language=N'python', #script=N'...' copy the <package_name> folder to the site-packages directory.
This is, sort of, equivalent to setup.py install, but bypasses the creation of intermediate files and directories. So I am going to stick with this simple--though odious--approach.
I am hoping somebody more knowledgeable would offer a better way to solve this problem.
I'm trying to install pyev on Ubuntu using pip but I get this error. libev.so.4 does indeed exist, I found it at /usr/local/lib along with other libev.so files so I assume it's simply a matter of python isn't looking in the right spot or the .so's got put in the wrong place by mistake.
$ pip install pyev
Collecting pyev
Using cached pyev-0.9.0.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 20, in <module>
File "/tmp/pip-build-cBE9pk/pyev/setup.py", line 57, in <module>
check_version(libev_version(), min_libev_version, "libev")
File "/tmp/pip-build-cBE9pk/pyev/setup.py", line 48, in libev_version
libev_dll = cdll.LoadLibrary(libev_dll_name)
File "/usr/local/lib/python2.7/ctypes/__init__.py", line 443, in LoadLibrary
return self._dlltype(name)
File "/usr/local/lib/python2.7/ctypes/__init__.py", line 365, in __init__
self._handle = _dlopen(self._name, mode)
OSError: libev.so.4: cannot open shared object file: No such file or directory
So my question is where the libev.so files should live, or, if they're in the correct place, why can't python find them and how do I fix it?
EDIT: Python version 2.7.10, Pip version 7.1.0 both downloaded and installed today (Python manually, pip using get-pip.py)
locate libev gives me a long list of files for libevent, libevolution, and libevview. Pretty much the only thing related to libev was usr/lib/libevdocument.so.1, usr/lib/libevdocument.so.1.0.0, and /usr/share/doc/libevdocument1 (which included AUTHORS, NEWS.gz, README, TODO, changelog.Debian.gz, and copyright). Nothing in local for some reason
EDIT 2.0
Running it as sudo (can't believe I forgot that) gives me a weird error message "The directory /home/username/.cache/pip/http or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag." and then proceeded to redownload pyev-0.9.0.tar.gz and run into the same error of not finding libev.so.4.
Running it as 'sudo -H pip install pyev' gave me the first error message again (using the cached pyev). What does sudo -H do anyway? I couldn't find anything about it online or in the sudo help message.
Looking at another partition that has libev properly installed I was able to find the libev.so files under usr/lib rather than usr/local/lib. Copying them to that folder and running sudo -H pip install pyev seems to have properly installed pyev. Thanks for your help.
I'm trying to install a python package I've developed using the develop command of setuptools.
[sidenote: There is a bewilderingly vast quantity of information about this on the web (distutils, distutils2, setuptools, distribute). setuptools and develop are, as far as I can tell, the most modern/best practice way to use a piece of code that's in development. Perhaps I am wrong.]
Here's what I did:
(1) I placed an empty __init__.py in the directory with my Python code.
(2) I made a setup.py:
from setuptools import setup, find_packages
setup(name = "STEM_pytools",
version = "0.1",
packages = find_packages(),
author = "Timothy W. Hilton",
author_email = "my#email.address",
description = "visualization and data pre-/post-processing tools for STEM",
license = "",
keywords = "STEM",
url = "")
(3) I ran
python setup.py develop
That seemed to proceed without problems.
However, when I try to use the package, I get:
>>> import STEM_pytools
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named STEM_pytools
The same thing happens with the install command: it's output looks ok, then "No module named STEM_pytools". I'm tearing my hair out. Any suggestions appreciated!
I solved the problem, although I still don't entirely understand why it works now and did not work before. It seems my setup.py and the directory structure of my project were not interacting successfully.
This is the directory structure that worked with my setup.py:
STEMpytools/
setup.py
stem_pytools/
__init__.py
source1.py
source2.py
...
sourceN.py
This directory structure did not work, at least when paired with my setup.py:
STEMpytools/
setup.py
__init__.py
source1.py
source2.py
...
sourceN.py
This explanation helped me a lot: http://bashelton.com/2009/04/setuptools-tutorial/
Now, from the python interpreter, these both work:
import stem_pytools
import stem_pytools.source1
Experimenting on my system suggests it is necessary to place __init__.py and the package source code in a subdirectory one level below the root directory that contains setup.py. I'm not sure from the setuptools and distutils documentation why this is the case.
Got this django project that I assume would run on virtualenv. I installed virtualenv through pip install and created the env but when I try to feed the pip requirements file, I got this:
Directory 'tagging' is not installable. File 'setup.py' not found.
Storing complete log in /Users/XXXX/.pip/pip.log
Here's the entry on the log file:
------------------------------------------------------------
/Users/XXXX/Sites/SampleProject/bin/pip run on Wed Jul 21 06:35:02 2010
Directory 'tagging' is not installable. File 'setup.py' not found.
Exception information:
Traceback (most recent call last):
File "/Users/XXXX/Sites/SampleProject/lib/python2.6/site-packages/pip-0.7.2-py2.6.egg/pip/basecommand.py", line 120, in main
self.run(options, args)
File "/Users/XXXX/Sites/SampleProject/lib/python2.6/site-packages/pip-0.7.2-py2.6.egg/pip/commands/install.py", line 158, in run
for req in parse_requirements(filename, finder=finder, options=options):
File "/Users/XXXX/Sites/SampleProject/lib/python2.6/site-packages/pip-0.7.2-py2.6.egg/pip/req.py", line 1395, in parse_requirements
req = InstallRequirement.from_line(line, comes_from)
File "/Users/XXXX/Sites/SampleProject/lib/python2.6/site-packages/pip-0.7.2-py2.6.egg/pip/req.py", line 87, in from_line
% name)
InstallationError: Directory 'tagging' is not installable. File 'setup.py' not found.
Also, here's the requirements file I'm trying to feed:
# to use:
# mkvirtualenv %PROJECT% (or workon %PROJECT%)
# export PIP_RESPECT_VIRTUALENV=true
# pip install -r requirements.txt
# you'll also need:
# mongodb1.1.4
# imagemagick > 6.3.8
# -e svn+http://code.djangoproject.com/svn/django/trunk#egg=djangoipython
ipdb
PIL
django-extensions
django-debug-toolbar
pytz
tagging
Could it be a problem with PIP? I have installed it through easy_install and used it already to install some modules such as fabric and etc. with no problems.
Hope someone could lend a hand :) BTW, here's my local setup: OSX 10.6.4, Python 2.6.1, Django 1.3 alpha. Thanks!
Sounds like you have a tagging/ directory in the directory from which you are running pip, and pip thinks this directory (rather than the django-tagging project on PyPI) is what you want it to install. But there's no setup.py in that directory, so pip doesn't know how to install it.
If the name of the project you wanted to install from PyPI were actually "tagging", you'd need to move or rename the tagging/ directory, or else run pip from a different directory. But it's not; it's actually django-tagging: http://pypi.python.org/pypi/django-tagging So if you just change the entry in your requirements file from "tagging" to "django-tagging," it should work.
All of this is a bug in pip, really: it should assume something is a PyPI project name rather than a local directory, unless the name you give has an actual slash in it or appended to it.
Is it possible you've copied the "tagging" directory from this location in the django-tagging source? In that case, you actually need the root from this location which has "tagging" as a sub-directory and a setup.py file. Just checkout from trunk or unzip to a "django-tagging" directory and make sure that your requirements file points to the "django-tagging" directory.
I need to run some code on a Linux machine with Python 2.3.4
pre-installed. I'm not on the sudoers list for that machine, so I
built Python 2.6.4 into (a subdirectory in) my home directory. Then I
attempted to use virtualenv (for the first time), but got:
$ Python-2.6.4/python virtualenv/virtualenv.py ENV
New python executable in ENV/bin/python
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Installing setuptools.........
Complete output from command /apps/users/dspitzer/ENV/bin/python -c "#!python
\"\"\"Bootstrap setuptoo...
" /apps/users/dspitzer/virtualen...6.egg:
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
'import site' failed; use -v for traceback
Traceback (most recent call last):
File "<string>", line 67, in <module>
ImportError: No module named md5
----------------------------------------
...Installing setuptools...done.
Traceback (most recent call last):
File "virtualenv/virtualenv.py", line 1488, in <module>
main()
File "virtualenv/virtualenv.py", line 529, in main
use_distribute=options.use_distribute)
File "virtualenv/virtualenv.py", line 619, in create_environment
install_setuptools(py_executable, unzip=unzip_setuptools)
File "virtualenv/virtualenv.py", line 361, in install_setuptools
_install_req(py_executable, unzip)
File "virtualenv/virtualenv.py", line 337, in _install_req
cwd=cwd)
File "virtualenv/virtualenv.py", line 590, in call_subprocess
% (cmd_desc, proc.returncode))
OSError: Command /apps/users/dspitzer/ENV/bin/python -c "#!python
\"\"\"Bootstrap setuptoo...
" /apps/users/dspitzer/virtualen...6.egg failed with error code 1
Should I be setting PYTHONHOME to some value? (I intentionally named
my ENV "ENV" for lack of a better name.)
Not knowing if I can ignore those errors, I tried installing nose
(0.11.1) into my ENV:
$ cd nose-0.11.1/
$ ls
AUTHORS doc/ lgpl.txt nose.egg-info/ selftest.py*
bin/ examples/ MANIFEST.in nosetests.1 setup.cfg
build/ functional_tests/ NEWS PKG-INFO setup.py
CHANGELOG install-rpm.sh* nose/ README.txt unit_tests/
$ ~/ENV/bin/python setup.py install
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Traceback (most recent call last):
File "setup.py", line 1, in <module>
from nose import __version__ as VERSION
File "/apps/users/dspitzer/nose-0.11.1/nose/__init__.py", line 1, in <module>
from nose.core import collector, main, run, run_exit, runmodule
File "/apps/users/dspitzer/nose-0.11.1/nose/core.py", line 3, in <module>
from __future__ import generators
ImportError: No module named __future__
Any advice?
Have you actually run "make install" on your custom python build? Usually you'll want to do something like
./configure --prefix=/path/to/installdir (other options)
make
make install
Note Prefix can be any directory you have full write-permissions to, for example I very often use $HOME/apps on shared-hosting environments.
Then run /path/to/installdir/bin/python, not the one from your build directory. This should create the correct variables, and after that you can install virtualenv. Might be best to install virtualenv using its setup.py:
cd virtualenv_source_dir
/path/to/installdir/bin/python setup.py install
This may require installing setuptools first, using the same method.
Then finally:
# Just to be safe
export PATH="/path/to/installdir/bin:$PATH"
virtualenv ~/ENV
~/ENV/bin/pip install somepackage # (and such)
In addition to Crast's suggestion of making sure you actually installed your custom compiled Python, you should also check that the custom Python can actually find its libraries. This is the hint you're getting with the message about PYTHONHOME. The import errors suggest you need to set in your .bashrc or appropriate shell configuration export PYTHONHOME=/path/to/python_installation.
Additionally, when you are trying to tell virtualenv to use a non-default version of python, you need to use the -p,--python flag, e.g.,
virtualenv --python=/path/to/python_installation/bin/python myenv
See also the related question, "Use different Python version with virtualenv".
I had the same error when trying to install on an existing directory that already had easy_install in lib/python2.6. I had to put a link from lib64/python2.6 to lib/python2.6.
I am not saying my fix is the right fix, rather, I'm pointing to another reason why you might get this error.
Now you can easily install Python as an unpriviledged user using Anaconda: http://continuum.io/downloads
It's similar to this question. Once of the answers details making a new environment, so you don't need to use virtual-env and avoid the occasional gotchas: Installing Anaconda into a Virtual Environment.
conda create -n myenv1 ipython scipy
I don't have enough rep to add this as a comment on #Crast's answer and this question is 4 years old, but this might be useful to someone. In Windows, you have to path out to python.exe, but it seems that in Linux/OS X you just path to the folder. Example:
Windows:
virtualenv -p <PATH TO YOUR DESIRED PYTHON.EXE> venv
Creates a virtual environment in subfolder "venv" in current directory.