On Linux, every time I try to install a python package I get an error that the site-packages directory is missing. For example:
error: [Errno 2] No such file or directory: '/opt/python/cp39-cp39/Lib/site-packages/'
One simple solution is to just create the missing directory; however, this is not possible in certain instances. For example, when running python -m build, a venv with a random name is created to build the package in - therefore, I cannot create the site-package folder since there is no pause where I can modify the new venv.
I have tested this on multiple machines, and this issue also appears in GitHub hosted CI/CD runners. This is especially difficult to resolve in CI/CD pipelines because I have no control over additional venvs created by build, cibuildwheel and other PyPA tools.
This error occurs when I run pip install ., python -m build and several other pip commands that involve building wheel packages. The error always occurs while pip is building the wheel.
I would appreciate any suggestions!
Related
I would like to easily export one Python project from one PC to other. When I created the project, I used a virtual environment in order to avoid problems with different package versions.
What I did was to just copy the project folder and paste it in the destination PC. Once I opened the project with Pycharm, I activated the virtual environment with project_path/venv/Scripts/activate, but when I tried to execute any Script, it said it didn´t find the modules.
Which is the workflow I should follow in order to create projects and be able to run them from multiple PC-s without needing to install all the dependencies?
Since you did not specify your Python version I will provide a solution working for both Python 2.x and 3.x.
My suggestion is to create a requirements.txt file containing all your requirements.
This file can be easily prepared using the output from the command:
pip freeze
Then you can paste the output in your requirements.txt file and when you are going to install your Python code on another PC you can simply:
pip install -r requirements.txt
To install your requirements again.
Depending on your project it could be possible, for example, to create a single EXE file (if you are using Windows machines) but more detailed is needed if this is the case.
In case you are using Python 3 the method that is at the moment arguably more popular in the Python community is Pipenv.
Here's its relevant documentation.
And here you can read a simple example of a workflow.
if you are using python3 then use pipenv. It will automatically create Pipfile and Pipfile.lock. That will insure reinstalling dependencies on different machine will have the same packages.
basic and helpful commands:
pipenv shell # activate virutalenv
pipenv install # will install dependencies in Pipfile
pipenv install requests # will install requests lib. and will auto update Pipfile and Pipfile.lock
I created a small Python package. As I'm making modifications, I've found that I need to do the following in order for the changes to take effect:
cd to the package's root directory (same as where setup.py is located)
pip uninstall <package_name>
delete the build directory that was created when the package was last installed
python setup.py install
The key is step #3. I've noticed that the changes don't stick if the build directory isn't removed first. However, I don't know why this step is necessary and I can't find any references to having to do this and why. This is true for both Windows and Mac OSX. Can somebody please provide a rationale or point me in the right direction?
python setup.py install is implicitly equivalent to two subcommands:
build, which puts the files to install into the build directory, and
install, which copies files from the build/lib directory into the installation directory.
It appears that the first step is only performed when necessary, i.e. when you run python setup.py install without first removing the build directory, you are installing stale modules from the previous build.
To have changes take effect immediately while developing, you should run python setup.py develop, which creates a link in the installation directory back to the source code.
When would the -e, or --editable option be useful with pip install?
For some projects the last line in requirements.txt is -e .. What does it do exactly?
As the man page says it:
-e,--editable <path/url>
Install a project in editable mode (i.e. setuptools "develop mode") from a local project path or a VCS url.
So you would use this when trying to install a package locally, most often in the case when you are developing it on your system. It will just link the package to the original location, basically meaning any changes to the original package would reflect directly in your environment.
Some nuggets around the same here and here.
An example run can be:
pip install -e .
or
pip install -e ~/ultimate-utils/ultimate-utils-proj-src/
note the second is the full path to where the setup.py would be at.
Concrete example of using --editable in development
If you play with this test package as in:
cd ~
git clone https://github.com/cirosantilli/vcdvcd
cd vcdvcd
git checkout 5dd4205c37ed0244ecaf443d8106fadb2f9cfbb8
python -m pip install --editable . --user
it outputs:
Obtaining file:///home/ciro/bak/git/vcdvcd
Installing collected packages: vcdvcd
Attempting uninstall: vcdvcd
Found existing installation: vcdvcd 1.0.6
Can't uninstall 'vcdvcd'. No files were found to uninstall.
Running setup.py develop for vcdvcd
Successfully installed vcdvcd-1.0.6
The Can't uninstall 'vcdvcd' is normal: it tried to uninstall any existing vcdvcd to then replace them with the "symlink-like mechanism" that is produced in the following steps, but failed because there were no previous installations.
Then it generates a file:
~/.local/lib/python3.8/site-packages/vcdvcd.egg-link
which contains:
/home/ciro/vcdvcd
.
and acts as a "symlink" to the Python interpreter.
So now, if I make any changes to the git source code under /home/ciro/vcdvcd, it reflects automatically on importers who can from any directory do:
python -c 'import vcdvcd'
Note however that at my pip version at least, binary files installed with --editable, such as the vcdcat script provided by that package via scripts= on setup.py, do not get symlinked, just copied to:
~/.local/bin/vcdcat
just like for regular installs, and therefore updates to the git repository won't directly affect them.
By comparison, a regular non --editable install from the git source:
python -m pip uninstall vcdvcd
python -m pip install --user .
produces a copy of the installed files under:
~/.local/lib/python3.8/site-packages/vcdvcd
Uninstall of an editable package as done above requires a new enough pip as mentioned at: How to uninstall editable packages with pip (installed with -e)
Tested in Python 3.8, pip 20.0.2, Ubuntu 20.04.
Recommendation: develop directly in-tree whenever possible
The editable setup is useful when you are testing your patch to a package through another project.
If however you can fully test your change in-tree, just do that instead of generating an editable install which is more complex.
E.g., the vcdvcd package above is setup in a way that you can just cd into the source and do ./vcdcat without pip installing the package itself (in general, you might need to install dependencies from requirements.txt though), and the import vcdvcd that that executable does (or possibly your own custom test) just finds the package correctly in the same directory it lives in.
From Working in "development" mode:
Although not required, it’s common to locally install your project in
“editable” or “develop” mode while you’re working on it. This allows
your project to be both installed and editable in project form.
Assuming you’re in the root of your project directory, then run:
pip install -e .
Although somewhat cryptic, -e is short for
--editable, and . refers to the current working directory, so together, it means to install the current directory (i.e. your
project) in editable mode.
Some additional insights into the internals of setuptools and distutils from “Development Mode”:
Under normal circumstances, the distutils assume that you are going to
build a distribution of your project, not use it in its “raw” or
“unbuilt” form. If you were to use the distutils that way, you would
have to rebuild and reinstall your project every time you made a
change to it during development.
Another problem that sometimes comes up with the distutils is that you
may need to do development on two related projects at the same time.
You may need to put both projects’ packages in the same directory to
run them, but need to keep them separate for revision control
purposes. How can you do this?
Setuptools allows you to deploy your projects for use in a common
directory or staging area, but without copying any files. Thus, you
can edit each project’s code in its checkout directory, and only need
to run build commands when you change a project’s C extensions or
similarly compiled files. You can even deploy a project into another
project’s checkout directory, if that’s your preferred way of working
(as opposed to using a common independent staging area or the
site-packages directory).
To do this, use the setup.py develop command. It works very similarly
to setup.py install, except that it doesn’t actually install anything.
Instead, it creates a special .egg-link file in the deployment
directory, that links to your project’s source code. And, if your
deployment directory is Python’s site-packages directory, it will also
update the easy-install.pth file to include your project’s source
code, thereby making it available on sys.path for all programs using
that Python installation.
It is important to note that pip uninstall can not uninstall a module that has been installed with pip install -e. So if you go down this route, be prepared for things to get very messy if you ever need to uninstall. A partial solution is to (1) reinstall, keeping a record of files created, as in sudo python3 -m setup.py install --record installed_files.txt, and then (2) manually delete all the files listed, as in e.g. sudo rm -r /usr/local/lib/python3.7/dist-packages/tdc7201-0.1a2-py3.7.egg/ (for release 0.1a2 of module tdc7201). This does not 100% clean everything up however; even after you've done it, importing the (removed!) local library may succeed, and attempting to install the same version from a remote server may fail to do anything (because it thinks your (deleted!) local version is already up to date).
As suggested in previous answers, there is no symlinks that are getting created.
How does '-e' option work? -> It just updates the file "PYTHONDIR/site-packages/easy-install.pth" with the project path specified in the 'command pip install -e'.
So each time python search for a package it will check this directory as well => any changes to the files in this directory is instantly reflected.
I try to run pip wheel azure-mgmt=0.20.1, but whenever I run it I get following pip wheel error, which is very clear:
error: [Error 183] Cannot create a file when that file already exists: 'build\\bdist.win32\\wheel\\azure_mgmt-0.20.0.data\\..'
So my question is where or how I can find that path? I want to delete that existing file. I have been searching my local computer, searched for default path in Google, but still didn't find any solution.
Also is it possible to tell pip wheel to output full log? As you can see that full error path is not displayed. I'm using virtualenv.
We can see the description of virtual env at the official python guide:
To help manage external package dependencies, Azure Git deployment supports the creation of virtual environments.
When Azure detects a requirements.txt in the root of the repository, it automatically creates a virtual environment named env. This only occurs on the first deployment, or during any deployment after the selected Python runtime has changed.
You can directly modify the dependencies and versions of them in requirement.txt, then deploy your python app to Azure Web App via git, Azure will update the python packages automatically. You can check the packages in the virtual env folder which path is env\Lib\site-packages in the root directory of your site. You can login on the kudu console of your site to check your files of you site online, the URL should be: https://{your_site_name}.scm.azurewebsites.net/DebugConsole .
Additionally, according your description, it seems that you use the global python environment to run pip install command which may directly install packages in your global python environment. To install packages in your virtual env, you need to run the similar command env\scripts\pip install -r requirements.txt in your root directory of your application. Please refer to Web app development - Windows - command line for more information.
have you tried uninstalling and reinstalling?
I tried pip wheel azure-mgmt and that installed -0.20.1 for me.
The directory for mine is /Users/me/wheelhouse, so you could look there. I found that in the initial log of the build.
#Amir,
One option is that you could generate the requirement.txt file and remove your virtual environment if you used Visual studio to develop your application. Then you can add a new virtual environment for your project and install all packages from requirement.txt file. Or after removed your virtual environment, you can try pip wheel azure-mgmt command.
And another option is that you can follow this similar issue：https://vilimpoc.org/blog/2014/01/18/time-robbing-python-errors/
The blogger modified the LOCALAPPDATA path to resolve this issue. Please try it.
What are the differences between the below commands
python setup.py install develop
Doesn't work for me error No such file or directory: 'build/bdist.macosx-10.7-intel/egg/test-easy-install-37886.pth'
python setup.py develop
Works for me appears to make an .egg link file
python setup.py install
Works for me appears to make a .egg file which in .zip file format
Develop is a setuptools / distribute feature that allows you to add a project
to your Python environment without installing it - so you can continue
its "development"
In other words, when you call "python setup.py develop", setuptools will
compile the metadata and hook your project into Python's site-package,
but the packages and modules that will be used are the one in the
directory where you've run that command.
This is useful to continue working on your code and testing it without
having to run "python setup.py install" on every run
With develop, Python 'pseudo-installs' a package by running the setup.py script instead of install. The difference is a modification of the environment (it doesn't with develop), so a package can be imported from it's current location instead of a site-package directory. The advantage of this is you can develop packages that are being used by other packages, and you can modify source code in place with develop.
As far as "setup.py install develop", I've never seen anyone use that before, sorry.
source
source
source
python setup.py install develop
Is a wrong command.
When you use develop you use the current code when you run your application.
When you use install and then modify you code, your modifications will not be taken in account while running your app. until you rerun install or develop.