I'm new to Python, so I think my question is very fundamental and is asked a few times before but I cannot really find something (maybe because I do not really know how to search for that problem).
I installed a module in Python (reportlab). Now I wanted to modify a python script in that module but it seems that the python interpreter does not notice the updates in the script. Ironically the import is successful although Python actually should not find that package because I deleted it before. Does Python uses something like a Cache or any other storage for the modules? How can I edit modules and use those updated scripts?
From what you are saying, you downloaded a package and installed it using either a local pip or setup.py. When you do so, it copies all the files into your python package directory. So after an install, you can delete the source folder because python is not looking here.
If you want to be able to modify, edit, something and see changes, you have to install it in editable mode. Inside the main folder do:
python setup.py develop
or
pip install -e .
This will create a symbolic link to you python package repository. You will be able to modify sources.
Careful for the changes to be effective, you have to restart your python interpreter. You cannot just import again the module or whatever else.
Related
I am a beginner in Linux and I am trying to do something very basic (yet it does not seem to work).
I have installed a python3 package. I wish to run this Python package on a specific folder from my linux machine. The idea is that this package should restructure all my files in a specific format (this is the link to the package https://github.com/SIMEXP/Data2Bids ).
In the demo of the tutorial, it is mentioned that once I install the package:
pip3 install data2bids
I should just use cd on my desired folder that needs restructuring and then simply run data2bids.
After doing so, I received a command not found error.
I have tried to run the package using the python data2bids command (thinking that maybe I should mention it is a python package. I have tried to use the full path of where the package was installed in the command, yet it did not work. I even tried to install the package in the same folder as my directory that needs restructuring, with no results.
I believe that this is a very basic problem, and the solution may be very simple. Within a linux machine, how does one exactly run a python module in a specific folder?
I am also a rookie in Linux, here's what i can think of.
First, the input is:
A directory containing some files in any extension, with names containing at minimum the information of modality and patient number. A .JSON configuration file explaining how the filenames should be read.
I assume you ran the command without the configuration file.
Second, Linux has built-in python whose version is 2.x, try to type python in command line and print python version, if it's 3.x, try to query pip installed list to see if the package was installed successfully, if not, see if the installed python 3.x directory is contained in PATH.
I am working in Python 3.6+ and want to check if a few different modules are installed from within my script. If not, I want to attempt to install them with a few caveats:
1) the proper way to do this, if I remember reading it correctly, is to look into 'packaging and versioning' .. possibly with setuptools .. im not really sure. There is a Digital ocean page that is hard to follow. It discusses this but I keep running into an issue with documents around this topic: they are all based around the assumption that the project will be uploaded to pypip for use with pip. I specifically do not want this. I want to distribute this directly to individuals, by hand. Maybe in the future have it available in a closed, not-open-to-everyone github repo.
Currently in my script I'm using a try and except. Try to import these modules, if they don't exist i run this exception which i don't know if it works.
except ImportError:
from pip._internal import main as pip
pip(['install', colorama])
import colorama
print('colorama imported successfully')
and for what its worth - i have no idea what pip(['install', colorama]) is doing.
The packaging aspect seems to include imported modules with your code. How does one preform this function? Instead of checking if colorama is installed and then attempting to launch a subprocess to install it .. how do i just include the entire thing assuming this is the 'right' way to do this?
One thing that's usually done to avoid this problem is to build your program in a virtual environment which you know to contain the correct python scripts - and then either
package the entire virtual environment with your project as a unit, or
write a requirements.txt file that lists all the packages (and versions) that are expected to be installed before the user runs the program (you'd install everything on that list by doing pip install -r requirements.txt on the command line before running the program with python script_name.py)
Ideally, you'd then have your script fail if the required dependencies aren't there, and have the user install them manually to fix the issue.
Here's python 3's documentation on virtual environments
What you're doing now is unconventional - if it's working, it's working, but it's not great practice. Biggest reason for that is that your script, in its current state, is installing software on the user's machine without their consent, i.e. the user did not tell the program to install this software and was not told that the software was necessary, but the program is installing it anyway. In your case this may be harmless, but in general it's something to stay away from because it can get into really shady territory.
I'm new to using python modules.
I'm currently working on a python 2.7 script that will be deployed to many remote computers (which have python 2.7 on them). The problem is that the script needs to use a module, which I am not allowed to install on those computers.
I'm wondering if it is possible to include the module files in the same package as my script (possibly have them compiled first), and then have the script import the library from that local folder, thus achieving a "portable" script.
If that is possible, how would I go about doing that?
Specifics: I'm running 2.7.11 on Windows needing to use Paramiko.
I'm asking this question because the similar questions that I can find either do not answer mine, or expect me to be familiar with core python structures with which I am not. I also DON'T want to include the entirety of python and then install the module onto that, something I see is often called Portable Python. I just want to send my script and the module and nothing more.
Many thanks!
To install modules in a specific directory, you can try pip install module --target=.
By default python search for those modules in same directory as the script first, then, if not available, it will search for python install lib files.
I'm working on a python project that contains a number of routines I use repeatedly. Instead of rewriting code all the time, I just want to update my package and import it; however, it's nowhere near done and is constantly changing. I host the package on a repo so that colleagues on various machines (UNIX + Windows) can pull it into their local repos and use it.
It sounds like I have two options, either I can keeping installing the package after every change or I can just add the folder directory to my system's path. If I change the package, does it need to be reinstalled? I'm using this blog post as inspiration, but the author there doesn't stress the issue of a continuously changing package structure, so I'm not sure how to deal with this.
Also if I wanted to split the project into multiple files and bundle it as a package, at what level in the directory structure does the PTYHONPATH need to be at? To the main project directory, or the .sample/ directory?
README.rst
LICENSE
setup.py
requirements.txt
sample/__init__.py
sample/core.py
sample/helpers.py
docs/conf.py
docs/index.rst
tests/test_basic.py
tests/test_advanced.py
In this example, I want to be able to just import the package itself and call the modules within it like this:
import sample
arg = sample.helper.foo()
out = sample.core.bar(arg)
return out
Where core contains a function called foo
PYTHONPATH is a valid way of doing this, but in my (personal) opinion it's more useful if you have a whole different place where you keep your python variables. Like /opt/pythonpkgs or so.
For projects where I want it to be installed and also I have to keep developing, I use develop instead of install in setup.py:
When installing the package, don't do:
python setup.py install
Rather, do:
python setup.py develop
What this does is that it creates a synlink/shortcut (I believe it's called egglink in python) in the python libs (where the packages are installed) to point to your module's directory. Hence, as it's a shortcut/symlink/egglink when ever you change a python file, it will immediately reflect the next time you import that file.
Note: Using this, if you delete the repository/directory you ran this command from, the package will cease to exist (as its only a shortcut)
The equivalent in pip is -e (for editable):
pip install -e .
Instead of:
pip install .
This is probably a question that has a very easy and straightforward answer, however, despite having a few years programming experience, for some reason I still don't quite get the exact concepts of what it means to "build" and then to "install". I know how to use them and have used them a lot, but have no idea about the exact processes which happen in the background...
I have looked across the web, wikipedia, etc... but there is no one simple answer to it, neither can I find one here.
A good example, which I tried to understand, is adding new modules to python:
http://docs.python.org/2/install/index.html#how-installation-works
It says that "the build command is responsible for putting the files to install into a build directory"
And then for the install command: "After the build command runs (whether you run it explicitly, or the install command does it for you), the work of the install command is relatively simple: all it has to do is copy everything under build/lib (or build/lib.plat) to your chosen installation directory."
So essentially what this is saying is:
1. Copy everything to the build directory and then...
2. Copy everything to the installation directory
There must be a process missing somewhere in the explanation...complilation?
Would appreciate some straightforward not too techy answer but in as much detail as possible :)
Hopefully I am not the only one who doesn't know the detailed answer to this...
Thanks!
Aivoric
Building means compiling the source code to binary in a sandbox location where it won't affect your system if something goes wrong, like a build subdirectory inside the source code directory.
Install means copying the built binaries from the build subdirectory to a place in your system path, where they become easily accessible. This is rarely done by a straight copy command, and it's often done by some package manager that can track the files created and easily uninstall them later.
Usually, a build command does all the compiling and linking needed, but Python is an interpreted language, so if there are only pure Python files in the library, there's no compiling step in the build. Indeed, everything is copied to a build directory, and then copied again to a final location. Only if the library depends on code written in other languages that needs to be compiled you'll have a compiling step.
You want a new chair for your living-room and you want to make it yourself. You browse through a catalog and order a pile of parts. When they arrives at your door, you can't immediately use them. You have to build the chair at your workshop. After a bit of elbow-grease, you can sit down in it. Afterwards, you install the chair in your living-room, in a convenient place to sit down.
The chair is a program you want to use. It arrives at your house as source code. You build it by compiling it into a runnable program. You install it by making it easier to use.
The build and install commands you are refering to come from setup.py file right?
Setup.py (http://docs.python.org/2/distutils/setupscript.html)
This file is created by 3rd party applications / extensions of Python. They are not part of:
Python source code (bunch of c files, etc)
Python libraries that come bundled with Python
When a developer makes a library for python that he wants to share to the world he creates a setup.py file so the library can be installed on any computer that has python. Maybe this is the MISSING STEP
Setup.py sdist
This creates a python module (the tar.gz files). What this does is copy all the files used by the python library into a folder. Creates a setup.py file for the module and archives everything so the library can be built somewhere else.
Setup.py build
This builds the python module back into a library (SPECIFICALLY FOR THIS OS).
As you may know, the computer that the python library originally came from will be different from the library that you are installing on.
It might have a different version of python
It might have a different operating system
It might have a different processor / motherboard / etc
For all the reasons listed above the code will not work on another computer. So setup.py sdist creates a module with only the source files needed to rebuild the library on another computer.
What setup.py does exactly is similar to what a makefile would do. It compiles sources / creates libraries all that stuff.
Now we have a copy of all the files we need in the library and they will work on our computer / operating system.
Setup.py install
Great we have all the files needed. But they won't work. Why? Well they have to be added to Python that's why. This is where install comes in. Now that we have a local copy of the library we need to install it into python so you can use it like so:
import mycustomlibrary
In order to do this we need to do several things including:
Copy files to their library folders in our version of python.
Make sure library can be imported using import command
Run any special install instructions for this library. (seting up paths, etc)
This is the most complicated part of the task. What if our library uses BeautifulSoup? This is not a part of Python Library. We'd have to install it in a way such that our library and any others can use BeautifulSoup without interfering with each other.
Also what if python was installed someplace else? What if it was installed on a server with many users?
Install handles all these problems transparently. What is does is make the library that we just built able to run. All you have to do is use the import command, install handles the rest.