I am somewhat new to Python and even newer to distutils.
I wanted to create a distribution of my program for other users in my office, so I used setup from distutils.core. I set up my directory structure, created a manifest.in file and a setup.py file. Everything seemed to go as planned. The result is that I have a .zip file containing the directory structure that I intended.
The maifest file was not contained in the .zip file (I assume it was only needed by distutils), but the setup.py file remained in the .zip file. Why is that? Is setup.py needed by the end-user?
Thank you,
-RS
In the normal case, users install your app by running python setup.py install, or something that effectively does the same thing (like pip install foo).
Of course there are cases where they don't need setup.py—e.g., because they're installing a pre-packaged binary egg or Windows installer or whatever—but most packages have to work for the normal case. So, the default packaging commands include it. In the docs, Specifying the files to distribute says:
If you don’t supply an explicit list of files (or instructions on how to generate one), the sdist command puts a minimal default set into the source distribution:
… setup.py (or whatever you called your setup script) …
Related
I have a python project which takes in a bunch of pdf files from a directory, scrapes data from them, and then does some matching of that scraped data with some data in a CSV file.
The whole work has 2-3 python scripts, used as modules, and also uses dependencies of pdftotext, pandas, NumPy, etc.
Now I can pip freeze my Conda env and it can give me a requirements.txt file with all packages to install.
However, I want this main python script (which calls other modules and runs the whole project) to be run by a less technical person who doesn't work on pandas and other such python stuff.
So is there a way I can make this whole project as an executable file that encapsulates all dependencies, packages, scripts, and just running that executable in the terminal should run the whole project without having the other person install all dependencies themselves using requirements.txt file.
I can't use docker unfortunately as that is not permitted right now for my work.
I was thinking buck build if that works?
https://buck.build/
or if there is an easy way?
Thanks!
One approach is to package the Python application directory as a .zip file and execute that. Zip files that have a __main__.py entry point can be run this way.
This can be done easily in version 2.6 and up. Additional “zipapp” support was added in 3.6.
The main challenge has to do with compatibility for non-pure-Python libraries. What you zip up needs to be compatible with the machine where it will be run.
pip install cx_freeze
cxfreeze main.py --target-name your_exe_name
Replace your_exe_name. It will generate a build folder with your .exe in it.
I am trying to create a Python distribution where I have to include both the source and the compiled binary. (Yes, I read arguments against/for adding .pyc, but my use case requires the .pyc to be added). Running my steps in Windows, both the source and compiled binaries are added in the output file (I used both sdist and bdist_wheel).
Say I have the following structure:
root
+--folderA
+--alpha
+--beta
+--__init__.py
+--folderB
folderA contains source codes while folderB contains other files within its subdirectories.
Steps done:
Modules are compiled using compileall
Sources in alpha are removed. Sources in beta are kept.
Run python setup.py sdist|bdist_wheel
I used find_packages() in setup.py to detect the modules. Modules in alpha are not detected while those in beta are found.
In the results .tgz and .whl files in Windows, all the needed files are there. All is good.
However, when the same procedure is done in Linux (Ubuntu to be specific), only the modules in beta are added and some modules in folderB but not other files of different type and the module alpha. sdist will give only the source while bdist and bdist_wheel will give the .pycs only. I understand that sdist is for distributing source files while bdist is for binary files.
My question is why is the behavior different in Windows and is it possible to produce the same output in Linux (source and .pyc along with other files)?
I am using Python 2.7.
To include files in sdist add them to MANIFEST.in:
global-include *.py *.pyc
Including *.pyc into wheels works for me on Linux without any special configuration, I just do
python setup.py build
python -m compileall build
python setup.py sdist
python setup.py bdist_wheel
PS. If you're trying to hide sources from people — you're on the wrong track. *.pyc files can be decompiled. There is a disassembler in standard library and there are many decompilers and uncompilers out there.
I am currently writing a command line application in Python, which needs to be made available to end users in such a way that it is very easy to download and run. For those on Windows, who may not have Python (2.7) installed, I intend to use PyInstaller to generate a self-contained Windows executable. Users will then be able to simply download "myapp.exe" and run myapp.exe [ARGUMENTS].
I would also like to provide a (smaller) download for users (on various platforms) who already have Python installed. One option is to put all of my code into a single .py file, "myapp.py" (beginning with #! /usr/bin/env python), and make this available. This could be downloaded, then run using myapp.py [ARGUMENTS] or python myapp.py [ARGUMENTS]. However, restricting my application to a single .py file has several downsides, including limiting my ability to organize the code and making it difficult to use third-party dependencies.
Instead I would like to distribute the contents of several files of my own code, plus some (pure Python) dependencies. Are there any tools which can package all of this into a single file, which can easily be downloaded and run using an existing Python installation?
Edit: Note that I need these applications to be easy for end users to run. They are not likely to have pip installed, nor anything else which is outside the Python core. Using PyInstaller, I can generate a file which these users can download from the web and run with one command (or, if there are no arguments, simply by double-clicking). Is there a way to achieve this ease-of-use without using PyInstaller (i.e. without redundantly bundling the Python runtime)?
I don't like the single file idea because it becomes a maintenance burden. I would explore an approach like the one below.
I've become a big fan of Python's virtual environments because it allows you to silo your application dependencies from the OS's installation. Imagine a scenario where the application you are currently looking to distribute uses a Python package requests v1.0. Some time later you create another application you want to distribute that uses requests v2.3. You may end up with version conflicts on a system where you want to install both applications side-by-side. Virtual environments solve this problem as each application would have its own package location.
Creating a virtual environment is easy. Once you have virtualenv installed, it's simply a matter of running, for example, virtualenv /opt/application/env. Now you have an isolated python environment for your application. Additionally, virtual environments are very easy to clean up, simply remove the env directory and you're done.
You'll need a setup.py file to install your application into the environment. Say your application uses requests v2.3.0, your custom code is in a package called acme, and your script is called phone_home. Your directory structure looks like this:
acme/
__init__.py
models.py
actions.py
scripts/
phone_home
setup.py
The setup.py would look something like this:
from distutils.core import setup
install_requires = [
'requests==2.3.0',
]
setup(name='phone_home',
version='0.0.1',
description='Sample application to phone home',
author='John Doe',
author_email='john#doe.com',
packages=['acme'],
scripts=['scripts/phone_home'],
url='http://acme.com/phone_home',
install_requires=install_requires,
)
You can now make a tarball out of your project and host it however you wish (your own web server, S3, etc.):
tar cvzf phone_home-0.0.1.tar.gz .
Finally, you can use pip to install your package into the virtual environment you created:
/opt/application/env/bin/pip install http://acme.com/phone_home-0.0.1.tar.gz
You can then run phone_home with:
/opt/application/env/bin/phone_home
Or create a symlink in /usr/local/bin to simply call the script using phone_home:
ln -s /opt/application/env/bin/phone_home /usr/local/bin/phone_home
All of the steps above can be put in a shell script, which would make the process a single-command install.
And with slight modification this approach works really well for development environments; i.e. using pip to install / reference your development directory: pip install -e . where . refers to the current directory and you should be in your project directory alongside setup.py.
Hope this helps!
You could use pip as suggested in the comments. You need to create a MANIFEST.in and setup.py in your project to make it installable. You can also add modules as prerequisites. More info can be found in this question (not specific to Django):
How do I package a python application to make it pip-installable?
This will make your module available in Python. You can then have users run a file that runs your module, by either python path/run.py, ./path/run.py (with +x permission) or python -c "some code here" (e.g. for an alias).
You can even have users install from a git public reporitory, like this
pip install git+https://bitbucket.org/yourname/projectname.git
...in which case they also need git.
I have a simple python script, which imports various other modules I've written (and so on). Due to my environment, my PYTHONPATH is quite long. I'm also using Python 2.4.
What I need to do is somehow package up my script and all the dependencies that aren't part of the standard python, so that I can email a single file to another system where I want to execute it. I know the target version of python is the same, but it's on linux where I'm on Windows. Otherwise I'd just use py2exe.
Ideally I'd like to send a .py file that somehow embeds all the required modules, but I'd settle for automatically building a zip I can just unzip, with the required modules all in a single directory.
I've had a look at various packaging solutions, but I can't seem to find a suitable way of doing this. Have I missed something?
[edit] I appear to be quite unclear in what I'm after. I'm basically looking for something like py2exe that will produce a single file (or 2 files) from a given python script, automatically including all the imported modules.
For example, if I have the following two files:
[\foo\module.py]
def example():
print "Hello"
[\bar\program.py]
import module
module.example()
And I run:
cd \bar
set PYTHONPATH=\foo
program.py
Then it will work. What I want is to be able to say:
magic program.py
and end up with a single file, or possibly a file and a zip, that I can then copy to linux and run. I don't want to be installing my modules on the target linux system.
I found this useful:
http://blog.ablepear.com/2012/10/bundling-python-files-into-stand-alone.html
In short, you can .zip your modules and include a __main__.py file inside, which will enable you to run it like so:
python3 app.zip
Since my app is small I made a link from my main script to __main__.py.
Addendum:
You can also make the zip self-executable on UNIX-like systems by adding a single line at the top of the file. This may be important for scripts using Python3.
echo '#!/usr/bin/env python3' | cat - app.zip > app
chmod a+x app
Which can now be executed without specifying python
./app
Use stickytape module
stickytape scripts/blah --add-python-path . > /tmp/blah-standalone
This will result with a functioning script, but not necessarily human-readable.
You can try converting the script into an executable file.
First, use:
pip install pyinstaller
After installation type ( Be sure you are in your file of interest directory):
pyinstaller --onefile --windowed filename.py
This will create an executable version of your script containing all the necessary modules. You can then transfer (copy and paste) this executable to the PC or machine you want to run your script.
I hope this helps.
You should create an egg file. This is an archive of python files.
See this question for guidance: How to create Python egg file
Update: Consider wheels in 2019
The only way to send a single .py is if the code from all of the various modules were moved into the single script and they your'd have to redo everything to reference the new locations.
A better way of doing it would be to move the modules in question into subdirectories under the same directory as your command. You can then make sure that the subdirectory containing the module has a __init__.py that imports the primary module file. At that point you can then reference things through it.
For example:
App Directory: /test
Module Directory: /test/hello
/test/hello/__init__.py contents:
import sayhello
/test/hello/sayhello.py contents:
def print_hello():
print 'hello!'
/test/test.py contents:
#!/usr/bin/python2.7
import hello
hello.sayhello.print_hello()
If you run /test/test.py you will see that it runs the print_hello function from the module directory under the existing directory, no changes to your PYTHONPATH required.
If you want to package your script with all its dependencies into a single file (it won't be a .py file) you should look into virtualenv. This is a tool that lets you build a sandbox environment to install Python packages into, and manages all the PATH, PYTHONPATH, and LD_LIBRARY_PATH issues to make sure that the sandbox is completely self-contained.
If you start with a virgin Python with no additional libraries installed, then easy_install your dependencies into the virtual environment, you will end up with a built project in the virtualenv that requires only Python to run.
The sandbox is a directory tree, not a single file, but for distribution you can tar/zip it. I have never tried distributing the env so there may be path dependencies, I'm not sure.
You may need to, instead, distribute a build script that builds out a virtual environment on the target machine. zc.buildout is a tool that helps automate that process, sort of like a "make install" that is tightly integrated with the Python package system and PyPI.
I've come up with a solution involving modulefinder, the compiler, and the zip function that works well. Unfortunately I can't paste a working program here as it's intermingled with other irrelevant code, but here are some snippets:
zipfile = ZipFile(os.path.join(dest_dir, zip_name), 'w', ZIP_DEFLATED)
sys.path.insert(0, '.')
finder = ModuleFinder()
finder.run_script(source_name)
for name, mod in finder.modules.iteritems():
filename = mod.__file__
if filename is None:
continue
if "python" in filename.lower():
continue
subprocess.call('"%s" -OO -m py_compile "%s"' % (python_exe, filename))
zipfile.write(filename, dest_path)
Have you taken into considerations Automatic script creation of distribute the official packaging solution.
What you do is create a setup.py for you program and provide entry points that will be turned into executables that you will be able run. This way you don't have to change your source layout while still having the possibility to easily distribute and run you program.
You will find an example on a real app of this system in gunicorn's setup.py
I have been programming in Python for a while now, and have created some utilities that I use a lot. Whenever I start a new project, I start writing, and as I need these utilities I copy them from where ever I think the latest version of the particular utility is. I have enough projects now that I am losing track of where the latest version is. And, I will upgrade one of these scripts to fix a problem in a specific situation, and then wish it had propagated back to all of the other projects that use that script.
I am thinking the best way to solve this problem is to create a directory in the site-packages directory, and put all of my utility modules in there. And then add this directory to the sys.path directory list.
Is this the best way to solve this problem?
How do modify my installation of Python so that this directory is always added to sys.path, and I don't have to explicitly modify sys.path at the beginning of each module that needs to use these utilities?
I'm using Python 2.5 on Windows XP, and Wing IDE.
The site-packages directory within the Python lib directory should always be added to sys.path, so you shouldn't need to modify anything to take care of that. That's actually just what I'd recommend, that you make yourself a Python package within that directory and put your code in there.
Actually, something you might consider is packaging up your utilities using distutils. All that entails is basically creating a setup.py file in the root of the folder tree where you keep your utility code. The distutils documentation that I just linked to describes what should go in setup.py. Then, from within that directory, run
python setup.py install
to install your utility code into the system site-packages directory, creating the necessary folder structure automatically. Or you can use
python setup.py install --user
to install it into a site-packages folder in your own user account.
Add your directory to the PYTHONPATH environment variable. For windows, see these directions.
If it's not in site-packages then you can add a file with the extension .pth to your site-packages directory.
The file should have one path per line, that you want included in sys.path