I followed the AWS guide to prepare a deployment package for my lambda function. The generated zip file is around 9 - 10MB which includes pip, setuptools, pylint. Are they really required?
Here are the commands.
virtualenv v-env
source v-env/bin/activate
pip install xmltodict
pip install requests
deactivate
cd v-env/lib/python3.7/site-packages/
zip -r9 ../../../../function.zip .
Edit: Remove installing boto as it is provided by AWS already
Well, as you may see the guides provide standars and, obviously, guidance for a clean and nice coding or project deployment.
Pylint has a lot of features that help you out while using Python as your programming language, such as checking coding standars, error detection, refactoring help in order to prevend duplicated code, among other tools.
Setuptools is really useful too. Is a development process library designed to facilitate packaging Python projects by enhancing the Python standard library distribution utilities, I encourage you to use it in order to wrap your processes and models in order to have a strong modular project.
And pip is a Package Manager for python packages or modules. You can add, download, delete and a lot more things with it by simple using few words on a line of code. This package manager is useful, you can download wheels, zip's and modules from the internet and easily install them by just using
pip install <module or library name>
So, by answering your question, if you downloaded and installed a package for AWS supported in Python and it installed those libraries I must think those are being used across the modules you want to use.
You can always check the source code in order to be sure.
If the libraries aren't really being used, they aren't necessary since there are several libraries and packages that do what those libraries do.
Hope it helps, happy coding.
Related
I have a git repository with a single python script which I use for some task. I want to create a 'package' for this script and then install it. Previously I was using cmake to do this, but I'm wondering what the pythonic way of doing it is.
I tried using setuptools' console_scripts keyword argument, but that didn't work.
This script should get installed into some ./bin directory.
The pythonic way here would be to create an actual package https://packaging.python.org/en/latest/tutorials/packaging-projects/ and either building something like a .whl and installing that using pip or conda (whichever you use for managing your virtual environment), or publishing the package on PyPI for general use and easy installation on any machine with internet access.
Packaging a project is fairly standard and the linked documentation is official. Getting a package onto PyPI is fairly straightforward as well, although there are many options you can add to make the installation and the accompanying page nicer - there's many guides online on how to publish a package on PyPI, but none of then official documentation, so I suggest just searching for one.
The answer here was to use the scripts keyword instead of console_scripts. my setup.py was then able to properly install the script.
I am trying to use Pythonanywhere to run some of my scripts. Eventually I would like to turn the entire code into a webapp, but I am not there yet.
Anyway, I would like to install a couple of bioinformatics packages on Pythonanywhere. Biopython has wrappers for these tools, but you still need the actual tools instaled. The tools in question do not have pip or easyinstall compatibility. So I think I will have to download/compile from source? However, I keep on running into "no permission" issues. Would a virtualenv be a solution?
As always, any help is greatly appreciated
Virtualenv works for Python packages.
However if you're trying to use non Python packages/libraries it won't be of much use.
Since the tools don't have easyinstall or pip compatibility I don't think using virtualenv will work out.
Download and compile from source should work though.
Note that if Biopython is installable via easyinstall or pip then use virtualenv to get that.
I'm an author of a pure Python library that aims to be also convenient to use from a command line. For Windows users it would be nice just installing the package from an .exe or .msi package.
However I cannot get the installer to install package dependencies (especially the dependency on setuptools itself, so that running the software fails with an import error on pkg_resources). I don't believe that providing an easy .exe installer makes much sense, if the user then needs to manually install setuptools and other libraries on top. I'd rather tell them how to add easy_install to their PATH and go through this way (http://stackoverflow.com/questions/1449494/how-do-i-install-python-packages-on-windows).
I've build .exe packages in the past, but don't remember if that ever worked the way I'd preferred it to.
It is quite common to distribute packages that have dependencies, especially those as you have, but I understand your wish to make installation as simple as possible.
Have a look at deployment bootstrapper, a tool dedicated to solving the problem of delivering software including its prerequisites.
Regardless of what packaging method you eventually choose, maintain your sanity by staying away from including MSIs in other MSI in any way. That just does not work because of transactional installation requirements and locking of the Windows Installer database.
I have a Python library I've written that interfaces with a Java application. Note that this is specifically a library for other developers to use, there is no main() involved. I've found plenty of information on packaging Jython web applications into jars to be deployed on application servers, but I can't seem to find anything on packaging a library to be installed for others to use.
This is a pretty simple scenario of a single package with 4 modules. Effectively what I'm looking for is the standard CPython methodology of having a setup.py using distutils and running:
python setup.py sdist
How can I accomplish the same via Jython?
If it matters:
jython --version
Jython 2.2.1 on java1.6.0_20
There is an implementation of disutils in jython but it doesn't work for me. For now you might have to write a script to copy your python files into jython/Lib/site-packages, which I don't think will be too hard if you're just installing python source, see shutil.copytree.
it's strongly recommend to use virtualenv so you can get most up-to-date setuptools and other utils by pip.
I also suggest you to take a look at jip, which will help you to define java dependencies in setup.py. The dependencies will be resolved automatically when user install your module with:
pip install ...
or
jython setup.py install
Please upgrade to jython 2.5.x to run it.
I recently began learning Python, and I am a bit confused about how packages are distributed and installed.
I understand that the official way of installing packages is distutils: you download the source tarball, unpack it, and run: python setup.py install, then the module will automagically install itself
I also know about setuptools which comes with easy_install helper script. It uses eggs for distribution, and from what I understand, is built on top of distutils and does the same thing as above, plus it takes care of any dependencies required, all fetched from PyPi
Then there is also pip, which I'm still not sure how it differ from the others.
Finally, as I am on a windows machine, a lot of packages also offers binary builds through a windows installer, especially the ones that requires compiling C/Fortran code, which otherwise would be a nightmare to manually compile on windows (assumes you have MSVC or MinGW/Cygwin dev environment with all necessary libraries setup.. nonetheless try to build numpy or scipy yourself and you will understand!)
So can someone help me make sense of all this, and explain the differences, pros/cons of each method. I'd like to know how each keeps track of packages (Windows Registry, config files, ..). In particular, how would you manage all your third-party libraries (be able to list installed packages, disable/uninstall, etc..)
I use pip, and not on Windows, so I can't provide comparison with the Windows-installer option, just some information about pip:
Pip is built on top of setuptools, and requires it to be installed.
Pip is a replacement (improvement) for setuptools' easy_install. It does everything easy_install does, plus a lot more (make sure all desired distributions can be downloaded before actually installing any of them to avoid broken installs, list installed distributions and versions, uninstall, search PyPI, install from a requirements file listing multiple distributions and versions...).
Pip currently does not support installing any form of precompiled or binary distributions, so any distributions with extensions requiring compilation can only be installed if you have the appropriate compiler available. Supporting installation from Windows binary installers is on the roadmap, but it's not clear when it will happen.
Until recently, pip's Windows support was flaky and untested. Thanks to a lot of work from Dave Abrahams, pip trunk now passes all its tests on Windows (and there's a continuous integration server helping us ensure it stays that way), but a release has not yet been made including that work. So more reliable Windows support should be coming with the next release.
All the standard Python package installation mechanisms store all metadata about installed distributions in a file or files next to the actual installed package(s). Distutils uses a distribution_name-X.X-pyX.X.egg-info file, pip uses a similarly-named directory with multiple metadata files in it. Easy_install puts all the installed Python code for a distribution inside its own zipfile or directory, and places an EGG-INFO directory inside that directory with metadata in it. If you import a Python package from the interactive prompt, check the value of package.__file__; you should find the metadata for that package's distribution nearby.
Info about installed distributions is only stored in any kind of global registry by OS-specific packaging tools such as Windows installers, Apt, or RPM. The standard Python packaging tools don't modify or pay attention to these listings.
Pip (or, in my opinion, any Python packaging tool) is best used with virtualenv, which allows you to create isolated per-project Python mini-environments into which you can install packages without affecting your overall system. Every new virtualenv automatically comes with pip installed in it.
A couple other projects you may want to be aware of as well (yes, there's more!):
distribute is a fork of setuptools which has some additional bugfixes and features.
distutils2 is intended to be the "next generation" of Python packaging. It is (hopefully) adopting the best features of distutils/setuptools/distribute/pip. It is being developed independently and is not ready for use yet, but eventually should replace distutils in the Python standard library and become the de facto Python packaging solution.
Hope all that helped clarify something! Good luck.
I use windows and python. It is somewhat frustrating, because pip doesn't always work to install things. Python is moving to pip, so I still use it. Pip is nice, because you can uninstall items and use
pip freeze > requirements.txt
pip install -r requirements.txt
Another reason I like pip is for virtual environments like venv with python 3.4. I have found venv a lot easier to use on windows than virtualenv.
If you cannot install a package you have to find the binary for it. http://www.lfd.uci.edu/~gohlke/pythonlibs/
I have found these binaries to be very useful.
Pip is trying to make something called a wheel for binary installations.
pip install wheel
wheel convert path\to\binary.exe
pip install converted_wheel.whl
You will also have to do this for any required libraries that do not install and are required for that package.