This question already has answers here:
Installing lxml, libxml2, libxslt on Windows 8.1
(4 answers)
Closed 6 years ago.
Actually Iam trying to installing lxml because of
UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("html.parser"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.
So Iam trying to use lxml.but I could not install due to this error
I tried to install manually by downloading the latest version also.But couldn't make it.Can any one please help me to sort this out
Probabily you didin't installed the requirements for lxml package. Follow this tutorial: http://lxml.de/installation.html
Pay extra attention to this excert:
Most MS Windows systems lack the necessarily tools to build software,
starting with a C compiler already. Microsoft leaves it to users to
install and configure them, which is usually not trivial and means
that distributors cannot rely on these dependencies being available on
a given system. In a way, you get what you've paid for and make others
pay for it.
Due to the additional lack of package management of this platform, it
is best to link the library dependencies statically if you decide to
build from sources, rather than using a binary installer. For that,
lxml can use the binary distribution of libxml2 and libxslt, which it
downloads automatically during the static build. It needs both libxml2
and libxslt, as well as iconv and zlib, which are available from the
same download site. Further build instructions are in the source build
documentation.
Related
The advantage of wheels over eggs is clear (see section why not egg? https://pypi.python.org/pypi/wheel).
However, it is not entirely clear to me what is the advantage of using wheels over tar.gz. I might be missing something obvious like "they are the same".
As I see it both can be installed directly using pip (even in Windows), have similar size and when packaging require a similar effort.
It sounds to me like the kind of questions you might get when justifying a packaging methodology.
EDIT:
Just found an example where tar.gz might be better than wheels. CherryPy (https://pypi.python.org/pypi/CherryPy) provides wheels for Python 3.x only, so if you want to have a local repository to serve CherryPy for Python 2.7 and 3.x dependencies, it seems to make more sense to store the tarball. Is this correct? (just to add a couple of "case-based" justification to the discussion)
This answered it for me (directly from the wheel PEP):
Python needs a package format that is easier to install than sdist.
Python's sdist packages are defined by and require the distutils and
setuptools build systems, running arbitrary code to build-and-install,
and re-compile, code just so it can be installed into a new
virtualenv. This system of conflating build-install is slow, hard to
maintain, and hinders innovation in both build systems and installers.
Wheel attempts to remedy these problems by providing a simpler
interface between the build system and the installer. The wheel binary
package format frees installers from having to know about the build
system, saves time by amortizing compile time over many installations,
and removes the need to install a build system in the target
environment.
https://www.python.org/dev/peps/pep-0427/#rationale
Note the tarballs we're speaking of are what are referred to as "sdists" above.
From Python Wheels
Advantages of wheels
• Faster installation for pure python and native C extension packages.
• Avoids arbitrary code execution for installation. (Avoids setup.py)
• Installation of a C extension does not require a compiler on Windows or OS X.
• Allows better caching for testing and continuous integration.
• Creates .pyc files as part of installation to ensure they match the python interpreter used.
• More consistent installs across platforms and machines.
Make sure wheel is installed.
python3 -m pip install wheel
I followed the AWS guide to prepare a deployment package for my lambda function. The generated zip file is around 9 - 10MB which includes pip, setuptools, pylint. Are they really required?
Here are the commands.
virtualenv v-env
source v-env/bin/activate
pip install xmltodict
pip install requests
deactivate
cd v-env/lib/python3.7/site-packages/
zip -r9 ../../../../function.zip .
Edit: Remove installing boto as it is provided by AWS already
Well, as you may see the guides provide standars and, obviously, guidance for a clean and nice coding or project deployment.
Pylint has a lot of features that help you out while using Python as your programming language, such as checking coding standars, error detection, refactoring help in order to prevend duplicated code, among other tools.
Setuptools is really useful too. Is a development process library designed to facilitate packaging Python projects by enhancing the Python standard library distribution utilities, I encourage you to use it in order to wrap your processes and models in order to have a strong modular project.
And pip is a Package Manager for python packages or modules. You can add, download, delete and a lot more things with it by simple using few words on a line of code. This package manager is useful, you can download wheels, zip's and modules from the internet and easily install them by just using
pip install <module or library name>
So, by answering your question, if you downloaded and installed a package for AWS supported in Python and it installed those libraries I must think those are being used across the modules you want to use.
You can always check the source code in order to be sure.
If the libraries aren't really being used, they aren't necessary since there are several libraries and packages that do what those libraries do.
Hope it helps, happy coding.
Assume that someone wants to package a Python (Cython) library that depends on the C++ boost library.
What is the best way to configure the setup.py so that the user is properly informed that it is required to install the boost library (i.e., apt-get install libboost-dev in Ubuntu, etc in other OSes)? Or is it a better practice to include the boost library in the python package distribution?
The question is better asked as
What is the best way to distribute a Python extension including
an external library dependency.
This is better dealt with binary wheel packages.
User does not need to know anything about setup.py, which is used for building and installing source code. User just needs to download and install a binary wheel package.
Including just the header files does not solve the problem of needing the library to build with and link to. It also opens up issues with version incompatibilities.
So setup.py need not have anything special about any of this, it just needs to know where to find headers which will be a sub-dir in your project if the library is included and which libraries to link with.
The documentation should include instructions on how to build from source, for which more than just boost is needed (python header files, appropriate compilers etc).
Tools like auditwheel then take care of bundling external library dependencies into the binary wheel, so end-users need not have the library installed to use your package.
See also manylinux for distributing binary Python extensions and this demo project.
The installation of the virtual environment of my python application takes too much time during deployment due to a large amount of dependencies. In order to minize that time, I want to include the dependencies residing in the virtual environment in git, so that they are already there on deployment.
The main issue with that is that dependencies with C code need to be rebuild because of architecture differences between machines.
Is there a way to rebuild all dependencies that need compilation in my virtual environment?
wheel format is what you need
Popular example is lxml, which when installing from source on Linux takes about 3 minutes to get downloaded, compiled and installed.
Using wheel format and installing from local wheel file for lxml installs within fraction of a second.
For detailed instructions how I use it see corrected link to Detailed SO answer how to configure pip incl. instructions how to take advantage of wheels
For more information:
pythonwheels page listing already available wheels.
wheel ReadTheDocs
using pip to build a wheel
Some notes:
pure python packages can be distributed in wheel format regardless of target platform (apart from being possibly python version dependent).
compiled python packages shall be build on the same platform, where you are going to install them. There might be some cross-compile options, but I do not have real experience with that.
some do consider wheel of "package format of the future", others claim, that it is supposed to be build on your own side and use your own wheels. The later case is for lxml not being provided as a wheel - see launchpad issue related to lxml in wheel format. Consider adding there yourself as an affected person, if you care.
Once you manage using wheels the first time, you will love it.
I recently began learning Python, and I am a bit confused about how packages are distributed and installed.
I understand that the official way of installing packages is distutils: you download the source tarball, unpack it, and run: python setup.py install, then the module will automagically install itself
I also know about setuptools which comes with easy_install helper script. It uses eggs for distribution, and from what I understand, is built on top of distutils and does the same thing as above, plus it takes care of any dependencies required, all fetched from PyPi
Then there is also pip, which I'm still not sure how it differ from the others.
Finally, as I am on a windows machine, a lot of packages also offers binary builds through a windows installer, especially the ones that requires compiling C/Fortran code, which otherwise would be a nightmare to manually compile on windows (assumes you have MSVC or MinGW/Cygwin dev environment with all necessary libraries setup.. nonetheless try to build numpy or scipy yourself and you will understand!)
So can someone help me make sense of all this, and explain the differences, pros/cons of each method. I'd like to know how each keeps track of packages (Windows Registry, config files, ..). In particular, how would you manage all your third-party libraries (be able to list installed packages, disable/uninstall, etc..)
I use pip, and not on Windows, so I can't provide comparison with the Windows-installer option, just some information about pip:
Pip is built on top of setuptools, and requires it to be installed.
Pip is a replacement (improvement) for setuptools' easy_install. It does everything easy_install does, plus a lot more (make sure all desired distributions can be downloaded before actually installing any of them to avoid broken installs, list installed distributions and versions, uninstall, search PyPI, install from a requirements file listing multiple distributions and versions...).
Pip currently does not support installing any form of precompiled or binary distributions, so any distributions with extensions requiring compilation can only be installed if you have the appropriate compiler available. Supporting installation from Windows binary installers is on the roadmap, but it's not clear when it will happen.
Until recently, pip's Windows support was flaky and untested. Thanks to a lot of work from Dave Abrahams, pip trunk now passes all its tests on Windows (and there's a continuous integration server helping us ensure it stays that way), but a release has not yet been made including that work. So more reliable Windows support should be coming with the next release.
All the standard Python package installation mechanisms store all metadata about installed distributions in a file or files next to the actual installed package(s). Distutils uses a distribution_name-X.X-pyX.X.egg-info file, pip uses a similarly-named directory with multiple metadata files in it. Easy_install puts all the installed Python code for a distribution inside its own zipfile or directory, and places an EGG-INFO directory inside that directory with metadata in it. If you import a Python package from the interactive prompt, check the value of package.__file__; you should find the metadata for that package's distribution nearby.
Info about installed distributions is only stored in any kind of global registry by OS-specific packaging tools such as Windows installers, Apt, or RPM. The standard Python packaging tools don't modify or pay attention to these listings.
Pip (or, in my opinion, any Python packaging tool) is best used with virtualenv, which allows you to create isolated per-project Python mini-environments into which you can install packages without affecting your overall system. Every new virtualenv automatically comes with pip installed in it.
A couple other projects you may want to be aware of as well (yes, there's more!):
distribute is a fork of setuptools which has some additional bugfixes and features.
distutils2 is intended to be the "next generation" of Python packaging. It is (hopefully) adopting the best features of distutils/setuptools/distribute/pip. It is being developed independently and is not ready for use yet, but eventually should replace distutils in the Python standard library and become the de facto Python packaging solution.
Hope all that helped clarify something! Good luck.
I use windows and python. It is somewhat frustrating, because pip doesn't always work to install things. Python is moving to pip, so I still use it. Pip is nice, because you can uninstall items and use
pip freeze > requirements.txt
pip install -r requirements.txt
Another reason I like pip is for virtual environments like venv with python 3.4. I have found venv a lot easier to use on windows than virtualenv.
If you cannot install a package you have to find the binary for it. http://www.lfd.uci.edu/~gohlke/pythonlibs/
I have found these binaries to be very useful.
Pip is trying to make something called a wheel for binary installations.
pip install wheel
wheel convert path\to\binary.exe
pip install converted_wheel.whl
You will also have to do this for any required libraries that do not install and are required for that package.