I am attempting to install Scrapy on my Windows 7 64 bit machine. I started by following the instructions here on Scrapy's documentation.
I got up until the command 'pip install Scrapy'. Everything works except that it cannot find 'libxml2':
Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
I then visited this website to get the binaries of libxml2:
ftp://ftp.zlatkovic.com/libxml/64bit/
The instructions for installation of libxml2 are here: https://www.zlatkovic.com/libxml.en.html
They state that you should unzip the binaries and place the contents of the BIN folder in a path such as C:\WINDOWS. I did this. However, after attempting to install Scrapy again, I continue to receive the same error. Is there something I am missing?
Just installled Scrapy on Windows10 x64. It installs only with Python 2.7 on Windows. I used tutotial from here.
Download from the official site and install Python 2.7
Open Command promt (cmd) with full admin permissions and type:
c:\python27\python.exe c:\python27\tools\scripts\win_add2path.py
and press Enter
Install Microsoft Visual C++ for Python 2.7
Install OpenSSL (sorry, can't attach link because of small reputation). Download it to some folder, e.g. C:\Python27\Scripts and execute command from there: pip install pyOpenSSL-16.0.0-py2.py3-none-any.whl (the name of file could not be the same)
Install lxml just like previous file: pip install lxml-3.6.0-cp27-cp27m-win32.whl
Install Pywin32
And finally, install Scrapy - pip install Scrapy
Can recommend to download and install all needed soft for x32 architecture (even if you have x64 system).
It is recommended to use conda to install Scrapy on Windows.
You can download Miniconda from here: http://conda.pydata.org/miniconda.html
Once you have conda installed you can follow this guide to learn about its subcommands: http://conda.pydata.org/docs/test-drive.html
In short, you can use this commands in your terminal:
Create an environment for scrapy: conda create -n scrapyenv python=2
Enable the scrapy environment: activate scrapyenv
Install scrapy: conda install -c scrapinghub scrapy
Edit: Scrapy's conda package has been moved to conda-forge channel. Use the command: conda install -c conda-forge scrapy
Hi this is what I did to get it installed under Win7/Win10:
easy_install lxml
pip install Scrapy
install pywin32 as indicated by the docs
Then, follow the tutorial to be sure everything is working
Related
I'm getting an error Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed? when trying to install lxml through pip.
c:\users\f\appdata\local\temp\xmlXPathInitqjzysz.c(1) : fatal error C1083: Cannot open include file: 'libxml/xpath.h': No such file or directory
*********************************************************************************
Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
*********************************************************************************
error: command 'C:\\Users\\f\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\VC\\Bin\\cl.exe' failed with exit status 2
I don't find any libxml2 dev packages to install via pip.
Using Python 2.7 and Python 3.x on x86 in a virtualenv under Windows 10.
Install lxml from http://www.lfd.uci.edu/~gohlke/pythonlibs/#lxml for your python version. It's a precompiled WHL with required modules/dependencies.
The site lists several packages, when e.g. using Win32 Python 3.11, use lxml‑4.9.0‑cp311‑cp311‑win32.whl.
Download the file, and then install with:
pip install C:\path\to\downloaded\file\lxml‑4.9.0‑cp311‑cp311‑win32.whl
I had this issue and realised that whilst I did have libxml2 installed, I didn't have the necessary development libraries required by the python package. Installing them solved the problem:
sudo apt-get install libxml2-dev libxslt1-dev
sudo pip install lxml
Try to use:
easy_install lxml
That works for me, win10, python 2.7.
On Mac OS X El Capitan I had to run these two commands to fix this error:
xcode-select --install
pip install lxml
Which ended up installing lxml-3.5.0
When you run the xcode-select command you may have to sign a EULA (so have an X-Term handy for the UI if you're doing this on a headless machine).
In case anyone else has the same issue as this on
Centos, try:
yum install python-lxml
Ubuntu
sudo apt-get install -y python-lxml
worked for me.
set STATICBUILD=true && pip install lxml
run this command instead, must have VS C++ compiler installed first
https://blogs.msdn.microsoft.com/pythonengineering/2016/04/11/unable-to-find-vcvarsall-bat/
It works for me with Python 3.5.2 and Windows 7
I tried install a lib that depends lxml and nothing works. I see a message when build was started: "Building without Cython", so after install cython with apt-get install cython, lxml was installed.
I had this issue and realized that while I did have libxml2 installed, I didn't have the necessary development libraries required by the python package.
1) Installing them solved the problem:
The site to download the file: Download
2) After Installing the file save it in a accessible folder
pip install *path to that file*
For some reason it doesn't work in python 3.11, but 3.10 works.
On windows, to install a module with a previous version, use
py -3.10 -m pip install lxml
if you want to install it in a venv, then use
py -3.10 -m venv .venv
.venv/Scripts/pip.exe install lxml
if you've set up the venv, then you can just use
pip install lxml
You also need to run the python program with that version. If you set up a venv, then you don't need to do this.
py -3.10 file.py
It is not strange for me that none of the solutions above came up, but I saw how the igd installation removed the new version and installed the old one, for the solution I downloaded this archive:https://pypi.org/project/igd/#files
and changed the recommended version of the new version: 'lxml==4.3.0' in setup.py
It works!
I got the same error for python 32 bit. After install 64bit, the problem was fixed.
I am using venv.
In my case it was enough to add lxml==4.6.3 to requirements.txt.
One library wanted earlier version and this was causing this error, so when I forced pip to use newest version (currently 4.6.3) installation was successful.
I am stuck on this installation of scikit-image (aka skimage). I tried multiple ways:
Installation from a git hub folder (using the requirements.txt)
Installation from a whl file
Installation with pip install scikit-image
All three trials failed during the import: import skimage
ImportError: cannot import name 'geometry'
It seems that scikit-image has not been built correctly.
Your install of scikit-image appears to be broken.
Try re-installing the package following the instructions at:
https://scikit-image.org/docs/stable/install.html
I went through internet but did not find solutions besides the ones above.
Does anyone went through that before?
One possibility seems to be to pip install with -egg, but I found that for Mac rather than Windows.
EDIT:
Hi everyone, so I found a solution but this is kind of very hard and I still don't understand why it did not work before.
I just:
uninstall anaconda
uninstall python
install python (3.8)
install Anaconda (I have trouble with Spyder now^^)
If you want to code for computer vision/Image processing/machine learning tasks, then it can be done in pycharm with conda environment very easily. There is no need to install python separately to run Anaconda.
First, download and install pycharm. Next, If you use windows then download Anaconda 64 bit python 3.7 version from here,
https://www.anaconda.com/distribution/#windows
You can find some details about managing environment and helpful links here,
How to create some environments for tensorflow in anaconda?
https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html
Create a conda environment first using Anaconda Prompt command line,
conda create -n cvenv python=3.7
Now activate that environment using. By default base is activated.
conda activate cvenv
Next, install the packages you want,
conda install -c conda-forge scikit-learn
conda install -c conda-forge scikit-image
conda install -c conda-forge pillow
conda install -c conda-forge opencv
I use conda-forge packages as they are more recent. Finally, open pycharm and create a new project by selecting existing python interpreter in conda environment. If none exists then select,
Browse > Add Python Interpreter > Conda Environment > Interpreter > Browse > Anaconda3 installation folder > envs folder > cvenv folder > Python.exe
You can test by creating a python file and writing import skimage.
I`m disparately trying to install win32.client which I know is a part of pywin32 module.
I have tried with:
pip install
pip install and downloaded different versions of .whl files
.zip install from cmd (see pics bellow)
My version of python is : Info of my python version
When I tried with pip install:
I get this error
When I tried with pip install and downloaded .whl file:
I get this error
When I tried with .zip file:
enter image description here
Do someone has any ideea why is not working ?
I also uninstalled python and installed again
I suggest you uninstall from pip and install with the .exe installer from the official repository.
This has always worked for me. Remember to select the right version for your version of python.
You may try installing with:
conda install -c anaconda pywin32
But from the warning;
You can upgrade to a newer version of Python to solve this.
I would upgrade to python 3 and try again.
I tried to setup Scrapy on Windows 7 by steps described at http://doc.scrapy.org/en/latest/intro/install.html . On my PC was installed Python 3.5.1. Although Scrapy not support this python version it was installed successfully with latest Anaconda but fails to run spider script. I find that Scrapy only works with Python 3.3.+ version so uninstall version 3.5.1, uninstall Anaconda, install python 3.3.5, install pywin32 and install pip. pip fails pip install Scrapy, so I install Anaconda and run conda install -c scrapinghub scrapy Scrapy installed, but I saw that libs installed was for python 3.5 like: scrapy: 1.1.0-py35_0
Now I run the
c:\python\olxscrapy>scrapy crawl OlxCatalogSpider
and get error
File "C:\Anaconda3\lib\site-packages\twisted\internet\stdio.py", line 30, in
module>
from twisted.internet import _win32stdio
ImportError: cannot import name '_win32stdio'
How to make Scrapy run with python 3.3.+
On this blog:
https://blog.scrapinghub.com/2016/05/25/data-extraction-with-scrapy-and-python-3/
it says Scrapy on Python 3 doesn't work in Windows environments yet
Edit:
I recently installed scrapy on Ubuntu for Python 3.5 and received a lot of errors. The errors stopped after: "sudo apt-get install python3.5-dev".
I add the follow package and it works:
pip install twisted-win==0.5.5
Try to create a virtual env:
pip install virtualenv (instalation)
virtualenv -p python3.3.5 envName (creation with specific python version)
source ./envName/bin/activate (activate virtual env)
This way you can guarantee that's the right python version. Also scrapy has some requirements that can't be installed via pip and this may cause your pip install scrapy to fail
So install at your computer:
python-dev
libxslt1-dev
libxslt1.1
libxml2-dev
libxml2
libssl-dev
After this you finaly be able to install scrapy via pip inside your virtual env (probably)
Sry for my poor English isn't my native lang. Hope this work =]
Installation of Scrapy on Windows may facing error while installing Twisted.
Download Twisted according to your Python and windows version on this site http://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted
Turn to your download folder and pip install <downloaded filename>
pip install scrapy
I am getting started with Scrapy, but I have two problems with installation on Linux Mint 17.2 (Ubuntu based version).
I don't get which is the difference from installing pip install
scrapy and sudo apt-get install scrapy
When i do install one of the two and I try to follow the first tutorial of Scrapy using the command scrapy startproject tutorialit gives me error /usr/bin: No such file or directory.
I have tried to uninstall and reinstall many times but still doesn't work.
1. Instalation Source
Both commands pip install scrapy and sudo apt-get install scrapy will install Scrapy on your computer, but the versions may be different. The pip option installs the latest version for Scrapy 1.0, while the one in your repositories its probably outdated.
If anyway you want to install the package from the repositories and still keep it updated, you can add the Scrapy repository:
http://doc.scrapy.org/en/1.0/topics/ubuntu.html
echo 'deb http://archive.scrapy.org/ubuntu scrapy main' | sudo tee /etc/apt-sources.list.d/scrapy.list
sudo apt-get update && sudo apt-get install scrapy
Souce: http://doc.scrapy.org/en/1.0/topics/ubuntu.html
2. Check the PATH variable
Depending on the way you install Scrapy, the binaries folder for the installation might be different. In my case I have it in /usr/local/bin.
Display your PATH variable with echo "$PATH", and check if the folder with the Scrapy binary is included.
You can add more directories to the variable with export PATH=$PATH:/path/to/dir
The installation guide tells not to use the packages provided by Ubuntu:
Don’t use the python-scrapy package provided by Ubuntu, they are
typically too old and slow to catch up with latest Scrapy.
Instead, use the official Ubuntu
Packages,
which already solve all dependencies for you and are continuously
updated with the latest bug fixes.
As mentioned you should install it using the Ubuntu packages on this page instead.
Besides the previous steps, I also had to install service-identity:
sudo pip install service-identity