I tried to setup Scrapy on Windows 7 by steps described at http://doc.scrapy.org/en/latest/intro/install.html . On my PC was installed Python 3.5.1. Although Scrapy not support this python version it was installed successfully with latest Anaconda but fails to run spider script. I find that Scrapy only works with Python 3.3.+ version so uninstall version 3.5.1, uninstall Anaconda, install python 3.3.5, install pywin32 and install pip. pip fails pip install Scrapy, so I install Anaconda and run conda install -c scrapinghub scrapy Scrapy installed, but I saw that libs installed was for python 3.5 like: scrapy: 1.1.0-py35_0
Now I run the
c:\python\olxscrapy>scrapy crawl OlxCatalogSpider
and get error
File "C:\Anaconda3\lib\site-packages\twisted\internet\stdio.py", line 30, in
module>
from twisted.internet import _win32stdio
ImportError: cannot import name '_win32stdio'
How to make Scrapy run with python 3.3.+
On this blog:
https://blog.scrapinghub.com/2016/05/25/data-extraction-with-scrapy-and-python-3/
it says Scrapy on Python 3 doesn't work in Windows environments yet
Edit:
I recently installed scrapy on Ubuntu for Python 3.5 and received a lot of errors. The errors stopped after: "sudo apt-get install python3.5-dev".
I add the follow package and it works:
pip install twisted-win==0.5.5
Try to create a virtual env:
pip install virtualenv (instalation)
virtualenv -p python3.3.5 envName (creation with specific python version)
source ./envName/bin/activate (activate virtual env)
This way you can guarantee that's the right python version. Also scrapy has some requirements that can't be installed via pip and this may cause your pip install scrapy to fail
So install at your computer:
python-dev
libxslt1-dev
libxslt1.1
libxml2-dev
libxml2
libssl-dev
After this you finaly be able to install scrapy via pip inside your virtual env (probably)
Sry for my poor English isn't my native lang. Hope this work =]
Installation of Scrapy on Windows may facing error while installing Twisted.
Download Twisted according to your Python and windows version on this site http://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted
Turn to your download folder and pip install <downloaded filename>
pip install scrapy
I'm trying to run a scraping program I wrote for in python using scrapy on an ubuntu machine. Scrapy is installed. I can import until python no problem and when try pip install scrapy I get
Requirement already satisfied (use --upgrade to upgrade): scrapy in /system/linux/lib/python2.7/dist-packages
When I try to run scrapy from the command, with scrapy crawl ... for example, I get.
The program 'scrapy' is currently not installed.
What's going on here? Are the symbolic links messed up? And any thoughts on how to fix it?
Without sudo, pip installs into $HOME/.local/bin, $HOME/.local/lib, etc. Add the following line to your ~/.bashrc or ~/.profile (or the appropriate place for other shells):
export PATH="${PATH}:${HOME}/.local/bin"
then open a new terminal or reload .bashrc, and it should find the command.
I had the same error. Running scrapy in a virtual environment solved it.
Create a virtual env : python3 -m venv env
Activate your env : source env/bin/activate
Install Scrapy with pip : pip install scrapy
Start your crawler : scrapy crawl your_project_name_here
For example my project name was kitten, I just did the following in step 4
scrapy crawl kitten
NOTE: I did this on Mac OS running Python 3+
I tried the following sudo pip install scrapy , however was promtly advised by Ubuntu 16.04 that it was already installed.
I had to first use sudo pip uninstall scrapy, then sudo pip install scrapy for it to successfully install.
Now you should successfully be able to run scrapy.
I faced the same problem and solved using following method. I think scrapy is not usable by the current user.
Uninstall scrapy.
sudo pip uninstall scrapy
Install scrapy again using -H.
sudo -H pip install scrapy
Should work properly.
If you install scrapy only in virtualenv, then scrapy command isn't exists in your system bin directory. You could check it like this:
$ which scrapy
For me it is in(because I sudo installed it):
/usr/local/bin/scrapy
You could try full path to your scrapy.
For example if it is installed in virtualenv:
(env) linux#hero:~dev/myscrapy$ python env/bin/scrapy
Note: We recommend installing Scrapy inside a virtual environment on all platforms.
I had the same issue. sudo pip install scrapy fixed my problem, although I don't know why must use sudo.
make sure you activate command that is
"Scripts\activate.bat"
A good way to work around is using pyenv to manage the python version.
$ brew install pyenv
# Any version 3.6 or above
$ pyenv install 3.7.3
$ pyenv global 3.7.3
# Update Environment to reflect correct python version controlled by pyenv
$ echo -e '\nif command -v pyenv 1>/dev/null 2>&1; then\n eval "$(pyenv init -)"\nfi' >> ~/.zshrc
# Refresh Terminal
# or source $~/.zshrc
$ exec $0
$ which python
/Users/mbbroberg/.pyenv/shims/python
$ python -V
Python 3.7.3
# Install scrapy
$ pip install scrapy
$ scrapy --version
Reference:
https://opensource.com/article/19/5/python-3-default-mac
scrapy crawl is not how you start a scrapy program. You start it by doing
scrapy startproject myprojectname
Then to actually start a scrapy program go into myprojectname/spiders and then you can call
scrapy crawl "yourspidername"
To have scrapy create a spider you can cd into your directory and execute
scrapy genspider mydomain mydomain.com
Additionally, you can test if your scrapy actually works by executing
scrapy shell "google.com"
All this information can be found in their Documentation.
If something happens then you have actually installed scrapy and you are crawling (haha) your way to success!
P.S. Scrapy does not work well on Python3 so if you're running on there and you still have troubles, use Python 2.7!
I am getting started with Scrapy, but I have two problems with installation on Linux Mint 17.2 (Ubuntu based version).
I don't get which is the difference from installing pip install
scrapy and sudo apt-get install scrapy
When i do install one of the two and I try to follow the first tutorial of Scrapy using the command scrapy startproject tutorialit gives me error /usr/bin: No such file or directory.
I have tried to uninstall and reinstall many times but still doesn't work.
1. Instalation Source
Both commands pip install scrapy and sudo apt-get install scrapy will install Scrapy on your computer, but the versions may be different. The pip option installs the latest version for Scrapy 1.0, while the one in your repositories its probably outdated.
If anyway you want to install the package from the repositories and still keep it updated, you can add the Scrapy repository:
http://doc.scrapy.org/en/1.0/topics/ubuntu.html
echo 'deb http://archive.scrapy.org/ubuntu scrapy main' | sudo tee /etc/apt-sources.list.d/scrapy.list
sudo apt-get update && sudo apt-get install scrapy
Souce: http://doc.scrapy.org/en/1.0/topics/ubuntu.html
2. Check the PATH variable
Depending on the way you install Scrapy, the binaries folder for the installation might be different. In my case I have it in /usr/local/bin.
Display your PATH variable with echo "$PATH", and check if the folder with the Scrapy binary is included.
You can add more directories to the variable with export PATH=$PATH:/path/to/dir
The installation guide tells not to use the packages provided by Ubuntu:
Don’t use the python-scrapy package provided by Ubuntu, they are
typically too old and slow to catch up with latest Scrapy.
Instead, use the official Ubuntu
Packages,
which already solve all dependencies for you and are continuously
updated with the latest bug fixes.
As mentioned you should install it using the Ubuntu packages on this page instead.
Besides the previous steps, I also had to install service-identity:
sudo pip install service-identity
I am trying to download pip onto my mac by following the instructions on the pip installation guide and I am coming up with this error after running the following command
$python get-pip.py
/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/
MacOS/Python: can't open file 'get-pip.py': [Errno 2] No such file or directory
This is happening after I download the 'get-pip.py' doc as the instructions suggest. Do I need to put this file in a certain location before I continue? I am relatively new to downloading programs through the terminal.
Thanks for the help!
It is recommended (highly) that you NOT use the version of Python that ships with your Mac. Instead use HomeBrew and install a "custom" version of Python (usually the latest). Then proceed to use virtualenv and optionally virtualenvwrapper
Prerequisites:
First, install Xcode from the App Store (it's FREE).
Install HomeBrew:
ruby -e "$(curl -fsSL https://raw.github.com/Homebrew/homebrew/go/install)"
Install Python:
brew install python
This will install pip for you as well in /usr/local/bin/.
Install virtualenv:
pip install virtualenv
virtualenv Basic Usage:
virtualenv /path/to/my/env
cd /path/to/my/env
source ./bin/activate
# hack on your python project
deactivate # to go back to your normal shell
Please follow instructions for virtualenv for more details.
virtualenvwrapper is also really convenient and worthwhile learning.
Update :
More explanation at #dval 's comment
$ curl -O https://raw.github.com/pypa/pip/master/contrib/get-pip.py
and then execute
$ python get-pip.py
None of the above solutions worked for me, so I decided to straight out clean install Python 3.6 from the downloads page at python.org.
After you have completed the Python installer, go into Terminal and type in:
curl -O https://bootstrap.pypa.io/get-pip.py
Wait for the download to complete and then type in:
python3 get-pip.py --user
Then for your pip commands you will use 'pip3'. For example:
pip3 install awsebcli --upgrade --user
After python and pip have been installed they should be in your user Library. So update your PATH in terminal like so:
export PATH=~/Library/Python/3.6/bin:$PATH
I have a bash_profile shell so I also ran the following command in terminal to load script into my current session:
source ~/.bash_profile
After this, verify that your pip installed component was successful.
For example:
eb --version
See AWS for the above reference.
Curl did not work for me. I had to use "wget".
$ wget https://raw.github.com/pypa/pip/master/contrib/get-pip.py
and then execute
$ python get-pip.py
I tried importing requests:
import requests
But I get an error:
ImportError: No module named requests
Requests is not a built in module (does not come with the default python installation), so you will have to install it:
OSX/Linux
Python 2: sudo pip install requests
Python 3: sudo pip3 install requests
if you have pip installed (pip is the package installer for python and should come by default with your python installation).
If pip is installed but not in your path you can use python -m pip install requests (or python3 -m pip install requests for python3)
Alternatively you can also use sudo easy_install -U requests if you have easy_install installed.
Linux
Alternatively you can use your systems package manager:
For centos: sudo yum install python-requests
For Debian/Ubuntu Python2: sudo apt-get install python-requests
For Debian/Ubuntu Python3: sudo apt-get install python3-requests
Windows
Use pip install requests (or pip3 install requests for python3) if you have pip installed and Pip.exe added to the Path Environment Variable. If pip is installed but not in your path you can use python -m pip install requests (or python3 -m pip install requests for python3)
Alternatively from a cmd prompt, use > Path\easy_install.exe requests, where Path is your Python*\Scripts folder, if it was installed. (For example: C:\Python32\Scripts)
If you manually want to add a library to a windows machine, you can download the compressed library, uncompress it, and then place it into the Lib\site-packages folder of your python path. (For example: C:\Python27\Lib\site-packages)
From Source (Universal)
For any missing library, the source is usually available at https://pypi.python.org/pypi/. You can download requests here: https://pypi.python.org/pypi/requests
On mac osx and windows, after downloading the source zip, uncompress it and from the termiminal/cmd run python setup.py install from the uncompressed dir.
(source)
It's not obvious to me which version of Python you are using.
If it's Python 3, a solution would be sudo pip3 install requests
To install requests module on Debian/Ubuntu for Python2:
$ sudo apt-get install python-requests
And for Python3 the command is:
$ sudo apt-get install python3-requests
Brew users can use reference below,
command to install requests:
python3 -m pip install requests
Homebrew and Python
pip is the package installer for Python and you need the package requests.
This may be a liittle bit too late but this command can be run even when pip path is not set. I am using Python 3.7 running on Windows 10 and this is the command
py -m pip install requests
and you can also replace 'requests' with any other uninstalled library
If you are using Ubuntu, there is need to install requests
run this command:
pip install requests
if you face permission denied error, use sudo before command:
sudo pip install requests
In my case requests was already installed, but needed an upgrade. The following command did the trick
$ sudo pip install requests --upgrade
On OSX, the command will depend on the flavour of python installation you have.
Python 2.x - Default
sudo pip install requests
Python 3.x
sudo pip3 install requests
On Windows Open Command Line
pip3 install requests
I had the same issue, so I copied the folder named "requests" from https://pypi.python.org/pypi/requests#downloadsrequests download to
"/Library/Python/2.7/site-packages".
Now when you use: import requests, it should work fine.
In the terminal/command-line:
pip install requests
then use it inside your Python script by:
import requests
or else if you want to use pycharm IDE to install a package:
go to setting from File in menu
next go to Python interpreter
click on pip
search for requests package and install it
Adding Third-party Packages to the Application
Follow this link
https://cloud.google.com/appengine/docs/python/tools/libraries27?hl=en#vendoring
step1 : Have a file by named a file named appengine_config.py in the root of your project, then add these lines:
from google.appengine.ext import vendor
Add any libraries installed in the "lib" folder.
vendor.add('lib')
Step 2: create a directory and name it "lib" under root directory of project.
step 3: use pip install -t lib requests
step 4 : deploy to app engine.
Try sudo apt-get install python-requests.
This worked for me.
If you are using anaconda as your python package manager, execute the following:
conda install -c anaconda requests
Installing requests through pip didn't help me.
The only thing that worked for me:
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python get-pip.py
pip install requests
Facing the same issue but unable to fix it with the above solution, so I tried this way and it worked:-
curl https://bootstrap.pypa.io/pip/2.7/get-pip.py --output get-pip.py
sudo python2 get-pip.py
python -m pip install requests
For windows just give path as cd and path to the "Scripts" of python and then execute the command easy_install.exe requests.Then try import requests...
I have had this issue a couple times in the past few months. I haven't seen a good solution for fedora systems posted, so here's yet another solution. I'm using RHEL7, and I discovered the following:
If you have urllib3 installed via pip, and requests installed via yum you will have issues, even if you have the correct packages installed. The same will apply if you have urllib3 installed via yum, and requests installed via pip. Here's what I did to fix the issue:
sudo pip uninstall requests
sudo pip uninstall urllib3
sudo yum remove python-urllib3
sudo yum remove python-requests
(confirm that all those libraries have been removed)
sudo yum install python-urllib3
sudo yum install python-requests
Just be aware that this will only work for systems that are running Fedora, Redhat, or CentOS.
Sources:
This very question (in the comments to this answer).
This github issue.
Python Common installation issues
These commands are also useful if Homebrew screws up your path on macOS.
python -m pip install requests
or
python3 -m pip install requests
Multiple versions of Python installed in parallel?
You must make sure your requests module is not being installed in a more recent version of python.
When using python 3.7, run your python file like:
python3 myfile.py
or enter python interactive mode with:
python3
Yes, this works for me. Run your file like this: python3 file.py
I have installed python2.7 and python3.6
Open Command Line to ~/.bash_profile I find that #Setting PATH for Python 3.6 , So
I change the path to PATH="/usr/local/Cellar/python/2.7.13/bin:${PATH}" ,
(please make sure your python2.7's path) ,then save.
It works for me.
if you want request import on windows:
pip install request
then beautifulsoup4 for:
pip3 install beautifulsoup4
Please try the following. If one doesn't work, skip to the next method.
pip install requests
or...
pip3 install requests
or...
python -m pip install requests
or...
python3 -m pip install requests
or...
python -m pip3 install requests
If all of these don't work, please leave a comment!
How does this work? Depending on the operating system you currently use, the pip command may vary or not work on some. These are the commands you may try in order for a fix.
In case you hit pip install requests and had an output massage of Requirement already satisfied but yet you still get the error: ImportError: No module named requests.
This is likely to happen when you find yourself in a different interpreter/virtual environment.
You can copy and append the path of the module into your working environment.
Note: This path usually comes with the message Requirement already satisfied
Before import requests, you should import sys and then append the copied path.
Example:
Command Prompt:
pip install requests
Output:
Requirement already satisfied: requests in /usr/local/lib/python3.9/site-packages
import sys
sys.path.append("/usr/local/lib/python3.9/site-packages")
import requests
I solved this problem.You can try this method.
In this file '.bash_profile', Add codes like alias python=/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
Type this command in Command Prompt (Windows) or Terminal (Linux/macOS):
pip install requests
My answer is basically the same as #pi-k. In my case my program worked locally but failed to build on QA servers. (I suspect devops had older versions of the package blocked and my version must have been too out-of-date) I just decided to upgrade everything
$ pip install pip-review
$ pip-review --local --interactive
You get an import error because requests are not a built-in module instead, it is created by someone else and you need to install the requests.
use the following command on your terminal then it will work correctly.
pip install requests
Install python requests library and this error will be solved.
I found that my issue was VSCode was reading from the wrong Python Interpreter. This youtube tutorial solved it for me.
If you are using anaconda
step 1:
where python
step 2:
open anaconda prompt in administrator mode
step 3:
cd <python path>
step 4:
install the package in this location