Develop Reference

Python 2.7: No module named setup tools

I am trying to clone and install the keras package from github. When I run sudo python setup.py install, it says no module named setuptools. I looked at some posts that were for python 3; however, this solutions did not work for me. I am working on a terminal from a windows computer. I am using python 2.7. I tried the following commands: python -m pip install -U pip setuptools from this website: https://packaging.python.org/installing/. However, nothing seemed to work.

If you are working on Windows then you should be typing batch commands, not bash (Linux, OSX), therefore, sudo ... should not be used by you.
Firstly, I would recommend to install pip, it will make package update and installation a lot easier. Save it as get-pip.py and run the following in the directory the file is located python get-pip.py. Official pip website.
Then install upgrade pip which will also install the latest versions of setuptools and wheel with this python -m pip install -U pip setuptools.

Scrapy commandline tool not installed by pip

I installed scrapy via sudo pip install scrapy. It installed the python modules into site-packages and I can import scrapy in my python environment. However, attempting to use the command line tool throws an error:
scrapy startproject demo
has error The program 'scrapy' is not currently installed. and tells me to install python-scrapy.
whereis scrapy has no output. Got tired of trying to track down the install path, so I ran find -name "*crap*", which also turned up nothing useful. It seems that the commandline tool wasn't installed by pip. What am I missing with this pip install?

The problem is sudo pip install scrapy installs scrapy in a directory not accessible by the current user, if you are not root.
You need to remove scrapy first sudo pip uninstall scrapy then reinstall with the -H sudo flag sudo -H pip install scrapy this will make it such that your command line can detect the scrapy installation.

This also does not answer the question why is scrapy command line tool not available, but if scrapy is importable as you comment, you can use:
$ python -m scrapy.cmdline version -v
$ python -m scrapy.cmdline shell <url>
scrapy is an alias to this in fact, as specified in Scrapy's setup.py entry_points section, and should have been setup by pip install.

This doesn't answer the question of what's wrong with the pip install, but to anyone with a working scrapy package but non-functional commandline command, you can create a script to run scrapy command line tool for you:
#! /usr/bin/python2.7
# path to python 2.7 (python 3 doesn't work well with scrapy atm)
import sys
import scrapy.cmdline
sys.exit(scrapy.cmdline.execute())
saved in a file (with execute permissions) called scrapy somewhere in your $PATH.

Verify whether you have these packages or not:
w3lib, cssselect, parsel, attrs, pyasn1-modules, service-identity, PyDispatcher, queuelib, zope.interface, constantly, incremental, Twisted, scrapy
I used :
$ pip install scrapy
on ubuntu 16.04 and all these packages were installed by it. After this I tried:
$ scrapy startproject demo
and it worked for me with this output:
New Scrapy project 'demo', using template directory '/home/*machine_name*/anaconda2/lib/python2.7/site-packages/scrapy/templates/project', created in:
/home/*machine_name*/demo
You can start your first spider with:
cd demo
scrapy genspider example example.com

Scrapy is not installed on you machine. If you want to install first run these command Which is used to install python-dev on you system
sudo apt-get install build-essential libssl-dev libffi-dev python-dev libxml2-dev
before these commands you should have run upgrade commands
sudo apt-get update
and
sudo apt-get upgrade
after these run
pip install scrapy
when it finish run following to check whether scrapy is installed or not
scrapy version
if version prompts you have installed scrapy successfully.

Trouble installing Scrapy on Windows 64 bit

I am attempting to install Scrapy on my Windows 7 64 bit machine. I started by following the instructions here on Scrapy's documentation.
I got up until the command 'pip install Scrapy'. Everything works except that it cannot find 'libxml2':
Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
I then visited this website to get the binaries of libxml2:
ftp://ftp.zlatkovic.com/libxml/64bit/
The instructions for installation of libxml2 are here: https://www.zlatkovic.com/libxml.en.html
They state that you should unzip the binaries and place the contents of the BIN folder in a path such as C:\WINDOWS. I did this. However, after attempting to install Scrapy again, I continue to receive the same error. Is there something I am missing?

Just installled Scrapy on Windows10 x64. It installs only with Python 2.7 on Windows. I used tutotial from here.
Download from the official site and install Python 2.7
Open Command promt (cmd) with full admin permissions and type:
c:\python27\python.exe c:\python27\tools\scripts\win_add2path.py
and press Enter
Install Microsoft Visual C++ for Python 2.7
Install OpenSSL (sorry, can't attach link because of small reputation). Download it to some folder, e.g. C:\Python27\Scripts and execute command from there: pip install pyOpenSSL-16.0.0-py2.py3-none-any.whl (the name of file could not be the same)
Install lxml just like previous file: pip install lxml-3.6.0-cp27-cp27m-win32.whl
Install Pywin32
And finally, install Scrapy - pip install Scrapy
Can recommend to download and install all needed soft for x32 architecture (even if you have x64 system).

It is recommended to use conda to install Scrapy on Windows.
You can download Miniconda from here: http://conda.pydata.org/miniconda.html
Once you have conda installed you can follow this guide to learn about its subcommands: http://conda.pydata.org/docs/test-drive.html
In short, you can use this commands in your terminal:
Create an environment for scrapy: conda create -n scrapyenv python=2
Enable the scrapy environment: activate scrapyenv
Install scrapy: conda install -c scrapinghub scrapy
Edit: Scrapy's conda package has been moved to conda-forge channel. Use the command: conda install -c conda-forge scrapy

Hi this is what I did to get it installed under Win7/Win10:
easy_install lxml
pip install Scrapy
install pywin32 as indicated by the docs
Then, follow the tutorial to be sure everything is working

pip install mechanize works fine; why permission denied for pip2.7 install?

I'm on an AWS EC2 instance.
Both Python 2.6 & Python 2.7 are installed.
My PYTHONPATH points to 2.7.
I want to install the mechanize module.
sudo pip install mechanize
puts it in Python 2.6 :(
So I tried:
sudo pip2.7 install mechanize
sudo: pip2.7: command not found
How can I install mechanize?

I would setup a pyvenv installation of python.
This will give you full control over the python environment - plus if you ever mess it up - your default system version of python is not effected.
To do this
cd ~
pyvenv py27
source py27/bin/activate
pip install --upgrade pip
And for mechanize
pip install mechanize

Though is a little messy, this worked:
sudo /usr/local/bin/pip2.7 install mechanize

Scrapy's Initialization error

I am getting started with Scrapy, but I have two problems with installation on Linux Mint 17.2 (Ubuntu based version).
I don't get which is the difference from installing pip install
scrapy and sudo apt-get install scrapy
When i do install one of the two and I try to follow the first tutorial of Scrapy using the command scrapy startproject tutorialit gives me error /usr/bin: No such file or directory.
I have tried to uninstall and reinstall many times but still doesn't work.

1. Instalation Source
Both commands pip install scrapy and sudo apt-get install scrapy will install Scrapy on your computer, but the versions may be different. The pip option installs the latest version for Scrapy 1.0, while the one in your repositories its probably outdated.
If anyway you want to install the package from the repositories and still keep it updated, you can add the Scrapy repository:
http://doc.scrapy.org/en/1.0/topics/ubuntu.html
echo 'deb http://archive.scrapy.org/ubuntu scrapy main' | sudo tee /etc/apt-sources.list.d/scrapy.list
sudo apt-get update && sudo apt-get install scrapy
Souce: http://doc.scrapy.org/en/1.0/topics/ubuntu.html
2. Check the PATH variable
Depending on the way you install Scrapy, the binaries folder for the installation might be different. In my case I have it in /usr/local/bin.
Display your PATH variable with echo "$PATH", and check if the folder with the Scrapy binary is included.
You can add more directories to the variable with export PATH=$PATH:/path/to/dir

The installation guide tells not to use the packages provided by Ubuntu:
Don’t use the python-scrapy package provided by Ubuntu, they are
typically too old and slow to catch up with latest Scrapy.
Instead, use the official Ubuntu
Packages,
which already solve all dependencies for you and are continuously
updated with the latest bug fixes.
As mentioned you should install it using the Ubuntu packages on this page instead.
Besides the previous steps, I also had to install service-identity:
sudo pip install service-identity