I receive an error when running scrapy on Raspberry Pi 3.
I have successfully installed it, but when I try to startproject or crawl with a previously created spider, I get the following error:
Traceback (most recent call last):
File "/usr/local/bin/scrapy", line 7, in <module>
from scrapy.cmdline import execute
File "/usr/local/lib/python3.4/dist-packages/scrapy/cmdline.py", line 9, in <module>
from scrapy.crawler import CrawlerProcess
File "/usr/local/lib/python3.4/dist-packages/scrapy/crawler.py", line 7, in <module>
from twisted.internet import reactor, defer
File "/usr/local/lib/python3.4/dist-packages/twisted/internet/reactor.py", line 38, in <module>
from twisted.internet import default
File "/usr/local/lib/python3.4/dist-packages/twisted/internet/default.py", line 56, in <module>
install = _getInstallFunction(platform)
File "/usr/local/lib/python3.4/dist-packages/twisted/internet/default.py", line 44, in _getInstallFunction
from twisted.internet.epollreactor import install
File "/usr/local/lib/python3.4/dist-packages/twisted/internet/epollreactor.py", line 24, in <module>
from twisted.internet import posixbase
File "/usr/local/lib/python3.4/dist-packages/twisted/internet/posixbase.py", line 18, in <module>
from twisted.internet import error, udp, tcp
File "/usr/local/lib/python3.4/dist-packages/twisted/internet/tcp.py", line 28, in <module>
from twisted.internet._newtls import (
File "/usr/local/lib/python3.4/dist-packages/twisted/internet/_newtls.py", line 21, in <module>
from twisted.protocols.tls import TLSMemoryBIOFactory, TLSMemoryBIOProtocol
File "/usr/local/lib/python3.4/dist-packages/twisted/protocols/tls.py", line 65, in <module>
from twisted.internet._sslverify import _setAcceptableProtocols
File "/usr/local/lib/python3.4/dist-packages/twisted/internet/_sslverify.py", line 1865, in <module>
"ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:DH+CHACHA20:ECDH+AES256:DH+AES256:"
File "/usr/local/lib/python3.4/dist-packages/twisted/internet/_sslverify.py", line 1845, in fromOpenSSLCipherString
SSL.SSLv23_METHOD, SSL.OP_NO_SSLv2 | SSL.OP_NO_SSLv3)
File "/usr/local/lib/python3.4/dist-packages/twisted/internet/_sslverify.py", line 1797, in _expandCipherString
ctx.set_cipher_list(cipherString.encode('ascii'))
TypeError: must be str, not bytes
I have no idea why I am getting this or how to fix it, please help?
SOLUTION:
Thanks everyone for the help, I got Scrapy to work on Raspberry Pi 3 in the end by following these steps:
First install virtualenv:
sudo pip install virtualenv
Then create a virtualenv and active it for Scrapy:
virtualenv scrapyenv
source scrapyenv/bin/activate
Then I ran and updated everything in there:
apt-get update
apt-get upgrade
Install all dependencies:
apt-get install libffi-dev
apt-get install libxml2-dev
apt-get install libxslt1-dev
apt-get install libssl-dev
apt-get install python-dev
Then install Scrapy
sudo pip install scrapy
I then updated my pyOpenSSL with this:
pip -vvvv install --upgrade pyOpenSSL
This created a lot of log files and took a bit of time, after that scrapy worked fine with the normal scrapy commands and I have also run a spider - all works.
What user are you using?
i think you need use sudo.
also update your ssl. looks like the ssl connection has problems..
Have you run :
pip install --verbose twisted
Also update your openssl
pip -vvvv install --upgrade pyOpenSSL,
please copy the output here to check if is updated
Regards
Related
I can't get pip2 to work.
I need pip2, because I need to use an older tensorflow version which is not supported by python3 pip. When trying to run pip2 install I get this:
Traceback (most recent call last):
File "/usr/bin/pip2", line 11, in <module>
load_entry_point('pip==20.1.1', 'console_scripts', 'pip')()
File "/usr/lib/python2.7/site-packages/pip/_internal/cli/main.py", line 73, in main
command = create_command(cmd_name, isolated=("--isolated" in cmd_args))
File "/usr/lib/python2.7/site-packages/pip/_internal/commands/__init__.py", line 104, in create_command
module = importlib.import_module(module_path)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/usr/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 24, in <module>
from pip._internal.cli.req_command import RequirementCommand, with_cleanup
File "/usr/lib/python2.7/site-packages/pip/_internal/cli/req_command.py", line 21, in <module>
from pip._internal.req.constructors import (
File "/usr/lib/python2.7/site-packages/pip/_internal/req/constructors.py", line 21, in <module>
from pip._vendor.pkg_resources import RequirementParseError, parse_requirements
ImportError: cannot import name RequirementParseError
I uninstalled pip2 and reinstalled it. Also did the same for pip(3). But this didn't change a thing. I could not find anything useful on this error message and feel a bit helpless. Any ideas?
update forgot to mention, that I use archlinux. I installed it with pacman -S python2-pip
Did you try installing packages using virtualenv ? pacman ? or using only pip ?
Which version of tensorflow are you trying to install ?
You can try install libraries using a recent pip2 version + virtualenv using the following commands:
pacman -S python2 python2-pip python-virtualenv
virtualenv -p python2 py2
. py2/bin/activate
pip install -U pip setuptools wheel
pip install tensorflow ipython
You may also try uninstall python2 first and reinstall it from scratch:
pacman -Rs python2 python2-pip
This link suggests that the problem might be due to a version of setuptools installed in ~/.local . Maybe you can check that.
Nevermind, setuptools was installed in my ~/.local directory for some
reason, superceding the system install, I removed that and pip2 runs
fine now.
Please see this link:
https://askubuntu.com/questions/984868/error-in-installing-sudo-pip-install-v-protobuf-3-0-0a2
The library is different but the error is the same.
I've recently updated Python to version 3.7 from version 3.5, and made it the default in my bash shell. I've also updated pip and made sure it uses Python3.7.
Now, whenever I try to install any module using pip I get this error"
$pip3 install google
Traceback (most recent call last): File "/home/AnnaK/.local/bin/pip3", line 7, in <module>
from pip._internal.cli.main import main File "/usr/lib/python3/dist-packages/pip/__init__.py", line 16, in <module>
from pip.vcs import git, mercurial, subversion, bazaar # noqa File "/usr/lib/python3/dist-packages/pip/vcs/subversion.py", line 9, in <module>
from pip.index import Link File "/usr/lib/python3/dist-packages/pip/index.py", line 30, in <module>
from pip.wheel import Wheel, wheel_ext File "/usr/lib/python3/dist-packages/pip/wheel.py", line 39, in <module>
from pip._vendor.distlib.scripts import ScriptMaker File "/usr/share/python-wheels/distlib-0.2.2-py2.py3-none-any.whl/distlib/scripts.py", line 14, in <module> File "/usr/share/python-wheels/distlib-0.2.2-py2.py3-none-any.whl/distlib/compat.py", line 66, in <module> ImportError: cannot import name 'HTTPSHandler' from 'urllib.request' (/usr/lib/python3.7/urllib/request.py)
I get the same error if I try to simply run pip. This didn't happen when I was using python3.5. I did have this problem when briefly using python2.7, but I fixed it using https://askubuntu.com/questions/581999/pip-raises-importerror-cannot-import-name-httpshandler. That solution doesn't seem to work for python3.7.
I've tried to reinstall pip, reinstall python3, update my Windows Ubuntu subsystem, and reinstall libssl-dev. I'm stumped. Any help is appreciated!
You must have inadvertently upgraded your system pip (probably through something like sudo pip install pip --upgrade)
pip x.x adjusts where its internals are situated. The pip3 command you're seeing is one provided by your package maintainer and is not a file managed by pip.
You can read more about this on pip's issue tracker
You'll probably want to not upgrade your system pip and instead use a virtualenv.
To recover the pip3 binary you'll need to sudo python3 -m pip uninstall pip && sudo apt install python3-pip --reinstall.
It should work fine if you use the first approach python3 -m pip ... instead of pip3 ....
When I attempt to run some test code my professor gave me using Python3.7 on Linux Mint 19 I get the following error
RuntimeError: Bad code object in .pyc file
Error in sys.excepthook:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
from apport.fileutils import likely_packaged, get_recent_crashes
File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
from apport.report import Report
File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in <module>
import apport.fileutils
File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 23, in <module>
from apport.packaging_impl import impl as packaging
File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 24, in <module>
import apt
File "/usr/lib/python3/dist-packages/apt/__init__.py", line 23, in <module>
import apt_pkg
ModuleNotFoundError: No module named 'apt_pkg'
Original exception was:
RuntimeError: Bad code object in .pyc file
At first, I only installed Python3.7 minimal and this error didn't occur, but I needed to install the full version of Python3.7 to use some modules that come with it, and that started this error. I already tried installing Python3.7 dev and it did nothing. I also tried
sudo apt-get remove --purge python3.7-apt
and it said it couldn't even find apt. So I tried
sudo apt-get install python3.7-apt
and it couldn't even find the package. I also tried installing python3-apt and it did nothing. I would uninstall python3-apt first but apparently that could harm the OS because Mint uses Python3.6 internally. Any help resolving this would be much appreciated!
I had the same issue with python3.7
I literally just reinstalled python3-apt and the error disappeard
sudo apt remove python3-apt
then
sudo apt install python3-apt
I'm not sure why pip always throw the exception that 'can not find pyparsing'.
even When I try to download pyparsing i get:
pip install pyparsing
Traceback (most recent call last):
File "/usr/bin/pip", line 5, in <module>
from pkg_resources import load_entry_point
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 72, in <module>
import packaging.requirements
File "/usr/local/lib/python2.7/dist-packages/packaging/requirements.py", line 9, in <module>
from pyparsing import stringStart, stringEnd, originalTextFor, ParseException
ImportError: No module named pyparsing
It doesn't make sense?!
Even after I did what Shahin and Paul said, I still get the same error. So I removed pip and reinstall it:
sudo apt-get purge python-pip
curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"
python get-pip.py
Try Downloading the wheel file and install
wget https://pypi.python.org/packages/6a/8a/718fd7d3458f9fab8e67186b00abdd345b639976bc7fb3ae722e1b026a50/pyparsing-2.2.0-py2.py3-none-any.whl#md5=7247e7896688eff4bc8c7fc5d0cdd2b0
pip install pyparsing-2.2.0-py2.py3-none-any.whl
In case you face any issue with wget command download , you can get the wheel file from the following link
https://pypi.python.org/pypi/pyparsing/2.2.0#downloads
After running
pip install requests[security]
Running any pip command outputs the error below. I've tried apt-get removing and reinstalling python-pip as well as python-pyopenssl. Wondering if anyone has advice or has ran into this problem? I can't upgrade python or Debian and would like to get pip working again.
Debian 6.0.10
Python 2.6.6
python-pip 1.5.6
$ pip --version
Traceback (most recent call last):
File "/usr/local/bin/pip", line 9, in <module>
load_entry_point('pip==1.5.6', 'console_scripts', 'pip')()
File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 305, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 2244, in load_entry_point
return ep.load()
File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 1954, in load
entry = __import__(self.module_name, globals(),globals(), ['__name__'])
File "/usr/local/lib/python2.6/dist-packages/pip-1.5.6-py2.6.egg/pip/__init__.py", line 11, in <module>
from pip.vcs import git, mercurial, subversion, bazaar # noqa
File "/usr/local/lib/python2.6/dist-packages/pip-1.5.6-py2.6.egg/pip/vcs/mercurial.py", line 9, in <module>
from pip.download import path_to_url
File "/usr/local/lib/python2.6/dist-packages/pip-1.5.6-py2.6.egg/pip/download.py", line 22, in <module>
from pip._vendor import requests, six
File "/usr/local/lib/python2.6/dist-packages/pip-1.5.6-py2.6.egg/pip/_vendor/requests/__init__.py", line 53, in <module>
from .packages.urllib3.contrib import pyopenssl
File "/usr/local/lib/python2.6/dist-packages/pip-1.5.6-py2.6.egg/pip/_vendor/requests/packages/urllib3/contrib/pyopenssl.py", line 49, in <module>
from ndg.httpsclient.ssl_peer_verification import SUBJ_ALT_NAME_SUPPORT
File "/usr/local/lib/python2.6/dist-packages/ndg/httpsclient/ssl_peer_verification.py", line 14, in <module>
from ndg.httpsclient.subj_alt_name import SubjectAltName
File "/usr/local/lib/python2.6/dist-packages/ndg/httpsclient/subj_alt_name.py", line 102, in <module>
class AnotherName(univ.Sequence):
File "/usr/local/lib/python2.6/dist-packages/ndg/httpsclient/subj_alt_name.py", line 105, in AnotherName
namedtype.NamedType('value', univ.Any().subtype(
AttributeError: 'module' object has no attribute 'Any'
Unfortunately the removal didn't magically happen with sudo apt-get autoremove. Of course I also tried the standard pip uninstall ndg-httpsclient with no luck first.
In the end, I had to go into the python program directory and forcibly removing it.
$ cd usr/local/lib/python2.7/dist-packages
$ sudo rm -rf ndg_httpsclient-0.3.3-py2.7.egg-info
$ sudo rm -rf ndg_httpsclient-0.3.3-py2.7-nspkg.pth
$ sudo rm -rf ndg
Was able to fix this by simply doing: sudo apt-get autoremove
Please reference this bug here: https://github.com/cedadev/ndg_httpsclient/issues/5
I suspect the problem is that pip on Ubuntu is using ndg_httpsclient, and the current ndg_httpsclient conflicts with the
old pyasn1, so pip breaks during installation.
Anyway, valid versions of pyasn1 to choose from are : 0.0.9a0,
0.0.10a0, 0.0.11a0, 0.0.12a0, 0.0.13a0, 0.0.13b0, 0.0.13, 0.1.1, 0.1.2, 0.1.3, 0.1.4, 0.1.5, 0.1.6, 0.1.7, 0.1.8, 0.1.9
And I have found a semi-reasonable workaround to the problem.
However 0.0.13 doesnt install on Python 3, so I think the minimum
should be at least 0.1.1.
However I see that cryptography requires pyasn1>=0.1.8 , so it is
likely most people will be using that soon.