Running Scrapy on Raspberry Pi 3, python 3.4 - python

I receive an error when running scrapy on Raspberry Pi 3.
I have successfully installed it, but when I try to startproject or crawl with a previously created spider, I get the following error:
Traceback (most recent call last):
File "/usr/local/bin/scrapy", line 7, in <module>
from scrapy.cmdline import execute
File "/usr/local/lib/python3.4/dist-packages/scrapy/cmdline.py", line 9, in <module>
from scrapy.crawler import CrawlerProcess
File "/usr/local/lib/python3.4/dist-packages/scrapy/crawler.py", line 7, in <module>
from twisted.internet import reactor, defer
File "/usr/local/lib/python3.4/dist-packages/twisted/internet/reactor.py", line 38, in <module>
from twisted.internet import default
File "/usr/local/lib/python3.4/dist-packages/twisted/internet/default.py", line 56, in <module>
install = _getInstallFunction(platform)
File "/usr/local/lib/python3.4/dist-packages/twisted/internet/default.py", line 44, in _getInstallFunction
from twisted.internet.epollreactor import install
File "/usr/local/lib/python3.4/dist-packages/twisted/internet/epollreactor.py", line 24, in <module>
from twisted.internet import posixbase
File "/usr/local/lib/python3.4/dist-packages/twisted/internet/posixbase.py", line 18, in <module>
from twisted.internet import error, udp, tcp
File "/usr/local/lib/python3.4/dist-packages/twisted/internet/tcp.py", line 28, in <module>
from twisted.internet._newtls import (
File "/usr/local/lib/python3.4/dist-packages/twisted/internet/_newtls.py", line 21, in <module>
from twisted.protocols.tls import TLSMemoryBIOFactory, TLSMemoryBIOProtocol
File "/usr/local/lib/python3.4/dist-packages/twisted/protocols/tls.py", line 65, in <module>
from twisted.internet._sslverify import _setAcceptableProtocols
File "/usr/local/lib/python3.4/dist-packages/twisted/internet/_sslverify.py", line 1865, in <module>
"ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:DH+CHACHA20:ECDH+AES256:DH+AES256:"
File "/usr/local/lib/python3.4/dist-packages/twisted/internet/_sslverify.py", line 1845, in fromOpenSSLCipherString
SSL.SSLv23_METHOD, SSL.OP_NO_SSLv2 | SSL.OP_NO_SSLv3)
File "/usr/local/lib/python3.4/dist-packages/twisted/internet/_sslverify.py", line 1797, in _expandCipherString
ctx.set_cipher_list(cipherString.encode('ascii'))
TypeError: must be str, not bytes
I have no idea why I am getting this or how to fix it, please help?
SOLUTION:
Thanks everyone for the help, I got Scrapy to work on Raspberry Pi 3 in the end by following these steps:
First install virtualenv:
sudo pip install virtualenv
Then create a virtualenv and active it for Scrapy:
virtualenv scrapyenv
source scrapyenv/bin/activate
Then I ran and updated everything in there:
apt-get update
apt-get upgrade
Install all dependencies:
apt-get install libffi-dev
apt-get install libxml2-dev
apt-get install libxslt1-dev
apt-get install libssl-dev
apt-get install python-dev
Then install Scrapy
sudo pip install scrapy
I then updated my pyOpenSSL with this:
pip -vvvv install --upgrade pyOpenSSL
This created a lot of log files and took a bit of time, after that scrapy worked fine with the normal scrapy commands and I have also run a spider - all works.

What user are you using?
i think you need use sudo.
also update your ssl. looks like the ssl connection has problems..
Have you run :
pip install --verbose twisted
Also update your openssl
pip -vvvv install --upgrade pyOpenSSL,
please copy the output here to check if is updated
Regards

Related

pip2: ImportError: cannot import name RequirementParseError

I can't get pip2 to work.
I need pip2, because I need to use an older tensorflow version which is not supported by python3 pip. When trying to run pip2 install I get this:
Traceback (most recent call last):
File "/usr/bin/pip2", line 11, in <module>
load_entry_point('pip==20.1.1', 'console_scripts', 'pip')()
File "/usr/lib/python2.7/site-packages/pip/_internal/cli/main.py", line 73, in main
command = create_command(cmd_name, isolated=("--isolated" in cmd_args))
File "/usr/lib/python2.7/site-packages/pip/_internal/commands/__init__.py", line 104, in create_command
module = importlib.import_module(module_path)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/usr/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 24, in <module>
from pip._internal.cli.req_command import RequirementCommand, with_cleanup
File "/usr/lib/python2.7/site-packages/pip/_internal/cli/req_command.py", line 21, in <module>
from pip._internal.req.constructors import (
File "/usr/lib/python2.7/site-packages/pip/_internal/req/constructors.py", line 21, in <module>
from pip._vendor.pkg_resources import RequirementParseError, parse_requirements
ImportError: cannot import name RequirementParseError
I uninstalled pip2 and reinstalled it. Also did the same for pip(3). But this didn't change a thing. I could not find anything useful on this error message and feel a bit helpless. Any ideas?
update forgot to mention, that I use archlinux. I installed it with pacman -S python2-pip
Did you try installing packages using virtualenv ? pacman ? or using only pip ?
Which version of tensorflow are you trying to install ?
You can try install libraries using a recent pip2 version + virtualenv using the following commands:
pacman -S python2 python2-pip python-virtualenv
virtualenv -p python2 py2
. py2/bin/activate
pip install -U pip setuptools wheel
pip install tensorflow ipython
You may also try uninstall python2 first and reinstall it from scratch:
pacman -Rs python2 python2-pip
This link suggests that the problem might be due to a version of setuptools installed in ~/.local . Maybe you can check that.
Nevermind, setuptools was installed in my ~/.local directory for some
reason, superceding the system install, I removed that and pip2 runs
fine now.
Please see this link:
https://askubuntu.com/questions/984868/error-in-installing-sudo-pip-install-v-protobuf-3-0-0a2
The library is different but the error is the same.

Python3.7 on Windows Ubuntu Bash shell ImportError: cannot import name 'HTTPSHandler' from 'urllib.request' (/usr/lib/python3.7/urllib/request.py)

I've recently updated Python to version 3.7 from version 3.5, and made it the default in my bash shell. I've also updated pip and made sure it uses Python3.7.
Now, whenever I try to install any module using pip I get this error"
$pip3 install google
Traceback (most recent call last): File "/home/AnnaK/.local/bin/pip3", line 7, in <module>
from pip._internal.cli.main import main File "/usr/lib/python3/dist-packages/pip/__init__.py", line 16, in <module>
from pip.vcs import git, mercurial, subversion, bazaar # noqa File "/usr/lib/python3/dist-packages/pip/vcs/subversion.py", line 9, in <module>
from pip.index import Link File "/usr/lib/python3/dist-packages/pip/index.py", line 30, in <module>
from pip.wheel import Wheel, wheel_ext File "/usr/lib/python3/dist-packages/pip/wheel.py", line 39, in <module>
from pip._vendor.distlib.scripts import ScriptMaker File "/usr/share/python-wheels/distlib-0.2.2-py2.py3-none-any.whl/distlib/scripts.py", line 14, in <module> File "/usr/share/python-wheels/distlib-0.2.2-py2.py3-none-any.whl/distlib/compat.py", line 66, in <module> ImportError: cannot import name 'HTTPSHandler' from 'urllib.request' (/usr/lib/python3.7/urllib/request.py)
I get the same error if I try to simply run pip. This didn't happen when I was using python3.5. I did have this problem when briefly using python2.7, but I fixed it using https://askubuntu.com/questions/581999/pip-raises-importerror-cannot-import-name-httpshandler. That solution doesn't seem to work for python3.7.
I've tried to reinstall pip, reinstall python3, update my Windows Ubuntu subsystem, and reinstall libssl-dev. I'm stumped. Any help is appreciated!
You must have inadvertently upgraded your system pip (probably through something like sudo pip install pip --upgrade)
pip x.x adjusts where its internals are situated. The pip3 command you're seeing is one provided by your package maintainer and is not a file managed by pip.
You can read more about this on pip's issue tracker
You'll probably want to not upgrade your system pip and instead use a virtualenv.
To recover the pip3 binary you'll need to sudo python3 -m pip uninstall pip && sudo apt install python3-pip --reinstall.
It should work fine if you use the first approach python3 -m pip ... instead of pip3 ....

Python3.7 'No module named apt_pkg' error on Linux Mint 19

When I attempt to run some test code my professor gave me using Python3.7 on Linux Mint 19 I get the following error
RuntimeError: Bad code object in .pyc file
Error in sys.excepthook:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
from apport.fileutils import likely_packaged, get_recent_crashes
File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
from apport.report import Report
File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in <module>
import apport.fileutils
File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 23, in <module>
from apport.packaging_impl import impl as packaging
File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 24, in <module>
import apt
File "/usr/lib/python3/dist-packages/apt/__init__.py", line 23, in <module>
import apt_pkg
ModuleNotFoundError: No module named 'apt_pkg'
Original exception was:
RuntimeError: Bad code object in .pyc file
At first, I only installed Python3.7 minimal and this error didn't occur, but I needed to install the full version of Python3.7 to use some modules that come with it, and that started this error. I already tried installing Python3.7 dev and it did nothing. I also tried
sudo apt-get remove --purge python3.7-apt
and it said it couldn't even find apt. So I tried
sudo apt-get install python3.7-apt
and it couldn't even find the package. I also tried installing python3-apt and it did nothing. I would uninstall python3-apt first but apparently that could harm the OS because Mint uses Python3.6 internally. Any help resolving this would be much appreciated!
I had the same issue with python3.7
I literally just reinstalled python3-apt and the error disappeard
sudo apt remove python3-apt
then
sudo apt install python3-apt

Python: Why it always asks to install pyparsing whenever I want install something?

I'm not sure why pip always throw the exception that 'can not find pyparsing'.
even When I try to download pyparsing i get:
pip install pyparsing
Traceback (most recent call last):
File "/usr/bin/pip", line 5, in <module>
from pkg_resources import load_entry_point
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 72, in <module>
import packaging.requirements
File "/usr/local/lib/python2.7/dist-packages/packaging/requirements.py", line 9, in <module>
from pyparsing import stringStart, stringEnd, originalTextFor, ParseException
ImportError: No module named pyparsing
It doesn't make sense?!
Even after I did what Shahin and Paul said, I still get the same error. So I removed pip and reinstall it:
sudo apt-get purge python-pip
curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"
python get-pip.py
Try Downloading the wheel file and install
wget https://pypi.python.org/packages/6a/8a/718fd7d3458f9fab8e67186b00abdd345b639976bc7fb3ae722e1b026a50/pyparsing-2.2.0-py2.py3-none-any.whl#md5=7247e7896688eff4bc8c7fc5d0cdd2b0
pip install pyparsing-2.2.0-py2.py3-none-any.whl
In case you face any issue with wget command download , you can get the wheel file from the following link
https://pypi.python.org/pypi/pyparsing/2.2.0#downloads

Pip broken after installing requests[security]

After running
pip install requests[security]
Running any pip command outputs the error below. I've tried apt-get removing and reinstalling python-pip as well as python-pyopenssl. Wondering if anyone has advice or has ran into this problem? I can't upgrade python or Debian and would like to get pip working again.
Debian 6.0.10
Python 2.6.6
python-pip 1.5.6
$ pip --version
Traceback (most recent call last):
File "/usr/local/bin/pip", line 9, in <module>
load_entry_point('pip==1.5.6', 'console_scripts', 'pip')()
File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 305, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 2244, in load_entry_point
return ep.load()
File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 1954, in load
entry = __import__(self.module_name, globals(),globals(), ['__name__'])
File "/usr/local/lib/python2.6/dist-packages/pip-1.5.6-py2.6.egg/pip/__init__.py", line 11, in <module>
from pip.vcs import git, mercurial, subversion, bazaar # noqa
File "/usr/local/lib/python2.6/dist-packages/pip-1.5.6-py2.6.egg/pip/vcs/mercurial.py", line 9, in <module>
from pip.download import path_to_url
File "/usr/local/lib/python2.6/dist-packages/pip-1.5.6-py2.6.egg/pip/download.py", line 22, in <module>
from pip._vendor import requests, six
File "/usr/local/lib/python2.6/dist-packages/pip-1.5.6-py2.6.egg/pip/_vendor/requests/__init__.py", line 53, in <module>
from .packages.urllib3.contrib import pyopenssl
File "/usr/local/lib/python2.6/dist-packages/pip-1.5.6-py2.6.egg/pip/_vendor/requests/packages/urllib3/contrib/pyopenssl.py", line 49, in <module>
from ndg.httpsclient.ssl_peer_verification import SUBJ_ALT_NAME_SUPPORT
File "/usr/local/lib/python2.6/dist-packages/ndg/httpsclient/ssl_peer_verification.py", line 14, in <module>
from ndg.httpsclient.subj_alt_name import SubjectAltName
File "/usr/local/lib/python2.6/dist-packages/ndg/httpsclient/subj_alt_name.py", line 102, in <module>
class AnotherName(univ.Sequence):
File "/usr/local/lib/python2.6/dist-packages/ndg/httpsclient/subj_alt_name.py", line 105, in AnotherName
namedtype.NamedType('value', univ.Any().subtype(
AttributeError: 'module' object has no attribute 'Any'
Unfortunately the removal didn't magically happen with sudo apt-get autoremove. Of course I also tried the standard pip uninstall ndg-httpsclient with no luck first.
In the end, I had to go into the python program directory and forcibly removing it.
$ cd usr/local/lib/python2.7/dist-packages
$ sudo rm -rf ndg_httpsclient-0.3.3-py2.7.egg-info
$ sudo rm -rf ndg_httpsclient-0.3.3-py2.7-nspkg.pth
$ sudo rm -rf ndg
Was able to fix this by simply doing: sudo apt-get autoremove
Please reference this bug here: https://github.com/cedadev/ndg_httpsclient/issues/5
I suspect the problem is that pip on Ubuntu is using ndg_httpsclient, and the current ndg_httpsclient conflicts with the
old pyasn1, so pip breaks during installation.
Anyway, valid versions of pyasn1 to choose from are : 0.0.9a0,
0.0.10a0, 0.0.11a0, 0.0.12a0, 0.0.13a0, 0.0.13b0, 0.0.13, 0.1.1, 0.1.2, 0.1.3, 0.1.4, 0.1.5, 0.1.6, 0.1.7, 0.1.8, 0.1.9
And I have found a semi-reasonable workaround to the problem.
However 0.0.13 doesnt install on Python 3, so I think the minimum
should be at least 0.1.1.
However I see that cryptography requires pyasn1>=0.1.8 , so it is
likely most people will be using that soon.

Categories

Resources