I am trying to use scrapy. I did pip install scrapy. My python version is 2.7.9. After installing when I typed scrapy in the terminal it gave the following error:
File "/usr/bin/scrapy", line 7, in <module>
from scrapy.cmdline import execute
File "/usr/lib/python2.7/site-packages/scrapy/__init__.py", line 48, in <module>
from scrapy.spiders import Spider
File "/usr/lib/python2.7/site-packages/scrapy/spiders/__init__.py", line 10, in <module>
from scrapy.http import Request
File "/usr/lib/python2.7/site-packages/scrapy/http/__init__.py", line 10, in <module>
from scrapy.http.request import Request
File "/usr/lib/python2.7/site-packages/scrapy/http/request/__init__.py", line 12, in <module>
from scrapy.utils.url import escape_ajax
File "/usr/lib/python2.7/site-packages/scrapy/utils/url.py", line 9, in <module>
from six.moves.urllib.parse import (ParseResult, urlunparse, urldefrag,
ImportError: No module named urllib.parse
I know scrapy needs python 2.7 but urllib.parse is introduced in python 3, before that it was urlparse. Looking at the error it seems the error is in the scrapy installation. What to do? I have uninstalled and reinstalled scrappy several times. But the problem is still there.
We can use seperate imports for python 2 and python 3.
try:
from urllib.parse import urlparse
except ImportError:
from urlparse import urlparse
Got this answer from here.
no module named urllib.parse (How should I install it?)
Related
I'm trying to run my Python file. (py name.py)
from bs4 import BeautifulSoup as BS
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\site-packages\bs4\__init__.py", line 29, in <module>
from .builder import builder_registry
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\site-packages\bs4\builder\__init__.py", line 294, in <module>
from . import _htmlparser
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\site-packages\bs4\builder\_htmlparser.py", line 7, in <module>
from html.parser import (
ImportError: cannot import name 'HTMLParseError' from 'html.parser' (C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\html\parser.py)
I installed this library using "pip".
After the library is completely removed and installed again, this error appears when you run this code.
File "name.py", line 9, in <module>
from bs4 import BeautifulSoup as BS
ModuleNotFoundError: No module named 'bs4'
I solved the problem. First I deleted all BS4 mentions from my PC. Then in CMD moved on path:
C:\Users\Administrator\AppData\Local\Programs\Python\Python37\Scripts.
And spent there the command:
pip install beautifulsoup4
I have installed beautifulsoup for Python, but it gives me this error when I import the library:
Traceback (most recent call last):
File "D:/Playroom/WebScraper_01.py", line 2, in <module>
from bs4 import BeautifulSoup
File "C:\Python\lib\site-packages\bs4\__init__.py", line 29, in <module>
from .builder import builder_registry
File "C:\Python\lib\site-packages\bs4\builder\__init__.py", line 294, in <module>
from . import _htmlparser
File "C:\Python\lib\site-packages\bs4\builder\_htmlparser.py", line 7, in <module>
from html.parser import (
ImportError: cannot import name 'HTMLParseError'
Does anyone know why?
If you get the ImportError “No module named HTMLParser”, your problem is that you’re running the Python 2 version of the code under Python 3.
If you get the ImportError “No module named html.parser”, your problem is that you’re running the Python 3 version of the code under Python 2.
See: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#problems-after-installation
I'm trying to use the python-fitbit package with Python 3.4 to get information from Fitbit (https://github.com/orcasgit/python-fitbit).
Whenever I run it, I get the following error. Right now oauthlib is installed, and I've tried uninstalling and reinstalling it to no avail. Any thoughts would be appreciated.
File "/usr/local/lib/python3.4/dist-packages/fitbit/__init__.py", line 10, in <module>
from .api import Fitbit, FitbitOauth2Client
File "/usr/local/lib/python3.4/dist-packages/fitbit/api.py", line 12, in <module>
from requests_oauthlib import OAuth2, OAuth2Session
File "/usr/local/lib/python3.4/dist-packages/requests_oauthlib/__init__.py", line 1, in <module>
from .oauth1_auth import OAuth1
File "/usr/local/lib/python3.4/dist-packages/requests_oauthlib/oauth1_auth.py", line 6, in <module>
from oauthlib.common import extract_params
ImportError: No module named 'oauthlib'
Where in your file system did you install oauthlib?
Please make sure that oauthlib is in your python path.
In order to check your python path you can execute:
import sys; print sys.path
In the past, I would create a spider using scrapy by writing
scrapy startproject some_project
Recently, I cloned a repository that had a spider, and now when I navigate to the correct location and type
scrapy crawl some_spider -o output.csv -t csv
I get a import errors:
Traceback (most recent call last):
File "/usr/local/bin/scrapy", line 3, in <module>
from scrapy.cmdline import execute
File "/usr/lib/pymodules/python2.7/scrapy/__init__.py", line 58, in <module>
from scrapy.selector import Selector
File "/usr/lib/pymodules/python2.7/scrapy/selector/__init__.py", line 4, in <module>
from scrapy.selector.unified import *
File "/usr/lib/pymodules/python2.7/scrapy/selector/unified.py", line 7, in <module>
from scrapy.utils.misc import extract_regex
File "/usr/lib/pymodules/python2.7/scrapy/utils/misc.py", line 8, in <module>
from w3lib.html import replace_entities
ImportError: cannot import name replace_entities
I googled around and attempted to see what is going on with `replace_entities' but I can't find any information. Any help on why these import errors are occurring and any ideas on how to fix this would be greatly appreciated.
w3lib is Scrapy's dependency, quote from setup.py (version 0.24.4):
install_requires=[
'Twisted>=10.0.0',
'w3lib>=1.8.0',
'queuelib',
'lxml',
'pyOpenSSL',
'cssselect>=0.9',
'six>=1.5.2',
],
As you see, Scrapy requires w3lib to be of version 1.8.0 or above.
The solution would be to upgrade w3lib package:
pip install --upgrade w3lib
I'm trying to use Markdown with Django as shown in this tutorial. I've got the library installed and configured in Django using django.contrib.markup, but when I go to a page using markup, I get:
Error in 'markdown' filter: The Python markdown library isn't installed.
Then, through the interpreter I try running import markdown, which results in the following traceback:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/site-packages/markdown/__init__.py", line 43, in <module>
from .treeprocessors import build_treeprocessors
File "/usr/lib/python2.7/site-packages/markdown/treeprocessors.py", line 2, in <module>
from . import inlinepatterns
File "/usr/lib/python2.7/site-packages/markdown/inlinepatterns.py", line 47, in <module>
from urllib.parse import urlparse, urlunparse
ImportError: No module named parse
The Python documentation, however, tells me that urlparse has been renamed to urllib.parse in Python 3, which doesn't make any sense, because the Markdown library was installed through pip2, which put in into the python2.7 site-packages directory.
Changing that one line, by the way, results in more import errors.
No idea how this happened, but reinstalling Markdown worked. In retrospect, I probably should have tried that first.