I've been getting error bringing up Scrapy on my Linux box. I'm using Ubuntu 14.0 Desktop. I've installed setuptools and scrapy. For some reason when I put
scrapy --version
I get this error:
Traceback (most recent call last):
File "/usr/local/bin/scrapy", line 11, in <module>
sys.exit(execute())
File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 122, in execute
cmds = _get_commands_dict(settings, inproject)
File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 46, in _get_commands_dict
cmds = _get_commands_from_module('scrapy.commands', inproject)
File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 29, in _get_commands_from_module
for cmd in _iter_command_classes(module):
File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 20, in _iter_command_classes
for module in walk_modules(module_name):
File "/usr/local/lib/python2.7/dist-packages/scrapy/utils/misc.py", line 68, in walk_modules
submod = import_module(fullpath)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/usr/local/lib/python2.7/dist-packages/scrapy/commands/bench.py", line 2, in <module>
from scrapy.tests.spiders import FollowAllSpider
ImportError: No module named tests.spiders
Any Idea what would this be from?
There was an issue with build 0.24.3 commited on Github.
To solve your problem just type this in terminal:
sudo pip install scrapy --upgrade
This will upgrade your version of Scrapy to 0.24.4.
Referring to the stackoverflow.com/q/25218633/190597. That will fix the issue. Or at least in my case.
Try uninstalling the old version of scrapy with:
pip uninstall Scrapy
and then installing it again using pip with this command:
pip install scrapy==0.24.2
well do not forget to install python development packages beforehand. That will raise another issue if not installed properly.
Related
I might have multiple problems here but they both deal with virtualenv and python3 (3.6.13).
Problem 1:
I had installed virtualenv==20.14.1.
However, it's complaining that there's a version conflict for some reason.
$ python3 -V
Python 3.6.13
$ virtualenv -h
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 573, in _build_master
ws.require(__requires__)
File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 891, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 782, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (virtualenv 20.14.1 (/home/mylogin/.local/lib/python3.6/site-packages), Requirement.parse('virtualenv==15.0.1'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/bin/virtualenv", line 5, in <module>
from pkg_resources import load_entry_point
File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 3266, in <module>
#_call_aside
File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 3241, in _call_aside
f(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 3279, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 575, in _build_master
return cls._build_from_requirements(__requires__)
File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 588, in _build_from_requirements
dists = ws.resolve(reqs, Environment())
File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 777, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'virtualenv==15.0.1' distribution was not found and is required by the application
$
Problem 2:
I downgraded virtualenv to be 15.0.1 like it said it needed but then it complains that there's a missing module called dataclasses. For some reason, my co-workers don't have this problem but we can't figure out what is different between my setup and their setup.
$ virtualenv mydir -p python3.6
Running virtualenv with interpreter /usr/bin/python3.6
Using base prefix '/usr'
New python executable in /tmp/mydir/bin/python3.6
Also creating executable in /tmp/mydir/bin/python
Installing setuptools, pip, wheel...done.
/tmp/mydir$ source bin/activate
(mydir) myhost:/tmp/mydir$ python -V
Python 3.6.13
(mydir) myhost:/tmp/mydir$ python -m pip -V
Traceback (most recent call last):
File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/tmp/mydir/lib/python3.6/site-packages/pip/__main__.py", line 29, in <module>
from pip._internal.cli.main import main as _main
File "/tmp/mydir/lib/python3.6/site-packages/pip/_internal/cli/main.py", line 9, in <module>
from pip._internal.cli.autocompletion import autocomplete
File "/tmp/mydir/lib/python3.6/site-packages/pip/_internal/cli/autocompletion.py", line 10, in <module>
from pip._internal.cli.main_parser import create_main_parser
File "/tmp/mydir/lib/python3.6/site-packages/pip/_internal/cli/main_parser.py", line 8, in <module>
from pip._internal.cli import cmdoptions
File "/tmp/mydir/lib/python3.6/site-packages/pip/_internal/cli/cmdoptions.py", line 23, in <module>
from pip._internal.cli.parser import ConfigOptionParser
File "/tmp/mydir/lib/python3.6/site-packages/pip/_internal/cli/parser.py", line 12, in <module>
from pip._internal.configuration import Configuration, ConfigurationError
File "/tmp/mydir/lib/python3.6/site-packages/pip/_internal/configuration.py", line 20, in <module>
from pip._internal.exceptions import (
File "/tmp/mydir/lib/python3.6/site-packages/pip/_internal/exceptions.py", line 14, in <module>
from pip._vendor.rich.console import Console, ConsoleOptions, RenderResult
File "/tmp/mydir/lib/python3.6/site-packages/pip/_vendor/rich/console.py", line 7, in <module>
from dataclasses import dataclass, field
ModuleNotFoundError: No module named 'dataclasses'
This seems to be a chicken and the egg problem. I would run pip to install this missing module but pip is the program that is complaining about this missing module so I can't install it. Like I said above, none of my co-workers I asked know what the problem is.
For debugging purposes, I installed python 2.7.12 and virtualenv==20.14.1 to see what would happen. Apparently things work fine with python 2.7.
$ pip freeze | grep virtual
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
virtualenv==20.14.1
virtualenv-clone==0.5.3
virtualenvwrapper==4.8.4
myhost:/tmp$ virtualenv zzz -p python
Running virtualenv with interpreter /usr/bin/python
New python executable in /tmp/zzz/bin/python
Installing setuptools, pip, wheel...done.
myhost:/tmp$ cd zzz
myhost:/tmp/zzz$ source bin/activate
(zzz) myhost:/tmp/zzz$ python -V
Python 2.7.12
(zzz) myhost:/tmp/zzz$ pip -V
pip 20.3.4 from /tmp/zzz/local/lib/python2.7/site-packages/pip (python 2.7)
(zzz) myhost:/tmp/zzz$ pip freeze
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
(zzz) myhost:/tmp/zzz$
So the questions are
Why do I need to use virtualenv 15.0.1? Why can't I use a newer version of virtualenv?
How come python3 is complaining about the missing dataclasses module but python2 is ok?
Like mentioned in this pypi docs; https://pypi.org/project/dataclasses/ dataclasses only exist in 3.7 and above, which i would recommend to use instead, otherwise, you could also download this package in hopes that it fixes your problem. Are you limited to a python version?
I was trying to update python's pip command through the command line and accidentally pressed ctrl+c which cancelled the upgrade mid-process.
As soon as I realised what had happened I tried to rerun the command python -m install --upgrade pip and I got this error code:
(Currently running python 3.8.3) on Windows 10 Home
File "C:\Users\HP\AppData\Local\Programs\Python\Python38\lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\HP\AppData\Local\Programs\Python\Python38\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\HP\AppData\Local\Programs\Python\Python38\lib\site-packages\pip\__main__.py", line 29, in <module>
from pip._internal.cli.main import main as _main
File "C:\Users\HP\AppData\Local\Programs\Python\Python38\lib\site-packages\pip\_internal\cli\main.py", line 9, in <module>
from pip._internal.cli.autocompletion import autocomplete
File "C:\Users\HP\AppData\Local\Programs\Python\Python38\lib\site-packages\pip\_internal\cli\autocompletion.py", line 10, in <module>
from pip._internal.cli.main_parser import create_main_parser
File "C:\Users\HP\AppData\Local\Programs\Python\Python38\lib\site-packages\pip\_internal\cli\main_parser.py", line 8, in <module>
from pip._internal.cli import cmdoptions
File "C:\Users\HP\AppData\Local\Programs\Python\Python38\lib\site-packages\pip\_internal\cli\cmdoptions.py", line 21, in <module>
from pip._vendor.packaging.utils import canonicalize_name
ModuleNotFoundError: No module named 'pip._vendor'
Is there any way to fix this or should I reinstall python? I had no extra pip libraries installed except the requests library.
You can follow this guide. I also describe the steps here:
Find where your python is located. In Windows it might be located in a path similar to C:\Users\<your-name>\AppsData\Local\Programs\Python\Python38-<minor-version>
Download the get-pip.py script to the directory where your Python is located. You can just copy the content and paste it into a file with the same name
Open the terminal (or PowerShell) and change the directory to where your Python is located.
run the script as follows: python get-pip.py
Verify the installation by executing: pip -V
all,
I've installed the package:pywinauto successfully with "pip install pywinauto", but it always fails, why?
I did it in this way:
pip install pywinauto
and then under the windows cmd env, I run the python:
and then:
import pywinauto
I got the following errors:
....
>>> import pywinauto
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\site-packages\pywinauto\__init__.py", line 40, in <module>
from . import findwindows
File "C:\Python27\lib\site-packages\pywinauto\findwindows.py", line 42, in <module>
from . import controls
File "C:\Python27\lib\site-packages\pywinauto\controls\__init__.py", line 36,in <module>
from . import uiawrapper # register "uia" back-end (at the end of uiawrapper module)
File "C:\Python27\lib\site-packages\pywinauto\controls\uiawrapper.py", line 44, in <module>
from ..uia_defines import IUIA
File "C:\Python27\lib\site-packages\pywinauto\uia_defines.py", line 175, in <module>
pattern_ids = _build_pattern_ids_dic()
File "C:\Python27\lib\site-packages\pywinauto\uia_defines.py", line 163, in _build_pattern_ids_dic
if hasattr(IUIA().ui_automation_client, cls_name):
File "C:\Python27\lib\site-packages\pywinauto\uia_defines.py", line 50, in __call__
cls._instances[cls] = super(_Singleton, cls).__call__(*args, **kwargs)
File "C:\Python27\lib\site-packages\pywinauto\uia_defines.py", line 60, in __init__
self.UIA_dll = comtypes.client.GetModule('UIAutomationCore.dll')
File "C:\Python27\lib\site-packages\comtypes\client\_generate.py", line 97, in GetModule
tlib = comtypes.typeinfo.LoadTypeLibEx(tlib)
File "C:\Python27\lib\site-packages\comtypes\typeinfo.py", line 485, in LoadTypeLibEx
_oleaut32.LoadTypeLibEx(c_wchar_p(szFile), regkind, byref(ptl))
File "_ctypes/callproc.c", line 950, in GetResult
WindowsError: [Error -2147312566] Error loading type library/DLL
It looks like you're using old OS Windows version like Windows XP. MS UI Automation is included into Windows Vista and later. But you may install .NET Framework 3.0+ to get UIAutomationCore.dll available even on Windows XP. If you don't need MS UI Automation technology at all, just run pip uninstall comtypes and pywinauto will work with Win32 API only.
If installing .NET Framework 3.0+ doesn't help (or if it's already installed) and you need to use Win XP, install Windows update KB971513. It solved the loading library issue for me.
I need upgrade the Winxp to Win7 or Win8, or Win10.
I was trying to use virtualenv to switch between python versions before learning that I could use both python and python3 on my Mac.
I was able to fix my python 2.7 version so that still works fine however, now when I run python3, I get this error:
Failed to import the site module
Traceback (most recent call last):
File "/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site.py", line 544, in <module>
main()
File "/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site.py", line 530, in main
known_paths = addusersitepackages(known_paths)
File "/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site.py", line 282, in addusersitepackages
user_site = getusersitepackages()
File "/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site.py", line 258, in getusersitepackages
user_base = getuserbase() # this will also set USER_BASE
File "/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site.py", line 248, in getuserbase
USER_BASE = get_config_var('userbase')
File "/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/sysconfig.py", line 601, in get_config_var
return get_config_vars().get(name)
File "/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/sysconfig.py", line 580, in get_config_vars
import _osx_support
File "/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/_osx_support.py", line 4, in <module>
import re
File "/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/re.py", line 125, in <module>
import functools
File "/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/functools.py", line 21, in <module>
from collections import namedtuple
File "/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/collections/__init__.py", line 32, in <module>
from reprlib import recursive_repr as _recursive_repr
File "/usr/local/lib/python2.7/site-packages/reprlib/__init__.py", line 7, in <module>
raise ImportError('This package should not be accessible on Python 3. '
ImportError: This package should not be accessible on Python 3. Either you are trying to run from the python-future src folder or your installation of python-future is corrupted.
I've looked online for this error but most of the conversation was on fixing up the patch and then upgrading virtualenv. However, this still doesn't fix my issue.
python3 -V: Python 3.6.0
virtualenv --version: 15.1.0
env | egrep -i 'python|virtualenv': PYTHONPATH=/usr/local/lib/python2.7/site-packages:
Your environment contains PYTHONPATH=/usr/local/lib/python2.7/site-packages
This doesn't work with Python 3 for obvious reasons. To remove it:
unset PYTHONPATH
I solved my problem with below command
sudo pip install virtualenv --upgrade
virtualenv -p python3 env
I'm attempting to install pip for arcpy (arcgis 10.2 on windows 7). Running get-pip.py results in the following error message:
X:\python>python get-pip.py
Traceback (most recent call last):
File "get-pip.py", line 20061, in <module>
main()
File "get-pip.py", line 194, in main
bootstrap(tmpdir=tmpdir)
File "get-pip.py", line 82, in bootstrap
import pip
File "c:\temp\tmpou5fje\pip.zip\pip\__init__.py", line 26, in <module>
File "c:\temp\tmpou5fje\pip.zip\pip\utils\__init__.py", line 27, in <module>
File "c:\temp\tmpou5fje\pip.zip\pip\_vendor\pkg_resources\__init__.py", line 73, in <module>
File "c:\temp\tmpou5fje\pip.zip\pip\_vendor\packaging\specifiers.py", line 275, in <module>
File "c:\temp\tmpou5fje\pip.zip\pip\_vendor\packaging\specifiers.py", line 373, in Specifier
File "C:\Python27\ArcGIS10.2\Lib\re.py", line 190, in compile
return _compile(pattern, flags)
File "C:\Python27\ArcGIS10.2\Lib\re.py", line 242, in _compile
raise error, v # invalid expression
sre_constants.error: nothing to repeat
Using an administrator command prompt doesn't help. My real goal is to get win32com working under arcpy. I usual just copy the appropriate directories out of c:\python27\lib\site-packages to c:\python27\arcgis10.2\lib\site-packages to install a package under arcpy (why doesn't arcpy come with pip?) but that's not working for win32com, presumably do to a missing dll or other windows specific file.
I would recommend the following:
Get the setuptools module
Get the pip module`
And then run the following in command line (assuming windows)
path-to-python path-to-setuptools install
path-to-python path-to-pip install
I work on a closed network (away from the interwebs of old) and cannot use get-pip.py so I find it best to simply download the actual modules and hard install.
Keep us posted!
Copy get_pip.py to "C:\Python27\ArcGIS10.2", then perform command "python get-pip.py" in the directory.
Note that keep network connected in the process, so that auto-download and setup setuptools,wheels,etc.
Hope that can help you.
Try opening a CMD prompt and typing:
C:\Python27\ArcGIS10.2\python.exe -m pip install -U pip