unable to run scrapy in ec2 - python

I'm trying to run a code on an ec2 server.
It is a python scrapy project which is executed fine on my own pc.
when trying to run it in the ec2 i get this:
Traceback (most recent call last):
File "/usr/local/bin/scrapy", line 4, in <module>
execute()
File "/usr/local/lib/python2.7/site-packages/scrapy/cmdline.py", line 122, in execute
cmds = _get_commands_dict(settings, inproject)
File "/usr/local/lib/python2.7/site-packages/scrapy/cmdline.py", line 46, in _get_commands_dict
cmds = _get_commands_from_module('scrapy.commands', inproject)
File "/usr/local/lib/python2.7/site-packages/scrapy/cmdline.py", line 29, in _get_commands_from_module
for cmd in _iter_command_classes(module):
File "/usr/local/lib/python2.7/site-packages/scrapy/cmdline.py", line 20, in _iter_command_classes
for module in walk_modules(module_name):
File "/usr/local/lib/python2.7/site-packages/scrapy/utils/misc.py", line 68, in walk_modules
submod = import_module(fullpath)
File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/usr/local/lib/python2.7/site-packages/scrapy/commands/bench.py", line 3, in <module>
from scrapy.tests.mockserver import MockServer
File "/usr/local/lib/python2.7/site-packages/scrapy/tests/mockserver.py", line 6, in <module>
from twisted.internet import reactor, defer, ssl
File "/usr/local/lib/python2.7/site-packages/twisted/internet/ssl.py", line 59, in <module>
from OpenSSL import SSL
File "/usr/local/lib/python2.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
from OpenSSL import rand, crypto, SSL
File "/usr/local/lib/python2.7/site-packages/OpenSSL/rand.py", line 11, in <module>
from OpenSSL._util import (
File "/usr/local/lib/python2.7/site-packages/OpenSSL/_util.py", line 4, in <module>
binding = Binding()
File "/usr/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 87, in __init__
self._ensure_ffi_initialized()
File "/usr/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 106, in _ensure_ffi_initialized
libraries=libraries,
File "/usr/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/utils.py", line 39, in build_ffi
ffi = cffi.FFI()
File "/usr/local/lib/python2.7/site-packages/cffi/api.py", line 56, in __init__
**import _cffi_backend as backend
ImportError: libffi.so.5: cannot open shared object file: No such file or directory**
I've tried to update && reinstall all the mentioned libraries.
it always says it is most updated version.
this is my linux distro and default python is 2.7:
# cat /etc/*-release
NAME="Amazon Linux AMI"
VERSION="2015.03"
ID="amzn"
ID_LIKE="rhel fedora"
VERSION_ID="2015.03"
PRETTY_NAME="Amazon Linux AMI 2015.03"
ANSI_COLOR="0;33"
CPE_NAME="cpe:/o:amazon:linux:2015.03:ga"
HOME_URL="http://aws.amazon.com/amazon-linux-ami/"
Amazon Linux AMI release 2015.03
UPDATE: it looks like some sort of problem in scrapy dependencies.
trying
pip install scrapy
gave me the following error:
creating /usr/lib/python2.6/dist-packages/cssselect
error: could not create '/usr/lib/python2.6/dist-packages/cssselect': Permission denied
----------------------------------------
Command "/usr/bin/python2.6 -c "import setuptools,
tokenize;__file__='/tmp/pip-build- aoghBl/cssselect/setup.py'
;exec(compile(getattr(tokenize, 'open', open)
(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))"
install --record /tmp/pip-1Qpx2a-record/install-record.txt
--single-version-externally-managed --compile"
failed with error code 1 in /tmp/pip-build-aoghBl/cssselect

Your updated question is a permissions error. you likely need to run
sudo pip install cssselect
May need to do that for other dependancies. It's strange though that it's trying to install under python2.6...
Edit:
Given that it's trying to run under 2.6, something is wrong with your instance. I'd suggest terminating and recreating the instance. The following commands are what I ran to get scrapy dependancies and my project up and running
sudo su
yum groupinstall 'Development Tools'
yum install -y libffi-devel libxslt-devel libxml2-devel openssl-devel python-devel mysql-devel

I also had problems getting scrapy installed. I found the following pages helpful:
How to Install Scrapy in Linux with out Any Errors
Installing Scrapy on an Amazon CentOS AMI
In the end, I had to launch a "fresh" Amazon Linux instance, which successfully installed with instructions from those pages. My older instance had too many issues to resolve.

Related

Vaultier is unusable for docker/ubuntu/debian (Python)

I am looking for a good Password safe for my Company. I wanted to test Vaultier, but it newer works as expected. Neither with Docker or other Platforms. There is always and error, if its Docker i get that output.
I'm no python specialist, but making pip install --upgrade made the installation just missing the whole app.
i followed these guides https://www.vaultier.org/install/
sudo docker run -t -i --name vaultier -p 80:8088 rclick/vaultier:latest -e "VAULTIER_DOMAIN=vaultier.bla.com"
Error: invalid value for -e 'VAULTIER_DOMAIN=vaultier.bla.com': bad logging level name 'VAULTIER_DOMAIN=vaultier.bla.com'
For help, use /usr/bin/supervisord -h
Starting the Docker without this value, gives a (400 Bad Request)
Using Ubuntu:
Traceback (most recent call last):
File "/opt/vaultier/venv/bin/vaultier", line 5, in <module>
from pkg_resources import load_entry_point
File "/opt/vaultier/venv/local/lib/python2.7/site-packages/pkg_resources.py", line 2720, in <module>
parse_requirements(__requires__), Environment()
File "/opt/vaultier/venv/local/lib/python2.7/site-packages/pkg_resources.py", line 592, in resolve
raise VersionConflict(dist,req) # XXX put more info here
pkg_resources.VersionConflict: (six 1.4.1 (/opt/vaultier/venv/lib/python2.7/site-packages), Requirement.parse('six>=1.7'))
Using Debian:
Traceback (most recent call last):
File "/opt/vaultier/venv/local/lib/python2.7/site-packages/pip/basecommand.py", line 122, in main
status = self.run(options, args)
File "/opt/vaultier/venv/local/lib/python2.7/site-packages/pip/commands/install.py", line 295, in run
requirement_set.install(install_options, global_options, root=options.root_path)
File "/opt/vaultier/venv/local/lib/python2.7/site-packages/pip/req.py", line 1436, in install
requirement.install(install_options, global_options, *args, **kwargs)
File "/opt/vaultier/venv/local/lib/python2.7/site-packages/pip/req.py", line 707, in install
cwd=self.source_dir, filter_stdout=self._filter_install, show_stdout=False)
File "/opt/vaultier/venv/local/lib/python2.7/site-packages/pip/util.py", line 716, in call_subprocess
% (command_desc, proc.returncode, cwd))
InstallationError: Command /opt/vaultier/venv/bin/python2 -c "import setuptools, tokenize;__file__='/tmp/pip-build-08foqW/psycopg2/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-UUDq9M-record/install-record.txt --single-version-externally-managed --compile --install-headers /opt/vaultier/venv/include/site/python2.7 failed with error code 1 in /tmp/pip-build-08foqW/psycopg2
Any ideas on how to avoid that fail, or fixing this?
UPDATE
after the "six" problem solution i received a new error while running vaultier setup
File "/opt/vaultier/venv/bin/vaultier", line 9, in <module>
load_entry_point('Vaultier==0.7.5', 'console_scripts', 'vaultier')()
File "/opt/vaultier/venv/local/lib/python2.7/site-packages/vaultier/vaultier/runner.py", line 231, in main
settings_envvar='VAULTIER_CONF'
File "/opt/vaultier/venv/local/lib/python2.7/site-packages/logan/runner.py", line 169, in run_app
management.execute_from_command_line([runner_name, command] + command_args)
File "/opt/vaultier/venv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
utility.execute()
File "/opt/vaultier/venv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/opt/vaultier/venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
self.execute(*args, **options.__dict__)
File "/opt/vaultier/venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
output = self.handle(*args, **options)
File "/opt/vaultier/venv/local/lib/python2.7/site-packages/vaultier/vaultier/management/commands/setup.py", line 22, in handle
management.call_command('syncdb')
File "/opt/vaultier/venv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 159, in call_command
return klass.execute(*args, **defaults)
File "/opt/vaultier/venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
output = self.handle(*args, **options)
File "/opt/vaultier/venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 415, in handle
return self.handle_noargs(**options)
File "/opt/vaultier/venv/local/lib/python2.7/site-packages/south/management/commands/syncdb.py", line 68, in handle_noargs
migrations = migration.Migrations(app_label)
File "/opt/vaultier/venv/local/lib/python2.7/site-packages/south/migration/base.py", line 64, in __call__
self.instances[app_label] = super(MigrationsMetaclass, self).__call__(app_label_to_app_module(app_label), **kwds)
File "/opt/vaultier/venv/local/lib/python2.7/site-packages/south/migration/base.py", line 90, in __init__
self.set_application(application, force_creation, verbose_creation)
File "/opt/vaultier/venv/local/lib/python2.7/site-packages/south/migration/base.py", line 154, in set_application
module = importlib.import_module(self.migrations_module())
File "/opt/vaultier/venv/local/lib/python2.7/site-packages/django/utils/importlib.py", line 40, in import_module
__import__(name)
File "/opt/vaultier/venv/local/lib/python2.7/site-packages/kombu/transport/django/migrations/__init__.py", line 16, in <module>
raise ImproperlyConfigured(SOUTH_ERROR_MESSAGE)
django.core.exceptions.ImproperlyConfigured:
For South support, customize the SOUTH_MIGRATION_MODULES setting
to point to the correct migrations module:
SOUTH_MIGRATION_MODULES = {
'kombu_transport_django': 'kombu.transport.django.south_migrations',
}
UPDATE2 -FIX
I couldnt find any solution online, so i tried another version of south and see there ... it works. :)
>>> DB is initialized, you can now try to run Vaultier using 'vaultier runserver'
(venv)root#Vaultier:/opt/vaultier# vaultier runserver
In the requirements.txt from vaultier, or setup.py ... doesnt matter which one you use. just set "South==1.0.2"
Dont forget to set "Six==1.9"
Have fun!
I found the same problem as you, the steps I did in Ubuntu to fix were:
Download Vaultier last version: wget https://pypi.python.org/packages/source/V/Vaultier/Vaultier-0.7.5.tar.gz
Unpack & unzip it: tar -xzvf Vaultier-0.7.5.tar.gz
Edit setup.py file and find the line that contains six==1.4,
change for this six==1.9,
Install it running: python setup.py install
If you wanna use the Docker Image the current Vaultier documentation is kind of confusing.
Try this:
Pull the latest Image (do not run the image!)
sudo docker pull rclick/vaultier:latest
Run the Image (replace EXAMPLE.COM with your desired Domain):
sudo docker run --name vaultier -p 80:80 -e "VAULTIER_DOMAIN=EXAMPLE.COM" rclick/vaultier:latest
After struggling many days, i found out how to install it :
If you install it with "apt-get install vaultier", then you will have the South and Six version issue, so just run in parallel "while [ 1 ]; do sed -i 's/six==1.4.1/six==1.9/' /tmp/pip*/setup.py; sed -i 's/South==0.8.4/South==1.0.2/' /tmp/pip*/setup.py;done 2>/dev/null"
After the installation succeed, if you run "vaultier check" and you got "No module named vaultier.runner", set the PYTHON_PATH variable : "export PYTHONPATH="/opt/vaultier/venv/local/lib/python2.7/site-packages/vaultier"", go to /opt/vaultier/venv/bin and try again.
I got issues with sending email (to invite and share with others). The environment variable FROM_EMAIL or VAULTIER_FROM_EMAIL wasn't effective, so Vaultier was using "noreply#vauliter.YOURDOMAIN.LOCAL" in the "from:" of the SMTP exchange.
I manually entered the desired email in "/opt/vaultier/venv/lib/python2.7/site-packages/vaultier/vaultier/business/mailer.py" line 33 : self.from_email = 'vaultier#YOURDOMAIN.COM'
If i got enough time, i'll make an OVA based on a CentOS 7 fully operationnal.
By the way, it's a good app, looks nice and do well it's job. By the way, it's the only free one that you can host. There was another one but ugly and not so easy to use.

Getting the following error: "EnvironmentError: Unable to find libfuse"

I'm trying to load test my MQTT network on a CentOS7 machine by using this repositorie on github. I executed the following commands to install the python-mosquitto dependency.
# yum install python-pip
# pip install mosquitto
Then I built and installed the setup.py file located in the repository file:
# python setup.py build
# python setup.py install
When trying to use the package, an error was thrown:
# malaria publish -P 2 -n 100 -H localhost -s 10
Error:
# malaria publish -P 2 -n 100 -H localhost -s 10malaria publish -P 2 -n 100 -H localhost -s 10
Traceback (most recent call last):
File "/usr/bin/malaria", line 5, in <module>
pkg_resources.run_script('mqtt-malaria==0.1-77-g7ae4c97', 'malaria')
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 540, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 1455, in run_script
execfile(script_filename, namespace, namespace)
File "/usr/lib/python2.7/site-packages/mqtt_malaria-0.1_77_g7ae4c97-py2.7.egg/EGG-INFO/scripts/malaria", line 30, in <module>
import beem.main
File "/usr/lib/python2.7/site-packages/mqtt_malaria-0.1_77_g7ae4c97-py2.7.egg/beem/main.py", line 34, in <module>
import beem.cmds
File "/usr/lib/python2.7/site-packages/mqtt_malaria-0.1_77_g7ae4c97-py2.7.egg/beem/cmds/__init__.py", line 2, in <module>
import beem.cmds.subscribe
File "/usr/lib/python2.7/site-packages/mqtt_malaria-0.1_77_g7ae4c97-py2.7.egg/beem/cmds/subscribe.py", line 34, in <module>
import beem.listen
File "/usr/lib/python2.7/site-packages/mqtt_malaria-0.1_77_g7ae4c97-py2.7.egg/beem/listen.py", line 40, in <module>
import fuse
File "build/bdist.linux-x86_64/egg/fuse.py", line 69, in <module>
EnvironmentError: Unable to find libfuse
Googling the error didn't provide me any answers. Did I do something wrong in the install process? I don't have much experience with the usage/building of cloned Github Repositories.
If running CentOS you may want to do "yum install fuse-python", which will install fuse-libs as dependencies. It worked for me after doing that.
The issues page on the repository (the issue right before the one you linked to) mentions exactly this problem.
The solution, as you've found is to install fusepy separately, either from your package manager or with pip install fusepy.

Fail to open up Scrapy on Linux Ubuntu

I've been getting error bringing up Scrapy on my Linux box. I'm using Ubuntu 14.0 Desktop. I've installed setuptools and scrapy. For some reason when I put
scrapy --version
I get this error:
Traceback (most recent call last):
File "/usr/local/bin/scrapy", line 11, in <module>
sys.exit(execute())
File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 122, in execute
cmds = _get_commands_dict(settings, inproject)
File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 46, in _get_commands_dict
cmds = _get_commands_from_module('scrapy.commands', inproject)
File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 29, in _get_commands_from_module
for cmd in _iter_command_classes(module):
File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 20, in _iter_command_classes
for module in walk_modules(module_name):
File "/usr/local/lib/python2.7/dist-packages/scrapy/utils/misc.py", line 68, in walk_modules
submod = import_module(fullpath)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/usr/local/lib/python2.7/dist-packages/scrapy/commands/bench.py", line 2, in <module>
from scrapy.tests.spiders import FollowAllSpider
ImportError: No module named tests.spiders
Any Idea what would this be from?
There was an issue with build 0.24.3 commited on Github.
To solve your problem just type this in terminal:
sudo pip install scrapy --upgrade
This will upgrade your version of Scrapy to 0.24.4.
Referring to the stackoverflow.com/q/25218633/190597. That will fix the issue. Or at least in my case.
Try uninstalling the old version of scrapy with:
pip uninstall Scrapy
and then installing it again using pip with this command:
pip install scrapy==0.24.2
well do not forget to install python development packages beforehand. That will raise another issue if not installed properly.

Why can't I install the openstack nova client on OS X?

I am attempting to install the openstack nova client on my Mac (10.4.8)
nova = https://github.com/openstack/python-novaclient#command-line-api
python --version
Python 2.7.2
I successfully got nova installed (after installing pip)
When I run the client, I get the following error
foo#bar-macbook-pro:~$ nova
Traceback (most recent call last):
File "/usr/local/bin/nova", line 6, in <module>
from novaclient.shell import main
File "/Library/Python/2.7/site-packages/novaclient/__init__.py", line 15, in <module>
import pbr.version
ImportError: No module named pbr.version
In my research, I have found conflicting information about pbr, some say it is required for nova, while others say it isn't required for nova.
https://github.com/rackspace/pyrax/issues/121
When I attempt to install pbr, I see the following error.
foo#bar-macbook-pro:~$ sudo python ~/Downloads/pbr/setup.py install
Traceback (most recent call last):
File "setup.py", line 22, in <module>
**util.cfg_to_args())
File "/Volumes/WDBlack750/spencerowen/Downloads/pbr/pbr/util.py", line 241, in cfg_to_args
pbr.hooks.setup_hook(config)
File "/Volumes/WDBlack750/spencerowen/Downloads/pbr/pbr/hooks/__init__.py", line 27, in setup_hook
metadata_config.run()
File "/Volumes/WDBlack750/spencerowen/Downloads/pbr/pbr/hooks/base.py", line 29, in run
self.hook()
File "/Volumes/WDBlack750/spencerowen/Downloads/pbr/pbr/hooks/metadata.py", line 28, in hook
self.config['name'], self.config.get('version', None))
File "/Volumes/WDBlack750/spencerowen/Downloads/pbr/pbr/packaging.py", line 817, in get_version
version = _get_version_from_git(pre_version)
File "/Volumes/WDBlack750/spencerowen/Downloads/pbr/pbr/packaging.py", line 776, in _get_version_from_git
"git --git-dir=\"" + git_dir + "\" describe --always").replace(
File "/Volumes/WDBlack750/spencerowen/Downloads/pbr/pbr/packaging.py", line 220, in _run_shell_command
stderr=err_location)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1228, in _execute_child
raise child_exception
TypeError: must be encoded string without NULL bytes, not str
Is there anything apparent that would explain why I can not get the library installed?
Surely I must not be the first person to try and install nova on OS X.
Over a year later, I finally got this working on OS X Yosemite
sudo pip install python-novaclient
I did not have to install pbr.
I did the install based on venv:
virtualenv venv_name
source venv_name/bin/activate
pip install python-novaclient fabric
In my case, I had a mixup in which python I was using by way of fabric being installed globally.
Prior: rf -rf all my virtualenvs, rf -rf all references to novaclient (locally, and globally), and deleted a global install of fabric which was calling the novaclient.
Also as a precaution, I do not install pip globally, and only use it without sudo in virtualenvs.

Registry error when installing MySQL-python into virtualenv using pip and easy_install

The pip and easy install commands (install MySQL-python) provide the massive error listed below.
I've been looking around for an answer a while now on this site and through google. The recurring answer is to install python dev (pip install python-dev) however the response I receive is "no distributions at all found"
Does anyone know the correct syntax in order to do this? or could someone be kind enough to point out where I'm going wrong.
All help is greatly appreciated. Thanks
C:\virtualenvs\myvirtualenv2\Scripts\pip-script.py run on 04/13/11 16:28:18
Downloading/unpacking mySQL-python
Running setup.py egg_info for package mySQL-python
Traceback (most recent call last):
File "<string>", line 14, in <module>
File "C:\virtualenvs\myvirtualenv2\build\mySQL-python\setup.py", line 15, in <module>
metadata, options = get_config()
File "setup_windows.py", line 7, in get_config
serverKey = _winreg.OpenKey(_winreg.HKEY_LOCAL_MACHINE, options['registry_key'])
WindowsError: [Error 2] The system cannot find the file specified
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 14, in <module>
File "C:\virtualenvs\myvirtualenv2\build\mySQL-python\setup.py", line 15, in <module>
metadata, options = get_config()
File "setup_windows.py", line 7, in get_config
serverKey = _winreg.OpenKey(_winreg.HKEY_LOCAL_MACHINE, options['registry_key'])
WindowsError: [Error 2] The system cannot find the file specified
----------------------------------------
Command python setup.py egg_info failed with error code 1
Exception information:
Traceback (most recent call last):
File "C:\virtualenvs\myvirtualenv2\lib\site-packages\pip-1.0-py2.6.egg\pip\basecommand.py", line 126, in main
self.run(options, args)
File "C:\virtualenvs\myvirtualenv2\lib\site-packages\pip-1.0-py2.6.egg\pip\commands\install.py", line 223, in run
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
File "C:\virtualenvs\myvirtualenv2\lib\site-packages\pip-1.0-py2.6.egg\pip\req.py", line 980, in prepare_files
req_to_install.run_egg_info()
File "C:\virtualenvs\myvirtualenv2\lib\site-packages\pip-1.0-py2.6.egg\pip\req.py", line 216, in run_egg_info
command_desc='python setup.py egg_info')
File "C:\virtualenvs\myvirtualenv2\lib\site-packages\pip-1.0-py2.6.egg\pip\__init__.py", line 255, in call_subprocess
% (command_desc, proc.returncode))
InstallationError: Command python setup.py egg_info failed with error code 1
Try ActivePython:
pypm -E C:\virtualenvs\myvirtualenv2 install mysql-python
no compile orgy. no strange errors. no terror.
Installing mysqldb on windows is tricky. I can install it manually, after downloading and unpacking mySQL-python:
Go to MySQL-python-1.2.3 folder
Edit the file site.cfg
Run regedit in Start > Run and verify the proper path for MySQL, mine looks like SOFTWARE\MySQL AB\MySQL Server 5.1
Uncomment the line registry_key
Set it:
registry_key = SOFTWARE\MySQL AB\MySQL Server 5.1
Run python setup.py install under your virtualenv

Categories

Resources