Develop Reference

Running Selenium on EC2 works on server, but not from the browser

TLDR;
Python Selenium using ChromeDriver installed on EC2 instance not working when the script is called from the browser but working when same script called from the server. How to fix?
Hey guys,
I have installed Python 3.6, Selenium, ChromeDriver, and Google Chrome on my AWS EC2 instance. I have also enabled HTTP/2 and given ChromeDriver permissions.
I have a test Python script, let's call it test.py.
#!/usr/bin/env python
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.headless = True
driver = webdriver.Chrome('/usr/bin/chromedriver', options=options)
driver.get("https://www.youtube.com/")
element_text = driver.find_element_by_id("title").text
print(element_text)
In the terminal, I can simply run this script with:
python3 /var/app/current/scripts/test.py
and it works with no problem.
However, if I try to run this same script from the browser, (note that the URL is HTTPS), I get the following error:
[mpm_event:notice] AH00493: SIGUSR1 received. Doing graceful restart
[http2:warn] AH02951: mod_ssl does not seem to be enabled
[lbmethod_heartbeat:notice] AH02282: No slotmem from mod_heartmonitor
[mpm_event:notice] AH00489: Apache/2.4.41 (Amazon) configured -- resuming normal operations
[core:notice] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[mpm_event:warn] AH00488: long lost child came home!
Traceback (most recent call last):
File "/var/app/current/scripts/test.py", line 3, in <module>
from selenium import webdriver
ModuleNotFoundError: No module named 'selenium'
stating that there is no module named 'selenium'.
Note that I am calling it from the browser using PHP:
$output=shell_exec('python3 /var/app/current/scripts/test.py');
echo "<pre>$output</pre>";
I also know that the PHP is working because if I change test.py to simply:
import datetime from datetime
print(datetime.now())
It will print the date and time.
So, there seems to be a problem with Selenium in particular, but only from the browser.
Does anyone have any idea what this could be?
I appreciate all of your help / comments!
Thanks a lot :)
Brad
EDIT
I have since changed the script to:
sudo PYTHONUSERBASE=/home/ec2-user/.local python3 /var/app/current/scripts/test.py
and am now getting a slightly different error:
sudo: no tty present and no askpass program specified
I think I'll have to create a virtualenvironment and then use that when I call the script again.

file sourcing within cgi failed

My cgi file:
#!/bin/sh -f
source /Users/sfma/sastbx/build/setpaths_all.sh
python processQuery.py
Then "Internal Server Error" arises. I checked the error_log, and it says:
[Thu Jun 23 15:02:39.809441 2016] [cgi:error] [pid 1327] [client ::1:52131] AH01215: /Users/sfma/sastbx/build/bin/libtbx.path_utility: line 63: /Users/sfma/sastbx/build/../../Library/Enthought/Canopy_64bit/User/bin/python: Permission denied: /Library/WebServer/CGI-Executables/web.cgi, referer: http://localhost/modelRetrieval/query.html
Remove the source statement, it works fine. But I need to source this file for further use.
Edit
According to the error message, I think the error may be due to the python permission.
#!/bin/sh -f
#python processQuery.py #this works fine
/Users/sfma/Library/Enthought/Canopy_64bit/User/bin/python processQuery.py #this does not work. Permission denied.
This really puzzles me because /Users/sfma/Library/Enthought/Canopy_64bit/User/bin/python is exactly the python I'm using:
$which python
/Users/sfma/Library/Enthought/Canopy_64bit/User/bin/python

I finally solved this problem.
Since python works fine while /Users/sfma/Library/Enthought/canopy_64bit/User/bin/python leads to a problem, I hard coded some variables in files libtbx.path_utility and sastbx.python to "python" instead of what they used to be: "/Users/sfma/sastbx/build/../../Library/Enthought/canopy_64bit/User/bin/python"
Update
The reason why /Users/sfma/Library/Enthought/Canopy_64bit/User/bin/python is permission denied is that _www is now allowed to visit my Library folder. Check this. After making that directory available to all, this problem is finally solved.

Flask+Nginx+uWSGI : ImportError: No module named site

I installed as the http://www.reinbach.com/uwsgi-nginx-flask-virtualenv-mac-os-x.html link's tutorial and when executing the command uwsgi --ini deploy/deploy.ini, the terminal says there was an import error:
Set PYTHONHOME to /virtualenv/sample/
ImportError: No module named site
I have set my PYTHONHOME and PYTHONPATH as
export PYTHONPATH=$PYTHONPATH:/Library/Python/2.7/site-packages
export PYTHONHOME=$PYTHONHOME:/Library/Python/2.7
I cannot figure out what wrong with it.
Could someone help me with the problem?
The whole info in the terminal is shown as below if it is helpful:
(env)ios-devmatoMacBook-Pro:hello ios_dev$ uwsgi --ini deploy/deploy.ini
[uWSGI] getting INI configuration from deploy/deploy.ini
*** Starting uWSGI 1.9.10 (64bit) on [Fri May 17 16:42:22 2013] ***
compiled with version: 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00) on 17 May 2013 12:41:07
os: Darwin-11.4.2 Darwin Kernel Version 11.4.2: Thu Aug 23 16:25:48 PDT 2012; root:xnu-
1699.32.7~1/RELEASE_X86_64
nodename: ios-devmatoMacBook-Pro.local
machine: x86_64
clock source: unix
detected number of CPU cores: 4
current working directory: /Users/ios_dev/Desktop/sample/hello
detected binary path: /Users/ios_dev/Documents/little/little-web/little_web_dev/env/bin/uwsgi
your processes number limit is 709
your memory page size is 4096 bytes
detected max file descriptor number: 256
lock engine: OSX spinlocks
uwsgi socket 0 bound to TCP address 127.0.0.1:3031 fd 3
Python version: 2.7.1 (r271:86832, Jun 16 2011, 16:59:05) [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)]
Set PYTHONHOME to /virtualenv/sample/
ImportError: No module named site

Here's how I resolved the same error message (ImportError: No module named site) that I got while trying the Django and NGINX tutorial — uWSGI 2.0 documentation.
Deactivate the virtualenv:
deactivate
Install uWSGI system-wide (if not already installed system-wide)
sudo pip install uwsgi
Edit the uwsgi.ini file. I commented out the line with the:
home = /path/to/virtualenv
Run uWSGI --ini mysite_uwsgi.ini.

I read a lot of document about the question, but get no answer.
By coincidentally, I fix this problem by edit uid and gid as root.
It seem like a permissions problem. I don't know why, but it just work. Remember, it is very unsafe to run a product environment as root.

1、active the virtual you used
2、 pip install uwsgi
this the key action，then the
command -v wsgi
show this
/virtual-path/bin/uwsgi
3、use current user to run uwsgi, because other user is not active the virtualenv

Im My case I wasn't using virtualEnv. Just using django + ngnix. My solution was removing HOME variable into the *.ini configuration file:
sudo nano /etc/uwsgi/sites/c_app.ini
[uwsgi]
project = c_app
uid = ubuntu
base = /home/%(uid)
chdir = %(base)/%(project)
**home = %(base)/%(project)** (REMOVED IT)
module = %(project).wsgi:application
master = true
processes = 5
socket = /run/uwsgi/%(project).sock
chown-socket = %(uid):www-data
chmod-socket = 660
vacuum = true
then it works.

Calling Python from Maven in a Cygwin shell

When building with Maven from a Cygwin shell (bash on Windows via Cygwin), the path to /usr/bin is not resolving properly.
Let me explain. Cygwin comes with Python, which is accessible as a symlink from /usr/bin. This symlink should be accessible to Maven, because its location is in the PATH environment variable. Cygwin adds /usr/bin to the PATH environment variable. However, Maven is unable to find Python. E.g.
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (default) on project cloud-devcloud: Command execution failed. Cannot run program "python" (in directory "C:\cygwin\home\myuser\incubator-cloudstack\tools\devcloud"): CreateProcess error=2, The system cannot find the file specified -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
But Python is there:
$ which python
/usr/bin/python
Albeit as a symlink.
$ ls -al /usr/bin/python
lrwxrwxrwx 1 donall Domain Users 13 Sep 19 11:38 /usr/bin/python -> python2.6.exe
Is the issue that Python is a symlink? Or that Maven cannot access files in /usr/bin?

It seems like maven is running as a regular Windows-native application, not a "cygwin-aware application" (if such thing exists :-) )
This is indicated by the path used internally, which is printed in the log: C:\cygwin\home\myuser\incubator-cloudstack\tools\devcloud
For this reason the call from Maven will not recognize the cygwin path when locating the python executable. One possible solution is to add the directory containing python.exe to the system PATH (which means, the Windows PATH variable)

uWSGI + Django + Virtualenv unable to pick up _functools (import error)

OK, So I have tried this with & without a virtualenv:
uwsgi --home /home/auston/new_proj/ --socket /tmp/uwsgi2.sock --chmod-socket --module app_wsgi --pp /home/auston/new_proj/nikeshere --logto /tmp/uwsgi.log --master --processes 4 -P
Pretty much no matter what, I get this:
*** Starting uWSGI 0.9.6.5 (32bit) on [Thu Oct 21 08:05:44 2010] ***
compiled with version: 4.4.3
Python version: 2.6.6 (r266:84292, Oct 21 2010, 04:07:38)
[GCC 4.4.3]
your memory page size is 4096 bytes
allocated 412 bytes (0 KB) for 1 request's buffer.
Setting PythonHome to /home/auston/new_proj/...
binding on UNIX socket: /tmp/uwsgi2.sock
chmod() socket to 666 for lazy and brave users
your server socket listen backlog is limited to 64 connections
added /home/auston/new_proj/nikeshere to pythonpath.
initializing hooks...done.
['/home/auston/new_proj/nikeshere', '.', '', '/home/auston/new_proj/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg', '/home/auston/new_proj/lib/python2.6/site-packages/pip-0.8.1-py2.6.egg', '/home/auston/new_proj/lib/python26.zip', '/home/auston/new_proj/lib/python2.6', '/home/auston/new_proj/lib/python2.6/plat-linux2', '/home/auston/new_proj/lib/python2.6/lib-tk', '/home/auston/new_proj/lib/python2.6/lib-old', '/home/auston/new_proj/lib/python2.6/lib-dynload', '/usr/lib/python2.6', '/usr/lib/python2.6/plat-linux2', '/usr/lib/python2.6/lib-tk', '/home/auston/new_proj/lib/python2.6/site-packages', '/usr/local/lib/python2.6/dist-packages/pip-0.8.1-py2.6.egg', '/usr/local/lib/python2.6/site-packages', '/usr/local/lib/python2.6/dist-packages', '/usr/lib/python2.6/dist-packages', '/home/auston/new_proj/nikeshere', '/usr/local/lib/python2.6']
Traceback (most recent call last):
File "/home/auston/new_proj/nikeshere/app_wsgi.py", line 11, in <module>
import django.core.handlers.wsgi
File "/usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py", line 1, in <module>
from threading import Lock
File "/usr/lib/python2.6/threading.py", line 13, in <module>
from functools import wraps
File "/usr/lib/python2.6/functools.py", line 10, in <module>
from _functools import partial, reduce
ImportError: No module named _functools
If I change --home to /usr/local/lib/python/2.6 I get fail on my app_wsgi.py import of os. Here it is, below, just in case:
import sys
import os
sys.path.append(os.path.abspath(os.path.dirname(__file__)))
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()
Essentially I am asking, how can I get uWSGI to recognize functools OR get on the right path (path is in output above). I would appreciate any help you guys can give!!
P.S. Ubuntu 10.04 - uWSGI 0.9.6.5 - NGINX 0.8.53 - virtual env Python 2.6.5 - "regular (or system)" Python 2.6.6 - Django 1.2.3
UPDATE:
I was able to get uwsgi to start accepting requests if I omit the "--module" like so:
uwsgi --home /home/auston/new_proj --socket /tmp/uwsgi2.sock --chmod-socket --pp /home/auston/new_proj/nikeshere --logto /tmp/uwsgi.log --master --processes 4 -P
but now I get a app not found error:
"uWSGI Error
wsgi application not found"
I'm closer but I would still appreciate suggestions as the app is not found because i cannot include the module needed to load it!

So as noted above, the problem has been with the pythonpath & it's inability to find a module named _functools.
Apparently, _functools is a c module & I needed to append the it's path to the pythonpath in order for it to be found, so the difference from the original wsgi.py, is now:
import sys
sys.path.append('/usr/local/lib/python2.6/lib-dynload') # to load _functools
sys.path.append('/usr/local/lib/python2.6/site-packages') # to load django
sys.path.append('/usr/local/lib/python2.6/dist-packages') # cautionary to load django
sys.path.append('/usr/lib/python2.6') # to load os
import os
os.environ['DJANGO_SETTINGS_MODULE'] = 'iwin.settings'
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()
Very hacky, but it works for now...

I know it's old topic and versions of stack building blocks changed, but I had the same problem with not recognizing under uWSGi installed libs in virtualenv. The solution is to point home parameter to virtualenv, as shown below (taken from https://uwsgi.readthedocs.org/en/latest/tutorials/Django_and_nginx.html).
So for me command:
uwsgi --http :8000 --module ii.wsgi --home /home/dev/.virtualenvs/ii_env/
worked, while being in the django application (ii) directory.
# mysite_uwsgi.ini file
[uwsgi]
# Django-related settings
# the base directory (full path)
chdir = /path/to/your/project
# Django's wsgi file
module = project.wsgi
# the virtualenv (full path)
home = /path/to/virtualenv
# process-related settings
# master
master = true
# maximum number of worker processes
processes = 10
# the socket (use the full path to be safe
socket = /path/to/your/project/mysite.sock
# ... with appropriate permissions - may be needed
# chmod-socket = 664
# clear environment on exit
vacuum = true

Check out http://blog.zacharyvoase.com/2010/03/05/django-uwsgi-nginx/. He's using very similar set up.

I've been having very similar problem and I found this:
When you install virtuelenv, it 'installs' the Python standard library by creating symlinks to the original one (in like /usr/lib/python2.7). But when you check your virtualenv Python lib directory, there are symlinks created for only a few basic libraries. Your functools is probably not among them.
So the solution is to create the symlink manually. It is a PITA, because you may have to create a lot of symlinks, but it seems like a cleaner solution to me. You don't have to hack any source files and it's transparent.
The symlink should be created not in the root of the venv_directory, but in e.g.
venv_directory/lib/python2.7/site-packages/
Hope it works for you!