Develop Reference

python import works in terminal and fails in browser - from google.cloud import storage

simple 3 line python script:
print("A")
from google.cloud import storage
print("B")
works: in terminal session -- output is A then B
works: sudo _www 3line.py -- output is A then B
Fails: called from a php, output is "A" and then nothing. No error logs produced in apacheDev-error_log.
It seems that from google.ANYTHING import is just plain stopping.
Failure occurs in Chrome and FireFox.
I think the problem is related to the browser environment, which is why I first tried the sudo to _www --- just hacking. It worked, so I thought that the Mac OS bash environment was handling it right under the _www id. Just seems that I am unaware of something related to requirements about browsers running python imports from google.cloud....
Any ideas?
testing.php is below:
$command = escapeshellcmd('/Users/admin/AJweb/3line.py');
echo $command;
$output = shell_exec($command);
echo $output;

Calling Matlab scripts from Django with Python's Popen class

I'm developing a Django app which runs Matlab scripts with Python's Popen class. The python script that calls Matlab scripts lives in the main folder of my Django app (with views.py). When I call the script from command line, it runs like a charm but when I make a request from the client in order to run the corresponding python script, I receive the following warning:
"< M A T L A B (R) > Copyright 1984-2018 The MathWorks, Inc. R2018a (9.4.0.813654) 64-bit (glnxa64) February 23, 2018 To get started, type one of these: helpwin, helpdesk, or demo. For product information, visit www.mathworks.com. >> [Warning: Unable to create preferences folder in /var/www/.matlab/R2018a. Preferences folder location must be writable. Using a temporary preferences folder for this MATLAB session. See the preferences documentation for more details.] >>
My app uses a Python virtual environment and it is being deployed with Apache web server.
Here is my python script that calls Matlab scripts:
import os
import subprocess as sp
import pymat_config
def pymat_run():
pwd = pymat_config.pwd_config['pwd']
cmd1 = "-r \"Arg_in = '/path/to/my/main/folder/input.txt'; Arg_out = '/path/to/my/main/folder/file.txt'; matlab_script1\""
baseCmd1 = ['/usr/local/MATLAB/R2018a/bin/matlab', '-nodesktop', '-nosplash', '-nodisplay', 'nojvm', cmd1]
os.chdir('/path/to/matlab_script1')
sudo_cmd = sp.Popen(['echo', pwd], stdout=sp.PIPE)
exec1 = sp.Popen(['sudo', '-S'] + baseCmd1, stdin=sudo_cmd.stdout, stdout=sp.PIPE, stderr=sp.PIPE)
out, err = exec1.communicate()
return out
Any suggestions ?

Finally I managed to find the solution of that issue by myself. The problem came from the kind of user who called the Matlab's script. When I was running the above script from a Python interpreter or from the shell, it was the user (with the user password) who was running the script while when I was calling the script from the client the user was the web server's user: www-data.
So at first to avoid the above warning I gave permissions to www-data user to the /var/www folder with the following command:
sudo chown -R www-data /var/www/
After that, the "Warning" disappeared but the script still didn't run because it was asking for www-data's password internally and taking user's password from pymat_config file.
To solve this, I edited /etc/sudoers file in order for www-data to be able to call Matlab scripts without asking password. So I added the following line:
www-data ALL=(ALL) NOPASSWD: /usr/local/MATLAB/R2018a/bin/matlab
and now it runs like a charm !

ec2 run scripts every boot

I have followed a few posts on here trying to run either a python or shell script on my ec2 instance after every boot not just the first boot.
I have tried the:
[scripts-user, always] to /etc/cloud/cloud.cfg file
Added script to ./scripts/per-boot folder
and
adding script to /etc/rc.local
Yes the permissions were changed to 755 for /etc/rc.local
I am attempting to pipe the output of the file into a file located in the /home/ubuntu/ directory and the file does not contain anything after boot.
If I run the scripts (.sh or .py) manually they work.
Any suggestions or request for additional info to help?

So the current solution appears to be a method I wrote off in my initial question post as I may have not performed the setup exactly as outline in the link below...
This link -->
How do I make cloud-init startup scripts run every time my EC2 instance boots?
The link shows how to modify the /etc/cloud/cloud.cfg file to update scripts-user to [scripts-user, always]
Also that link says to add your *.sh file to /var/lib/cloud/scripts/per-boot directory.
Once you reboot your system your script should have executed and you can verify this in: sudo cat /var/log/cloud-init.log
if your script still fails to execute try to erase the instance state of your server with the following command: sudo rm -rf /var/lib/cloud/instance/*
--NOTE:--
It appears print commands from a python script do not pipe (>>) as expected but echo commands pipe easily
Fails to pipe
sudo python test.py >> log.txt
Pipes successfully
echo "HI" >> log.txt

Is this something along the lines that you want?
It copies the script to the instance, connects to the instance, and runs the script right away.
ec2 scp ~/path_to_script.py : instance_name -y && ec2 ssh instance_name -yc "python script_name.py" 1>/dev/null

I read that the use of rc.local is getting deprecated. One thing to try is a line in /etc/crontab like this:
#reboot full-path-of-script
If there's a specific user you want to run the script as, you can list it after #reboot.

Running sudo command via CGI (Python)

I am writing a test suite for a web application using Selenium.
In the course of which I need to test behaviour of the app in case a certain service is running or not.
I wanted to create a cgi call to a Python script turning that service on and off.
I know that the cgi call is in the context of the webserver (Apache) however thought that issuing sudo calls like so:
import subprocess
import os
command = 'sudo -S launchctl unload /Library/LaunchAgents/com.my.daemon.plist'
pwd = 'pwd123'
test1 = subprocess.Popen( command, shell=True, stdin=subprocess.PIPE)
test1.communicate(input=pwd)
test2 = os.system( 'echo %s|%s' % (pwd,command) )
would do the trick, well they don't I get return code 256.
What can I do to have this call be executed w/o touching the context in which Apache runs?
As for security: this will only run on a test machine.

The user that Apache runs as needs to be in the /etc/sudoers file, or belong to the sudo group, which I guess it usually doesn't. You also need to make it not ask for a password, which is configured in /etc/sudoers
For Ubuntu, check these out: https://askubuntu.com/questions/7477/how-can-i-add-a-new-user-as-sudoer-using-the-command-line
https://askubuntu.com/questions/147241/execute-sudo-without-password

It could potentially be a pathing issue..
Have you tried writing out the full path like this:
command = '/usr/bin/sudo -S launchctl unload /Library/LaunchAgents/com.my.daemon.plist'

command should be a list, not a string. Try with:
command = ['sudo', '-S', 'launchctl', 'unload', '/Library/LaunchAgents/com.my.daemon.plist']

Cant run sudo this way -- sudo needs a controlling terminal to run.

Drush commands not executing using Paramiko

I've followed the steps here http://jessenoller.com/2009/02/05/ssh-programming-with-paramiko-completely-different/
to connect to my server with ssh via Python. I can connect fine and send commands.
When I run stderr.readlines(), however, it shows me the error message below every time, even if the command seems to have executed correctly. I've closed the connection and restarted Python, and still the same result.
Here's a Python sample:
>>> stdin, stdout, stderr = myssh.exec_command("xyz")
>>> stderr.readlines()
['which: no php in (/usr/bin:/bin:/usr/sbin:/sbin:/big/dom/mydomain/pear/drush)\n', '/big/dom/mydomain/pear/drush/drush: line 89: exec: : not found\n', 'bash: xyz: command not found\n']
I have drush installed and it seems to work fine. If I type in "which php" on the server I'm told where it resides, instead of the error message above. I sent some other commands to purposefully get an error message to see if it cleared anything out. Instead it tacked things on at the end.
Following the error message, I went and looked at the drush file referenced. Here's line 89:
exec "$php" $php_options "$SCRIPT_PATH" --php="$php" --php-options="$php_options" "$#"
I believe the "which php" command comes from the $php variable in the chunk above this line
if [ ! -z "$DRUSH_PHP" ] ; then
# Use the DRUSH_PHP environment variable if it is available.
php="$DRUSH_PHP"
else
# Default to using the php that we find on the PATH.
# Note that we need the full path to php here for Dreamhost, which behaves oddly. See http://drupal.org/node/662926
php=`which php`
# We check for a command line (cli) version of php, and if found use that.
which php-cli >/dev/null 2>&1
if [ "$?" = 0 ] ; then
php=`which php-cli`
fi
# On MSYSGIT, we need to use "php", not the full path to php
if [ ! -z "$MSYSTEM" ] && [ "x${MSYSTEM:0:5}" = "xMINGW" ] ; then
php="php"
fi
fi
The full text of the file is here: http://pastebin.com/29AXmHKF
I get the same error if I try to execute any drush command. But drush commands work fine if I just log myself into the server directly w/o using python/paramiko.

The first thing I had to understand is what the $PATH variable held at the time of executing the command. I ran
>>> stdin, stdout, stderr = myssh.exec_command("echo $PATH")
>>> stderr.readlines()
and realized my $PATH was not the same as when I run echo $PATH directly on the server! I can only guess that extra paths get appended to the $PATH variable at some point after the channel is opened and my command is sent.
However, what $PATH did contain was the path to drush which I previously added to the .bashrc file in my home folder. So, all I had to do was also add the path to php there as well (even though that path is there when I run "echo $PATH" on the server).
Now I don't get the error message and I can execute drush commands.

I used Mike Ryan's solution (thanks Mike!) but found the information in stdout, not stderr.
stdin, stdout, stderr = server.ssh_client.exec_command("echo $PATH")
print stdout.readlines()

What happens if you ssh to that server interactively and run xyz?
You only get to read the error message when you actually read it, not when you send the command. (Thank you, Captain.)
The error output looks very much like as if your xyz is a PHP script that starts with a #!which php shebang line. But the shell cannot find any PHP executable. This may be due to PATH not being set correctly in the login script. Make sure you understand which login script runs when you ssh to the box (usually it's ~/.bash_profile and/or ~/.profile and not necessarily ~/.bashrc).