I have a bunch of python code that is basically executed via Click CLI framework entry points.
I am exploring how to make some of CLI functions into WebActions, and was looking IBM Cloud Functions, which is basically Apache OpenWhisk.
I am brand new to OpenWhisk and IBM CloudFunctions.
I am following the Help Docs here:
https://console.bluemix.net/docs/openwhisk/openwhisk_actions.html#creating-python-actions
trying to mimic the virtualenv method.
When I translate their basic example to be Click CLI Commands as follows:
(below is the contents of a file __main__.py which started out as a file named hello_too.py but changed as following along with IBM Docs)
import click
#click.command()
#click.argument('params', nargs=-1)
def main(params):
#name = args.get("name", "stranger")
greeting = "Hello " + "foo" + "!"
print(greeting)
return {"greeting": greeting}
if __name__ == "__main__":
main()
and then zip it and upload (as per their virtualenv example) as web action I get the following error
{
"error": "The action did not produce a valid JSON response: Internal Server Error"
}
I saw on some other blogs, that running python with -i is a good sanity check for OpenWhisk runtime.
When I run this code with -i I get a stack-trace around system exit.
Traceback (most recent call last):
File "hello_too.py", line 12, in <module>
main()
File "/Users/mcmasty/projects/ppf-github/experiments/ibm-api-connect/venv/lib/python2.7/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/Users/mcmasty/projects/ppf-github/experiments/ibm-api-connect/venv/lib/python2.7/site-packages/click/core.py", line 700, in main
ctx.exit()
File "/Users/mcmasty/projects/ppf-github/experiments/ibm-api-connect/venv/lib/python2.7/site-packages/click/core.py", line 484, in exit
sys.exit(code)
SystemExit: 0
but when I run the example code, non-Click enabled, the interactive interpreter does complain.
Any advice at easiest path to port Click CLI scripts to be OpenWhisk Actions / IBM Cloud Functions ?
I tried to get click standalone_mode to change SystemExit behavior, but could not seem to get it to work
I also tried this with naming the command echo and using --main echo option on OpenWhisk action create. (Same result)
I also tried many variations of returning JSON String (via json.dumps()), either through return or via writing to stdout, both in zip packaging and Docker image packaging... (same results)
Since the python dictionary is basically the hard-coded result, my best guess right now is that this stack trace when running click enabled script is the root of my problem when deploying to IBM Cloud Functions.
Thanks in advance.
Additional info in response to comments
Code provided above. That code is in in a file called __main__.py (as per the IBM Docs https://console.bluemix.net/docs/openwhisk/openwhisk_actions.html#creating-python-actions )
then following the IBM Docs its...
docker run --rm -v "$PWD:/tmp" openwhisk/python2action bash -c "cd tmp && virtualenv virtualenv && source virtualenv/bin/activate && pip install -r requirements.txt"
the only requirement in requirements.txt is click
then also following along the IBM Docs
zip -r hello_too.zip virtualenv __main__.py
and a sanity check
python -i hello_too.zip
Throws the SystemExit exception / stack trace similar to example above.
But
python hello_too.zip
completes normally.
Then deploy to Cloud Functions / Web Actions
ibmcloud wsk action create hello_too --kind python:2 hello_too.zip --web true
then invoke via command line
ibmcloud wsk action invoke --result hello_too
I get the following message:
{
"error": "The action did not produce a valid JSON response: Internal Server Error"
}
but the hard-coded response
return {"greeting": greeting}
is identical to their sample code in the "Creating and invoking a Python action section https://console.bluemix.net/docs/openwhisk/openwhisk_actions.html#creating-python-actions
so I am assuming this is not root-cause of the issue. (I ran their sample code, as outlined in the docs and returning a Python dict worked fine.)
Its just when I try to use click version of python, I am getting stuck.
The click module is causing a runtime error which kills the underlying Python process running the code.
The click module is designed to build command line interface tools. Python code for OpenWhisk Actions is dynamically evaluated and invoked by an existing Python script. You will need to re-factor your application to expose the core functions through raw functions than the click module.
Related
I'm developing a Django app which runs Matlab scripts with Python's Popen class. The python script that calls Matlab scripts lives in the main folder of my Django app (with views.py). When I call the script from command line, it runs like a charm but when I make a request from the client in order to run the corresponding python script, I receive the following warning:
"< M A T L A B (R) > Copyright 1984-2018 The MathWorks, Inc. R2018a (9.4.0.813654) 64-bit (glnxa64) February 23, 2018 To get started, type one of these: helpwin, helpdesk, or demo. For product information, visit www.mathworks.com. >> [Warning: Unable to create preferences folder in /var/www/.matlab/R2018a. Preferences folder location must be writable. Using a temporary preferences folder for this MATLAB session. See the preferences documentation for more details.] >>
My app uses a Python virtual environment and it is being deployed with Apache web server.
Here is my python script that calls Matlab scripts:
import os
import subprocess as sp
import pymat_config
def pymat_run():
pwd = pymat_config.pwd_config['pwd']
cmd1 = "-r \"Arg_in = '/path/to/my/main/folder/input.txt'; Arg_out = '/path/to/my/main/folder/file.txt'; matlab_script1\""
baseCmd1 = ['/usr/local/MATLAB/R2018a/bin/matlab', '-nodesktop', '-nosplash', '-nodisplay', 'nojvm', cmd1]
os.chdir('/path/to/matlab_script1')
sudo_cmd = sp.Popen(['echo', pwd], stdout=sp.PIPE)
exec1 = sp.Popen(['sudo', '-S'] + baseCmd1, stdin=sudo_cmd.stdout, stdout=sp.PIPE, stderr=sp.PIPE)
out, err = exec1.communicate()
return out
Any suggestions ?
Finally I managed to find the solution of that issue by myself. The problem came from the kind of user who called the Matlab's script. When I was running the above script from a Python interpreter or from the shell, it was the user (with the user password) who was running the script while when I was calling the script from the client the user was the web server's user: www-data.
So at first to avoid the above warning I gave permissions to www-data user to the /var/www folder with the following command:
sudo chown -R www-data /var/www/
After that, the "Warning" disappeared but the script still didn't run because it was asking for www-data's password internally and taking user's password from pymat_config file.
To solve this, I edited /etc/sudoers file in order for www-data to be able to call Matlab scripts without asking password. So I added the following line:
www-data ALL=(ALL) NOPASSWD: /usr/local/MATLAB/R2018a/bin/matlab
and now it runs like a charm !
i have compiled myscripts to myscripts.exe file using pyinstaller --onefile
myscripts.py contain
import os
os.popen("python celery -A tasks worker --loglevel=info -P solo -c 1")
Once i got .exe file
SimpleSC::InstallService "ERP" "ERP Data Cloud" "16" "2" "$INSTDIR\myscript1.exe" "" "" ""
SimpleSC::StartService "ERP" "" 30
i compiled using nsis got my setup.exe
Now when i see the service window i can see the service is added bt the status is blank, even i try to start the service manually i got an error
control request is not timely fashion
after install
!define MUI_FINISHPAGE_RUN "$INSTDIR\dist\myscripts.exe"
i am able to run myscripts.exe which start celery with no problem, But i want it to run in service .
now Question
Am i doing it completly worng way, Or i need to add something,
What am i missing.
A service has to call specific service functions. If you are unable to write a proper service you could try a helper utility like Srvany...
I have followed a few posts on here trying to run either a python or shell script on my ec2 instance after every boot not just the first boot.
I have tried the:
[scripts-user, always] to /etc/cloud/cloud.cfg file
Added script to ./scripts/per-boot folder
and
adding script to /etc/rc.local
Yes the permissions were changed to 755 for /etc/rc.local
I am attempting to pipe the output of the file into a file located in the /home/ubuntu/ directory and the file does not contain anything after boot.
If I run the scripts (.sh or .py) manually they work.
Any suggestions or request for additional info to help?
So the current solution appears to be a method I wrote off in my initial question post as I may have not performed the setup exactly as outline in the link below...
This link -->
How do I make cloud-init startup scripts run every time my EC2 instance boots?
The link shows how to modify the /etc/cloud/cloud.cfg file to update scripts-user to [scripts-user, always]
Also that link says to add your *.sh file to /var/lib/cloud/scripts/per-boot directory.
Once you reboot your system your script should have executed and you can verify this in: sudo cat /var/log/cloud-init.log
if your script still fails to execute try to erase the instance state of your server with the following command: sudo rm -rf /var/lib/cloud/instance/*
--NOTE:--
It appears print commands from a python script do not pipe (>>) as expected but echo commands pipe easily
Fails to pipe
sudo python test.py >> log.txt
Pipes successfully
echo "HI" >> log.txt
Is this something along the lines that you want?
It copies the script to the instance, connects to the instance, and runs the script right away.
ec2 scp ~/path_to_script.py : instance_name -y && ec2 ssh instance_name -yc "python script_name.py" 1>/dev/null
I read that the use of rc.local is getting deprecated. One thing to try is a line in /etc/crontab like this:
#reboot full-path-of-script
If there's a specific user you want to run the script as, you can list it after #reboot.
I have a Django project hosted on a remote server. This contains a file called tmp_file.py. There's a function called fetch_data() inside that file. Usually I follow the below approach to run that function.
# Inside Django Project
$ python manage.py shell
[Shell] from tmp_file import feth_data
[Shell] fetch_data()
Also the file doesn't contain __name__ section. So can't run as a stand alone script. What's the best way to perform this task using Ansible. I couldn't find anything useful from Ansible docs.
There's --command switch for shell django-admin command.
So you can try in Ansible:
- name: Fetch data
command: "django-admin shell --command='from tmp_file import feth_data; fetch_data()'"
args:
chdir: /path/to/tmp_file
Recently, when I was installing openstack on 3 vm on centos 7 using answer file I had the following error:
10.7.35.174_osclient.pp: [ ERROR ]
Applying Puppet manifests [ ERROR ]
ERROR : Error appeared during Puppet run: 10.7.35.174_osclient.pp
Error: Execution of '/usr/bin/yum -d 0 -e 0 -y list python-iso8601' returned 1: Error: No matching Packages to list
You will find full trace in log /var/tmp/packstack/20160318-124834-91QzZC/manifests/10.7.35.174_osclient.pp.log
Please check log file /var/tmp/packstack/20160318-124834-91QzZC/openstack-setup.log for more information
Additional information:
* Time synchronization installation was skipped. Please note that unsynchronized time on server instances might be problem for some OpenStack components.
* File /root/keystonerc_admin has been created on OpenStack client host 10.7.35.174. To use the command line tools you need to source the file.
* To access the OpenStack Dashboard browse to http://10.7.35.174/dashboard .
Please, find your login credentials stored in the keystonerc_admin in your home directory.
I have already manually installed that module, but the same problem occures anyway.
That command only runs like that:
/usr/bin/yum -d 0 -e 0 -y list python2-iso8601
Is there any way to parse it to python?
Do you have any ideas how to solve it?
Found that kilo version works fine.