I have a command line tool like as follow:
tool -o output -a authentication.txt -i input.txt
(it gets some files as input)
what I would like to do is to provide an web interface for this command line tool using flask or django since the tool is based on python.
So can you guide me how can I call the tool and gets its results from stdout!
Also tool has the config file and I want to open and edit it in web interface too.
My initial idea was to call python subprocess to call the app to gets its stdout but do not know how wise it is?s
There are many ways to do it!
I have no idea what you are trying to do. But basic example would be something like this. You mention that it is python program/app. You can import you command line tool as a package. If it is not package, then package it https://packaging.python.org/. Parameters you pass as arguments, you can pass in your app. In Flask you would do something like this.
pseudo app.py
import tool
#app.route("/")
def index():
return render_template("index.html")
#app.route('/tool')
def tool(input):
auth = tool.auth()
input = tool.input(input)
return output
Assuming Python 3.5:
import subprocess
res = subprocess.run(["tool", "-o", "output", "-a", "authentication.txt", "-i", "input.txt"])
res.stdout
In an earlier version, you could potentially use the following, which would raise a CalledProcessError on non-zero return codes.
print(subprocess.check_output(
"/usr/local/bin/spam bacon spam",
stderr=subprocess.STDOUT,
shell=True))
# => EGGS AND SPAM!
Do be careful with running commands using untrustworthy (e.g. user) input, make sure that arguments are properly sanitized. Remember to test for malicious input, such as attempting to escape the command to run something different.
Try to ensure that the system user (typically the application user / web server user in this case) that winds up executing the command can't muck up too much if/when something inevitably makes it through by restricting its rights on the machine.
Related
Iv'e been using the following shell command to read the image off a scanner named scanner_name and save it in a file named file_name
scanimage -d <scanner_name> --resolution=300 --format=tiff --mode=Color 2>&1 > <file_name>
This has worked fine for my purposes.
I'm now trying to embed this in a python script. What I need is to save the scanned image, as before, into a file and also capture any std output (say error messages) to a string
I've tried
scan_result = os.system('scanimage -d {} --resolution=300 --format=tiff --mode=Color 2>&1 > {} '.format(scanner, file_name))
But when I run this in a loop (with different scanners), there is an unreasonably long lag between scans and the images aren't saved until the next scan starts (the file is created as an empty file and is not filled until the next scanning command). All this with scan_result=0, i.e. indicating no error
The subprocess method run() has been suggested to me, and I have tried
with open(file_name, 'w') as scanfile:
input_params = '-d {} --resolution=300 --format=tiff --mode=Color 2>&1 > {} '.format(scanner, file_name)
scan_result = subprocess.run(["scanimage", input_params], stdout=scanfile, shell=True)
but this saved the image in some kind of an unreadable file format
Any ideas as to what may be going wrong? Or what else I can try that will allow me to both save the file and check the success status?
subprocess.run() is definitely preferred over os.system() but neither of them as such provides support for running multiple jobs in parallel. You will need to use something like Python's multiprocessing library to run several tasks in parallel (or painfully reimplement it yourself on top of the basic subprocess.Popen() API).
You also have a basic misunderstanding about how to run subprocess.run(). You can pass in either a string and shell=True or a list of tokens and shell=False (or no shell keyword at all; False is the default).
with_shell = subprocess.run(
"scanimage -d {} --resolution=300 --format=tiff --mode=Color 2>&1 > {} ".format(
scanner, file_name), shell=True)
with open(file_name) as write_handle:
no_shell = subprocess.run([
"scanimage", "-d", scanner, "--resolution=300", "--format=tiff",
"--mode=Color"], stdout=write_handle)
You'll notice that the latter does not support redirection (because that's a shell feature) but this is reasonably easy to implement in Python. (I took out the redirection of standard error -- you really want error messages to remain on stderr!)
If you have a larger working Python program this should not be awfully hard to integrate with a multiprocessing.Pool(). If this is a small isolated program, I would suggest you peel off the Python layer entirely and go with something like xargs or GNU parallel to run a capped number of parallel subprocesses.
I suspect the issue is you're opening the output file, and then running the subprocess.run() within it. This isn't necessary. The end result is, you're opening the file via Python, then having the command open the file again via the OS, and then closing the file via Python.
JUST run the subprocess, and let the scanimage 2>&1> filename command create the file (just as it would if you ran the scanimage at the command line directly.)
I think subprocess.check_output() is now the preferred method of capturing the output.
I.e.
from subprocess import check_output
# Command must be a list, with all parameters as separate list items
command = ['scanimage',
'-d{}'.format(scanner),
'--resolution=300',
'--format=tiff',
'--mode=Color',
'2>&1>{}'.format(file_name)]
scan_result = check_output(command)
print(scan_result)
However, (with both run and check_output) that shell=True is a big security risk ... especially if the input_params come into the Python script externally. People can pass in unwanted commands, and have them run in the shell with the permissions of the script.
Sometimes, the shell=True is necessary for the OS command to run properly, in which case the best recommendation is to use an actual Python module to interface with the scanner - versus having Python pass an OS command to the OS.
So i have a script from Python that connects to the client servers then get some data that i need.
Now it will work in this way, my bash script from the client side needs input like the one below and its working this way.
client.exec_command('/apps./tempo.sh' 2016 10 01 02 03))
Now im trying to get the user input from my python script then transfer it to my remotely called bash script and thats where i get my problem. This is what i tried below.
Below is the method i tried that i have no luck working.
import sys
client.exec_command('/apps./tempo.sh', str(sys.argv))
I believe you are using Paramiko - which you should tag or include that info in your question.
The basic problem I think you're having is that you need to include those arguments inside the string, i.e.
client.exec_command('/apps./tempo.sh %s' % str(sys.argv))
otherwise they get applied to the other arguments of exec_command. I think your original example is not quite accurate in how it works;
Just out of interest, have you looked at "fabric" (http://www.fabfile.org ) - this has lots of very handy funcitons like "run" which will run a command on a remote server (or lots of remote servers!) and return you the response.
It also gives you lots of protection by wrapping around popen and paramiko for hte ssh login etcs, so it can be much more secure then trying to make web services or other things.
You should always be wary of injection attacks - Im unclear how you are injecting your variables, but if a user calls your script with something like python runscript "; rm -rf /" that would have very bad problems for you It would instead be better to have 'options' on the command, which are programmed in, limiting the users input drastically, or at least a lot of protection around the input variables. Of course if this is only for you (or trained people), then its a little easier.
I recommend using paramiko for the ssh connection.
import paramiko
ssh_client = paramiko.SSHClient()
ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh_client.connect(server, username=user,password=password)
...
ssh_client.close()
And If you want to simulate a terminal, as if a user was typing:
chan=ssh_client.invoke_shell()
chan.send('PS1="python-ssh:"\n')
def exec_command(cmd):
"""Gets ssh command(s), execute them, and returns the output"""
prompt='python-ssh:' # the command line prompt in the ssh terminal
buff=''
chan.send(str(cmd)+'\n')
while not chan.recv_ready():
time.sleep(1)
while not buff.endswith(prompt):
buff+=ssh_client.chan.recv(1024)
return buff[:len(prompt)]
Example usage: exec_command('pwd')
And the result would even be returned to you via ssh
Assuming that you are using paramiko you need to send the command as a string. It seems that you want to pass the command line arguments passed to your Python script as arguments for the remote command, so try this:
import sys
command = '/apps./tempo.sh'
args = ' '.join(sys.argv[1:]) # all args except the script's name!
client.exec_command('{} {}'.format(command, args))
This will collect all the command line arguments passed to the Python script, except the first argument which is the script's file name, and build a space separated string. This argument string is them concatenated with the bash script command and executed remotely.
I read here that it might be possible to use python interpreter to access Odoo and test things interactively (https://www.odoo.com/forum/help-1/question/how-to-get-a-python-shell-with-the-odoo-environment-54096), but doing this in terminal:
ipython
import sys
import openerp
sys.argv = ['', '--addons-path=~/my-path/addons', '--xmlrpc-port=8067', '--log-level=debug', '-d test',]
openerp.cli.main()
it starts Odoo server, but I can't write anything in that terminal tab to use it interactively. If for example I write anything like print 'abc', I don't get any output. Am I missing something here?
Sometime I use "logging" library for print output on the console/terminal.
For example:
import logging
logging.info('Here is your message')
logging.warning('Here is your message')
For more details, You may checkout this reference link.
The closest thing I have found to interactive is put the line
import pdb; pdb.set_trace()
in the method I want to inspect, and then trigger that method.
It's clunky, but it works.
As an example, I was just enhancing the OpenChatter implementation for our copy of OpenERP, and during the "figure things out" stage I had that line in .../addons/mail/mail_thread.py::mail_thread.post_message so I could get a better idea of what was happening in that method.
The correct way to do this is with shell:
./odoo-bin shell -d <yourdatabase>
Please, be aware that if you already have an instance of odoo, the port will be busy. In that case, the instance you are opening should be using a different port. So the command should be something like this:
./odoo-bin shell --xmlrpc-port=8888 -d <yourdatabase>
But if you want to have your addons available in the new instance, yo can make something similar to the following:
./odoo-bin shell -c ~/odooshell.conf -d <yourdatabase>
This way you can have in your odooshell.conf whatever you need to have configured (port, addons_path, etc). This way you can work smoothly with your shell.
As I always use docker, this is what I do to have my shell configured in docker:
docker exec -ti <mycontainer> odoo shell -c /etc/odoo/odooshell.conf -d <mydatabase>
You will have the env available to do anything. You can create express python code to make whatever you need. The syntax is very similar to server actions. For example:
partner_ids = env['res.partner'].search([])
for partner in partner_ids:
partner['name'] = partner.name + '.'
env.cr.commit()
Remember to env.cr.commit() if you make any data change.
I'm trying to copy thousands files to a remote server. These files are generated in real-time within the script. I'm working on a Windows system and need to copy the files to a Linux server (hence the escaping).
I currently have:
import os
os.system("winscp.exe /console /command \"option batch on\" \"option confirm off\" \"open user:pass#host\" \"put f1.txt /remote/dest/\"")
I'm using Python to generate the files but need a way to persist the remote connection so that I can copy each file, to the server, as it is generated (as opposed to creating a new connection each time). That way, I'll only need to change the field in the put option thus:
"put f2 /remote/dest"
"put f3 /remote/dest"
etc.
I needed to do this and found that code similar to this worked well:
from subprocess import Popen, PIPE
WINSCP = r'c:\<path to>\winscp.com'
class UploadFailed(Exception):
pass
def upload_files(host, user, passwd, files):
cmds = ['option batch abort', 'option confirm off']
cmds.append('open sftp://{user}:{passwd}#{host}/'.format(host=host, user=user, passwd=passwd))
cmds.append('put {} ./'.format(' '.join(files)))
cmds.append('exit\n')
with Popen(WINSCP, stdin=PIPE, stdout=PIPE, stderr=PIPE,
universal_newlines=True) as winscp: #might need shell = True here
stdout, stderr = winscp.communicate('\n'.join(cmds))
if winscp.returncode:
# WinSCP returns 0 for success, so upload failed
raise UploadFailed
This is simplified (and using Python 3), but you get the idea.
Instead of using an external program (winscp) you could also use an python ssh-library like pyssh.
You would have to start persistent WinSCP sub-process in Python and feed the put commands to its standard input continuously.
I do not have Python example for this, but there's an equivalent JScript example:
https://winscp.net/eng/docs/guide_automation_advanced#inout
or C# example:
https://winscp.net/eng/docs/guide_dotnet#input
Though using WinSCP .NET assembly via its COM interface for Python would be a way easier:
https://winscp.net/eng/docs/library
I searched the site, but I didn't see anything quite matching what I was looking for. I created a stand-alone application that uses a web service I created. To run the client I use:
C:/scriptsdirecotry> "run-client.bat" param1 param2 param3 param4
How would I go about coding this in Python or F#. It seems like it should be pretty simple, but I haven't seen anything online that quite matches what I'm looking for.
Python is similar.
import os
os.system("run-client.bat param1 param2")
If you need asynchronous behavior or redirected standard streams.
from subprocess import *
p = Popen(['run-client.bat', param1, param2], stdout=PIPE, stderr=PIPE)
output, errors = p.communicate()
p.wait() # wait for process to terminate
In F#, you could use the Process class from the System.Diagnostics namespace. The simplest way to run the command should be this:
open System.Diagnostics
Process.Start("run-client.bat", "param1 param2")
However, if you need to provide more parameters, you may need to create ProcessStartInfo object first (it allows you to specify more options).
Or you can use fsi.exe to call a F# script (.fsx). Given the following code in file "Script.fsx"
#light
printfn "You used following arguments: "
for arg in fsi.CommandLineArgs do
printfn "\t%s" arg
printfn "Done!"
You can call it from the command line using the syntax:
fsi --exec .\Script.fsx hello world
The FSharp interactive will then return
You used following arguments:
.\Script.fsx
hello
world
Done!
There is more information about fsi.exe command line options at msdn: http://msdn.microsoft.com/en-us/library/dd233172.aspx