Per Python documentation, subprocess.call should be blocking and wait for the subprocess to complete. In this code I am trying to convert few xls files to a new format by calling Libreoffice on command line. I assumed that the call to subprocess call is blocking but seems like I need to add an artificial delay after each call otherwise I miss few files in the out directory.
what am I doing wrong? and why do I need the delay?
from subprocess import call
for i in range(0,len(sorted_files)):
args = ['libreoffice', '-headless', '-convert-to',
'xls', "%s/%s.xls" %(sorted_files[i]['filename'],sorted_files[i]['filename']), '-outdir', 'out']
call(args)
var = raw_input("Enter something: ") # if comment this line I dont get all the files in out directory
EDIT It might be hard to find the answer through the comments below. I used unoconv for document conversion which is blocking and easy to work with from an script.
It's possible likely that libreoffice is implemented as some sort of daemon/intermediary process. The "daemon" will (effectively1) parse the commandline and then farm the work off to some other process, possibly detaching them so that it can exit immediately. (based on the -invisible option in the documentation I suspect strongly that this is indeed the case you have).
If this is the case, then your subprocess.call does do what it is advertised to do -- It waits for the daemon to complete before moving on. However, it doesn't do what you want which is to wait for all of the work to be completed. The only option you have in that scenario is to look to see if the daemon has a -wait option or similar.
1It is likely that we don't have an actual daemon here, only something which behaves similarly. See comments by abernert
The problem is that the soffice command-line tool (which libreoffice is either just a link to, or a further wrapper around) is just a "controller" for the real program soffice.bin. It finds a running copy of soffice.bin and/or creates on, tells it to do some work, and then quits.
So, call is doing exactly the right thing: it waits for libreoffice to quit.
But you don't want to wait for libreoffice to quit, you want to wait for soffice.bin to finish doing the work that libreoffice asked it to do.
It looks like what you're trying to do isn't possible to do directly. But it's possible to do indirectly.
The docs say that headless mode:
… allows using the application without user interface.
This special mode can be used when the application is controlled by external clients via the API.
In other words, the app doesn't quit after running some UNO strings/doing some conversions/whatever else you specify on the command line, it sits around waiting for more UNO commands from outside, while the launcher just runs as soon as it sends the appropriate commands to the app.
You probably have to use that above-mentioned external control API (UNO) directly.
See Scripting LibreOffice for the basics (although there's more info there about internal scripting than external), and the API documentation for details and examples.
But there may be an even simpler answer: unoconv is a simple command-line tool written using the UNO API that does exactly what you want. It starts up LibreOffice if necessary, sends it some commands, waits for the results, and then quits. So if you just use unoconv instead of libreoffice, call is all you need.
Also notice that unoconv is written in Python, and is designed to be used as a module. If you just import it, you can write your own (simpler, and use-case-specific) code to replace the "Main entrance" code, and not use subprocess at all. (Or, of course, you can tear apart the module and use the relevant code yourself, or just use it as a very nice piece of sample code for using UNO from Python.)
Also, the unoconv page linked above lists a variety of other similar tools, some that work via UNO and some that don't, so if it doesn't work for you, try the others.
If nothing else works, you could consider, e.g., creating a sentinel file and using a filesystem watch, so at least you'll be able to detect exactly when it's finished its work, instead of having to guess at a timeout. But that's a real last-ditch workaround that you shouldn't even consider until eliminating all of the other options.
If libreoffice is being using an intermediary (daemon) as mentioned by #mgilson, then one solution is to find out what program it's invoking, and then directly invoke it yourself.
Related
This Python program enters fdisk. I see the output. fdisk is an interactive program. How do I get the Python program to pass an "m" to the first field and press enter?
import subprocess
a = "dev/sda"
x = subprocess.call(["fdisk", a])
print x
I'd rather not import a new module/library, but I could. I've tried different syntax with subprocess.call() and extra parameters in the above. Nothing seems to work. I get different errors. I've reviewed Python documentation. I want to feed input and press Enter in the subsequent, interactive menu options of fdisk.
Check out the pexpect library (I know you didn't want an extra module, but you want to use the best tool for the job). It's pure Python, with no compiled submodules, so installation is a snap. Basically, it does the same thing in Python as the classic Unix utility expect - spawns child applications, controls them, and responds to expected patterns in their output. It's great for automation, and especially application testing, where you can quickly feed the newest build of a command-line program a series of inputs and guide the interaction based on what output appears.
In case you just don't want another module at all, you can always fall back on the subprocess module's Popen() constructor. It spawns and creates a connection to a child process, allowing you to communicate with it as needed, and in fact pexpect relies a great deal on it. I personally think using pexpect is more intuitive than subprocess.Popen(), but that's just me. YMMV.
I'm trying to run an external, separate program from Python. It wouldn't be a problem normally, but the program is a game, and has a Python interpreter built into it. When I use subprocess.Popen, it starts the separate program, but does so under the original program's Python instance, so that they share the first Python console. I can end the first program fine, but I would rather have separate consoles (mainly because I have the console start off hidden, but it gets shown when I start the program from Python with subprocess.POpen).
I would like it if I could start the second program wholly on its own, as though I just 'double-clicked on it'. Also, os.system won't work because I'm aiming for cross-platform compatibility, and that's only available on Windows.
I would like it if I could start the second program wholly on its own, as though I just 'double-clicked on it'.
As of 2.7 and 3.3, Python doesn't have a cross-platform way to do this. A new shutil.open method may be added in the future (possibly not under that name); see http://bugs.python.org/issue3177 for details. But until then, you'll have to write your own code for each platform you care about.
Fortunately, what you're trying to do is simpler and less general than what shutil.open is ultimately hoped to provide, which means it's not that hard to code:
On OS X, there's a command called open that does exactly what you want: "The open command opens a file (or a directory or URL), just as if you had double-clicked the file's icon." So, you can just popen open /Applications/MyGame.app.
On Windows, the equivalent command is start, but unfortunately, that's part of the cmd.exe shell rather than a standalone program. Fortunately, Python comes with a function os.startfile that does the same thing, so just os.startfile(r'C:\Program Files\MyGame\MyGame.exe').
On FreeDesktop-compatible *nix systems (which includes most modern linux distros, etc.), there's a very similar command called xdg-open: "xdg-open opens a file or URL in the user's preferred application." Again, just popen xdg-open /usr/local/bin/mygame.
If you expect to run on other platforms, you'll need to do a bit of research to find the best equivalent. Otherwise, for anything besides Mac and Windows, I'd just try to popen xdg-open, and throw an error if that fails.
See http://pastebin.com/XVp46f7X for an (untested) example.
Note that this will only work to run something that actually can be double-clicked to launch in Finder/Explorer/Nautilus/etc. For example, if you try to launch './script.py', depending on your settings, it may just fire up a text editor with your script in it.
Also, on OS X, you want to run the .app bundle, not the UNIX executable inside it. (In some cases, launching a UNIX executable—whether inside an .app bundle or standalone—may work, but don't count on it.)
Also, keep in mind that launching a program this way is not the same as running it from the command line—in particular, it will inherit its environment, current directory/drive, etc. from the Windows/Launch Services/GNOME/KDE/etc. session, not from your terminal session. If you need more control over the child process, you will need to look at the documentation for open, xdg-open, and os.startfile and/or come up with a different solution.
Finally, just because open/xdg-open/os.startfile succeeds doesn't actually mean that the game started up properly. For example, if it launches and then crashes before it can even create a window, it'll still look like success to you.
You may want to look around PyPI for libraries that do what you want. http://pypi.python.org/pypi/desktop looks like a possibility.
Or you could look through the patches in issue 3177, and pick the one you like best. As far as I know, they're all pure Python, and you can easily just drop the added function in your own module instead of in os or shutil.
As a quick hack, you may be able to (ab)use webbrowser.open. "Note that on some platforms, trying to open a filename using this function, may work and start the operating system’s associated program. However, this is neither supported nor portable." In particular, IIRC, it will not work on OS X 10.5+. However, I believe that making a file: URL out of the filename actually does work on OS X and Windows, and also works on linux for most, but not all, configurations. If so, it may be good enough for a quick&dirty script. Just keep in mind that it's not documented to work, it may break for some of your users, it may break in the future, and it's explicitly considered abuse by the Python developers, so I wouldn't count on it for anything more serious. And it will have the same problems launching 'script.py' or 'Foo.app/Contents/MacOS/foo', passing env variables, etc. as the more correct method above.
Almost everything else in your question is both irrelevant and wrong:
It wouldn't be a problem normally, but the program is a game, and has a Python interpreter built into it.
That doesn't matter. If the game were writing to stdout from C code, it would do the exact same thing.
When I use subprocess.Popen, it starts the separate program, but does so under the original program's Python instance
No it doesn't. It starts an entirely new process, whose embedded Python interpreter is an entirely new instance of Python. You can verify that by, e.g., running a different version of Python than the game embeds.
so that they share the first Python console.
No they don't. They may share the same tty/cmd window, but that's not the same thing.
I can end the first program fine, but I would rather have separate consoles (mainly because I have the console start off hidden, but it gets shown when I start the program from Python with subprocess.POpen).
You could always pipe the child's stdout and stderr to, e.g., a logfile, which you could then view separately from the parent process's output, if you wanted to. But I think this is going off on a tangent that has nothing to do with what you actually care about.
Also, os.system won't work because I'm aiming for cross-platform compatibility, and that's only available on Windows.
Wrong; os.system is available on "Unix, Windows"--which is probably everywhere you care about. However, it won't work because it runs the child program in a subshell of your script, using the same tty. (And it's got lots of other problems—e.g., blocking until the child finishes.)
When I use subprocess.Popen, it starts the separate program, but does so under the original program's Python instance...
Incorrect.
... so that they share the first Python console.
This is the crux of your problem. If you want it to run in another console then you must run another console and tell it to run your program instead.
... I'm aiming for cross-platform compatibility ...
Sorry, there's no cross-platform way to do it. You'll need to run the console/terminal appropriate for the platform.
In Windows, that is.
I think the answer to this question is that I need to create a Windows service. This seems ludicrously heavyweight for what I am trying to do.
I'm just trying to slap together a little prototype here for my manager, I'm not going to be responsible for productizing it... in fact, it may never even BE productized; it might just be something that a few researchers play around with.
I have a CGI script that receives a file for upload, stores it to a temporary location, then launches a background process to do some serious number-crunching on the file. Then some Javascript stuff sits around calling other CGI scripts to check on the status and update the page as needed.
All of this works, except the damn web server won't close the connection as long as the subrocess is running. I've done some searching, and it appears the answer on Unix is to make it a daemon, but I'm stuck on Windows right now and I guess the answer there is to make it a Windows service?!? This seems incredibly heavyweight to just, you know, launch a damn process and then close the server connection.
That's really the only way?
Edit: Okay, found a nifty little hack over here (the choice (3) that the guy gives):
How to completely background a process in Perl CGI under IIS
I was able to modify this to make it even simpler, and although this is a klugey solution, it is perfect for the quick-and-dirty little prototype I am trying to make.
So I initially had my main script doing this:
subprocess.Popen("python.exe","myscript.py","arg1","arg2")
Which doesn't work, as I've described. Instead, I now have my main script emit this little bit of Javascript which runs after the document is fully loaded:
$("#somecrap").load("launchBackgroundProcess.py", {arg1:"foo",arg2:"bar"});
And then launchBackgroundProcess.py does the subprocess.Popen.
This solution would never scale, since it still leaves the browser connection open during the entire time the background task is running. But since this little thinger I am whipping up might someday have two simultaneous users at most (even then I doubt it) resources are not a concern. This allows the user to see the main page and get the Javascript updates even though there is still an http connection hanging open for no good reason.
Thanks for the answers! If I'm ever asked to productize this, I'll take at the resources Profane recommends.
If you haven't much experience with windows programming and don't wish to peruse the MSDN docs-- I don't blame you-- you may want to try to pick up a copy of Mark Hammond's cannonical guide to all things python and windows. It somehow never goes out-of-date on many of these sorts of recurring questions. Instead of launching the process with the every-platform solution, you'd probably be better off using the win32process module. Chapter 17 of the Hammond book covers this extensively, but you could probably get all you need by downloading the pywin ide (I think it comes bundled in the windows extensions which you can download from pypi), and looking through the help docs it has on python's windows' api. Here's an example of using the api, from a project I was working on recently. It may in fact do some of what you want with a little adaptation. You'd probably want to focus on CreationFlags. In particular, win32process.DETACHED_PROCESS is "often used to execute console programs in the background." Many other flags are available and conveniently wrapped however.
if subprocess.mswindows:
su=subprocess.STARTUPINFO()
su.dwFlags |= subprocess._subprocess.STARTF_USESHOWWINDOW
process = subprocess.Popen(['program', 'flag', 'flag2'], bufsize=-1,
stdout=subprocess.PIPE, startupinfo=su)
Simplest, but not most efficient way would be to just run another python executable
from subprocess import Popen
Popen("python somescript.py")
You can just use a system call using the "start" windows command. This way your python script will not wait for the completion of the started program.
CGI scripts are run with standard output redirected, either directly to the TCP socket or to a pipe. Typically, the connection won't close until the handle, and all copies of it, are closed. By default, the subprocess will inherit a copy of the handle.
There are two ways to prevent the connection from waiting on the subprocess. One is to prevent the subprocess from inheriting the handle, the other is for the subprocess to close its copy of the handle when it starts.
If the subprocess is in Perl, I think you could close the handle very simply:
close(STDOUT);
If you want to prevent the subprocess from inheriting the handle, you could use the SetHandleInformation function (if you have access to the Win32 API) or set bInheritHandles to FALSE in the call to CreateProcess. Alternatively, close the handle before launching the subprocess.
Is there a way to send some parameter from autohotkey to python.
Using Autohot key I read some number from the notepad and store in a variable and now I want to send this number to the python code in order to do some calculations.
My Autohotkey code is:
controlGetText, telphoneNumber, Edit1, Untitled - Notepad
And I want to send this telphoneNumber to python file.
Is there a way I can do that?
Do I need to create an exe file of a python and then call from autohotkey?
For example:
RunWait, C:\Button\button.exe telphoneNumber
Or do I need to run command prompt commands from autohotkey to run python program? Something like:
Run Cmd Python C:\Button\button.py telphoneNumber
I do not know which is the best way as I am newbie in Autohotkey.
Any suggestion will be appreciated.
EDIT:
However I succeded in sending parameter by using run command from autohotkey, which will execute the python file from command prompt.
Run Cmd \k "Python C:\Button\button.py %telphoneNumber%"
But still want to know if this is the right solution, or if there are others?
Inter-process communication would be capable of sending the information while the Python script is already running.
Forum thread: http://www.autohotkey.com/forum/topic21699.html (there's a nice documentation link in that post)
You could also use TCP/IP Network communication (like in the post below), but that probably wouldn't be quite as slick as using IPC.
Forum thread: http://www.autohotkey.com/forum/topic13829.html
The way you got it working is the easiest, and probably best, method of accomplishing what you want.
Communication between applications can be done with more methods then you probably can imagine, but as long as it doesn't have to be realtime you can call your programs with arguments, as it is easy and reliable.
Python COM server allows directly calling Python functions(with args and return) using AHK.
you use it like this: MsgBox % pythonComServer.method(args)
You do not need to have a python script already running.
ComObjCreate() will instantiate an instance of python.
I don't know how the inter-process communication is done in the background by pywin32, but using it is simple.
2 examples here: Call python function with arguments and get returned value in autohotkey
Imagine a web application that allows a logged in user to run a shell command on the web server at the press of a button. This is relatively simple in most languages via some standard library os tools.
But if that command is long running you don't want your UI to hang. Again this is relatively easy to deal with using some sort of background process or putting the command to be executed onto a message queue (and maybe saving the output and status somewhere for later consumption). Just return quickly saving we'll run that and get back to you.
What I'd like to do is show the output of said web ui triggered shell command as it happens. So vertically scrolling text like when running in a terminal.
I have a vague idea of how I might approach this, streaming the output to a websocket perhaps and simply printing the output to screen.
What I'd like to ask is:
Are their any plugins, libraries or applications that already do this. Something I can either use or read the source of. Ideally an open source python/django or ruby/rails tool, but other stacks would be interesting too.
I'm not sure if it's what you want, but there are some web based ssh clients out there. If you care about security and really just want dynamic feedback, you could look into comet or just have a frame with its own http session that doesn't end until it's done printing.
web-based ssh client would work, into the host (there are java ssh clients out there).
Ruby has a web-based terminal:
http://tryruby.org (link to the source is at the bottom of the page).
You could also embed Ruby via jruby: http://tim.lossen.de/2007/03/jruby/applet.html
http://github.com/jruby/jruby/blob/master/samples/irb-applet.html
I haven't heard of any libraries that do this, but you'll need to setup the system command and call out to the system. You will then need to "pump" the sysout and syserr standard inputs and pipe that data back out to your web client.
As an example for this style of problem, look into code snippits of how people use ruby/python/etc to transcode a video, i.e. http://kpumuk.info/ruby-on-rails/encoding-media-files-in-ruby-using-ffmpeg-mencoder-with-progress-tracking/ - my example was taken from this blog post.
class MediaFormatException < StandardError
end
def execute_mencoder(command)
progress = nil
IO.popen(command) do |pipe|
pipe.each("r") do |line|
if line =~ /Pos:[^(]*(s*(d+)%)/
p = $1.to_i
p = 100 if p > 100
if progress != p
progress = p
print "PROGRESS: #{progress}n"
$defout.flush
end
end
end
end
raise MediaFormatException if $?.exitstatus != 0
end
I don't know if this example is pulling data from both sysout and syserr, but you will definitely need to be pulling data from both of those interfaces, typically if the buffer fills up, the executing command might hang or fail (I have experienced this with Python). This method will also look different if the only thing you do is return line to the web client - in a terminal, the progress indicator of ffmpeg/mencoder remains stationary on the bottom line, but this method will give you a long list of progress indicator updates. Pipe line out to your terminal and you'll see what I'm referring to.
So, I've tried to answer my own question with code as I couldn't find anything to quite fit the bill. Hopefully it's useful to anyone coming across the same problem.
Redbeard 0X0A pointed me in the general direction, I was able to get a stand along ruby script doing what I wanted using popen. Extending this to using EventMachine (as it provided a convenient way of writing a websocket server) and using it's inbuilt popen method solved my problem.
More details here http://morethanseven.net/2010/09/09/Script-running-web-interface-with-websockets.html and the code at http://github.com/garethr/bolt/
Certainly not the best way to run shell commands, but likely the easiest:
#!/bin/sh
echo Content-Type: text/plain
echo
/usr/bin/uptime
http://www.sente.cc/scripts/uptime.cgi
Take a look at Galaxy (online demo) or Yabi.
Except from the requirement to be able to show output during the job run, they are both excellent solutions to this! They are also both written i Python (and Yabi even on django).
They were both built with bioinformatics in mind, but really are both general job runner/workflow tools.
They will let you specify parameters in a web interface, see queued/running/finished jobs in a separate column, and after the jobs are finished, inspect details and results, or re-run the job, with possibly changed parameters.
Galaxy is the easier one to install. The Galaxy installation boils down to downloading and run "sh run.sh"), and adding your own tool boils down to creating an XML file in the line of:
<tool id="mytool" name="My Tool" version="1.0.0">
<description>Does this and that</description>
<command>somecommand --aparam $aparam</command>
<inputs>
<param name="aparam" type="text" label="A parameter"/>
</inputs>
<outputs>
<data name="outfile" format="tabular"/>
</outputs>
</tool>
... and place it in the /tools folder, and add a line in the tool_conf.xml to tell galaxy of your new tool (There you can also get rid of the bioinformatics-tools, so they don't mess up your tools menu).
Yabi is more complicated to install (see the readme file), but the process might be smooth if you are on the right kind of system. On the other hand, it allows you even do the tool configuration in the web interface, rather than as an XML file like in Galaxy.
Galaxy still is the one with the biggest community though, which is reflected in the number of features/already integrated tools (See the toolshed for shared tools/wrapper).
websocketd looks like the perfect tool for that.