Develop Reference

Is there a way to continuously collect output from a Python script I'm running into a c++ program?

So, I'm currently trying to construct a c++ aplication that calls for a python script. The main idea is that the python script runs a loop and prints decisions based on user input. I want the cpp program to be able to wait and read(if there s an output from python). I tried to make them "talk" via a file but it turned out bad. Any ideas?
PS: im calling the script using system("start powershell.exe C:\\python.exe C:\\help.py");
If there is any better way please let me know! Thanks

You could write to a file from python and check every certain amount of time in the c++ program to check for any changes in the file.

No, there is no standard way to capture the output if you start the program using std::system.
The operating system facility that you're looking for is "stream". The standard C++ provides access only to the standard streams, which you can use if you redirect the output of the python program when you start the C++ program. Example shell command:
python help.py > cpp_program
If you do this, then you can continuously read the output from the standard input stream in the C++ program. There is no standard way to create extra streams in C++, although that possibility is typically provided by the operating system.

Displaying reports for files using Python

Question about general possibilities using Python here, I don't really know enough about programming to know whether it's something that's doable, and if so, how do I go about it.
I have a program which is a simple desktop program, which you load files into. The program can then output various properties of the thing that's in the file, and depending on what you ask it to do will output a report. It outputs the report in text format, but not as a file, and instead actually, just in the program itself displays the report. Like this:
My question is that if I want to get this text output for a large number of files, I'm currently manually loading the files individually into the program making the report, copying this to a text file, and saving the text file.
Basically I want to know whether it's extremely difficult to get Python to do this for me, or not. If it is doable, are the resources available for me to read about how it might be done? Are there conditions about being able to run my program and various commands from the Python command box?
Hope my question's clear enough. Sorry if it's a bit garbled.

The tricky part here is
The program can then output various properties of the thing that's in the file, and depending on what you ask it to do will output a report.
Basically, if the desktop application you use has a command line interface, it is possible and relatively easy.
If this program has command line option to open a document and output a report in any format (print the report on the standard output, write it into a file on the disk, etc.), you can call that commands from a script python for each files you set in a list.
If your software doesn't have a CLI (Command Line Interface), it might be possible but more diffficult. In that case, you have to automate actions by using a library that will emulate clicks on the Window of you software (1. Click on Open 2. Click, click, click to select the file to load 3. Click on the button to generate a report etc.) It's a pain, but it can be considered.
You will find plenty of resources to learn by yourself how to code a python script. You will probably need to learn about lists, loops, files manipulations and maybe the subprocess library which will let you call any command from your python script.
I suggest you to start with Python3 instead of Python2 because it has a better support for unicode that could quickly become an issue if you have non ascii characters in your input files or in reports from your software.
Good luck ;)

If the only way you can get report is selecting and copy/pasting it from program GUI, the situation just begs for AutoIt instead of Python.
With Python it would be much more difficult. Unless you want to improve your python knowledge or course...
Simulating keypresses, you can open specific file in program (through sending ctrl+o or alt and navigating file menu). Simulating mouse or keypress - start report generation. Then simulate a mouse click in text area, and perform something like:
(just a skeleton of script, probably need to be modified to suit your situation and needs)
send("^{a}^{c}") ; to select all and copy (if these keys are supported in this program
$text = ClipGet() ; get contents of clipboard
$fout = FileOpen("somefile.txt",2)
FileWrite($fout,$text)
FileClose($fout)
To fully automate the task, in script you can get a list of source files in specific folder, and run this macro for each of them, automatically naming resulting txt files.

Trying to embed a native WINDOWS command line in my Tkinter/Python app

I've done a bit of research and I've played with different graphical kits and was wondering if there was a way to display a native windows command line in my application. So, it'd be something like the command line in Jetbrains products (I think eclipse has one too), but the goal is to have 4 of them.
I've found a piece of linux software demonstrating the core concept of what I want to do and I've attached it at the bottom. Basically, I just want four terminals, and I'm going to have the app handle specific key bindings for "ease of access" tasks. I've used the linked software before and like it, but I thought as I'm exploring python it'd be a good learning experience to write something like this for windows as well!
Thanks in advance!
Edit - Google searching "Windows Tkinter Command Line" and the likes haven't been fruitful :) I promise I googled a LOT before posting here.

I dont know if there is anything readymade for this. Basically, you will need to have a text widget and simulate the output with the subprocess module to directly pass user input to run system commands and append results to the same text widget. You could also consider two windows - one for text input and one for result output.
Here is a link: Calling an external command in Python

subprocess.call does not wait for the process to complete

Per Python documentation, subprocess.call should be blocking and wait for the subprocess to complete. In this code I am trying to convert few xls files to a new format by calling Libreoffice on command line. I assumed that the call to subprocess call is blocking but seems like I need to add an artificial delay after each call otherwise I miss few files in the out directory.
what am I doing wrong? and why do I need the delay?
from subprocess import call
for i in range(0,len(sorted_files)):
args = ['libreoffice', '-headless', '-convert-to',
'xls', "%s/%s.xls" %(sorted_files[i]['filename'],sorted_files[i]['filename']), '-outdir', 'out']
call(args)
var = raw_input("Enter something: ") # if comment this line I dont get all the files in out directory
EDIT It might be hard to find the answer through the comments below. I used unoconv for document conversion which is blocking and easy to work with from an script.

It's possible likely that libreoffice is implemented as some sort of daemon/intermediary process. The "daemon" will (effectively1) parse the commandline and then farm the work off to some other process, possibly detaching them so that it can exit immediately. (based on the -invisible option in the documentation I suspect strongly that this is indeed the case you have).
If this is the case, then your subprocess.call does do what it is advertised to do -- It waits for the daemon to complete before moving on. However, it doesn't do what you want which is to wait for all of the work to be completed. The only option you have in that scenario is to look to see if the daemon has a -wait option or similar.
1It is likely that we don't have an actual daemon here, only something which behaves similarly. See comments by abernert

The problem is that the soffice command-line tool (which libreoffice is either just a link to, or a further wrapper around) is just a "controller" for the real program soffice.bin. It finds a running copy of soffice.bin and/or creates on, tells it to do some work, and then quits.
So, call is doing exactly the right thing: it waits for libreoffice to quit.
But you don't want to wait for libreoffice to quit, you want to wait for soffice.bin to finish doing the work that libreoffice asked it to do.
It looks like what you're trying to do isn't possible to do directly. But it's possible to do indirectly.
The docs say that headless mode:
… allows using the application without user interface.
This special mode can be used when the application is controlled by external clients via the API.
In other words, the app doesn't quit after running some UNO strings/doing some conversions/whatever else you specify on the command line, it sits around waiting for more UNO commands from outside, while the launcher just runs as soon as it sends the appropriate commands to the app.
You probably have to use that above-mentioned external control API (UNO) directly.
See Scripting LibreOffice for the basics (although there's more info there about internal scripting than external), and the API documentation for details and examples.
But there may be an even simpler answer: unoconv is a simple command-line tool written using the UNO API that does exactly what you want. It starts up LibreOffice if necessary, sends it some commands, waits for the results, and then quits. So if you just use unoconv instead of libreoffice, call is all you need.
Also notice that unoconv is written in Python, and is designed to be used as a module. If you just import it, you can write your own (simpler, and use-case-specific) code to replace the "Main entrance" code, and not use subprocess at all. (Or, of course, you can tear apart the module and use the relevant code yourself, or just use it as a very nice piece of sample code for using UNO from Python.)
Also, the unoconv page linked above lists a variety of other similar tools, some that work via UNO and some that don't, so if it doesn't work for you, try the others.
If nothing else works, you could consider, e.g., creating a sentinel file and using a filesystem watch, so at least you'll be able to detect exactly when it's finished its work, instead of having to guess at a timeout. But that's a real last-ditch workaround that you shouldn't even consider until eliminating all of the other options.

If libreoffice is being using an intermediary (daemon) as mentioned by #mgilson, then one solution is to find out what program it's invoking, and then directly invoke it yourself.

What is the best way of running shell commands from a web based interface?

Imagine a web application that allows a logged in user to run a shell command on the web server at the press of a button. This is relatively simple in most languages via some standard library os tools.
But if that command is long running you don't want your UI to hang. Again this is relatively easy to deal with using some sort of background process or putting the command to be executed onto a message queue (and maybe saving the output and status somewhere for later consumption). Just return quickly saving we'll run that and get back to you.
What I'd like to do is show the output of said web ui triggered shell command as it happens. So vertically scrolling text like when running in a terminal.
I have a vague idea of how I might approach this, streaming the output to a websocket perhaps and simply printing the output to screen.
What I'd like to ask is:
Are their any plugins, libraries or applications that already do this. Something I can either use or read the source of. Ideally an open source python/django or ruby/rails tool, but other stacks would be interesting too.

I'm not sure if it's what you want, but there are some web based ssh clients out there. If you care about security and really just want dynamic feedback, you could look into comet or just have a frame with its own http session that doesn't end until it's done printing.

web-based ssh client would work, into the host (there are java ssh clients out there).
Ruby has a web-based terminal:
http://tryruby.org (link to the source is at the bottom of the page).
You could also embed Ruby via jruby: http://tim.lossen.de/2007/03/jruby/applet.html
http://github.com/jruby/jruby/blob/master/samples/irb-applet.html

I haven't heard of any libraries that do this, but you'll need to setup the system command and call out to the system. You will then need to "pump" the sysout and syserr standard inputs and pipe that data back out to your web client.
As an example for this style of problem, look into code snippits of how people use ruby/python/etc to transcode a video, i.e. http://kpumuk.info/ruby-on-rails/encoding-media-files-in-ruby-using-ffmpeg-mencoder-with-progress-tracking/ - my example was taken from this blog post.
class MediaFormatException < StandardError
end
def execute_mencoder(command)
progress = nil
IO.popen(command) do |pipe|
pipe.each("r") do |line|
if line =~ /Pos:[^(]*(s*(d+)%)/
p = $1.to_i
p = 100 if p > 100
if progress != p
progress = p
print "PROGRESS: #{progress}n"
$defout.flush
end
end
end
end
raise MediaFormatException if $?.exitstatus != 0
end
I don't know if this example is pulling data from both sysout and syserr, but you will definitely need to be pulling data from both of those interfaces, typically if the buffer fills up, the executing command might hang or fail (I have experienced this with Python). This method will also look different if the only thing you do is return line to the web client - in a terminal, the progress indicator of ffmpeg/mencoder remains stationary on the bottom line, but this method will give you a long list of progress indicator updates. Pipe line out to your terminal and you'll see what I'm referring to.

So, I've tried to answer my own question with code as I couldn't find anything to quite fit the bill. Hopefully it's useful to anyone coming across the same problem.
Redbeard 0X0A pointed me in the general direction, I was able to get a stand along ruby script doing what I wanted using popen. Extending this to using EventMachine (as it provided a convenient way of writing a websocket server) and using it's inbuilt popen method solved my problem.
More details here http://morethanseven.net/2010/09/09/Script-running-web-interface-with-websockets.html and the code at http://github.com/garethr/bolt/

Certainly not the best way to run shell commands, but likely the easiest:
#!/bin/sh
echo Content-Type: text/plain
echo
/usr/bin/uptime
http://www.sente.cc/scripts/uptime.cgi

Take a look at Galaxy (online demo) or Yabi.
Except from the requirement to be able to show output during the job run, they are both excellent solutions to this! They are also both written i Python (and Yabi even on django).
They were both built with bioinformatics in mind, but really are both general job runner/workflow tools.
They will let you specify parameters in a web interface, see queued/running/finished jobs in a separate column, and after the jobs are finished, inspect details and results, or re-run the job, with possibly changed parameters.
Galaxy is the easier one to install. The Galaxy installation boils down to downloading and run "sh run.sh"), and adding your own tool boils down to creating an XML file in the line of:
<tool id="mytool" name="My Tool" version="1.0.0">
<description>Does this and that</description>
<command>somecommand --aparam $aparam</command>
<inputs>
<param name="aparam" type="text" label="A parameter"/>
</inputs>
<outputs>
<data name="outfile" format="tabular"/>
</outputs>
</tool>
... and place it in the /tools folder, and add a line in the tool_conf.xml to tell galaxy of your new tool (There you can also get rid of the bioinformatics-tools, so they don't mess up your tools menu).
Yabi is more complicated to install (see the readme file), but the process might be smooth if you are on the right kind of system. On the other hand, it allows you even do the tool configuration in the web interface, rather than as an XML file like in Galaxy.
Galaxy still is the one with the biggest community though, which is reflected in the number of features/already integrated tools (See the toolshed for shared tools/wrapper).

websocketd looks like the perfect tool for that.