Develop Reference

Run python function from command line or subprocess popen

I wanted to run a test function called test.counttest() that counts up to 10.
def counttest():
x = 0
for x in range(0,3):
x = x+1
print("Number: "+ str(x))
time.sleep(1)
I want to call just the function from the command line OR from subprocess popen. Not write the function, just call it. Everything I have google keeps bringing me back to how I can write a function from the command line which is NOT what I need.
I need to specifically run a function from subprocess popen so I can get the stdout in a forloop that can then be sent to a flask socket. (This is required)
Main point - How can Call (not write) a function from the command line or from subprocess?
Not this:
python -c 'import whatever then add code'
But something like this:
python "test.counttest()"
or like this:
subprocess.Popen(['python', ".\test.counttest()"],stdout=subprocess.PIPE, bufsize=1,universal_newlines=True)
EDIT:
This is for #Andrew Holmgren. Consider the following script:
def echo(ws):
data = ws.receive()
with subprocess.Popen(['powershell', ".\pingtest.ps1"],stdout=subprocess.PIPE, bufsize=1,universal_newlines=True) as process:
for line in process.stdout:
line = line.rstrip()
print(line)
try:
ws.send(line+ "\n")
except:
pass
this works perfectly for what I need as it:
takes the script's stdout and send's it to the ws.send() function which is a websocket.
However I need this same concept for a function instead. The only way I know how to get the stdout easily is from using subprocess.popen but if there is another way let me know. This is why I am trying to make a hackjob way of running a function through the subprocess module.
The question of Run python function from command line or subprocess popen relates in the fact that if I can get a function to run from subprocess, then I know how to get the stdout for a websocket.

Actually you have really a lot of questions inside this one.
How can I send output of function line-by-line to another one (and/or websocket)? Just avoid writing to stdout and communicate directly. yield (or other generator creation methods) are intended exatly for that.
import time
def counttest():
for i in range(10):
yield f'Item {i}'
time.sleep(1)
def echo(ws):
# data = ws.receive()
for row in counttest():
ws.send(row)
How to call a function func_name defined in file (suppose it's test.py) from command line? Being in directory with test.py, do
$ python -c 'from test import func_name; func_name()'
How to read from sys.stdout? The easiest will be to replace it with io.StringIO and restore thing back later.
from contextlib import redirect_stdout
import io
def echo(ws):
f = io.StringIO()
with redirect_stdout(f):
counttest()
output = f.getvalue()
ws.send(output)
It will return only after call_function(), so you cannot monitor real-time printed items.
Regarding
I need the stdout
I can say, that I'm sure your question is X-Y problem, thus I try to suggest alternatives. Solution you want will also work, but it's awkward. This will exactly run function counttest defined in test.py, capture its output and send it line-by-line to websocket. It will process output immediately when new line arrives. Note -u flag on python call (unbuffered), it's important.
import subprocess
import shlex
def echo(ws):
data = ws.receive()
with subprocess.Popen(shlex.split("python -u -c 'from test import counttest; counttest()'"),
stdout=subprocess.PIPE,
bufsize=1,
universal_newlines=True) as process:
for line in iter(process.stdout.readline, ''):
line = line.rstrip()
if not line:
break
print(line)
try:
ws.send(line + "\n")
except:
pass

Python execution order using subprocess.call

I have the following code in file abc.py:
import subprocess
def evaluate():
for i in range(5):
print("Printing", i)
subprocess.call(['ls', '-lrt'])
print("~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~")
evaluate()
Now when I call using python abc.py > out.dat, the output file contains the result of 'ls -lrt' five times which is followed by printing statements in the code.
Why is it happening so and what should I do if I need to get output as:
printing 0
(results of ls -lrt here)
~~~~~~~~~~~~~~~~~~~~~~~
printing 1
.
.
.
and so on..?
Thank you..

You need to flush your stream before you call a subprocess:
import subprocess, sys
def evaluate():
for i in range(5):
print("Printing", i)
sys.stdout.flush()
subprocess.call(['ls', '-lrt'])
print("~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~")
evaluate()
Flushing takes automatically place line-by-line as long as you write to a terminal (without redirection). Then you do not notice the problem because the print() flushes at the newline.
As soon as you redirect your output to a file the print() statement notices this and flushes automatically only when a certain buffer is full (and at termination time).
The subprocess (ls) does the same but uses its own buffer. And it terminates first, so its output is in the file first.

Read from stdin AND forward it to a subprocess in Python

I'm writing a wrapper script for a program that optionally accepts input from STDIN. My script needs to process each line of the file, but it also needs to forward STDIN to the program it is wrapping. In minimalist form, this looks something like this:
import subprocess
import sys
for line in sys.stdin:
# Do something with each line
pass
subprocess.call(['cat'])
Note that I'm not actually trying to wrap cat, it just serves as an example to demonstrate whether or not STDIN is being forwarded properly.
With the example above, if I comment out the for-loop, it works properly. But if I run it with the for-loop, nothing gets forwarded because I've already read to the end of STDIN. I can't seek(0) to the start of the file because you can't seek on streams.
One possible solution is to read the entire file into memory:
import subprocess
import sys
lines = sys.stdin.readlines()
for line in lines:
# Do something with each line
pass
p = subprocess.Popen(['cat'], stdin=subprocess.PIPE)
p.communicate(''.join(lines))
which works, but isn't very memory efficient. Can anyone think of a better solution? Perhaps a way to split or copy the stream?
Additional Constraints:
The subprocess can only be called once. So I can't read a line at a time, process it, and forward it to the subprocess.
The solution must work in Python 2.6

Does this work for you?
#!/usr/bin/env python2
import subprocess
import sys
p = subprocess.Popen(['cat'], stdin = subprocess.PIPE)
line = sys.stdin.readline()
####################
# Insert work here #
####################
line = line.upper()
####################
p.communicate(line)
Example:
$ echo "hello world" | ./wrapper.py
HELLO WORLD

stdin should not wait for "CTRL+D"

I got a simple python script which should read from stdin.
So if I redirect a stdout of a program to the stdin to my python script.
But the stuff that's logged by my program to the python script will only "reach" the python script when the program which is logging the stuff gets killed.
But actually I want to handle each line which is logged by my program as soon as it is available and not when my program which should actually run 24/7 quits.
So how can I make this happen? How can I make the stdin not wait for CTRL+D or EOF until they handle data?
Example
# accept_stdin.py
import sys
import datetime
for line in sys.stdin:
print datetime.datetime.now().second, line
# print_data.py
import time
print "1 foo"
time.sleep(3)
print "2 bar"
# bash
python print_data.py | python accept_stdin.py

Like all file objects, the sys.stdin iterator reads input in chunks; even if a line of input is ready, the iterator will try to read up to the chunk size or EOF before outputting anything. You can work around this by using the readline method, which doesn't have this behavior:
while True:
line = sys.stdin.readline()
if not line:
# End of input
break
do_whatever_with(line)
You can combine this with the 2-argument form of iter to use a for loop:
for line in iter(sys.stdin.readline, ''):
do_whatever_with(line)
I recommend leaving a comment in your code explaining why you're not using the regular iterator.

It is also an issue with your producer program, i.e. the one you pipe stdout to your python script.
Indeed, as this program only prints and never flushes, the data it prints is kept in the internal program buffers for stdout and not flushed to the system.
Add sys.stdout.flush() call right after you print statement in print_data.py.
You see the data when you quit the program as it automatically flushes on exit.
See this question for explanation,

As said by #user2357112 you need to use:
for line in iter(sys.stdin.readline, ''):
After that you need to start python with the -u flag to flush stdin and stdout immediately.
python -u print_data.py | python -u accept_stdin.py
You can also specify the flag in the shebang.

Popen does not give output immediately when available

I am trying to read from both stdout and stderr from a Popen and print them out. The command I am running with Popen is the following
#!/bin/bash
i=10
while (( i > 0 )); do
sleep 1s
echo heyo-$i
i="$((i-1))"
done
echo 'to error' >&2
When I run this in the shell, I get one line of output and then a second break and then one line again, etc. However, I am unable to recreate this using python. I am starting two threads, one each to read from stdout and stderr, put the lines read into a Queue and another thread that takes items from this queue and prints them out. But with this, I see that all the output gets printed out at once, after the subprocess ends. I want the lines to be printed as and when they are echo'ed.
Here's my python code:
# The `randoms` is in the $PATH
proc = sp.Popen(['randoms'], stdout=sp.PIPE, stderr=sp.PIPE, bufsize=0)
q = Queue()
def stream_watcher(stream, name=None):
"""Take lines from the stream and put them in the q"""
for line in stream:
q.put((name, line))
if not stream.closed:
stream.close()
Thread(target=stream_watcher, args=(proc.stdout, 'out')).start()
Thread(target=stream_watcher, args=(proc.stderr, 'err')).start()
def displayer():
"""Take lines from the q and add them to the display"""
while True:
try:
name, line = q.get(True, 1)
except Empty:
if proc.poll() is not None:
break
else:
# Print line with the trailing newline character
print(name.upper(), '->', line[:-1])
q.task_done()
print('-*- FINISHED -*-')
Thread(target=displayer).start()
Any ideas? What am I missing here?

Only stderr is unbuffered, not stdout. What you want cannot be done using the shell built-ins alone. The buffering behavior is defined in the stdio(3) C library, which applies line buffering only when the output is to a terminal. When the output is to a pipe, it is pipe-buffered, not line-buffered, and so the data is not transferred to the kernel and thence to the other end of the pipe until the pipe buffer fills.
Moreover, the shell has no access to libc’s buffer-controlling functions, such as setbuf(3) and friends. The only possible solution within the shell is to launch your co-process on a pseudo-tty, and pty management is a complex topic. It is much easier to rewrite the equivalent shell script in a language that does grant access to low-level buffering features for output streams than to arrange to run something over a pty.
However, if you call /bin/echo instead of the shell built-in echo, you may find it more to your liking. This works because now the whole line is flushed when the newly launched /bin/echo process terminates each time. This is hardly an efficient use of system resources, but may be an efficient use of your own.

IIRC, setting shell=True on Popen should do it.