How can I remove the output buffering from Sublime Text 3 when I build a Python 3 script? I would like real-time output.
I am using Sublime Text 3 with the Anaconda plugin, Python 3.6 and Linux Mint 18. When I run a simple script using control-b:
print('hello')
I get an instant output in a separate window called 'Build output'. When I use a script with a repeated output, such as:
from time import sleep
count = 0
print('starting')
while True:
print('{} hello'.format(count))
count += 1
sleep(0.5)
Initially I get a blank screen in 'Build output'. Some time later it populates with several hundred lines of output. It looks like the output is being buffered. When the buffer is full, it outputs all at once to the 'Build output' screen.
Edit
Sublime Text allows custom build configurations. The default Python build is for python 2. I entered a build configuration for Python 3 and missed the -u flag. The fix is to put the -u flag in the Python 3 build.
File: Python3.sublime-build
{
"shell_cmd": "/usr/bin/env python3 -u ${file}",
"selector": "source.python",
"file_regex": "^(...*?):([0-9]*):?([0-9]*)",
"working_dir": "${file_path}",
}
Save in sublime_install/Data/Packages/User/Python3.sublime-build
By default the exec command is used to execute the commands in build systems, and the exec command doesn't buffer output at all. There is more information in this answer (which also provides a version of exec that does line buffering) but in short exec launches one thread to handle stdout and one to handle stderr, and both forward whatever data they get to the panel as soon as they get it.
As such, a problem like the one you're describing here is generally caused by the program doing it's own buffering. Depending on the language and platform that you're using, buffering may change from what you expect in unexpected ways:
For example, see this text in the man page for stdout under Linux:
The stream stderr is unbuffered. The stream stdout is line-buffered when it points to a terminal. Partial lines will not appear until fflush(3) or exit(3) is called, or a newline is printed. This can produce unexpected results, especially with debugging output.
In the general case, the solution to this problem would be to modify the program itself to ensure that it's not buffering, and how you would do that depends on the language you're using and the platform that you're on. It could be something as simple as setting an environment variable or as complex as startup code that ensures that regardless of circumstance buffering is set as you expect it to be.
In the specific case of Python, the -u command line argument to the interpreter tells Python to keep things unbuffered:
-u : unbuffered binary stdout and stderr; also PYTHONUNBUFFERED=x
see man page for details on internal buffering relating to '-u'
The Python.sublime-build that ships with Sublime uses this argument to the python command to ensure that the output is unbuffered, and using that build system works as expected for your sample program.
I don't use the Anaconda package so I'm not sure if it provides it's own build systems or not, but you may want to check the build command that you're using to ensure that it uses -u.
Related
Running a long and time consuming number crunching process in the shell with a Python script. In the script, to indicate progress, I have inserted occassional print commands like
#!/usr/bin/env python3
#encoding:utf-8
print('Stage 1 completed')
Triggering the script in the shell by
user#hostname:~/WorkingDirectory$chmod 744 myscript.py && nohup ./myscript.py&
It redirects the output to nohup.out, but I cannot see the output until the entire script is done, probably because of stdout buffering. So in this scenario, how do I somehow adjust the buffering parameters to check the progress periodically? Basically, I want zero buffering, so that as soon a print command is issued in the python script, it will appear on nohup.out. Is that possible?
I know it is a rookie question and in addition to the exact solution, any easy to follow reference to the relevant material (which will help me master the buffering aspects of shell without getting into deeper Kernel or hardware level) will be greatly appreciated too.
If it is important, I am using #54~16.04.1-Ubuntu on x86_64
Python is optimised for reading in and printing out lots of data.
So standard input and output of the Python interpreter are buffered by default.
We can override this behavior some ways:
use interpretator python with option -u.
From man python:
-u Force stdin, stdout and stderr to be totally unbuffered. On systems where it matters, also put stdin, stdout and stderr in
binary mode. Note that there is internal buffering in xreadlines(), readlines() and file-object iterators ("for line in
sys.stdin") which is not influenced by this option. To work around this, you will want to use "sys.stdin.readline()" inside a
"while 1:" loop.
Run script in shell:
nohup python -u ./myscript.py&
Or modify shebang line of script to #!/usr/bin/python -u and then run:
nohup ./myscript.py&
use shell command stdbuf for turn off buffering stream
See man stdbuf.
Set unbuffered stream for output:
stdbuf --output=0 nohup ./myscript.py&
Set unbuffered stream for output and errors:
stdbuf -o0 -e0 nohup ./myscript.py&
I try to run this script:
file = open("console-output.txt", "w")
task = subprocess.Popen(sys.executable + " \"main.py\"", shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in iter(task.stdout.readline, ''):
print("Got data")
file.write(line)
file.flush()
file.close()
It works fine and prints the program output to the console-output.txt file. However it outputs every text at once at the end of the program. I would like to have a live output to my file so that I can see the output of long-running programs. Am I doing anything wrong or is this a bug? I am on Ubuntu 17.10 with Python 3.6.3 64Bit.
It seems to me like task.stdout.readline is blocking till the program is completely finished.
After a lot more research using different search terms, I found out that C (and therefore many interpreters and programs) detects whether the program output is a console or a pipe. It buffers every output, as soon as the buffer has enough empty space or is not flushed, if in pipe mode. To force an unbuffered behaviour, you just need to pass -u to the target python interpreter. If not using Python, you may want to try the stdbuf command pre-installed on almost all common linux platforms, available via the coreutils package in Mac OS X (you have to call gstdbuf instead of stdbuf ). After a lot of research, I found out that the only equivalent for stdbuf on linux could be the stdbuf.exe found in the git-scm for windows. However I did not test it yet.
I am running a C++ binary which runs a python script. I do this in the following way
stream = popen(cmd.c_str(), "r");
if (stream) {
while (!feof(stream))
if (fgets(buffer, max_buffer, stream) != NULL) data.append(buffer);
status = pclose(stream);
}
This python script also runs some shell commands using subprocess
subprocess.check_call(arg_list)
Suddenly I have started getting errors like "Vim: Warning: Output not to a terminal" and the C++ binary hangs.
What is the easiest way to fix it?
The popen call is going to read whatever the subprocess writes to its standard output. Some of that is likely useful (such as status messages). You would not get useful results from vim's standard output, because it would be partly escape sequences (for moving the cursor, changing the colors).
If you can modify the script so that it handles vim specially, there is a fix: modify the commands which run vim so that they redirect to /dev/tty. That would change a command-string that looks like this
vim filename
to
vim filename >/dev/tty
You can try using
export TERM=vt100
before you call Perl or as the first command in each Unix subprocess.
I would like to create a simple Python program that will concurrently execute 2 independent scripts. For now, the two scripts just print a sequence of numbers but my intention is to use this program to concurrently run a few Twitter streaming programs in the future.
I suspect I need to use subprocess.Popen but I cannot quite get my head around what arguments I should put in there. There was a similar question on StackOverflow but the code provided there (pasted below) doesn't print anything. I will appreciate your help.
My files are:
thread1.py
thread2.py
import subprocess
subprocess.Popen(['screen', './thread1.py']))
subprocess.Popen(['screen', './thread2.py'])
Use supervisord
supervisord is process control system just for the purpose of running multiple command line scripts.
It features:
multiple controlled processes
autorestarting failed runs
log stdout and stderr output
starting scripts in order (using priority)
command line utility to view latest log output, stop, start, restart the processes
This solution works only on *nix based systems, it is not available on Windows.
As wanderlust mentioned, why do you want to do it this way and not via linux command line?
Otherwise, the solution you post is doing what it is meant to, i.e, you are doing this at the command line:
screen ./thread1.py
screen ./thread2.py
This will open a screen session and run the program and output within this screen session, such that you will not see the output on your terminal directly. To trouble shoot your output, just execute the scripts without the screen call:
import subprocess
subprocess.Popen(['./thread1.py'])
subprocess.Popen(['./thread2.py'])
Content of thread1.py:
#!/usr/bin/env python
def countToTen():
for i in range(10):
print i
countToTen()
Content of thread2.py:
#!/usr/bin/env python
def countToHundreds():
for i in range(10):
print i*100
countToHundreds()
Then don't forget to do this on the command line:
chmod u+x thread*.py
You can also just open several Command Prompt windows to run several Python programs at once - just run one in each of them:
In each Command Prompt window, go to the correct directory (such as C:/Python27) and then type 'python YourCodeNo1.py' in one Command Prompt window, 'python YourCodeNo2.py' in the next one ect. .
I'm currently running 3 codes at one time in this way, without slowing any of them down.
I'm wanting to open a terminal from a Python script (not one marked as executable, but actually doing python3 myscript.py to run it), have the terminal run commands, and then keep the terminal open and let the user type commands into it.
EDIT (as suggested): I am primarily needing this for Linux (I'm using Xubuntu, Ubuntu and stuff like that). It would be really nice to know Windows 7/8 and Mac methods, too, since I'd like a cross-platform solution in the long-run. Input for any system would be appreciated, however.
Just so people know some useful stuff pertaining to this, here's some code that may be difficult to come up with without some research. This doesn't allow user-input, but it does keep the window open. The code is specifically for Linux:
import subprocess, shlex;
myFilePathString="/home/asdf asdf/file.py";
params=shlex.split('x-terminal-emulator -e bash -c "python3 \''+myFilePathString+'\'; echo \'(Press any key to exit the terminal emulator.)\'; read -n 1 -s"');
subprocess.call(params);
To open it with the Python interpreter running afterward, which is about as good, if not better than what I'm looking for, try this:
import subprocess, shlex;
myFilePathString="/home/asdf asdf/file.py";
params=shlex.split('x-terminal-emulator -e bash -c "python3 -i \''+myFilePathString+'\'"');
subprocess.call(params);
I say these examples may take some time to come up with because passing parameters to bash, which is being opened within another command can be problematic without taking a few steps. Plus, you need to know to use to quotes in the right places, or else, for example, if there's a space in your file path, then you'll have problems and might not know why.
EDIT: For clarity (and part of the answer), I found out that there's a standard way to do this in Windows:
cmd /K [whatever your commands are]
So, if you don't know what I mean try that and see what happens. Here's the URL where I found the information: http://ss64.com/nt/cmd.html