I am running a C++ binary which runs a python script. I do this in the following way
stream = popen(cmd.c_str(), "r");
if (stream) {
while (!feof(stream))
if (fgets(buffer, max_buffer, stream) != NULL) data.append(buffer);
status = pclose(stream);
}
This python script also runs some shell commands using subprocess
subprocess.check_call(arg_list)
Suddenly I have started getting errors like "Vim: Warning: Output not to a terminal" and the C++ binary hangs.
What is the easiest way to fix it?
The popen call is going to read whatever the subprocess writes to its standard output. Some of that is likely useful (such as status messages). You would not get useful results from vim's standard output, because it would be partly escape sequences (for moving the cursor, changing the colors).
If you can modify the script so that it handles vim specially, there is a fix: modify the commands which run vim so that they redirect to /dev/tty. That would change a command-string that looks like this
vim filename
to
vim filename >/dev/tty
You can try using
export TERM=vt100
before you call Perl or as the first command in each Unix subprocess.
Related
I try to get network adapter name (e.g "Ethernet") in python.
The only way I found and works is by entering
for /F "skip=3 tokens=3*" %G in ('netsh interface show interface') do echo %H
in the windows CMD.So I tried following:
os.system("for /F \"skip=3 tokens=3*\" %G in (\"netsh interface show interface\")")
But it gives Syntaxerror even tho I fixed the string quote stuff...
Subprocess module not working either.
Anybody has an idea?
You're having trouble with the , shell=True subprocess argument.
You don't want to see bash trying to
interpret that command.
Put the long command into ether.bat so you won't be
distracted by quoting issues, and can focus on
whether bash or cmd.exe are running your batch file.
This should suffice:
os.system('cmd ether.bat')
though it's better to use this interface:
interface = subprocess.check_output(['cmd', 'ether.bat'])
Python offers perfectly good text parsing facilities.
Consider dispensing with cmd.exe
and just run netsh interface show interface,
with a python loop that makes sense of the
returned text.
I'm attempting to run a Linux script through Python's subprocess module. Below is the subprocess command:
result = subprocess.run(['/dir/scripts/0_texts.sh'], shell=True)
print(result)
Here is the 0_texts.sh script file:
cd /dir/TXTs
pylanguagetool text_0.txt > comments_0.txt
The subprocess command executes the script file, writing a new comments_0.txt in the correct directory. However, there's an error in the execution. The comments_0.txt contains an error of "input file is required", and the subprocess result returns returncode=2. When I run the pylanguagetool text_0.txt > comments_0.txt directly in the terminal the command executes properly, with the comments_0.txt written with the proper input file of text_0.txt.
Any suggestions on what I'm missing?
There is some ambiguity here in that it's not obvious which shell is run each time 0_texts.sh is invoked, and whether it has the values you expect of environment variables like PATH, which could result in a different copy of pylanguagetool running from when you call it at the command line.
First I'd suggest removing the shell=True option in subprocess.run, which is only involving another, potentially different shell here. Next I would change subprocess.run(['/dir/scripts/0_texts.sh']) to subprocess.run(['bash', '/dir/scripts/0_texts.sh']) (or whichever shell you wanted to run, probably bash or dash) to remove that source of ambiguity. Finally, you can try using type pylanguagetool in the script, invoking pylanguagetool with its full path, or calling bash /dir/scripts/0_texts.sh from your terminal to debug the situation further.
A bigger-picture issue is, pyLanguageTool is a Python library, so you're almost certainly going to be better off calling its functions from your original Python script directly instead of using a shell script as an intermediary.
I have a MATLAB .m file and want a Python.py file to be able to run/call that .m file, however it seems calling MATLAB files using 'import matlab.engine' isn't supported in Python 3.9 (what Im running) and was cut after 3.7
My questions is what is the proper convention to call a MATLAB script in Python 3.9?
EDIT: When I say 'call a MATLAB script', I mean run the script on it's own. Using the .py file to open the MATLAB window kind of defeats the purpose of this project.
This answer is for Python 3.6 but it should be applicable to 3.9 as well. On a Linux machine (and MacOS, possibly with slight modifications), you can execute a system command which opens MATLAB, runs your script, and then closes MATLAB,
import subprocess
run_matlab = 'matlab -nodisplay -nojvm -nosplash -nodesktop -r "try, test_script, catch e, disp(getReport(e)), exit(1), end, exit(0);"'
subprocess.run(run_matlab, shell=True, check=True)
The purpose of the try-catch block is to properly exit from MATLAB after executing the script test_script in case when the script raises an exception - throws an error. One could instead just exit (exit or quit in MATLAB returns proper exit code by default on those two OS), but with a try-catch block there are more opportunities to fix, notify, or save things when an exception occurs. This may be generally useful when running outside an interactive session.
The rest of the command opens MATLAB with no display of any kind, runs the script test_script, and exits MATLAB - exit(1) signifying an exception was raised, and exit(0) meaning that all went well.
Without the try-catch the command is much simpler,
run_matlab = 'matlab -nodisplay -nojvm -nosplash -nodesktop -r "test_script; exit;"'
On the Python side, I use subprocess.run() (or .call()) since it is newer, and recommended. The option shell means that the command will be executed through the shell. check means that if the run_matlab process (command) throws an error, the Python script will also throw one. The default is False and along with the try-catch block may be used to swiftly ignore crashing code.
As #CrisLuengo said in his comment, starting with MATLAB 2019a (MATLAB 2018b indeed does not have it) there is no need for the elaborated command above. All you need is
run_matlab = 'matlab -batch test_script.m'
Looking at the documentation, -batch invokes all the necessary options, and more.
This is more of a hack fix, but you can edit the Python package for the matlab engine on your system and bypass the version checker. Since Python is backwards compatible it shouldn't be a problem, depending on your use case.
Open the following file on your system (or the equivalent file under matlab engine):
vim /usr/lib/python3.9/site-packages/matlab/engine/__init__.py
In that file, change line 31 to include Python 3.9:
_supported_versions = ['2_7', '3_6', '3_7', '3_8', '3_9']
Finally, in line 37, change the version number to a valid version.
_PYTHONVERSION = '3_7' # or another new, supported version
After this, you should be able to run matlab's engine as if Python 3.9 were supported.
How can I remove the output buffering from Sublime Text 3 when I build a Python 3 script? I would like real-time output.
I am using Sublime Text 3 with the Anaconda plugin, Python 3.6 and Linux Mint 18. When I run a simple script using control-b:
print('hello')
I get an instant output in a separate window called 'Build output'. When I use a script with a repeated output, such as:
from time import sleep
count = 0
print('starting')
while True:
print('{} hello'.format(count))
count += 1
sleep(0.5)
Initially I get a blank screen in 'Build output'. Some time later it populates with several hundred lines of output. It looks like the output is being buffered. When the buffer is full, it outputs all at once to the 'Build output' screen.
Edit
Sublime Text allows custom build configurations. The default Python build is for python 2. I entered a build configuration for Python 3 and missed the -u flag. The fix is to put the -u flag in the Python 3 build.
File: Python3.sublime-build
{
"shell_cmd": "/usr/bin/env python3 -u ${file}",
"selector": "source.python",
"file_regex": "^(...*?):([0-9]*):?([0-9]*)",
"working_dir": "${file_path}",
}
Save in sublime_install/Data/Packages/User/Python3.sublime-build
By default the exec command is used to execute the commands in build systems, and the exec command doesn't buffer output at all. There is more information in this answer (which also provides a version of exec that does line buffering) but in short exec launches one thread to handle stdout and one to handle stderr, and both forward whatever data they get to the panel as soon as they get it.
As such, a problem like the one you're describing here is generally caused by the program doing it's own buffering. Depending on the language and platform that you're using, buffering may change from what you expect in unexpected ways:
For example, see this text in the man page for stdout under Linux:
The stream stderr is unbuffered. The stream stdout is line-buffered when it points to a terminal. Partial lines will not appear until fflush(3) or exit(3) is called, or a newline is printed. This can produce unexpected results, especially with debugging output.
In the general case, the solution to this problem would be to modify the program itself to ensure that it's not buffering, and how you would do that depends on the language you're using and the platform that you're on. It could be something as simple as setting an environment variable or as complex as startup code that ensures that regardless of circumstance buffering is set as you expect it to be.
In the specific case of Python, the -u command line argument to the interpreter tells Python to keep things unbuffered:
-u : unbuffered binary stdout and stderr; also PYTHONUNBUFFERED=x
see man page for details on internal buffering relating to '-u'
The Python.sublime-build that ships with Sublime uses this argument to the python command to ensure that the output is unbuffered, and using that build system works as expected for your sample program.
I don't use the Anaconda package so I'm not sure if it provides it's own build systems or not, but you may want to check the build command that you're using to ensure that it uses -u.
Running a simple .py or .pyw python file causes python.exe to show up under Task Manager.
python myApp.py
python myApp.pyw
However when we try to run it without using the console, the script does not appear to run, nor does python.exe or pythonw.exe appears under Task Manager
pythonw myApp.pyw
pythonw myApp.py
How do we troubleshoot the problem? The system is running Python 2.7.8 x64.
tl;dr
To troubleshoot, use output redirection on invocation:
pythonw myApp.py 1>stdout.txt 2>stderr.txt
This will capture stdout output, such as from print(), in file stdout.txt, and stderr output (such as from unhandled exceptions), in file stderr.txt; from PowerShell, use
cmd /c pythonw myApp.py 1>stdout.txt 2>stderr.txt).
Note that the very act of redirecting stdout may actually make your script work again, if the only reason for its failure with pythonw was the use of print (in Python 2.x - see below).
Caveat: This output redirection technique seemingly does not work when invoking *.pyw scripts directly (as opposed to by passing the script file path to pythonw.exe). Do let me know if you know why and/or if it does work for you.
To fix your script:
Place the following at the top of any Python 2.x or 3.x script that you want to run with pythonw.exe:
import sys, os
if sys.executable.endswith("pythonw.exe"):
sys.stdout = open(os.devnull, "w");
sys.stderr = open(os.path.join(os.getenv("TEMP"), "stderr-"+os.path.basename(sys.argv[0])), "w")
This ensures the following when a script is run with pythonw.exe:
print() calls and explicit calls to sys.stdout() are effectively ignored (are no-ops).
Stderr output, including from an unhandled fatal exception, is sent to file
%TEMP%\stderr-<scriptFileName>; %TEMP% is a standard Windows environment variable that points to the current user's folder for temporary files.
In other words: With the above code in place, check file %TEMP%\stderr-<scriptFileName> after your script has failed silently when invoked with pythonw.exe.
For an explanation, read on.
On Windows, pythonw.exe is for launching GUI/no-UI-at-all scripts, which means that the
standard in- and output streams - sys.stdin, sys.stdout, sys.stderr are NOT available.
This has two nasty side effects:
Using print() - which targets sys.stdout by default - causes an exception in Python 2.x.
This problem has been fixed in Python 3.x.
Any unhandled exception - including one triggered by print() in 2.x - causes the script to abort silently.
Exception error messages go to sys.stderr by default, which is the very thing not available in this scenario.
The above code fixes these problems by:
sending stdout output to the null device, effectively ignoring any attempt to output to sys.stdout - whether explicitly, or implicitly via print().
sending all stderr output to a temporary file.
Differences between Python 2.x and Python 3.x:
When a script is run with pythonw.exe, sys.stdin, sys.stdout, and sys.stderr:
in Python 2.x: have invalid file descriptors
The eventual result when trying to write to sys.stdout or sys.stderr is the following exception: IOError: [Errno 9] Bad file descriptor
Pitfall: Due to output buffering, this exception may not surface until you've output, say, 4K bytes; you can provoke it instantly by invoking pythonw.exe with -u (for unbuffered output).
print() blindly tries to sys.stdout (by default), so it provokes this exception sooner or later.
in Python 3.x: are set to None
This is complemented with the 3.x print() function performing a no-op (doing nothing) when it finds that sys.stdout is None, so that print() statements can by default safely be used - they'll simply be ignored when run with pythonw.exe
However, it follows that trying to use sys.stdout.write() and sys.stderr.write() still results in an exception.
See here for more background.
Try adding the line import sys; sys.stderr = open("errlog.txt", "w") to the start of myApp.py. Then look in errlog.txt for a traceback or any other error messages.
I faced the same problem on a script of my own and found that when adding the output from Ross' answer the script would actually run.
It appears that for some reason that redirecting output fixes the problem. Since I'm not interested in writing the output to disk I've instead written it to /dev/null (or the platform equivalent) with:
if ( sys.platform == 'win32' and sys.executable.split( '\\' )[-1] == 'pythonw.exe'):
sys.stdout = open(os.devnull, 'w')
sys.stderr = open(os.devnull, 'w')
The if statement ensures it only happens when the script is launched from pythonw.exe. I'm not sure if it is related but it was important to do this before other imports (including e.g. import logging).
I was having similar problem.
After debugging step by step by writing to a log file, I discovered that pythonw.exe crashed after a statement that tried to use the call: sys.stdout.write(). It turns out, when run with pythonw.exe, sys.stdout is None.
If you are using functions of sys.stdout/stderr/stdin, and intend to use your program with pythonw.exe, adding a check for "None" is a good idea.
Im not sure I understand your problem but I think this is what you need to know
you need to right click on a py or pyw file and select open with ... find python.exe (probably C:\Python27\python.exe) .. check the box that says always open ... now you can just double click it if you want to run it
(usually the installer sets this up for you ...)
This is an old answer, but I want to leave my solution here also:
Open CMD (with elevated privileges or not - depends on your needs)
Change to directory of .py / .pyw script - this is important
Run pythonw with script as argument
cd E:\my\script\folder\
pythonw script.py
I had a similar problem after an upgrade to my computer RAM. Turns out I had to reinstall Pillow (library used for image processing). So make sure it is installed and if it's not, install it using "pip install Pillow" in cmd.