I am trying to open chromium to an authorization URL with a specific launch parameter. I was not able to find the solution to this launch parameter using the webbrowser library, so I moved onto os.system
browser_cmd = "chromium-browser --password-store=basic " + auth_url
os.system(browser_cmd)
This works up until the "&" in the URL. So chromium opens without bothering me with keyring nonsense, but only opens the URL until the first &. Is there a way of handling the URL and maintaining its integrity?
This is because & is special to the shell. The canonical way to run a subprocess from within Python 3 is:
import subprocess
subprocess.run(['chromium-browser', '--password-store=basic', auth_url],
check=True)
print('chromium-browser exited successfully')
This is exactly why you shouldn't use os.system for anything non-trivial. As the docs for that function say:
The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function. See the Replacing Older Functions with the subprocess Module section in the subprocess documentation for some helpful recipes.
If you follow that link, you'll see that you can write this as:
subprocess.call(['chromium-browser', '--password-store=basic', auth_url])
Because you're passing a list of arguments, rather than trying to put them together into a string that you can smuggle through the shell, you don't have to worry about quoting, or anything else.
By the way, you probably want to use run rather than call here, but for some reason the recipes still haven't been updated in the docs as of 3.7, and I didn't want to add confusion by showing something that doesn't match… Anyway, you should read the at least the quick-start "Using" section at the top of the docs.
If you really want to use os.system anyway for some reason, you will need to quote and/or escape the auth_url argument. Assuming you don't care about Windows, the best way to do this is with the shlex module:
browser_cmd = "chromium-browser --password-store=basic " + shlex.quote(auth_url)
os.system(browser_cmd)
If you do care about Windows, you can add posix=False to the quote call. The effects of the posix flag are documented under the shlex constructor. The default True value means it follows POSIX rules as closely as possible, which means it should be able to handle anything that could possibly be handled, as long as your shell is strictly compatible with sh (as, e.g., bash is, but Windows cmd is definitely not, and even tcsh or fish may not be). With False, it uses "compatibility mode". For simple examples like yours, it should work for most shells without fiddling, but if you need to get more complicated, you need to read Improved Compatibility with Shells. (And, for Windows cmd or PowerShell, there's a limit to how far you can push things.)
Related
I'm currently doing some work in a server (Ubuntu) without admin rights nor contact with the administrator. When using the help(command) in the python command line I get an error.
Here's an example:
>>> help(someCommand)
/bin/sh: most: command not found
So, this error indicates that most pager is not currently installed. However, the server I'm working on has "more" and "less" pagers installed. So, how can I change the default pager configuration for this python utility?
This one is annoyingly difficult to research, but I think I found it.
The built-in help generates its messages using the standard library pydoc module (the module is also intended to be usable as a standalone script). In that documentation, we find:
When printing output to the console, pydoc attempts to paginate the output for easier reading. If the PAGER environment variable is set, pydoc will use its value as a pagination program.
So, presumably, that's been set to most on your system. Assuming it won't break anything else on your system, just unset or change it. (It still pages without a value set - even on Windows. I assume it has a built-in fallback.)
You can make a custom most script that just invokes less (or even more).
The steps would be:
Set up a script called most, the contents of which are:
#!/bin/sh
less ${#:1} # wierdess is just "all arguments except argument 0"
Put that script in a location that is on your PATH
Then most filename should just run less on that file, and that command should get called from in your python interpreter.
To be honest though, I'd just use Karl's approach.
You can view the various pager options in the source code. That function can be replaced to return whatever is desired. For example:
import pydoc
pydoc.getpager = lambda: lambda text: pydoc.pipepager(text, 'less')
I have recently came across a few posts on stack overflow saying that subprocess is much better than os.system, however I am having difficulty finding the exact advantages.
Some examples of things I have run into:
https://docs.python.org/3/library/os.html#os.system
"The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function."
No idea in what ways it is more powerful though, I know it is easier in many ways to use subprocess but is it actually more powerful in some way?
Another example is:
https://stackoverflow.com/a/89243/3339122
The advantage of subprocess vs system is that it is more flexible (you can get the stdout, stderr, the "real" status code, better error handling, etc...).
This post which has 2600+ votes. Again could not find any elaboration on what was meant by better error handling or real status code.
Top comment on that post is:
Can't see why you'd use os.system even for quick/dirty/one-time. subprocess seems so much better.
Again, I understand it makes some things slightly easier, but I hardly can understand why for example:
subprocess.call("netsh interface set interface \"Wi-Fi\" enable", shell=True)
is any better than
os.system("netsh interface set interface \"Wi-Fi\" enabled")
Can anyone explain some reasons it is so much better?
First of all, you are cutting out the middleman; subprocess.call by default avoids spawning a shell that examines your command, and directly spawns the requested process. This is important because, besides the efficiency side of the matter, you don't have much control over the default shell behavior, and it actually typically works against you regarding escaping.
In particular, do not do this:
subprocess.call('netsh interface set interface "Wi-Fi" enable')
since
If passing a single string, either shell must be True (see below) or else the string must simply name the program to be executed without specifying any arguments.
Instead, you'll do:
subprocess.call(["netsh", "interface", "set", "interface", "Wi-Fi", "enable"])
Notice that here all the escaping nightmares are gone. subprocess handles escaping (if the OS wants arguments as a single string - such as Windows) or passes the separated arguments straight to the relevant syscall (execvp on UNIX).
Compare this with having to handle the escaping yourself, especially in a cross-platform way (cmd doesn't escape in the same way as POSIX sh), especially with the shell in the middle messing with your stuff (trust me, you don't want to know what unholy mess is to provide a 100% safe escaping for your command when calling cmd /k).
Also, when using subprocess without the shell in the middle you are sure you are getting correct return codes. If there's a failure launching the process you get a Python exception, if you get a return code it's actually the return code of the launched program. With os.system you have no way to know if the return code you get comes from the launched command (which is generally the default behavior if the shell manages to launch it) or it is some error from the shell (if it didn't manage to launch it).
Besides arguments splitting/escaping and return code, you have way better control over the launched process. Even with subprocess.call (which is the most basic utility function over subprocess functionalities) you can redirect stdin, stdout and stderr, possibly communicating with the launched process. check_call is similar and it avoids the risk of ignoring a failure exit code. check_output covers the common use case of check_call + capturing all the program output into a string variable.
Once you get past call & friends (which is blocking just as os.system), there are way more powerful functionalities - in particular, the Popen object allows you to work with the launched process asynchronously. You can start it, possibly talk with it through the redirected streams, check if it is running from time to time while doing other stuff, waiting for it to complete, sending signals to it and killing it - all stuff that is way besides the mere synchronous "start process with default stdin/stdout/stderr through the shell and wait it to finish" that os.system provides.
So, to sum it up, with subprocess:
even at the most basic level (call & friends), you:
avoid escaping problems by passing a Python list of arguments;
avoid the shell messing with your command line;
either you have an exception or the true exit code of the process you launched; no confusion about program/shell exit code;
have the possibility to capture stdout and in general redirect the standard streams;
when you use Popen:
you aren't restricted to a synchronous interface, but you can actually do other stuff while the subprocess run;
you can control the subprocess (check if it is running, communicate with it, kill it).
Given that subprocess does way more than os.system can do - and in a safer, more flexible (if you need it) way - there's just no reason to use system instead.
There are many reasons, but the main reason is mentioned directly in the docstring:
>>> os.system.__doc__
'Execute the command in a subshell.'
For almost all cases where you need a subprocess, it is undesirable to spawn a subshell. This is unnecessary and wasteful, it adds an extra layer of complexity, and introduces several new vulnerabilities and failure modes. Using subprocess module cuts out the middleman.
How can i run an external program, let's say "Firefox", from my python script and make sure that its process will remain alive after the termination of my python script? I want to make it crossplatform if it's doable.
There is no cross-platform way to do this with just the stdlib. However, if you write code for POSIX and for Windows, that's usually good enough, right?
On Windows, you want to pass a creationflags argument. Read the docs (both there and at MSDN) and decide whether you want a console-detached process, a new-console process, or a new-process-group process, then use the appropriate flag. You may also want to set some of the flags in startupinfo; again, MSDN will tell you what they mean.
On POSIX, if you just want the simplest behavior, and you're using 3.2+, you want to pass start_new_session=True. In earlier Python versions, or for other cases, you want to pass a preexec_fn that allows you to do whatever daemonization you want. That could be as little as os.setsid() (what start_new_session does), or a whole lot more. See PEP 3143 -- Standard daemon process library for a discussion of all of the different things you might want to do here.
So, the simplest version is:
def launch_in_background(args):
try:
subprocess.CREATE_NEW_PROCESS_GROUP
except AttributeError:
# not Windows, so assume POSIX; if not, we'll get a usable exception
p = subprocess.Popen(args, start_new_session=True)
else:
# Windows
p = subprocess.Popen(args, creationflags=subprocess.CREATE_NEW_PROCESS_GROUP)
If you're willing to go outside the stdlib, there are dozens of "shell-type functionality" libraries out there, many of which have some kind of "detach" functionality. Just search shell, cli, or subprocess at PyPI and find the one you like best.
Per Python documentation, subprocess.call should be blocking and wait for the subprocess to complete. In this code I am trying to convert few xls files to a new format by calling Libreoffice on command line. I assumed that the call to subprocess call is blocking but seems like I need to add an artificial delay after each call otherwise I miss few files in the out directory.
what am I doing wrong? and why do I need the delay?
from subprocess import call
for i in range(0,len(sorted_files)):
args = ['libreoffice', '-headless', '-convert-to',
'xls', "%s/%s.xls" %(sorted_files[i]['filename'],sorted_files[i]['filename']), '-outdir', 'out']
call(args)
var = raw_input("Enter something: ") # if comment this line I dont get all the files in out directory
EDIT It might be hard to find the answer through the comments below. I used unoconv for document conversion which is blocking and easy to work with from an script.
It's possible likely that libreoffice is implemented as some sort of daemon/intermediary process. The "daemon" will (effectively1) parse the commandline and then farm the work off to some other process, possibly detaching them so that it can exit immediately. (based on the -invisible option in the documentation I suspect strongly that this is indeed the case you have).
If this is the case, then your subprocess.call does do what it is advertised to do -- It waits for the daemon to complete before moving on. However, it doesn't do what you want which is to wait for all of the work to be completed. The only option you have in that scenario is to look to see if the daemon has a -wait option or similar.
1It is likely that we don't have an actual daemon here, only something which behaves similarly. See comments by abernert
The problem is that the soffice command-line tool (which libreoffice is either just a link to, or a further wrapper around) is just a "controller" for the real program soffice.bin. It finds a running copy of soffice.bin and/or creates on, tells it to do some work, and then quits.
So, call is doing exactly the right thing: it waits for libreoffice to quit.
But you don't want to wait for libreoffice to quit, you want to wait for soffice.bin to finish doing the work that libreoffice asked it to do.
It looks like what you're trying to do isn't possible to do directly. But it's possible to do indirectly.
The docs say that headless mode:
… allows using the application without user interface.
This special mode can be used when the application is controlled by external clients via the API.
In other words, the app doesn't quit after running some UNO strings/doing some conversions/whatever else you specify on the command line, it sits around waiting for more UNO commands from outside, while the launcher just runs as soon as it sends the appropriate commands to the app.
You probably have to use that above-mentioned external control API (UNO) directly.
See Scripting LibreOffice for the basics (although there's more info there about internal scripting than external), and the API documentation for details and examples.
But there may be an even simpler answer: unoconv is a simple command-line tool written using the UNO API that does exactly what you want. It starts up LibreOffice if necessary, sends it some commands, waits for the results, and then quits. So if you just use unoconv instead of libreoffice, call is all you need.
Also notice that unoconv is written in Python, and is designed to be used as a module. If you just import it, you can write your own (simpler, and use-case-specific) code to replace the "Main entrance" code, and not use subprocess at all. (Or, of course, you can tear apart the module and use the relevant code yourself, or just use it as a very nice piece of sample code for using UNO from Python.)
Also, the unoconv page linked above lists a variety of other similar tools, some that work via UNO and some that don't, so if it doesn't work for you, try the others.
If nothing else works, you could consider, e.g., creating a sentinel file and using a filesystem watch, so at least you'll be able to detect exactly when it's finished its work, instead of having to guess at a timeout. But that's a real last-ditch workaround that you shouldn't even consider until eliminating all of the other options.
If libreoffice is being using an intermediary (daemon) as mentioned by #mgilson, then one solution is to find out what program it's invoking, and then directly invoke it yourself.
How to send string/data to STDIN of a running process in python?
i'd like to create a front end for a CLI program. eg. i want to pass multiple string to this Pascal application:
program spam;
var a,b,c:string;
begin
while e <> "no" do
begin
writeln('what is your name?');
readln(a);
writeln('what is your quest?');
readln(b);
writeln('what is your favorite color?');
readln(c);
print(a,b,c);
end;
end.
how do i pass string to this program from python (using subprocess module in python). thankyou. sorry for my english.
If you want to control another interactive program, it could be worth trying the Pexpect module to do so. It is designed to look for prompt messages and so on, and interact with the program. Note that it doesn't currently work directly on Windows - it does work under Cygwin.
A possible non-Cygwin Windows variant is WinPexpect, which I found via this question. One of the answers on that question suggests the latest version of WinPexpect is at http://sage.math.washington.edu/home/goreckc/sage/wexpect/, but looking at the modification dates I think the BitBucket (the first link) is actually the latest.
As Windows terminals are somewhat different to Unix ones, I don't think there is a direct cross-platform solution. However, the WinPexpect docs say the only difference in the API between it and pexpect is the name of the spawn function. You could probably do something like the following (untested) code to get it to work in both:
try:
import pexpect
spawn = pexpect.spawn
except ImportError:
import winpexpect
spawn = winpexpect.winspawn
# NB. Errors may occur when you run spawn rather than (or as
# well as) when you import it, so you may have to wrap this
# up in a try...except block and handle them appropriately.
child = spawn('command and args')