I am trying to do the following command (to download Calibre through a python script):
sudo -v && wget -nv -O- https://raw.githubusercontent.com/kovidgoyal/calibre/master/setup/linux-installer.py | sudo python -c "import sys; main=lambda:sys.stderr.write('Download failed\n'); exec(sys.stdin.read()); main()"
Following some of the answers on here on how to PIPE, I have been doing this:
import subprocess
from subprocess import Popen, PIPE
wget = subprocess.Popen(["sudo -v wget -nv -O- https://raw.githubusercontent.com/kovidgoyal/calibre/master/setup/linux-installer.py"], stdout=PIPE)
run = subprocess.Popen(["sudo python -c "import sys; exec(sys.stdin.read()); main()""], stdin=wget.stdout)
I have tried changing many things but NOTHING is working. There are too many errors to put here. Can anyone correct this? Many thanks in advance.
All I get is the first
When you have parameters you need to break it into a list, e.g.
wget = subprocess.Popen(["wget -nv -O- https://raw.githubusercontent.com/kovidgoyal/calibre/master/setup/linux-installer.py"])
(**No such file or directory** error, because it looks for that whole string as a command/file)
Needs to become:
subprocess.Popen(['wget', '-nv', '-O-', 'https://raw.githubusercontent.com/kovidgoyal/calibre/master/setup/linux-installer.py'])
<subprocess.Popen object at 0x10e203950>
You can also use shlex.split() to split your command for you, e.g.
>>> import shlex
>>> shlex.split('wget -nv -O- https://raw.githubusercontent.com/kovidgoyal/calibre/master/setup/linux-installer.py')
['wget', '-nv', '-O-', 'https://raw.githubusercontent.com/kovidgoyal/calibre/master/setup/linux-installer.py']
Ref. https://docs.python.org/2/library/subprocess.html#popen-constructor
You also have "sudo python -c "import sys; exec(sys.stdin.read()); main()"", I don't think it's acceptable syntax because you have quotes inside quotes (the inside ones close out the first opening quote), so try 'sudo python -c "import sys; exec(sys.stdin.read()); main()"' with single quotes instead, that way you don't have to escape the quotes inside!
To avoid escaping quotes inside the command, you could use triple quotes:
from subprocess import check_call
check_call(r'''# http://calibre-ebook.com/download_linux
sudo -v &&
wget -nv -O- https://raw.githubusercontent.com/kovidgoyal/calibre/master/setup/linux-installer.py |
sudo python -c "import sys; main=lambda:sys.stderr.write('Download failed\n'); exec(sys.stdin.read()); main()"
''', shell=True)
shell=True causes the first parameter to be interpreted as a shell command.
In principle, you could simplify the command:
#!/usr/bin/env python3
import sys
from subprocess import check_call
from urllib.request import urlretrieve
# http://calibre-ebook.com/download_linux
path, _ = urlretrieve('https://raw.githubusercontent.com/kovidgoyal/calibre/master/setup/linux-installer.py')
check_call(['sudo', sys.executable, path])
but it may have subtle security implications e.g., urlretrieve() may skip server ssl certificate validation.
Related
What is the best way to execute the below command in Python in a single line?
echo $(readlink /sys/dev/block/$(mountpoint -d /))
Tried using individual os.system(cmd) by separating - "mountpoint -d /" first and taking the output and appending to "readlink /sys/dev/block/${0}".format(out.strip()) and doing an echo works. Tried using subprocess and subprocess.Popen and subprocess.check_output but it raises raise CalledProcessError
cmd = "echo $(readlink /sys/dev/block/$(mountpoint -d /))"
You have to call the subcommand separately. And you can use python methods to read the link:
import subprocess
import os
path = "/"
device = subprocess.run(["mountpoint", "-d", path], stdout=subprocess.PIPE, encoding="utf8").stdout.strip()
link = os.readlink("/sys/dev/block/" + device)
print(link)
You probably want to use something like the following:
cmd = "bash -c 'echo $(readlink /sys/dev/block/$(mountpoint -d /))'"
echo doesn't substitute $() blocks, that's what your shell does, so you have to call the shell. os.system(cmd) should work then.
I am running a python3 script which performs the following snippet on Debian 9:
os.environ["PA_DIR"] = "/home/myname/some_folder"
command_template = ("sudo java -Dconfig.file=$PA_DIR/google.conf "
"-jar ~/big/cromwell-42.jar run $PA_DIR/WholeGenomeGermlineSingleSample.wdl "
"-i {} -o $PA_DIR/au_options.json > FDP{}.log 2>&1")
command = command_template.format("test.json, "1")
os.system("screen -dm -S S{} bash -c '{}'".format("1", command))
The use of PA_DIR works as intended. When I tried it on command line:
PA_DIR="/home/myname/some_folder"
screen -dm -S S1 bash -c 'sudo java -Dconfig.file=$PA_DIR/google.conf -jar ~/big/cromwell-42.jar run $PA_DIR/WholeGenomeGermlineSingleSample.wdl -i test.json -o $PA_DIR/au_options.json > FDP1.log 2>&1'
it doesn't do variable substitution due to single quotes and I had to replace them with double quotes (it complains it cannot find the file /google.conf).
What is different when python runs it?
Thanks!
The Python os.system() invokes the underlying system function of the C library, which on POSIX systems is equivalent to do something like
sh -c "your_command and all its arguments"
So the command and all arguments are already surrounded by double-quotes, which does environment variable substitution. Any single quotes inside the string is irrelevant for the variable substitution.
You can test it easily. In a shell do something like
$ foo="bar"
$ echo "foo is '$foo'" # Will print foo is 'bar'
$ echo 'foo is "$foo"' # Will print foo is "$foo"
Waiting for your answer to daltonfury42, I'd bet the problem is, when running in a command line, you are not exporting the PA_DIR environment variable so it is not present in the second bash interpreter. And it behaves different beacuse of what Mihir answered.
If you run
PA_DIR=foo
you only declare a bash variable but it is not an environment variable. Then
bash -c "echo $PA_DIR"
this will output foo because your current interpreter interpolates $PA_DIR and then raises a second bash process with the command echo foo. But
bash -c 'echo $PA_DIR'
this prevents your bash interpreter from interpolating it so it raises a second bash process with the comand echo $PA_DIR. But in this second process the variable PA_DIR does not exist.
If you start your journey running
export PA_DIR=foo
this will become an environment variable that will be accessible to children processes, thus
bash -c 'echo $PA_DIR'
will output foo because the nested bash interpreter has access to the variable even if the parent bash interpreter did not interpolate it.
The same is true for any kind of children process. Try running
PA_DIR=foo
python3 -c 'import os; print(os.environ.get("PA_DIR"))'
python3 -c "import os; print(os.environ.get('PA_DIR'))"
export PA_DIR=foo
python3 -c 'import os; print(os.environ.get("PA_DIR"))'
python3 -c "import os; print(os.environ.get('PA_DIR'))"
in your shell. No quotes are involved here!
When you use the os.environ dictionary in a Python script, Python will export the variables for you. That's why you will see the variable interpolated by either
os.system("bash -c 'echo $PA_DIR'")
or
os.system('bash -c "echo $PA_DIR"')
But beware that in each case it is either the parent or either the children shell process who is interpolating the variable.
You must understand your process tree here:
/bin/bash # but it could be a zsh, fish, sh, ...
|- /usr/bin/python3 # presumably
|- /bin/sh # because os.system uses that
|- /bin/bash
If you want an environment variable to exist in the most nested process, you must export it anywhere in the upper tree. Or in that very process.
I have the following command that works in the shell:
$ pv itunes20140910.tbz | sudo tar xpj -C /tmp
However, when I try and do it in python, it doesn't work:
>>> import subprocess
>>> import shlex
>>> cmd=shlex.split('pv itunes20140910.tbz | sudo tar xpj -C /tmp')
>>> subprocess.call(cmd)
pv: invalid option -- 'C'
Try `pv --help' for more information.
1
What am I doing wrong here, and what would be the correct command to run in python?
The above answers didn't have the net effect of what I was looking for (the progress bar), though the command would run without error. Here is what worked for me:
>>> import shlex, subprocess
>>> p1 = subprocess.Popen(shlex.split('pv /tmp/itunes20140910.tbz'), stdout=subprocess.PIPE) #Set up the echo command and direct the output to a pipe
>>> subprocess.Popen(shlex.split('sudo tar xpj -C /tmp'), stdin=p1.stdout) #send p1's output to p2
Use shell=True argument. Otherwise | cannot be interpreted.
subprocess.call('pv itunes20140910.tbz | sudo tar xpj -C /tmp', shell=True)
I have this command as part of a bash script
$(python -c "import urllib, sys; print urllib.unquote(sys.argv[0])", "h%23g")
But when I run it, I get this:
-bash: -c: command not found
As though bash has missed reading the python, and is thinking -c is the name of the command. Exactly the same happens when using backticks.
How can I make bash recognise the python?
the Python command is returning the string "-c" from your $(...) structure, which bash then tries to execute.
for example
python -c "import urllib, sys; print urllib.unquote(sys.argv[0])"
prints "-c", so you are essentially asking bash to interpret $(-c), for which the error is valid.
I think you want something like the following:
$(python -c "import urllib, sys; print urllib.unquote(sys.argv[1])" "h%23g")
This will result in h#g, if this is all you have on a line then it will also attempt to run a command called h#g, so I'm assuming you are actually using this as a part of a larger command.
The issue with your version is that sys.argv[0] is the -c from the command, and urllib.unquote('-c') will just return '-c'.
From the documentation on sys.argv:
If the command was executed using the -c command line option to the interpreter, argv[0] is set to the string '-c'.
Combining that with info from the man page (emphasis mine):
-c command
Specify the command to execute (see next section). This terminates the option list (following options are passed as arguments to the command).
So, when you use -c, sys.argv[0] will be '-c', the argument provided to -c is the script so it will not be included in sys.argv, and any additional arguments are added to sys.argv starting at index 1.
I'm trying to execute a rsync command via subrocess & popen. Everything's ok until I don't put the rsh subcommand where things go wrong.
from subprocess import Popen
args = ['-avz', '--rsh="ssh -C -p 22 -i /home/bond/.ssh/test"', 'bond#localhost:/home/bond/Bureau', '/home/bond/data/user/bond/backups/']
p = Popen(['rsync'] + args, shell=False)
print p.wait()
#just printing generated command:
print ' '.join(['rsync']+args)
I've tried to escape the '--rsh="ssh -C -p 22 -i /home/bond/.ssh/test"' in many ways, but it seems that it's not the problem.
I'm getting the error
rsync: Failed to exec ssh -C -p 22 -i /home/bond/.ssh/test: No such file or directory (2)
If I copy/paste the same args that I output at the time, I'm getting a correct execution of the command.
Thanks.
What happens if you use '--rsh=ssh -C -p 22 -i /home/bond/.ssh/test' instead (I removed the double quotes).
I suspect that this should work. What happens when you cut/paste your line into the commandline is that your shell sees the double quotes and removes them but uses them to prevent -C -p etc. from being interpreted as separate arguments. when you call subprocess.Popen with a list, you've already partitioned the arguments without the help of the shell, so you no longer need the quotes to preserve where the arguments should be split.
Having the same problem, I googled this issue extensively. It would seem you simply cannot pass arguments to ssh with subprocess. Ultimately, I wrote a shell script to run the rsync command, which I could pass arguments to via subprocess.call(['rsyncscript', src, dest, sshkey]). The shell script was: /usr/bin/rsync -az -e "ssh -i $3" $1 $2
This fixed the problem.