I am trying to run a gerrit cherry pick query in python
query_to_run='git fetch https://gerritserver.com/projectname refs/changes/51/1151/1 ' + '&&' + ' git cherry-pick FETCH_HEAD'
I am getting error:
fatal: Couldn't find remote ref &&
Unexpected end of command stream
My code works with other gerrit queries but not this one, is it the && which is causing problem!
thanks
Pratibha
The && token has no meaning to Git or Gerrit but is interpreted by your shell. By default the subprocess module doesn't pass off commands to the shell but runs the process directly, so the string in query_to_run is sent as a single command. To force subprocess.Popen(), subprocess.check_call() or whatever you're using to pass the command to a shell, pass shell=True:
subprocess.check_call(query_to_run, shell=True)
However, the use of shell=True is discouraged and is unnecessary in this case. What && does is simply run one command and, if successful, run another command. It's basically equivalent to this sequence of Python statements:
subprocess.check_call(command1)
subprocess.check_call(command2)
Alternatively, if you prefer not have exceptions thrown when either of the commands fail:
subprocess.call(command1) != 0 and subprocess.call(command2) != 0
In addition to this, I strongly recommend making a good habit out of passing lists of arguments to process execution functions instead of strings. Passing strings works fine a lot of the time, but when arguments contain spaces you suddenly need to think about quoting.
Putting everything together, this is what I think your code should look like:
try:
subprocess.check_call(['git', 'fetch',
'https://gerritserver.com/projectname',
'refs/changes/51/1151/1'])
subprocess.check_call(['git', 'cherry-pick', 'FETCH_HEAD'])
except (EnvironmentError, subprocess.CalledProcessError):
# Suitable error handling here. I'm not sure about
# the possibility of EnvironmentError exceptions.
Also, a note on terminology: You're talking about Gerrit queries, but using that language might confuse people. By Gerrit query one usually means the Lucene query string entered into the search box in the UI (or the equivalent REST API).
Related
I am trying to run an ssh command within a python script using os.system to add a 0 at the end of a fully matched string in a remote server using ssh and sed.
I have a file called nodelist in a remote server that's a list that looks like this.
test-node-1
test-node-2
...
test-node-11
test-node-12
test-node-13
...
test-node-21
I want to use sed to make the following modification, I want to search test-node-1, and when a full match is found I want to add a 0 at the end, the file must end up looking like this.
test-node-1 0
test-node-2
...
test-node-11
test-node-12
test-node-13
...
test-node-21
However, when I run the first command,
hostname = 'test-node-1'
function = 'nodelist'
os.system(f"ssh -i ~/.ssh/my-ssh-key username#serverlocation \"sed -i '/{hostname}/s/$/ 0/' ~/{function}.txt\"")
The result becomes like this,
test-node-1 0
test-node-2
...
test-node-11 0
test-node-12 0
test-node-13 0
...
test-node-21
I tried adding a \b to the command like this,
os.system(f"ssh -i ~/.ssh/my-ssh-key username#serverlocation \"sed -i '/\b{hostname}\b/s/$/ 0/' ~/{function}.txt\"")
The command doesn't work at all.
I have to manually type in the node name instead of using a variable like so,
os.system(f"ssh -i ~/.ssh/my-ssh-key username#serverlocation \"sed -i '/\btest-node-1\b/s/$/ 0/' ~/{function}.txt\"")
to make my command work.
What's wrong with my command, why can't I do what I want it to do?
This code has serious security problems; fixing them requires reengineering it from scratch. Let's do that here:
#!/usr/bin/env python3
import os.path
import shlex # note, quote is only here in Python 3.x; in 2.x it was in the pipes module
import subprocess
import sys
# can set these from a loop if you choose, of course
username = "whoever"
serverlocation = "whereever"
hostname = 'test-node-1'
function = 'somename'
desired_cmd = ['sed', '-i',
f'/\\b{hostname}\\b/s/$/ 0/',
f'{function}.txt']
desired_cmd_str = ' '.join(shlex.quote(word) for word in desired_cmd)
print(f"Remote command: {desired_cmd_str}", file=sys.stderr)
# could just pass the below direct to subprocess.run, but let's log what we're doing:
ssh_cmd = ['ssh', '-i', os.path.expanduser('~/.ssh/my-ssh-key'),
f"{username}#{serverlocation}", desired_cmd_str]
ssh_cmd_str = ' '.join(shlex.quote(word) for word in ssh_cmd)
print(f"Local command: {ssh_cmd_str}", file=sys.stderr) # log equivalent shell command
subprocess.run(ssh_cmd) # but locally, run without a shell
If you run this (except for the subprocess.run at the end, which would require a real SSH key, hostname, etc), output looks like:
Remote command: sed -i '/\btest-node-1\b/s/$/ 0/' somename.txt
Local command: ssh -i /home/yourname/.ssh/my-ssh-key whoever#whereever 'sed -i '"'"'/\btest-node-1\b/s/$/ 0/'"'"' somename.txt'
That's correct/desired output; the funny '"'"' idiom is how one safely injects a literal single quote inside a single-quoted string in a POSIX-compliant shell.
What's different? Lots:
We're generating the commands we want to run as arrays, and letting Python do the work of converting those arrays to strings where necessary. This avoids shell injection attacks, a very common class of security vulnerability.
Because we're generating lists ourselves, we can change how we quote each one: We can use f-strings when it's appropriate to do so, raw strings when it's appropriate, etc.
We aren't passing ~ to the remote server: It's redundant and unnecessary because ~ is the default place for a SSH session to start; and the security precautions we're using (to prevent values from being parsed as code by a shell) prevent it from having any effect (as the replacement of ~ with the active value of HOME is not done by sed itself, but by the shell that invokes it; because we aren't invoking any local shell at all, we also needed to use os.path.expanduser to cause the ~ in ~/.ssh/my-ssh-key to be honored).
Because we aren't using a raw string, we need to double the backslashes in \b to ensure that they're treated as literal rather than syntactic by Python.
Critically, we're never passing data in a context where it could be parsed as code by any shell, either local or remote.
This question already has answers here:
When to wrap quotes around a shell variable?
(5 answers)
How to pass all arguments passed to my Bash script to a function of mine? [duplicate]
(7 answers)
Closed 3 years ago.
The problem:
I'm writing a program that performs actions in python based on links, and possibly expanding it to do things beyond that. This program is meant to be quickly used through bash. So, I'm using a custom function to do that.
function youtube() {python3 <youtube program path here> $1}
As for the python file; I'm using sys, os, and re in order to make it function. sys, in order to use both sys.exit() and var = sys.argv[<argNum>], the former in order to exit the program using custom exceptions, like error.searchError() or error.usageError(), and the later for actualling using the arguments from the command itself. os is just for os.system('{}'.format(<your command here>)). And re is for removing the spaces from the second argument, where my problem lies, and replacing them with '+', as per query = re.sub(' ', '+', query).
Now, as for the problem itself. As I mentioned before, the problem lies with the second bash argument, or sys.argv[2]. With sys.argv[0] being the python file, and sys.argv[1] being the option, in this case being -s.
sys.argv[2] is meant to be the actual youtube search query. But, according to whenever I use the command with all three arguments, youtube -s Hi this is a test., I get the following output, as per the custom error I made: No search query provided!. This only happens when python excepts an IndexError, which means that the program is not receiving the second argument from bash or zsh. What is actually supposed to happen, when the second arguments does exist, is:
os.system('open https://www.youtube.com/results?search_query=Hi+this+is+a+test.')
Which opens that link in my default browser. I have tried to add $2 to the custom function, and various ways of entering the second argument through the python source itself, including using a x = input('Search Query: ') statement. But that isn't optimal for what I'm doing.
The code:
The following is the source code for all the nonsense I just typed out.
The custom function:
function youtube() {python3 <python program path here> $1}
For those that have no idea what this means (i.e.; people that don't know much (or anything) about bash coding); The function method creates a bash object, in this case, youtube(). As for the code in the brackets ({}), this uses the function python3, which just pushes the program in argument 0 to the python 3.x interpreter, to open <python program path here>, which is a placeholder for the actual path of the program. As for $1, this is a variable that always equals the text inputted after the function name.
The custom errors:
class error:
def usageError():
usageError = '''Usage: youtube [-s] [<search_query>]
Help: Performs actions related to https://www.youtube.com
Options:
-s Opens https://www.youtube.com/results?search_query=<your query here>'''
print(usageError)
sys.exit()
def searchError():
searchError = 'No search query provided!'
print(searchError)
sys.exit()
Is this irrelevant? I'm not sure, but I'm putting it in anyway! Now, if you don't understand it, the following should explain it.
The error class contains all of the customs errors for this program, ok? Ok, so you get that, but what do these functions do? usageError is raised when argument 1 simply doesn't exist, and prints the usage information to the terminal. Then sys.exit()s the program, basically the equivalent of hitting Alt+f4 in video game. searchError, on the other hand, only happens if argument 2 doesn't exist, meaning there is no search query. It then tells you that you're stupid, and will need to actually enter your query for it to work.
Well, maybe not that exactly, but you get the point.
The Juicy Bits:
option = ''
try: option = sys.argv[1];
except IndexError: raise error.usageError()
if option == '-s':
try:
query = sys.argv[2]
query = re.sub(' ', '+', query)
os.system('open https://www.youtube.com/results?search_query={}'.format(query))
except IndexError: raise error.searchError();
Just to explain; First, the program creates the variable option and then sets it to an empty string. Then, it tries to set option to argument 1, or the option. If argument 1 doesn't exist, it raises the error error.usageError, as explained in The Custom Errors. After that, the program tries to create the variable query, and set it to argument 2, then replace all of the spaces in query with '+' signs. If all of that succeeds to happen, it then loads up the youtube search in your default browser. If not, it raises the error error.searchError().
The Edits
Edit 1. The error was in The Custom Function. Where I should have had an $#, I had an $1. As Jeshurun Roach explains in his answer, $1 only holds the argument 1, and no other arguments. While $# contains all variables.
function youtube() {python3 <python program path here> $#}
$1 refers to the first argument passed into the function. in bash, spaces delimit arguments. so in your example youtube -s Hi this is a test.,
$1 is -s,
$2 is Hi,
$3 is this etc...
What you're looking for is the $# symbol. This value stands for all the arguments.
But just plugging in $# instead of $1 won't fix all your problems. in your python script, each argument will be broken up again by spaces, just like the bash function.
To fix this, you can put quotes around the text after the flag like so: youtube -s 'Hi this is a test.'.
If you call your program like this: youtube -s something cool, then sys.argv[2] is going to be "something".
I'd suggest wrapping your query in quotes. For example youtube -s "something cool".
I'm using a radio sender on my RPi to control some light-devices at home. I'm trying to implement a time control and had successfully used the program "at" in the past.
#!/usr/bin/python
import subprocess as sp
##### some code #####
sp.call(['at', varTime, '<<<', '\"sudo', './codesend', '111111\"'])
When I execute the program, i receive the
errmsg:
syntax error. Last token seen: <
Garbled time
This codesnipped works fine with every command by itself (as long every parameter is from type string).
It's neccessary to call "at" in this way: at 18:25 <<< "sudo ./codesend 111111" to hold the command in the queue (viewable in "atq"),
because sudo ./codesend 111111 | at 18:25 just executes the command directly and writes down the execution in "/var/mail/user".
My question ist, how can I avoid the syntax error.
I'm using a lot of other packages in this program, so I have to stay with Python
I hope someone has a solution for this problem or can help to find my mistake.
Many thanks in advance
Preface: Shared Code
Consider the following context to be part of both branches of this answer.
import subprocess as sp
try:
from shlex import quote # Python 3
except ImportError:
from pipes import quote # Python 2
# given the command you want to schedule, as an array...
cmd = ['sudo', './codesend', '111111']
# ...generate a safely shell-escaped string.
cmd_str = ' '.join(quote(x) for x in cmd))
Solution A: Feed Stdin In Python
<<< is shell syntax. It has no meaning to at, and it's completely normal and expected for at to reject it if given as a literal argument.
You don't need to invoke a shell, though -- you can do the same thing directly from native Python:
p = sp.Popen(['at', vartime], stdin=sp.PIPE)
p.communicate(cmd_str)
Solution B: Explicitly Invoke A Shell
Moreover, <<< isn't /bin/sh syntax -- it's an extension honored in bash, ksh, and others; so you can't reliably get it just by adding the shell=True flag (which uses /bin/sh and so guarantees only POSIX-baseline features). If you want it, you need to explicitly invoke a shell with the feature, like so:
bash_script = '''
at "$1" <<<"$2"
'''
sp.call(['bash', '-c', bash_script,
'_', # this is $0 for that script
vartime, # this is its $1
cmd_str, # this is its $2
])
In either case, note that we're using shlex.quote() or pipes.quote() (as appropriate for our Python release) when generating a shell command from an argument list; this is critical to avoid creating shell injection vulnerabilities in our software.
So i have a script from Python that connects to the client servers then get some data that i need.
Now it will work in this way, my bash script from the client side needs input like the one below and its working this way.
client.exec_command('/apps./tempo.sh' 2016 10 01 02 03))
Now im trying to get the user input from my python script then transfer it to my remotely called bash script and thats where i get my problem. This is what i tried below.
Below is the method i tried that i have no luck working.
import sys
client.exec_command('/apps./tempo.sh', str(sys.argv))
I believe you are using Paramiko - which you should tag or include that info in your question.
The basic problem I think you're having is that you need to include those arguments inside the string, i.e.
client.exec_command('/apps./tempo.sh %s' % str(sys.argv))
otherwise they get applied to the other arguments of exec_command. I think your original example is not quite accurate in how it works;
Just out of interest, have you looked at "fabric" (http://www.fabfile.org ) - this has lots of very handy funcitons like "run" which will run a command on a remote server (or lots of remote servers!) and return you the response.
It also gives you lots of protection by wrapping around popen and paramiko for hte ssh login etcs, so it can be much more secure then trying to make web services or other things.
You should always be wary of injection attacks - Im unclear how you are injecting your variables, but if a user calls your script with something like python runscript "; rm -rf /" that would have very bad problems for you It would instead be better to have 'options' on the command, which are programmed in, limiting the users input drastically, or at least a lot of protection around the input variables. Of course if this is only for you (or trained people), then its a little easier.
I recommend using paramiko for the ssh connection.
import paramiko
ssh_client = paramiko.SSHClient()
ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh_client.connect(server, username=user,password=password)
...
ssh_client.close()
And If you want to simulate a terminal, as if a user was typing:
chan=ssh_client.invoke_shell()
chan.send('PS1="python-ssh:"\n')
def exec_command(cmd):
"""Gets ssh command(s), execute them, and returns the output"""
prompt='python-ssh:' # the command line prompt in the ssh terminal
buff=''
chan.send(str(cmd)+'\n')
while not chan.recv_ready():
time.sleep(1)
while not buff.endswith(prompt):
buff+=ssh_client.chan.recv(1024)
return buff[:len(prompt)]
Example usage: exec_command('pwd')
And the result would even be returned to you via ssh
Assuming that you are using paramiko you need to send the command as a string. It seems that you want to pass the command line arguments passed to your Python script as arguments for the remote command, so try this:
import sys
command = '/apps./tempo.sh'
args = ' '.join(sys.argv[1:]) # all args except the script's name!
client.exec_command('{} {}'.format(command, args))
This will collect all the command line arguments passed to the Python script, except the first argument which is the script's file name, and build a space separated string. This argument string is them concatenated with the bash script command and executed remotely.
I am trying to call a shell (Bash) script from python. The script is in my /home/user/bin directory with execute permission for group & user, i.e., -rwxr-xr--. I am using subprocess.check_call(["/home/user/bin/script.sh %s %s" % (subj,-6)],shell=True) and this is generating an exit status 127 code. Adding stderr=subprocess.STDOUT to the command does nothing to elucidate. Here is the exact output:
CalledProcessError: Command
'['/home/.../bin/MNE_setup_source_space.sh kubi_td104 -6']'
returned non-zero exit status 127`
I believe this might be a PATH related issue, is that correct? I don't know how to resolve this. If I am already passing in the absolute path to the executable how can there be a PATH issue?
Thanks in advance
Do not use shell=True. Do not pass arguments as part of argv[0]. Pass your argument vector as a vector -- which is to say, in Python, a list:
subprocess.check_call(["/home/user/bin/script.sh", str(subj), "-6"])
If you were going to use shell=True, you would do it like so:
subprocess.check_call("/home/user/bin/script.sh %s %s" % (subj,-6), shell=True)
...which is to say, you wouldn't use a list form at all.
To clarify why what you're currently trying is failing -- because you're using shell=True, it's trying to pass only the first list element as a script, and additional arguments as extra argv elements which would only be read or interpreted if the script passed in the first argument chose to look at them (as by referring to "$1", "$2", or the like).
shell=True is only needed in very rare circumstances where you need a shell to perform redirections or logic before starting the program you're trying to run, and comes with serious security concerns if any unvetted input is incorporated into the command being run. Do not use it unless you're very, very sure you need to.