I have a file on a remote server that needs to have a line updated in it. I am trying to do the update via python but appear to be having a character escape issue.
The line in the file I am trying to update is:
BEGRCVDDATE=02/01/2018 00:00 am
The line of code in python script I am using to try and make the update:
os.popen('ssh %s sed -i s/'BEGRCVDDATE=[0-9][0-9]\/[0-9][0-9]\/[0-9][0-9][0-9][0-9] [0-9][0-9]:[0-9][0-9] [a-z][a-z]'/'BEGRCVDDATE=%s'/ %s' % (ip, rcvdate, file_path))
The above code is throwing the following error:
sed: -e expression #1, char 37: unknown option to `s'
I am however able to run the sed command outside of the python script which leads me to believe this is a character escaping issue. Below works outside of the python script.
ssh <ip> "sed -i s/'BEGRCVDDATE=[0-9][0-9]\/[0-9][0-9]\/[0-9][0-9][0-9][0-9] [0-9][0-9]:[0-9][0-9] [a-z][a-z]'/'BEGRCVDDATE=BEGRCVDDATE=03\\/08\\/2018 00:00 pm'/ /tmp/test.txt"
I have tried various combinations of quotes and back-slashes to try and get around the issue I am seeing without success.
If anyone can help me resolve the issue I am seeing it would be much appreciated.
Note: due to python version in my environment os.popen is being using instead of subprocess.
import subprocess, pipes
# Generate your list of arguments *as a list of Python strings*
rcvdate = '02/01/2018 00:00 am'
cmd=['sed', '-i',
's#BEGRCVDDATE=[0-9][0-9]/[0-9][0-9]/[0-9][0-9][0-9][0-9] [0-9][0-9]:[0-9][0-9] [a-z][a-z]#BEGRCVDDATE=%s#' % (rcvdate,),
file_path,
]
# Ask Python itself to correctly form a shell command from that list
cmd_str = ' '.join([pipes.quote(s) for s in cmd])
# Pass that shell command as an argument to `ssh`.
subprocess.call(['ssh', ip, cmd_str])
...or, if you truly don't have the subprocess module, you'll need a second round of shell escaping:
ssh_cmd = ['ssh', ip, cmd_str]
ssh_cmd_str = ' '.join([pipes.quote(s) for s in ssh_cmd])
os.system(ssh_cmd_str)
Note that I changed your sed expression to use #, not /, as a sigil. This means that literal backslashes are no longer needed.
Related
In bash when I used
myscript.sh
file="/tmp/vipin/kk.txt"
curl -L "myabcurlx=10&id-11.com" > $file
cat $file
./myscript.sh gives me below output
1,2,33abc
2,54fdd,fddg3
3,fffff,gfr54
When I tried to fetch it using python and tried below code -
mypython.py
command = curl + ' -L ' + 'myabcurlx=10&id-11.com'
output = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE).stdout.read().decode('ascii')
print(output)
python mypython.py throw error, Can you please point out what is wrong with my code.
Error :
/bin/sh: line 1: &id=11: command not found
Wrong Parameter
command = curl + ' -L ' + 'myabcurlx=10&id-11.com'
Print out what this string is, or just think about it. Assuming that curl is the string 'curl' or '/usr/bin/curl' or something, you get:
curl -L myabcurlx=10&id-11.com
That’s obviously not the same thing you typed at the shell. Most importantly, that last argument is not quoted, and it has a & in the middle of it, which means that what you’re actually asking it to do is to run curl in the background and then run some other program that doesn’t exist, as if you’d done this:
curl -L myabcurlx=10 &
id-11.com
Obviously you could manually include quotes in the string:
command = curl + ' -L ' + '"myabcurlx=10&id-11.com"'
… but that won’t work if the string is, say, a variable rather than a literal in your source—especially if that variable might have quote characters within it.
The shlex module has helpers to quoting things properly.
But the easiest thing to do is just not try to build a command line in the first place. You aren’t using any shell features here, so why add the extra headaches, performance costs, problems with the shell getting in the way of your output and retcode, and possible security issues for no benefit?
Make the arguments a list rather than a string:
command = [curl, '-L', 'myabcurlx=10&id-11.com']
… and leave off the shell=True
And it just works. No need to get spaces and quotes and escapes right.
Well, it still won’t work, because Popen doesn’t return output, it’s a constructor for a Popen object. But that’s a whole separate problem—which should be easy to solve if you read the docs.
But for this case, an even better solution is to use the Python bindings to libcurl instead of calling the command-line tool. Or, even better, since you’re not using any of the complicated features of curl in the first place, just use requests to make the same request. Either way, you get a response object as a Python object with useful attributes like text and headers and request.headers that you can’t get from a command line tool except by parsing its output as a giant string.
import subprocess
fileName="/tmp/vipin/kk.txt"
with open(fileName,"w") as f:
subprocess.read(["curl","-L","myabcurlx=10&id-11.com"],stdout=f)
print(fileName)
recommended approaches:
https://docs.python.org/3.7/library/urllib.request.html#examples
http://docs.python-requests.org/en/master/user/install/
I am using Vim (8.0) and tmux (2.3) together in the following way: In a tmux session I have a window split to 2 panes, one pane has some text file open in Vim, the other pane has some program to which I want to send lines of text. A typical use case is sending lines from a Python script to IPython session running in the other pane.
I am doing this by a Vim script which uses python, code snippet example (assuming the target tmux pane is 0):
py import vim
python << endpython
cmd = "print 1+2"
vim_cmd = "silent !tmux send-keys -t 0 -l \"" + cmd + "\"" # -l for literal
vim.command(vim_cmd)
endpython
This works well, except for when cmd has some characters which has to be escaped, like %, #, $, etc. The cmd variable is read from the current line in the text file opened in Vim, so I can do something like cmd = cmd.replace('%', '\%') etc., but this has 2 disadvantages: first, I don't know all the vim characters which have to be escaped, so it has been trial and error up until now, and second, the characters " is not escaped properly - that is in the string Vim gets, the " just disappears, even if I do cmd = cmd.replace('"', '\"').
So, is there a general way to tell Vim to not interpret anything, just get a raw string and send it as is? If not, why is the " not escaped properly?
Vimscript
You're looking for the shellescape() function. If you use the :! Vim command, the {special} argument needs to be 1.
silent execute '!tmux send-keys -t 0 -l ' . shellescape(cmd, 1)
But as you're not interested in (displaying) the shell output, and do not need to interact with the launched script, I would switch from :! to the lower-level system() command.
call system('tmux send-keys -t 0 -l ' . shellescape(cmd))
Python
The use of Python (instead of pure Vimscript) doesn't have any benefits (at least in the small snippet in your question). Instead, if you embed the Python cmd variable in a Vimscript expression, now you also need a Python function to escape the value as a Vimscript string (something like '%s'" % str(cmd).replace("'", "''"))). Alternatively, you could maintain the value in a Vim variable, and access it from Python via vim.vars.
I am running a Python script which takes the dump of CSVs from a Postgres database and then I want to escape double quotes in all these files. So I am using sed to do so.
In my Python code:
sed_for_quotes = 'sed -i s/\\"//g /home/ubuntu/PTOR/csvdata1/'+table+'.csv'
subprocess.call(sed_for_quotes, shell=True)
The process completes without any error, but when I load these tables to Redshift, I get error No delimiter found and upon checking the CSV, I find that one of the rows is only half-loaded,for example if it is a timestamp column, then only half of it is loaded, and there is no data after that in the table (while the actual CSV has that data before running sed). And that leads to the No delimiter found error.
But when I run sed -i s/\"//g filename.csvon these files in the shell it works fine and the csv after running sed has all the rows. I have checked that there is no problem with the data in the files.
What is the reason for this not working in a Python program ? I have also tried using sed -i.bak in the python program but that makes no difference.
Please Note that I am using an extra backslash(\) in the Python code because I need to escape the other backslash.
Other approaches tried:
Using subprocess.Popen without any buffer size and with positive buffer size, but that didn't help
Using subprocess.Popen(sed_for_quotes,bufsize=-4096) (negative buffer size) worked for one of
the files which was giving the error, but then encountered the same
problem in another file.
Do not use intermediate shell when you do not need to. And check for return code of the subprocess to make sure it completed successfully (check_call does this for you)
path_to_file = ... # e.g. '/home/ubuntu/PTOR/csvdata1/' + table + '.csv'
subprocess.check_call(['sed', '-i', 's/"//g', path_to_file])
By "intermediate" shell I mean the shell process run by subprocess that parses the command (± splits by whitespace but not only) and runs it (runs sed in this example). Since you precisely know what arguments sed should be invoked with, you do not need all this and it's best to avoid that.
Put your sed into a shell script, e.g.
#!/bin/bash
# Parameter $1 = Filename
sed -i 's/"//g' "$1"
Call your shell script using subprocess:
sed_for_quotes = 'my_sed_script /home/ubuntu/PTOR/csvdata1/'+table+'.csv'
Use docs.python.org/3.6: shlex.split
shlex.split(s, comments=False, posix=True)
Split the string s using shell-like syntax.
I'm unable to execute the following line:
os.system("timeout 1s bash -c \"ffmpeg -i \""+path+\"+" | <some_<other_cmd>\"")
So the purpose of this command is to set a timeout for the whole command, i.e. pipelining some ffmpeg information from a path.
The problem is because bash -c "CMD" is expected, but the command also contains " ".
Is there another way of defining the \"path\", because the path can contain spaces? Or another solution which can resolve my problem?
Thanks in advance!
Triple sinqle quotes can do the trick (so that you do not have to escape doublequotes):
os.system('''timeout 1s bash -c "ffmpeg -i "+path+"+" | cat''')
But in general.. Why not use subprocess.call that has saner syntax?
Has answered similar question in other posts: 1 and 2
you can use subprocess related functions, which all support timeout parameter, to replace os.system
such as subprocess.check_output
ffmpegCmd = "ffmpeg -I %s | %s" % (path, someOtherCmd)
outputBytes = subprocess.check_output(ffmpegCmd, shell=True, timeout=1)
outputStr = outputBytes.decode("utf-8") # change utf-8 to your console encoding if necessary
So, I am trying to issue this command from a python script that collects cpu information across a predetermined number of nodes in a cluster. Here I use a fanout of 2 and only run it on nodes b127 through b129 for testing purposes.
pdsh -f2 -w b[127-129] 'python -c "import multiprocessing
num_cpu = multiprocessing.cpu_count()
stat_fd = open('/proc/stat')
stat_fd.close()"'
I printed the command and this is what it shows on the terminal. Thus, telling me that the quotes and commands are properly formatted. I get this string by executing the following code:
python_command = "'python -c "\
+ "\"import multiprocessing\n"\
+ "num_cpu = multiprocessing.cpu_count()\n"\
+ "stat_fd = open(\'/proc/stat\')\n"\
+ "stat_fd.close()\"'"
command = "pdsh -f2 -w b[127-129] " + python_command
print command
Unfortunately, the line with open(\'/proc/stat\') seems to be the problem as that is the only line that causes the parser to fail due to the nested single quotes. I've tried numerous combinations of quoting and escaping in order to make it work to no avail. I've omitted some code between the opening and closing of the file to minimize the posted code.
I searched around and found this link, but found it was too simple of an example because I could replicate those commands. And yes, I know I can use bash commands to get what I want done and I may end up doing so, but this one has me beating my head on the wall. I also have scripts that gather data using top and ps so I don't need an explanation using them. I greatly appreciate the help in advance.
Try this:
python_command = """pdsh -f2 -w b[127-129] 'python -c "import multiprocessing
num_cpu = multiprocessing.cpu_count()
stat_fd = open(\\"/proc/stat\\")
stat_fd.close()"'"""
In Python, you can use triple quotes ("""...""" or '''...''') for strings containing new lines and single/double quotes.
The last level of quotes (on the open() line) will need to be escaped so that they don't conflict with outer quotes. You also need to escape the backslashes so they aren't immediately consumed by Python when interpreting the string: \\".