Develop Reference

How to get data from web in python using curl?

In bash when I used
myscript.sh
file="/tmp/vipin/kk.txt"
curl -L "myabcurlx=10&id-11.com" > $file
cat $file
./myscript.sh gives me below output
1,2,33abc
2,54fdd,fddg3
3,fffff,gfr54
When I tried to fetch it using python and tried below code -
mypython.py
command = curl + ' -L ' + 'myabcurlx=10&id-11.com'
output = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE).stdout.read().decode('ascii')
print(output)
python mypython.py throw error, Can you please point out what is wrong with my code.
Error :
/bin/sh: line 1: &id=11: command not found
Wrong Parameter

command = curl + ' -L ' + 'myabcurlx=10&id-11.com'
Print out what this string is, or just think about it. Assuming that curl is the string 'curl' or '/usr/bin/curl' or something, you get:
curl -L myabcurlx=10&id-11.com
That’s obviously not the same thing you typed at the shell. Most importantly, that last argument is not quoted, and it has a & in the middle of it, which means that what you’re actually asking it to do is to run curl in the background and then run some other program that doesn’t exist, as if you’d done this:
curl -L myabcurlx=10 &
id-11.com
Obviously you could manually include quotes in the string:
command = curl + ' -L ' + '"myabcurlx=10&id-11.com"'
… but that won’t work if the string is, say, a variable rather than a literal in your source—especially if that variable might have quote characters within it.
The shlex module has helpers to quoting things properly.
But the easiest thing to do is just not try to build a command line in the first place. You aren’t using any shell features here, so why add the extra headaches, performance costs, problems with the shell getting in the way of your output and retcode, and possible security issues for no benefit?
Make the arguments a list rather than a string:
command = [curl, '-L', 'myabcurlx=10&id-11.com']
… and leave off the shell=True
And it just works. No need to get spaces and quotes and escapes right.
Well, it still won’t work, because Popen doesn’t return output, it’s a constructor for a Popen object. But that’s a whole separate problem—which should be easy to solve if you read the docs.
But for this case, an even better solution is to use the Python bindings to libcurl instead of calling the command-line tool. Or, even better, since you’re not using any of the complicated features of curl in the first place, just use requests to make the same request. Either way, you get a response object as a Python object with useful attributes like text and headers and request.headers that you can’t get from a command line tool except by parsing its output as a giant string.

import subprocess
fileName="/tmp/vipin/kk.txt"
with open(fileName,"w") as f:
subprocess.read(["curl","-L","myabcurlx=10&id-11.com"],stdout=f)
print(fileName)
recommended approaches:
https://docs.python.org/3.7/library/urllib.request.html#examples
http://docs.python-requests.org/en/master/user/install/

Escaping a sed expression to execute over ssh from Python

I have a file on a remote server that needs to have a line updated in it. I am trying to do the update via python but appear to be having a character escape issue.
The line in the file I am trying to update is:
BEGRCVDDATE=02/01/2018 00:00 am
The line of code in python script I am using to try and make the update:
os.popen('ssh %s sed -i s/'BEGRCVDDATE=[0-9][0-9]\/[0-9][0-9]\/[0-9][0-9][0-9][0-9] [0-9][0-9]:[0-9][0-9] [a-z][a-z]'/'BEGRCVDDATE=%s'/ %s' % (ip, rcvdate, file_path))
The above code is throwing the following error:
sed: -e expression #1, char 37: unknown option to `s'
I am however able to run the sed command outside of the python script which leads me to believe this is a character escaping issue. Below works outside of the python script.
ssh <ip> "sed -i s/'BEGRCVDDATE=[0-9][0-9]\/[0-9][0-9]\/[0-9][0-9][0-9][0-9] [0-9][0-9]:[0-9][0-9] [a-z][a-z]'/'BEGRCVDDATE=BEGRCVDDATE=03\\/08\\/2018 00:00 pm'/ /tmp/test.txt"
I have tried various combinations of quotes and back-slashes to try and get around the issue I am seeing without success.
If anyone can help me resolve the issue I am seeing it would be much appreciated.
Note: due to python version in my environment os.popen is being using instead of subprocess.

import subprocess, pipes
# Generate your list of arguments *as a list of Python strings*
rcvdate = '02/01/2018 00:00 am'
cmd=['sed', '-i',
's#BEGRCVDDATE=[0-9][0-9]/[0-9][0-9]/[0-9][0-9][0-9][0-9] [0-9][0-9]:[0-9][0-9] [a-z][a-z]#BEGRCVDDATE=%s#' % (rcvdate,),
file_path,
]
# Ask Python itself to correctly form a shell command from that list
cmd_str = ' '.join([pipes.quote(s) for s in cmd])
# Pass that shell command as an argument to `ssh`.
subprocess.call(['ssh', ip, cmd_str])
...or, if you truly don't have the subprocess module, you'll need a second round of shell escaping:
ssh_cmd = ['ssh', ip, cmd_str]
ssh_cmd_str = ' '.join([pipes.quote(s) for s in ssh_cmd])
os.system(ssh_cmd_str)
Note that I changed your sed expression to use #, not /, as a sigil. This means that literal backslashes are no longer needed.

How can I get raw string when I run a command that includes color from bash script

I'm running a heroku-cli command as follows:
heroku run python manage.py my_command -a my_app_id
It outputs a string, e.g. "abcdefgh".
However, when I run it from a bash script and echo the output to a file:
var=`heroku run python manage.py my_command -a my_app_id` && echo $var > test.txt
the file ends up containing the following:
^[[?25l^[[?25habcdefgh ^[[?25h
I'm not quite sure what these extra characters are - possibly color code or reset code?
Any idea how I can get the raw string. I've tried many things.

The ANSI codes you are seeing seem to consist of octal 33, a square bracket, a question mark, an integer and a letter. You might try something like
sed 's/\o33\[\?[0-9]\{2\}[hl]//g'
See also: https://en.wikipedia.org/wiki/ANSI_escape_code

python -c vs python -<< heredoc

I am trying to run some piece of Python code in a Bash script, so i wanted to understand what is the difference between:
#!/bin/bash
#your bash code
python -c "
#your py code
"
vs
python - <<DOC
#your py code
DOC
I checked the web but couldn't compile the bits around the topic. Do you think one is better over the other?
If you wanted to return a value from Python code block to your Bash script then is a heredoc the only way?

The main flaw of using a here document is that the script's standard input will be the here document. So if you have a script which wants to process its standard input, python -c is pretty much your only option.
On the other hand, using python -c '...' ties up the single-quote for the shell's needs, so you can only use double-quoted strings in your Python script; using double-quotes instead to protect the script from the shell introduces additional problems (strings in double-quotes undergo various substitutions, whereas single-quoted strings are literal in the shell).
As an aside, notice that you probably want to single-quote the here-doc delimiter, too, otherwise the Python script is subject to similar substitutions.
python - <<'____HERE'
print("""Look, we can have double quotes!""")
print('And single quotes! And `back ticks`!')
print("$(and what looks to the shell like process substitutions and $variables!)")
____HERE
As an alternative, escaping the delimiter works identically, if you prefer that (python - <<\____HERE)

If you are using bash, you can avoid heredoc problems if you apply a little bit more of boilerplate:
python <(cat <<EoF
name = input()
print(f'hello, {name}!')
EoF
)
This will let you run your embedded Python script without you giving up the standard input. The overhead is mostly the same of using cmda | cmdb. This technique is known as Process Substitution.
If want to be able to somehow validate the script, I suggest that you dump it to a temporary file:
#!/bin/bash
temp_file=$(mktemp my_generated_python_script.XXXXXX.py)
cat > $temp_file <<EoF
# embedded python script
EoF
python3 $temp_file && rm $temp_file
This will keep the script if it fails to run.

If you prefer to use python -c '...' without having to escape with the double-quotes you can first load the code in a bash variable using here-documents:
read -r -d '' CMD << '--END'
print ("'quoted'")
--END
python -c "$CMD"
The python code is loaded verbatim into the CMD variable and there's no need to escape double quotes.

How to use here-docs with input
tripleee's answer has all the details, but there's Unix tricks to work around this limitation:
So if you have a script which wants to process its standard input, python -c is pretty much your only option.
This trick applies to all programs that want to read from a redirected stdin (e.g., ./script.py < myinputs) and also take user input:
python - <<'____HERE'
import os
os.dup2(1, 0)
print(input("--> "))
____HERE
Running this works:
$ bash heredocpy.sh
--> Hello World!
Hello World!
If you want to get the original stdin, run os.dup(0) first. Here is a real-world example.
This works because as long as either stdout or stderr are a tty, one can read from them as well as write to them. (Otherwise, you could just open /dev/tty. This is what less does.)
In case you want to process inputs from a file instead, that's possible too -- you just have to use a new fd:
Example with a file
cat <<'____HERE' > file.txt
With software there are only two possibilites:
either the users control the programme
or the programme controls the users.
____HERE
python - <<'____HERE' 4< file.txt
import os
for line in os.fdopen(4):
print(line.rstrip().upper())
____HERE
Example with a command
Unfortunately, pipelines don't work here -- but process substitution does:
python - <<'____HERE' 4< <(fortune)
import os
for line in os.fdopen(4):
print(line.rstrip().upper())
____HERE

Parsing inline python command via linux using pdsh

So, I am trying to issue this command from a python script that collects cpu information across a predetermined number of nodes in a cluster. Here I use a fanout of 2 and only run it on nodes b127 through b129 for testing purposes.
pdsh -f2 -w b[127-129] 'python -c "import multiprocessing
num_cpu = multiprocessing.cpu_count()
stat_fd = open('/proc/stat')
stat_fd.close()"'
I printed the command and this is what it shows on the terminal. Thus, telling me that the quotes and commands are properly formatted. I get this string by executing the following code:
python_command = "'python -c "\
+ "\"import multiprocessing\n"\
+ "num_cpu = multiprocessing.cpu_count()\n"\
+ "stat_fd = open(\'/proc/stat\')\n"\
+ "stat_fd.close()\"'"
command = "pdsh -f2 -w b[127-129] " + python_command
print command
Unfortunately, the line with open(\'/proc/stat\') seems to be the problem as that is the only line that causes the parser to fail due to the nested single quotes. I've tried numerous combinations of quoting and escaping in order to make it work to no avail. I've omitted some code between the opening and closing of the file to minimize the posted code.
I searched around and found this link, but found it was too simple of an example because I could replicate those commands. And yes, I know I can use bash commands to get what I want done and I may end up doing so, but this one has me beating my head on the wall. I also have scripts that gather data using top and ps so I don't need an explanation using them. I greatly appreciate the help in advance.

Try this:
python_command = """pdsh -f2 -w b[127-129] 'python -c "import multiprocessing
num_cpu = multiprocessing.cpu_count()
stat_fd = open(\\"/proc/stat\\")
stat_fd.close()"'"""
In Python, you can use triple quotes ("""...""" or '''...''') for strings containing new lines and single/double quotes.
The last level of quotes (on the open() line) will need to be escaped so that they don't conflict with outer quotes. You also need to escape the backslashes so they aren't immediately consumed by Python when interpreting the string: \\".