Develop Reference

How to get data from web in python using curl?

In bash when I used
myscript.sh
file="/tmp/vipin/kk.txt"
curl -L "myabcurlx=10&id-11.com" > $file
cat $file
./myscript.sh gives me below output
1,2,33abc
2,54fdd,fddg3
3,fffff,gfr54
When I tried to fetch it using python and tried below code -
mypython.py
command = curl + ' -L ' + 'myabcurlx=10&id-11.com'
output = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE).stdout.read().decode('ascii')
print(output)
python mypython.py throw error, Can you please point out what is wrong with my code.
Error :
/bin/sh: line 1: &id=11: command not found
Wrong Parameter

command = curl + ' -L ' + 'myabcurlx=10&id-11.com'
Print out what this string is, or just think about it. Assuming that curl is the string 'curl' or '/usr/bin/curl' or something, you get:
curl -L myabcurlx=10&id-11.com
That’s obviously not the same thing you typed at the shell. Most importantly, that last argument is not quoted, and it has a & in the middle of it, which means that what you’re actually asking it to do is to run curl in the background and then run some other program that doesn’t exist, as if you’d done this:
curl -L myabcurlx=10 &
id-11.com
Obviously you could manually include quotes in the string:
command = curl + ' -L ' + '"myabcurlx=10&id-11.com"'
… but that won’t work if the string is, say, a variable rather than a literal in your source—especially if that variable might have quote characters within it.
The shlex module has helpers to quoting things properly.
But the easiest thing to do is just not try to build a command line in the first place. You aren’t using any shell features here, so why add the extra headaches, performance costs, problems with the shell getting in the way of your output and retcode, and possible security issues for no benefit?
Make the arguments a list rather than a string:
command = [curl, '-L', 'myabcurlx=10&id-11.com']
… and leave off the shell=True
And it just works. No need to get spaces and quotes and escapes right.
Well, it still won’t work, because Popen doesn’t return output, it’s a constructor for a Popen object. But that’s a whole separate problem—which should be easy to solve if you read the docs.
But for this case, an even better solution is to use the Python bindings to libcurl instead of calling the command-line tool. Or, even better, since you’re not using any of the complicated features of curl in the first place, just use requests to make the same request. Either way, you get a response object as a Python object with useful attributes like text and headers and request.headers that you can’t get from a command line tool except by parsing its output as a giant string.

import subprocess
fileName="/tmp/vipin/kk.txt"
with open(fileName,"w") as f:
subprocess.read(["curl","-L","myabcurlx=10&id-11.com"],stdout=f)
print(fileName)
recommended approaches:
https://docs.python.org/3.7/library/urllib.request.html#examples
http://docs.python-requests.org/en/master/user/install/

Python script doesn't delete file from archive -- printed command via terminal works fine

I'm creating an archive in Python using this code:
#Creates archive using string like [proxy_16-08-15_08.57.07.tar]
proxyArchiveLabel = 'proxy_%s' % EXECUTION_START_TIME + '.tar'
log.info('Packaging %s ...' % proxyArchiveLabel)
#Removes .tar from label during creation
shutil.make_archive(proxyArchiveLabel.rsplit('.',1)[0], 'tar', verbose=True)
So this creates an archive fine in the local directory. The problem is, there's a specific directory in my archive I want to remove, due to it's size and lack of necessity for this task.
ExecWithLogging('tar -vf %s --delete ./roles/jobs/*' % proxyArchiveLabel)
# ------------
def ExecWithLogging(cmd):
print cmd
p = subprocess.Popen(cmd.split(' '), env=os.environ, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while(True):
log.info(p.stdout.readline().strip())
if(p.poll() is not None):
break
However, this seems to do basically nothing. The size remains the same. If I print cmd inside of the ExecWithLogging, and copy/past that command to a terminal in the working directory of the script, it works fine. Just to be sure, I also tried hard-coding the full path to where the archive is created as part of the tar -vf %s --delete command, but still nothing seemed to happen.
I do get this output in my INFO log: tar: Pattern matching characters used in file names, so I'm kind of thinking Popen is interpreting my command incorrectly somehow... (or rather, I'm more likely passing in something incorrectly).
Am I doing something wrong? What else can I try?

You may have to use the --wildcards option in the tar command, which enables pattern matching. This may well be what you are seeing in your log, be it somewhat cryptically.
Edit:
In answer to your question Why? I suspect that the shell is performing the wildcard expansion whilst the command proffered through Popen is not. The --wildcard option for tar, forces tar to perform the wildcard expansion.
For a more detailed explanation see here:
Tar and wildcards

Problems executing commands with (') character in python

I'm doing a program that copies some files from a music playlist. I execute the command just like this:
command = 'cp "%s" "%s"' % (songPath,plPath)
os.system(command)
The problem is that when I execute that if the song's path has a ' character command can not be executed. It says:
cp: cannot stat `/home/myname/Music/Oasis/(What\'s The Story) Morning Glory/03 Wonderwall.mp3': No such file or directory
I checked the songPath and has no \ character before the '
Does anyone know how to avoid the program adding that \ character?
Thank you in advance!

Use subprocess.call instead:
ret_val = subprocess.call(['cp',songPath,plPath])
This avoids the shell so your arguments should be passed the cp in the exact form that you gave them.

os.system commands

I am developing a program which needs to use os.system because of the old Python limitations. Currently I'm stuck in one small spot.
os.system("C:\\FIOCheck\\xutil.exe -i get phy" +HBEA + ">C:\\FIOCheck\\HBEAResult.txt")
This is the piece of code I am trying to work through. It will access an external program, which has some parameters. HBEA is the variable I am trying to pass (which is received earlier in the program). The program then takes whatever the .exe created and pipes it to an external file. At this point, the variable HBEA is not being passed to the command line, so the .exe never runs, which causes the .txt to be blank. Since the file is blank, I cannot grab data from it and therefore cannot complete the program.
Any ideas?
EDIT:
So I attempted the following code per some suggestions:
cmd = "C:\\FIOCheck\\xutil.exe -i get phy " +HBEA + ">C:\\FIOCheck\\HBEAResult.txt"
print cmd
os.system(cmd)
The following output was generated:
50012BE00004BDFF #HBEA variable
C:\FIOCheck\xutil.exe -i get phy 50012BE00004BDFF>C:\FIOCheck\HBEAResult.txt #the cmd var
However this still isn't passing the value through. Is the HBEA variable too long?
SOLVED
This program worked with some editing from the best answer. The commands were passing correctly, however the way I formatted it was not correct. The new formatting looks like:
cmd = "C:\\FIOCheck\\xutil.exe -i " + HBEA + " get ver" + ">C:\\FIOCheck\\HBEAResult.txt"
os.system(cmd)
Thanks for the help!

os.system("C:\\FIOCheck\\xutil.exe -i get phy" +HBEA + ">C:\\FIOCheck\\HBEAResult.txt")
should that be
os.system("C:\\FIOCheck\\xutil.exe -i get phy " +HBEA + ">C:\\FIOCheck\\HBEAResult.txt")
and you can always build the string first
cmd = "C:\\FIOCheck\\xutil.exe -i get phy " +HBEA + ">C:\\FIOCheck\\HBEAResult.txt"
print cmd
os.system(cmd)

Python not outputing to file when script is run in vb program

I am making a vb text editor that can run scripting code. I have successfully gotten it to run the code and show up but I would like to be able to have the output from the the code redirected to a file without having to do anything to the python code. Is that possible?
This is the code I'm using to try it but it does not write anything to the file:
Shell(compiler & " """ & fileName & " "" > C:\output.txt")
the compiler is the location of python.exe in the python install folder and the file name is the file I'm running.
The problem could also be in the way I am trying to do it with the shell command.

After some more research into the the matter I found a way to make it work.
Here is the working code:
Shell("cmd.exe /c " & compiler & " """ & fileName & " "" > ""C:\output.txt"" ", vbNormalFocus)
So I am actually running the line through the command prompt which is being called by the shell function. My assumption was that shell function was using the command prompt to execute the string argument but I guess I was wrong. Also, just to note, the output file name is also surrounded by quotes which is different than my original post.