Currently, I'm trying to convert CURL request to Python script.
curl $(curl -u username:password -s https://api.example.com/v1.1/reports/11111?fields=download | jq ".report.download" -r) > "C:\sample.zip"
I have tried pycurl, with no success, due to knowledge limitation.
As a solution, I have found, that it is possible to run commands through python.
https://www.raspberrypi.org/forums/viewtopic.php?t=112351
import os
os.system("curl -K.........")
And other solution ( based on the search more common) using subprocess
import subprocess
subprocess.call(['command', 'argument'])
Currently, I'm not sure where to move and how to adapt this solution to my sitionation.
import os
os.system("curl $(curl -u username:password -s https://api.example.com/v1.1/reports/11111?fields=download | jq '.report.download' -r) > 'C:\sample.zip'")
'curl' is not recognized as an internal or external command,
operable program or batch file.
255
P.S. - Update v1
Any suggestion?
import requests
response = requests.get('https://api.example.com/v1.1/reports/11111?fields=download | jq ".report.download" -r', auth=('username', 'password'))
This work without "| jq ".report.download" this part, but this is the main part which gives at the end only link to download the file.
ANy way around it?
The error 'curl' is not recognized as an internal or external command means that python couldn't find the location where curl is installed. If you have already installed curl, try giving the full path to where curl is installed. For example, if curl.exe is located in C:\System32, the try
import os
os.system("C:\System32\curl $(curl -u username:password -s https://api.example.com/v1.1/reports/11111?fields=download | jq '.report.download' -r) > 'C:\sample.zip'")
But thats definitely not pythonic way of doing things. I would instead suggest to use requests module.
You need to invoke requests module twice for this, first to download the json content from https://api.example.com/v1.1/reports/11111?fields=download, get a new url pointed byreport.download and then invoke requests again to download data from the new url.
Something along these lines should get you going
import requests
url = 'https://api.example.com/v1.1/reports/11111'
response = requests.get(url, params=(('fields', 'download'),),
auth=('username', 'password'))
report_url = response.json()['report']['download']
data = requests.get(report_url).content
with open('C:\sample.zip', 'w') as f:
f.write(data)
You can use this site to convert the actual curl part of your command to something that works with requests: https://curl.trillworks.com/
From there, just use the .json() method of the request object to do whatever processing you need to be doing.
Finally, can save like so:
import json
with open('C:\sample.zip', 'r') as f:
json.dump(data, f)
Related
I need to create AWS lambda function to execute the python program.
I need to incorporate the following shell command in it.
curl https://ip-ranges.amazonaws.com/ip-ranges.json | jq -r '.prefixes[] | select(.region=="ap-southeast-1") | .ip_prefix'
Could someone guide me on this.
To simply shell out to curl and jq to get that data,
import subprocess
data = subprocess.check_output("""curl https://ip-ranges.amazonaws.com/ip-ranges.json | jq -r '.prefixes[] | select(.region=="ap-southeast-1") | .ip_prefix'""", shell=True)
but you really probably shouldn't do that since e.g. there's no guarantee you have curl and jq in the Lambda execution environment (not to mention the overhead).
Instead, if you have the requests library,
import requests
resp = requests.get("https://ip-ranges.amazonaws.com/ip-ranges.json")
resp.raise_for_status()
prefixes = {
r["ip_prefix"]
for r in resp.json()["prefixes"]
if r["region"] == "ap-southeast-1"
}
When I execute this [say filename as curl.py and execute by python curl.py]
import subprocess
import json
subprocess.call([
'curl',
'--digest',
'--user',
'user:pass',
'https://url'],
)
The output is a JSON file in my case.
I see the output on my terminal, but I want it to be stored in an object in the same python file.
How can I achieve this?
As per the comments, here's an alternative without curl. I am using requests here (pip install requests):
import requests
url = 'http://httpbin.org/digest-auth/auth/user/pass'
r = requests.get(url, auth=requests.auth.HTTPDigestAuth('user', 'pass'))
print(r.json()) # no need to unpack JSON manually!
I want to execute a curl with command in python.
Usually, I just need enter the command in terminal and press return key.
The command shows below:
curl -H "`oauth2l header --json key.json mobileinsights`" https://mobileinsights.googleapis.com/v2/networks
The result is in json format.
Use the subprocess module to run your shell command.
import subprocess
result = subprocess.check_output('curl -H "`oauth2l header --json
key.json mobileinsights`" https://mobileinsights.googleapis.com/v2/networks', shell=True)
Then, use the json module to parse the JSON data returned by the server.
import json
result_json = json.loads(result)
You may load an OS and JSON load modules and then run
os.execute(curl URL) store it in any variable then convert it into JSON format with JSON load module
I am trying to download a file using python, imitating the same behavior as this curl command:
curl ftp://username:password#example.com \
--retry 999 \
--retry-max-time 0
-o 'target.txt' -C -
How would this look in python ?
Things I have looked into:
Requests : no ftp support
Python-wget: no download resume support
requests-ftp : no download resume support
fileDownloader : broken(?)
I am guessing one would need to build this from scratch and go low level with pycurl or urllib2 or something similar.
I am trying to create this script in python and I feel lost.. Should I just call curl from python subprocess ?
Any point to the write direction would be much appreciated
you can use python's inbuilt ftplib
Here is the code:
from ftplib import FTP
ftp = FTP('example.com', 'username', 'password') #logs in
ftp.retrlines() # to see the list of files and directories ftp.cwd('to change to any directory')
ftp.retrbinary('RETR filename', open('Desktop\filename', 'wb').write) # start downloading
ftp.close() # close the connection
Auto resume is supported. I even tried turning off my wifi and checked if the download is resuming.
You can refer to /Python27/Lib/ftplib.py for default GLOBAL_TIME_OUT settings.
there is this library for downloading files from ftp server
fileDownloader.py
to download the file
downloader = fileDownloader.DownloadFile(‘http://example.com/file.zip’, “C:UsersusernameDownloadsnewfilename.zip”, (‘username’,’password’))
downloader.download()
to resume download
downloader = fileDownloader.DownloadFile(‘http://example.com/file.zip’, “C:UsersusernameDownloadsnewfilename.zip”, (‘username’,’password’))
downloader.resume()
I've tried using the following commands to download the ctrl alt del comics.
$ for filename in $(seq 20021023 20100503); do wget http://www.ctrlaltdel-online.com/comics/"$filename".jpg; done
I get the following error code, bash: syntax error near unexpected token 'do'
I've also tried using cURL, using this command,
curl http://ctrlaltdel-online.com/comics[20021023..20100503].jpg
I get the following error code, curl: (3) [globbing] error: bad range specification after pos 37
What's wrong, and how can I fix it?
As msw pointed out, crawling a site could be either illegal, unethical, irritating to the author, or perfectly fine. Please use your scripting powers responsibly and for Good (tm). Asking permission would certainly be a nice thing to do.
Note that the ctrlaltdel-online.com web server seems to return HTTP 403 forbidden to wget with the normal wget User-Agent string. Emulating something Firefox-ish seems to bypass that (although I bet they are just explicitly denying wget, which indicates they most likely forbid this type of access).
USERAGENT='Mozilla/5.0 Firefox/3.6.3'
for DAYS in $(seq 365)
do
NEXT=`date -d "${DAYS} days ago" +%Y%m%d`
wget -U "${USERAGENT}" "http://www.cad-comic.com/comics/cad/${NEXT}.jpg"
done
Replace 365 with a larger number to go back more than a year. The wget output might be annoying, so you can pass it -q to make it quiet.
i was writing the same script. Here it is.
import sys
import re
import urllib
import os
import ctypes
from urllib import FancyURLopener
class MyOpener(FancyURLopener):
version = 'Mozilla/5.0 (Windows; U; Windows NT 5.1; it;rv:1.8.1.11)Gecko/20071127 Firefox/2.0.0.11'
def getlinks(add,m,opener):
ufile=opener.open(add)
html=ufile.read()
dates=re.findall('href="/cad/(\d+)">',html)
links=[]
for date in dates:
if date[4:6]==m:
links.append('http://www.cad-comic.com/cad/'+date)
links.reverse()
print 'Total {} comics found.'.format(len(links))
#print len(links)
return links
def getstriplink(link,opener):
ufile=opener.open(link)
html=ufile.read()
url=re.search('img src="(.+)" alt="(.+)" title=',html)
date=link[-8:]
return(url.group(1),url.group(2),date)
def main():
y=raw_input('Enter year 2002 - current(yyyy) ')
m=raw_input('Enter month(only months 12,11 and 10 for 2002)(mm) ')
add='http://www.cad-comic.com/cad/archive/'+y
opener=MyOpener()
links=getlinks(add,m,opener)
f=open('/media/aux1/pythonary/cad'+str(y)+str(m)+'.html','w')
print 'downloading'
for link in links:
url=getstriplink(link,opener)
#date=url[0][-8:]
date=url[2]
opener.retrieve(url[0],'/media/aux1/pythonary/getcad_files/strip'+date)
sys.stdout.flush()
print'.',
f.write('<h2>'+url[1]+' '+date+'</h2>'+'<p><img src="getcad_files/strip'+date+'"/></p>')
f.close()
if __name__ == '__main__':
main()