I want to make a post request to a HTTPS-site that should respond with a .csv file. I have this Python code:
try:
#conn = httplib.HTTPSConnection(host="www.site.com", port=443)
=> Gives an BadStatusLine: ' ' error
conn = httplib.HTTPConnection("www.site.com");
params = urllib.urlencode({'val1':'123','val2':'abc','val3':'1b3'})
conn.request("POST", "/nps/servlet/exportdatadownload", params)
content = conn.getresponse()
print content.reason, content.status
print content.read()
conn.close()
except:
import sys
print sys.exc_info()[:2]
Output:
Found 302
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<HTML><HEAD>
<TITLE>302 Found</TITLE>
</HEAD><BODY>
<H1>Found</H1>
The document has moved here.<P>
<HR>
<ADDRESS>Oracle-Application-Server-10g/10.1.3.5.0 Oracle-HTTP-Server Server at mp-www1.mrco.be Port 7778</ADDRESS>
</BODY></HTML>
What am I doing wrong? How do I 'catch' the returning .csv file?
I tried the POST request with an Chrome Extentions (Advanced Rest Client, and that's working...)
The HTML output with the 302 error is because you're connecting to the site using HTTP instead of HTTPS, which is in your code here:
conn = httplib.HTTPConnection("www.site.com");
Presumably you're doing that because of some other error with the commented out section of your code.
If I were trying to do this I would use Requests, which comes with very clear documentation. With Requests the code would be something like:
import requests
url = "https://www.example.com/nps/servlet/exportdatadownload"
payload = { "val1": "123", "val2": "abc", "val3": "1b3" }
r = requests.post(url, data=payload, verify=True)
The CSV file should be in r.content, which can be written to a file.
Related
Im trying to export images by a custom node export server. server is running and when I send requests directly to server everything is fine.
exporting: {
url: "http://ip:7779"
},
But for some security reasons first I need to send request to my flask server, I checked incoming request and it contain this values:
CombinedMultiDict([ImmutableMultiDict([]), ImmutableMultiDict([('width', u'0'), ('scale', u'2'), ('type', u'image/png'), ('filename', u'chart'), ('svg', u'<svg xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" class="highcharts-root" ...omited!')])
and content type is like multipart/form-data; boundary=----WebKitFormBoundaryt7Gcilm12pNBmSab
so I changed my code:
url: "/highchart"
},
#app.route('/highchart', methods=["GET", "POST"])
#login_required
def highchart_export():
try:
data = {}
for i in request.values:
data[i] = request.values[i]
response = requests.post('http://0.0.0.0:7779',data=data)
return response.text, response.status_code, response.headers.items()
except:
print traceback.format_exc(sys.exc_info())
But it not work. I just get an image with text "It appears that we don't support this file format."
It seems in my code some headers and part of body is missing so for a clean redirect I used this code:
response = requests.request(
method=request.method,
url='http://127.0.0.1:7779',
headers={key: value for (key, value) in request.headers},
data=request.get_data(),
cookies=request.cookies,
allow_redirects=False)
headers = [(name, value) for (name, value) in response.raw.headers.items()]
return Response(response.content, response.status_code, headers)
And now output is fine.
import requests,json
verify='/users/myprofile/Downloads/.me/ca_bundle.crt'
data = {"Authorization": "Basic ODE2YzMzM2ZmYzUwNDg1MThlNTJmMWMyMWJiYzZhOTQ6NWExNDFhMGExNGI4NGQ2OWIyMjUxYTI2OGYwZDJlYjk="}
url = "https://api.pge.com/datacustodian/test/oauth/v2/token?grant_type=client_credentials"
response = requests.post(url, cert=('/users/David/Downloads/bei.me/private.key', data)
#print("status code", response.status_code)
Error i am getting:
File "SSLTest.py", line 12
^
SyntaxError: unexpected EOF while parsing
How do i include my public cert when making a call to utility company's api? They had me submit my public certificate
You should put:
response = requests.post(url, cert='/users/David/Downloads/bei.me/private.key', data)
In addition:
CA Certificate are not mandatory for public websites
PS: your data variable looks like an header, so you might want:
response = requests.post(url, cert='/users/David/Downloads/bei.me/private.key', headers=data)
I'm trying to make a program that will allow me to submit username and password on a website. For this, I am using DVWA(Damn Vulnerable Web Application) which is running on localhost:8080.
But whenever I try to send post request, it always returns an error.
csrf token is incorrect
Here's my code:
import requests
url = 'http://192.168.43.1:8080/login.php'
data_dict = {"username": "admin", "password": "password", "Login": "Login"}
response = requests.post(url, data_dict)
print(response.text)
You need to make GET request for that URL first, and parse the correct "CSRF" value from the response (in this case user_token). From response HTML, you can find hidden value:
<input type="hidden" name="user_token" value="28e01134ddf00ec2ea4ce48bcaf0fc55">
Also, it seems that you need to include cookies from first GET request for following request - this can be done automatically by using request.Session() object. You can see cookies by for example print(resp.cookies) from first response.
Here is modified code. I'm using BeautifulSoup library for parsing the html - it finds correct input field, and gets value from it.
POST method afterwards uses this value in user_token parameter.
from bs4 import BeautifulSoup
import requests
with requests.Session() as s:
url = 'http://192.168.43.1:8080/login.php'
resp = s.get(url)
parsed_html = BeautifulSoup(resp.content, features="html.parser")
input_value = parsed_html.body.find('input', attrs={'name':'user_token'}).get("value")
data_dict = {"username": "admin", "password": "password", "Login": "Login", "user_token":input_value}
response = s.post(url, data_dict)
print(response.content)
I am using python 2.7.5. I am using urllib2.urlopen() to make a client request to local server. When I write this piece of code seperately on my machine..it works smoothly
import json
import urllib2
urls = [ "https://localhost/rest/service1",
"https://localhost/rest/service2" ]
for url in urls:
url_encoded = urllib2.quote(url, safe="-:?=/")
token = 'f42fa4d43db94c2782feabf84fa2cc90'
headers = { "Content-Type": "application/json",
"X-Auth-Token": token }
try:
request = urllib2.Request( url_encoded, headers=headers )
response = urllib2.urlopen( request )
reply = response.read()
data = json.loads(reply)
print data
except Exception as ex:
msg = ex.msg
print msg
Same piece of code invoked in my project using WSGI rest framework gives this error:
-> response = urllib2.urlopen( request )
(Pdb)
URLError: URLError...:579)'),)
Please guide me in this.. Why is this happening ?
I want to make a post request to a HTTPS-site that should respond with a .csv file.
I have this Python code:
url = 'https://www.site.com/servlet/datadownload'
values = {
'val1' : '123',
'val2' : 'abc',
'val3' : '1b3',
}
data = urllib.urlencode(values)
req = urllib2.Request(url,data)
response = urllib2.urlopen(req)
myfile = open('file.csv', 'wb')
shutil.copyfileobj(response.fp, myfile)
myfile.close()
But 'm getting the error:
BadStatusLine: '' (in httplib.py)
I've tried the post request with the Chrome Extension: Advanced REST client (screenshot) and that works fine.
What could be the problem and how could I solve it? (is it becasue of the HTTPS?)
EDIT, refactored code:
try:
#conn = httplib.HTTPSConnection(host="www.site.com", port=443)
=> Gives an BadStatusLine: '' error
conn = httplib.HTTPConnection("www.site.com");
params = urllib.urlencode({'val1':'123','val2':'abc','val3':'1b3'})
conn.request("POST", "/nps/servlet/exportdatadownload", params)
content = conn.getresponse()
print content.reason, content.status
print content.read()
conn.close()
except:
import sys
print sys.exc_info()[:2]
Output:
Found 302
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<HTML><HEAD>
<TITLE>302 Found</TITLE>
</HEAD><BODY>
<H1>Found</H1>
The document has moved here.<P>
<HR>
<ADDRESS>Oracle-Application-Server-10g/10.1.3.5.0 Oracle-HTTP-Server Server at mp-www1.mrco.be Port 7778</ADDRESS>
</BODY></HTML>
What am I doing wrong?
Is there a reason you've got to use urllib? Requests is simpler, better in almost every way, and abstracts away some of the cruft that makes urllib hard to work with.
As an example, I'd rework you example as something like:
import requests
resp = requests.post(url, data=values, allow_redirects=True)
At this point, the response from the server is available in resp.text, and you can do what you'd like with it. If requests wasn't able to POST properly (because you need a custom SSL certificate, for example), it should give you a nice error message that tells you why.
Even if you can't do this in your production environment, do this in a local shell to see what error messages you get from requests, and use that to debug urllib.
The BadStatusLine: '' (in httplib.py) gives away that there might be something else going on here. This may happen when the server sends no reply back at all, and just closes the connection.
As you mentioned that you're using an SSL connection, this might be particularly interesting to debug (with curl -v URL if you want).
If you find out that curl -2 URL (which forces the use of SSLv2) seems to work, while curl -3 URL (SSLv3), doesn't, you may want to take a look at issue #13636 and possibly #11220 on the python bugtracker. Depending on your Python version & a possibly misconfigured webserver, this might be causing a problem: the SSL defaults have changed in v2.7.3.
conn = httplib.HTTPSConnection(host='www.site.com', port=443, cert_file=_certfile)
params = urllib.urlencode({'cmd': 'token', 'device_id_st': 'AAAA-BBBB-CCCC',
'token_id_st':'DDDD-EEEE_FFFF', 'product_id':'Unit Test',
'product_ver':"1.6.3"})
conn.request("POST", "servlet/datadownload", params)
content = conn.getresponse().read()
#print response.status, response.reason
conn.close()
The server may not like the missing headers, particularly user-agent and content-type. The Chrome image shows what is used for these. Maybe try adding the headers:
import httplib, urllib
host = 'www.site.com'
url = '/servlet/datadownload'
values = {
'val1' : '123',
'val2' : 'abc',
'val3' : '1b3',
}
headers = {
'User-Agent': 'python',
'Content-Type': 'application/x-www-form-urlencoded',
}
values = urllib.urlencode(values)
conn = httplib.HTTPSConnection(host)
conn.request("POST", url, values, headers)
response = conn.getresponse()
data = response.read()
print 'Response: ', response.status, response.reason
print 'Data:'
print data
This is untested code, and you may want to experiment by adding other header values to match your screenshot. Hope it helps.