I am trying to execute below script where I try to send POST request. I have replaced some values in headers part in order to post it here. The issue I have is related to body of my request which I read from xmlFile.xml. File is in the same directory as my script. XML is written in one line and begins with following line:
<?xml version="1.0"?>
Could you please help? I cannot understand why it is returning 400 Bad Request. XML separately is working fine, but not from within the py script.
#!/usr/bin/python
import httplib
def do_request(xmlFile):
request = open(xmlFile, "r").read()
conn = httplib.HTTPConnection("ipAddress", port)
conn.putrequest("POST", "selector HTTP/1.1")
conn.putheader("Content-Length", "%d" % len(request))
conn.putheader("Content-Type", "text/xml")
conn.putheader("Host", "ipAddress")
conn.putheader("User-Agent", "userAgent")
conn.endheaders()
conn.send(request)
response = conn.getresponse()
print(response.status, response.reason)
data = response.read()
print(data)
conn.close()
do_request('xmlFile.xml')
IIRC the second argument to putrequest() should be the path part of the url (/ for the root). But you're making it much more complicated than it has to be - you could use conn.request(method, path, params, headers) as showed here, or (even better) just use python-requests actually (even the official httplib doc recommend it).
Related
I have a webservice running at "tcs-webdev2:8200/scheduler/requestgroup" to which new build requests can be sent using an xml file(sample xml file below). I need some guidance
on how requests to a webserive via n xml file work.
Sample xml file:-
<BuildInfo>
<BuildSource>DEV_CI</BuildSource>
<SoftwareProductBuild>MAAAAANLGD0000211.1_101</SoftwareProductBuild>
<PriorrootBuild>MAAAAANLGD0000211.1</PriorrootBuild>
<NewSIBuilds>
<Image>
<Type>LNX</Type>
<SoftwareImageBuild>buildlocation</SoftwareImageBuild>
<Location>\\\sever\buildlocation\checkout</Location>
<Variant>Default</Variant>
<LoadType>Debug</LoadType>
</Image>
</NewSIBuilds>
</BuildInfo>
It depends on your web service, how exactly you need to send the request, but you have to something like the following:
import httplib
with open("your_xml_filename.xml") as f:
body = f.read()
headers = {"Content-type": "application/xml"}
conn = httplib.HTTPConnection("tcs-webdev2", 8200)
conn.request("POST", "/scheduler/requestgroup", body, headers)
response = conn.getresponse()
print( response.status )
print( response.read())
conn.close()
It assumes that tcs-webdev2 is a valid hostname (if not, you can use the IP address). Also this request is an HTTP POST, your service might need different request type. Also some additional headers and authentication might be needed.
I would like to make a POST request to upload a file to a web service (and get response) using Python. For example, I can do the following POST request with curl:
curl -F "file=#style.css" -F output=json http://jigsaw.w3.org/css-validator/validator
How can I make the same request with python urllib/urllib2? The closest I got so far is the following:
with open("style.css", 'r') as f:
content = f.read()
post_data = {"file": content, "output": "json"}
request = urllib2.Request("http://jigsaw.w3.org/css-validator/validator", \
data=urllib.urlencode(post_data))
response = urllib2.urlopen(request)
I got a HTTP Error 500 from the code above. But since my curl command succeeds, it must be something wrong with my python request?
I am quite new to this topic and my question may have very simple answers or mistakes.
Personally I think you should consider the requests library to post files.
url = 'http://jigsaw.w3.org/css-validator/validator'
files = {'file': open('style.css')}
response = requests.post(url, files=files)
Uploading files using urllib2 is not impossible but quite a complicated task: http://pymotw.com/2/urllib2/#uploading-files
After some digging around, it seems this post solved my problem. It turns out I need to have the multipart encoder setup properly.
from poster.encode import multipart_encode
from poster.streaminghttp import register_openers
import urllib2
register_openers()
with open("style.css", 'r') as f:
datagen, headers = multipart_encode({"file": f})
request = urllib2.Request("http://jigsaw.w3.org/css-validator/validator", \
datagen, headers)
response = urllib2.urlopen(request)
Well, there are multiple ways to do it. As mentioned above, you can send the file in "multipart/form-data". However, the target service may not be expecting this type, in which case you may try some more approaches.
Pass the file object
urllib2 can accept a file object as data. When you pass this type, the library reads the file as a binary stream and sends it out. However, it will not set the proper Content-Type header. Moreover, if the Content-Length header is missing, then it will try to access the len property of the object, which doesn't exist for the files. That said, you must provide both the Content-Type and the Content-Length headers to have the method working:
import os
import urllib2
filename = '/var/tmp/myfile.zip'
headers = {
'Content-Type': 'application/zip',
'Content-Length': os.stat(filename).st_size,
}
request = urllib2.Request('http://localhost', open(filename, 'rb'),
headers=headers)
response = urllib2.urlopen(request)
Wrap the file object
To not deal with the length, you may create a simple wrapper object. With just a little change you can adapt it to get the content from a string if you have the file loaded in memory.
class BinaryFileObject:
"""Simple wrapper for a binary file for urllib2."""
def __init__(self, filename):
self.__size = int(os.stat(filename).st_size)
self.__f = open(filename, 'rb')
def read(self, blocksize):
return self.__f.read(blocksize)
def __len__(self):
return self.__size
Encode the content as base64
Another way is encoding the data via base64.b64encode and providing Content-Transfer-Type: base64 header. However, this method requires support on the server side. Depending on the implementation, the service can either accept the file and store it incorrectly, or return HTTP 400. E.g. the GitHub API won't throw an error, but the uploaded file will be corrupted.
I'm trying to connect to my Django server using a client side python script. Right now I'm just trying to connect to a view and retrieve the HttpResponse. The following works fine
import urllib2
import urllib
url = "http://localhost:8000/minion/serve"
request = urllib2.Request(url)
response = urllib2.urlopen(request)
html = response.read()
print html
However if I change it to
import urllib2
import urllib
url = "http://localhost:8000/minion/serve"
values = {'name': 'Bob'}
data = urllib.urlencode(values)
request = urllib2.Request(url, data)
response = urllib2.urlopen(request)
html = response.read()
print html
I get urllib2.HTTPError: HTTP Error 500: INTERNAL SERVER ERROR. Am I doing something wrong here? Here are the tutorials I was trying to follow, http://techmalt.com/?p=212 http://docs.python.org/2/howto/urllib2.html.
EDIT: tried to make the following change as per That1Guy's suggestion (other lines left the same)
request = urllib2.Request(url, data)
response = urllib2.urlopen(request, data)
This returned the same error message as before.
EDIT: It seems to work if I change the page I'm viewing so the error isn't in the client side script. So in light of that revelation, here's the server side view that's being accessed:
def serve(request):
return HttpResponse("You've been served!")
As you can see, it's very straight forward.
EDIT: Tested to see if Internal Error might be caused by missing csrf token, but csrf_exempt decorator failed to resolve error.
Finally figured it out with this. How to POST dictionary from python script to django url?
Turns out my url was missing a trailing slash. It's the little things that always get you I guess.
I'm trying to replace curl with Python & the requests library. With curl, I can upload a single XML file to a REST server with the curl -T option. I have been unable to do the same with the requests library.
A basic scenario works:
payload = '<person test="10"><first>Carl</first><last>Sagan</last></person>'
headers = {'content-type': 'application/xml'}
r = requests.put(url, data=payload, headers=headers, auth=HTTPDigestAuth("*", "*"))
When I change payload to a bigger string by opening an XML file, the .put method hangs (I use the codecs library to get a proper unicode string). For example, with a 66KB file:
xmlfile = codecs.open('trb-1996-219.xml', 'r', 'utf-8')
headers = {'content-type': 'application/xml'}
content = xmlfile.read()
r = requests.put(url, data=content, headers=headers, auth=HTTPDigestAuth("*", "*"))
I've been looking into using the multipart option (files), but the server doesn't seem to like that.
So I was wondering if there is a way to simulate curl -T behaviour in Python requests library.
UPDATE 1:
The program hangs in textmate, but throws an UnicodeEncodeError error on the commandline. Seems that must be the problem. So the question would be: is there a way to send unicode strings to a server with the requests library?
UPDATE 2:
Thanks to the comment of Martijn Pieters the UnicodeEncodeError went away, but a new issue turned up.
With a literal (ASCII) XML string, logging shows the following lines:
2012-11-11 15:55:05,154 INFO Starting new HTTP connection (1): my.ip.address
2012-11-11 15:55:05,294 DEBUG "PUT /v1/documents?uri=/example/test.xml HTTP/1.1" 401 211
2012-11-11 15:55:05,430 DEBUG "PUT /v1/documents?uri=/example/test.xml HTTP/1.1" 201 0
Seems the server always bounces the first authentication attempt (?) but then accepts the second one.
With a file object (open('trb-1996-219.xml', 'rb')) passed to data, the logfile shows:
2012-11-11 15:50:54,309 INFO Starting new HTTP connection (1): my.ip.address
2012-11-11 15:50:55,105 DEBUG "PUT /v1/documents?uri=/example/test.xml HTTP/1.1" 401 211
2012-11-11 15:51:25,603 WARNING Retrying (0 attempts remain) after connection broken by 'BadStatusLine("''",)': /v1/documents?uri=/example/test.xml
So, first attempt is blocked as before, but no second attempt is made.
According to Martijn Pieters (below), the second issue can be explained by a faulty server (empty line).
I will look into this, but if someone has a workaround (apart from using curl) I wouldn't mind hearing it.
And I am still surprised that the requests library behaves so differently for small string and file object. Isn't the file object serialized before it gets to the server anyway?
To PUT large files, don't read them into memory. Simply pass the file as the data keyword:
xmlfile = open('trb-1996-219.xml', 'rb')
headers = {'content-type': 'application/xml'}
r = requests.put(url, data=xmlfile, headers=headers, auth=HTTPDigestAuth("*", "*"))
Moreover, you were opening the file as unicode (decoding it from UTF-8). As you'll be sending it to a remote server, you need raw bytes, not unicode values, and you should open the file as a binary instead.
Digest authentication always requires you to make at least two request to the server. The first request doesn't contain any authentication data. This first request will fail with a 401 "Authorization required" response code and a digest challenge (called a nounce) to be used for hashing your password etc. (the exact details don't matter here). This is used to make a second request to the server containing your credentials hashed with the challenge.
The problem is in the this two step authentication: your large file was already send with the first unauthorized request (send in vain) but on the second request the file object is already at the EOF position. Since the file size was also send in the Content-length header of the second request, this causes the server to wait for a file that will never be send.
You could solve it using a requests Session and first make a simple request for authentication purposes (say a GET request). Then make a second PUT request containing the actual payload using the same digest challenge form the first request.
sess = requests.Session()
sess.auth = HTTPDigestAuth("*", "*")
sess.get(url)
headers = {'content-type': 'application/xml'}
with codecs.open('trb-1996-219.xml', 'r', 'utf-8') as xmlfile:
sess.put(url, data=xmlfile, headers=headers)
i used requests in python to upload an XML file using the commands.
first to open the file use open()
file = open("PIR.xsd")
fragment = file.read()
file.close()
copy the data of XML file in the payload of the requests and post it
payload = {'key':'PFAkrzjmuZR957','xmlFragment':fragment}
r = requests.post(URL,data=payload)
to check the html validation code
print (r.text)
I'm trying to create a super-simplistic Virtual In / Out Board using wx/Python. I've got the following code in place for one of my requests to the server where I'll be storing the data:
data = urllib.urlencode({'q': 'Status'})
u = urllib2.urlopen('http://myserver/inout-tracker', data)
for line in u.readlines():
print line
Nothing special going on there. The problem I'm having is that, based on how I read the docs, this should perform a Post Request because I've provided the data parameter and that's not happening. I have this code in the index for that url:
if (!isset($_POST['q'])) { die ('No action specified'); }
echo $_POST['q'];
And every time I run my Python App I get the 'No action specified' text printed to my console. I'm going to try to implement it using the Request Objects as I've seen a few demos that include those, but I'm wondering if anyone can help me explain why I don't get a Post Request with this code. Thanks!
-- EDITED --
This code does work and Posts to my web page properly:
data = urllib.urlencode({'q': 'Status'})
h = httplib.HTTPConnection('myserver:8080')
headers = {"Content-type": "application/x-www-form-urlencoded",
"Accept": "text/plain"}
h.request('POST', '/inout-tracker/index.php', data, headers)
r = h.getresponse()
print r.read()
I am still unsure why the urllib2 library doesn't Post when I provide the data parameter - to me the docs indicate that it should.
u = urllib2.urlopen('http://myserver/inout-tracker', data)
h.request('POST', '/inout-tracker/index.php', data, headers)
Using the path /inout-tracker without a trailing / doesn't fetch index.php. Instead the server will issue a 302 redirect to the version with the trailing /.
Doing a 302 will typically cause clients to convert a POST to a GET request.