According to the Basecamp API documentation, files should be uploaded using HTTP POST with content type set to application/octet-stream and the request body containing the binary contents of the file (see http://developer.37signals.com/basecamp/). I'd like to stream the file rather than reading the whole thing into memory. I'm using Python 2.7.
I can see a few possibilities:
Do this using the low-level socket API.
Use urllib2 with Poster (http://atlee.ca/software/poster/) to handle the file streaming. However, with Poster you register special openers for the file streaming and I'm already using my own opener explicitly (passing it to build_opener) to handle authentication on the Basecamp server. Also Poster docs only talk about posting form data and it's not clear to me yet (still reading the source code) whether it can handle octet-stream.
Use httplib. This looks like it will give me more low-level handling for the POST data (so I can use octet-stream) but I still don't see an easy way to stream the file.
I found Python: HTTP Post a large file with streaming but it sounds like unless I want to use a form data format I'd have to patch httplib (!). That post is a year old though so I'm hoping that there is now a better way.
Right now I'm investigating creating my own mixin like Poster does, but wondering: is this really so hard? Isn't there an easier way to handle what seems to me like a relatively standard use case?
I ended up using Twisted for this since I needed the upload to happen asynchronously anyway. This excellent blog post explains the general procedure: http://marianoiglesias.com.ar/python/file-uploading-with-multi-part-encoding-using-twisted/. I simply wrote my own producer instead to write raw binary data as the POST payload.
Related
I am new to working with APIs in general and am writing code in python that needs to consume/interact with an API someone else has set up. I was wondering if there is any package out there that would build some sort of custom client class to interact with an API given a file outlining the API in some way (like a json or something where each available endpoint and http verb could be outlined in terms of stuff like allowed payload json schema for posts, general params allowed and their types, expected response json schema, the header key/value for a business verb, etc.). It would be helpful if I could have one master file outlining the endpoints available and then some package uses that to generate a client class we can use to consume the API as described.
In my googling most API packages I have found in python are much more focused on the generation of APIs but this isn't what I want.
Basically I believe you are looking for the built in requests package.
response = requests.get(f'{base_url}{endpoint}',
params={'foo': self.bar,
'foo_2':self.bar_2},
headers={'X-Api-Key': secret}
)
And from here, you can build you own class, pass it to a dataframe or whatever.
In the requests package is basically everything you need. Status handling, exception handling everything you need.
Please check the docs.
https://pypi.org/project/requests/
I have two jks files truststore.jks and keystore.jks that I use when sending REST request with java based client , now I want to use Python but I didn't find a way to use them to authenticate so How can I use them in Python ?
You didn't provide much of info (e.g. what you tried before), so my answer will be not precise.
I think what you are looking for is urllib2.urlopen() (probably using Request object to tune up request properties), note SSL-related function parameters. But first you'll probably need to convert jks files to format accepted by Python (I guess it's OpenSSL format).
I need to implement a small test utility which consumes extremely simple SOAP XML (HTTP POST) messages. This is a protocol which I have to support, and it's not my design decision to use SOAP (just trying to prevent those "why do you use protocol X?" answers)
I'd like to use stuff that's already in the basic python 2.6.x installation. What's the easiest way to do that? The sole SOAP message is really simple, I'd rather not use any enterprisey tools like WSDL class generation if possible.
I already implemented the same functionality earlier in Ruby with just plain HTTPServlet::AbstractServlet and REXML parser. Worked fine.
I thought I could a similar solution in Python with BaseHTTPServer, BaseHTTPRequestHandler and the elementree parser, but it's not obvious to me how I can read the contents of my incoming SOAP POST message. The documentation is not that great IMHO.
You could write a WSGI function (see wsgiref) and parse inside it an HTTP request body using the xml.etree.ElementTree module.
SOAP is basically very simple, I'm not sure that it deserves a special module. Just use a standard XML processing library you like.
I wrote something like this in Boo, using a .Net HTTPListener, because I too had to implement someone else's defined WSDL.
The WSDL I was given used document/literal form (you'll need to make some adjustments to this information if your WSDL uses rpc/encoded). I wrapped the HTTPListener in a class that allowed client code to register callbacks by SOAP action, and then gave that class a Start method that would kick off the HTTPListener. You should be able to do something very similar in Python, with a getPOST() method on BaseHTTPServer to:
extract the SOAP action from the HTTP
headers
use elementtree to extract the SOAP
header and SOAP body from the POST'ed
HTTP
call the defined callback for the
SOAP action, sending these extracted values
return the response text given by the
callback in a corresponding SOAP
envelope; if the callback raises an
exception, catch it and re-wrap it as
a SOAP fault
Then you just implement a callback per SOAP action, which gets the XML content passed to it, parses this with elementtree, performs the desired action (or mock action if this is tester), and constructs the necessary response XML (I was not too proud to just create this explicitly using string interpolation, but you could use elementtree to create this by serializing a Python response object).
It will help if you can get some real SOAP sample messages in order to help you not tear out your hair, especially in the part where you create the necessary response XML.
I've been trying to use suds for Python to call a SOAP WSDL. I just need to call the service programmatically and write the output XML document. However suds automatically parses this data into it's own pythonic data format. I've been looking through the examples and the documentation, but I can't seem to find a way to return the XML document that the SOAP service gives me.
Is there an easy way to do this I'm overlooking? Is there an easier way to do this in Python than suds?
At this early stage in suds development, the easiest way to get to the raw XML content is not what one would expect.
The examples on the site show us with something like this:
client = Client(url)
result = client.service.Invoke(subm)
however, the result is a pre-parsed object that is great for access by Python, but not for XML document access. Fortunately the Client object still has the original SOAP message received stored.
result = client.last_received()
print result
Will give you the actual SOAP message received back.
You could take a look at a library such as soaplib: its a really nice way to consume (and serve) SOAP webservices in Python. The latest version has some code to dynamically generate Python bindings either dynamically (at runtime) or statically (run a script against some WSDL).
[disclaimer: I'm the maintainer of the project! - I didn't write the bulk of it though]
I have an application that should communicate status information to a server. This information is effectively a large dictionary with string keys.
The server will run a web application based on Turbogears, so the server-side method called accepts an arbitrary number of keyword arguments.
In addition to the actual data, some data related to authentication (id, password..) should be transmitted. One approach would be to simply urlencode a large dictionary containing all this and send it in a request to the server.
urllib2.urlencode(dataPlusId)
But actually, the method doing the authentication and accepting the data set does not have to know much about the data. The data could be transmitted and accepted transparently and handed over to another method working with the data.
So my question is: What is the best way to transmit a large dictionary of data to a server in general? And, in this specific case, what is the best way to deal with authentication here?
I agree with all the answers about avoiding pickle, if safety is a concern (it might not be if the sender gets authenticated before the data's unpickled -- but, when security's at issue, two levels of defense may be better than one); JSON is often of help in such cases (or, XML, if nothing else will do...!-).
Authentication should ideally be left to the webserver, as SpliFF recommends, and SSL (i.e. HTTPS) is generally good for that. If that's unfeasible, but it's feasible to let client and server share a "secret", then sending the serialized string in encrypted form may be best.
I think the best way is to encode your data in an appropriate transfer format (you should not use pickle, as it's not save, but it can be binary) and transfer it as a multipart post request
What I do not know if you can make it work with repoze.who. If it does not support sign in and function call in one step, you'll perhaps have to verify the credentials yourself.
If you can wrap your data in xml you could also use XML-RPC.
Why don't you serialize the dictionary to a file, and upload the file? This way, the server can read the object back into a dictionary .
Do a POST of your python data (use binary as suggested in other answers) and handle security using your webserver. Apache and Microsoft servers can both do authentication using a wide variety of methods (SSL client certs, Password, System accounts, etc...)
Serialising/Deserialising to text or XML is probably overkill if you're just going to turn it back to dictionary again).
I'd personally use SimpleJSON at both ends and just post the "file" (it would really just be a stream) over as multipart data.
But that's me. There are other options.
Have you tried using pickle on the data ?