I have an application that should communicate status information to a server. This information is effectively a large dictionary with string keys.
The server will run a web application based on Turbogears, so the server-side method called accepts an arbitrary number of keyword arguments.
In addition to the actual data, some data related to authentication (id, password..) should be transmitted. One approach would be to simply urlencode a large dictionary containing all this and send it in a request to the server.
urllib2.urlencode(dataPlusId)
But actually, the method doing the authentication and accepting the data set does not have to know much about the data. The data could be transmitted and accepted transparently and handed over to another method working with the data.
So my question is: What is the best way to transmit a large dictionary of data to a server in general? And, in this specific case, what is the best way to deal with authentication here?
I agree with all the answers about avoiding pickle, if safety is a concern (it might not be if the sender gets authenticated before the data's unpickled -- but, when security's at issue, two levels of defense may be better than one); JSON is often of help in such cases (or, XML, if nothing else will do...!-).
Authentication should ideally be left to the webserver, as SpliFF recommends, and SSL (i.e. HTTPS) is generally good for that. If that's unfeasible, but it's feasible to let client and server share a "secret", then sending the serialized string in encrypted form may be best.
I think the best way is to encode your data in an appropriate transfer format (you should not use pickle, as it's not save, but it can be binary) and transfer it as a multipart post request
What I do not know if you can make it work with repoze.who. If it does not support sign in and function call in one step, you'll perhaps have to verify the credentials yourself.
If you can wrap your data in xml you could also use XML-RPC.
Why don't you serialize the dictionary to a file, and upload the file? This way, the server can read the object back into a dictionary .
Do a POST of your python data (use binary as suggested in other answers) and handle security using your webserver. Apache and Microsoft servers can both do authentication using a wide variety of methods (SSL client certs, Password, System accounts, etc...)
Serialising/Deserialising to text or XML is probably overkill if you're just going to turn it back to dictionary again).
I'd personally use SimpleJSON at both ends and just post the "file" (it would really just be a stream) over as multipart data.
But that's me. There are other options.
Have you tried using pickle on the data ?
Related
I have a TCP server written in SwiftNIO, based on this documentation.
I want my client to be written in python from which I can send multiple JSON strings & can receive similar/different multiple JSON string(s) as a response periodically for a few minutes.
In which format do I need to convert those JSON strings from the python client & how do I get the same JSON string on the SwiftNIO server (and vice versa)?
If I were you, I'd use HTTP using the Vapor web server and any Python HTTP library such as requests. If you do that, then your job will be pretty straightforward. The Vapor community is also super helpful in their Discord chat.
If you really want to do this in a low-level library like SwiftNIO then that's of course possible but you'll need to design a so called "wire protocol" for the framing (ie. when does one JSON message start and end). SwiftNIO is very well equipped for these things but you'll likely need to learn a bunch of things.
You could for example use NIO Extras' LineBasedFrameDecoder and send each JSON (make sure it doesn't contain newlines) followed by a \n. Or you could say that you prepend the JSON by say a 32 bit length field (which you could decode using the LengthFieldBasedFrameDecoder. There are many options...
You could also implement JSON-RPC and you could get some inside in this example which is also explained in this talk.
I want to get a stream object from Azure Inheritance Iterator ItemPaged - ItemPaged[TableEntity] to stream (Python). Is it possible?
https://learn.microsoft.com/en-us/python/api/azure-core/azure.core.paging.itempaged?view=azure-python
https://learn.microsoft.com/en-us/python/api/azure-core/azure.core.paging.itempaged?view=azure-python
#Updated 11.08.2021
I have a realization to backup Azure Tables to Azure Blob - Current process to backup Azure Tables. But I want to improve this process and I am considering different options. I try to get the stream from Azure Tables to use create_blob_from_stream
I assume you want to stream bytes from the HTTP response, and not the use the iterator of objects you receive.
Each API in the SDK supports a keyword argument call raw_response_hook that gives you access to the HTTP response object, and then let you use a stream download API if you want to. Note that since the payload is considered to represent objects, it will be pre-loaded in memory no matter what, but you can still use a stream syntax nonetheless.
The callback is simply one parameter:
def response_callback(response):
# Do something with the response
requests_response = response.internal_response
# Use "requests" API now
for chunk in requests_response.iter_content():
work_with_chunk(chunk)
Note that this is pretty advanced, you may encounter difficulties and this might not fit what you want precisely. We are working on a new pattern on SDK to simplify complex scenario like that, but it's not shipped yet. You would be able to send and receive raw requests using a send_request method, which gives you absolute control on all aspect of the query, like explaining you just want to stream (no pre-load in memory) or disabling the deserialization by default.
Feel free to open an issue on the Azure SDK for Python repo if you have additional questions or clarification: https://github.com/Azure/azure-sdk-for-python/issues
Edit with new suggestions: TableEntity is a dict like class, so you can json.dumps as string, or json.dump as a stream while using the ItemPaged<TableEntity>. If JSON dumps raise an exception, you can try our JSON encoder in azure.core.serialization.AzureJSONEncoder: https://github.com/Azure/azure-sdk-for-python/blob/1ffb583d57347257159638ae5f71fa85d14c2366/sdk/core/azure-core/tests/test_serialization.py#L83
(I work at MS in the Azure SDK for Python team.)
Ref:
https://docs.python-requests.org/en/master/api/#requests.Response.iter_content
https://azuresdkdocs.blob.core.windows.net/$web/python/azure-core/1.17.0/azure.core.pipeline.policies.html#azure.core.pipeline.policies.CustomHookPolicy
I have two jks files truststore.jks and keystore.jks that I use when sending REST request with java based client , now I want to use Python but I didn't find a way to use them to authenticate so How can I use them in Python ?
You didn't provide much of info (e.g. what you tried before), so my answer will be not precise.
I think what you are looking for is urllib2.urlopen() (probably using Request object to tune up request properties), note SSL-related function parameters. But first you'll probably need to convert jks files to format accepted by Python (I guess it's OpenSSL format).
As the title says, is there a way to expose a function to both the JSON-RPC and the XML-RPC interface? Preferably one server running on a single port would answer to both types of requests.
Thanks!
One of the usual ways of returning different formats is to specify the type you want in the url in some way. Most common are:
http://example.com/some/page.<format>
or
http://example.com/some/page?output=<format>
And then your returned object should be transformed into the wanted format:
// somewhere at the end of the method handling the request
return Formater(format_param).format(response_object)
According to the Basecamp API documentation, files should be uploaded using HTTP POST with content type set to application/octet-stream and the request body containing the binary contents of the file (see http://developer.37signals.com/basecamp/). I'd like to stream the file rather than reading the whole thing into memory. I'm using Python 2.7.
I can see a few possibilities:
Do this using the low-level socket API.
Use urllib2 with Poster (http://atlee.ca/software/poster/) to handle the file streaming. However, with Poster you register special openers for the file streaming and I'm already using my own opener explicitly (passing it to build_opener) to handle authentication on the Basecamp server. Also Poster docs only talk about posting form data and it's not clear to me yet (still reading the source code) whether it can handle octet-stream.
Use httplib. This looks like it will give me more low-level handling for the POST data (so I can use octet-stream) but I still don't see an easy way to stream the file.
I found Python: HTTP Post a large file with streaming but it sounds like unless I want to use a form data format I'd have to patch httplib (!). That post is a year old though so I'm hoping that there is now a better way.
Right now I'm investigating creating my own mixin like Poster does, but wondering: is this really so hard? Isn't there an easier way to handle what seems to me like a relatively standard use case?
I ended up using Twisted for this since I needed the upload to happen asynchronously anyway. This excellent blog post explains the general procedure: http://marianoiglesias.com.ar/python/file-uploading-with-multi-part-encoding-using-twisted/. I simply wrote my own producer instead to write raw binary data as the POST payload.