I have a TCP server written in SwiftNIO, based on this documentation.
I want my client to be written in python from which I can send multiple JSON strings & can receive similar/different multiple JSON string(s) as a response periodically for a few minutes.
In which format do I need to convert those JSON strings from the python client & how do I get the same JSON string on the SwiftNIO server (and vice versa)?
If I were you, I'd use HTTP using the Vapor web server and any Python HTTP library such as requests. If you do that, then your job will be pretty straightforward. The Vapor community is also super helpful in their Discord chat.
If you really want to do this in a low-level library like SwiftNIO then that's of course possible but you'll need to design a so called "wire protocol" for the framing (ie. when does one JSON message start and end). SwiftNIO is very well equipped for these things but you'll likely need to learn a bunch of things.
You could for example use NIO Extras' LineBasedFrameDecoder and send each JSON (make sure it doesn't contain newlines) followed by a \n. Or you could say that you prepend the JSON by say a 32 bit length field (which you could decode using the LengthFieldBasedFrameDecoder. There are many options...
You could also implement JSON-RPC and you could get some inside in this example which is also explained in this talk.
I would greatly benefit from a list of all the available headers that App Service can forward to my (keyword PYTHON) Function. Or if someone knows how to "list-all", that would be awesome.
Through asking questions on SO, I see that the request IP addressed can be gleaned using:
req.headers.get("X-FORWARDED-FOR").
I need the Hostname that a request is coming from.
Looks like this is possible using C# Functions. But I either did it wrong using req.headers.Host or its not available for Python.
Is it possible using Python?
For this requirement, you just need to use req.headers.get("host"). I test it in my side, it works fine on azure portal.
I'm currently building a python gRPC server that serializes tons of different proto messages into json to store them into a no-sql db. I'd like to simplify extension of this server such that we can add new types without rewriting the gRPC server and redeploying. Ideally, we would like to define a new message, put it in a proto file and update only the client. The server should expect any type at first but knows a .proto file or folder where to look for when it comes to serializing/deserializing.
I've read about the Any type and I'm exploring whether this is my way to do this. There is some documentation on it but very few examples to work with. One thing that I don't quite get is how to store/retrieve the type of an "Any" field.
All documentation use https as protocol for the type of an Any field (e.g. type.googleapis.com/google.protobuf.Duration). This is also the default. How would it look like if I use the local file system? How would I store this in the proto message on the client side?
How can I retrieve the type on the server side?
Where can I find a similar example?
Apologies, this is only a partial answer.
I've recently begun using Any in a project and can provide some perspective. I have a similar (albeit simpler) requirement to what you outline. Enveloped message content but, in my case, clients are required to ship a descriptor to the server and identify a specific method to help it (un)marshal etc.
I've been using Google's new Golang APIv2 and only familiar with it from Golang and Rust (not Python). The documentation is lacking but the Golang documents will hopefully help:
anypb
protoregistry
I too struggled with understanding the concept (implementation) of the global registry and so I hacked the above solution. The incoming message metadata provides sufficient context to the server that it can construct the message type and marshal the bytes into it.
According to the Basecamp API documentation, files should be uploaded using HTTP POST with content type set to application/octet-stream and the request body containing the binary contents of the file (see http://developer.37signals.com/basecamp/). I'd like to stream the file rather than reading the whole thing into memory. I'm using Python 2.7.
I can see a few possibilities:
Do this using the low-level socket API.
Use urllib2 with Poster (http://atlee.ca/software/poster/) to handle the file streaming. However, with Poster you register special openers for the file streaming and I'm already using my own opener explicitly (passing it to build_opener) to handle authentication on the Basecamp server. Also Poster docs only talk about posting form data and it's not clear to me yet (still reading the source code) whether it can handle octet-stream.
Use httplib. This looks like it will give me more low-level handling for the POST data (so I can use octet-stream) but I still don't see an easy way to stream the file.
I found Python: HTTP Post a large file with streaming but it sounds like unless I want to use a form data format I'd have to patch httplib (!). That post is a year old though so I'm hoping that there is now a better way.
Right now I'm investigating creating my own mixin like Poster does, but wondering: is this really so hard? Isn't there an easier way to handle what seems to me like a relatively standard use case?
I ended up using Twisted for this since I needed the upload to happen asynchronously anyway. This excellent blog post explains the general procedure: http://marianoiglesias.com.ar/python/file-uploading-with-multi-part-encoding-using-twisted/. I simply wrote my own producer instead to write raw binary data as the POST payload.
I have an application that should communicate status information to a server. This information is effectively a large dictionary with string keys.
The server will run a web application based on Turbogears, so the server-side method called accepts an arbitrary number of keyword arguments.
In addition to the actual data, some data related to authentication (id, password..) should be transmitted. One approach would be to simply urlencode a large dictionary containing all this and send it in a request to the server.
urllib2.urlencode(dataPlusId)
But actually, the method doing the authentication and accepting the data set does not have to know much about the data. The data could be transmitted and accepted transparently and handed over to another method working with the data.
So my question is: What is the best way to transmit a large dictionary of data to a server in general? And, in this specific case, what is the best way to deal with authentication here?
I agree with all the answers about avoiding pickle, if safety is a concern (it might not be if the sender gets authenticated before the data's unpickled -- but, when security's at issue, two levels of defense may be better than one); JSON is often of help in such cases (or, XML, if nothing else will do...!-).
Authentication should ideally be left to the webserver, as SpliFF recommends, and SSL (i.e. HTTPS) is generally good for that. If that's unfeasible, but it's feasible to let client and server share a "secret", then sending the serialized string in encrypted form may be best.
I think the best way is to encode your data in an appropriate transfer format (you should not use pickle, as it's not save, but it can be binary) and transfer it as a multipart post request
What I do not know if you can make it work with repoze.who. If it does not support sign in and function call in one step, you'll perhaps have to verify the credentials yourself.
If you can wrap your data in xml you could also use XML-RPC.
Why don't you serialize the dictionary to a file, and upload the file? This way, the server can read the object back into a dictionary .
Do a POST of your python data (use binary as suggested in other answers) and handle security using your webserver. Apache and Microsoft servers can both do authentication using a wide variety of methods (SSL client certs, Password, System accounts, etc...)
Serialising/Deserialising to text or XML is probably overkill if you're just going to turn it back to dictionary again).
I'd personally use SimpleJSON at both ends and just post the "file" (it would really just be a stream) over as multipart data.
But that's me. There are other options.
Have you tried using pickle on the data ?