I'm trying to build client-server app in Python.
My client use requests module to connect to the server and upload json and files.
Server use tornado framework. When server receive data from client, they start processing and send result to client by parts.
Example of my post handler:
class PostAd(tornado.web.RequestHandler):
def post(self):
jdata = self.get_body_arguments('json', False)[0]
jdata = json.loads(jdata)
id = self.insert_ad(jdata)
fpath_list = self.save_files(self.request.files.values(), id)
self.insert_file_path(id, fpath_list)
self.write("Successfully posted into SQL with sql id: {0}".format(id))
self.flush()
self.write("Are u there?")
self.finish()
in the client requests used to post data
r=agent.post("http://localhost:8888/api/v1/add-ad", data={"json": thread_data}, files=files)
in this way I cannot receive data by pieces because r=agent.post will wait until server will close connection but I need to check returned values every time when tornado server will send me data with self.flush() command (in my example I expect to get two answers, first one: "Successfully posted into SQL with sql id: 100" and second: "Are u there?").
is it possible to do it with requests module or I need to use something else here?
I don't know what agent.post() is, but you can do this with tornado's HTTP client and the streaming_callback option. You'll have to format the request body yourself, though, since Tornado doesn't have built-in client-side support for multipart file uploads.
await AsyncHTTPClient().fetch(url, body=encoded_body, streaming_callback=print)
There is no guarantee that the chunks observed by streaming_callback will align with the calls to flush, so you should format the data so that the client can determine where messages begin or end.
Related
I use the python socket.io client and I would like to know if it is possible to retrieve the session cookie, if so, how?
Another question, is it possible to get the answer from an .emit without going through an #sio.event or #sio.on()? As with the websockets library:
websocket.send('Hi')
response = websocket.rcv()
print(response) -> "Hi :)"
If not, is it possible to create an event/on that retrieves messages from the server that do not contain an event name?
For example I send from the client: sio.emit("GetNbrPlayers") but the server answers me [{"data": "5"}] without an event before the data (it is a server/api that uses socket.io too), I would like to get this message but impossible with an event/on because there is no event name in the answer.
Thank you in advance!
it is possible to retrieve the session cookie,
What do you mean by session cookie? If you mean something like the Flask session cookie, then no, there is no session cookie. The user session is stored in the server, not in a cookie.
If you mean the sid cookie that contains the session id assigned to the client, you can obtain this id from the sid attribute of your client object, for example as sio.sid.
is it possible to get the answer from an .emit without going through an #sio.event or #sio.on()?
Yes, you can use the "ack" feature of the Socket.IO protocol for this. If the server is a Python server, you can just return the response from your handler function. For example:
#sio.event
def GetNbrPlayers():
return [{"data": "5"}]
In the Python client you have two ways to receive this information. You can use a callback function:
def get_response(data):
print(data)
sio.emit("GetNbrPlayers", callback=get_response)
Or you can use call() instead of emit() to combine the emit and the callback handling into a single function call:
data = sio.call("GetNbrPlayers")
I am using Python 3.6, asyncio and the websockets library. I am trying to build a client for a websocket-based service which works as follows:
The client can send JSON requests with a custom id, a method and some params. The service will reply with a JSON payload with the same id echoed, and data as a result of the method call.
I would like to have an abstraction on top of this device that would work sort of like this:
wsc = get_websocket_connection()
async def call_method(method, **params):
packet = make_json_packet(method, params)
await wsc.send(packet)
resp = await wsc.recv()
return decode_json_packet(resp)
async def working_code():
separate_request = asyncio.ensure_future(call_method("quux"))
first_result = await call_method("foo", x=1)
second_result = await call_method("bar", y=first_result)
print(second_result)
return await separate_request
Now, I expect the separate_request to wait asynchronously while first_result and second_results are processed. But I have no guarantee that the wsc.recv() call will return the matching response; in fact, I have no guarantees that the service returns the responses in order of requests.
I can use the id field to disambiguate the responses. But how can I write the call_method() so that it manages the requests internally and resumes the "right" coroutine when the corresponding reply is received?
when I've done this sort of thing before I've tended to split things out into two parts:
"sending code" (can be multiple threads) this sets up where responses should go to (i.e. a dict of ids to functions or Futures), then sends the request and blocks for the response
"receiving code" (probably one thread per socket) that monitors all inbound traffic and passes responses off to whichever code is interested in the id. this is also a sensible place to handle the socket being closed unexpectedly which should push an exception out as appropriate
this is probably a few hundred lines of code and pretty application specific…
I'm looking for some advice, or a relevant tutorial regarding the following:
My task is to set up a flask route that POSTs to API endpoint X, receives a new endpoint Y in X's response, then GETs from endpoint Y repeatedly until it receives a certain status message in the body of Y's response, and then returns Y's response.
The code below (irrelevant data redacted) accomplishes that goal in, I think, a very stupid way. It returns the appropriate data occasionally, but not reliably. (It times out 60% of the time.) When I console log very thoroughly, it seems as though I have bogged down my server with multiple while loops running constantly, interfering with each other.
I'll also receive this error occasionally:
SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /book
import sys, requests, time, json
from flask import Flask, request
# create the Flask app
app = Flask(__name__)
# main booking route
#app.route('/book', methods=['POST']) #GET requests will be blocked
def book():
# defining the api-endpoints
PRICING_ENDPOINT = ...
# data to be sent to api
data = {...}
# sending post request and saving response as response object
try:
r_pricing = requests.post(url = PRICING_ENDPOINT, data = data)
except requests.exceptions.RequestException as e:
return e
sys.exit(1)
# extracting response text
POLL_ENDPOINT = r_pricing.headers['location']
# setting data for poll
data_for_poll = {...}
r_poll = requests.get(POLL_ENDPOINT, data = data_for_poll)
# poll loop, looking for 'UpdatesComplete'
j = 1
poll_json = r_poll.json()
update_status = poll_json['Status']
while update_status == 'UpdatesPending':
time.sleep(2)
j = float(j) + float(1)
r_poll = requests.get(POLL_ENDPOINT, data = data_for_poll)
poll_json = r_poll.json()
update_status = poll_json['Status']
return r_poll.text
This is more of an architectural issue more than a Flask issue. Long-running tasks in Flask views are always a poor design choice. In this case, the route's response is dependent on two endpoints of another server. In effect, apart from carrying the responsibility of your app, you are also carrying the responsibility of another server.
Since the application's design seems to be a proxy for another service, I would recommend creating the proxy in the right way. Just like book() offers the proxy for PRICING_ENDPOINT POST request, create another route for POLL_ENDPOINT GET request and move the polling logic to the client code (JS).
Update:
If you cannot for some reason trust the client (browser -> JS) with the POLL_ENDPOINT information in a hidden proxy like situation, then maybe move the polling to a task runner like Celery or Python RQ. Although, it will introduce additional components to your application, it would be the right way to go.
Probably you get that error because of the HTTP connection time out with your API server that is looping. There are some standards for HTTP time connection and loop took more time that is allowed for the connection. The first (straight) solution is to "play" with Apache configs and increase the HTTP connection time for your wsgi. You can also make a socket connection and in it check the update status and close it while the goal was achieved. Or you can move your logic to the client side.
As a follow up to another question I asked, I have a basic question about the easiest way to get a webapp2 python server to provide json data that is too large (about 100 kb) to send as a Channel API message to a client .
The webapp2 server generates several data files over several minutes based on a client request, and I am thinking that I would like the Channel API to send messages with the url to the client when the data is ready, and the client (a GWT app) could perform a http GET request to get the data. Each data file is unique to the client and therefore the server will have to have a request handler that will give the appropriate data file for the client.
Can you write a request handler that can provide the correct data file directly from another request handler for that particular client when the request is called? Or Do I need to store the data using Cloud SQL or the Data Store first until the client asks for it? Here is some incomplete sample code of what I would like to do:
class MainPage(webapp2.RequestHandler):
def get(self):
## This opens the GWT app
class Service_handler(webapp2.RequestHandler):
def get(self, parameters):
## This is called by the GWT app and generates the data to be
## sent to the client.
## A channel API message is sent to the client with the url
## for each data file generated.
class kml_handler(webapp2.RequestHandler):
def get(self, client_id):
## I would like to return the correct data here when it is
## called by the client. Do I need to store the data in
## Cloud SQL or the Data Store and then retrieve it
## or can this handler take the results directly from the
## Service_handler as soon as it is generated?
app = webapp2.WSGIApplication([
webapp2.Route(r'/', handler=MainPage),
webapp2.Route(r'/Service/', handler=Service_handler),
webapp2.Route(r'/_ah/channel/<connected>/', handler = connection_handler),
webapp2.Route(r'/kml/<client_id>', handler = kml_handler)
],
debug=True)
You can write files to the blobstore and serve those files from the blobstore.
Here is an example:
https://developers.google.com/appengine/docs/python/blobstore/overview#Complete_Sample_App
I have a front-end web server written in Twisted Web, that interfaces with another web server. Clients upload files to my front-end server, which then sends the files along to the back-end server. I'd like to receive the uploaded file, and then send an immediate response to the client before sending the file on to the back-end server. That way the client doesn't have to wait for both uploads to occur before getting a response.
I'm trying to do this by starting the upload to the back-end server in a separate thread. The problem is, after sending a response to the client, I'm no longer able to access the uploaded file from the Request object. Here's an example:
class PubDir(Resource):
def render_POST(self, request):
if request.args["t"][0] == 'upload':
thread.start_new_thread(self.upload, (request,))
### Send response to client while the file gets uploaded to the back-end server:
return redirectTo('http://example.com/uploadpage')
def upload(self, request):
postheaders = request.getAllHeaders()
try:
postfile = cgi.FieldStorage(
fp = request.content,
headers = postheaders,
environ = {'REQUEST_METHOD':'POST',
'CONTENT_TYPE': postheaders['content-type'],
}
)
except Exception as e:
print 'something went wrong: ' + str(e)
filename = postfile["file"].filename
file = request.args["file"][0]
#code to upload file to back-end server goes here...
When I try this, I get an error: I/O operation on closed file.
You need to actually copy the file into a buffer in memory or into a tempfile on disk before you finish the request object (which is what happens when you redirect).
So you are starting your thread and handing it the request object, it's maybe opening a connection to your backend server and beginning to copy when you redirect which finishes the request and closes any associated tempfiles and you're in trouble.
Instead of passing the whole request to your thread a quick test would be trying to just pass the content of the request to your thread:
thread.start_new_thread(self.upload, (request.content.read(),))