I got timeout error on my GAE server when it tries to send large files to an EC2 REST server. I found Backends Python API would be a good solution to my example but I had some problems on configuring it.
Following some instructions, I have added a simple backends.yaml in my project folder. But I still received the following error, which seems like I failed to create a backend instance.
File "\Google\google_appengine\google\appengine\api\background_thread\background_thread.py", line 84, in start_new_background_thread
raise ERROR_MAP[error.application_error](error.error_detail)
FrontendsNotSupported
Below is my code, and my question is:
Currently, I got timeout error in OutputPage.py, how do I let this script run on a backend instance?
update
Following Jimmy Kane's suggestions, I created a new script przm_batchmodel_backend.py for the backend instance. After staring my GAE, now it I have two ports (a default and a backend) running my site. Is that correct?
app.yaml
- url: /backend.html
script: przm_batchmodel.py
backends.yaml
backends:
- name: mybackend
class: B1
instances: 1
options: dynamic
OutputPage.py
from przm import przm_batchmodel
from google.appengine.api import background_thread
class OutputPage(webapp.RequestHandler):
def post(self):
form = cgi.FieldStorage()
thefile = form['upfile']
#this is the old way to initiate calculations
#html= przm_batchmodel.loop_html(thefile)
przm_batchoutput_backend.przmBatchOutputPageBackend(thefile)
self.response.out.write(html)
app = webapp.WSGIApplication([('/.*', OutputPage)], debug=True)
przm_batchmodel.py
def loop_html(thefile):
#parses uploaded csv and send its info. to the REST server, the returned value is a html page.
data= csv.reader(thefile.file.read().splitlines())
response = urlfetch.fetch(url=REST_server, payload=data, method=urlfetch.POST, headers=http_headers, deadline=60)
return response
przm_batchmodel_backend.py
class BakendHandler(webapp.RequestHandler):
def post(self):
t = background_thread.BackgroundThread(target=przm_batchmodel.loop_html, args=[thefile])
t.start()
app = webapp.WSGIApplication([('/backend.html', BakendHandler)], debug=True)
You need to create an application 'file'/script for the backend to work. Just like you do with the main.
So something like:
app.yaml
- url: /backend.html
script: przm_batchmodel.py
and on przm_batchmodel.py
class BakendHandler(webapp.RequestHandler):
def post(self):
html = 'test'
self.response.out.write(html)
app = webapp.WSGIApplication([('/backend.html', OutputPage)], debug=True)
May I also suggest using the new feature modules which are easier to setup?
Edit due to comment
Possible the setup was not your problem.
From the docs
Code running on a backend can start a background thread, a thread that
can "outlive" the request that spawns it. They allow backend instances
to perform arbitrary periodic or scheduled tasks or to continue
working in the background after a request has returned to the user.
You can only use backgroundthread on backends.
So edit again. Move the part of the code that is:
t = background_thread.BackgroundThread(target=przm_batchmodel.loop_html, args=[thefile])
t.start()
self.response.out.write(html)
To the backend app
Related
I have the following code which works successfully when I deploy it on App Engine. Except I want to run it with a cron.
import pandas as pd
import requests
from flask import Flask
app = Flask(__name__)
#app.route('/')
def sfLibraries():
return 'Success'
if __name__ == '__main__':
app.run(host='127.0.0.1', port=8080, debug=True)
cron.yaml
cron:
- description: "SF Loader"
url: /task/loader
schedule: every 1 hours
app.yaml
runtime: python38
handlers:
- url: /task/loader
script: auto
I know that if I change #app.route('/') to #app.route('/task/loader') that the cron will work successfully, yet I lose the capability to go to the site when I first deploy it and see success.
gcloud app browse
Did not detect your browser. Go to this link to view your app:
https://<<<my_synapse>>>.uc.r.appspot.com
https://<<<my_synapse>>>.uc.r.appspot.com:
Not Found
The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.
How can I have it both ways -both able to go to the site via gcloud app browse and see success as well as run the cron successfully?
I mentioned WSGI Handler because I found this pertinent post but not sure how WSGI Handler might help or improve from what Flask is doing Setting up cron job in google app engine python
Create two different routes to accomplish what you need. The functions must have a different name in this case.
#app.route('/')
def sfLibraries():
return 'Success'
#app.route('/task/loader')
def sfLibrariesSecond():
return 'Success'
Assign different actions for each one.
They could also both do the exact same thing. If that was the case, then assign two routes to the same function.
#app.route('/')
#app.route('/task/loader')
def sfLibraries():
return 'Success'
I'm trying to trigger a python module (market order for Oanda) using web hooks(from trading view).
Similar to this
1) https://www.youtube.com/watch?v=88kRDKvAWMY&feature=youtu.be
and this
2)https://github.com/Robswc/tradingview-webhooks-bot
But my broker is Oanda so I'm using python to place the trade. This link has more information.
https://github.com/hootnot/oanda-api-v20
The method is web hook->ngrok->python. When a web hook is sent, the ngrok (while script is also running) shows a 500 internal service error and that the server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.
This is what my script says when its running (see picture);
First says some stuff related the market order then;
running script picture
One thing I noticed is that after Debug it doesn't say Running on... (so maybe my flask is not active?
Here is the python script;
from flask import Flask
import market_orders
# Create Flask object called app.
app = Flask(__name__)
# Create root to easily let us know its on/working.
#app.route('/')
def root():
return 'online'
#app.route('/webhook', methods=['POST'])
def webhook():
if request.method == 'POST':
# Parse the string data from tradingview into a python dict
print(market_orders.myfucn())
else:
print('do nothing')
if __name__ == '__main__':
app.run()
Let me know if there is any other information that would be helpful.
Thanks for your help.
I fixed it!!!! Google FTW
The first thing I learned was how to make my module a FLASK server. I followed these websites to figure this out;
This link helped me set up the flask file in a virtual environment. I also moved my Oanda modules to this new folder. And opened the ngrok app while in this folder via the command window. I also ran the module from within the command window using flask run.
https://topherpedersen.blog/2019/12/28/how-to-setup-a-new-flask-app-on-a-mac/
This link showed me how to set the FLASK_APP and the FLASK_ENV
Flask not displaying http address when I run it
Then I fixed the internal service error by adding return 'okay' after print(do nothing) in my script. This I learned from;
Flask Value error view function did not return a response
I have a flask application with a single endpoint, like this (simplified below):
#app.route('/classify', methods=['POST'])
def classify():
p = g.model_loader.get_model()
json = request.get_json()
text = json['text']
return p.classify(text)
def main():
model_loader = ResourceLoader()
with app.app_context():
g.model_loader = model_loader
app.run()
if __name__ == '__main__':
main()
The application needs to load a machine learning model into memory once, in the main method, and then use that model to classify text that is being sent via POST to a flask endpoint. I've tried to do this using an application context, which works locally, but doesn't work on heroku. My understanding currently is that this is because the application context isn't shared across workers. How can I give the classify endpoint access to the model loader?
EDIT: I didn't word the initial question correctly. Each worker should run main() and should thus have access to model_loader in its own application context. However, when I run this on heroku, I get AttributeError: '_AppCtxGlobals' object has no attribute 'model_loader'. Does the application context differ on heroku?
You can't load it into memory once, because Heroku workers might be on completely different machines. You need to run this code in every worker, or store the data somewhere it can be read by any process.
I am having some problems understanding how a application written in python jsonrpc2 is related to a wgsi application.
I have a json rpc test application in a file called greeting.py
It is a simple test case
def hello(name=None,greeting=None):
# Print to stdout the greeting
result = "From jsonrpc you have: {greeting} , {name}".format(greeting=greeting,name=name)
# print result
# You can basically now return the string result
return result
Using the jsonrpc2 module I am able to POST json to this function which then returns a json response.
Sample post :
self.call_values_dict_webpost = dict(zip(["jsonrpc","method","id","params"],["2.0","greeting.hello","2",["Hari","Hello"]]))
Response returned as json:
u"jsonrpc": u"2.0", u"id": u"2", u"result": u"From jsonrpc you have: Hello , Hari"
I start the server with an entry point defined in the jsonrpc2 module which essentially does the following
from jsonrpc2 import JsonRpcApplication
from wsgiref.simple_server import make_server
app = JsonRpcApplication()
app.rpc.add_module("greeting")
httpd = make_server(host, port, app)
httpd.serve_forever()
I can currently run this jsonrpc2 server as a standalone "web app" and test it approproately.
I wanted to understand how to go from this simple function web app to a wsgi web app that reads and writes json without using a web framework such as flask or django ( which I know some of)
I am looking for whether there is a simple conceptual step that makes my function above compatible with a wsgi "callable" : or am I just better off using flask or django to read/receive json "POST" and write json response.
I don't know that particular module, but it looks like your app object is the WSGI application. All you do in that code is instantiate the app, then create a server for it via wsgiref. So instead of doing that, just point your real WSGI server - Apache/mod_wsgi, or gunicorn, or whatever - to that app object in exactly the same way as you would serve Flask or Django.
How do I get App Engine to generate the URL of the server it is currently running on?
If the application is running on development server it should return
http://localhost:8080/
and if the application is running on Google's servers it should return
http://application-name.appspot.com
You can get the URL that was used to make the current request from within your webapp handler via self.request.url or you could piece it together using the self.request.environ dict (which you can read about on the WebOb docs - request inherits from webob)
You can't "get the url for the server" itself, as many urls could be used to point to the same instance.
If your aim is really to just discover wether you are in development or production then use:
'Development' in os.environ['SERVER_SOFTWARE']
Here is an alternative answer.
from google.appengine.api import app_identity
server_url = app_identity.get_default_version_hostname()
On the dev appserver this would show:
localhost:8080
and on appengine
your_app_id.appspot.com
If you're using webapp2 as framework chances are that you already using URI routing in you web application.
http://webapp2.readthedocs.io/en/latest/guide/routing.html
app = webapp2.WSGIApplication([
webapp2.Route('/', handler=HomeHandler, name='home'),
])
When building URIs with webapp2.uri_for() just pass _full=True attribute to generate absolute URI including current domain, port and protocol according to current runtime environment.
uri = uri_for('home')
# /
uri = uri_for('home', _full=True)
# http://localhost:8080/
# http://application-name.appspot.com/
# https://application-name.appspot.com/
# http://your-custom-domain.com/
This function can be used in your Python code or directly from templating engine (if you register it) - very handy.
Check webapp2.Router.build() in the API reference for a complete explanation of the parameters used to build URIs.