Develop Reference

App Engine Standard Python3 Handler Error Locally Only

Running the dev server for App Engine standard python 3 is not routing requests properly.
dev_appserver.py app.yaml
The app.yaml file has 3 handlers.
runtime: python37
instance_class: F1
inbound_services:
- warmup
handlers:
- url: /api/.*
script: auto
secure: always
- url: /
static_files: public/index.html
upload: public/index.html
secure: always
- url: /
static_dir: public
secure: always
Locally the requests to /api/whatever all return 404 errors.
When I deploy the app to GCP the requests succeed.
The reasoning for my setup is statically hosting Angular 7 app while also hosting an API the angular app calls.
Since the issue is only associated with the dev server, I think this is a bug. There is a similar python 2 example here: https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/appengine/standard/angular/app.yaml
Has anyone else run into this? Any workarounds?
Update: As requested in the comments here is an example main.py file.
# [START gae_python37_app]
import logging
from flask import request, url_for
from flask_api import FlaskAPI, status, exceptions
# Create flask app
app = FlaskAPI(__name__)
#app.route("/api/whatever", methods=["GET"])
def doSomething():
response = {"message":"placeholder"}
return response
if __name__ == "__main__":
# This is used when running locally only. When deploying to Google App
# Engine, a webserver process such as Gunicorn will serve the app.
app.run(host="127.0.0.1", port=8080, debug=True)
# [END gae_python37_app]

UPDATE: I attempted to recreate the issue and there indeed seems to be a discrepancy between the dev_appserver.py and the deployed version. I have created this issue on Google’s Issue Tracker to be properly followed up by the App Engine engineering team.
The answer below is only valid if your root directory doesn't contain a static_dir path
The issue is with the way that routing is implemented. I attempted to replicate the issue that you were having and I got the same results.
In my case, I was able to resolve it by changing the route from /api/.* to ('/api/<path:path>'), since the function associated with that handle will then be properly defined as a catch-all 1.
Please refer to the code[2] provided along with the links provided1.
1 https://flask.palletsprojects.com/en/1.1.x/quickstart/#routing
main.py
from flask import Flask
app = Flask(__name__)
#app.route('/')
def hello():
"""Return a friendly HTTP greeting."""
return 'Hello World!'
#app.route('/api/<path:path>')
def catch_all(path):
return 'Hello api page!'`
if __name__ == '__main__':
# This is used when running locally only. When deploying to Google App
# Engine, a webserver process such as Gunicorn will serve the app. This
# can be configured by adding an entrypoint to app.yaml.
app.run(host='127.0.0.1', port=8080, debug=True)`
app.yaml
runtime: python37
handlers:
- url: /
script: auto
- url: /api/.*
script: auto

GAE app.yaml Script Handling for scripts in subdirectories (python)

I am setting up a new GAE project, and I can't get my scripts in sub-directories to work.
EDIT: If I go to localhost:8080/testing_desc.html I don't any errors, just a blank page (view-source is blank, too). Scripts in my root dir work normally. There is a __init__.py in both the root and sub-dir.
Python script example ("/testing/testing_desc.py"):
import webapp2 as webapp
class DescTstPage(webapp.RequestHandler):
def get(self):
html = 'This should work.'
self.response.out.write(html)
app = webapp.WSGIApplication([('/.*', DescTstPage)], debug=True)
app.yaml:
application: blah
version: blah
runtime: python27
api_version: 1
default_expiration: "5d 12h"
threadsafe: false
libraries:
- name: webapp2
version: latest
handlers:
- url: / <-- This one works
script: main.app
- url: /index.html <-- This does NOT work (??)
script: main.app
- url: /(.*?)_desc.html <-- Also does NOT work
script: \1/\1_desc.app
#file not found
- url: /.*
script: file_not_found.app
I have also tried a simpler version of the yaml:
-url: /testing_desc.html
script: /testing/testing_desc.app

When you use WSGI Pyhon27 this will work. Use a dot for the seperator :
-url: /testing_desc.html
script: /testing.testing_desc.app

You are passing a list of routes to your WSGIApplicaiton call. Those routes must match whatever URL is being handled.
('/', DescTstPage) just matches the '/' URL to your handler. /.* matches all routes.
What routes are you setting in testing/testing_desc.app? They must match /testing_desc.html.

To get my scripts working in sub-dir, I changed the app.yaml and /testing/testing_desc.py to this:
app.yaml:
- url: /testing.html
script: testing/testing_desc.py
/testing/testing_desc.py:
app = webapp.WSGIApplication([('/.*', DescTstPage),], debug=True)
def main():
run_wsgi_app(app)
if __name__ == '__main__':
main()
ATM I do not understand how to make the routing work with sub-dir, so I will use this.

The answer is here:
How can i use script in sub folders on gae?
Its not explicitly stated, but in the answer you must change /foo/ to /foo in the WSGIApplication call, which maps to www.bar.com/foo. If you want to map to www.bar.com/foo.html you need to change /foo to /foo.* in the WSGIApplication call AND the url handler in app.yaml to - url: /foo\.html

My API doesn't show in the explorer and the logs just shows 500

As the title indicates my API isn't visible in the explorer and all my logs show is the following:
INFO 2013-03-08 13:39:08,182 dev_appserver.py:723] Internal redirection to http://127.0.0.1:8080/_ah/spi/BackendService.getApiConfigs
INFO 2013-03-08 13:39:08,198 dev_appserver.py:3104] "GET /_ah/api/discovery/v1/apis HTTP/1.1" 500 -
The relevant handler from my app.yaml file looks as follows:
13 # Endpoint handlers
14 - url: /_ah/spi/.*
15 script: main.app
And my code from main.py is as follows:
from google.appengine.ext import endpoints
from protorpc import messages
class Location(messages.Message):
reg_id = messages.StringField(1)
phone_number = messages.StringField(2)
latitude = messages.StringField(3)
longitude = messages.StringField(4)
#endpoints.api(name='locations', version='v1', description='Location API for where are you app')
class LocationApi(remote.Service):
#endpoints.method(Location, Location, name='location.insert', path='location', http_method='POST')
def insert(self, request):
return request
app = endpoints.api_server([LocationApi])
Anyone got a clue about what I'm doing wrong?

Please check the following:
As you should be able to see from the logs, you're getting an import error. In main.py add this
from protorpc import remote
From the documentation:
Note: If you want to test authenticated calls to your API using the Google API Explorer, you must also supply its client ID, which is available through the Endpoints library as endpoints.API_EXPLORER_CLIENT_ID.
Check your logs again to make sure the code actually runs. You're getting a 500 because of a Python exception being raise and there may be other issues with your code, though it doesn't seem like from what you've posted.

It works now that I list the endpoint handler before the other handlers.
This works:
handlers:
# Endpoint handler
- url: /_ah/spi/.*
script: endpoints.app
# Page handlers
- url: /.*
script: home.app
This doesn't work:
handlers:
# Page handlers
- url: /.*
script: home.app
# Endpoint handler
- url: /_ah/spi/.*
script: endpoints.app

The problem is that your python file can't find the import for:
from protorpc import remote
Therefore use the terminal, skip the GUI, navigate to the appengine sdk directory and put your project in there. For mac it is:
/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/

Is there any difference between splitting application in app.yaml or webapp2 URI router

Consider the two following scenarios:
There are two url handlers in app.yaml
handlers:
- url: /main
script: main.app1
- url: /secondary
script: secondary.app2
and URI router in main.py
app1 = webapp2.WSGIApplication([('/main', MainHandler)])
and another in secondary.py
app2 = webapp2.WSGIApplication([('/secondary', SecondaryHandler)])
vs
There is one url handler in app.yaml
handlers:
- url: /.*
script: main.app
and the URI router decides the handler
app = webapp2.WSGIApplication([
('/main', MainHandler),
('/secondary', SecondaryHandler)
])
Is there any difference in how the App Engine imports the two scenarios? If all requests are for MainHandler, does the App Engine import the files associated with SecondaryHandler at all in the first scenario or does an instance always import each handler when it is first initialized?
Obviously these are different ways to partition the application logically, but I'm asking if there are any associated performance considerations.

You can use a lazy handler in webapp2 to optimize loading and use a single app.
See this link : https://webapp2.readthedocs.io/en/latest/guide/routing.html#lazy-handlers

Google App Engine static pages Python 2.5 directories etc

I am new at this and am planning to move my stuff from a paid web service to GAE (nothing dynamic just static pages). Believe me I have spent countless hours trying to make this work but am at an impasse whereby I achieve one result at the exclusion of another and visa versa.
I am sure it is a simple answer and that I am violating some basic principles. What I want is that the app engine page (mygaeid.appspot.com) delivers a static landing page such that other pages are available with the addition of a suffix e.g. mygaeid.appspot.com/oranges.html mygaeid.appspot.com/grapes.html etc.
I am unable to achieve this such that I either am able to get the other pages when I add the suffix e.g. mygaeid.appspot.com/apples.html; mygaeid.appspot.com/oranges.html but not the landing page OR with a slightly different yaml the landing page (mygaeid.appspot.com) works but there is no access to the other pages (mygaeid.appspot.com/oranges.html etc) that have a suffix.
The py file (abcdefg.py) is below and is common to the two different yamls that follow:
import os
from google.appengine.ext import webapp
from google.appengine.ext.webapp import util
from google.appengine.ext.webapp import template
class MainHandler(webapp.RequestHandler):
def get (self, q):
if q is None:
q = 'htmdir/apples.html'
path = os.path.join (os.path.dirname (__file__), q)
self.response.headers ['Content-Type'] = 'text/html'
self.response.out.write (template.render (path, {}))
def main ():
application = webapp.WSGIApplication ([('/(.*html)?', MainHandler)], debug=True)
util.run_wsgi_app (application)
if __name__ == '__main__':
main ()
Using the following yaml the landing page (mygaeid.appspot.com) works perfectly (delivering the content of apples.html), but I cannot access the page if I add a suffix e.g. mygaeid.appspot.com/apples.html or mygaeid.appspot.com/static/htmdir/apples.html etc, as soon as I add the suffix it does not work. In the directory (htmdir) I have placed apples.html along with other html pages e.g. oranges.html etc and I cannot access any of them with this yaml.
application: mygaeid
version: 1
runtime: python
api_version: 1
handlers:
- url: /(.*\.(html))
static_files: static/htmdir/\1
upload: static/htmdir/(.*\.(html))
- url: /css
static_dir: css
- url: /js
static_dir: js
- url: /images
static_dir: images
- url: .*
script: abcdefg.py
If I modify the yaml as follows then the landing page (mygaeid.appspot.com) does not work but when I add the suffixes then it works perfectly e.g. mygaeid.appspot.com/apples.html; mygaeid.appspot.com/oranges.html etc deliver the appropriate pages:
- url: /(.*\.(html))
static_files: htmdir/\1
upload: htmdir/(.*\.(html))
Finally if I dispense with the directories altogether and using the same abcdefg.py (without the directory) the following very simple yaml actually delivers the results I want but is very unruly as all the files are stuffed in the root directory.
application: mygaeid
version: 1
runtime: python
api_version: 1
handlers:
- url: /(.*\.(png|js|css))
static_files: \1
upload: (.*\.(png|js|css))
- url: .*
script: abcedfg.py
any help would be much appreciated on figuring this out.
thanks
thanks wooble and thanks dave I went back yet again and carefully read the logs Wooble's solution works but I needed to put the htmdir (that contains the html) inside a directory called static. GAE is a great (and free) solution for static websites
your help and feedback is very much appreciated
SiteDirectory
-mygaeid
-static
-htmdir
-js
-css
-images
app.yaml
index.yaml
(py file was removed)

If you declare files as static in app.yaml, they are not available to your application's handlers.
However, if they're really static, using the django template engine to "render" them is kind of silly; just add mappings in app.yaml to display the static files, including one to display apples.html for /:
application: mygaeid
version: 1
runtime: python
api_version: 1
handlers:
- url: /(.*\.html)
static_files: static/htmdir/\1
upload: static/htmdir/.*\.html
- url: /css
static_dir: css
- url: /js
static_dir: js
- url: /images
static_dir: images
- url: /
static_files: static/htmdir/apples.html
upload: static/htmdir/apples\.html
(no python files needed)

Woobie has the right answer. Let me phrase it differently.
When you put .html in static_files, they're served by separate services that are dedicated to serving static content. Your app will not be able to see those files. The app can only see files that are resources.
Django templates must be resources, not static files. Otherwise, the application can't see them, and the template.render(path, ... will fail.
A next step to getting your problem solved is (if you haven't done so aleady) is to update your post to show your current app.yaml, and also to show us what's being reported up in the application's logs.

There are technical reasons why it works this way
The app.yaml config functions in a very simple top->bottom procedural manner.
Matching happens in the following order:
1. - url: /(.*\.(html))
static_files: static/htmdir/\1
upload: static/htmdir/(.*\.(html))
2. - url: /css
static_dir: css
3. - url: /js
static_dir: js
4. - url: /images
static_dir: images
5. - url: .*
script: abcdefg.py
To put it simply, if the file has a .html suffix it gets matched in step 1 and reroutes that request from mygaeid.appspot.com/.html to mygaeid.appspot.com/htmdir/.html. The *.html handler in step 5 never gets hit because all *.html routes are already spoken for.
In addition -- as the answers have already covered -- directories marked as static will not be available locally to your app handlers. I'll try to address the technical reasons why.
app.yaml acts as a configuration file for GAE's reverse proxy. Static files only change when they're uploaded so they're ideal for caching. By immediately pushing the static files to a cache server when they're deployed, GAE increases loading performance and removes unnecessary load from the app servers.
There's a good reason that static requests are counted separately and cost less than regular app requests. Every time you request a static file you're essentially pulling fetching the file from GAE's CDN. If you were to only fetch static files, then your server would have no reason to spool up in the first place.
The obvious downside to that approach is that those files don't physically exist on the same server as your app so you can't fetch or upload them directly in your app.
Note: Also following what other answers have covered. Don't mark your template folder as static. When your app goes to load the template -- instead of grabbing it from an adjacent directory -- it'll have to send out a web request and fetch the file from a remote location. Obviously, fetching a remote file instead of a local file is going to increase load time and latency.