Develop Reference

How to rename Flask WSGI App during development?

I would like to know how can I change the name of the Flask WSGI App during the development stage.
Using the Flask Mega-Tutorial as reference, I was able to successfully setup a "Hello World" app.
Digressions from the tutorial:
Use pipenv as my Python virtual environment manager (instead of venv)
Name of the app is astronomer.py.
Now, I want to build on top of the existing app and customize the code to my requirements; starting with the app name that I have defined in the .flaskenv file as FLASK_APP env var.
Accordingly, I have updated the name of the root level Python script from astronomer.py (in the tutorial) to galielo.py (for my use). After changing the corresponding value of FLASK_APP and restarting the flask server via $ pipenv run flask run, the app crashes with the following error:
$ pipenv run flask run [12:29:33]
* Serving Flask app "astronomer.py" (lazy loading)
* Environment: development
* Debug mode: on
* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
* Restarting with stat
* Debugger is active!
* Debugger PIN: 302-012-958
127.0.0.1 - - [09/Oct/2019 12:29:57] "GET / HTTP/1.1" 500 -
Traceback (most recent call last):
File "/Users/kshitij10496/.local/share/virtualenvs/galileo-iQPdbs28/lib/python3.7/site-packages/flask/cli.py", line 240, in locate_app
__import__(module_name)
ModuleNotFoundError: No module named 'astronomer'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/kshitij10496/.local/share/virtualenvs/galileo-iQPdbs28/lib/python3.7/site-packages/flask/cli.py", line 338, in __call__
self._flush_bg_loading_exception()
File "/Users/kshitij10496/.local/share/virtualenvs/galileo-iQPdbs28/lib/python3.7/site-packages/flask/cli.py", line 326, in _flush_bg_loading_exception
reraise(*exc_info)
File "/Users/kshitij10496/.local/share/virtualenvs/galileo-iQPdbs28/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/Users/kshitij10496/.local/share/virtualenvs/galileo-iQPdbs28/lib/python3.7/site-packages/flask/cli.py", line 314, in _load_app
self._load_unlocked()
File "/Users/kshitij10496/.local/share/virtualenvs/galileo-iQPdbs28/lib/python3.7/site-packages/flask/cli.py", line 330, in _load_unlocked
self._app = rv = self.loader()
File "/Users/kshitij10496/.local/share/virtualenvs/galileo-iQPdbs28/lib/python3.7/site-packages/flask/cli.py", line 388, in load_app
app = locate_app(self, import_name, name)
File "/Users/kshitij10496/.local/share/virtualenvs/galileo-iQPdbs28/lib/python3.7/site-packages/flask/cli.py", line 250, in locate_app
raise NoAppException('Could not import "{name}".'.format(name=module_name))
flask.cli.NoAppException: Could not import "astronomer".
Debugging
After logging into the virtual env and checking for the value of the env var FLASK_APP, I get the old value of astronomer.py. This explains why the application is not starting. However, I'm not able to understand why this is happening?
I even tried using "eager-loading" the app using: $ pipenv run flask run --eager-loading
Still, the app does not start with the same error message ofcourse.
I was able to solve this manually by unsetting the env var FLASK_APP from within the virtual env and restarting the flask server. I'm curious to know about why the app is not loading the file .flaskenv at initialization and if there is an automated way to do this?

With Pipenv, I think things are a little bit different when it comes to environment variables. As per the documentation, there is a builtin mechanism for loading a .env file:
If a .env file is present in your project, $ pipenv shell and $ pipenv run will automatically load it, for you
So I guess you should rename your file from .flaskenv to .env and then safely remove the python-dotenv dependency.

Google Python cloud-dataflow instances broke without new deployment (failed pubsub import)

I have defined a few different Cloud Dataflow jobs for Python in the Google AppEngine Flex Environment. I have defined my requirements in a requirements.txt file, included my setup.py file, and everything was working just fine. My last deployment was on May 3rd, 2018. Looking through logs, I see that one of my jobs began failing on May 22nd, 2018. The job fails with a stack trace resulting from a bad import, seen below.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 582, in do_work
work_executor.execute()
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py", line 166, in execute
op.start()
File "apache_beam/runners/worker/operations.py", line 294, in apache_beam.runners.worker.operations.DoOperation.start (apache_beam/runners/worker/operations.c:10607)
def start(self):
File "apache_beam/runners/worker/operations.py", line 295, in apache_beam.runners.worker.operations.DoOperation.start (apache_beam/runners/worker/operations.c:10501)
with self.scoped_start_state:
File "apache_beam/runners/worker/operations.py", line 300, in apache_beam.runners.worker.operations.DoOperation.start (apache_beam/runners/worker/operations.c:9702)
pickler.loads(self.spec.serialized_fn))
File "/usr/local/lib/python2.7/dist-packages/apache_beam/internal/pickler.py", line 225, in loads
return dill.loads(s)
File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 277, in loads
return load(file)
File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 266, in load
obj = pik.load()
File "/usr/lib/python2.7/pickle.py", line 858, in load
dispatch[key](self)
File "/usr/lib/python2.7/pickle.py", line 1090, in load_global
klass = self.find_class(module, name)
File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 423, in find_class
return StockUnpickler.find_class(self, module, name)
File "/usr/lib/python2.7/pickle.py", line 1124, in find_class
__import__(module)
File "/usr/local/lib/python2.7/dist-packages/dataflow_pipeline/tally_overages.py", line 27, in <module>
from google.cloud import pubsub
File "/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub.py", line 17, in <module>
from google.cloud.pubsub_v1 import PublisherClient
File "/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/__init__.py", line 17, in <module>
from google.cloud.pubsub_v1 import types
File "/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/types.py", line 26, in <module>
from google.iam.v1.logging import audit_data_pb2
ImportError: No module named logging
So the main issue seems to come from the pubsub dependency relying on importing google.iam.v1.logging, which is installed from grpc-google-iam-v1.
Here is my requirements.txt file
Flask==0.12.2
apache-beam[gcp]==2.1.1
gunicorn==19.7.1
google-cloud-dataflow==2.1.1
google-cloud-datastore==1.3.0
pytz
google-cloud-pubsub
google-gax
grpc-google-iam-v1
googleapis-common-protos
google-cloud==0.32
six==1.10.0
protobuf
I am able to run everything locally just fine by doing the following from my project.
$ virtualenv --no-site-packages .
$ . bin/activate
$ pip install --ignore-installed -r requirements.txt
$ python main.py
No handlers could be found for logger "oauth2client.contrib.multistore_file"
INFO:werkzeug: * Running on http://0.0.0.0:8080/ (Press CTRL+C to quit)
INFO:werkzeug: * Restarting with stat
No handlers could be found for logger "oauth2client.contrib.multistore_file"
WARNING:werkzeug: * Debugger is active!
INFO:werkzeug: * Debugger PIN: 317-820-645
specifically, I am able to do the following locally just fine
$ python
>>> from google.cloud import pubsub
>>> import google.iam.v1.logging
>>> google.iam.v1.logging.__file__
'/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/google/iam/v1/logging/__init__.pyc'
So I know that the installation of the grpc-google-iam-v1 package is working just fine locally.. the required files are there.
My questions are
Why is the install of grpc-google-iam-v1 on the Google AppEngine Flex Environment not installing all of the files correctly? I must be missing the /site-packages/google/iam/v1/logging directory.
Why would this randomly start failing? I didn't do any more deploys, the same code was running and working on the 21st and then it broke on the 22nd on May.
I was able to get the pipeline running again after changing the requirements.txt file to
Flask==0.12.2
apache-beam[gcp]
google-cloud-dataflow
gunicorn==19.7.1
google-cloud-datastore==1.3.0
pytz
google-cloud-pubsub
google-gax
grpc-google-iam-v1
googleapis-common-protos
google-cloud==0.32
six==1.10.0
protobuf
so simply removing the version requirements from apache-beam[gcp] and google-cloud-dataflow did the trick.

Building on the solution provided by John Allard, removing the version from the requirements.txt will automatically default to the latest version. Thus, with no version specified for apache-beam[gcp] , google-cloud-dataflow and google-cloud-pubsub they will all run on the latest version and solve the dependency issue. The requirements.txt will look like the following:
Flask==0.12.2
apache-beam[gcp]
gunicorn==19.7.1
google-cloud-dataflow
google-cloud-datastore==1.3.0
pytz
google-cloud-pubsub
google-gax
grpc-google-iam-v1
googleapis-common-protos
google-cloud==0.32
six==1.10.0
protobuf

Flask import is failing in standard environment of Google App Engine

I have specified an app.yaml file as follows:
runtime: python
env: standard
# this assumes that the entrypoint is app.py
entrypoint: gunicorn -b :$PORT main:app
threadsafe: true
After running gcloud app deploy I'm faced with a 500 error and the logs reveal the following
Traceback (most recent call last):
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "/base/data/home/apps/g~test-website/20170924t080410.404328292497306090/app.py", line 3, in <module>
from flask import Flask, render_template
ImportError: No module named flask
Can anyone help me configure a standard environment

You are combining syntax for the flexible environment and the standard environment in your app.yaml
Look at the documentation for the app.yaml and stick to the standard environment reference. The env parameter is only used for flex and entrypoint is only for flex. The runtime parameter should be python27 as python means the deprecated Python 2.5 runtime.
Given that you're trying to use Flask, you should probably read the getting started guide that is specifically for Flask.

Flask debug depends on tkinter, sometimes

When I try to import non-standard modules into my Flask app with debug mode on and run it locally, the Flask server crashes with an ImportError for _tkinter. If I remove the import for the non-standard module, or turn off debug mode, everything works as expected.
Examples
The following runs fine, and "Hello, World" can be seen at localhost:5000
from flask import Flask
app = Flask(__name__)
#app.route("/")
def home():
return "Hello, World"
if __name__ == '__main__':
app.run(debug=True)
If I add the following line at the top of the file
import dateparser
I get the following Traceback when I run the file:
flask#ubuntu:~/tkerr$ python app.py
* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
* Restarting with stat
Traceback (most recent call last):
File "app.py", line 10, in <module>
app.run(debug=True)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 772, in run
run_simple(host, port, self, **options)
File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 622, in run_simple
reloader_type)
File "/usr/local/lib/python2.7/dist-packages/werkzeug/_reloader.py", line 269, in run_with_reloader
reloader.run()
File "/usr/local/lib/python2.7/dist-packages/werkzeug/_reloader.py", line 159, in run
for filename in chain(_iter_module_files(), self.extra_files):
File "/usr/local/lib/python2.7/dist-packages/werkzeug/_reloader.py", line 70, in _iter_module_files
for package_path in getattr(module, '__path__', ()):
File "/usr/lib/python2.7/dist-packages/six.py", line 116, in __getattr__
_module = self._resolve()
File "/usr/lib/python2.7/dist-packages/six.py", line 105, in _resolve
return _import_module(self.mod)
File "/usr/lib/python2.7/dist-packages/six.py", line 76, in _import_module
__import__(name)
File "/usr/lib/python2.7/lib-tk/tkCommonDialog.py", line 11, in <module>
from Tkinter import *
File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 42, in <module>
raise ImportError, str(msg) + ', please install the python-tk package'
ImportError: No module named _tkinter, please install the python-tk package
For confirmation, with debug mode turned off everything runs as expected, even if I use the dateparser module. For example, the following shows "2015-01-01 00:00:00"
import dateparser
from flask import Flask
app = Flask(__name__)
#app.route("/")
def home():
return str(dateparser.parse("1 jan"))
if __name__ == '__main__':
app.run()
Importing any standard Python modules, such as
import json
works fine. But any other modules installed through pip cause the same issue.
If I run (as prompted by the error)
sudo apt-get install python-tk
Then the Traceback is replaced with an ImportError for _winreg
flask#ubuntu:~/tkerr$ python app.py
* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
* Restarting with stat
Traceback (most recent call last):
File "app.py", line 11, in <module>
app.run(debug=True)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 772, in run
run_simple(host, port, self, **options)
File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 622, in run_simple
reloader_type)
File "/usr/local/lib/python2.7/dist-packages/werkzeug/_reloader.py", line 269, in run_with_reloader
reloader.run()
File "/usr/local/lib/python2.7/dist-packages/werkzeug/_reloader.py", line 159, in run
for filename in chain(_iter_module_files(), self.extra_files):
File "/usr/local/lib/python2.7/dist-packages/werkzeug/_reloader.py", line 70, in _iter_module_files
for package_path in getattr(module, '__path__', ()):
File "/usr/lib/python2.7/dist-packages/six.py", line 116, in __getattr__
_module = self._resolve()
File "/usr/lib/python2.7/dist-packages/six.py", line 105, in _resolve
return _import_module(self.mod)
File "/usr/lib/python2.7/dist-packages/six.py", line 76, in _import_module
__import__(name)
ImportError: No module named _winreg
Details
Python 2.7.7; Ubuntu 14.04 (VMWare VM); Flask/Werkzeug 0.10.1
A solution would be appreciated, but more so I'd love to gain insight into what could cause Flask to require tkinter, considering it displays all debug output from within the web browser.
edit Add output from pip freeze
Flask==0.10.1
Jinja2==2.7.3
MarkupSafe==0.23
PAM==0.4.2
Pillow==2.3.0
PyMySQL==0.6.6
PyYAML==3.11
Twisted-Core==13.2.0
Twisted-Web==13.2.0
Werkzeug==0.10.1
adium-theme-ubuntu==0.3.4
apt-xapian-index==0.45
argparse==1.2.1
beautifulsoup4==4.4.1
ccsm==0.9.11.3
chardet==2.0.1
colorama==0.2.5
command-not-found==0.3
compizconfig-python==0.9.11.3
dateparser==0.3.0
debtagshw==0.1
defer==1.0.6
dirspec==13.10
duplicity==0.6.23
feedparser==5.1.3
html5lib==0.999
httplib2==0.8
itsdangerous==0.24
lockfile==0.8
lxml==3.3.3
oauthlib==0.6.1
oneconf==0.3.7
pexpect==3.1
piston-mini-client==0.7.5
pyOpenSSL==0.13
pycrypto==2.6.1
pycups==1.9.66
pycurl==7.19.3
pygobject==3.12.0
pyserial==2.6
pysmbc==1.0.14.1
python-apt==0.9.3.5
python-dateutil==2.4.2
python-debian==0.1.21-nmu2ubuntu2
pyxdg==0.25
reporter==0.1.2
reportlab==3.0
requests==2.2.1
sessioninstaller==0.0.0
six==1.5.2
software-center-aptd-plugins==0.0.0
system-service==0.1.6
unity-lens-photos==1.0
urllib3==1.7.1
virtualenv==1.11.4
wsgiref==0.1.2
xdiagnose==3.6.3build2
youtube-dl==2014.02.17
zope.interface==4.0.5

I just tried running the following code on my Windows 10 machine and it worked fine for me.
Flask-Test.py
import dateparser
from flask import Flask
app = Flask(__name__)
#app.route("/")
def home():
return str(dateparser.parse("1 jan"))
if __name__ == '__main__':
app.run(debug=True)
I think there may be an issue with the python environment that you're currently working in. Perhaps try creating a virtual environment to operate out of using virtualenv to make sure that your environment is setup correctly.
If you would like to following along with the steps that I took to setup up a virtual environment to test the code, see below. The commands used might be slightly different when running from Ubuntu.
pip install virtualenv
Navigate to the folder where you have your test code saved and create the Virtual Environment using the following command
virtualenv env
Activate the virtual environment by navigating to the 'activate' file. In my case it was located at
.\venv\Scripts\activate
Download Flask and dateparser using pip
pip install Flask
pip install dateparser
Finally, activate the test file through the virtual environment
python .\Flasky-Test.py
Navigating to localhost returned a value of 2015-01-01 00:00:00 for me ok.
Edit
I just tried running your list of packages and managed to replicate the error that you were receiving. I believe the problem might be with the version of six that you are using. Try running
pip install six --upgrade
And let me know if that fixes the problem for you - it did for me.

Import Errors in Virtualenv

I'm trying to run python setup.py a small script on a Linode instance where I'm running Flask + sqlalchemy (mysql).
#filename - setup.py
from daaru import db
def init_db():
db.drop_all()
db.create_all()
init_db()
This is the error I'm getting -
Traceback (most recent call last):
File "setup.py", line 21, in <module>
init_db()
File "setup.py", line 9, in init_db
db.drop_all()
File "/home/captain/public/daaru/v1/local/lib/python2.7/site-packages/flask_sqlalchemy.py", line 830, in drop_all
self._execute_for_all_tables(app, bind, 'drop_all')
File "/home/captain/public/daaru/v1/local/lib/python2.7/site-packages/flask_sqlalchemy.py", line 814, in _execute_for_all_tables
op(bind=self.get_engine(app, bind), tables=tables)
File "/home/captain/public/daaru/v1/local/lib/python2.7/site-packages/flask_sqlalchemy.py", line 763, in get_engine
return connector.get_engine()
File "/home/captain/public/daaru/v1/local/lib/python2.7/site-packages/flask_sqlalchemy.py", line 443, in get_engine
self._engine = rv = sqlalchemy.create_engine(info, **options)
File "/home/captain/public/daaru/v1/local/lib/python2.7/site-packages/sqlalchemy/engine/__init__.py", line 338, in create_engine
return strategy.create(*args, **kwargs)
File "/home/captain/public/daaru/v1/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 64, in create
dbapi = dialect_cls.dbapi(**dbapi_args)
File "/home/captain/public/daaru/v1/local/lib/python2.7/site-packages/sqlalchemy/connectors/mysqldb.py", line 52, in dbapi
return __import__('MySQLdb')
ImportError: No module named MySQLdb
The funny part is that when I log into the server, activate the virtual env (named v1), and run init_db in ipython everything works as expected. But when I run it as a script, it gives me an ImportError for the module named MySQLdb. MySQLdb has a systemwide installation using apt-get install python-mysqldb as this package is not available via pip.
What am I doing wrong here?
Running pip freeze on the server shows me this:
Flask==0.9
Flask-SQLAlchemy==0.16
Jinja2==2.6
SQLAlchemy==0.7.9
Werkzeug==0.8.3
argparse==1.2.1
wsgiref==0.1.2
Is the error coming because mysqldb package is not found in the venv? Why isn't then the systemwide installation of that package being used?

I used this blog post to install mysql-python in my venv. Everything works likes a charm now. Thanks.
I've also added a fabfile to automate deployment -
from fabric.api import *
env.user = "host"
env.hosts = ["hostname"]
env.directory = "/home/captain/public/daaru"
env.activate = "source /home/captain/public/daaru/v1/bin/activate"
def deploy():
with cd(env.directory):
run("git pull")
run("sudo service apache2 reload")
def virtualenv(command):
with cd(env.directory), prefix(env.activate):
run(command)
def populate_db():
virtualenv("python setup.py")
def freeze():
""" command for testing virtualenv """
virtualenv("pip freeze")

I just ran into this, and the simple, dirty workaround for me was to cp -r /usr/lib/python2.7/dist-packages/*mysql* $VIRTUALENV/lib/python2.7/dist-packages/ and cp -r /usr/lib/python2.7/dist-packages/*MySQL* $VIRTUALENV/lib/python2.7/dist-packages/, which picked up all needed parts.
Looking into it more it turned out that pip install MySQL-python failed in the virtualenv due to libmysqlclient-dev not being installed via apt. Hope this helps.