Develop Reference

Flask: matplotlib's. pyplot.save_fig() gives file_not_found error for new path

I'm not sure if this is an issue with flask or an issue with matplotlib. I've been able to access images in html and javascript, but now I can't seem to do it in python.
I have a directory tree structure like the following
\---Neuroethics_Behavioral_Task # 'root' directory for my flask app. Contents are shown below
+---jspsych-6.0
| +---css
| ## cutting this directory off here, jspsych is just a framework I'm using, not relevant
+---static
| +---images
| +---plugins
| | \---template
| +---textdata
| \---trial_stimuli
+---templates
\---__pycache__
I've just been using a script called experiment.py that lives in the topmost directory of my flask app (so in the /Neuroethics_Behavioral_Task/ folder). It executes a function called "generate_stim_for_trials(trials)". Create_stimuli uses matplotlib to create a bunch of pngs, each representing stimuli, and it's supposed to save these images in the /static/images/ directory.
I'm omitting most of the code related to the stimulus creation for... "ownership/copyright" purposes. Also that code really shouldn't be relevant All that needs to be said is that I create a pandas dataframe and plot it in matplotlib.
def generate_stim_for_trials(trials):
import matplotlib
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import json
import os
lst_of_jsons_for_stimuli = []
stim_metadata_path = "/static/textdata/stimulus_data.json"
for each in range(trials):
stimuli_in_static = "images/stimulus_img" + str(each) + ".png"
stimuli_path = url_for('static', filename=stimuli_in_static)
# started ommitting code here...
df = # df variable gets created
df.plot(kind='barh', legend = False, stacked=True, color=['#e26a6a', '#3CB371', '#A9A9A9'])
plt.axis('off')
plt.margins(x=0)
plt.margins(y=0)
plt.grid(b=None)
plt.savefig(stimuli_path, bbox_inches='tight', pad_inches=-1) # error
The error I get is ultimately: FileNotFoundError: [Errno 2] No such file or directory: '/static/images/stimulus_img0.png'. I have the full error message but it's pretty long (and if moderators don't think it's useful, feel free to remove it/tell me to remove it).
Traceback (most recent call last):
File "D:\miniconda3\lib\site-packages\flask\app.py", line 2463, in __call__
return self.wsgi_app(environ, start_response)
File "D:\miniconda3\lib\site-packages\flask\app.py", line 2449, in wsgi_app
response = self.handle_exception(e)
File "D:\miniconda3\lib\site-packages\flask\app.py", line 1866, in handle_exce
ption
reraise(exc_type, exc_value, tb)
File "D:\miniconda3\lib\site-packages\flask\_compat.py", line 39, in reraise
raise value
File "D:\miniconda3\lib\site-packages\flask\app.py", line 2446, in wsgi_app
response = self.full_dispatch_request()
File "D:\miniconda3\lib\site-packages\flask\app.py", line 1951, in full_dispat
ch_request
rv = self.handle_user_exception(e)
File "D:\miniconda3\lib\site-packages\flask\app.py", line 1820, in handle_user
_exception
reraise(exc_type, exc_value, tb)
File "D:\miniconda3\lib\site-packages\flask\_compat.py", line 39, in reraise
raise value
File "D:\miniconda3\lib\site-packages\flask\app.py", line 1949, in full_dispat
ch_request
rv = self.dispatch_request()
File "D:\miniconda3\lib\site-packages\flask\app.py", line 1935, in dispatch_re
quest
return self.view_functions[rule.endpoint](**req.view_args)
File "D:\Projects\CogNeW\Neuroethics_Behavioral_Task\experiment.py", line 21,
in exp_js_psych_app
generate_stim_for_trials(total_trials)
File "D:\Projects\CogNeW\Neuroethics_Behavioral_Task\experiment.py", line 93,
in generate_stim_for_trials
plt.savefig(stimuli_path, bbox_inches='tight', pad_inches=-1)
File "D:\miniconda3\lib\site-packages\matplotlib\pyplot.py", line 722, in save
fig
res = fig.savefig(*args, **kwargs)
File "D:\miniconda3\lib\site-packages\matplotlib\figure.py", line 2180, in sav
efig
self.canvas.print_figure(fname, **kwargs)
File "D:\miniconda3\lib\site-packages\matplotlib\backends\backend_qt5agg.py",
line 88, in print_figure
super().print_figure(*args, **kwargs)
File "D:\miniconda3\lib\site-packages\matplotlib\backend_bases.py", line 2082,
in print_figure
**kwargs)
File "D:\miniconda3\lib\site-packages\matplotlib\backends\backend_agg.py", lin
e 530, in print_png
cbook.open_file_cm(filename_or_obj, "wb") as fh:
File "D:\miniconda3\lib\contextlib.py", line 81, in __enter__
return next(self.gen)
File "D:\miniconda3\lib\site-packages\matplotlib\cbook\__init__.py", line 447,
in open_file_cm
fh, opened = to_filehandle(path_or_file, mode, True, encoding)
File "D:\miniconda3\lib\site-packages\matplotlib\cbook\__init__.py", line 432,
in to_filehandle
fh = open(fname, flag, encoding=encoding)
FileNotFoundError: [Errno 2] No such file or directory: '/static/images/stimulus
_img0.png'
Again, this python file experiment.py exists at the topmost level of the flask module, in the "neuroethics_behavioral_task" directory. So it should be able to understand the path, /static/images since the static folder is also at that same level. But it doesn't.
Furthermore, here I'm trying to create/save a new file called stimulus_img0.png. So indeed, this file shouldn't exist yet -- it's getting created.
I can't tell if this is a matplotlib or flask issue, and I"m not sure how to resolve it because I'm not sure where my python program is actually looking at this point.
Edit 1
Well the 'mystery' is solved.
You know how you're supposed to do flask run or python -m flask run from a directory above your flask app?
If I just make the line plt.savefig('stimuli_img0.png'), the image(s) get saved in the directory above my flask application, aka the directory from which I'm executing python -m flask run.
So now my question is... What actually controls where python scripts in Flask save to? Is this actually the default behavior? I haven't encountered this in the html or js files in this flask app.

Authentication Error when using Flask to connect to ParseServer

What I am trying to achieve is pretty simple.
I want to use Flask to create a web app that connects to a remote Server via API calls (specifically ParseServer).
I am using a third-party library to achieve this and everything works perfectly when I am running my code in a stand-alone script. But when I add my code into the Flask I suddenly can't authenticate with the Server
Or to be more precise I get an 'unauthorized' error when executing an API call.
It seems to me that in Flask, the registration method used by the APi library is not remembered.
I tried many things of putting the registration and initialization code in different places in Flask, nothing worked.
I asked a similar question in the Github of the Library with no help.
So I guess I have two questions that could help me solve this
1) Where should I put a registration method and import of the files from this library?
&
2) What can I do to identify the issue specifically, eg. to know precisely what's wrong?
Here's some code
The Flask code is here
#app.route('/parseinsert')
def run_parse_db_insert():
"""The method for testing implementation and design of the Parse Db
"""
pc = ParseCommunication()
print(pc.get_all_names_rating_table())
return 'done'
The ParseCommunication is my Class that deals with Parse. If I run ParseCommunication from that script, with the same code as above in the main part, everything works perfectly.
I run the Flask app with dev_appserver.py from Google App Engine.
My folder structure
/parseTest
/aplication
views.py
app.yaml
run.py
My run.py code
import os
import sys
sys.path.insert(1, os.path.join(os.path.abspath('.'), 'lib'))
sys.path.insert(1, os.path.join(os.path.abspath('.'), 'application'))
import aplication
Let me know what else I could provide to help out.
Thank you in Advance
EDIT:
A stack trace as requested.
It's mostly related to the library (from what I can say?)
ERROR 2016-09-28 06:45:50,271 app.py:1587] Exception on /parseinsert [GET]
Traceback (most recent call last):
File "/home/theshade/Devel/ParseBabynames/parseTest/lib/flask/app.py", line 1988, in wsgi_app
response = self.full_dispatch_request()
File "/home/theshade/Devel/ParseBabynames/parseTest/lib/flask/app.py", line 1641, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/theshade/Devel/ParseBabynames/parseTest/lib/flask/app.py", line 1544, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/theshade/Devel/ParseBabynames/parseTest/lib/flask/app.py", line 1639, in full_dispatch_request
rv = self.dispatch_request()
File "/home/theshade/Devel/ParseBabynames/parseTest/lib/flask/app.py", line 1625, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/theshade/Devel/ParseBabynames/parseTest/aplication/views.py", line 34, in run_parse_db_insert
name = pc.get_user('testuser1')
File "/home/theshade/Devel/ParseBabynames/parseTest/aplication/parseCommunication.py", line 260, in get_user
return User.Query.get(username=uname)
File "/home/theshade/Devel/ParseBabynames/parseTest/lib/parse_rest/query.py", line 58, in get
return self.filter(**kw).get()
File "/home/theshade/Devel/ParseBabynames/parseTest/lib/parse_rest/query.py", line 150, in get
results = self._fetch()
File "/home/theshade/Devel/ParseBabynames/parseTest/lib/parse_rest/query.py", line 117, in _fetch
return self._manager._fetch(**options)
File "/home/theshade/Devel/ParseBabynames/parseTest/lib/parse_rest/query.py", line 41, in _fetch
return [klass(**it) for it in klass.GET(uri, **kw).get('results')]
File "/home/theshade/Devel/ParseBabynames/parseTest/lib/parse_rest/connection.py", line 108, in GET
return cls.execute(uri, 'GET', **kw)
File "/home/theshade/Devel/ParseBabynames/parseTest/lib/parse_rest/connection.py", line 102, in execute
raise exc(e.read())
ResourceRequestLoginRequired: {"error":"unauthorized"}

Parse requires keys and env variables. Check this line:
API_ROOT = os.environ.get('PARSE_API_ROOT') or 'https://api.parse.com/1'
Your error is in line 102 at:
https://github.com/milesrichardson/ParsePy/blob/master/parse_rest/connection.py
Before you can parse, you need to register:
from parse_rest.connection import register
APPLICATION_ID = '...'
REST_API_KEY = '...'
MASTER_KEY = '...'
register(APPLICATION_ID, REST_API_KEY, master_key=MASTER_KEY)

"Random" SocketError/Connection Refused errors on py2neo queries

Hullo, hope this doesn't end up being too trivial.
The relevant parts of my stack are Gunicorn/Celery, neomodel (0.3.6), and py2neo (1.5). Neo4j version is 1.9.4, bound on 0.0.0.0:7474 (all of this is on linux, Ubuntu 13.04 I think)
So my gunicorn/celery servers are fine most of the time, except occasionally, I get the following error:
ConnectionRefusedError(111, 'Connection refused')
Stacktrace (most recent call last):
File "flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "flask/_compat.py", line 33, in reraise
raise value
File "flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "Noomsa/web/core/util.py", line 156, in inner
user = UserMixin().get_logged_in()
File "Noomsa/web/core/util.py", line 117, in get_logged_in
user = models.User.index.get(username=flask.session["user"])
File "neomodel/index.py", line 50, in get
nodes = self.search(query=query, **kwargs)
File "neomodel/index.py", line 41, in search
return [self.node_class.inflate(n) for n in self._execute(str(query))]
File "neomodel/index.py", line 28, in _execute
return self.__index__.query(query)
File "py2neo/neo4j.py", line 2044, in query
self.__uri__, quote(query, "")
File "py2neo/rest.py", line 430, in _send
raise SocketError(err)
So, as you can see, I do a call to User.index.get (The first call in the request response), and get a socket error. Sometimes. Most of the time, it connects fine. The error occurs amongst all Flask views/Celery tasks that use the neo4j connection (and not just doing User.index.get ;)).
So far, the steps I've taken have involved moneky patching the neomodel connection function to check that the GraphDatabaseService object is created per thread, and to automatically reconnect (and authenticate) to the neo4j server every 30 or so seconds. This may have reduced the frequency of the errors, but they still occur.
Looking for the error online, it seems to be mostly people trying to connect to the wrong interface/ip/port. However, given that the majority of my requests go through, I don't feel like that is the case here.
Any ideas? I don't think it's related, but my database seems to have 38k orphaned nodes; that's probably worthy of another question in its own right.
EDIT: I should add, this seems to disappear when running gunicorn/celery with workers=1, instead of workers=$CPU_N. Can't see why it should matter, as apparently neo4j is set up to handle $N_CPU*10 connections by default.

This looks like a networking or web stack configuration problem so I don't think I can help from a py2neo perspective. I'd recommend upgrading to py2neo 1.6 though as the client HTTP code has been completely rewritten and it might handle a reconnection more gracefully.

Flask : TypeError: 'str' object is not callable

I have a generated flask application that is giving me this traceback:
Traceback (most recent call last):
File "/home/.virtualenvs/j/lib/python2.7/site-packages/flask/app.py", line 1701, in __call__
return self.wsgi_app(environ, start_response)
File "/home/.virtualenvs/j/lib/python2.7/site-packages/flask/app.py", line 1689, in wsgi_app
response = self.make_response(self.handle_exception(e))
File "/home/.virtualenvs/j/lib/python2.7/site-packages/flask/app.py", line 1687, in wsgi_app
response = self.full_dispatch_request()
File "/home/.virtualenvs/j/lib/python2.7/site-packages/flask/app.py", line 1360, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/.virtualenvs/j/lib/python2.7/site-packages/flask/app.py", line 1358, in full_dispatch_request
rv = self.dispatch_request()
File "/home/.virtualenvs/j/lib/python2.7/site-packages/flask/app.py", line 1344, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
TypeError: 'str' object is not callable
but I cannot determine why or what is causing this at this point. The app starts and appears to run, but chokes on this traceback anytime I try to visit a route.
What is this and what would be causing this? I don't what object is a str and why it is not callable. This is the first I've seen something like this.

That means that self.view_functions[rule.endpoint] is a string, not a function, thus cannot be called (with (...) operator). That said, it is difficult to tell what the issue without your code: Flask expects a function to be called and it gets a string. Did you use the #app.route decorator?
Try printing the value of self.view_functions[rule.endpoint] and see what comes out.

This is, of course, one of these things that can have multiple different fixes (without seeing all of the original source code).
I got the same error in the HTML template that you did:
TypeError: 'str' object is not callable
My fix was to delete my (foolish) attempt to obtain the original request variables:
my_form = request.form
and just assign them by hand:
my_form.blah = request.form['blah']
Interestingly, the my_form = request.form executed without a problem, it only surfaced as an error in the HTML template.

Trouble with Cork authentication for Bottle app: TypeError("'NoneType' object does not support item assignment",)

I'm having some trouble implementing the Cork library with a Bottle app. I've tried setting up the example app, and I made some minor modifications to get it to run in an WSGI container. The main change is that I changed this:
if __name__ == "__main__":
main()
to this:
application = default_app()
When I authenticate with the admin user, I get the following:
Traceback (most recent call last):
File "/path_to_app/lib/python2.7/bottle-0.11.4-py2.7.egg/bottle.py", line 763, in _handle
return route.call(**args)
File "/path_to_app/lib/python2.7/bottle-0.11.4-py2.7.egg/bottle.py", line 1572, in wrapper
rv = callback(*a, **ka)
File "/path_to_app/cork-example/simple_webapp.py", line 47, in login
aaa.login(username, password, success_redirect='/', fail_redirect='/login')
File "build/bdist.linux-x86_64/egg/cork/cork.py", line 209, in login
self._setup_cookie(username)
File "build/bdist.linux-x86_64/egg/cork/cork.py", line 610, in _setup_cookie
session['username'] = username
TypeError: 'NoneType' object does not support item assignment
When I attempt to view the main page, I get this error:
Traceback (most recent call last):
File "/path_to_app/lib/python2.7/bottle-0.11.4-py2.7.egg/bottle.py", line 763, in _handle
return route.call(**args)
File "/path_to_app/lib/python2.7/bottle-0.11.4-py2.7.egg/bottle.py", line 1572, in wrapper
rv = callback(*a, **ka)
File "/path_to_app/cork-example/simple_webapp.py", line 90, in index
aaa.require(fail_redirect='/login')
File "build/bdist.linux-x86_64/egg/cork/cork.py", line 267, in require
cu = self.current_user
File "build/bdist.linux-x86_64/egg/cork/cork.py", line 417, in current_user
username = session.get('username', None)
AttributeError: 'NoneType' object has no attribute 'get'
Is there a newbie mistake I might be making? Or is something else wrong?

I figured it out. It was a newbie mistake. The Bottle documentation says that if you want to use Bottle with mod_wsgi, toward the end of the file, you should use this:
application = default_app()
After getting a better understanding of how this all works, I realized I was sending the wrong application object to mod_wsgi. Instead, I should have used this:
application = app #app was defined earlier in the example application