A data file was getting corrupted when I terminated the program and realised that it was never properly closed.
It is quite critical that it does not get corrupted. So I added a statement to close the file.
Now, it seems like the file gets opened twice and then closed. That's one operation too many. There are of course many read-write operations in-between but it should only open and close the files once.
Here is what I have done to the standarize web.py template:
import web
import pandas as pd
store = pd.HDFStore('data_file.h5')
urls = (
'/', 'index'
)
class index:
def __init__(self):
self.__df = store['df']
def GET(self):
# several read-write, and modify operations on self.__df
return "Hello, world!"
if __name__ == "__main__":
try:
app = web.application(urls, globals())
app.run()
finally:
store.close()
Now, if I move the line which opens the store down inside the try statement at the bottom, it complains since it compiles the class but I can't find the variable store.
I tried initialising store with None at the top but it didn't work either. Then I tried putting that line up at the top in the function and calling it from the bottom, however, that didn't bring it into scope.
I was thinking of making it a global variable, which would probably do the trick, is that the right approach?
See web.py running twice. As mentioned there, avoid using globals as they don't do what you think they do... app.py runs twice, once on startup and a second time within web.appplication(urls, globals()). If you set autoreload=False in web.applications() call, it won't load the file twice.
Another solution is to attach your store to web.config, which is globally available.
if __name__ == "__main__":
try:
web.config.store = pd.HDFStore('data_file.h5')
app = web.application(urls, globals())
app.run()
finally:
web.config.store.close()
...and reference that global in your __init__
class index:
def __init__(self):
self.__df = web.config.store['df']
Related
This question already has an answer here:
How to process requests from multiiple users using ML model and FastAPI?
(1 answer)
Closed 15 days ago.
I have a Python file named main.py. I am running it on Python 3.9.13 on Windows.
import uvicorn
from fastapi import FastAPI
app = FastAPI()
#app.post('/c')
async def c(b: str):
print(a)
if __name__ == '__main__':
a = load_embeddings('embeddings')
uvicorn.run('main:app', host='127.0.0.1', port=80)
Running this, then invoking POST /c will cause a 500 error with NameError 'a' is not defined.
However it is obvious that a will be defined first before the server is ran. If I move a outside of the if __name__ == '__main__': then it works, but it causes load_embeddings to be ran multiple times for unknown reasons (3 exact). Since load_embeddings for me takes long time, I do not want the duplicate execution.
I wish to look for either of these as a solution to my issue: stop whatever outside if __name__ == '__main__': from executing multiple times, OR make a defined globally when it is being defined under if __name__ == '__main__':.
Note: variable names are intentionally renamed for ease of reading. Please do not advise me anything on coding style/naming conventions. I know the community is helpful but that's not the point here, thanks.
You can resolve the issue by moving the a variable definition inside the c function. Then, you can add a check inside the function to load the embeddings only if they have not been loaded yet. You can achieve this by using a global variable, which will keep track of whether the embeddings have been loaded or not.
Here is an example:
import uvicorn
from fastapi import FastAPI
app = FastAPI()
EMBEDDINGS_LOADED = False
def load_embeddings(filename):
# Load embeddings code here
...
#app.post('/c')
async def c(b: str):
global EMBEDDINGS_LOADED
if not EMBEDDINGS_LOADED:
load_embeddings('embeddings')
EMBEDDINGS_LOADED = True
print(a)
if __name__ == '__main__':
uvicorn.run('main:app', host='127.0.0.1', port=80)
So I am trying to create an API that constantly reads from a CSV and returns information about it when requested. So far, I have created a flask API that reads the CSV file once and returns correctly. However, I can't seem to make it constantly update. My working code is something like this.
app = flask.Flask(__name__)
app.config["DEBUG"] = True
dfchat = pd.read_csv(path)
escaper = None
# for now, this is just to make sure the program keeps running even if there is an error
def escape_route():
global escaper
while escaper != "Y":
escaper = str(input("Exit now? Enter \'Y\': \n")).strip()
os._exit(os.X_OK)
def sample_function(dfchat):
#app.route('/sample_text', methods=['GET'])
def sample_endpoint():
# this function filters dfchat and returns whatever
def main():
global dfchat
escape_route_thread = threading.Thread(target = escape_route)
escape_route_thread.start()
sample_function(dfchat)
app.run()
main()
I have tried creating another thread that updates the CSV file:
def retrieve_database():
global dfchat
while True:
time.sleep(0.1)
dfchat = pd.read_csv(path)
along with:
escape_route_thread = threading.Thread(target = retrieve_database)
escape_route_thread.start()
in the main function.
But that fails to update the dfchat data frame when the API launches. I have tested the thread by itself and it does update and return an updated data frame.
From what I understand so far, once an API runs, python code cannot change the API itself.
So,
Is there a way to update a running API with just python?
I'm asking for just python because I will not be able to manually enter a link like "/refresh" to do this. It has to be done by python.
Am I missing something?
Thank you very much for helping!
Edit:
I also tried to update the csv file for every API call. But that does but work either:
def sample_function():
dfchat = pd.read_csv(path)
#app.route('/sample_text', methods=['GET'])
def sample_endpoint():
# this function filters dfchat and returns whatever
Code defined at the root of the script (as was dfchat definition in your example) is executed once at the moment you start the flask server.
Code inside a Flask app route (function decorated with #app.route(...)) is executed at each API call to this route.
from flask import Flask
app = Flask(__name__)
path = "path/to/your/csv/file.csv"
#app.route('/sample_text', methods=['GET'])
def sample_endpoint():
dfchat = pd.read_csv(path)
# do what you have to do with the DF
Also note that Flask handles errors without stopping the API, and has great documentation to help you : https://flask.palletsprojects.com/en/2.0.x/quickstart/#a-minimal-application
So I realized that the solution is really simple. I'm new to CS and APIs in general so I did not realize how #app.route worked in Flask. Outside of #app.route cannot change a route but updating a variable inside a route does work. I accidentally kept updating dfchat outside of #app.route.
def sample_function():
# instead of putting dfchat here
#app.route('/sample_text', methods=['GET'])
def sample_endpoint():
dfchat = pd.read_csv(path) # it should go here.
# this function filters dfchat and returns whatever
Thank you yco for helping me realize this.
I have taken input from a form and passed them to this function from the kivy file (on_press property). The form data is fetched properly in the execute function but it won't get logged in myapp.log
Here's the code:
import logging
import selenium
class UIf(GridLayout):
def execute(self, *args):
print("First probe")
logging.basicConfig(filename="myapp.log", level = logging.DEBUG, format='%(asctime)s:%(message)s')
print("Second probe")
for name in args:
print("Third probe")
logging.debug(name)
class MyApp(App):
def build(self):
return UIf()
if __name__ == '__main__':
runapp = MyApp()
runapp.run()
Be sure to make your basicConfig call before importing any of kivy code, because it does its own, that could conflict with your parameters (especially output file).
Also, you might need to the debug level to DEBUG in kivy config, or to reset yourself the log level after the kivy import, because it reset it to its value.
I agree it's a bit intrusive and we should probably consider this a bug, though i don't know what the best way to both have good defaults and stay out of the way of people with opinions would be.
I'm dealing with global variables in Python. The code should work fine, but there is a problem. I have to use global variable for instance of class Back. When I run the application it says that back is None which should be not true because the second line in setup() function - 'back = Back.Back()'
# -*- coding: utf-8 -*-
from flask import Flask
from flask import request
from flask import render_template
import Search
import Back
app = Flask(__name__)
global back
back = None
#app.route('/')
def my_form():
return render_template('my-form.html')
def setup():
global back
back = Back.Back()
def is_ascii(s):
return all(ord(c) < 128 for c in s)
#app.route('/', methods=['POST'])
def search():
from time import time
pattern = request.form['text']
startTime = time()
pattern=pattern.lower()
arr = []
if len(pattern)<1:
arr.append('Incorrect input')
currentTime = time()-startTime
return render_template('my-form.html', arr=arr, time=currentTime)
arr = []
search = Search.Search(pattern,back)
results = search.getResults()
..................
return render_template('my-form.html', arr=arr, time=currentTime, pattern=pattern)
app.debug=True
if __name__ == '__main__':
setup()
app.run()
Why is the back variable None instead of instance of Back class? Thanks
The Flask development server runs your module twice. Once to run the server itself, and another time in a child process so it can reload your whole script each time you make a change to it. It is that second process that won't run the __main__ guarded code and the global is left as None.
You'll get the same problem if you used another WSGI server; it'd import your file as a module, not as the initial script and the __main__ guard is not executed.
Use a #app.before_first_request function instead; it is guaranteed to be executed when the very first request is handled for this process. This also keeps your global working if you moved to a proper WSGI container that used multiple child processes to scale your site:
#app.before_first_request
def setup():
global back
back = Back.Back()
the last few weaks, I am playing a little bit with the Web.py framework. As my application is now getting bigger and bigger, I want to restructure the sourcecode and put code fragments in different classes. Now, I don't really know where I should create my object instances if I need them in different web.py classes. Let us assume, my sourcecode looks like:
import web
import myclass
urls = (
'/', 'index',
'/test', 'test'
)
#should i make my instance global...
my = myclass.myClass()
class test:
def __init__(self):
#...or should i make my instance local: my = myclass.myClass()
pass
def GET(self):
item = my.getItem()
return item
def POST(self):
pass
class index:
def __init__(self):
#...or should i make my instance local: my = myclass.myClass()
pass
def GET(self):
date = my.getDate()
return date
if __name__ == "__main__":
app = web.application(urls, globals())
app.run()
Now, I want to access the methods getItem() and getDate() (which belong to the instance my), if the appropriate sites in my webbrowser are called. My question is now: Should I make the instance global or is it better, if I make it local? I really don't like global instances, but I don't see any other way as to make it global. Sure, it would be possible, to create a local instance, but then, every time the page loads, a new instance would be created, right? Normally, this wouldn't be a problem, but myclass accesses a serial port, so I need to make sure, that only one instance is created.
Am I missing something or is a global instance the only possible solution to accomplish this?
After some research, I came to the conclusion, that global instances are the way to go here. However, one must be careful with global instances if they are used together with the web.py auto reload mode. In auto reload mode, a global instance is created every time a new page loads. If you want to avoid that, you have to use something like this:
import web
import serial
urls = ("/(.*)", "test"
)
web.config.debug = True
if(web.config.debug):
#in debug mode, make sure that global serial instance is only created at start up
if not hasattr(serObj, "_email"):
serObj = serial.Serial(0, 9600, parity=serial.PARITY_NONE)
web._serObj = serObj
else:
serObj = web._serObj
class test:
def GET(self):
return "Test"
def POST(self):
pass
if __name__ == "__main__":
app = web.application(urls, globals())
app.run()