Develop Reference

Updating DataFrame while API is running in Python

So I am trying to create an API that constantly reads from a CSV and returns information about it when requested. So far, I have created a flask API that reads the CSV file once and returns correctly. However, I can't seem to make it constantly update. My working code is something like this.
app = flask.Flask(__name__)
app.config["DEBUG"] = True
dfchat = pd.read_csv(path)
escaper = None
# for now, this is just to make sure the program keeps running even if there is an error
def escape_route():
global escaper
while escaper != "Y":
escaper = str(input("Exit now? Enter \'Y\': \n")).strip()
os._exit(os.X_OK)
def sample_function(dfchat):
#app.route('/sample_text', methods=['GET'])
def sample_endpoint():
# this function filters dfchat and returns whatever
def main():
global dfchat
escape_route_thread = threading.Thread(target = escape_route)
escape_route_thread.start()
sample_function(dfchat)
app.run()
main()
I have tried creating another thread that updates the CSV file:
def retrieve_database():
global dfchat
while True:
time.sleep(0.1)
dfchat = pd.read_csv(path)
along with:
escape_route_thread = threading.Thread(target = retrieve_database)
escape_route_thread.start()
in the main function.
But that fails to update the dfchat data frame when the API launches. I have tested the thread by itself and it does update and return an updated data frame.
From what I understand so far, once an API runs, python code cannot change the API itself.
So,
Is there a way to update a running API with just python?
I'm asking for just python because I will not be able to manually enter a link like "/refresh" to do this. It has to be done by python.
Am I missing something?
Thank you very much for helping!
Edit:
I also tried to update the csv file for every API call. But that does but work either:
def sample_function():
dfchat = pd.read_csv(path)
#app.route('/sample_text', methods=['GET'])
def sample_endpoint():
# this function filters dfchat and returns whatever

Code defined at the root of the script (as was dfchat definition in your example) is executed once at the moment you start the flask server.
Code inside a Flask app route (function decorated with #app.route(...)) is executed at each API call to this route.
from flask import Flask
app = Flask(__name__)
path = "path/to/your/csv/file.csv"
#app.route('/sample_text', methods=['GET'])
def sample_endpoint():
dfchat = pd.read_csv(path)
# do what you have to do with the DF
Also note that Flask handles errors without stopping the API, and has great documentation to help you : https://flask.palletsprojects.com/en/2.0.x/quickstart/#a-minimal-application

So I realized that the solution is really simple. I'm new to CS and APIs in general so I did not realize how #app.route worked in Flask. Outside of #app.route cannot change a route but updating a variable inside a route does work. I accidentally kept updating dfchat outside of #app.route.
def sample_function():
# instead of putting dfchat here
#app.route('/sample_text', methods=['GET'])
def sample_endpoint():
dfchat = pd.read_csv(path) # it should go here.
# this function filters dfchat and returns whatever
Thank you yco for helping me realize this.

Is this the correct way to localize a python app

So, I have a python app which I would like to translate in a few languages. The way the app loads its messages is by using a messages.py module and a few classes like for a example.
class MenuMessages:
hello_world = "This is a menu"
The app will use a user defined settings (which I can query through the database) too choose which language the user wants to use the app on.
This app serves more than one user at once (so its not an app which is run by a single user).
An example function would be something as follow
import MenuMessages from messages
def connect():
"""Example function"""
config = get_config(args)
## Way to check user lang settings
language = config.get("lang", "en")
## Right now default messages
print(MenuMessages.hello_world)
One idea that I have is to make a folder (lets call it languages) and put the same messages.py file structure, translate them and call them english.py, spanish.py etc
My issue with this is that I would need to find a way to switch between them everytime in a method given the user settings such as.
from languages.english import MenuMessages as en_menu_messages
from languages.spanish import MenuMessages as sp_menu_messages
def connect():
"""Example function"""
config = get_config(args)
## Way to check user lang settings
lang = config.get("lang", "en")
if lang == "en":
MenuMessages = en_menu_messages
elif lang == "sp":
MenuMessages = sp_menu_messages
## Right now default messages
print(MenuMessages.hello_world)
Do you guys think this is a efficient way to approach this issue?
I want to implement it in the best way possible so I do not have to change it in the future.

Downloading PDF's and tracking downloads with Python

I'm creating an application that downloads PDF's from a website and saves them to disk. I understand the Requests module is capable of this but is not capable of handling the logic behind the download (File size, progress, time remaining etc.).
I've created the program using selenium thus far and would like to eventually incorporate this into a GUI Tkinter app eventually.
What would be the best way to handle the downloading, tracking and eventually creating a progress bar?
This is my code so far:
from selenium import webdriver
from time import sleep
import requests
import secrets
class manual_grabber():
""" A class creating a manual downloader for the Roger Technology website """
def __init__(self):
""" Initialize attributes of manual grabber """
self.driver = webdriver.Chrome('\\Users\\Joel\\Desktop\\Python\\manual_grabber\\chromedriver.exe')
def login(self):
""" Function controlling the login logic """
self.driver.get('https://rogertechnology.it/en/b2b')
sleep(1)
# Locate elements and enter login details
user_in = self.driver.find_element_by_xpath('/html/body/div[2]/form/input[6]')
user_in.send_keys(secrets.username)
pass_in = self.driver.find_element_by_xpath('/html/body/div[2]/form/input[7]')
pass_in.send_keys(secrets.password)
enter_button = self.driver.find_element_by_xpath('/html/body/div[2]/form/div/input')
enter_button.click()
# Click Self Service Area button
self_service_button = self.driver.find_element_by_xpath('//*[#id="bs-example-navbar-collapse-1"]/ul/li[1]/a')
self_service_button.click()
def download_file(self):
"""Access file tree and navigate to PDF's and download"""
# Wait for all elements to load
sleep(3)
# Find and switch to iFrame
frame = self.driver.find_element_by_xpath('//*[#id="siteOutFrame"]/iframe')
self.driver.switch_to.frame(frame)
# Find and click tech manuals button
tech_manuals_button = self.driver.find_element_by_xpath('//*[#id="fileTree_1"]/ul/li/ul/li[6]/a')
tech_manuals_button.click()
bot = manual_grabber()
bot.login()
bot.download_file()
So in summary, I'd like to make this code download PDF's on a website, store them in a specific directory (named after it's parent folder in the JQuery File Tree) and keep tracking of the progress (file size, time remaining etc.)
Here is the DOM:
I hope this is enough information. Any more required please let me know.

How to *efficiently* monitor a webpage modified with javascript?

I'm trying to monitor an element on a website that's generated via javascript. The problem of downloading a javascript modified page has been handled before, I borrowed the following code that solves the problem with PyQt.
But when I set this code to run every 20 seconds, my network traffic averages 70KB/s down 5 KB/s up. The actual page saved is only 80KB, but is javascript heavy.
6GB a day is not reasonable, my ISP has data limits and I already toe the line.
Is there a way to modify this code so that, for example, it only executes the javascript that corresponds to a specific element on the page? If so, how would I go about figuring out what I need to execute? And would that have a significant effect on the network traffic I'm seeing?
Alternately, how SHOULD I be doing this? I considered making a chrome extension, as Chrome would already be handling the javascript for me, but then I have to figure out how to integrate it with the rest of my project, and that's brand new territory for me. If there's a better way I'd rather do that.
#Borrowed from http://stackoverflow.com/questions/19161737/cannot-add-custom-request-headers-in-pyqt4
#which is borrowed from http://blog.motane.lu/2009/07/07/downloading-a-pages-content-with-python-and-webkit/
import sys, signal
from PyQt4.QtCore import QUrl
from PyQt4.QtGui import QApplication
from PyQt4.QtWebKit import QWebPage
from PyQt4.QtNetwork import QNetworkAccessManager, QNetworkRequest, QNetworkReply
cookie = ''#snipped, the cookie I have to send is about as long as this bit of code...
class MyNetworkAccessManager(QNetworkAccessManager):
def __init__(self, url):
QNetworkAccessManager.__init__(self)
request = QNetworkRequest(QUrl(url))
self.reply = self.get(request)
def createRequest(self, operation, request, data):
request.setRawHeader('User-Agent', 'Mozilla/5.0')
request.setRawHeader("Cookie",cookie);
return QNetworkAccessManager.createRequest( self, operation, request, data )
class Crawler( QWebPage ):
def __init__(self, url, file):
QWebPage.__init__( self )
self._url = url
self._file = file
self.manager = MyNetworkAccessManager(url)
self.setNetworkAccessManager(self.manager)
def crawl( self ):
signal.signal( signal.SIGINT, signal.SIG_DFL )
self.loadFinished.connect(self._finished_loading)
self.mainFrame().load( QUrl( self._url ) )
def _finished_loading( self, result ):
file = open( self._file, 'w' )
file.write( self.mainFrame().toHtml() )
file.close()
exit(0)
def main(url,file):
app = QApplication([url,file])
crawler = Crawler(url, file)
crawler.crawl()
sys.exit( app.exec_() )

First, be clear about your motivation here. Something is changing every 20 seconds and you want to tell the server when one field changes. So
1). Do you need to send the whole page or just the contents of one field. I'm not clear what you are currently doing, but if you are sending 80k every 20 seconds this seems like overkill.
2). Does the server need to know immediately? What are the consequences of sending the state every minute rather than every 20 seconds. You miss some changes, but does that matter?
You haven't really told what you're doing or why, so we can't comment on that.
My first thought is that if the server just wants to know about one field then make an ajax call with just the payload you care about. To me more efficient send a summary every few minutes. One composite record is much cheaper than several small records.

Inserting multiple model instances using a single db.put() on Google App Engine

Edit: Sorry I didn't clarify this, it's a Google App Engine related question.
According to this, I can give db.put() a list of model instances and ask it to input them all into the datastore. However, I haven't been able do this successfully. I'm still a little new with Python, so go easy on me
list_of_models = []
for i in range(0, len(items) - 1):
point = ModelName()
... put the model info here ...
list_of_models.append(point)
db.put(list_of_models)
Could anyone point out where I'm going wrong?

Please define what you mean by "going wrong" -- the tiny pieces of code you're showing could perfectly well be part of an app that's quite "right". Consider e.g.:
class Hello(db.Model):
name = db.StringProperty()
when = db.DateTimeProperty()
class MainHandler(webapp.RequestHandler):
def get(self):
self.response.out.write('Hello world!')
one = Hello(name='Uno', when=datetime.datetime.now())
two = Hello(name='Due', when=datetime.datetime.now())
both = [one, two]
db.put(both)
this does insert the two entities correctly each time that get method is called, for example if a sample app continues with:
def main():
application = webapp.WSGIApplication([('/', MainHandler)],
debug=True)
wsgiref.handlers.CGIHandler().run(application)
if __name__ == '__main__':
main()
as in a typical "hello world" app engine app. You can verify the correct addition of both entities with the datastore viewer of the sdk console, or of course by adding another handler which gets the entities back and shows them, etc etc.
So please clarify!