my goal is to be able to use beutifulsoup4 with Python3.4 in a flask web-app all with a elastic-beanstalk server and be able to use beautifulsoup within my main script. I have succeeded in getting a elastic beanstalk web app with flask and python 3.4 up and running smoothly with this code:
from flask import Flask
import time
# print a nice greeting.
def say_hello(username = "World"):
return '<p>Hello %s!</p>\n' % username
# some bits of text for the page.
header_text = '''
<html>\n<head> <title>EB Flask Test</title> </head>\n<body>'''
instructions = '''
<p><em>Hint</em>: This is a RESTful web service! Append a username
to the URL (for example: <code>/Thelonious</code>) to say hello to
someone specific.</p>\n'''
home_link = '<p>Back</p>\n'
footer_text = '</body>\n</html>'
# EB looks for an 'application' callable by default.
application = Flask(__name__)
# add a rule for the index page.
application.add_url_rule('/', 'index', (lambda: header_text +
say_hello() + instructions + footer_text))
# add a rule when the page is accessed with a name appended to the site
# URL.
application.add_url_rule('/<username>', 'hello', (lambda username:
header_text + say_hello(username) + home_link + footer_text))
if __name__ == "__main__":
# Setting debug to True enables debug output. This line should be
# removed before deploying a production app.
application.debug = True
application.run()
requirements file looks like this:
flask==0.12.2
beautifulsoup4==4.6
lxml==4.1
up until this point every thing is OK with no errors, but once I import beautifulsoup in line 3 of the python code the web page gives me this error:
Internal Server Error
The server encountered an internal error or misconfiguration and was unable
to complete your request.
Please contact the server administrator at root#localhost to inform them of the time this error occurred, and the actions you performed just before this error.
More information about this error may be available in the server error log.
if anyone is able to provide advice or let me know if i am being to broad pleas let me know:)
thanks again, john D.
Related
I'm trying to trigger a python module (market order for Oanda) using web hooks(from trading view).
Similar to this
1) https://www.youtube.com/watch?v=88kRDKvAWMY&feature=youtu.be
and this
2)https://github.com/Robswc/tradingview-webhooks-bot
But my broker is Oanda so I'm using python to place the trade. This link has more information.
https://github.com/hootnot/oanda-api-v20
The method is web hook->ngrok->python. When a web hook is sent, the ngrok (while script is also running) shows a 500 internal service error and that the server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.
This is what my script says when its running (see picture);
First says some stuff related the market order then;
running script picture
One thing I noticed is that after Debug it doesn't say Running on... (so maybe my flask is not active?
Here is the python script;
from flask import Flask
import market_orders
# Create Flask object called app.
app = Flask(__name__)
# Create root to easily let us know its on/working.
#app.route('/')
def root():
return 'online'
#app.route('/webhook', methods=['POST'])
def webhook():
if request.method == 'POST':
# Parse the string data from tradingview into a python dict
print(market_orders.myfucn())
else:
print('do nothing')
if __name__ == '__main__':
app.run()
Let me know if there is any other information that would be helpful.
Thanks for your help.
I fixed it!!!! Google FTW
The first thing I learned was how to make my module a FLASK server. I followed these websites to figure this out;
This link helped me set up the flask file in a virtual environment. I also moved my Oanda modules to this new folder. And opened the ngrok app while in this folder via the command window. I also ran the module from within the command window using flask run.
https://topherpedersen.blog/2019/12/28/how-to-setup-a-new-flask-app-on-a-mac/
This link showed me how to set the FLASK_APP and the FLASK_ENV
Flask not displaying http address when I run it
Then I fixed the internal service error by adding return 'okay' after print(do nothing) in my script. This I learned from;
Flask Value error view function did not return a response
I have incorporated an authentication functionality into my code based off of the official Dash-Auth docs. This app is to be hosted on heroku. Couple of things happen (Code is below) :
app = dash.Dash('app',server=server)
app = dash.Dash('auth')
auth = dash_auth.BasicAuth(
app,
(('abcde','1234',),)
)
Locally, the authentication works flawlessly, except that once you
login into the app it saves the login info as cookies. Hence, if you
would refresh the page or probably paste the link to a new window it
wont ask for the login info again unless and until you clear you
cookies in the browser.
Once the app is pushed to the heroku master it successfully deploys
it, unfortunalely the app does not open due to an application error.
On checking the heroku logs the error shown is below. This error is not shown if hosted locally.
'TypeError: 'type' object is not subscriptable '.
As per the post I have removed dash.ly so im sure the error has nothing to do with it. As per this post around the 8th comment someone raises the issue of the login but the reply is not definitive.
UPDATE1
: I believe there is a clash between the app = dash.Dash('app',server=server) and app = dash.Dash('auth') since the moment I remove the code,
app = dash.Dash('auth')
auth = dash_auth.BasicAuth(
app,
(('abcde','1234',),)
)
The app seems to run fine even after deployment to heroku.
Try these 3 points:
Change the name of your app "auth" to __name__
app = dash.Dash(__name__)
Do NOT call the dash.Dash() function 2 times. Change this:
app = dash.Dash('app',server=server)
app = dash.Dash('auth')
auth = dash_auth.BasicAuth(
app,
(('abcde','1234',),)
)
To this:
app = dash.Dash(__name__,server=server)
auth = dash_auth.BasicAuth(
app,
(('abcde','1234',),)
)
Remove those 2 last commas from your password pairs. Then it will be like this:
app = dash.Dash(__name__,server=server)
auth = dash_auth.BasicAuth(
app,
(('abcde','1234'))
)
i follow the tutorial based on a service i wan't to use ( Drone Deploy) :
https://dronedeploy.gitbooks.io/dronedeploy-apps/content/server_example.html
i made a topic on their forum for this issue 2 days ago.( and repo on Github)
But maybe someone can help me here.
I got an issue with an Heroku server app i do.
i use Python and the Tornado module.
The fact is i got a 404 error
My terminal ( when test it localy ) ask me :
"WARNING:tornado.access:404 GET / (::1) 0.75ms"
so the "GET" function is not working , maybe because of HTTP access control (CORS) i try many fix , but none worked.
Maybe i made something wrong ,or forgot something.
update :
Recently someone of the Drone deploy said my tornado routes aren't being picked up by heroku.
After following many tutorial and read the Tornado documentation i have no idea how to routes this.
the url of the server : https://stardrone-server.herokuapp.com/
this the code :
import os
import requests
import tornado.ioloop
import tornado.web
GEOCODE_FMT = 'https://maps.googleapis.com/maps/api/geocode/json?address={address}&key={key}'
class GeocodeHandler(tornado.web.RequestHandler):
"""Proxy a call to the Google Maps geocode API"""
def set_default_headers(self):
# allow cross-origin requests to be made from your app on DroneDeploy to your web server
self.set_header("Access-Control-Allow-Origin", "https://www.dronedeploy.com")
self.set_header("Access-Control-Allow-Headers", "x-requested-with")
# add more allowed methods when adding more handlers (POST, PUT, etc.)
self.set_header("Access-Control-Allow-Methods", "GET, OPTIONS")
def get(self):
api_key = os.environ.get("MyApiKey")
address = self.get_query_argument("address")
url = GEOCODE_FMT.format(address=address, key=api_key)
# fetch results of the geocode from Google
response = requests.get(url)
# send the results back to the client
self.write(response.content)
def options(self):
# no body
self.set_status(204)
self.finish()
def main():
application = tornado.web.Application([
(r"/geocode/", GeocodeHandler)
])
port = int(os.environ.get("PORT", 5000))
application.listen(port)
tornado.ioloop.IOLoop.current().start()
if __name__ == "__main__":
main()
Maybe i missed something.
Any help, advices are appreciated
Thanks
John
Latest update: The problem had indeed to do with permission and user groups, today I learned why we do not simply use root for everything. Thanks to Jakub P. for reminding me to look into the apache error logs and thanks to domoarrigato for providing helpful insight and solutions.
What's up StackOverflow.
I followed the How To Deploy a Flask Application on an Ubuntu VPS tutorial provided by DigitalOcean, and got my application to successfully print out Hello, I love Digital Ocean! when being reached externally by making a GET request to my public server IP.
All good right? Not really.
After that, I edit the tutorial script and write a custom flask application, I test the script in my personal development environment and it runs without issue, I also test it on the DigitalOcean server by having it deploy on localhost and making another GET request.
All works as expected until I try to access it from my public DigitalOcean IP, now suddenly I am presented with a 500 Internal Server Error.
What is causing this issue, and what is the correct way to debug in this case?
What have I tried?
Setting app.debug = True gives the same 500 Internal Server Error without a debug report.
Running the application on the localhost of my desktop pc and DigitalOcean server gives no error, the script executes as expected.
The tutorial code runs and executed fine, and making a GET request to the Digital Ocean public IP returns the expected response.
I can switch between the tutorial application and my own application and clearly see that I am only getting the error wit my custom application. However the custom application still presents no issues running on localhost.
My code
from flask import Flask, request, redirect
from netaddr import IPNetwork
import os
import time
app = Flask(__name__)
APP_ROOT = os.path.dirname(os.path.abspath(__file__))
# Custom directories
MODULES = os.path.join(APP_ROOT, 'modules')
LOG = os.path.join(APP_ROOT, 'log')
def check_blacklist(ip_adress):
ipv4 = [item.strip() for item in open(MODULES + '//ipv4.txt').readlines()]
ipv6 = [item.strip() for item in open(MODULES + '//ipv6.txt').readlines()]
for item in ipv4 + ipv6:
if ip_adress in IPNetwork(item):
return True
else:
pass
return False
#app.route('/')
def hello():
ip_adress = request.environ['REMOTE_ADDR']
log_file = open(LOG + '//captains_log.txt', 'a')
with log_file as f:
if check_blacklist(ip_adress):
f.write('[ {}: {} ][ FaceBook ] - {} .\n'
.format(time.strftime("%d/%m/%Y"), time.strftime("%H:%M:%S"), request.environ))
return 'Facebook'
else:
f.write('[ {}: {} ][ Normal User ] - {} .\n'
.format(time.strftime("%d/%m/%Y"), time.strftime("%H:%M:%S"), request.environ))
return 'Normal Users'
if __name__ == '__main__':
app.debug = True
app.run()
The tutorial code:
from flask import Flask
app = Flask(__name__)
#app.route("/")
def hello():
return "Hello, I love Digital Ocean!"
if __name__ == "__main__":
app.run()
Seems like the following line could be a problem:
log_file = open(LOG + '//captains_log.txt', 'a')
if the path its looking for is: '/var/www/flaskapp/flaskapp/log//captains_log.txt'
that would make sense that an exception is thrown there. Possible that the file is in a different place, on the server, or a / needs to be removed - make sure the open command will find the correct file.
If captains_log.txt is outside the flask app directory, you can copy it in and chown it. if the txt file needs to be outside the directory then you'll have to add the user to the appropriate group, or open up permissions on the actual directory.
chown command should be:
sudo chown www:www /var/www/flaskapp/flaskapp/log/captains_log.txt
and it might be smart to run:
sudo chown -r www:www /var/www
I got timeout error on my GAE server when it tries to send large files to an EC2 REST server. I found Backends Python API would be a good solution to my example but I had some problems on configuring it.
Following some instructions, I have added a simple backends.yaml in my project folder. But I still received the following error, which seems like I failed to create a backend instance.
File "\Google\google_appengine\google\appengine\api\background_thread\background_thread.py", line 84, in start_new_background_thread
raise ERROR_MAP[error.application_error](error.error_detail)
FrontendsNotSupported
Below is my code, and my question is:
Currently, I got timeout error in OutputPage.py, how do I let this script run on a backend instance?
update
Following Jimmy Kane's suggestions, I created a new script przm_batchmodel_backend.py for the backend instance. After staring my GAE, now it I have two ports (a default and a backend) running my site. Is that correct?
app.yaml
- url: /backend.html
script: przm_batchmodel.py
backends.yaml
backends:
- name: mybackend
class: B1
instances: 1
options: dynamic
OutputPage.py
from przm import przm_batchmodel
from google.appengine.api import background_thread
class OutputPage(webapp.RequestHandler):
def post(self):
form = cgi.FieldStorage()
thefile = form['upfile']
#this is the old way to initiate calculations
#html= przm_batchmodel.loop_html(thefile)
przm_batchoutput_backend.przmBatchOutputPageBackend(thefile)
self.response.out.write(html)
app = webapp.WSGIApplication([('/.*', OutputPage)], debug=True)
przm_batchmodel.py
def loop_html(thefile):
#parses uploaded csv and send its info. to the REST server, the returned value is a html page.
data= csv.reader(thefile.file.read().splitlines())
response = urlfetch.fetch(url=REST_server, payload=data, method=urlfetch.POST, headers=http_headers, deadline=60)
return response
przm_batchmodel_backend.py
class BakendHandler(webapp.RequestHandler):
def post(self):
t = background_thread.BackgroundThread(target=przm_batchmodel.loop_html, args=[thefile])
t.start()
app = webapp.WSGIApplication([('/backend.html', BakendHandler)], debug=True)
You need to create an application 'file'/script for the backend to work. Just like you do with the main.
So something like:
app.yaml
- url: /backend.html
script: przm_batchmodel.py
and on przm_batchmodel.py
class BakendHandler(webapp.RequestHandler):
def post(self):
html = 'test'
self.response.out.write(html)
app = webapp.WSGIApplication([('/backend.html', OutputPage)], debug=True)
May I also suggest using the new feature modules which are easier to setup?
Edit due to comment
Possible the setup was not your problem.
From the docs
Code running on a backend can start a background thread, a thread that
can "outlive" the request that spawns it. They allow backend instances
to perform arbitrary periodic or scheduled tasks or to continue
working in the background after a request has returned to the user.
You can only use backgroundthread on backends.
So edit again. Move the part of the code that is:
t = background_thread.BackgroundThread(target=przm_batchmodel.loop_html, args=[thefile])
t.start()
self.response.out.write(html)
To the backend app