Develop Reference

Using multiprocessing module with BaseHTTPRequestHandler

I'm trying to use the python multiprocessing module to run a server in another Thread using the http.server.BaseHTTPRequestHandler module. I am stuck though and am running into a '_thread.lock' issue.
I don't want to use the threading module because I'd rather use true multi-threading with the multi-processing module.
If anyone knows what I am doing incorrectly or can point me to a good library to use that would be awesome.
import multiprocessing
from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler
if __name__ == '__main__':
httpd = ThreadingHTTPServer(('localhost', 4433), BaseHTTPRequestHandler)
manager = multiprocessing.Manager()
manager.http_server = httpd
running_server = multiprocessing.Process(target=manager.http_server.serve_forever)
running_server.start()
Stack Trace:
File "/Users/redacted/python/test2/test1.py", line 10, in <module>
running_server.start()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
return Popen(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
super().__init__(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
reduction.dump(process_obj, fp)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_thread.lock' object

Python uses pickle to pass objects to another process when using multiprocess module. In your case, the thread lock used in the httpserver is not pickleable. So it reports the error.
What you can do is start the http server in another process completely like this:
import multiprocessing
from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler
def startServer():
httpd = ThreadingHTTPServer(('localhost', 4433), BaseHTTPRequestHandler)
httpd.serve_forever()
if __name__ == '__main__':
manager = multiprocessing.Manager()
running_server = multiprocessing.Process(target=startServer)
running_server.start()
Also, you might want to try a different port other than 4433. I cannot connect to this port on my windows machine. But if I use 8000 everything works fine.

Permission denied while starting a flask app at port 80 in AWS EC2 instance

I have developed a flask based app which is running successfully at my local machine / network.
I deployed it to an AWS EC2 instance. Started it like this:
flask run --host <private ip within AWS>
but when I tried to access an implemented API through Postman, I got following error in Postman console:
Error: connect ECONNREFUSED <IPv4 Public IP within AWS>
Noticed that it might be due to the fact that flask app typically runs on port 5000, however the EC2 has inbound rule of type http bound to port 80.
So I changed my flask code to:
if __name__ == "__main__":
app = create_app()
app.run(host='<private ip within AWS>', port=80)
And tried to run it like this in AWS ec2 instance:
python3 apiapp.py
However I got following stack trace while trying to run the app:
Traceback (most recent call last):
File "apiapp.py", line 49, in <module>
app.run(host='<private ip within AWS>', port=80)
File "/home/ec2-user/AwesomeAPIs/venv/lib64/python3.7/site-packages/flask/apiapp.py", line 990, in run
run_simple(host, port, self, **options)
File "/home/ec2-user/AwesomeAPIs/venv/lib64/python3.7/site-packages/werkzeug/serving.py", line 1052, in run_simple
inner()
File "/home/ec2-user/AwesomeAPIs/venv/lib64/python3.7/site-packages/werkzeug/serving.py", line 1005, in inner
fd=fd,
File "/home/ec2-user/AwesomeAPIs/venv/lib64/python3.7/site-packages/werkzeug/serving.py", line 848, in make_server
host, port, app, request_handler, passthrough_errors, ssl_context, fd=fd
File "/home/ec2-user/AwesomeAPIs/venv/lib64/python3.7/site-packages/werkzeug/serving.py", line 740, in __init__
HTTPServer.__init__(self, server_address, handler)
File "/usr/lib64/python3.7/socketserver.py", line 452, in __init__
self.server_bind()
File "/usr/lib64/python3.7/http/server.py", line 137, in server_bind
socketserver.TCPServer.server_bind(self)
File "/usr/lib64/python3.7/socketserver.py", line 466, in server_bind
self.socket.bind(self.server_address)
PermissionError: [Errno 13] Permission denied
Am I missing anything obvious over here?

Auto reloading is not working in Bottle framework

I am currently getting started with Bottle framework (doing the Hello World example and afterwards have to build a RESTful API). The problem is the fact that reloader doesn't work. When I make a change in the code and reload the page where the change should show nothing happens. It works on my friends' computers so I'm am a bit confused.
Using python 2.7.
from bottle import route, run
#route('/hello')
def hello():
return "Hello World!"
run(host='localhost', port=8080, debug=True, reloader =True)
EDIT:Also what i noticed is that when i save the change in the script while the server is still listening i get this:
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 60472)
Traceback (most recent call last):
File "C:\Python27\lib\SocketServer.py", line 290, in _handle_request_noblock
self.process_request(request, client_address)
File "C:\Python27\lib\SocketServer.py", line 318, in process_request
self.finish_request(request, client_address)
File "C:\Python27\lib\SocketServer.py", line 331, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\Python27\lib\SocketServer.py", line 652, in __init__
self.handle()
File "C:\Python27\lib\wsgiref\simple_server.py", line 116, in handle
self.raw_requestline = self.rfile.readline(65537)
File "C:\Python27\lib\socket.py", line 480, in readline
data = self._sock.recv(self._rbufsize)
KeyboardInterrupt
----------------------------------------

There is an interesting clue if you use Windows OS:
Keep in mind that in windows this must be
under if name == "main": due to the way the multiprocessing
module works.
So it should look like this
from bottle import route, run
#route('/hello')
def hello():
return "Hello World!"
if __name__ == "__main__":
run(host='localhost', port=8080, debug=True, reloader=True)

GAE/Flask [Errno 13] Permission denied

As the title suggests, I am seeing this error when my flask app tries to run.
I am hosting the application locally using dev_appserver.
The error occurs when I visit the site and it tries to run the app. It appears that GAE is trying and failing to bind a socket for some reason.
I suspect that this may have something to do with OAuth2. Maybe it requires an SSL connection?
I don't even know where to begin solving this as none of the other posts about this are experiencing the same variation of the issue.
Edit: Here's a screenshot of the console confirming that the GAE server launches successfully on a different port; still doesn't resolve it
Traceback (most recent call last):
File "C:\Users\XXX\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\runtime\wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "C:\Users\XXX\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\runtime\wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "C:\Users\XXX\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\runtime\wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "C:\Users\XXX\PycharmProjects\ad-assignment\main.py", line 51, in <module>
app.run()
File "C:\Users\XXX\PycharmProjects\ad-assignment\lib\flask\app.py", line 843, in run
run_simple(host, port, self, **options)
File "C:\Users\XXX\PycharmProjects\ad-assignment\lib\werkzeug\serving.py", line 694, in run_simple
inner()
File "C:\Users\XXX\PycharmProjects\ad-assignment\lib\werkzeug\serving.py", line 656, in inner
fd=fd)
File "C:\Users\XXX\PycharmProjects\ad-assignment\lib\werkzeug\serving.py", line 550, in make_server
passthrough_errors, ssl_context, fd=fd)
File "C:\Users\XXX\PycharmProjects\ad-assignment\lib\werkzeug\serving.py", line 464, in __init__
HTTPServer.__init__(self, (host, int(port)), handler)
File "C:\Python27\Lib\SocketServer.py", line 417, in __init__
self.server_bind()
File "C:\Python27\Lib\BaseHTTPServer.py", line 108, in server_bind
SocketServer.TCPServer.server_bind(self)
File "C:\Python27\Lib\SocketServer.py", line 431, in server_bind
self.socket.bind(self.server_address)
File "C:\Users\XXX\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\dist27\socket.py", line 222, in meth
return getattr(self._sock,name)(*args)
File "C:\Users\XXX\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\api\remote_socket\_remote_socket.py", line 676, in bind
raise _SystemExceptionFromAppError(e)
error: [Errno 13] Permission denied
INFO 2016-12-16 21:41:51,631 module.py:788] default: "GET /oauth2callback?code=x/xxxxxxxxxxxxxxxxx HTTP/1.1" 500 -
Code (as seen in Google's OAuth2 usage guide):
import flask
app = flask.Flask(__name__)
#app.route('/')
def index():
...
#app.route('/oauth2callback')
def oauth2callback():
...
if __name__ == 'main':
import uuid
app.secret_key = str(uuid.uuid4())
app.debug = False
app.run()

We have a tutorial that walks you through adding Firebase Authentication to your Python app running with Flask. Firebase Authentication is the preferred identity toolkit now. You can of course still use a pure OAuth2 flow, but Firebase Auth also provides multi-provider authentication if that's something you were considering adding to your app anyways. If you just want to dive into the sample's code its here on GitHub.
If you just want to stick with straight OAuth, you might want to look at your Flask code itself. Getting flask to run is pretty easy on App Engine. My guess is that you're calling some code that you don't need to (flask.run()) or you aren't importing your library properly (see appengine_config.py).

Openshift python gear wont deploy

I have a problem with my most recent openshift gear , i get the python error errno 13 , Permission denied. I have checked around on here and found that most people who have gotten the same issue have just forgot the if __name__ == "__main__": statement before the app.run(), but i still have the same issue. According to the log on openshift, the error seems to be rooting from the socket.py built in to python.
This is my second gear on openshift and both this and my first one is built using flask. The first one works just fine but my second one wont work ...
This is the error message I get in the python log from openshift:
* Running on http://127.0.0.1:5000/
Traceback (most recent call last):
File "app.py", line 24, in <module>
app.run()
File "/var/lib/openshift/54d242655973ca23980001d1/python/virtenv /lib/python2.7/site-packages/flask/app.py", line 772, in run
run_simple(host, port, self, **options)
File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/werkzeug/serving.py", line 617, in run_simple
inner()
File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/werkzeug/serving.py", line 599, in inner
passthrough_errors, ssl_context).serve_forever()
File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/werkzeug/serving.py", line 408, in make_server
passthrough_errors, ssl_context)
File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/werkzeug/serving.py", line 334, in __init__
HTTPServer.__init__(self, (host, int(port)), handler)
File "/opt/rh/python27/root/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
self.server_bind()
File "/opt/rh/python27/root/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
SocketServer.TCPServer.server_bind(self)
File "/opt/rh/python27/root/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
self.socket.bind(self.server_address)
File "/opt/rh/python27/root/usr/lib64/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
socket.error: [Errno 13] Permission denied
EDIT 2:
So i changed the ip to OPENSHIFT_PYTHON_IP but the errorlog looked exactly the same. This is my wsgi.py code
#!/usr/bin/python
import os
virtenv = os.environ['OPENSHIFT_PYTHON_DIR'] + '/virtenv/'
virtualenv = os.path.join(virtenv, 'bin/activate_this.py')
try:
execfile(virtualenv, dict(__file__=virtualenv))
except IOError:
pass
def run_gevent_server(app, ip, port=8080):
from gevent.pywsgi import WSGIServer
WSGIServer((ip, port), app).serve_forever()
def run_simple_httpd_server(app, ip, port=8080):
from wsgiref.simple_server import make_server
make_server(ip, port, app).serve_forever()
from server import app as application
#
# Below for testing only
#
if __name__ == '__main__':
ip = os.environ['OPENSHIFT_PYTHON_IP']
port = 8080
zapp = impo.load_source('application', 'wsgi/application')
try:
run_gevent_server(application, ip, port)
except:
run_simple_httpd_server(zapp.application, ip, port)

What port/ip address are you using? You must bind to port 8080 of your openshift ip address, you can't bind to 127.0.0.1, it is not allowed.