Develop Reference

Cannot reach "worker" process in Heroku from web process: `ConnectionRefusedError`

I have a web process and an api process (a "worker" process similar to what is described in the docs and these other docs). However, I don't understand how the networking between different processes works.
Both processes are listening on 0.0.0.0, and the worker process is bound to port 5000. However, when I try to make a request from the web process, I get a ConnectionRefusedError:
ConnectionError: HTTPConnectionPool(host='0.0.0.0', port=5000): Max retries exceeded with url: /tokenize?sentence=hello (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd307019dc0>: Failed to establish a new connection: [Errno 111] Connection refused'))
Am I supposed to figure out the IP of the other process? Am I doing something wrong here?
Procfile:
api: python app.py
web: voila UI.ipynb --port $PORT --Voila.ip=0.0.0.0
app.py:
from flask import Flask, request
from ie_utils import tokenize
app = Flask(__name__)
#app.route("/")
def home():
return {
"message": "Hello world!",
"version": "0.1",
}
if __name__ == "__main__":
import os
port = 5000
app.run(host="0.0.0.0", port=port)
Relevant code in UI.ipynb:
import requests
requests.get("http://0.0.0.0:5000/")
Full source code: https://github.com/astrojuanlu/ie-titanic-utils-a/tree/test-procfile

I don't understand how the networking between different processes works
Unless you are using Private Spaces (part of Heroku's enterprise offering), it doesn't:
The Common Runtime provides strong isolation by firewalling all dynos off from one another. The only traffic that can reach a dyno is web requests forwarded from the router to web processes listening on the port number specified in the $PORT environment variable. Worker and one-off dynos cannot receive inbound requests.
I'm not totally clear what you're trying to do here, but you won't be able to do it by making an HTTP request to localhost. You might have better luck running this as two separate apps—in that case your API could be its own web process and your Voilà app could make requests to the API app's hostname.
Side note: even on a system where this is permitted your request should not go to 0.0.0.0. That isn't a real IP address.
Telling Flask to bind to 0.0.0.0 really means that it should listen on all available IP addresses. When you want to make a request you should use one of the real hostnames or IP addresses that your system is using, e.g. localhost or 127.0.0.1 or one of its public IP addresses or hostnames.

flask server with ssl_context freezes if it receives http request

I'm trying to create a simple flask server that redirects any http requests to https. I've created a certificate and key file and registered a before_request hook to see if the request is secure and redirect appropriately, following advise this SO answer.
The flask server responds to https requests as expected. However, when I send an http request, the before_request hook never gets called and ther server hangs forever. If I send the http request from the browser, I see an "ERR_EMPTY_RESPONSE". The server doesn't even respond to https requests afterwards. No logs are printed either.
Running the app with gunicorn didn't help either. The only difference was that gunicorn is able to detect that the worker is frozen and eventually kills and replaces it. I've also tried using flask-talisman, with the same results.
Below is the code I'm running
### server.py
from flask import Flask, request, redirect
def verify_https():
if not request.is_secure:
url = request.url.replace("http://", "https://", 1)
return redirect(url, 301)
def create_flask_app():
app = Flask(__name__)
app.before_request(verify_https)
app.add_url_rule('/', 'root', lambda: "Hello World")
return app
if __name__ == '__main__':
app = create_flask_app()
app.run(
host="0.0.0.0",
port=5000,
ssl_context=('server.crt', 'server.key')
)
Running it with either python3.8 server.py or gunicorn --keyfile 'server.key' --certfile 'server.crt' --bind '0.0.0.0:5000' 'server:create_flask_app()' and opening a browser window to localhost:5000 causes the server to hang.

Talking about freezes, its not. Flask and gunicorn can serve only one variant of connection. So it's not freezing because your browser canceled the request and is idling.
I think it is better to use a faster web server, for example, Nginx, if you want to change HTTP to HTTPS. I would recommend it to you.
But it's possible to trigger your verify_https function if you run multiple instances of gunicorn at the same time.
I took your example, generated a certificate, and then run this script in my console (it contains a background job and can be runned in twoo separate ter)
gunicorn --bind '0.0.0.0:80' 'server:create_flask_app()' & gunicorn --certfile server.crt --keyfile server.key --bind '0.0.0.0:443' 'server:create_flask_app()'
now chrome goes to the secure page as expected.

Typically servers don't listen for both http and https on the same port. I have a similar requirement for my personal portfolio, but I use nginx to forward http requests (port 80) to https (port 443) and then the https server passes it off to my uwsgi backend, which listens on port 3031. That's probably more complex than you need, but a possible solution. If you go that route I would recommend letsencrypt for your certificate needs. It will set up the certificates AND the nginx.conf for you.
If you don't want to go the full nginx/apache route I think your easiest solution is the one suggested here on that same thread that you linked.

uvicorn suppresses python's syslog for gelf driver

I have a docker container running logging with gelf to a logging instance via udp -- all fine!
The container is based on Ubuntu 18 where rsyslog is running as a service, which works well.
Inside the container is a FastAPI application running with uvicorn webserver. It also works perfectly and uvicorn is perfectly logging to the logging instance.
Here comes what is not working, but usually works with non-FastAPI python projects. I use python's syslog to log more stuff.
The app with syslog looks like this (I created an easy example to debug for myself):
from fastapi import FastAPI
import syslog
syslog.openlog(facility=syslog.LOG_LOCAL0)
app = FastAPI()
syslog.syslog(syslog.LOG_INFO, 'startup done')
#app.get("/")
async def root():
syslog.syslog(syslog.LOG_INFO, 'get hello')
return {"message": "Hello World"}
The logs at the logging instance don't show the syslog messages. Only uvicorn's messages:
INFO: Started server process [21]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:80 (Press CTRL+C to quit)
INFO: 172.17.0.1:35346 - "GET / HTTP/1.1" 200 OK
For further debugging I checked rsyslog's log file, and it contains the syslog messages:
Dec 23 17:21:39 /uvicorn: startup done
Dec 23 17:21:50 /uvicorn: get hello
and here is the rsyslog configuration at /etc/rsyslog.d
local0.* {
action(type="omfile" file="/var/log/test.log" fileOwner="syslog" fileGroup="syslog" fileCreateMode="0640")
stop
}
What am I missing here?
Why is gelf ignoring rsyslog?
What do I need to understand about uvicorn concerning syslog?
or what can I do?
Thanks

The problem results from gelf ignoring syslog. A simple print in python will be recognized though.
Solution: I built myself a log function in python which will log to syslog and print out whatever I want to log.

How to get MQTT client to work on Appengine (python)

Trying to run a simple MQTT client (not the broker) on a paid GAE app. However a on_connect callback never occurs in the following:
worker.py
import webapp2
import paho.mqtt.client as paho
class WorkerHandler(webapp2.RequestHandler):
def on_subscribe(client, userdata, mid, granted_qos):
print("Subscribed: "+str(mid)+" "+str(granted_qos))
def on_message(client, userdata, msg):
print(msg.topic+" "+str(msg.qos)+" "+str(msg.payload))
def on_connect(client, userdata, flags, rc):
client.subscribe("$SYS/#")
print "Subscribed to wildcard"
def get(self):
client = paho.Client()
client.on_connect = self.on_connect
client.on_subscribe = self.on_subscribe
client.on_message = self.on_message
client.connect("iot.eclipse.org")
print "connected to broker"
client.loop_forever()
app = webapp2.WSGIApplication([
(r'/_ah/start', WorkerHandler),
])
In the dev environment it fails silently with just a message after a minute or so
INFO 2017-04-04 01:51:40,958 module.py:813] worker: "GET /_ah/start HTTP/1.1" 500 220
INFO 2017-04-04 01:51:41,869 module.py:1759] New instance for module "worker" serving on: http://localhost:8080
connected to broker
WARNING 2017-04-04 01:52:10,860 module.py:1927] All instances may not have restarted
This is configured as a "backend"/service and the yaml looks like this:
worker.yaml
service: worker
runtime: python27
api_version: 1
threadsafe: true
instance_class: B8
manual_scaling:
instances: 1
handlers:
# If a service has an _ah/start handler, it should be listed first.
- url: /_ah/start
script: worker.app
Note: In the dev environment, socket.py is being imported directly from python install .../2.7/lib/python2.7/socket.py

You're attempting to run a standalone script as your GAE app worker service. It won't work.
Your worker.py needs to contain a WSGI application called app to match your worker.yaml configuration.
From the script row in the Handlers element table:
A script: directive must be a python import path, for example,
package.module.app that points to a WSGI application. The last component of a script: directive using a Python module path is
the name of a global variable in the module: that variable must be a
WSGI app, and is usually called app by convention.
The error you get most likely indicates that the attempt to start the worker's module WSGI app fails.
After your update to bring back the WSGI app the reason for the error message becomes even clearer: the WorkerHandler.get() method doesn't respond to the /_ah/start request, because it's stuck in client.loop_forever().
From Startup:
Each service instance is created in response to a start request, which
is an empty HTTP GET request to /_ah/start. App Engine sends this
request to bring an instance into existence; users cannot send a
request to /_ah/start. Manual and basic scaling instances must
respond to the start request before they can handle another request.
...
When an instance responds to the /_ah/start request with an HTTP
status code of 200–299 or 404, it is considered to have successfully
started and can handle additional requests. Otherwise, App Engine
terminates the instance. Manual scaling instances are restarted
immediately, while basic scaling instances are restarted only when
needed for serving traffic.

Twisted.internet.error.CannotListenError,Address already in use

I tried to run scrapy spiders in a flask app, and there is a API in github Arachne.But there is something wrong when I run the demo project in my mac.The app.py looks like this:
app = Arachne(__name__)
resource = WSGIResource(reactor, reactor.getThreadPool(), app)
site = Site(resource,
logFormatter=http.combinedLogFormatter,
logPath="logs/"+datetime.now().strftime("%Y-%m-%d.web.log"))
reactor.listenTCP(8080, site)
if __name__ == '__main__':
reactor.run()
But when I run this file,there is an error:
twisted.internet.error.CannotListenError: Couldn't listen on any:8080: [Errno 48] Address already in use.
I tried to solve this by the method which is mentioned in this question:twisted python server port already in use.And the result after I use the command to kill the process in my terminal like this:
sunzhendeMacBook-Pro:~ Rabbit$ lsof -n -iTCP:8080 -sTCP:LISTEN
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
Python 5151 Rabbit 3u IPv4 0x738fcf82e37369b9 0t0 TCP *:http-alt (LISTEN)
However,when I run the demo project the Address already in use problem still there.I've also tried to change another port number, but the program has no result if I change the port.
So I wonder who has ever used Arachne API has encountered this problem and can give me a solution.Or does anybody has other good way to run multi spiders in flask app?