I have a twistd plugin that listens on a port and does very simple things. The problem is that when I start it, if the post is not available it just sits there with the process running, but doing nothing. I need the process to exit immediately in this case so the larger system can notice and deal with the problem
I have code like this:
def makeService(options):
root = Resource() # Not what I actually have...
factory = server.Site(root)
server_string = b'tcp:{0}:interface={1}'.format(options['port'], options['interface'])
endpoint = endpoints.serverFromString(reactor, server_string)
service = internet.StreamServerEndpointService(endpoint, factory)
return service
This results in:
2016-12-19T11:42:21-0600] [info] [3082] [-] Log opened.
[2016-12-19T11:42:21-0600] [info] [3082] [-] twistd 15.5.0 (/home/matthew/code-venvs/wgcbap/bin/python 2.7.6) starting up.
[2016-12-19T11:42:21-0600] [info] [3082] [-] reactor class: twisted.internet.epollreactor.EPollReactor.
[2016-12-19T11:42:21-0600] [critical] [3082] [-] Unhandled Error
Traceback (most recent call last):
File "/home/matthew/code-venvs/wgcbap/local/lib/python2.7/site-packages/twisted/scripts/_twistd_unix.py", line 394, in startApplication
service.IService(application).privilegedStartService()
File "/home/matthew/code-venvs/wgcbap/local/lib/python2.7/site-packages/twisted/application/service.py", line 278, in privilegedStartService
service.privilegedStartService()
File "/home/matthew/code-venvs/wgcbap/local/lib/python2.7/site-packages/twisted/application/internet.py", line 352, in privilegedStartService
self._waitingForPort = self.endpoint.listen(self.factory)
File "/home/matthew/code-venvs/wgcbap/local/lib/python2.7/site-packages/twisted/internet/endpoints.py", line 457, in listen
interface=self._interface)
--- <exception caught here> ---
File "/home/matthew/code-venvs/wgcbap/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 121, in execute
result = callable(*args, **kw)
File "/home/matthew/code-venvs/wgcbap/local/lib/python2.7/site-packages/twisted/internet/posixbase.py", line 478, in listenTCP
p.startListening()
File "/home/matthew/code-venvs/wgcbap/local/lib/python2.7/site-packages/twisted/internet/tcp.py", line 984, in startListening
raise CannotListenError(self.interface, self.port, le)
twisted.internet.error.CannotListenError: Couldn't listen on 127.0.0.1:9999: [Errno 98] Address already in use.
And it continues to run, doing nothing....
Adding a line service._raiseSynchronously = True just above the return works, but seems to be undocumented and feels dirty.
Is there an approved way to do this?
Related
I have a server running locally. When I run it on AWS EC2 and send a request from outside on port 8000, I get the following error:
$ uvicorn sql_app.main:app --host="0.0.0.0" --port=8000
INFO: Started server process [9806]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
WARNING: Invalid HTTP request received.
Traceback (most recent call last):
File "/home/ec2-user/.local/lib/python3.7/site-packages/uvicorn/protocols/http/h11_impl.py", line 170, in handle_events
event = self.conn.next_event()
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_connection.py", line 443, in next_event
exc._reraise_as_remote_protocol_error()
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_util.py", line 76, in _reraise_as_remote_protocol_error
raise self
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_connection.py", line 425, in next_event
event = self._extract_next_receive_event()
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_connection.py", line 367, in _extract_next_receive_event
event = self._reader(self._receive_buffer)
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_readers.py", line 73, in maybe_read_from_IDLE_client
request_line_re, lines[0], "illegal request line: {!r}", lines[0]
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_util.py", line 88, in validate
raise LocalProtocolError(msg)
h11._util.RemoteProtocolError: illegal request line: bytearray(b'\x16\x03\x01\x02\x00\x01\x00\x01\xfc\x03\x03\x91\xa5\xe2Y\xf0\xa1\xdd\x1d+\x08\x1c\r\x15X\x1d#\x1e/\xb1N\x00\xb5\xe5\xec\xf3F\x1fm\x03\xa1{> \xa80\xb4\x14\x1aUs\xaa\xcd\xc3<s\xcd\xd1\x17\xdf3\x0e\xdbh\xd1c\x88}\x8c\x1f\xa5\x15\x9aa\x14I\x00 ')
WARNING: Invalid HTTP request received.
Traceback (most recent call last):
File "/home/ec2-user/.local/lib/python3.7/site-packages/uvicorn/protocols/http/h11_impl.py", line 170, in handle_events
event = self.conn.next_event()
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_connection.py", line 443, in next_event
exc._reraise_as_remote_protocol_error()
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_util.py", line 76, in _reraise_as_remote_protocol_error
raise self
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_connection.py", line 425, in next_event
event = self._extract_next_receive_event()
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_connection.py", line 367, in _extract_next_receive_event
event = self._reader(self._receive_buffer)
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_readers.py", line 68, in maybe_read_from_IDLE_client
raise LocalProtocolError("illegal request line")
h11._util.RemoteProtocolError: illegal request line
WARNING: Invalid HTTP request received.
Traceback (most recent call last):
File "/home/ec2-user/.local/lib/python3.7/site-packages/uvicorn/protocols/http/h11_impl.py", line 170, in handle_events
event = self.conn.next_event()
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_connection.py", line 443, in next_event
exc._reraise_as_remote_protocol_error()
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_util.py", line 76, in _reraise_as_remote_protocol_error
raise self
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_connection.py", line 425, in next_event
event = self._extract_next_receive_event()
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_connection.py", line 367, in _extract_next_receive_event
event = self._reader(self._receive_buffer)
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_readers.py", line 68, in maybe_read_from_IDLE_client
raise LocalProtocolError("illegal request line")
h11._util.RemoteProtocolError: illegal request line
WARNING: Invalid HTTP request received.
It would be very nice if you could tell me how to do it on port 80.
I was getting the same arcane WARNING: Invalid HTTP request received. error with an unhelpful stack trace. I tried all of the environment variable tweaks recommended and none worked (see FastAPI issue #680, uvicorn issue #441).
My issue was that when I was calling my FastAPI microservice I was using https when my microservice did not have HTTPS support. I changed the url from https to http and it started working as expected.
Note that if your service requires HTTPS support you can add HTTPS support as Ilgizar Murzakov suggests.
Had the same issue. Disabling http2 on application load balancer helped me.
I had the same issue and solved it by adding HTTPS support. I configured NGINX with LetsEncrypt certificate. Make sure that the ports that you use are open at your security group settings.
this is because uvicorn updated to display detailed error. uvicorn #886
then, use uvicorn 0.13.1 which is before the update. this version won't display the error.
P.S: this update is to fix this error. and uvicorn haven't fixed the error for now. uvicorn #1296
I've written this code and then ran it via the CMD. However when I ran the code this occurred.
Bottle v0.12.18 server starting up (using WSGIRefServer())...
Listening on http://localhost:80/
Hit Ctrl-C to quit.
Traceback (most recent call last):
File "bottle_01.py", line 7, in <module>
run(host='localhost', port = 80, debug=True)
File "C:\Users\Owner\Desktop\BottleWebApp\bottle.py", line 3137, in run
server.run(app)
File "C:\Users\Owner\Desktop\BottleWebApp\bottle.py", line 2789, in run
srv = make_server(self.host, self.port, app, server_cls, handler_cls)
File "C:\Users\Owner\AppData\Local\Programs\Python\Python38-32\lib\wsgiref\simple_server.py", line 154, in make_server
server = server_class((host, port), handler_class)
File "C:\Users\Owner\AppData\Local\Programs\Python\Python38-32\lib\socketserver.py", line 452, in __init__
self.server_bind()
File "C:\Users\Owner\AppData\Local\Programs\Python\Python38-32\lib\wsgiref\simple_server.py", line 50, in server_bind
HTTPServer.server_bind(self)
File "C:\Users\Owner\AppData\Local\Programs\Python\Python38-32\lib\http\server.py", line 138, in server_bind
socketserver.TCPServer.server_bind(self)
File "C:\Users\Owner\AppData\Local\Programs\Python\Python38-32\lib\socketserver.py", line 466, in server_bind
self.socket.bind(self.server_address)
OSError: [WinError 10013] An attempt was made to access a socket in a way forbidden by its access permissions
Now I asked my professor but he's unfamiliar with the issue himself. I searched stackoverflow and I tried to disable my firewall but that didn't seem to help. Any suggestions? Is it an administration issue perhaps? My original code is below.
from bottle import route, run, static_file
#route('/')
def index():
return static_file('webpage_01.html' , root= 'C:/Users/Owner/Desktop/BottleWebApp')
run(host='localhost', port = 80, debug=True)
Does this problem go away when you try port 8000 instead of 80?
You're trying to bind to port 80. I don't know Windows, but on Linux your code would fail because port 80 (like all ports below 1024) is privileged--only root can bind to them. That's why you'll see most tutorials and web framework defaults use a high port number, typically 8000 or 8080.
References:
https://www.w3.org/Daemon/User/Installation/PrivilegedPorts.html
https://en.wikipedia.org/wiki/Registered_port
gunicorn version 19.9.0
Got the following gunicorn config:
accesslog = "access.log"
worker_class = 'sync'
workers = 1
worker_connections = 1000
timeout = 300
graceful_timeout = 300
keepalive = 300
proc_name = 'server'
bind = '0.0.0.0:8080'
name = 'server.py'
preload = True
log_level = "info"
threads = 7
max_requests = 0
backlog = 100
As you can see, the server is configured to run 7 threads.
The server is started with:
gunicorn -c gunicorn_config.py server:app
Here are the number of lines and thread IDs from our log file at the beginning (with the last line being the thread of the main server):
10502 140625414080256
10037 140624842843904
9995 140624859629312
9555 140625430865664
9526 140624851236608
9409 140625405687552
2782 140625422472960
6 140628359804736
So 7 threads are processing the requests. (Already we can see that thread 140625422472960 is processing substantially fewer requests than the other threads.)
But after the lines examined above, thread 140625422472960 just vanishes and the log file only has:
19602 140624859629312
18861 140625405687552
18766 140624851236608
18765 140624842843904
12523 140625414080256
2111 140625430865664
(excluding the main thread here)
From the server logs we could see that the thread received a request and started processing it, but never finished. The client received no response either.
There is no error/warning in the log file, nor in stderr.
And running the app for a little longer, two more threads are gone:
102 140624842843904
102 140624851236608
68 140624859629312
85 140625405687552
How to debug this?
Digging into the stderr logs further, finally found something like this exception stack trace:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 134, in handle
req = six.next(parser)
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/parser.py", line 41, in __next__
self.mesg = self.mesg_class(self.cfg, self.unreader, self.req_count)
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 181, in __init__
super(Request, self).__init__(cfg, unreader)
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 54, in __init__
unused = self.parse(self.unreader)
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 230, in parse
self.headers = self.parse_headers(data[:idx])
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 74, in parse_headers
remote_addr = self.unreader.sock.getpeername()
OSError: [Errno 107] Transport endpoint is not connected
[2018-11-04 17:57:55 +0330] [31] [ERROR] Socket error processing request.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 134, in handle
req = six.next(parser)
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/parser.py", line 41, in __next__
self.mesg = self.mesg_class(self.cfg, self.unreader, self.req_count)
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 181, in __init__
super(Request, self).__init__(cfg, unreader)
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 54, in __init__
unused = self.parse(self.unreader)
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 230, in parse
self.headers = self.parse_headers(data[:idx])
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 74, in parse_headers
remote_addr = self.unreader.sock.getpeername()
OSError: [Errno 107] Transport endpoint is not connected
This is due to this gunicorn bug.
An interim solution until this bug is fixed is to monkey patch gunicorn as done by asantoni.
I have a RabbitMQ 3.6.1 server on Ubuntu 14.04 running properly. I tried to configure an SSL listener according to official documentation. No problems during the startup.
However when trying to establish a connection, I get the following error on Python/pika side (full transcript below):
pika.exceptions.AuthenticationError: EXTERNAL
What does EXTERNAL mean here? How to debug / get further details of the error?
Course of actions (to test I used a Vagrant box and a local connection):
RabbitMQ starts SSL Listener on port 5671 (per /var/log/rabbitmq/rabbit#rabbitmq-server.log):
started SSL Listener on [::]:5671
I execute the pika.BlockingConnection on the client side.
On the server side I can see an incoming connection:
=INFO REPORT==== 17-Apr-2016::17:07:15 ===
accepting AMQP connection <0.2788.0> (127.0.0.1:48404 -> 127.0.0.1:5671)
Client fails with:
pika.exceptions.AuthenticationError: EXTERNAL
Server timeouts:
=ERROR REPORT==== 17-Apr-2016::17:07:25 ===
closing AMQP connection <0.2788.0> (127.0.0.1:48404 -> 127.0.0.1:5671):
{handshake_timeout,frame_header}
Full transcript of the client side:
>>> import pika, ssl
>>> from pika.credentials import ExternalCredentials
>>> ssl_options = ({"ca_certs": "/etc/rabbitmq/certs/testca/cacert.pem",
... "certfile": "/etc/rabbitmq/certs/client/cert.pem",
... "keyfile": "/etc/rabbitmq/certs/client/key.pem",
... "cert_reqs": ssl.CERT_REQUIRED,
... "server_side": False})
>>> host = "localhost"
>>> connection = pika.BlockingConnection(
... pika.ConnectionParameters(
... host, 5671, credentials=ExternalCredentials(),
... ssl=True, ssl_options=ssl_options))
Traceback (most recent call last):
File "<stdin>", line 4, in <module>
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/blocking_connection.py", line 339, in __init__
self._process_io_for_connection_setup()
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/blocking_connection.py", line 374, in _process_io_for_connection_setup
self._open_error_result.is_ready)
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/blocking_connection.py", line 410, in _flush_output
self._impl.ioloop.poll()
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/select_connection.py", line 602, in poll
self._process_fd_events(fd_event_map, write_only)
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/select_connection.py", line 443, in _process_fd_events
handler(fileno, events, write_only=write_only)
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/base_connection.py", line 364, in _handle_events
self._handle_read()
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/base_connection.py", line 415, in _handle_read
self._on_data_available(data)
File "/usr/local/lib/python2.7/dist-packages/pika/connection.py", line 1347, in _on_data_available
self._process_frame(frame_value)
File "/usr/local/lib/python2.7/dist-packages/pika/connection.py", line 1414, in _process_frame
if self._process_callbacks(frame_value):
File "/usr/local/lib/python2.7/dist-packages/pika/connection.py", line 1384, in _process_callbacks
frame_value) # Args
File "/usr/local/lib/python2.7/dist-packages/pika/callback.py", line 60, in wrapper
return function(*tuple(args), **kwargs)
File "/usr/local/lib/python2.7/dist-packages/pika/callback.py", line 92, in wrapper
return function(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/pika/callback.py", line 236, in process
callback(*args, **keywords)
File "/usr/local/lib/python2.7/dist-packages/pika/connection.py", line 1298, in _on_connection_start
self._send_connection_start_ok(*self._get_credentials(method_frame))
File "/usr/local/lib/python2.7/dist-packages/pika/connection.py", line 1077, in _get_credentials
raise exceptions.AuthenticationError(self.params.credentials.TYPE)
pika.exceptions.AuthenticationError: EXTERNAL
>>>
The Python / pika code in the question is correct.
The error:
pika.exceptions.AuthenticationError: EXTERNAL
is reported when client certificate authorisation is not enabled on the RabbitMQ server side. The word EXTERNAL in the error refers to the authentication mechanism as described here.
To enable:
rabbitmq-plugins enable rabbitmq_auth_mechanism_ssl
I've tried to simplify this as much as possible but I'm still getting an error. I have a simple http server (bottle) that upon receiving a post request executes a function which is supposed to quickly fork itself. The parent process simply returns a job ID and closes while the child process continues to process same data (which is a list of URLs). I've removed all the input and output functions and hard coded the data but my program is still crashing. The funny part is when I alter the program to run directly at the command line rather then start an http server and wait for bottle to execute it everything works fine!
#!/usr/bin/python
#This is a comment
import sys, time, bottle, os
from threading import Thread
from Queue import Queue
from bottle import route, run, request, abort
num_fetch_threads = 2
url_queue = Queue()
def fetchURLContent(i, q):
while True:
#print '%s: Looking for URLs in queue' % i
url = q.get()
#print 'URL found: %s' % url[0]
q.task_done()
time.sleep(1)
#route('/', method='POST') # or #route('/login', method='POST')
def main():
urls = ['http://www.yahoo.com', 'http://www.google.com']
newpid = os.fork()
if newpid == 0:
for i in range(num_fetch_threads):
worker = Thread(target=fetchURLContent, args=(i, url_queue))
worker.setDaemon(True)
worker.start()
print 'Queuing: ', url
for url in urls:
url_queue.put(url)
time.sleep(2)
print 'main thread waiting...'
url_queue.join()
print 'Done'
else:
print "Your job id is 5"
return
def webServer():
run(host='33.33.33.10', port=8080)
if __name__ == "__main__":
print 'Listening on 8080...'
webServer()
The error message I get is as follows:
Listening on 8080...
Bottle v0.11.3 server starting up (using WSGIRefServer())...
Listening on http://33.33.33.10:8080/
Hit Ctrl-C to quit.
33.33.33.1 - - [19/Oct/2012 21:21:24] "POST / HTTP/1.1" 200 0
Traceback (most recent call last):
File "/usr/lib/python2.7/wsgiref/handlers.py", line 86, in run
self.finish_response()
File "/usr/lib/python2.7/wsgiref/handlers.py", line 128, in finish_response
self.finish_content()
File "/usr/lib/python2.7/wsgiref/handlers.py", line 246, in finish_content
self.send_headers()
9 url_queue = Queue()
File "/usr/lib/python2.7/wsgiref/handlers.py", line 268, in send_headers
self.send_preamble()
File "/usr/lib/python2.7/wsgiref/handlers.py", line 189, in send_preamble
self._write('HTTP/%s %s\r\n' % (self.http_version,self.status))
File "/usr/lib/python2.7/wsgiref/handlers.py", line 389, in _write
self.stdout.write(data)
File "/usr/lib/python2.7/socket.py", line 324, in write
self.flush()
File "/usr/lib/python2.7/socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
----------------------------------------
Exception happened during processing of request from ('33.33.33.1', 57615)
Traceback (most recent call last):
File "/usr/lib/python2.7/SocketServer.py", line 284, in _handle_request_noblock
self.process_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 310, in process_request
self.finish_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 323, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.7/SocketServer.py", line 640, in __init__
self.finish()
File "/usr/lib/python2.7/SocketServer.py", line 693, in finish
self.wfile.flush()
File "/usr/lib/python2.7/socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
----------------------------------------
Any ideas?
Your main() function terminates immediately without returning anything. Bottle writes an empty HTTP response to the socket and the web server closes the connection.
Your forked off process stays a bit longer in main(), but then terminates too and causes Bottle to write another empty response to the already closed socket. Thats the error you get (broken pipe).
Forking at that point cannot work. HTTP does not allow more than one response per request. You can either block until all work is done and then send a response, or send the response immediately and do the work in a differed thread.