I've been googling for days trying to find a straight answer for why this is happening, but can't find anything useful. I have a web2py application that simply reads a database and makes some requests to a REST api. It is a healthcheck monitor so it refreshes itself every minute. There are about 20 or so users at any given time. Here is the error I'm seeing very consistently in the log file:
ERROR:Rocket.Errors.Port8080:Traceback (most recent call last):
File "/opt/apps/web2py/gluon/rocket.py", line 562, in listen
sock = self.wrap_socket(sock)
File "/opt/apps/web2py/gluon/rocket.py", line 506, in wrap_socket
ssl_version = ssl.PROTOCOL_SSLv23)
File "/usr/local/lib/python2.7/ssl.py", line 342, in wrap_socket
ciphers=ciphers)
File "/usr/local/lib/python2.7/ssl.py", line 121, in __init__
self.do_handshake()
File "/usr/local/lib/python2.7/ssl.py", line 281, in do_handshake
self._sslobj.do_handshake()
error: [Errno 104] Connection reset by peer
Based on some googling the most promising piece of information is that someone is trying to connect through a firewall and so it is killing the connection, however I don't understand why it's taking the actual application down. The process is still running, but no one can connect and I have to restart web2py.
I will be very appreciative of any input here. I'm beyond frustration.
Thanks!
The most common source of Connection reset by peer errors is that the remote client decides he doesn't want to contact you anymore, and cancels the interaction (with shutdown/an RST packet). This happens if the user navigates to a different page while the site is loading.
In your case, the remote host gave up on the connection even before you got to read or write anything on it. With the current web2py, this should only output the warning you're seeing, and not terminate anything.
If you have the current web2py, the error of not being able to connect is unrelated to these error messages. If you have an old version of web2py, you should update.
Related
Thanks for taking the time to read through this question.
I've spent quite a bit of time trying to find an answer to this and some time digging through code in pydobc and mysqldb to no avail.
Summary of Problem:
I can connect to a MySQL server using the same method+connection string from many machines, but cannot use the same method from a RHEL server. I continually get an Access denied error; it seems unlikely this is actually the issue though.
Background:
I am using sqlalchemy to connect to a MySQL server hosted on a remote network. I am able to connect to the MySQL server using the same credentials from MySQL Workbench, MySQL Shell, my local Jupyter Notebook, local Python scripts, a virtual Ubuntu 18.04 WSL1 instance, and the troublesome RHEL server- but I cannot use the same connection method on RHEL as the rest of the instances.
I am positive my user has access to read/write tables on the MySQL server (and ostensibly make ODBC connections since I believe I'm doing so from my laptop), so despite the error below, I'm having a hard time believing it is actually a true permission error. That said, I'm happy to listen to any suggestions.
Problem:
When connecting from any of the sources listed above except for the RHEL server, I am able to utilize this code to create an engine and subsequently a connection:
engine = sqlalchemy.create_engine('mysql+mysqldb://{user}:{password}#{server}:{port}/{database}?charset=utf8mb4'.format(user=user
,password=password
,server=server
,port=port
,database=database))
con = engine.connect()
I am then able to use that connection to do any number of things- pd.to_sql(), direct code executions- works like a charm.
The issue comes up when I try to use that same connection info on the RHEL Server, I get the following error:
>>> engine.connect()
Traceback (most recent call last):
File "/python_dir/python3.8/site-packages/sqlalchemy/engine/base.py", line 3211, in _wrap_pool_connect
return fn()
File "/python_dir/python3.8/site-packages/sqlalchemy/pool/base.py", line 307, in connect
return _ConnectionFairy._checkout(self)
File "/python_dir/python3.8/site-packages/sqlalchemy/pool/base.py", line 767, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/python_dir/python3.8/site-packages/sqlalchemy/pool/base.py", line 425, in checkout
rec = pool._do_get()
File "/python_dir/python3.8/site-packages/sqlalchemy/pool/impl.py", line 146, in _do_get
self._dec_overflow()
File "/python_dir/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
compat.raise_(
File "/python_dir/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/python_dir/python3.8/site-packages/sqlalchemy/pool/impl.py", line 143, in _do_get
return self._create_connection()
File "/python_dir/python3.8/site-packages/sqlalchemy/pool/base.py", line 253, in _create_connection
return _ConnectionRecord(self)
File "/python_dir/python3.8/site-packages/sqlalchemy/pool/base.py", line 368, in __init__
self.__connect()
File "/python_dir/python3.8/site-packages/sqlalchemy/pool/base.py", line 611, in __connect
pool.logger.debug("Error on connect(): %s", e)
File "/python_dir/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
compat.raise_(
File "/python_dir/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/python_dir/python3.8/site-packages/sqlalchemy/pool/base.py", line 605, in __connect
connection = pool._invoke_creator(self)
File "/python_dir/python3.8/site-packages/sqlalchemy/engine/create.py", line 578, in connect
return dialect.connect(*cargs, **cparams)
File "/python_dir/python3.8/site-packages/sqlalchemy/engine/default.py", line 584, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/python_dir/python3.8/site-packages/MySQLdb/__init__.py", line 130, in Connect
return Connection(*args, **kwargs)
File "/python_dir/python3.8/site-packages/MySQLdb/connections.py", line 187, in __init__
super().__init__(*args, **kwargs2)
MySQLdb._exceptions.OperationalError: (1045, "Access denied for user 'USER'#'SERVER_ADDRESS' (using password: YES)")
Obviously I've changed a few things in the error for security, but that last line is the only real error I see:
MySQLdb._exceptions.OperationalError: (1045, "Access denied for user 'USER'#'SERVER_ADDRESS' (using password: YES)")
Connecting from RHEL
Despite the failure using the connection info above, I am able to connect to the MySQL server from the RHEL server with a different connection setup:
engine = sqlalchemy.create_engine('mysql+pyodbc://my_dsn')
By using pyodbc, I'm able to connect utilizing a DSN (I think DSN is the right term- this is getting a bit murky for me) that I have configured on the RHEL server in /etc/odbc.ini
Here's an example of what I have for that specific entry (I've retained capitalizations + blanks where they exist in the file, but have changed all sensitive fields):
[ODBC Data Sources]
my_dsn = My Special DSN
[my_dsn]
Driver = /dir/libmyodbc5.so
Description = Description
SERVER = Server
PORT = Port
USER = User
Password = Password
Database = Database
OPTION = 3
SOCKET =
The only major difference that I see between the DSN configuration listed above and the connection string that I'm providing to mysqldb is that the DSN lets me specify the OPTION parameter. I can't figure out how pyodbc uses this, though.
I've tried digging through the documentation from quite a few sources (sqlalchemy,pyodbc,MySQL,StackOverflow,SuperUser, direct source code) to find if/how to specify the OPTION for mysqldb to no avail. I'm not sure if that OPTION is really making that big of a difference, but it's the only piece I can't provide.
That being said, I did find this question that explains what the options are a bit more and this corresponding page that the answer came from. Although that doesn't help me much.
Goal
Ideally, I'd like to be able to run my connections the exact same regardless of server or OS. I'm not sure if that's possible, but it's what I'm attempting to do. I would rather not set up the same DSN on every machine that needs to connect (so the pyodbc option I'm using on RHEL isn't ideal), but no other connection method from that RHEL server will work.
Troubleshooting
I don't have a stellar understanding of the drivers/Linux config, so there's definitely a possibility that I've done something wrong here. Here's some stuff I've tried:
Tried many different connection strings and different drivers/connection info specified
Tried installing various different MySQL drivers on RHEL (with varied success)
Read dozens of questions and articles on similar issues
Other janky configuration attempts that didn't help
Wept softly over my keyboard
Configuration Information:
Problem server: RHEL Version 7
Other machines: Ubuntu 18.04, Windows 10
Python: 3.8
SQLAlchemy: RHEL- 1.4.20, Windows- 1.4.15, Ubuntu- 1.4.22
pyodbc: RHEL- 4.0.30, Windows- 4.0.30, Ubuntu- 4.0.31
If I can provide any more information, please let me know! Really appreciate any ideas.
I'm having trouble connecting to a db I set up in Atlas MongoDB.
I have already read multiple other questions but no answer made my connection work.
I want to point out that I am trying to access it behind a company's proxy (if that has anything to do with it).
My code is this and fails only on the last line where I try to count the documents:
from pymongo import MongoClient
client = MongoClient(
"mongodb+srv://name:pass#iliastrialcluster-1tl2y.azure.mongodb.net/test?retryWrites=true&w=majority&ssl=true&ssl_cert_reqs=CERT_NONE"
)
db = client.get_database('sample_airbnb')
print(db)
listings_and_reviews = db.listingAndReviews
print(listings_and_reviews)
listings_and_reviews.count_documents({})
The error I get is:
Traceback (most recent call last):
File "C:~/mongo_connection1.py", line 11, in <module>
listings_and_reviews.count_documents({})
File "C:~\lib\site-packages\pymongo\collection.py", line 1721, in count_documents
_cmd, self._read_preference_for(session), session)
File "C:~\lib\site-packages\pymongo\mongo_client.py", line 1454, in _retryable_read
read_pref, session, address=address)
File "C:~\lib\site-packages\pymongo\mongo_client.py", line 1253, in _select_server
server = topology.select_server(server_selector)
File "C:~\lib\site-packages\pymongo\topology.py", line 235, in select_server
address))
File "C:~\lib\site-packages\pymongo\topology.py", line 193, in select_servers
selector, server_timeout, address)
File "C:~\lib\site-packages\pymongo\topology.py", line 209, in _select_servers_loop
self._error_message(selector))
pymongo.errors.ServerSelectionTimeoutError: iliastrialcluster-shard-00-00-1tl2y.azure.mongodb.net:27017: timed out,iliastrialcluster-shard-00-02-1tl2y.azure.mongodb.net:27017: timed out,iliastrialcluster-shard-00-01-1tl2y.azure.mongodb.net:27017: timed out
ServerSelectionTimeout error is the client's way of telling you you can't connect to the server. The primary cause of these errors when using MongoDB Atlas is the failure to enable access for the IP address of the node the client is running on.
To verify try connecting using the MongoDB shell mongo using the same connection string. if you get a connection failed then you know it's not a Python problem. Your client code looks OK BTW so I am pretty sure this is what it is.
The connection to the server is lazily evaluated so we don't try to initiate a connection until you make an actual request. In this case the count_documents call. This is why this is the call that generates the error.
#JoeDrumgoole Thanks for your answer. The problem was the proxy after all, after deactivating it the connection works fine. Good to know that the connection to the server is lazily evaluated.
I am trying to get my bottle server so that when one person in a game logs out, everyone can immediately see it. As I am using long polling, there is a request open with all the users.
The bit I am having trouble with is catching the exception that is thrown when the user leaves the page from the long polling that can no longer connect to the page. The error message is here.
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 438, in handle_one_response
self.run_application()
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 425, in run_application
self.process_result()
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 416, in process_result
self.write(data)
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 373, in write
self.socket.sendall(msg)
File "/usr/lib/python2.7/dist-packages/gevent/socket.py", line 509, in sendall
data_sent += self.send(_get_memory(data, data_sent), flags)
File "/usr/lib/python2.7/dist-packages/gevent/socket.py", line 483, in send
return sock.send(data, flags)
error: [Errno 32] Broken pipe
<WSGIServer fileno=3 address=0.0.0.0:8080>: Failed to handle request:
request = GET /refreshlobby/1 HTTP/1.1 from ('127.0.0.1', 53331)
application = <bottle.Bottle object at 0x7f9c05672750>
127.0.0.1 - - [2013-07-07 10:59:30] "GET /refreshlobby/1 HTTP/1.1" 200 160 6.038377
The function to handle that page is this.
#route('/refreshlobby/<id>')
def refreshlobby(id):
while True:
yield lobby.refresh()
gevent.sleep(1)
I tried catching the exception within the function, and in a decorator which I put to wrap #route, neither of which worked. I tried making an #error(500) decorator, but that didn't trigger, either. It seems that this is to do with the internals of bottle.
Edit: I know now that I need to be catching socket.error, but I don't know whereabouts in my code
The WSGI runner
Look closely at the traceback: this in not happening in your function, but in the WSGI runner.
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 438, in handle_one_response
self.run_application()
The way the WSGI runner works, in your case, is:
Receives a request
Gets a partial response from your code
Sends it to the client (this is where the exception is raised)
Repeats steps 2-3
You can't catch this exception
This error is not raised in your code.
It happens when you try to send a response to a client that closed the connection.
You'll therefore not be able to catch this error from within your code.
Alternate solutions
Unfortunately, it's not possible to tell from within the generator (your code) when it stops being consumed.
It's also not a good idea to rely on your generator being garbage collected.
You have a couple other solutions.
"Last seen"
Another way to know when an user disconnects would probably be to record a "last seen", after your yield statement.
You'll be able to identify clients that disconnected if their last seen is far in the past.
Other runner
Another, non-WSGI runner, will be more appropriate for a realtime application. You could give tornado a try.
After updating from 1.7.5 (where everything worked fine) I'm getting a HTTP Error 403: Forbidden when trying to open any sites via localhost. Strange thing is I have pretty much the same setup at home as here at work and everything works there... Might be an issue with proxy server we're using at work, since that's the only difference I can think of? Here's the error log I'm getting, so if anyone knows what's going on please help (;
Traceback (most recent call last):
File "U:\Dev\GAE\lib\cherrypy\cherrypy\wsgiserver\wsgiserver2.py", line 1302, in communicate
req.respond()
File "U:\Dev\GAE\lib\cherrypy\cherrypy\wsgiserver\wsgiserver2.py", line 831, in respond
self.server.gateway(self).respond()
File "U:\Dev\GAE\lib\cherrypy\cherrypy\wsgiserver\wsgiserver2.py", line 2115, in respond
response = self.req.server.wsgi_app(self.env, self.start_response)
File "U:\Dev\GAE\google\appengine\tools\devappserver2\wsgi_server.py", line 246, in __call__
return app(environ, start_response)
File "U:\Dev\GAE\google\appengine\tools\devappserver2\request_rewriter.py", line 311, in _rewriter_middleware
response_body = iter(application(environ, wrapped_start_response))
File "U:\Dev\GAE\google\appengine\tools\devappserver2\python\request_handler.py", line 89, in __call__
self._flush_logs(response.get('logs', []))
File "U:\Dev\GAE\google\appengine\tools\devappserver2\python\request_handler.py", line 220, in _flush_logs
apiproxy_stub_map.MakeSyncCall('logservice', 'Flush', request, response)
File "U:\Dev\GAE\google\appengine\api\apiproxy_stub_map.py", line 94, in MakeSyncCall
return stubmap.MakeSyncCall(service, call, request, response)
File "U:\Dev\GAE\google\appengine\api\apiproxy_stub_map.py", line 320, in MakeSyncCall
rpc.CheckSuccess()
File "U:\Dev\GAE\google\appengine\api\apiproxy_rpc.py", line 156, in _WaitImpl
self.request, self.response)
File "U:\Dev\GAE\google\appengine\ext\remote_api\remote_api_stub.py", line 200, in MakeSyncCall
self._MakeRealSyncCall(service, call, request, response)
File "U:\Dev\GAE\google\appengine\ext\remote_api\remote_api_stub.py", line 226, in _MakeRealSyncCall
encoded_response = self._server.Send(self._path, encoded_request)
File "U:\Dev\GAE\google\appengine\tools\appengine_rpc.py", line 393, in Send
f = self.opener.open(req)
File "U:\Dev\Python\lib\urllib2.py", line 410, in open
response = meth(req, response)
File "U:\Dev\Python\lib\urllib2.py", line 523, in http_response
'http', request, response, code, msg, hdrs)
File "U:\Dev\Python\lib\urllib2.py", line 448, in error
return self._call_chain(*args)
File "U:\Dev\Python\lib\urllib2.py", line 382, in _call_chain
result = func(*args)
File "U:\Dev\Python\lib\urllib2.py", line 531, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 403: Forbidden
INFO 2013-04-19 12:28:52,576 server.py:561] default: "GET / HTTP/1.1" 500 -
INFO 2013-04-19 12:28:52,619 server.py:561] default: "GET /favicon.ico HTTP/1.1" 304 -
Also, the launcher throws an error when closing:
Traceback (most recent call last):
File "launcher\mainframe.pyc", line 327, in OnStop
File "launcher\taskcontroller.pyc", line 167, in Stop
File "launcher\dev_appserver_task_thread.pyc", line 82, in stop
File "launcher\taskthread.pyc", line 107, in stop
File "launcher\platform.pyc", line 397, in KillProcess
pywintypes.error: (5, 'TerminateProcess', 'Access is denied.')
I had this very same issue with my MacOSX when using a proxy server using Google App Engine Launcher 1.8.6. Apparently there's an issue with "proxy_bypass" on "urllib2.py".
There are two possible solutions:
Downgrade to 1.7.5, but, who wants to downgrade?
Edit "[GAE Instalattion path]/google/appengine/tools/appengine_rpc.py" and look for the line that says
opener.add_handler(fancy_urllib.FancyProxyHandler())
In my computer it was line 578, and then put a hash (#) at the beginning of the line, like this:
`#opener.add_handler(fancy_urllib.FancyProxyHandler())`
Save the file, stop and then restart your application. Now dev_appserver.py shouldn't try to use any proxy server at all.
If your application uses any external resources like a SOAP Webservice or something like that and you can't reach the server without the proxy server, then you'll have to downgrade. Please keep in mind that external javascript files (like facebook SDK or similar) are loaded from your browser, not from your application.
Since I'm not using any external REST or SOAP services it worked for me!
Hopefully it will work for you as well.
Try either:
-Accessing it through a different proxy. I.E a . proxy within a proxy
-Accessing it through your local IP i.e 192.168.1.1
I faced the same issue with version 1.9.5. Seems that the API proxy is sending some RPCs to the proxy server, which are then being rejected with HTTP 403 (since proxy servers are generally configured to reject connection attempts to arbitrary ports). In my case I was using the urlfetch module in my app to access external web pages, so disabling the proxy server was not a choice for me.
This is how I worked around the issue some time back (most probably it was based on comments found under this issue, but I cannot remember the exact sources).
NOTE:
For this approach to work, you'll have to know the hostname/IP address and default port of your proxy server, and change them appropriately in the code if you happen to connect to a different proxy server.
When you are not behind the proxy server, you will have to revert the applied changes in order to return to a working state (if you want internet access inside your app).
Here it goes:
Disable proxy settings for the Python (Google App Engine Launcher) environment in some way. (In my case it was easy since I was launching the dev_appserver.py from a Terminal shell (on Linux), and the unset http_proxy and unset https_proxy commands did the trick.)
Edit {App Engine SDK root}/google/appengine/api/urlfetch_stub.py. Find the code block
if _CONNECTION_SUPPORTS_TIMEOUT:
connection = connection_class(host, timeout=deadline)
else:
connection = connection_class(host)
(lines 376-379 in my case) and replace it with:
if _CONNECTION_SUPPORTS_TIMEOUT:
if host[:9] == 'localhost' or host[:9] == '127.0.0.1':
connection = connection_class(host, timeout=deadline)
else:
connection = connection_class('your_proxy_host_goes_here', your_proxy_port_number_goes_here, timeout=deadline)
else:
if host[:9] == 'localhost' or host[:9] == '127.0.0.1':
connection = connection_class(host)
else:
connection = connection_class('your_proxy_host_goes_here', your_proxy_port_number_goes_here)
replacing the placeholders your_proxy_host_goes_here and your_proxy_port_number_goes_here with appropriate values.
(I believe this code can be written more elegantly, though... any Python geeks out there? :) )
In my case, I also had to delete the existing compiled file urlfetch_stub.pyc (located in the same directory as urlfetch_stub.py) because the SDK didn't seem to pick up the changes until I did so.
Now you can use dev_appserver to launch your app, and use urlfetch-backed services within the app, free from HTTP 403 errors.
I'm writing the code to attend a service. I receive POST requests to a particular service and start processing. The whole process is rather simple; I loop trough items in the request and add each item to the database. The problem arise when I have to process a lot of items and the loop takes like three minutes to finish, then when I try to respond:
status = '200 OK'
headers = [('Content-type', 'application/json'),('Access-Control-Allow-Origin','*')]
start_response(status, headers)
return json.dumps(response)
I get this error:
Exception happened during processing of request from ('XXX.XXX.XXX.XXX', 49172)
Traceback (most recent call last):
File "/usr/local/lib/python2.7/SocketServer.py", line 284, in _handle_request_noblock
self.process_request(request, client_address)
File "/usr/local/lib/python2.7/SocketServer.py", line 310, in process_request
self.finish_request(request, client_address)
File "/usr/local/lib/python2.7/SocketServer.py", line 323, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/local/lib/python2.7/SocketServer.py", line 640, in __init__
self.finish()
File "/usr/local/lib/python2.7/SocketServer.py", line 693, in finish
self.wfile.flush()
File "/usr/local/lib/python2.7/socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
I don't know if this helps, but the POST request is a forwarded POST made from a browser to a different domain (that's why the a Access-Control-Allow-Origin) and all the accesses to the database are made using a single object that interacts with the database using SQLAlchemy (can be seen similar to a Java EE DAO pattern).
How do I avoid this error?
You maybe are violating the idea behind REST.
If the proccessing could take some time, the service may want to answer it with an 202 Accepted Response! For a full overview of http response codes follow this link: http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
202 Accepted
The request has been accepted for processing, but the processing has not been completed. The request might or might not eventually be acted upon, as it might be disallowed when processing actually takes place. There is no facility for re-sending a status code from an asynchronous operation such as this.
The 202 response is intentionally non-committal. Its purpose is to allow a server to accept a request for some other process (perhaps a batch-oriented process that is only run once per day) without requiring that the user agent's connection to the server persist until the process is completed. The entity returned with this response SHOULD include an indication of the request's current status and either a pointer to a status monitor or some estimate of when the user can expect the request to be fulfilled.
I could be wrong, but it just looks to me like the socket is timing out. You shouldn't leave the client hanging for a response for more than three minutes.
Instead, you should validate the data and send a message stating that it was received. If necessary, you can use something like AJAX to tell the client that the data was entered after it was received.