I am using xmlrpc to contact a local server. On the client side, Sometimes the following socket timeout error and happens and its not a consistent error.
Why is it happening? What could be the reason for socket timeout?
<class 'socket.timeout'>: timed out
args = ('timed out',)
errno = None
filename = None
message = 'timed out'
strerror = None
Traceback on the server side is as follows
Exception happened during processing of request from ('127.0.0.1', 34855)
Traceback (most recent call last):
File "/usr/lib/python2.4/SocketServer.py", line 222, in handle_request
self.process_request(request, client_address)
File "/usr/lib/python2.4/SocketServer.py", line 241, in process_request
self.finish_request(request, client_address)
File "/usr/lib/python2.4/SocketServer.py", line 254, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.4/SocketServer.py", line 521, in __init__
self.handle()
File "/usr/lib/python2.4/BaseHTTPServer.py", line 314, in handle
self.handle_one_request()
File "/usr/lib/python2.4/BaseHTTPServer.py", line 308, in handle_one_request
method()
File "/usr/lib/python2.4/SimpleXMLRPCServer.py", line 441, in do_POST
self.send_response(200)
File "/usr/lib/python2.4/BaseHTTPServer.py", line 367, in send_response
self.send_header('Server', self.version_string())
File "/usr/lib/python2.4/BaseHTTPServer.py", line 373, in send_header
self.wfile.write("%s: %s\r\n" % (keyword, value))
File "/usr/lib/python2.4/socket.py", line 256, in write
self.flush()
File "/usr/lib/python2.4/socket.py", line 243, in flush
self._sock.sendall(buffer)
error: (32, 'Broken pipe')
I killed the server and restarted it. Its working fine now.
What could be the reason?
My machine's RAM went full yesterday night by a process and came back to normal today morning.
Will this error be because of some swapping of processes?
Looks like the client socket it timing out waiting for the server to respond. Is it possible that your server might take a lot time to respond some times? Also, if the server is causing the machine to go into swap, that would slow it down making a timeout possible.
If I remember right, socket timeout is not set in xmlrpc in python. Are you doing socket.setdefaulttimeout somewhere in your code?
If it is expected that your server will take time once in a while, then you could set a higher timeout value using above.
HTH
Related
I'm writing a chat program. Each of my clients has an open get request to the server in a separate thread (and another thread for posting their own messages). I don't want to have a lot of overhead. That is, clients don't send get requests frequently to see if there have been any unseen messages. Instead, they always have exactly one open get request to get the new messages, and as soon as the server responded to them with new unseen messages, they immediately send another get request to the server to stay updated and so on.
So on the client-side, I have something like this:
def coms():
headers = {'data': myAut.strip()}
resp = requests.get("http://localhost:8081/receive", headers=headers,timeout=1000000)
print(resp.text)
t = threading.Thread(target=coms, args=())
t.start()
On the server-side, I have something like this:
def do_GET(self):
if self.path == '/receive':
auth=self.headers['data']
#Using auth, find who has sent this message
u=None
for i in range(len(users)):
print(users[i].aut,auth)
if users[i].aut==auth:
u=users[i]
break
t=threading.Thread(target=long_Poll,args=(u,self))
t.start()
and
def long_Poll(client,con):
while True:
if len(client.unreadMessages) != 0:
print("IM GONNA RESPOND")
con.end_headers()
con.wfile.write(bytes(client.unreadMessages, "utf8"))
client.unreadMessages=[]
break
con.send_response(200)
con.end_headers()
And the logic behind this is that the servers want to do the long-polling, that is, it keeps GET/receive requests open in another busy-waiting thread. When any client sends a message to the server via POST/message it just adds this new message to other clients unseenMessages and so once their thread is running, they come out of the while True: loop, and the server gives them the new messages. In other words, I want to hold client's GET/receive open and not respond it as long as I want.
This process might take so long time. Maybe the chatroom is idle and there is no messages for a long time.
Right now the problem I have is that as soon as my client sends its first GET/receive message, it gets this error, even though I have set the timeout value in GET/receive request to be so much.
C:\Users\erfan\Desktop\web\client\venv\Scripts\python.exe C:\Users\erfan\Desktop\web\client\Client.py
Hossein
Welcome to chatroom Hossein ! Have a nice time !
Exception in thread Thread-1:
Traceback (most recent call last):
File "C:\Users\erfan\Desktop\web\client\venv\lib\site-packages\urllib3\connectionpool.py", line 677, in urlopen
chunked=chunked,
File "C:\Users\erfan\Desktop\web\client\venv\lib\site-packages\urllib3\connectionpool.py", line 426, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "C:\Users\erfan\Desktop\web\client\venv\lib\site-packages\urllib3\connectionpool.py", line 421, in _make_request
httplib_response = conn.getresponse()
File "C:\Users\erfan\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1321, in getresponse
response.begin()
File "C:\Users\erfan\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 296, in begin
version, status, reason = self._read_status()
File "C:\Users\erfan\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 265, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\erfan\Desktop\web\client\venv\lib\site-packages\requests\adapters.py", line 449, in send
timeout=timeout
File "C:\Users\erfan\Desktop\web\client\venv\lib\site-packages\urllib3\connectionpool.py", line 727, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "C:\Users\erfan\Desktop\web\client\venv\lib\site-packages\urllib3\util\retry.py", line 410, in increment
raise six.reraise(type(error), error, _stacktrace)
File "C:\Users\erfan\Desktop\web\client\venv\lib\site-packages\urllib3\packages\six.py", line 734, in reraise
raise value.with_traceback(tb)
File "C:\Users\erfan\Desktop\web\client\venv\lib\site-packages\urllib3\connectionpool.py", line 677, in urlopen
chunked=chunked,
File "C:\Users\erfan\Desktop\web\client\venv\lib\site-packages\urllib3\connectionpool.py", line 426, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "C:\Users\erfan\Desktop\web\client\venv\lib\site-packages\urllib3\connectionpool.py", line 421, in _make_request
httplib_response = conn.getresponse()
File "C:\Users\erfan\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1321, in getresponse
response.begin()
File "C:\Users\erfan\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 296, in begin
version, status, reason = self._read_status()
File "C:\Users\erfan\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 265, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\erfan\AppData\Local\Programs\Python\Python37\lib\threading.py", line 917, in _bootstrap_inner
self.run()
File "C:\Users\erfan\AppData\Local\Programs\Python\Python37\lib\threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\erfan\Desktop\web\client\Client.py", line 13, in coms
resp = requests.get("http://localhost:8081/receive", headers=headers,timeout=1000000)
File "C:\Users\erfan\Desktop\web\client\venv\lib\site-packages\requests\api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "C:\Users\erfan\Desktop\web\client\venv\lib\site-packages\requests\api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\erfan\Desktop\web\client\venv\lib\site-packages\requests\sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\erfan\Desktop\web\client\venv\lib\site-packages\requests\sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "C:\Users\erfan\Desktop\web\client\venv\lib\site-packages\requests\adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
===========================================================================
UPDATE:
The strange part is whenever I edit the GET/receive module to this:
def do_GET(self):
while True:
pass
everything works fine.
But when I do :
def do_GET(self):
t=threading.Thread(target=long_Poll,args=(self))
t.start()
def long_Poll(con):
client =None
while True:
pass
It gives the same error to the client!
I mean the problem is because I pass the self object to another function to respond? maybe it interrupts the connection? I remember having the same problem in Java socket programming where I would encounter to some bugs sometimes when I wanted to use a socket to communicate in two functions? However, here I only want to communicate in the long-polling function not anywhere else.
=======================================
update:
I also put my server and client code here. For brevity, I post the paste.ubuntu links here.
Client:
https://paste.ubuntu.com/p/qJmRjYy4Y9/
Server:
https://paste.ubuntu.com/p/rVyHPs4Rjz/
First time a client types, he enters his name and after that he starts sending GET/receive requests. Client can then send his messages to other people by sending POST/message requests. Any time a user send a message to the server, the server finds him (by his auth) and updates all other clients unseenMessages so that whenever their long-polling thread continued, they'll get the new messages and their clients also send another GET/receive message immediately.
I have found the answer. I was trying to have a multithreaded server using single thread syntax!
I followed this thread for having a multithreaded HTTP server
Multithreaded web server in python
When running unittests I often get error: [Errno 32] Broken pipe. It seems to be a harmless error that happens during testing, but I haven't been able to prevent or otherwise suppress it.
Some things that I have tried include changing SIGPIPE handling to SIG_DFL and running the app with threaded=True. If I were to try/except I don't know which code to wrap since this is in the context of unittesting.
I don't care about catching/preventing the error as much as just suppressing its output while all the other tests finish running. What else should I try?
Edit:
Here is some example code that often, but does not always result in the error:
def test_register(self):
self.driver.get(self.get_server_url() + url_for(u'register'))
body = self.driver.find_element_by_id(u'body')
username_input = body.find_element_by_id(u'username')
username_input.send_keys(self.USER1_DISPLAY_USERNAME)
privacy_policy = body.find_element_by_id(u'privacy_policy')
privacy_policy.click()
#NOTE: not shown - more lines filling out form elements exactly like the above lines
self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
register_button = body.find_element_by_id(u'onclick-register')
register_button.click()
And here is an example of the error message:
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 44668)
Traceback (most recent call last):
File "/usr/lib/python2.7/SocketServer.py", line 599, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.7/SocketServer.py", line 657, in __init__
self.finish()
File "/usr/lib/python2.7/SocketServer.py", line 716, in finish
self.wfile.close()
File "/usr/lib/python2.7/socket.py", line 283, in close
self.flush()
File "/usr/lib/python2.7/socket.py", line 307, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
----------------------------------------
This seems to only happen with integration tests. I'm using Selenium and a LiveServerTestCase with a LIVESERVER_PORT port set to 8943. The mention of port 44668 in the error message looks suspicious.
My problem sounds exactly like Suppress print output in unittests but none of those solutions work.
I found a solution by redirecting stderr and stdout.
In create_app() add the following:
_stderr, _stdout = sys.stderr, sys.stdout
null = open(os.devnull,'wb')
sys.stdout = sys.stderr = null
I have a web service(REST) where one request might take up to 30 sec to return an answer (lots of calculation). There is a risk, that during the calculation, the client webbrowser aborts(?) the existing connection and retries. Here is the console-output of the server-side:
Exception happened during processing of request from ('127.0.0.1', 53209)
Traceback (most recent call last):
File "C:\Users\tmx\Anaconda2\lib\SocketServer.py", line 290, in _handle_request_noblock
self.process_request(request, client_address)
File "C:\Users\tmx\Anaconda2\lib\SocketServer.py", line 318, in process_request
self.finish_request(request, client_address)
File "C:\Users\tmx\Anaconda2\lib\SocketServer.py", line 331, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\Users\tmx\Anaconda2\lib\SocketServer.py", line 654, in __init__
self.finish()
File "C:\Users\tmx\Anaconda2\lib\SocketServer.py", line 713, in finish
self.wfile.close()
File "C:\Users\tmx\Anaconda2\lib\socket.py", line 283, in close
self.flush()
File "C:\Users\tmx\Anaconda2\lib\socket.py", line 307, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 10053] An established connection was aborted by the software in your host machine
One option is what I thought of is to somehow notify the client that "I'm alivem but the request is still needs some more time", or to somehow set the timeout on server side. What are the possibilities?
It's difficult to run code in Flask after you've already returned some data. Your options are to either use something like a task queue (see Celery), or to yield your response in multiple parts.
Views in Flask can return strings, but they can also return iterables that contain strings. So you could return "abc", ["abc"], or a generator that will yield "abc". If you do your processing between yields, data will get sent to the client while the request is still running.
Take a look at the following example:
def generator_that_does_the_calculation():
sleep(1)
yield "I'm alive, but I need some time\n"
sleep(1)
yield "Still alive here\n"
sleep(1)
yield "Done\n"
#app.route('/calculate')
def calculate():
return Response(generator_that_does_the_calculation())
Can anyone help me with my python WebSockets server for some reason whenever I try to send JSON data there are many errors, here is a link to my code:
http://textuploader.com/v5kn
My Client hasn't changed since the last update and doesn't send anything after connection to the server so it should be fine.
Heres the error:
connection established ('127.0.0.1', 51777)
Handshaking...
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 51429)
Traceback (most recent call last):
File "C:\Python27\lib\SocketServer.py", line 599, in process_request_thread
self.finish_request(request, client_address)
File "C:\Python27\lib\SocketServer.py", line 334, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\Python27\lib\SocketServer.py", line 655, in __init__
self.handle()
File "C:/Users/Heslop/Desktop/Projects/SnakeServer/server.py", line 64, in handle
self.read_next_message()
File "C:/Users/Heslop/Desktop/Projects/SnakeServer/server.py", line 68, in read_next_message
length = ord(self.rfile.read(2)[1]) & 127
IndexError: string index out of range
In fact you can try the third-lib for ServerSide-ClientSide msg(frame) encoding and decoding for you, so you can focus on app-building.
Here is my little wrapping for it: https://github.com/HelloLyfing/py-websocket-helper
Since I am the only one who ever used this wrapper, so if you got any problems, please let me know.
I've tried to simplify this as much as possible but I'm still getting an error. I have a simple http server (bottle) that upon receiving a post request executes a function which is supposed to quickly fork itself. The parent process simply returns a job ID and closes while the child process continues to process same data (which is a list of URLs). I've removed all the input and output functions and hard coded the data but my program is still crashing. The funny part is when I alter the program to run directly at the command line rather then start an http server and wait for bottle to execute it everything works fine!
#!/usr/bin/python
#This is a comment
import sys, time, bottle, os
from threading import Thread
from Queue import Queue
from bottle import route, run, request, abort
num_fetch_threads = 2
url_queue = Queue()
def fetchURLContent(i, q):
while True:
#print '%s: Looking for URLs in queue' % i
url = q.get()
#print 'URL found: %s' % url[0]
q.task_done()
time.sleep(1)
#route('/', method='POST') # or #route('/login', method='POST')
def main():
urls = ['http://www.yahoo.com', 'http://www.google.com']
newpid = os.fork()
if newpid == 0:
for i in range(num_fetch_threads):
worker = Thread(target=fetchURLContent, args=(i, url_queue))
worker.setDaemon(True)
worker.start()
print 'Queuing: ', url
for url in urls:
url_queue.put(url)
time.sleep(2)
print 'main thread waiting...'
url_queue.join()
print 'Done'
else:
print "Your job id is 5"
return
def webServer():
run(host='33.33.33.10', port=8080)
if __name__ == "__main__":
print 'Listening on 8080...'
webServer()
The error message I get is as follows:
Listening on 8080...
Bottle v0.11.3 server starting up (using WSGIRefServer())...
Listening on http://33.33.33.10:8080/
Hit Ctrl-C to quit.
33.33.33.1 - - [19/Oct/2012 21:21:24] "POST / HTTP/1.1" 200 0
Traceback (most recent call last):
File "/usr/lib/python2.7/wsgiref/handlers.py", line 86, in run
self.finish_response()
File "/usr/lib/python2.7/wsgiref/handlers.py", line 128, in finish_response
self.finish_content()
File "/usr/lib/python2.7/wsgiref/handlers.py", line 246, in finish_content
self.send_headers()
9 url_queue = Queue()
File "/usr/lib/python2.7/wsgiref/handlers.py", line 268, in send_headers
self.send_preamble()
File "/usr/lib/python2.7/wsgiref/handlers.py", line 189, in send_preamble
self._write('HTTP/%s %s\r\n' % (self.http_version,self.status))
File "/usr/lib/python2.7/wsgiref/handlers.py", line 389, in _write
self.stdout.write(data)
File "/usr/lib/python2.7/socket.py", line 324, in write
self.flush()
File "/usr/lib/python2.7/socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
----------------------------------------
Exception happened during processing of request from ('33.33.33.1', 57615)
Traceback (most recent call last):
File "/usr/lib/python2.7/SocketServer.py", line 284, in _handle_request_noblock
self.process_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 310, in process_request
self.finish_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 323, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.7/SocketServer.py", line 640, in __init__
self.finish()
File "/usr/lib/python2.7/SocketServer.py", line 693, in finish
self.wfile.flush()
File "/usr/lib/python2.7/socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
----------------------------------------
Any ideas?
Your main() function terminates immediately without returning anything. Bottle writes an empty HTTP response to the socket and the web server closes the connection.
Your forked off process stays a bit longer in main(), but then terminates too and causes Bottle to write another empty response to the already closed socket. Thats the error you get (broken pipe).
Forking at that point cannot work. HTTP does not allow more than one response per request. You can either block until all work is done and then send a response, or send the response immediately and do the work in a differed thread.