Develop Reference

How to use TCP transport in pysnmp-4.4.6 version

I am using PySNMP 4.4.6 and I want to change carrier to TCP.
That means I want to poll SNMP data using TCP protocol.
I'm getting the following error when I switch carrier to TCP:
File "send-trap-over-ipv4-and-ipv6.py", line 76, in <module>
transportDispatcher.runDispatcher()
File "/usr/local/lib/python2.7/dist-packages/pysnmp/carrier/asyncore/dispatch.py", line 50, in runDispatcher
raise PySnmpError('poll error: %s' % ';'.join(format_exception(*exc_info())))
pysnmp.error.PySnmpError: poll error: Traceback (most recent call last):
; File "/usr/local/lib/python2.7/dist-packages/pysnmp/carrier/asyncore/dispatch.py", line 46, in runDispatcher
use_poll=True, map=self.__sockMap, count=1)
; File "/usr/lib/python2.7/asyncore.py", line 220, in loop
poll_fun(timeout, map)
; File "/usr/lib/python2.7/asyncore.py", line 201, in poll2
readwrite(obj, flags)
; File "/usr/lib/python2.7/asyncore.py", line 123, in readwrite
obj.handle_error()
; File "/usr/lib/python2.7/asyncore.py", line 110, in readwrite
obj.handle_write_event()
; File "/usr/lib/python2.7/asyncore.py", line 468, in handle_write_event
self.handle_write()
; File "/usr/local/lib/python2.7/dist-packages/pysnmp/carrier/asyncore/dgram/base.py", line 155, in handle_write
raise error.CarrierError('sendto() failed for %s: %s' % (transportAddress, sys.exc_info()[1]))
;CarrierError: sendto() failed for 172.16.152.177:162: [Errno 2] No such file or directorycaused by <class 'socket.error'>: [Errno 2] No such file or directory
caused by <class 'pysnmp.carrier.error.CarrierError'>: sendto() failed for 172.16.152.177:162: [Errno 2] No such file or directorycaused by <class 'socket.error'>: [Errno 2] No such file or directory

Unfortunately, pysnmp 4.x does not presently offer TCP transport support. There is an ongoing work on that matter, though. Odds are to have streaming transport support in pysnmp 5.x.

Celery upgrade (3.1->4.1) - Connection reset by peer

We are working with celery at the last year, with ~15 workers, each one defined with concurrency between 1-4.
Recently we upgraded our celery from v3.1 to v4.1
Now we are having the following errors in each one of the workers logs, any ideas what can cause to such error?
2017-08-21 18:33:19,780 94794 ERROR Control command error: error(104, 'Connection reset by peer') [file: pidbox.py, line: 46]
Traceback (most recent call last):
File "/srv/dy/venv/lib/python2.7/site-packages/celery/worker/pidbox.py", line 42, in on_message
self.node.handle_message(body, message)
File "/srv/dy/venv/lib/python2.7/site-packages/kombu/pidbox.py", line 129, in handle_message
return self.dispatch(**body)
File "/srv/dy/venv/lib/python2.7/site-packages/kombu/pidbox.py", line 112, in dispatch
ticket=ticket)
File "/srv/dy/venv/lib/python2.7/site-packages/kombu/pidbox.py", line 135, in reply
serializer=self.mailbox.serializer)
File "/srv/dy/venv/lib/python2.7/site-packages/kombu/pidbox.py", line 265, in _publish_reply
**opts
File "/srv/dy/venv/lib/python2.7/site-packages/kombu/messaging.py", line 181, in publish
exchange_name, declare,
File "/srv/dy/venv/lib/python2.7/site-packages/kombu/messaging.py", line 203, in _publish
mandatory=mandatory, immediate=immediate,
File "/srv/dy/venv/lib/python2.7/site-packages/amqp/channel.py", line 1748, in _basic_publish
(0, exchange, routing_key, mandatory, immediate), msg
File "/srv/dy/venv/lib/python2.7/site-packages/amqp/abstract_channel.py", line 64, in send_method
conn.frame_writer(1, self.channel_id, sig, args, content)
File "/srv/dy/venv/lib/python2.7/site-packages/amqp/method_framing.py", line 178, in write_frame
write(view[:offset])
File "/srv/dy/venv/lib/python2.7/site-packages/amqp/transport.py", line 272, in write
self._write(s)
File "/usr/lib64/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
error: [Errno 104] Connection reset by peer
BTW: our tasks in the form:
#app.task(name='EXAMPLE_TASK'],
bind=True,
base=ConnectionHolderTask)
def example_task(self, arg1, arg2, **kwargs):
# task code

We are also having massive issues with celery... I spend 20% of my time just dancing around weird idle-hang/crash issues with our workers sigh
We had a similar case that was caused by a high concurrency combined with a high worker_prefetch_multiplier, as it turns out fetching thousands of tasks is a good way to frack the connection.
If that's not the case: try to disable the broker pool by setting broker_pool_limit to None.
Just some quick ideas that might (hopefully) help :-)

Python socket ConnectionResetError: [Errno 54] Connection reset by peer vs socket.error: [Errno 104] Connection reset by peer

I'm having trouble debugging my code because I cannot understand the socket error being raised.
Here is the traceback.
Traceback (most recent call last):
File "clickpression.py", line 517, in <module> presser.main()
File "clickpression.py", line 391, in main
File "clickpression.py", line 121, in clickpress self.refresh_proxies(country=country)
File "clickpression.py", line 458, in refresh_proxies self.proxies = self.get_proxies(country=country)
File "helpers.py", line 72, in wrapper return func(*args, **kwargs)
File "clickpression.py", line 264, in get_proxies self.settings.SUPER_PROXY).read().decode('utf-8')
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 161, in urlopen return opener.open(url, data, timeout)
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 463, in open response = self._open(req, data)
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 481, in _open '_open', req)
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 441, in _call_chain result = func(*args)
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 1210, in http_open return self.do_open(http.client.HTTPConnection, req)
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 1185, in do_open r = h.getresponse()
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1171, in getresponse response.begin()
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 351, in begin version, status, reason = self._read_status()
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 313, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socket.py", line 374, in readinto return self._sock.recv_into(b)
ConnectionResetError: [Errno 54] Connection reset by peer
According to the errno library Errno 54 is errno.EXFULL which in the python 3 documentation is explained as exchange full.
To my understanding the Connection reset by peer is Errno 104 i.e errno.ECONNRESET.
So what does errno.EXFULL mean? and why does socket raise the error with a connection reset by peer description instead of exchange full. And or how are the two errors errno.EXFULL and errno.ECONNRESET related?
PS: I read that the errno 54 might be related to http proxy (I'm using a proxy in my code). If so, how?

According to the errno library Errno 54 is errno.EXFULL
Did you determine that by examining errno.errorcode[54]? Anyway - this errno library might be at fault. You could verify the meaning of an error code on your system by looking into errno.h, e. g. with the help of gcc:
gcc -xc -imacros errno.h -Wp,-P -E <(echo ECONNRESET)
Also, the Python documentation says:
To translate a numeric error code to an error message, use
os.strerror().
It may well be that error number 54 is ECONNRESET on your system, and that os.strerror(54) will attest that.
Now that you have verified that os.strerror(54) returns 'Exchange full', I am puzzled why the error number 54 and the error string Connection reset by peer do not match. If that happens on a system with strace or something similar, I would further check which error is returned by the operating system through use of strace -e network on the affected process.
Regarding your question about EXFULL: Its meaning seems somewhat system dependent; e. g. on Linux, EXFULL is returned from only a handful places in the kernel, the only network-related place being in br_if.c concerning network bridges, when no available bridge port number is found (other places are in USB and SCSI drivers).

I tried to use python to crew coin market on OKEX.com using WebSocket,cause the url is an outer address,i used a vpn service provided by us,but it still can work. here is the code an traceback.
from ws4py.client.threadedclient import WebSocketClient
class DummyClient(WebSocketClient):
def opened(self):
# self.send("{'event': 'addChannel', 'channel': 'ok_sub_futureusd_btc_ticker_this_week'}") #发送请求数据格式
# self.send("www.baidu.com")
self.send("{'event':'addChannel','channel':'ok_sub_spot_bch_btc_ticker'}")
def closed(self, code, reason=None):
print("Closed down", code, reason)
#服务器返回消息
def received_message(self, m):
print("recv:", m)
if __name__ == '__main__':
try:
# 服务器连接地址wss://real.okex.com:10440/websocket/okexapi
# ws = DummyClient('wss://real.okcoin.cn:10440/websocket/okcoinapi', protocols=['chat'])
ws = DummyClient('wss://real.okex.com:10440/websocket/okexapi', protocols=['chat'])
ws.connect()
#ws.send("my test...")
ws.run_forever()
except KeyboardInterrupt:
ws.close()

You can try this code to your project:
import ssl
ssl._create_default_https_context = ssl._create_unverified_context
if it not work,make sure the server open TLSv1 support.

Deploying Flower to Heroku

I'm following the instructions on https://github.com/jorilallo/celery-flower-heroku to deploy Flower celery monitoring app to Heroku.
After configuring and deploying my app I see the following in heroku logs:
Traceback (most recent call last):
File "/app/.heroku/python/bin/flower", line 9, in <module>
load_entry_point('flower==0.7.0', 'console_scripts', 'flower')()
File "/app/.heroku/python/lib/python2.7/site-packages/flower/__main__.py", line 11, in main
flower.execute_from_commandline()
File "/app/.heroku/python/lib/python2.7/site-packages/celery/bin/base.py", line 306, in execute_from_commandline
return self.handle_argv(self.prog_name, argv[1:])
File "/app/.heroku/python/lib/python2.7/site-packages/flower/command.py", line 99, in handle_argv
return self.run_from_argv(prog_name, argv)
File "/app/.heroku/python/lib/python2.7/site-packages/flower/command.py", line 75, in run_from_argv
**app_settings)
File "/app/.heroku/python/lib/python2.7/site-packages/flower/app.py", line 40, in __init__
max_tasks_in_memory=max_tasks)
File "/app/.heroku/python/lib/python2.7/site-packages/flower/events.py", line 60, in __init__
state = shelve.open(self._db)
File "/app/.heroku/python/lib/python2.7/shelve.py", line 239, in open
return DbfilenameShelf(filename, flag, protocol, writeback)
File "/app/.heroku/python/lib/python2.7/shelve.py", line 223, in __init__
Shelf.__init__(self, anydbm.open(filename, flag), protocol, writeback)
File "/app/.heroku/python/lib/python2.7/anydbm.py", line 85, in open
return mod.open(file, flag, mode)
File "/app/.heroku/python/lib/python2.7/dumbdbm.py", line 250, in open
return _Database(file, mode)
File "/app/.heroku/python/lib/python2.7/dumbdbm.py", line 71, in __init__
f = _open(self._datfile, 'w')
IOError: [Errno 2] No such file or directory: 'postgres://USERNAME:PASSWORD#ec2-HOST.compute-1.amazonaws.com:5432/DBNAME.dat'
Notice the .dat appendix there? No idea where it comes from, its not present int my DATABASE_URL env variable.
Furthermore, the error above is with flower 0.7. I also tried installing 0.6, with which I do get further (namely the DB is correctly recognized and connection established), but I then get the following warnings once flower starts:
2014-06-19T15:14:02.464424+00:00 app[web.1]: [E 140619 15:14:02 state:138] Failed to inspect workers: '[Errno 104] Connection reset by peer', trying again in 128 seconds
2014-06-19T15:14:02.464844+00:00 app[web.1]: [E 140619 15:14:02 events:103] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 128 seconds.
Loading flower in my browser does show a few tabs of stuff, but there is no data.
How do I resolve these issues?

Flower doesn't support database persistence. It saves the state to file(s) using shelve module.

python xmlrpc timeout error

I am using xmlrpc to contact a local server. On the client side, Sometimes the following socket timeout error and happens and its not a consistent error.
Why is it happening? What could be the reason for socket timeout?
<class 'socket.timeout'>: timed out
args = ('timed out',)
errno = None
filename = None
message = 'timed out'
strerror = None
Traceback on the server side is as follows
Exception happened during processing of request from ('127.0.0.1', 34855)
Traceback (most recent call last):
File "/usr/lib/python2.4/SocketServer.py", line 222, in handle_request
self.process_request(request, client_address)
File "/usr/lib/python2.4/SocketServer.py", line 241, in process_request
self.finish_request(request, client_address)
File "/usr/lib/python2.4/SocketServer.py", line 254, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.4/SocketServer.py", line 521, in __init__
self.handle()
File "/usr/lib/python2.4/BaseHTTPServer.py", line 314, in handle
self.handle_one_request()
File "/usr/lib/python2.4/BaseHTTPServer.py", line 308, in handle_one_request
method()
File "/usr/lib/python2.4/SimpleXMLRPCServer.py", line 441, in do_POST
self.send_response(200)
File "/usr/lib/python2.4/BaseHTTPServer.py", line 367, in send_response
self.send_header('Server', self.version_string())
File "/usr/lib/python2.4/BaseHTTPServer.py", line 373, in send_header
self.wfile.write("%s: %s\r\n" % (keyword, value))
File "/usr/lib/python2.4/socket.py", line 256, in write
self.flush()
File "/usr/lib/python2.4/socket.py", line 243, in flush
self._sock.sendall(buffer)
error: (32, 'Broken pipe')
I killed the server and restarted it. Its working fine now.
What could be the reason?
My machine's RAM went full yesterday night by a process and came back to normal today morning.
Will this error be because of some swapping of processes?

Looks like the client socket it timing out waiting for the server to respond. Is it possible that your server might take a lot time to respond some times? Also, if the server is causing the machine to go into swap, that would slow it down making a timeout possible.
If I remember right, socket timeout is not set in xmlrpc in python. Are you doing socket.setdefaulttimeout somewhere in your code?
If it is expected that your server will take time once in a while, then you could set a higher timeout value using above.
HTH