Develop Reference

Mac : Application exited with code 255, illegal signal 0X04

I am working on a python app which includes a CEFPython3 browser inside the app. The app is getting correctly compiled and when I run our_app.py, everything works just fine. But when I package it with py2app, then I get these errors. What exactly is the problem? I tried searching other threads, but each user seems to have a different issue.
Note : In our app, we are setting a flag for GPU for CEFPython3 as disable-gpu=true
Error log :
{\rtf1\ansi\ansicpg1252\cocoartf1561\cocoasubrtf200
{\fonttbl\f0\fswiss\fcharset0 Helvetica;}
{\colortbl;\red255\green255\blue255;}
{\*\expandedcolortbl;;}
\paperw11900\paperh16840\margl1440\margr1440\vieww28600\viewh15000\viewkind0
\pard\tx566\tx1133\tx1700\tx2267\tx2834\tx3401\tx3968\tx4535\tx5102\tx5669\tx6236\tx6803\pardirnatural\partightenfactor0
\f0\fs24 \cf0 May 24 13:04:48 Akshays-MacBook-Air syslogd[46]: ASL Sender Statistics\
May 24 13:05:28 Akshays-MacBook-Air com.apple.xpc.launchd[1] (org.pythonmac.unspecified.APP_NAME.12348[954]): Service exited with abnormal code: 255\
May 24 13:05:37 Akshays-MacBook-Air sudo[964]: akshaysulakhe : TTY=ttys000 ; PWD=/Users/akshaysulakhe/Downloads/APP_NAMEwx ; USER=root ; COMMAND=/sbin/dmesg\
May 24 13:05:51 Akshays-MacBook-Air sudo[971]: akshaysulakhe : TTY=ttys000 ; PWD=/Users/akshaysulakhe/Downloads/APP_NAMEwx ; USER=root ; COMMAND=/sbin/dmesg\
May 24 13:06:09 Akshays-MacBook-Air com.apple.xpc.launchd[1] (com.apple.quicklook[973]): Endpoint has been activated through legacy launch(3) APIs. Please switch to XPC or bootstrap_check_in(): com.apple.quicklook\
May 24 13:07:04 Akshays-MacBook-Air sudo[974]: akshaysulakhe : TTY=ttys000 ; PWD=/Users/akshaysulakhe/Downloads/APP_NAMEwx ; USER=root ; COMMAND=/bin/launchctl remove com.cylance.agent_service\
May 24 13:07:10 Akshays-MacBook-Air com.apple.xpc.launchd[1] (org.pythonmac.unspecified.APP_NAME.12352[966]): Service exited with abnormal code: 255\
May 24 13:07:14 Akshays-MacBook-Air com.apple.xpc.launchd[1] (org.pythonmac.unspecified.APP_NAME.12356[976]): Service exited with abnormal code: 255\
May 24 13:07:19 Akshays-MacBook-Air com.apple.xpc.launchd[1] (com.apple.quicklook[981]): Endpoint has been activated through legacy launch(3) APIs. Please switch to XPC or bootstrap_check_in(): com.apple.quicklook\
May 24 13:07:35 Akshays-MacBook-Air com.apple.xpc.launchd[1] (org.pythonmac.unspecified.APP_NAME.12356[987]): Service exited with abnormal code: 255\
May 24 13:08:30 Akshays-MacBook-Air com.apple.xpc.launchd[1] (com.apple.quicklook[992]): Endpoint has been activated through legacy launch(3) APIs. Please switch to XPC or bootstrap_check_in(): com.apple.quicklook\
May 24 13:09:27 Akshays-MacBook-Air TextEdit[993]: assertion failed: 17D102: libxpc.dylib + 72637 [F7E5F1BC-614B-39CB-B6CE-92A9C7B7EC0B]: 0x89\
May 24 13:09:28 Akshays-MacBook-Air com.apple.xpc.launchd[1] (com.apple.imfoundation.IMRemoteURLConnectionAgent): Unknown key for integer: _DirtyJetsamMemoryLimit\
May 24 13:12:19 Akshays-MacBook-Air com.apple.xpc.launchd[1] (com.apple.imfoundation.IMRemoteURLConnectionAgent): Unknown key for integer: _DirtyJetsamMemoryLimit\
}
Crash_log :
{\rtf1\ansi\ansicpg1252\cocoartf1561\cocoasubrtf200
{\fonttbl\f0\fswiss\fcharset0 Helvetica;}
{\colortbl;\red255\green255\blue255;}
{\*\expandedcolortbl;;}
\paperw11900\paperh16840\margl1440\margr1440\vieww10800\viewh8400\viewkind0
\pard\tx566\tx1133\tx1700\tx2267\tx2834\tx3401\tx3968\tx4535\tx5102\tx5669\tx6236\tx6803\pardirnatural\partightenfactor0
\f0\fs24 \cf0 Process: APP_NAME [846]\
Path: /Users/USER/Downloads/*/APP_NAME.app/Contents/MacOS/APP_NAME\
Identifier: org.pythonmac.unspecified.APP_NAME\
Version: 0.0.0 (0.0.0)\
Code Type: X86-64 (Native)\
Parent Process: ??? [1]\
Responsible: APP_NAME [846]\
User ID: 501\
\
Date/Time: 2018-05-24 12:44:16.609 +0530\
OS Version: Mac OS X 10.13.3 (17D102)\
Report Version: 12\
Anonymous UUID: 83F4A79A-60D6-23EF-2BDC-2B052BA014A7\
\
Sleep/Wake UUID: CE376F54-DDE9-4CB5-960D-A08762CAFD0C\
\
Time Awake Since Boot: 7900 seconds\
Time Since Wake: 1900 seconds\
\
System Integrity Protection: enabled\
\
Crashed Thread: 0 Dispatch queue: com.apple.main-thread\
\
Exception Type: EXC_BAD_INSTRUCTION (SIGILL)\
Exception Codes: 0x0000000000000001, 0x0000000000000000\
Exception Note: EXC_CORPSE_NOTIFY\
\
Termination Signal: Illegal instruction: 4\
Termination Reason: Namespace SIGNAL, Code 0x4\
Terminating Process: exc handler [0]\
\
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread\
0 org.chromium.ContentShell.framework 0x0000000107e5e1f7 0x106a38000 + 21127671\
1 org.chromium.ContentShell.framework 0x0000000107e87e4e 0x106a38000 + 21298766\
2 org.chromium.ContentShell.framework 0x0000000107e87c44 0x106a38000 + 21298244\
3 org.chromium.ContentShell.framework 0x0000000106a3a2c4 cef_initialize + 276\
4 cefpython_py36.so 0x0000000106931213 CefInitialize(CefMainArgs const&, CefStructBase<CefSettingsTraits> const&, scoped_refptr<CefApp>, void*) + 163\
5 cefpython_py36.so 0x00000001068d20ff __pyx_pf_14cefpython_py36_22Initialize(_object*, _object*, _object*, _object*) + 43535\
6 cefpython_py36.so 0x00000001068c74d4 __pyx_pw_14cefpython_py36_23Initialize(_object*, _object*, _object*) + 164\
7 org.python.python 0x0000000101867685 _PyCFunction_FastCallDict + 229\
8 org.python.python 0x00000001018678a8 _PyCFunction_FastCallKeywords + 136\
9 org.python.python 0x00000001018f05e4 call_function + 612\
10 org.python.python 0x00000001018f2506 _PyEval_EvalFrameDefault + 6918\
11 org.python.python 0x00000001018f032e fast_function + 606\
VIRTUAL REGION \
REGION TYPE SIZE COUNT (non-coalesced) \
=========== ======= ======= \
Activity Tracing 256K 2 \
CoreUI image file 116K 2 \
Kernel Alloc Once 8K 2 \
MALLOC 52.9M 46 \
MALLOC guard page 32K 9 \
MALLOC_LARGE (reserved) 512K 3 reserved VM address space (unallocated)\
STACK GUARD 56.0M 7 \
Stack 10.5M 7 \
VM_ALLOCATE 40K 4 \
__DATA 40.6M 321 \
__FONT_DATA 4K 2 \
__LINKEDIT 201.1M 36 \
__TEXT 329.0M 319 \
__UNICODE 560K 2 \
mapped file 33.5M 5 \
shared memory 628K 10 \
=========== ======= ======= \
TOTAL 725.8M 761 \
TOTAL, minus reserved VM space 725.3M 761 \
\
}
setup.py from py2app :
from setuptools import setup
APP = ['Our_APP.py']
DATA_FILES = []
OPTIONS = {}
setup(
app=APP,
data_files=DATA_FILES,
options={'py2app': OPTIONS},
setup_requires=['py2app','cefpython3'],
)

See how the exit code 255 is handled in the wxpython.py example:
if MAC:
# On Mac things work differently, other steps are required
self.browser.CloseBrowser()
self.clear_browser_references()
self.Destroy()
global g_count_windows
g_count_windows -= 1
if g_count_windows == 0:
cef.Shutdown()
wx.GetApp().ExitMainLoop()
# Call _exit otherwise app exits with code 255 (Issue #162).
# noinspection PyProtectedMember
os._exit(0)
Full source:
https://github.com/cztomczak/cefpython/blob/2fb9ae156802c12b9941f61661318386dbf3bfe2/examples/wxpython.py#L167
Corresponding issue in the tracker:
https://github.com/cztomczak/cefpython/issues/162

Scapy: How to access a Custom Layer

I am trying to understand how to add a custom dissector in Scapy. I am using Python 3.4 and Scapy3 if that has any bearing on the result.
I have a stupid class, and the packet.show2() command correctly renders the nested packet. But I can not access the new Layers field values.
Scary Class and bind_layer follows...
from scapy.all import *
#Create simple Class
class DUMBO(Packet):
fields_desc = [
ShortField('ears',0),
ShortField('legs',0),
ShortField('trunk',0)
]
#Inform TCP that ports 9898 are this protocol
bind_layers(TCP, DUMBO, sport=9898, dport=9898)
I make a packet like this
#Make a Packet
pack=IP()/TCP(sport=9898, dport=9898)/Raw(load=b'\x00\x02\x00\x04\x00\x01')
Looking at the Packet I have created using ls yields
version : BitField = 4 (4)
ihl : BitField = None (None)
tos : XByteField = 0 (0)
len : ShortField = None (None)
id : ShortField = 1 (1)
flags : FlagsField = 0 (0)
frag : BitField = 0 (0)
ttl : ByteField = 64 (64)
proto : ByteEnumField = 6 (0)
chksum : XShortField = None (None)
src : Emph = '127.0.0.1' (None)
dst : Emph = '127.0.0.1' ('127.0.0.1')
options : PacketListField = [] ([])
--
sport : ShortEnumField = 9898 (20)
dport : ShortEnumField = 9898 (80)
seq : IntField = 0 (0)
ack : IntField = 0 (0)
dataofs : BitField = None (None)
reserved : BitField = 0 (0)
flags : FlagsField = 2 (2)
window : ShortField = 8192 (8192)
chksum : XShortField = None (None)
urgptr : ShortField = 0 (0)
options : TCPOptionsField = {} ({})
--
load : StrField = b'\x00\x02\x00\x04\x00\x01' (b'')
And display it using Show2 it all looks good
pack.show2()
###[ IP ]###
version = 4
ihl = 5
tos = 0x0
len = 46
id = 1
flags =
frag = 0
ttl = 64
proto = tcp
chksum = 0x7cc7
src = 127.0.0.1
dst = 127.0.0.1
\options \
###[ TCP ]###
sport = monkeycom
dport = monkeycom
seq = 0
ack = 0
dataofs = 5
reserved = 0
flags = S
window = 8192
chksum = 0x447f
urgptr = 0
options = []
###[ DUMBO ]###
ears = 2
legs = 4
trunk = 1
I now want to access the DUMBO Layer fields
But
PACK[DUMBO].ears
Is not correct - as the packet when displayed as pack.show() still has the Payload as Raw....
What am I missing ??

Ok - This is my solution....
pack=IP()/TCP(sport=19898, dport=19898)/Raw(load=b'\x00\x02\x00\x04\x00\x01')
#Cast this packet back
pack=IP(bytes(pack))
pack.show2()
pack.show()
if DUMBO in pack:
print('Elephant in the house')
print('Ears -> {}'.format(pack[DUMBO].ears))
If anyone else can improve on this I would be happy on seeing the solution.

Note: I'm just getting started with Scapy, so I can't promise this is the correct/only way to go.
As per Documentation: Add new protocols to Scapy, put the code with the protocol definition in a seperate python file. Make sure you also set the required headers. Then place that file either in scapy/layers or scapy/contrib.
After that, the protocol can be loaded with load_layer(...) or load_contrib(...) where you plan on using it.
For DUMBO we'll go with contrib.
dumbo.py:
# scapy.contrib.description = Dumbo the elephant
# scapy.contrib.status = loads
from scapy.packet import Packet, bind_layers
from scapy.fields import ShortField
from scapy.layers.inet import TCP
#Create simple Class
class DUMBO(Packet):
fields_desc = [
ShortField('ears',0),
ShortField('legs',0),
ShortField('trunk',0)
]
#Inform TCP that ports 9898 are this protocol
bind_layers(TCP, DUMBO, sport=9898, dport=9898)
Now let's use it:
$ scapy
>>> load_contrib("dumbo")
>>> pack1=IP()/TCP(sport=9898, dport=9898)/DUMBO(b'\x00\x02\x00\x04\x00\x01')
>>> pack2=IP()/TCP(sport=9898, dport=9898)/DUMBO(ears=2, legs=4, trunk=1)
>>> pack1
<IP frag=0 proto=tcp |<TCP sport=9898 dport=9898 |<DUMBO ears=2 legs=4 trunk=1 |>>>
>>> pack2
<IP frag=0 proto=tcp |<TCP sport=9898 dport=9898 |<DUMBO ears=2 legs=4 trunk=1 |>>>
>>> pack1[DUMBO].ears
2
>>> pack2[DUMBO].ears
2
Hope this helps somebody who stumbles upon this question.
Versions used: Python v3.8.5 ; Scapy v2.4.5

Can not connect to an abstract unix socket in python

I have a server written in c++ which creates and binds to an abstract unix socket with a namespace address of "\0hidden". I also have a client which is written in c++ also and this client can successfully connect to my server. BTW, I do not have the source code of this client. Now I am trying to connect to my server using a client I have written in python with no success. I do not understand why my python client is not working. I am posting the relevant parts of my server and client codes.
Server
#define UD_SOCKET_PATH "\0hidden"
struct sockaddr_un addr;
int fd,cl;
if ( (fd = socket(AF_UNIX, SOCK_STREAM, 0)) == -1)
{
syslog(LOG_CRIT, "Error creating socket!");
exit(1);
}
memset(&addr, 0, sizeof(addr));
addr.sun_family = AF_UNIX;
strncpy(addr.sun_path, UD_SOCKET_PATH, sizeof(addr.sun_path)-1);
unlink(UD_SOCKET_PATH);
if (::bind(fd, (struct sockaddr*)&addr, sizeof(addr)) == -1)
{
syslog(LOG_CRIT, "Bind error");
exit(1);
}
if (listen(fd, MAX_CONN_PENDING) == -1)
{
syslog(LOG_CRIT, "Listen error");
exit(1);
}
syslog(LOG_INFO, "Start listening.");
And my client code
#! /opt/python/bin/python
import os
import socket
import sys
# Create a UDS socket
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
server_address = "\0hidden"
print >>sys.stderr, 'connecting to %s' % server_address.decode("utf-8")
try:
sock.connect(server_address)
except socket.error, msg:
print >>sys.stderr, msg
sys.exit(1)
After running the client I get the following error output:
connecting to hidden
[Errno 111] Connection refused
And for some extra information I am posting the relevant parts of the strace outputs of my working c++ client and non-working python client:
Working c++ client:
socket(PF_FILE, SOCK_STREAM, 0) = 3
connect(3, {sa_family=AF_FILE, path=#""}, 110) = 0
fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb77d7000
write(1, "Sent message is: 00014 www.googl"..., 38) = 38
write(3, "00014 www.google.com", 20) = 20
recv(3, "014 Search Engines", 99, 0) = 18
write(1, "014 Search Engines\n", 19) = 19
close(3) = 0
exit_group(0) = ?
None working python client:
socket(PF_FILE, SOCK_STREAM, 0) = 3
connect(3, {sa_family=AF_FILE, path=#"hidden"...}, 9) = -1 ECONNREFUSED (Connection refused)
write(2, "Traceback (most recent call last"..., 35) = 35
write(2, " File \"./uds.py\", line 13, in <"..., 40) = 40
open("./uds.py", O_RDONLY|O_LARGEFILE) = 4
fstat64(4, {st_mode=S_IFREG|0755, st_size=839, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7792000
read(4, "#! /opt/python/bin/python\nimport"..., 4096) = 839
write(2, " ", 4) = 4
write(2, "sock.connect('\\0hidden')\n", 25) = 25
close(4) = 0
munmap(0xb7792000, 4096) = 0
write(2, " File \"/opt/python/lib/python2."..., 64) = 64
open("/opt/python/lib/python2.7/socket.py", O_RDONLY|O_LARGEFILE) = 4
fstat64(4, {st_mode=S_IFREG|0755, st_size=20234, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7792000
read(4, "# Wrapper module for _socket, pr"..., 4096) = 4096
read(4, "oo long.\"\n errorTab[10064] = "..., 4096) = 4096
write(2, " ", 4) = 4
write(2, "return getattr(self._sock,name)("..., 39) = 39
close(4) = 0
munmap(0xb7792000, 4096) = 0
write(2, "socket", 6) = 6
write(2, ".", 1) = 1
write(2, "error", 5) = 5
write(2, ": ", 2) = 2
write(2, "[Errno 111] Connection refused", 30) = 30
write(2, "\n", 1) = 1
rt_sigaction(SIGINT, {SIG_DFL, [], 0}, {0x810fbe0, [], 0}, 8) = 0
close(3) = 0
exit_group(1) = ?
And also when I run my c++ client, I get this strace output from my server:
0, NULL) = 12
futex(0x80646a4, FUTEX_CMP_REQUEUE_PRIVATE, 1, 2147483647, 0x8064688, 360) = 10
futex(0x8064688, FUTEX_WAKE_PRIVATE, 1) = 1
accept(5,
But when I run my python client, no output is shown on strace. So it seems like I am trying to connect to a wrong address, but my address is defined as "\0hidden" in both my server and my client.

Your C++ doesn't do quite what you think it does. This line:
strncpy(addr.sun_path, UD_SOCKET_PATH, sizeof(addr.sun_path)-1);
Copies a single null character '\0' into addr.sun_path. Note this line in the manpage for strncpy():
If the length of src is less than n, strncpy() writes additional null
bytes to dest to ensure that a total of n bytes are written.
As a result your C++ actually connects to an abstract domain socket at "\0". Python does the right thing here and connects to an abstract domain socket at "\0hidden".

Why did I get a Segmentation Fault in Python when /tmp is mounted with noexec?

I am running a custom Python 2.7.3 application on CherryPy in Linux. When I used a service script in /etc/init.d/ to start or stop the service, I encountered a Segmentation Fault (SIGSEGV). Strangely, I did not receive a SIGSEGV if I ran the start or stop command manually from the shell, using "python /path/to/file.py --stop". The service script executes the same command.
After some debugging, by chance, I discovered that my /tmp was mounted with a "noexec" option. I removed the "noexec" option and the application was able to start and stop via the service scripts without any segmentation faults.
When I first encountered the issue, I ran strace and generated a core dump. Nothing from either tool gave me any indication that /tmp was the culprit. My question is this: How could I have used strace or gdb to help me identify that "noexec" on /tmp was causing the segmentation faults?
Here is some output from gdb when analyzing the core dump:
(gdb) bt full
#0 PyObject_Malloc (nbytes=4) at Objects/obmalloc.c:788
bp = 0x7f6b0fd1c6e800 \Address 0x7f6b0fd1c6e800 out of bounds\
pool = 0x7f6b0fd1c000
next = \value optimized out\
size = 0
#1 0x00007f6b0f7fd8e6 in _PyUnicode_New (length=1) at Objects/unicodeobject.c:345
new_size = 4
unicode = 0x3873480
#2 0x00007f6b0f7fdd4e in PyUnicodeUCS2_FromUnicode (u=0x38367cc, size=)
at Objects/unicodeobject.c:461
unicode = \value optimized out\
(There is a lot more output, this is just the first few lines)
Here is some output from strace on the failure:
3046 open("/usr/local/python2.7/lib/python2.7/site-packages/oauthlib/common.py", O_RDONLY) = 9
3046 fstat(9, {st_mode=S_IFREG|0644, st_size=13310, ...}) = 0
3046 open("/usr/local/python2.7/lib/python2.7/site-packages/oauthlib/common.pyc", O_RDONLY) = 10
3046 fstat(10, {st_mode=S_IFREG|0644, st_size=16043, ...}) = 0
3046 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fbc9ff9d000
3046 read(10, "\3\363\r\n}\321\322Tc\0\0\0\0\0\0\0\0\5\0\0\0##\2\0sd\2\0\0d\0"..., 4096) = 4096
3046 fstat(10, {st_mode=S_IFREG|0644, st_size=16043, ...}) = 0
3046 read(10, "\0\0\0C#\2\0s\330\0\0\0t\0\0|\0\0t\1\0\203\2\0s\36\0t\0\0|\0"..., 8192) = 8192
3046 read(10, "thon2.7/site-packages/oauthlib/c"..., 4096) = 3755
3046 read(10, "", 4096) = 0
3046 close(10) = 0
3046 munmap(0x7fbc9ff9d000, 4096) = 0
3046 --- SIGSEGV (Segmentation fault) # 0 (0) ---
After fixing the problem, here's a snippet from strace, from the same point where it tries to load oauthlib/common.pyc - notice that the only difference appears to be a brk() before munmap():
3416 open("/usr/local/python2.7/lib/python2.7/site-packages/oauthlib/common.pyc", O_RDONLY) = 10
3416 fstat(10, {st_mode=S_IFREG|0644, st_size=16043, ...}) = 0
3416 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5791f2c000
3416 read(10, "\3\363\r\n}\321\322Tc\0\0\0\0\0\0\0\0\5\0\0\0##\2\0sd\2\0\0d\0"..., 4096) = 4096
3416 fstat(10, {st_mode=S_IFREG|0644, st_size=16043, ...}) = 0
3416 read(10, "\0\0\0C#\2\0s\330\0\0\0t\0\0|\0\0t\1\0\203\2\0s\36\0t\0\0|\0"..., 8192) = 8192
3416 read(10, "thon2.7/site-packages/oauthlib/c"..., 4096) = 3755
3416 read(10, "", 4096) = 0
3416 brk(0x372f000) = 0x372f000
3416 close(10) = 0
3416 munmap(0x7f5791f2c000, 4096) = 0
3416 close(9) = 0
What information can help me point the blame at /tmp's mount options?

For those who would have the same problem as I had and find this page:
My CherryPy server.py ran fine on my Win10 system on python3.8 but failed with segmentation fault on my Linux system which had python3.6.1.
Switching to python3.8 on Linux solved my problem.

how can I release mongodb connections?

I have a mongo server with high read/write in a short time. I used python and pymongo, when I wake up this morning I found no connection can make to mongod master cause it's connections reached 19992, its a pretty much scary number
even I stopped all the program, the connection number seems no change
insert query update delete getmore command flushes mapped vsize res faults locked % idx miss % qr|qw ar|aw netIn netOut conn repl time
localhost:27417 0 0 0 0 2 1 0 624m 2.87g 287m 0 0 0 0|0 7|0 162b 1k 19992 M 10:36:16
> db.serverStatus(1)
{
"host" : "42yl:27417",
"version" : "1.8.1",
"process" : "mongod",
"uptime" : 71732,
"uptimeEstimate" : 71470,
"localTime" : ISODate("2011-05-26T03:02:48.301Z"),
"globalLock" : {
"totalTime" : 71732232290,
"lockTime" : 149471421,
"ratio" : 0.002083741384148133,
"currentQueue" : {
"total" : 0,
"readers" : 0,
"writers" : 0
},
"activeClients" : {
"total" : 7,
"readers" : 7,
"writers" : 0
}
},
"mem" : {
"bits" : 32,
"resident" : 258,
"virtual" : 910,
"supported" : true,
"mapped" : 624
},
"connections" : {
"current" : 19792,
"available" : 208
},
"extra_info" : {
"note" : "fields vary by platform",
"heap_usage_bytes" : 562688,
"page_faults" : 965
},
"indexCounters" : {
"btree" : {
"accesses" : 12789,
"hits" : 12789,
"misses" : 0,
"resets" : 0,
"missRatio" : 0
}
},
"backgroundFlushing" : {
"flushes" : 1195,
"total_ms" : 848633,
"average_ms" : 710.1531380753138,
"last_ms" : 101,
"last_finished" : ISODate("2011-05-26T03:02:18.691Z")
},
"cursors" : {
"totalOpen" : 7,
"clientCursors_size" : 7,
"timedOut" : 0
},
"network" : {
"bytesIn" : 685742402,
"bytesOut" : 2742190274,
"numRequests" : 3800041
},
"repl" : {
"ismaster" : true
},
"opcounters" : {
"insert" : 104225,
"query" : 9,
"update" : 925044,
"delete" : 45734,
"getmore" : 1642979,
"command" : 1119290
},
"asserts" : {
"regular" : 0,
"warning" : 56,
"msg" : 0,
"user" : 0,
"rollovers" : 0
},
"writeBacksQueued" : false,
"ok" : 1
}
I checked the socket connections
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:60000 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27424 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28417 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28418 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28419 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28420 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28421 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28422 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28423 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28424 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:38422 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:58422 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:5432 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27417 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27418 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27419 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:8443 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27420 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27421 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27422 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27423 0.0.0.0:* LISTEN
tcp 0 0 222.73.61.42:27420 222.73.61.43:38249 ESTABLISHED
tcp 0 0 127.0.0.1:27417 127.0.0.1:56699 ESTABLISHED
tcp 0 0 127.0.0.1:27417 127.0.0.1:56698 ESTABLISHED
tcp 0 0 127.0.0.1:27417 127.0.0.1:56697 ESTABLISHED
tcp 0 0 127.0.0.1:27417 127.0.0.1:56696 ESTABLISHED
tcp 0 0 127.0.0.1:27417 127.0.0.1:56702 ESTABLISHED
tcp 0 0 127.0.0.1:27417 127.0.0.1:56701 ESTABLISHED
tcp 0 0 127.0.0.1:27417 127.0.0.1:56700 ESTABLISHED
tcp 0 0 222.73.61.42:27422 222.73.61.43:33616 ESTABLISHED
tcp 0 0 222.73.61.42:27417 222.73.61.43:60218 ESTABLISHED
tcp 0 0 222.73.61.42:27423 222.73.61.43:33035 ESTABLISHED
tcp 0 3324 222.73.61.42:58422 119.85.195.88:54295 ESTABLISHED
tcp 0 0 222.73.61.42:27424 222.73.61.43:55825 ESTABLISHED
tcp 0 0 222.73.61.42:54279 222.215.136.8:80 ESTABLISHED
tcp 0 0 222.73.61.42:27418 222.73.61.43:37093 ESTABLISHED
tcp 0 0 222.73.61.42:27419 222.73.61.43:38346 ESTABLISHED
tcp 0 0 127.0.0.1:56702 127.0.0.1:27417 ESTABLISHED
tcp 0 0 127.0.0.1:56701 127.0.0.1:27417 ESTABLISHED
tcp 0 0 127.0.0.1:56700 127.0.0.1:27417 ESTABLISHED
tcp 0 0 127.0.0.1:56699 127.0.0.1:27417 ESTABLISHED
tcp 0 0 127.0.0.1:56698 127.0.0.1:27417 ESTABLISHED
tcp 0 0 127.0.0.1:56697 127.0.0.1:27417 ESTABLISHED
tcp 0 0 127.0.0.1:56696 127.0.0.1:27417 ESTABLISHED
tcp 0 0 222.73.61.42:27421 222.73.61.43:39843 ESTABLISHED
udp 0 0 0.0.0.0:48514 0.0.0.0:*
udp 0 0 222.73.61.42:50721 61.128.128.68:53 ESTABLISHED
udp 0 0 127.0.0.1:52274 127.0.0.1:52274 ESTABLISHED
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] STREAM LISTENING 9081684 /var/run/nscd/socket
unix 2 [ ACC ] STREAM LISTENING 18011686 /tmp/mongodb-27417.sock
unix 2 [ ACC ] STREAM LISTENING 18011689 /tmp/mongodb-27422.sock
unix 2 [ ACC ] STREAM LISTENING 18011691 /tmp/mongodb-28422.sock
unix 2 [ ACC ] STREAM LISTENING 18011697 /tmp/mongodb-27420.sock
unix 2 [ ACC ] STREAM LISTENING 18011702 /tmp/mongodb-28417.sock
unix 2 [ ACC ] STREAM LISTENING 18011693 /tmp/mongodb-27421.sock
unix 2 [ ACC ] STREAM LISTENING 18011695 /tmp/mongodb-28421.sock
unix 2 [ ACC ] STREAM LISTENING 18011699 /tmp/mongodb-28420.sock
unix 2 [ ACC ] STREAM LISTENING 18011710 /tmp/mongodb-27419.sock
unix 2 [ ACC ] STREAM LISTENING 18011713 /tmp/mongodb-28419.sock
unix 2 [ ACC ] STREAM LISTENING 18011716 /tmp/mongodb-27418.sock
unix 2 [ ACC ] STREAM LISTENING 18011719 /tmp/mongodb-28418.sock
unix 2 [ ACC ] STREAM LISTENING 18011722 /tmp/mongodb-27424.sock
unix 2 [ ACC ] STREAM LISTENING 18011725 /tmp/mongodb-28424.sock
unix 2 [ ACC ] STREAM LISTENING 18011728 /tmp/mongodb-27423.sock
unix 2 [ ACC ] STREAM LISTENING 18011731 /tmp/mongodb-28423.sock
unix 2 [ ACC ] STREAM LISTENING 12771288 /tmp/.s.PGSQL.5432
unix 2 [ ] DGRAM 3651 #/org/kernel/udev/udevd
unix 5 [ ] DGRAM 16472048 /dev/log
unix 2 [ ] STREAM CONNECTED 18706425 /var/run/nscd/socket
unix 2 [ ] DGRAM 16792651
unix 2 [ ] DGRAM 16472057
unix 2 [ ] DGRAM 16472052

We have come across same problem before, I think there is a problem with linux system tuning for TCP_KEEPALIVE_TIME, which specify a timeout for a given tcp connection. For your case, you have a high read/write in a short time, which can make a even lower timeout config for tcp connection.
By using following command, it may help you:
Checking current config:
[root#monitor-hk-1 ~]# cat /proc/sys/net/ipv4/tcp_keepalive_time
7200
Changing config:
echo 300 > /proc/sys/net/ipv4/tcp_keepalive_time
Below are some internal memo by my company:
After some researching for 30,Jul's failure of mongo on testbed, ip 118.26, I found something familiar with production:
1. Total Connections reach 970+ which mongo start to block all the incoming connections.
2. Check netstat, only nearly 100 something connections are kept listening or waiting.
3. Check iostat, and cpu, memory, not a high utilization rate, several percent around 10.
Log:
Tue Jul 30 10:19:03.575 [initandlisten] connection accepted from 192.168.118.18:52858 #261673 (974 connections now open)
Tue Jul 30 10:19:03.575 [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable
Tue Jul 30 10:19:03.575 [initandlisten] can't create new thread, closing connection
After checking official manual, I found that our tcp keep-alive value may be too high:
[root#monitor-hk-1 ~]# cat /proc/sys/net/ipv4/tcp_keepalive_time
7200
I suggest to change the such value to 300 for a short keep-alive for tcp connections.
Can be done with following command:
echo 300 > /proc/sys/net/ipv4/tcp_keepalive_time
Notes: This can only change this value temporarily. If we reboot the system, it would be reset to default value. If we want to make persistent change, please reference at the Linux operation guide.
Hope this can help you. More Info:
Linux TCP I/O Tuning:
MongoDB Official Doc for TCP Tuning

i googled a bit for you and found on mongodb official site this: mongodb-docs
try using:
waitQueueTimeoutMS=ms
"The amount of time a thread can wait for a connection to become available before timing out. This only applies if the connection pool has reached the maximum size and all connections are already in use."
or:
waitQueueMultiple=n
"The drivers impose a limit on how many threads can be waiting for a connection at the same time. This limit is expressed as a multiple of maxPoolSize."

If you navigate to /tmp/ , you should see 'mongodb-.sock'. you can remove that. Also, there might be a mongod.lock in the dbpath. Removing that will free it up.

USE:
Service mongodb stop
else, kill mongodb connections