web application firewall development - python

I have an assignment to develop a web application firewall. I have been researching for some source codes about that.My main source was ModSecurity.
Main question is that:
-Which framework or programming language I can use, to develop a web application firewall? Which one would be the most useful?
-Can I use Django & Python?
It would be a starting point for the project research.

OK, so my guess was basically correct, although I thought it was protecting an app with no or bad security, but it's more about protecting against attacks. In that Case, Django is definitely wrong. It's clearly doable in Python, but don't expect to be able to handle 100.000 requests per second. :) But if it's research and development, I think Python can be a great choice, as it's fast to develop in, and with tools like Cython it can be quite fast to run as well. Should you then end up making a finished product that does need extreme performance, you can take the algorithms and translate them to C/C++.
I'd look into Twisted in your case. That may be the correct solution.
"It will be used on the server side to control the user transactions via HTTP."
Most web application frameworks have security settings. These are usually not called "firewalls", and you didn't answer my questions, so I'm going to guess here:
You are writing a web proxy that will act to filter out requests which do not have the correct permission because there is an application which does not have any access control at all. Is this correct?
Yes, you can do that in Python. Django is probably not the correct solution. If you need to implement access control with login pages and user management, then you probably want SQL and templating and a lightweight Python framework could be helpful. Otherwise Twisted or just doing it with the functionality in standard lib might be the correct solution.

this is what you need to do, for start use Linux, much easier to handle then Windows, you have 2 options, the first is to implement NetFilter driver by hooking to the system calls (More Complex!) , the second is to use libnetfilter_queue library and iptables to transfer the packet to a user space application, the main idea is to do deep analysis to the payload beside checking the IP and TCP header, very similar IDS,IPS systems but it's focusing on web application security holes.
i don't think you can do this with python without deeper interfering, what you asking here is pretty tricky, it's involved the lowers levels of the system...
you can start analyze the data using this example:
configure the iptables
#: iptables -A INPUT -j NFQUEUE --queue-balance 0:3
#: iptables -A OUTPUT -j NFQUEUE --queue-balance 4:8
queues 0 to 3 is for all inputs (better to divided to different queue, there is limit for packets the queue can hold),the other are for all outputs
writing application in C (the iptables transfer the packet from the kernel to user space)
filterQueue.c
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <netinet/in.h>
#include <linux/types.h>
#include <string.h>
/* for ethernet header */
#include<net/ethernet.h>
/* for UDP header */
#include<linux/udp.h>
/* for TCP header */
#include<linux/tcp.h>
/* for IP header */
#include<linux/ip.h>
/* -20 (maximum priority) */
#include <sys/time.h>
#include <sys/resource.h>
/* for NF_ACCEPT */
#include <linux/netfilter.h>
/* for Threads */
#include <pthread.h>
/* for Queue */
#include <libnetfilter_queue/libnetfilter_queue.h>
#define NUM_THREADS 15
pthread_t threads[NUM_THREADS];
void printTCP(unsigned char *buffer) {
unsigned short iphdrlen;
struct iphdr *iph = (struct iphdr *) (buffer + sizeof(struct ethhdr));
iphdrlen = iph->ihl * 4;
struct tcphdr *tcph = (struct tcphdr *) (buffer + iphdrlen
+ sizeof(struct ethhdr));
int header_size = sizeof(struct ethhdr) + iphdrlen + tcph->doff * 4;
printf("| Packet Type: TCP \n");
printf("|-Source Port : %u\n", ntohs(tcph->source));
printf("|-Destination Port : %u\n", ntohs(tcph->dest));
printf("|-Sequence Number : %u\n", ntohl(tcph->seq));
printf("|-Acknowledge Number : %u\n", ntohl(tcph->ack_seq));
printf("|-Header Length : %d DWORDS or %d BYTES\n",
(unsigned int) tcph->doff, (unsigned int) tcph->doff * 4);
printf("|-CWR Flag : %d\n", (unsigned int) tcph->cwr);
printf("|-ECN Flag : %d\n", (unsigned int) tcph->ece);
printf("|-Urgent Flag : %d\n", (unsigned int) tcph->urg);
printf("|-Acknowledgement Flag : %d\n", (unsigned int) tcph->ack);
printf("|-Push Flag : %d\n", (unsigned int) tcph->psh);
printf("|-Reset Flag : %d\n", (unsigned int) tcph->rst);
printf("|-Synchronise Flag : %d\n", (unsigned int) tcph->syn);
printf("|-Finish Flag : %d\n", (unsigned int) tcph->fin);
printf("|-Window : %d\n", ntohs(tcph->window));
printf("|-Checksum : %d\n", ntohs(tcph->check));
printf("|-Urgent Pointer : %d\n", tcph->urg_ptr);
}
void printUDP(unsigned char *buffer) {
unsigned short iphdrlen;
struct iphdr *iph = (struct iphdr *) (buffer + sizeof(struct ethhdr));
iphdrlen = iph->ihl * 4;
struct udphdr *udph = (struct udphdr*) (buffer + iphdrlen
+ sizeof(struct ethhdr));
int header_size = sizeof(struct ethhdr) + iphdrlen + sizeof udph;
printf("| Packet Type: UDP \n");
printf("|-Source Port : %u\n", ntohs(udph->source));
printf("|-Destination Port : %u\n", ntohs(udph->dest));
printf("|-UDP Length : %u\n", ntohs(udph->len));
printf("|-UDP Checksum : %u\n", ntohs(udph->check));
}
char * getText(unsigned char * data, char Size) {
char * text = malloc(Size);
int i = 0;
for (i = 0; i < Size; i++) {
if (data[i] >= 32 && data[i] <= 128)
text[i] = (unsigned char) data[i];
else
text[i] = '.';
}
return text;
}
u_int32_t analyzePacket(struct nfq_data *tb, int *blockFlag) {
//packet id in the queue
int id = 0;
//the queue header
struct nfqnl_msg_packet_hdr *ph;
//the packet
char *data;
//packet size
int ret;
//extracting the queue header
ph = nfq_get_msg_packet_hdr(tb);
//getting the id of the packet in the queue
if (ph)
id = ntohl(ph->packet_id);
//getting the length and the payload of the packet
ret = nfq_get_payload(tb, &data);
if (ret >= 0) {
printf("Packet Received: %d \n", ret);
/* extracting the ipheader from packet */
struct sockaddr_in source, dest;
unsigned short iphdrlen;
struct iphdr *iph = ((struct iphdr *) data);
iphdrlen = iph->ihl * 4;
memset(&source, 0, sizeof(source));
source.sin_addr.s_addr = iph->saddr;
memset(&dest, 0, sizeof(dest));
dest.sin_addr.s_addr = iph->daddr;
printf("|-Source IP: %s\n", inet_ntoa(source.sin_addr));
printf("|-Destination IP: %s\n", inet_ntoa(dest.sin_addr));
printf("|-Checking for Protocol: \n");
if (iph->protocol == 6) {
printTCP(data);
} else if (iph->protocol == 17) {
printUDP(data);
}
printf("|-Extracting Payload: \n");
char * text = getText(data, ret);
//filtering requests for facebook
if (text && text[0] != '\0') {
printf("\n %s \n", text);
ret = strstr(text, "facebook");
if (ret == 0)
//not found in string
*blockFlag = 0;
else
//found in string
*blockFlag = 1;
}
//release the packet
free(text);
}
//return the queue id
return id;
}
int packetHandler(struct nfq_q_handle *qh, struct nfgenmsg *nfmsg, struct nfq_data *nfa,
void *data) {
printf("entering callback \n");
//when to drop
int blockFlag = 0;
//analyze the packet and return the packet id in the queue
u_int32_t id = analyzePacket(nfa, &blockFlag);
//this is the point where we decide the destiny of the packet
if (blockFlag == 0)
return nfq_set_verdict(qh, id, NF_ACCEPT, 0, NULL);
else
return nfq_set_verdict(qh, id, NF_DROP, 0, NULL);
}
void *QueueThread(void *threadid) {
//thread id
long tid;
tid = (long) threadid;
struct nfq_handle *h;
struct nfq_q_handle *qh;
char buf[128000] __attribute__ ((aligned));
//pointers and descriptors
int fd;
int rv;
int ql;
printf("open handle to the netfilter_queue - > Thread: %d \n", tid);
h = nfq_open();
if (!h) {
fprintf(stderr, "cannot open nfq_open()\n");
return NULL;
}
//unbinding previous procfs
if (nfq_unbind_pf(h, AF_INET) < 0) {
fprintf(stderr, "error during nfq_unbind_pf()\n");
return NULL;
}
//binding the netlink procfs
if (nfq_bind_pf(h, AF_INET) < 0) {
fprintf(stderr, "error during nfq_bind_pf()\n");
return NULL;
}
//connet the thread for specific socket
printf("binding this socket to queue '%d'\n", tid);
qh = nfq_create_queue(h, tid, &packetHandler, NULL);
if (!qh) {
fprintf(stderr, "error during nfq_create_queue()\n");
return NULL;
}
//set queue length before start dropping packages
ql = nfq_set_queue_maxlen(qh, 100000);
//set the queue for copy mode
if (nfq_set_mode(qh, NFQNL_COPY_PACKET, 0xffff) < 0) {
fprintf(stderr, "can't set packet_copy mode\n");
return NULL;
}
//getting the file descriptor
fd = nfq_fd(h);
while ((rv = recv(fd, buf, sizeof(buf), 0)) && rv >= 0) {
printf("pkt received in Thread: %d \n", tid);
nfq_handle_packet(h, buf, rv);
}
printf("unbinding from queue Thread: %d \n", tid);
nfq_destroy_queue(qh);
printf("closing library handle\n");
nfq_close(h);
return NULL;
}
int main(int argc, char *argv[]) {
//set process priority
setpriority(PRIO_PROCESS, 0, -20);
int rc;
long balancerSocket;
for (balancerSocket = 0; balancerSocket < NUM_THREADS; balancerSocket++) {
printf("In main: creating thread %ld\n", balancerSocket);
//send the balancer socket for the queue
rc = pthread_create(&threads[balancerSocket], NULL, QueueThread,
(void *) balancerSocket);
if (rc) {
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
while (1) {
sleep(10);
}
//destroy all threads
pthread_exit(NULL);
}

Unless this is just some kind of academic exercise and Python helps you get it done fast, I don't think a high level language like Python is the best choice for a firewall (I don't even know if it's possible honestly). If you're planning some sort of proxy/filter application, that might be fine, but Django isn't needed either way.

Django is a web application framework. I don't see anyone writing a firewall implementation using it.

C, C++, Golang, Lua are all optional languages to develop a Web Application Firewall or Gateway, but django is not suitable for it.
C, C++ can develop nginx plugin or backend WAF.
Golang can develop gateway with WAF, such as Janusec Application Gateway.
Lua can extend nginx access control and work as a WAF.

Related

C socket recv function returning undefined error (errno)

I am trying to make a file transfer program. I am currently programming the C server and using a short Python script as the client. When I try sending the filename from Python to C, the recv function (in C) returns an undefined error. It has received half of the filename once. Here is the code:
C server (problematic recv function is pointed out):
#define BUFFER 2000000
#define HEADERSIZE 20
#define PORT 9342
#define FGET "FGET"
#define UPLD "UPLD"
typedef struct timeval timeval_t;
int i;
fd_set readlist, sockets;
void* sendf(void* sock){
printf("File Descriptor: %d\n", *(int*)sock);
fflush(stdout);
char* filename = (char*) malloc(21*sizeof(char));
ssize_t result = recv(*(int*)sock, filename, 20+1, 0);
if (result != 0){
printf("%s", strerror(errno));
exit(0);
}
printf("Filename: ");
puts(filename);
FILE* sfile = fopen(filename, "rb");
free(filename);
fseek(sfile, 0, SEEK_END);
unsigned long long filesize = ftell(sfile);
fseek(sfile, 0, SEEK_SET);
char* content = malloc(BUFFER);
for(i=0; i<=ceil(filesize/BUFFER); ++i){
if (filesize >= BUFFER){
fread(content, BUFFER, 1, sfile);
filesize-=BUFFER;
fseek(sfile, filesize-1, SEEK_END);
} else{
fread(content, filesize, 1, sfile);
}
content = realloc(content, BUFFER);
}
fclose(sfile);
char* header = malloc(HEADERSIZE);
sprintf(header, "%20llu", filesize);
send(*(int*)sock, header, 20, 0);
send(*(int*)sock, content, filesize, 0);
free(filename);
free(content);
free(header);
return NULL;
}
void* receivef(void* sock){
printf("Function not written yet");
return NULL;
}
int main(){
chdir("/Users/reimeytal/Desktop/Fileserver");
int sock = socket(AF_INET, SOCK_STREAM, 0);
pthread_t thread;
timeval_t timeout;
timeout.tv_sec = 1;
timeout.tv_usec = 0;
struct sockaddr_in server;
server.sin_port = htons(PORT);
server.sin_addr.s_addr = INADDR_ANY;
server.sin_family = AF_INET;
FD_ZERO(&sockets);
FD_SET(sock, &sockets);
bind(sock, (struct sockaddr*)&server, sizeof(server));
listen(sock, 5);
int maxsock = sock;
int clientsock;
char* msghdr = (char*) malloc(5*sizeof(char));
while (1){
readlist = sockets;
select(FD_SETSIZE, &readlist, NULL, NULL, &timeout);
for(i=0; i<=maxsock; i++){
if(FD_ISSET(i, &readlist)){
if(i==sock){
clientsock = accept(sock, NULL, NULL);
FD_SET(clientsock, &sockets);
if(clientsock>sock){
maxsock = clientsock;
}
} else{
printf("Incoming message from %d\n", i);
fflush(stdout);
recv(i, msghdr, 5, 0);
clientsock = i;
if(strcmp(msghdr, FGET) == 0){
pthread_create(&thread, NULL, sendf, (int *)&clientsock);
}/*else if(strcmp(msghdr, UPLD) == 0){
pthread_create(&thread, NULL, receivef, (int*)&i);
}else{
puts("ERROR: Received invalid message header");
}*/
msghdr = (char *) realloc(msghdr, 5*sizeof(char));
}
}
}
}
return 0;
Result when running:
Incoming message from 4
File Descriptor: 4
Incoming message from 4
Incoming message from 4
Undefined error: 0
Python client:
import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect(("localhost", 9342))
sock.send(b'FGET\0')
sock.send(b'abcdeftghijklmno.txt\0')
If anyone needs my to clarify/add something, I'd be happy to do so. Thanks
There's a mistake here:
ssize_t result = recv(*(int*)sock, filename, 20+1, 0);
if (result != 0){
printf("%s", strerror(errno));
exit(0);
}
recv returns the number of bytes written, so you should check if (result <= 0).
You are getting a result code 0 because the Python app closed the connection after sending the data. Apart from that all should work.
EDIT: Removed part about timing issue that was incorrect.

send packets to ipv6 link local multicast addr

I write some sample code in different languages to send packets to ipv6 link local multicast addr(ff02::fb), but it doesn't always work.
the python version:
#!/usr/bin/env python
MYPORT = 5353
MYGROUP_6 = 'ff02::fb'
MYTTL = 1 # Increase to reach other networks
import time
import struct
import socket
import sys
def main():
sender(MYGROUP_6)
def sender(group):
addrinfo = socket.getaddrinfo(group, None)[0]
s = socket.socket(addrinfo[0], socket.SOCK_DGRAM)
# Set Time-to-live (optional)
ttl_bin = struct.pack('#i', MYTTL)
s.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_MULTICAST_HOPS, ttl_bin)
while True:
data = repr(time.time())
s.sendto(data + '\0', (addrinfo[4][0], MYPORT))
time.sleep(1)
if __name__ == '__main__':
main()
this one works on linux(3.13.0-24-generic), but get error "no route to host" on OS X(10.10 Yosemite).
Then I write a simple Go program to try again:
package main
import (
"fmt"
"net"
)
func main() {
addr, err := net.ResolveUDPAddr("udp6", "[ff02::fb]:5353")
if err != nil {
fmt.Printf("ResolveUDPAddr err: %v\n", err)
return
}
if conn, err := net.DialUDP("udp6", nil, addr); err == nil {
if _, err = conn.Write([]byte("hello")); err != nil {
fmt.Printf("Write failed, %v\n", err)
}
} else {
fmt.Printf("DialUDP err: %v\n", err)
}
return
}
this one gets error "connect: invalid argument" on linux, and the same error "no route to host" with sample above on OS X. Finally, I try the low-level C language:
/*
* Examples:
* >sender 224.0.22.1 9210 6000 1000
* >sender ff15::1 2001 65000 1
*/
#include <stdio.h>
#include <stdlib.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <sys/un.h>
#include <netinet/tcp.h>
#include <arpa/inet.h>
#include <netdb.h>
#include <unistd.h> /* for usleep() */
#define SOCKET int
int mcast_send_socket(char* multicastIP, char* multicastPort, int multicastTTL, struct addrinfo **multicastAddr) {
SOCKET sock;
struct addrinfo hints = { 0 }; /* Hints for name lookup */
/*
Resolve destination address for multicast datagrams
*/
hints.ai_family = PF_UNSPEC;
hints.ai_socktype = SOCK_DGRAM;
hints.ai_flags = AI_NUMERICHOST;
int status;
if ((status = getaddrinfo(multicastIP, multicastPort, &hints, multicastAddr)) != 0 )
{
fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(status));
return -1;
}
/*
Create socket for sending multicast datagrams
*/
if ( (sock = socket((*multicastAddr)->ai_family, (*multicastAddr)->ai_socktype, 0)) < 0 ) {
perror("socket() failed");
freeaddrinfo(*multicastAddr);
return -1;
}
/*
Set TTL of multicast packet
*/
if ( setsockopt(sock,
(*multicastAddr)->ai_family == PF_INET6 ? IPPROTO_IPV6 : IPPROTO_IP,
(*multicastAddr)->ai_family == PF_INET6 ? IPV6_MULTICAST_HOPS : IP_MULTICAST_TTL,
(char*) &multicastTTL, sizeof(multicastTTL)) != 0 ) {
perror("setsockopt() failed");
freeaddrinfo(*multicastAddr);
return -1;
}
/*
set the sending interface
*/
if((*multicastAddr)->ai_family == PF_INET) {
in_addr_t iface = INADDR_ANY; /* well, yeah, any */
if(setsockopt (sock,
IPPROTO_IP,
IP_MULTICAST_IF,
(char*)&iface, sizeof(iface)) != 0) {
perror("interface setsockopt() sending interface");
freeaddrinfo(*multicastAddr);
return -1;
}
}
if((*multicastAddr)->ai_family == PF_INET6) {
unsigned int ifindex = 0; /* 0 means 'default interface'*/
if(setsockopt (sock,
IPPROTO_IPV6,
IPV6_MULTICAST_IF,
(char*)&ifindex, sizeof(ifindex)) != 0) {
perror("interface setsockopt() sending interface");
freeaddrinfo(*multicastAddr);
return -1;
}
}
return sock;
}
static void DieWithError(char* errorMessage)
{
fprintf(stderr, "%s\n", errorMessage);
exit(EXIT_FAILURE);
}
int main(int argc, char *argv[])
{
SOCKET sock;
struct addrinfo *multicastAddr;
char* multicastIP; /* Arg: IP Multicast address */
char* multicastPort; /* Arg: Server port */
char* sendString; /* Arg: String to multicast */
int sendStringLen; /* Length of string to multicast */
int multicastTTL; /* Arg: TTL of multicast packets */
int defer_ms; /* miliseconds to defer in between sending */
int i;
if ( argc < 5 || argc > 6 )
{
fprintf(stderr, "Usage: %s <Multicast Address> <Port> <packetsize> <defer_ms> [<TTL>]\n", argv[0]);
exit(EXIT_FAILURE);
}
multicastIP = argv[1]; /* First arg: multicast IP address */
multicastPort = argv[2]; /* Second arg: multicast port */
sendStringLen = atoi(argv[3]);
defer_ms = atoi(argv[4]);
/* just fill this with some byte */
sendString = calloc(sendStringLen, sizeof(char));
for(i = 0; i<sendStringLen; ++i)
sendString[i]= 's';
multicastTTL = (argc == 6 ? /* Fourth arg: If supplied, use command-line */
atoi(argv[5]) : 1); /* specified TTL, else use default TTL of 1 */
sock = mcast_send_socket(multicastIP, multicastPort, multicastTTL, &multicastAddr);
if(sock == -1 )
DieWithError("mcast_send_socket() failed");
int nr=0;
for (;;) /* Run forever */
{
int* p_nr = (int*)sendString;
*p_nr = htonl(nr);
if ( sendto(sock, sendString, sendStringLen, 0,
multicastAddr->ai_addr, multicastAddr->ai_addrlen) != sendStringLen )
DieWithError("sendto() sent a different number of bytes than expected");
fprintf(stderr, "packet %d sent\n", nr);
nr++;
usleep(defer_ms*1000);
}
/* NOT REACHED */
return 0;
}
this C version doesn't work on OS X either, it complains "Can't assign requested address" on setsockopt(sock, IPPROTO_IPV6, IPV6_MULTICAST_IF, (char*)&ifindex, sizeof(ifindex)), seems OS X doesn't support to choose interface automatically.
EDIT: Adding a scope id(en0) makes the go&C version work on OS X, but python(2.7.6) version still complains: "No route to host".

urllib2.urlopen(url).read() timed out in virtual machine Os x 10.11(Vmware Workstation 12 pro)

I have a http resource whose size is 3GB.
I have some codes like below.
#the url is actually a http resource which is 3GB.
res = urllib2.urlopen(url, timeout = 10)
data = res.read(1024)
while data:
data = res.read(1024)
In Vmware workstation 11 or below, it works fine.But in Vmware workstation 12, it gives me the error.
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 384, in read
data = self._sock.recv(left)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 612, in read
s = self.fp.read(amt)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 384, in read
data = self._sock.recv(left)
socket.timeout: timed out
I use safari to download the resource in Vmware workstation 12, it works fine. And
if the resource is less than some size such as 10K, it also works fine.
They fixed it in VMware Fusion 8.5.7! See https://communities.vmware.com/thread/544049
I can't really provide you an answer right now and what I have to say is a bit longer than a comment, but I'm experiencing a similar issue in VMWare Fusion Pro 8.5 on 10.12 with Python's urllib2. It has nothing to do with urllib2.
I started receiving this issue randomly during transfer sessions and, after some Wireshark debugging, determined that it was due to the TCP window reaching 0 on the receiver. For some reason, it never updates again.
If you don't know what a TCP window is, it's basically the size of the receive buffer on one end of a TCP connection. That buffer should expand and contract as a congestion control mechanism during normal transfer, but what shouldn't happen is getting the window stuck at 0.
The reason your sessions work for transfers less than 10k is because the default TCP window is usually about 8k. Anything less than that are you won't even fill up the receiving buffer. Anymore more and you're basically hoping you process the data faster than you receive it.
To reproduce the issue on my local machine, here are two [terribly] written C programs you can compile with cc client.c -o client and cc server.c -o server. Run the client in the VM and the server on your local machine.
server.c:
/* server.c */
/* A simple server in the internet domain using TCP
The port number is passed as an argument */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
void error(const char *msg)
{
perror(msg);
exit(1);
}
int main(int argc, char *argv[])
{
int sockfd, newsockfd, portno;
socklen_t clilen;
char buffer[1024];
struct sockaddr_in serv_addr, cli_addr;
int n, total;
if (argc < 2) {
fprintf(stderr,"ERROR, no port provided\n");
exit(1);
}
sockfd = socket(AF_INET, SOCK_STREAM, 0);
if (sockfd < 0)
error("ERROR opening socket");
bzero((char *) &serv_addr, sizeof(serv_addr));
portno = atoi(argv[1]);
serv_addr.sin_family = AF_INET;
serv_addr.sin_addr.s_addr = INADDR_ANY;
serv_addr.sin_port = htons(portno);
if (bind(sockfd, (struct sockaddr *) &serv_addr,
sizeof(serv_addr)) < 0)
error("ERROR on binding");
listen(sockfd,5);
clilen = sizeof(cli_addr);
newsockfd = accept(sockfd,
(struct sockaddr *) &cli_addr,
&clilen);
if (newsockfd < 0)
error("ERROR on accept");
memset(buffer, '0xAB', sizeof(buffer));
total = 0;
for (;;) {
n = write(newsockfd, buffer, sizeof(buffer));
if (n < 0)
error("ERROR writing to socket");
else
total += n;
printf("wrote %d / %d\n", n, total);
}
close(newsockfd);
close(sockfd);
return 0;
}
client.c:
/* client.c */
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netdb.h>
void error(const char *msg)
{
perror(msg);
exit(0);
}
int main(int argc, char *argv[])
{
fd_set set;
int sockfd, portno, n, total, rv;
struct sockaddr_in serv_addr;
struct hostent *server;
struct timeval timeout;
char buffer[256];
if (argc < 3) {
fprintf(stderr,"usage %s hostname port\n", argv[0]);
exit(0);
}
portno = atoi(argv[2]);
sockfd = socket(AF_INET, SOCK_STREAM, 0);
if (sockfd < 0)
error("ERROR opening socket");
server = gethostbyname(argv[1]);
if (server == NULL) {
fprintf(stderr,"ERROR, no such host\n");
exit(0);
}
bzero((char *) &serv_addr, sizeof(serv_addr));
serv_addr.sin_family = AF_INET;
bcopy((char *)server->h_addr,
(char *)&serv_addr.sin_addr.s_addr,
server->h_length);
serv_addr.sin_port = htons(portno);
if (connect(sockfd,(struct sockaddr *) &serv_addr,sizeof(serv_addr)) < 0)
error("ERROR connecting");
bzero(buffer, 256);
FD_ZERO(&set);
FD_SET(sockfd, &set);
sleep(1);
timeout.tv_sec = 1;
timeout.tv_usec = 0;
total = 0;
for (;;) {
rv = select(sockfd + 1, &set, NULL, NULL, &timeout);
if (rv == -1) {
perror("select\n");
} else if(rv == 0) {
printf("timeout\n");
break;
} else {
n = read(sockfd, buffer, 256);
if (n < 0)
error("ERROR reading from socket");
total += n;
printf("read %d / %d\n", n, total);
}
}
close(sockfd);
return 0;
}
These programs are both taken directly from http://www.linuxhowtos.org/C_C++/socket.htm with modification to report additional stats and to force a stall out.
Here is a screenshot from Wireshark demonstrating the TCP Window reducing to 0 and sticking:
My current theory is that there is some kind of bug in the network stack on the VMWare side on the client, but it's difficult to tell. So far I've tried using three different virtual network interfaces (e1000, e1000e, vlance) and still had the same issue with each of them.
I'm going to attempt to try various vmx options to reduce the likelihood that the issue occurs, but this is obviously a killer for a stable system and my use case (Virtualized Jenkins slaves for CI) simply won't allow this kind of bug.
I'll report back if I'm able to learn anything new.
EDIT: I posted a bug in the VMWare Community board: https://communities.vmware.com/message/2648727
EDIT again: They fixed it in VMware Fusion 8.5.7! See the same link as above.

Scan for surrounding WiFi devices and store MAC addresses in database in Python

The ultimate goal here is to be able to use a Raspberry Pi with a wireless adapter set to monitor mode, to scan all surrounding WiFi devices (even if they only have WiFi turned on and not associated), and keep a log of MAC addresses and timestamps in a local database (e.g. MySQL).
I have been using Python (specifically subprocess) and airodump-ng (and airmon-ng) to try to accomplish this, I am wondering if there are any better ideas or tools which can help me achieve this, even another language, C or Java or some other tool.
Well i hope you know about wireshark tool ,but what you want is the details about your neigboring access points and save it into a file. So lets start,
first you need to capture the packets floating around and filter it, right? you can do it by with help of pcap.
When you capture the packets, the radiotap header will have all the information you need, with some google search you learn more about it or go through each packets in wireshark.The packets are like different layers of wrapping. i can give you a sample code of unwrapping a packet..
static int count = 1; /* packet counter */
/* declare pointers to packet headers */
const struct ieee80211_hdr *ethernet; /* The ethernet header [1] */
const struct ieee80211_radiotap_header *rt;
uint8_t *llc;
unsigned int radioTapSize ;
int ieee80211Size = 24;
rt = (struct ieee80211_radiotap_header*)(packet);
//Hex to Dec
std::stringstream ss;
ss << std::hex << rt->it_len;
ss >> radioTapSize;
//std::cout << radioTapSize << std::endl;
ethernet = (struct ieee80211_hdr*)(packet+radioTapSize);
llc = (uint8_t *)(packet+radioTapSize+ieee80211Size);
// LLC Value = aa:aa:03:00:00:00:08:00
//printf("%x\n",llc[0]);
if(llc[0] == 0xaa && llc[1] == 0xaa && llc[2] == 0x03 && llc[3] == 0x00 && llc[4] == 0x00
&& llc[5] == 0x00 && llc[6] == 0x08 && llc[7] == 0x00)
{
printf("\nPacket number %d:\n", count);
count++;
//radioTapSize = rt->it_len;
printf("Radiotap Length : %x\n",radioTapSize);
std::cout << "Radiotap Length : " << radioTapSize << std::endl;
printf("Radiotap pad : %x\n",rt->it_pad);
printf("Radiotap present : %x\n",rt->it_present);
printf("Radiotap version : %x\n",rt->it_version);
int i=0;
printf("Farme Control : %x\n",htons(ethernet->frame_control));
printf("Duration ID : %x\n",htons(ethernet->duration_id));
printf("Address 1: ");
for (i = 0; i < 6; ++i)
{
if(i != 5)
printf(" %x:",ethernet->addr1[i]);
else
printf(" %x\n",ethernet->addr1[i]);
}
printf("Address 2: ");
for (i = 0; i < 6; ++i)
{
if(i != 5)
printf(" %x:",ethernet->addr2[i]);
else
printf(" %x\n",ethernet->addr2[i]);
}
printf("Address 3: ");
for (i = 0; i < 6; ++i)
{
if(i != 5)
printf(" %x:",ethernet->addr3[i]);
else
printf(" %x\n",ethernet->addr3[i]);
}
printf("LLC: ");
for (i = 0; i < 8; ++i)
{
if(i != 7)
printf(" %x:",llc[i]);
else
printf(" %x\n",llc[i]);
}
}
look, this is just an example how a packet is unwrapped.
go through pcap stuff and you can easily filter all the informations from a packet sent by your neigbors. i can give you links to some useful things
to know more about sending and receiving packets.
I hope you understand it else feel free to ask.

Python 3 socket client sending data and C++ socket server receiving data with offset?

I have made a small test to send an integer from a python application (client socket) to a c++ application (socket server) both are TCP stream sockets. I have also tested sending the same integer from the client socket in python to a server socket in python.
What I do is I send an array of 4 bytes (char in c++), each char has the integer shifted right (>>) like this (python syntax):
...
[my_int >> i & 0xff for i in (24,16,8,0)]:
...
The problem is that when sending from the client socket in python to the server socket in python the data "arrives ok", for instance if I send the integer 1390248303
the python server socket first prints the bytes stream received then for each byte it prints its ascii code and then I do:
sum([l[3-i] << 8*i for i in (3,2,1,0)])
to "rebuild" the integer and this is the result (which is ok):
RECEIVED: b'R\xdd\x81o'
82
221
129
111
RECEIVED: 1390248303
BUT the C++ server socket, in which I do the same but with more verbose in the code:
...
int sum = 0;
int term = 0;
for(int i = 3;i > -1;i--)
{
printf("%d\n",msg[3-i]);
term = msg[3-i];
//if (term < 0)
// term = 256 + term;
suma += term << 8*i;
};
printf("Received: %d\n",sum);
...
Outputs
82
-35
-127
111
Received: 1373405551
Did you see that the 2 bytes in the middle are different to the 2 bytes in the middle corresponding to what the server socket in python outputs? Not only that but if I add 256 to them they became the same:
-35 + 256 = 221
-127 + 256 = 129
What is the reason of this behaviour? Thanks beforehand for any hint!
Here is the code of the applications:
python client socket:
import socket
client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client_socket.connect(("localhost", 7000))
my_int = 1390248303
my_bytes = bytearray()
for e in [my_int >> i & 0xff for i in (24,16,8,0)]:
my_bytes.append(e)
print("To be sent:", my_bytes)
client_socket.send(my_bytes)
print("Sent:", my_bytes)
client_socket.close()
python server socket:
import socket
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_socket.bind(("", 7000))
server_socket.listen(5)
print("TCPServer Waiting for client on port 7000")
while 1:
client_socket, address = server_socket.accept()
print("I got a connection from ", address)
while 1:
data = client_socket.recv(32)
print("RECEIVED:",data)
l = []
for e in data:
l.append(e)
print(e)
print("RECEIVED:",sum([l[3-i] << 8*i for i in (3,2,1,0)]))
if (data == b''):
break;
break;
C++ server socket:
#define WIN32_LEAN_AND_MEAN
#include <WinSock2.h>
#include <WS2tcpip.h>
#include <stdio.h>
#include <stdlib.h>
// link with Ws2_32.lib
#pragma comment(lib, "Ws2_32.lib")
#define DEFAULT_PORT "7000"
//"27015"
#define DEFAULT_BUFFER_LENGTH 32
//512
int main() {
WSADATA wsaData;
// Initialize Winsock
int iResult = WSAStartup(MAKEWORD(2,2), &wsaData);
if(iResult != 0)
{
printf("WSAStartup failed: %d\n", iResult);
return 1;
}
struct addrinfo *result = NULL,
hints;
ZeroMemory(&hints, sizeof(hints));
hints.ai_family = AF_INET; // Internet address family is unspecified so that either an IPv6 or IPv4 address can be returned
hints.ai_socktype = SOCK_STREAM; // Requests the socket type to be a stream socket for the TCP protocol
hints.ai_protocol = IPPROTO_TCP;
hints.ai_flags = AI_PASSIVE;
// Resolve the local address and port to be used by the server
iResult = getaddrinfo(NULL, DEFAULT_PORT, &hints, &result);
if (iResult != 0)
{
printf("getaddrinfo failed: %d\n", iResult);
WSACleanup();
return 1;
}
SOCKET ListenSocket = INVALID_SOCKET;
// Create a SOCKET for the server to listen for client connections
ListenSocket = socket(result->ai_family, result->ai_socktype, result->ai_protocol);
if (ListenSocket == INVALID_SOCKET)
{
printf("Error at socket(): %d\n", WSAGetLastError());
freeaddrinfo(result);
WSACleanup();
return 1;
}
// Setup the TCP listening socket
iResult = bind(ListenSocket, result->ai_addr, (int)result->ai_addrlen);
if (iResult == SOCKET_ERROR)
{
printf("bind failed: %d", WSAGetLastError());
freeaddrinfo(result);
closesocket(ListenSocket);
WSACleanup();
return 1;
}
freeaddrinfo(result);
// To listen on a socket
if ( listen(ListenSocket, SOMAXCONN) == SOCKET_ERROR)
{
printf("listen failed: %d\n", WSAGetLastError());
closesocket(ListenSocket);
WSACleanup();
return 1;
}
SOCKET ClientSocket;
ClientSocket = INVALID_SOCKET;
// Accept a client socket
ClientSocket = accept(ListenSocket, NULL, NULL);
if (ClientSocket == INVALID_SOCKET)
{
printf("accept failed: %d\n", WSAGetLastError());
closesocket(ListenSocket);
WSACleanup();
return 1;
}
char recvbuf[DEFAULT_BUFFER_LENGTH];
int iSendResult;
// receive until the client shutdown the connection
do {
iResult = recv(ClientSocket, recvbuf, DEFAULT_BUFFER_LENGTH, 0);
if (iResult > 0)
{
char msg[DEFAULT_BUFFER_LENGTH];
memset(&msg, 0, sizeof(msg));
strncpy(msg, recvbuf, iResult);
printf("Received: %s\n", msg);
//Here is where I implement the python code:
//sum([l[3-i] << 8*i for i in (3,2,1,0)]));
int sum = 0;
int term = 0;
for(int i = 3;i > -1;i--)
{
printf("%d\n",msg[3-i]);
term = msg[3-i];
//if (term < 0)
// term = 256 + term;
sum += term << 8*i;
};
printf("Received: %d\n",sum);
iSendResult = send(ClientSocket, recvbuf, iResult, 0);
if (iSendResult == SOCKET_ERROR)
{
printf("send failed: %d\n", WSAGetLastError());
closesocket(ClientSocket);
WSACleanup();
getchar();
return 1;
}
printf("Bytes sent: %ld\n", iSendResult);
}
else if (iResult == 0)
printf("Connection closed\n");
else
{
printf("recv failed: %d\n", WSAGetLastError());
closesocket(ClientSocket);
WSACleanup();
getchar();
return 1;
}
} while (iResult > 0);
// Free the resouces
closesocket(ListenSocket);
WSACleanup();
getchar();
//while (true){};
return 0;
}
msg[] is declared as char which is not guaranteed to be unsigned. Use unsigned char.
Both of lines
printf("%d\n",msg[3-i]);
term = msg[3-i];
cast to signed integer. %d formats to signed integer, use %u. term is declared as int, make it unsigned int.
Following #Ante advices, I changed the code in C++ server socket, I also changed recvbuf to unsigned char and sum from int to unsigned int for consistency (after all I am waiting to receiving unsigned chars and "rebuild" an unsigned int) but it also worked leaving both recvbuf and sum as they were. I left commented what was before.
The problem was that for representing an integer so big I was actually using an unsigned integer and the bytes that were being sent were ascii codes whose range is 0 - 255 (the same range of unsigned char) and char range is -127 - 126.
Nevertheless, sockets do not care about data types, they just send binary data so I was receiveng unsigned chars that when bein put in a char they "overflow" and go negative (technically it is because of how two's complement works I think).
A few more notes about the corrected code:
1) This
if (term < 0)
term = 256 + term;
is no longer necessary (actually I was fixing manually the issue of the overflow).
2) I had to use cast to char* (char ) in order to be able to use recv,strncpy and send which take char as parameters not unsigned char* . This work and I think it is not hackish because both a pointer to char and a pointer to unsigned char point to a datatype with the same size in memory (8 bits). If this is wrong or could lead to unwanted or unexpected behaviour please correct me.
//char recvbuf[DEFAULT_BUFFER_LENGTH];
unsigned char recvbuf[DEFAULT_BUFFER_LENGTH];
int iSendResult;
// receive until the client shutdown the connection
do {
//iResult = recv(ClientSocket, recvbuf, DEFAULT_BUFFER_LENGTH, 0);
iResult = recv(ClientSocket, (char *)recvbuf, DEFAULT_BUFFER_LENGTH, 0);
if (iResult > 0)
{
//char msg[DEFAULT_BUFFER_LENGTH];
unsigned char msg[DEFAULT_BUFFER_LENGTH];
memset(&msg, 0, sizeof(msg));
//strncpy((msg, recvbuf, iResult);
strncpy((char *)msg, (char *)recvbuf, iResult);
printf("Received: %s\n", msg);
//sum([l[3-i] << 8*i for i in (3,2,1,0)]));
//int sum = 0;
unsigned int sum = 0;
//int term = 0;
unsigned int term = 0;
for(int i = 3;i > -1;i--)
{
//printf("%d\n",msg[3-i]);
printf("%u\n",msg[3-i]);
term = msg[3-i];
sum += term << 8*i;
};
//printf("Received: %d\n",sum);
printf("Received: %u\n",sum);
//iSendResult = send(ClientSocket, recvbuf, iResult, 0);
iSendResult = send(ClientSocket, (char *) recvbuf, iResult, 0);

Categories

Resources