Develop Reference

Python: how to calculate data received and send between two ipaddresses and ports

I guess it's socket programming. But I have never done socket programming expect for running the tutorial examples while learning Python. I need some more ideas to implement this.
What I specifically need is to run a monitoring program of a server which will poll or listen to traffic being exchange from different IPs across different popular ports. For example, how do I get data received and sent through port 80 of 192.168.1.10 and 192.168.1.1 ( which is the gateway).
I checked out a number of ready made tools like MRTG, Bwmon, Ntop etc but since we are looking at doing some specific pattern studies, we need to do data capturing within the program.
Idea is to monitor some popular ports and do a study of network traffic across some periods and compare them with some other data.
We would like to figure a way to do all this with Python....

You probably want to use scapy for that. Just sniff all ethernet traffic on a particular interface, drop everything that is not TCP and doesn't match the port.
Not sure if scapy can already track TCP connections (stuff like recognizing duplicate sequence numbers, extracting just the payload stream) but I would guess it probably can, and if not it's not too hard to hack together a good-enough TCP connection tracker that works for 95% of the traffic.
Alternatives would be to use sockets directly (look for raw sockets) or libpcap, which can both be done from Python. You may also want to check out the filter experssion syntax of the 'tcpdump' commandline tool, maybe it can do what you want already.
I bet there are more specialized high-level tools for this, but I don't know them.
PS: if you don't know wireshark yet, go check it out and play around with it first. It can follow TCP streams and will teach you what TCP connection tracking means. Maybe its commandline binary, tshark, can be used to extract TCP streams for what you want.

IPTraf is an ncurses based IP LAN monitoring tool. Has a capability to generate network statistics including TCP,UDP,ICMP and some more.
Since you're thinking to execute it from python, you may consider to use screen (screen manager with VT100/ANSI terminal emulation) to overcome ncurses issues and you may want to pass logging and interval parameters to IPTraf which forces iptraf to log to a file in a given interval. Little bit tricky but eventually you can have what you are looking for by basically parsing the log file.

twisted - print IP datagrams from/to proxy

I have a twisted proxy from here: Python Twisted proxy - how to intercept packets .
It prints the HTTP data, and I would like also to intercept and examine the raw IP datgrams. How to hook the callback for the IP packets?
http://twistedmatrix.com/documents/11.0.0/api/twisted.pair.ip.IPProtocol.html

Twisted doesn't have a built-in friendly way to hook in a listener on a raw IP socket (SOCK_RAW). This is for several reasons:
using SOCK_RAW can be tricky and it can work in non-obvious ways;
in most environments, using such a socket requires elevated privileges;
and the packets you actually get through a raw socket differ a lot between operating systems (e.g., you won't get any raw TCP-protocol IP packets on *BSD/Darwin through a raw socket, even if you're root).
The best way to capture raw datagrams in general, in a remotely portable manner, is with libpcap. Here is a link to someone who appears to have combined pcap and Twisted in a reasonably intelligent way; that may help.

Twisted doesn't include comprehensive support for operating at the IP level. There is some support for parsing IP datagrams, as you found, but no built-in support for hooking into platform support for sending or receiving these.
You might want to take a look at scapy.

(Python) Send structured packet over network

i want to build a chat application which supports text messaging, group messaging, file transfer(like netmeeting). when i send it over TCP socket i saw that data is not structured all the data send as string over TCP. I want to send it in a structured way with few headers like name:,ip:,data:,data_type_flag:(file or text message) etc... one stackoverflow member told me to use TELEPATHY but i can't get a simple tutorial to understand. how can i send structured data over socket? or can any one suggest me a good tutorial to implement telepathy properly. i want to communicate over network as peer-to-peer rather than dedicated server.. Thanks

Try google protcol buffers or apache thrift. There are many examples for how to use them.
As for your comment about "peer to peer", please realize that even in peer-to-peer one of the peers is always acting as a server (sometimes both are).

TCP is a transport layer protocol, as opposed to application layer. This means that TCP is not responsible for the types of data you send, only the raw bits. HTTP has headers and other metadata because it is application level.
For a project like the one you're talking about, you will want to implement your own application layer protocol, but this is not exactly a trivial task. I would look at the python source code in the httplib module for an example of how to implement such a protocol, but note that this is likely fairly different still from what you want, as you will want persistent socket connections to be a first-class citizen in a peer-to-peer chat protocol like the one you're describing.
Another option is to use one of the various RPC libraries, eg xmlrpclib, which will handle a decent amount of the required low-level network things for you (although not file transfer; there are other libraries like the ftplib that can do this).

Pickle your data before you send it, and unpickle it on the other end?
http://docs.python.org/library/pickle.html

Simple server/client string exchange protocol

i am looking for an abstract and clean way to exchange strings between two python programs. The protocol is really simple: client/server sends a string to the server/client and it takes the corresponding action - via a handler, i suppose - and replies OR NOT to the other side with another string. Strings can be three things: an acknowledgement, signalling one side that the other on is still alive; a pickled class containing a command, if going from the "client" to the "server", or a response, if going from the "server" to the "client"; and finally a "lock" command, that signals a side of the conversation that the other is working and no further questions should be asked until another lock packet is received.
I have been looking at the python's built in SocketServer.TCPServer, but it's way too low level, it does not easily support reconnection and the client has to use the socket interface, which i preferred to be encapsulated.
I then explored the twisted framework, particularly the LineOnlyReceiver protocol and server examples, but i found the initial learning curve to be too steep, the online documentation assuming a little too much knowledge and a general lack of examples and good documentation (except the 2005 O'reilly book, is this still valid?).
I then tryied the pyliblo library, which is perfect for the task, alas it is monodirectional, there is no way to "answer" a client, and i need the answer to be associated to the specific command.
So my question is: is there an existing framework/library/module that allows me to have a client object in the server, to read the commands from and send the replies to, and a server object in the client, to read the replies from and send the commands to, that i can use after a simple setup (client, the server address is host:port, server, you are listening on port X) having the underlying socket, reconnection engine and so on handled?
thanks in advance to any answer (pardon my english and inexperience, this is my first question)

Python also provides an asyncchat module that simplifies much of the server/client behavior common to chat-like communications.

What you want to do seems a lot like RPC, so the things that come to my mind are XMLRPC or JSON RPC, if you dont want to use XML .
Python has a XMLRPC library that you can use, it uses HTTP as the transport so it also solves your problem of not being too low level. However if you could provide more detail in terms of what you exactly want to do perhaps we can give a better solution.

Network programming in Python

What library should I use for network programming? Is sockets the best, or is there a higher level interface, that is standard?
I need something that will be pretty cross platform (ie. Linux, Windows, Mac OS X), and it only needs to be able to connect to other Python programs using the same library.

You just want to send python data between nodes (possibly on separate computers)? You might want to look at SimpleXMLRPCServer. It's based on the inbuilt HTTP server, which is based on the inbuilt Socket server, neither of which are the most industrial-strength servers around, but it's easy to set up in a hurry:
from SimpleXMLRPCServer import SimpleXMLRPCServer
server = SimpleXMLRPCServer(("localhost", 9876))
def my_func(a,b):
return a + b
server.register_function(my_func)
server.serve_forever()
And easy to connect to:
import xmlrpclib
s = xmlrpclib.ServerProxy('http://localhost:9876')
print s.my_func(2,3)
>>> 5
print type(s.my_func(2,3))
>>> <type 'int'>
print s.my_func(2,3.0):
>>> 7.0
Twisted is popular for industrial applications, but it's got a brutal learning curve.

There is a framework that you may be interested in: Twisted

the answer depends on what you are trying to do.
"What library should I use for network programming?" is pretty vague.
for example, if you want to do HTTP, you might look at such standard libraries as urllib, urllib2, httplib, sockets. It all depends on which protocol you are looking to use, and which network layer you want to work at.
there are libraries in python for various network tasks... email, web, rpc, etc etc...
for starters, look over the standard library reference manual and see which tasks you want to do, then go from there: http://docs.python.org/library/index.html

As previously mentioned, Twisted is the most popular (by far). However, there are a lot of other alternative worth exploring. Tornado and Diesel are probably the top two contenders. A more complete comparison is found here.

Personally I just use asyncore from the standard library, which is a bit like a very cut-down version of Twisted, but this is because I prefer a simple and low level interface. If you want a higher level interface, especially just to communicate with another instance of your own program, you don't necessarily have to worry about the networking layer, and can consider something higher level like RPyC or pyro instead. The network then becomes an implementation detail and you can concentrate on just sending the information.

A lot of people like Twisted. I was a huge fan for awhile, but on working with it a bit and thinking about it more I've become disenchanted. It's complex, and last I looked, a lot of it had the assumption that your program would always be able to send data leading to possible situations in which your program grows memory usage endlessly buffering data to send that isn't being picked up by the remote side or isn't being picked up fast enough.
In my opinion, it depends a lot on what kind of network programming you want to do. A lot of times you don't really care about getting stuff done while you're waiting for IO. HTTP, for example, is very request-response oriented, and if you're only talking to a single server there is little reason to need something like Twisted and plain sockets or Python's built-in HTTP libraries will work fine.
If you're writing a server of any kind, you almost certainly need to be event-driven. Twisted has a slight edge there, but it still seems overly complex to me. Bittorrent, for example, was written in Python and doesn't use Twisted at all.
Another factor favoring Twisted is that there is code for a lot of protocols already written for it. So if you want to speak an existing protocol a lot of hard work may already have been done for you.

The socket module in the standard lib is in my opinion a good choice if you don't need high performance.
It is a very famous API that is known by almost every developpers of almost every languages. It's quite sipmple and there is a lot of information available on the internet. Moreover, it will be easier for other people to understand your code.
I guess that an event-driven framework like Twisted has better performance but in basic cases standard sockets are enough.
Of course, if you use a higher-level protocol (http, ftp...), you should use the corresponding implementation in the python standard library.

Socket is low level api, it is mapped directly to operating system interface.
Twisted, Tornado ... are high level framework (of course they are built on socket because socket is low level).
When it come to TCP/IP programming, you should have some basic knowledge to make a decision about what you shoud use:
Will you use well-known protocol like HTTP, FTP or create your own protocol?
Blocking or non-blocking? Twisted, Tornado are non-blocking framework (basically like nodejs).
Of course, socket can do everything because every other framework is base on its ;)