i want to build a chat application which supports text messaging, group messaging, file transfer(like netmeeting). when i send it over TCP socket i saw that data is not structured all the data send as string over TCP. I want to send it in a structured way with few headers like name:,ip:,data:,data_type_flag:(file or text message) etc... one stackoverflow member told me to use TELEPATHY but i can't get a simple tutorial to understand. how can i send structured data over socket? or can any one suggest me a good tutorial to implement telepathy properly. i want to communicate over network as peer-to-peer rather than dedicated server.. Thanks
Try google protcol buffers or apache thrift. There are many examples for how to use them.
As for your comment about "peer to peer", please realize that even in peer-to-peer one of the peers is always acting as a server (sometimes both are).
TCP is a transport layer protocol, as opposed to application layer. This means that TCP is not responsible for the types of data you send, only the raw bits. HTTP has headers and other metadata because it is application level.
For a project like the one you're talking about, you will want to implement your own application layer protocol, but this is not exactly a trivial task. I would look at the python source code in the httplib module for an example of how to implement such a protocol, but note that this is likely fairly different still from what you want, as you will want persistent socket connections to be a first-class citizen in a peer-to-peer chat protocol like the one you're describing.
Another option is to use one of the various RPC libraries, eg xmlrpclib, which will handle a decent amount of the required low-level network things for you (although not file transfer; there are other libraries like the ftplib that can do this).
Pickle your data before you send it, and unpickle it on the other end?
http://docs.python.org/library/pickle.html
Related
I'm new to twisted, and I'm having trouble working out how I should organise my code. The client connects to a TCP(SSL) control channel and then will try to connect to the same IP:port on UDP for a low-latency data channel, based on encryption settings provided over TCP. If it can't, the TCP control channel will be used for the data. I'd like to write a reusable client such that people can override a class with functions such as dataReceived, controlMessageXReceived, sendControlMessageX, sendDataMessage etc, with whether the UDP channel is in use or not abstracted away into my code.
I currently have a Protocol that can understand the TCP control channel; for testing purposes I've overridden ConnectionMade() there to send set-up messages and confirm everything works (it can understand the server and vice versa) but I have no idea how to integrate that into a wider context.
(For the curious, this is a client for Mumble - this protocol specification is here, and I'm trying to update this horrible pile of umaintainable (multithreaded) code into something modern)
Consider mirroring the protocol/transport separation already present in Twisted.
Protocol doesn't know anything about TCP. It just knows how to handle a stream of bytes. It's the transport that knows about TCP (or TLS, or UNIX sockets, or something else).
There is an explicit interface between Protocol and its transport (actually, there are two - IProtocol lets a transport know what it can do to the protocol object and ITransport lets the protocol know what it can do to the transport object).
Invent an interface that makes sense for the application you're working with. For example, Protocol has dataReceived because "some bytes arrived" is one of the things that happens with "a stream of bytes". What things can happen in Mumble? For example these things might be, "a user connected to the server" or "a message arrived in the channel you're in". Your interface might have a method for each of these.
Now application developers can implement their own novel behavior by writing an implementation of this interface - which is explicitly and completely defined - and then plugging that implementation into your library (for example, perhaps your library could offer a connectToMumbleServer(address, mumbleApplicationObject) API).
Your library knows exactly what it's allowed to do with the application object because the interface is explicitly defined. If you repeat this process for the opposite direction then the application developer will know what they can do to the mumble server using your library, too (eg "join a channel" or "send a packet of audio data").
You could provide a base class (like Protocol) for applications to subclass but this is a very minor bit of convenience. In case you haven't recently, open up twisted/internet/protocols.py and look at the implementation of the Protocol class. There's almost nothing there and none of what is there is very complicated or difficult to replicate. If application developers had to start off subclassing object and type out all of the methods themselves they wouldn't be at much of a disadvantage.
I am writing an implementation of an NAT and have the need to use raw sockets. I have become accustomed to the Twisted architecture and like how it handles concurrent connections.
Data coming into a Twisted protocol is manipulated, NATed, tabulated, and sent out the raw socket. Data coming into the raw socket is manipulated, looked up, NATed, and directed to the appropriate protocol instance.
Would having a single raw socket suffice? What if a large number of connections came in at the same time. Doesn't twisted handle that, or is twisted pretty much a non advantage in connectionless protocols. If there is an advantage could anyone direct me to a raw sockets twisted example
Twisted supports connectionless protocols just fine. See, for example, listenUDP.
There are modules for manipulating IP-level protocol data in twisted.pair, but not all of it works; in particular, tuntap support does not work.
There are no examples of this that I know of, but as I understand it, a single raw socket should be fine. You will, however, need to write your own transport, wrapping the socket up in an IReadDescriptor / IWriteDescriptor and using IReactorFDSet. However, if you're adept enough to know you need raw sockets in the first place, this shouldn't be too hard.
Dig around in the twisted source code and you'll find twisted.pair, which isn't really maintained anymore, but gives you about 90% of what you need to do raw sockets.
I have some example code somewhere showing how to use /dev/bpf on BSD with it. The only caveat is that it's pure python so no tcpdump-style packet filters (port 80 and host blah.com) - you just have to drink from the fire hose.
i am looking for an abstract and clean way to exchange strings between two python programs. The protocol is really simple: client/server sends a string to the server/client and it takes the corresponding action - via a handler, i suppose - and replies OR NOT to the other side with another string. Strings can be three things: an acknowledgement, signalling one side that the other on is still alive; a pickled class containing a command, if going from the "client" to the "server", or a response, if going from the "server" to the "client"; and finally a "lock" command, that signals a side of the conversation that the other is working and no further questions should be asked until another lock packet is received.
I have been looking at the python's built in SocketServer.TCPServer, but it's way too low level, it does not easily support reconnection and the client has to use the socket interface, which i preferred to be encapsulated.
I then explored the twisted framework, particularly the LineOnlyReceiver protocol and server examples, but i found the initial learning curve to be too steep, the online documentation assuming a little too much knowledge and a general lack of examples and good documentation (except the 2005 O'reilly book, is this still valid?).
I then tryied the pyliblo library, which is perfect for the task, alas it is monodirectional, there is no way to "answer" a client, and i need the answer to be associated to the specific command.
So my question is: is there an existing framework/library/module that allows me to have a client object in the server, to read the commands from and send the replies to, and a server object in the client, to read the replies from and send the commands to, that i can use after a simple setup (client, the server address is host:port, server, you are listening on port X) having the underlying socket, reconnection engine and so on handled?
thanks in advance to any answer (pardon my english and inexperience, this is my first question)
Python also provides an asyncchat module that simplifies much of the server/client behavior common to chat-like communications.
What you want to do seems a lot like RPC, so the things that come to my mind are XMLRPC or JSON RPC, if you dont want to use XML .
Python has a XMLRPC library that you can use, it uses HTTP as the transport so it also solves your problem of not being too low level. However if you could provide more detail in terms of what you exactly want to do perhaps we can give a better solution.
I decided to improve my knowledge about python network programming and here is the deal: I have a simple server for Windows, which interacts with a client from a mobile device using wi-fi. Also I have a packet sniffer (Wireshark).
Now I want to ask, what do I need to write the Linux version of this server? How to determine the structure of packets, establish the connection? What do I need to use - sockets, Twisted, maybe Tornado?
Start with the SocketServer module and build from there.
Note that this will take a lot of guesswork if there is no documentation about the protocol. If you're lucky, they are using XML or HTML. If not, you will have to make the existing server send a lot of test data which you have to manipulate in some way (by changing fields and see what changes in the data stream).
Good luck!
I have built a Python server to which various clients can connect, and I need to set a predefined series of messages from clients to the server - For example the client passes in a name to the server when it first connects.
I was wondering what the best way to approach this is? How should I build a simple protocol for their communication?
Should the messages start with a specific set of bytes to mark them out as part of this protocol, then contain some sort of message id? Any suggestions or further reading appreciated.
First, you need to decide whether you want your protocol to be human readable (much more overhead) or binary. If the first, you probably want to use regular expressions to decode the messages. For this, use the python module re. If the latter, the module struct is your friend.
Second, if you are building a protocol that is somehow stateful (e.g. first we do a handshake, then we transfer data, then we check checksums and say goodbye) you probably want to create a some sort of FSM to track the state.
Third, if protocol design is not a familiar subject, read some simple protocol specifications, for example by IETF
If this is not a learning excercise, you might want to build up from something else, like Python Twisted
Depending on the requirements, you might want to consider using JSON: use "newline" terminated strings with JSON encoding. The transport protocol could be HTTP: with this, you could have access to all the "connection related" facilities (e.g. status codes) and have JSON encoded payload.
The advantages of using JSON over HTTP:
human readable (debugging etc.)
libraries available for all languages/platforms
cross-platform
browser debuggable (to some extent)
Of course, there are many other ways to skin this cat but the time to working prototype using this approach is very low. This is worth considering if your requirements (which aren't very detailed here) can be met.
Read some protocols, and try to find one that looks like what you need. Does it need to be message-oriented or stream-oriented? Does it need request order to be preserved, does it need requests to be paired with responses? Do you need message identifiers? Retries, back-off? Is it an RPC protocol, a message queue protocol?
See http://www.faqs.org/docs/artu/ch05s02.html and http://www.faqs.org/docs/artu/ch05s03.html for a good overview and discussion on data file formats and protocols.