I had a tcp proxy in python the version is 2.6.
It works fine in any cases with following logic
client ---> proxy ---> server
I wrapped the tcp with tls from proxy to server.
client ---> proxy ==++ssl++==> server
That works fine in some cases and fails in others.
The error is that the server is waiting for more information from the client, but client sends nothing more. At the 26th round trip.(Certainly, the round trip number of successful case also larger than 26.)
I cannot tell more about the detail but I thought the SSL should be transparent to the logic.
Any Idea that part of the functionality fails? How should I debug it?
Edit: In python 2.6, the tls version can only be 1.0.
It is hard to tell what you are doing without any example demonstrating the problem but depending on how your application works SSL/TLS is not just a transparent replacement for TCP sockets. While it might be transparent in most cases if you use only blocking sockets it gets different with non-blocking I/O. In this case you have to deal with user space buffering where select will not report available data even thought there are unread data. You also have to deal with situations where you temporarily fail to write because the TLS stack needs a read first or the other way.
For more details about differences with non-blocking I/O and select see Behavior of python's select() with partial recv() on SSL socket or select and ssl in python. Additionally non-blocking I/O needs special handling with accept and connect too but I doubt that there is useful support for it in the old python version you are using.
Related
I am making a SSL server, and I don't use python's library as I want to make some unorthodox changes to the process. Because of that, I cannot simply start a TCP connection since I need to transfer the encryption details with the handshake, which I can't do over sockets. So I am using scapy to make the handshake itself, but after that I would like to continue working with a TCP socket without going through the process of the handshake again. Is that possible?
If I understand your question correctly, you exchanged a few segments using scapy and now want to manufacture a normal full-blown socket out of them.
This is not easily possible: for all practical purposes your TCP is oblivious to whatever you sent in your packets and it doesn't keep any state for this TCP connection: all the state is in your application.
That said, there is a thing called TCP_REPAIR in Linux that lets you put a socket in a given state.
When this option is used, a socket is switched into a special mode, in
which any action performed on it does not result in anything defined
by an appropriate protocol actions, but rather directly puts the
socket into a state, in which the socket is expected to be at the end
of the successfully finished operation.
If you set sequence numbers correctly, the socket should "just work".,
One also needs to restore the TCP sequence numbers. To do so, the
TCP_REPAIR_QUEUE and TCP_QUEUE_SEQ options were introduced.
Of course all this is specific to a modern Linux; other operating systems may or may not have similar mechanisms.
I am currently working on a server + client combo on python and I'm using TCP sockets. From networking classes I know, that TCP connection should be closed step by step, first one side sends the signal, that it wants to close the connection and waits for confirmation, then the other side does the same. After that, socket can be safely closed.
I've seen in python documentation function socket.shutdown(flag), but I don't see how it could be used in this standard method, theoretical of closing TCP socket. As far as I know, it just blocks either reading, writing or both.
What is the best, most correct way to close TCP socket in python? Are there standard functions for closing signals or do I need to implement them myself?
shutdown is useful when you have to signal the remote client that no more data is being sent. You can specify in the shutdown() parameter which half-channel you want to close.
Most commonly, you want to close the TX half-channel, by calling shutdown(1). In TCP level, it sends a FIN packet, and the remote end will receive 0 bytes if blocking on read(), but the remote end can still send data back, because the RX half-channel is still open.
Some application protocols use this to signal the end of the message. Some other protocols find the EOM based on data itself. For example, in an interactive protocol (where messages are exchanged many times) there may be no opportunity, or need, to close a half-channel.
In HTTP, shutdown(1) is one method that a client can use to signal that a HTTP request is complete. But the HTTP protocol itself embeds data that allows to detect where a request ends, so multiple-request HTTP connections are still possible.
I don't think that calling shutdown() before close() is always necessary, unless you need to explicitly close a half-channel. If you want to cease all communication, close() does that too. Calling shutdown() and forgetting to call close() is worse because the file descriptor resources are not freed.
From Wikipedia: "On SVR4 systems use of close() may discard data. The use of shutdown() or SO_LINGER may be required on these systems to guarantee delivery of all data." This means that, if you have outstanding data in the output buffer, a close() could discard this data immediately on a SVR4 system. Linux, BSD and BSD-based systems like Apple are not SVR4 and will try to send the output buffer in full after close(). I am not sure if any major commercial UNIX is still SVR4 these days.
Again using HTTP as an example, an HTTP client running on SVR4 would not lose data using close() because it will keep the connection open after request to get the response. An HTTP server under SVR would have to be more careful, calling shutdown(2) before close() after sending the whole response, because the response would be partly in the output buffer.
According to the python documentation which says:
Strictly speaking, you’re supposed to use shutdown on a socket before
you close it. The shutdown is an advisory to the socket at the other
end. Depending on the argument you pass it, it can mean “I’m not going
to send anymore, but I’ll still listen”, or “I’m not listening, good
riddance!”. Most socket libraries, however, are so used to programmers
neglecting to use this piece of etiquette that normally a close is the
same as shutdown(); close(). So in most situations, an explicit
shutdown is not needed.
I think the most correct way to close a TCP connection would be to use shutdown before closing a connection, because close is not atomic! This can make some bugs. Suppose you're using close function without shutdown and the data didn't send to the server correctly, at the same time python closes the connection and server can't reply to client, now the socket at the other end may hang indefinitely.
I'm working with a python application that makes remote procedure calls, using Twisted Perspective broker's callRemote, on a TCP connection. From a system call trace, it appears that multiple remote procedure calls from the sender could be aggregated together into a single sendto() call on the socket. The same behavior was observed with the receiver's response as well. I would've thought that as long as the socket was write-able and if there was some data to send, Perspective broker would send it out on the socket. But it does not appear to be the case.
Does Twisted's Perspective broker aggregate multiple RPC messages together for a specific reason, before they are sent on the socket ? In other words, does Twisted do something similar to Nagle's algorithm in TCP ?
If the above is true, is there an option to turn off this behavior ?
Twisted performs write buffering in the underlying twisted.internet.abstract.FileDescriptor object. You can try changing the twisted.internet.abstract.FileDescriptor.SEND_LIMIT attribute to something smaller to force it to write to the socket more frequently.
See the Twisted bug 4089 for discussion about the SEND_LIMIT and bufferSize attributes.
I am working on a project where I have a client server model in python. I set up a server to monitor requests and send back data. PYZMQ supports: tcp, udp, pgm, epgm, inproc and ipc. I have been using tcp for interprocess communication, but have no idea what i should use for sending a request over the internet to a server. I simply need something to put in:
socket.bind(BIND_ADDRESS)
DIAGRAM: Client Communicating over internet to server running a program
Any particular reason you're not using ipc or inproc for interprocess communication?
Other than that, generally, you can consider tcp the universal communicator; it's not always the best choice, but no matter what (so long as you actually have an IP address) it will work.
Here's what you need to know when making a choice between transports:
PGM/EPGM are multicast transports - the idea is that you send one message and it gets delivered as a single message until the last possible moment where it will be broken up into multiple messages, one for each receiver. Unless you absolutely know you need this, you don't need this.
IPC/Inproc are for interprocess communication... if you're communicating between different threads in the same process, or different processes on the same logical host, then these might be appropriate. You get the benefit of a little less overhead. If you might ever add new logical hosts, this is probably not appropriate.
Russle Borogove enumerates the difference between TCP and UDP well. Typically you'll want to use TCP. Only if absolute speed is more important than reliability then you'll use UDP.
It was always my understanding that UDP wasn't supported by ZMQ, so if it's there it's probably added by the pyzmq binding.
Also, I took a look at your diagram - you probably want the server ZMQ socket to bind and the client ZMQ socket to connect... there are some reasons why you might reverse this, but as a general rule the server is considered the "reliable" peer, and the client is the "transient" peer, and you want the "reliable" peer to bind, the "transient" peer to connect.
Over the internet, TCP or UDP are the usual choices. I don't know if pyzmq has its own delivery guarantees on top of the transport protocol. If it doesn't, TCP will guarantee in-order delivery of all messages, while UDP may drop messages if the network is congested.
If you don't know what you want, TCP is the simplest and safest choice.
I'm trying to communicate with a device that uses the same protocol whether you're talking to it over a TCP socket or a serial port. Either way, it's an ASCII command-based interface, in which you type your command at a prompt, followed by a newline, you get a response with a newline, and then a new prompt.
> IDENTIFY
DEVICE_TYPE_RESPONSE
> TEST POWER
OK
>
The program can and should block until it gets the new prompt, but there needs to be a timeout so you don't wait forever in case of sudden device death. That timeout should be seconds so as not to falsely flag on a network glitch, but shouldn't force me to wait seconds once I've gotten my prompt character.
I'd love some way to abstract the interface so as to not care about what the underlying communications layer is. Just pass it either an open socket or an open serial port and let everything happen. Is there some good way to do this? Preferably capable of running under Python 2.6.
You could implement the protocol in Twisted which allows you to use TCP or the serial port as a transport without changing your protocol implementation. Twisted also allows you to set timeouts/delayed callbacks.
Python's serial package provides a lot of useful stuff including some serial over TCP/IP bridges. If you want to talk to it using ASCII/Telnet then you probably want the to use the '--convert' option with the Simple Serial to Network (TCP/IP) redirector.
Also you might want to take a look at this other related question on Converting serial port data to TCP/IP in a linux environment