Develop Reference

How to use a Python 3 socket client/server setup over the internet (not locally)?

I am trying to implement a simple chat program that uses sockets to transmit data via a UDP connection. However, I can't figure out how to correctly set it up so that people from outside my local network can access it if I am hosting it on my laptop. I am utilizing port 5000, and have port-forwarded that port on my router for my laptop. The port-forwarding doesn't seem to be the issue; at least the "Port Forward Network Utilities" from portforward.com seems to detect it as properly forwarded. Maybe I am mixing up the IP addresses I need to host from and connect with? The code in question is below:
import socket
import threading
import sys
class Server:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
connections = []
def __init__(self):
self.sock.bind(('192.168.1.5', 5000))
self.sock.listen(1)
def handler(self, c, a):
while True:
data = c.recv(1024)
for connection in self.connections:
print(data.decode())
connection.send(data)
if not data:
break
def run(self):
while True:
c, a = self.sock.accept()
cThread = threading.Thread(target=self.handler, args=(c, a))
cThread.daemon = True
cThread.start()
self.connections.append(c)
print(self.connections)
class Client:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
usr_name = ""
def sendMsg(self):
while True:
self.sock.send(bytes(self.usr_name + ": " + input("-> "), 'utf-8'))
def __init__(self, address):
self.sock.connect((address, 5000))
self.usr_name = input("Enter a username: ")
iThread = threading.Thread(target=self.sendMsg)
iThread.daemon = True
iThread.start()
while True:
data = self.sock.recv(1024)
if not data:
break
print(data.decode())
if len(sys.argv) > 1:
client = Client(sys.argv[1])
else:
server = Server()
server.run()
As you can see, I have my current local IP address inputted for hosting the server, while the client asks for an IP to connect to. I'm not sure what to do now for hosting this over the internet, but I have tried every IP combination I can think of and it returns a number of errors.
Thanks in advance.
Edit:
The two main errors I was getting are:
Timeout Error [WinError 10060]
My friend received this when trying to connect from another network
[WinError 10061]
I would receive this when trying to connect using my public IP from the same computer
I'm sorry that I can't be more detailed in my errors and provide a full printout, and I will try to update this if I'm able to replicate them.
Edit:
I was able to rewrite it and get it to work, I don't need anymore help with this.
Thanks.

You're port-forwarding UDP port 5000 to 5000.
But you're opening TCP streams, not UDP. That's what SOCK_STREAM means. If you want UDP, you need to use SOCK_DGRAM.
So, you need to make these two consistent. The only problem is, I'm not sure which one you actually want here.
On the one hand, your code is doing connection-oriented recv, and seems to be assuming reliable transmission, which means you probably want TCP.
On the other hand, your code seems to be assuming that each recv(1024) is going to get exactly one send from the other side, which is only true for UDP; TCP sockets are byte streams, not message streams. When you do a recv(1024), you could easily get just the first 12 bytes of an 80-byte line, which means it could end in the middle of a UTF-8 character, which means decode will throw an exception.
I think you want TCP, but with a framing protocol on top of it. The simplest protocol that would probably make sense here is lines of text. Which is pretty easy to do on your own, but even easier to do with socket.makefile, given that you're dedicating a thread to each connection.

Simple TELNET in Python - loop paused if no data coming

I'm trying to build a very simple TELNET client in Python and I'm getting problem on the last part: sending/receiving data to/from the server.
With the code I have, if no data arrives at the very beginnig, the loop get paused and I can't even send commands.
Here the interested part of the code:
# Infinite cycle that allows user to get and send data from/to the host
while True:
incoming_data = my_socket.recv(4096)
if not incoming_data:
print('Problem occurred - Connection closed')
my_socket.close()
sys.exit()
else:
# display data sent from the host trough the stdout
sys.stdout.write(incoming_data)
# Commands sent to the host
command = sys.stdin.readline()
my_socket.send(command)
(I think the program kinda of works if I try to connect to some hosts that send data at the beginning.)
The idea would be have two loops, running at the same time, getting data or sending data, but I can't get it to work.
I can't use the telnet library and I don't want to use the select library (only sys and socket).

You want to use the threading library.
The following program runs the receiving in one thread and the sending in another:
import socket
from threading import Thread
def listen(conn):
while True:
received = conn.recv(1024).decode()
print("Message received: " + received)
def send(conn):
while True:
to_send = input("Input message to send: ").encode()
conn.sendall(to_send)
host = "127.0.0.1"
port = 12345
sock = socket.socket()
sock.connect((host, port))
Thread(target=listen, args=[sock]).start()
Thread(target=send, args=[sock]).start()
This program is for Python 3. Python 2 is very similar, except print() works differently, and you don't need to encode() and decode() everything being sent through a socket.
The listen and send functions are run in parallel, so that as soon as data arrives, it is printed, but you can also send data at any time. Practically, you would probably want to make some changes so that the data isn't just printed over the input prompt. However, this would be hard just in a command line application.
Research queues for control over data passing between threads.
Let me know if you have any more questions.

Python - Use an Unix socket which already exists

I run a Python program on an embedded system. This system runs a server that creates UNIX sockets.
def com(request):
my_socket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
my_socket.connect("/var/run/.json-cgi")
my_socket.send(request)
reply = my_socket.recv(4096)
return repr(reply)
/var/run/.json-cgi is the socket created by the server. I send a http request that I wrote before and I passed in the arg of the function.
When I print the reply, I just get ''.
I wonder if my program creates a new socket that has the same path than the server's one or if it uses the socket of the server.
I want to use the socket created by the server.
Did I forget something ?

How to feed information to a Python daemon?

I have a Python daemon running on a Linux system.
I would like to feed information such as "Bob", "Alice", etc. and have the daemon print "Hello Bob." and "Hello Alice" to a file.
This has to be asynchronous. The Python daemon has to wait for information and print it whenever it receives something.
What would be the best way to achieve this?
I was thinking about a named pipe or the Queue library but there could be better solutions.

Here is how you can do it with a fifo:
# receiver.py
import os
import sys
import atexit
# Set up the FIFO
thefifo = 'comms.fifo'
os.mkfifo(thefifo)
# Make sure to clean up after ourselves
def cleanup():
os.remove(thefifo)
atexit.register(cleanup)
# Go into reading loop
while True:
with open(thefifo, 'r') as fifo:
for line in fifo:
print "Hello", line.strip()
You can use it like this from a shell session
$ python receiver.py &
$ echo "Alice" >> comms.fifo
Hello Alice
$ echo "Bob" >> comms.fifo
Hello Bob

There are several options
1) If the daemon should accept messages from other systems, make the daemon an RPC server - Use xmlrpc/jsonrpc.
2) If it is all local, you can use either TCP sockets or Named PIPEs.
3) If there will be a huge set of clients connecting concurrently, you can use select.epoll.

python has a built-in rpc library (using xml for data encoding). the documentation is well written; there is a complete example there:
https://docs.python.org/2.7/library/xmlrpclib.html
(python 2.7) or
https://docs.python.org/3.3/library/xmlrpc.server.html#module-xmlrpc.server
(python 3.3)
that may be worth considering.

Everyone mentioned FIFO-s (that's named pipes in Linux terminology) and XML-RPC, but if you learning these things right now, you have to check TCP/UDP/Unix sockets as well, since they are platform independent (at least, TCP/UDP sockets are). You can check this tutorial for a working example or the Python documentation if you want to go deper in this direction. It's also useful since most of the modern communication platforms (XML-RPC, SOAP, REST) uses these basic things.

There are a few mechanisms you could use, but everything boils down to using IPC (inter-process communication).
Now, the actual mechanism you will use depends on the details of what you can achieve, a good solution though would be to use something like zmq.
Check the following example on pub/sub on zmq
http://learning-0mq-with-pyzmq.readthedocs.org/en/latest/pyzmq/patterns/pubsub.html
also this
http://learning-0mq-with-pyzmq.readthedocs.org/en/latest/pyzmq/multisocket/zmqpoller.html
for the non-blocking way.

I'm not good in python so I would like to share
**Universal Inter process communcation **
nc a.k.a netcat is a server client model program which allow to send data such as text,files over network.
Advantages of nc
Very easy to use
IPC even between different programming langauges
Inbuilt on most linux OS
Example
On deamon
nc -l 1234 > output.txt
From other program or shell/terminal/script
echo HELLO | nc 127.0.0.1 1234
nc can be python by using the system command calling function ( may be os.system ) and read the stdout.

Why not use signals?
I am not a python programmer but presumably you can register a signal handler within your daemon and then signal it from the terminal. Just use SIGUSR or SIGHUP or similar.
This is the usual method you use to rotate logfiles or similar.

One solution could be to use the asynchat library which simplify calls between a server and a client.
Here is an example you could use (adapted from this site)
In deamon.py, a ChatServer object is created. Each time a connection is done, a ChatHandler object is created, inherited from asynchat.async_chat. This object collects data and fills it in self.buffer.
When a special string call the terminator is encountered, data is supposed to be complete and method found_terminator is called. It is in this method that you write your own code.
In sender.py, you create a ChatClient object, inherited from asynchat.async_chat, setup the connection in the constructor, define the terminator (in case the server answers !) and call the push method to send your data. You must append your terminator string to your data for the server to know when it can stop reading data.
daemon.py :
import asynchat
import asyncore
import socket
# Terminator string can be changed here
TERMINATOR = '\n'
class ChatHandler(asynchat.async_chat):
def __init__(self, sock):
asynchat.async_chat.__init__(self, sock=sock)
self.set_terminator(TERMINATOR)
self.buffer = []
def collect_incoming_data(self, data):
self.buffer.append(data)
def found_terminator(self):
msg = ''.join(self.buffer)
# Change here what the daemon is supposed to do when a message is retrieved
print 'Hello', msg
self.buffer = []
class ChatServer(asyncore.dispatcher):
def __init__(self, host, port):
asyncore.dispatcher.__init__(self)
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
self.bind((host, port))
self.listen(5)
def handle_accept(self):
pair = self.accept()
if pair is not None:
sock, addr = pair
print 'Incoming connection from %s' % repr(addr)
handler = ChatHandler(sock)
server = ChatServer('localhost', 5050)
print 'Serving on localhost:5050'
asyncore.loop()
sender.py :
import asynchat
import asyncore
import socket
import threading
# Terminator string can be changed here
TERMINATOR = '\n'
class ChatClient(asynchat.async_chat):
def __init__(self, host, port):
asynchat.async_chat.__init__(self)
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
self.connect((host, port))
self.set_terminator(TERMINATOR)
self.buffer = []
def collect_incoming_data(self, data):
pass
def found_terminator(self):
pass
client = ChatClient('localhost', 5050)
# Data sent from here
client.push("Bob" + TERMINATOR)
client.push("Alice" + TERMINATOR)

How to achieve tcpflow functionality (follow tcp stream) purely within python

I am writing a tool in python (platform is linux), one of the tasks is to capture a live tcp stream and to
apply a function to each line. Currently I'm using
import subprocess
proc = subprocess.Popen(['sudo','tcpflow', '-C', '-i', interface, '-p', 'src', 'host', ip],stdout=subprocess.PIPE)
for line in iter(proc.stdout.readline,''):
do_something(line)
This works quite well (with the appropriate entry in /etc/sudoers), but I would like to avoid calling an external program.
So far I have looked into the following possibilities:
flowgrep: a python tool which looks just like what I need, BUT: it uses pynids
internally, which is 7 years old and seems pretty much abandoned. There is no pynids package
for my gentoo system and it ships with a patched version of libnids
which I couldn't compile without further tweaking.
scapy: this is a package manipulation program/library for python,
I'm not sure if tcp stream
reassembly is supported.
pypcap or pylibpcap as wrappers for libpcap. Again, libpcap is for packet
capturing, where I need stream reassembly which is not possible according
to this question.
Before I dive deeper into any of these libraries I would like to know if maybe someone
has a working code snippet (this seems like a rather common problem). I'm also grateful if
someone can give advice about the right way to go.
Thanks

Jon Oberheide has led efforts to maintain pynids, which is fairly up to date at:
http://jon.oberheide.org/pynids/
So, this might permit you to further explore flowgrep. Pynids itself handles stream reconstruction rather elegantly.See http://monkey.org/~jose/presentations/pysniff04.d/ for some good examples.

Just as a follow-up: I abandoned the idea to monitor the stream on the tcp layer. Instead I wrote a proxy in python and let the connection I want to monitor (a http session) connect through this proxy. The result is more stable and does not need root privileges to run. This solution depends on pymiproxy.
This goes into a standalone program, e.g. helper_proxy.py
from multiprocessing.connection import Listener
import StringIO
from httplib import HTTPResponse
import threading
import time
from miproxy.proxy import RequestInterceptorPlugin, ResponseInterceptorPlugin, AsyncMitmProxy
class FakeSocket(StringIO.StringIO):
def makefile(self, *args, **kw):
return self
class Interceptor(RequestInterceptorPlugin, ResponseInterceptorPlugin):
conn = None
def do_request(self, data):
# do whatever you need to sent data here, I'm only interested in responses
return data
def do_response(self, data):
if Interceptor.conn: # if the listener is connected, send the response to it
response = HTTPResponse(FakeSocket(data))
response.begin()
Interceptor.conn.send(response.read())
return data
def main():
proxy = AsyncMitmProxy()
proxy.register_interceptor(Interceptor)
ProxyThread = threading.Thread(target=proxy.serve_forever)
ProxyThread.daemon=True
ProxyThread.start()
print "Proxy started."
address = ('localhost', 6000) # family is deduced to be 'AF_INET'
listener = Listener(address, authkey='some_secret_password')
while True:
Interceptor.conn = listener.accept()
print "Accepted Connection from", listener.last_accepted
try:
Interceptor.conn.recv()
except: time.sleep(1)
finally:
Interceptor.conn.close()
if __name__ == '__main__':
main()
Start with python helper_proxy.py. This will create a proxy listening for http connections on port 8080 and listening for another python program on port 6000. Once the other python program has connected on that port, the helper proxy will send all http replies to it. This way the helper proxy can continue to run, keeping up the http connection, and the listener can be restarted for debugging.
Here is how the listener works, e.g. listener.py:
from multiprocessing.connection import Client
def main():
address = ('localhost', 6000)
conn = Client(address, authkey='some_secret_password')
while True:
print conn.recv()
if __name__ == '__main__':
main()
This will just print all the replies. Now point your browser to the proxy running on port 8080 and establish the http connection you want to monitor.