Develop Reference

How do I configure docker ports?

I have a C# program which reads a TCP stream from port 19777. It runs under docker, and has this in its yml file:
ports:
- 19777:19777`
Now I want a python program to intercept this stream, process, it and then send it to the C# program. I have tried a (run external to docker) python socket program which writes to a different port, 19888.
host = '0.0.0.0'
port = 19888
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((host, port))
s.listen(1)
while True:
print('\nListening for a client at',host , port)
conn, addr = s.accept()
print('\nConnected by', addr)
To test it I got a Python program to connect to this fine.
I then changed the C# and yml to use 19888, but I get:
Error starting userland proxy: listen tcp 127.0.0.1:19888: bind: address already in use
It seems to me that a solution might be to run the python server in docker but I am not sure whether this would work and how to configure the ports.
Any help is much appreciated.

Socket module (python) works but doesn't use specified port number?

I'm using the socket module from Python 3.7 (shouldn't matter, as I tried activating a different Python version from different venv's).
The problem is that I've created a TCP connection listening at port 65432, an arbitrary number that I selected for this simple demo.
server.py looks like the following:
import socket
HOST = '127.0.0.1' # Standard loopback interface address (localhost)
PORT = 65432 # Non-privileged ports are > 1024
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.bind((HOST, PORT))
s.listen()
conn, addr = s.accept()
with conn:
print('Connected by', addr)
while True:
data = conn.recv(1024)
if not data:
break
conn.sendall(data)
client.py is relatively straightforward as it makes a connection with 127.0.0.1:65432.
import socket
HOST = '127.0.0.1' # The server's hostname or IP address
PORT = 65432 # Port used by the server
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.connect((HOST, PORT))
# Send its message and then read the server's reply and prints it
s.sendall(b'Hello, world')
data = s.recv(1024)
print('Received', repr(data))
Executing server.py to open the port 65432 for listening (in first console) and then executing client.py to send a simple 'hello world' message (in a second console). This is what got printed to the first console:
Connected by ('127.0.0.1', 56051)
So far so good. Port 56051 connecting to port 65432, right? No.
I execute netstat -am (command tool utility to see state of sockets on the host machine) and found this:
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address Foreign Address (state)
tcp4 0 0 127.0.0.1.51495 *.* LISTEN
Instead of 127.0.0.1.65432 as local address, it is using port 51495 instead.
Doing another verification check, this time firing off lsof -i -n:
COMMAND PID FD TYPE DEVICE SIZE/OFF NODE NAME
Code\x20H 51214 37u IPv4 0x1af15eb424ba89f3 0t0 TCP 127.0.0.1:51495 (LISTEN)
Both verifications confirmed that port 51495 is being used instead of 65432 as specified in my server.py and client.py scripts. Any leads or tips? Many thanks in advance!

65432 is the port number of your server socket, not your client socket. As the client end is not attached with any specific port number, it will be dynamically allocated with port number, every time you run the client code.
As far as I understood, you mentioned -
Connected by ('127.0.0.1', 56051)
is shown on the first console which is your server console. so this port number is port number of client socket. not the server socket.
In the server code, you are using, s.accept(), this function returns the connection temporary id and the address of the client which made the request. same thing you are trying to print in the code.

As #ottomeister pointed out, the process name was the first giveaway. The process name should have been Python but it showed VS Code instead, which is indicative that the port 51495 is opened by the VS Code process and has nothing to do with our socket module code.
The way the context manager was setup means that the connection will be closed the moment the last line (in this case, socket.sendall()) is executed. So the server socket is not active anymore.
I run netstat after the client socket has connected, by this point the server port is closed.
When I monitor the ports status while the server port is open (before the client socket connects with it) then sure enough 65432 is what appeared. This is confirmed in netstat, lsof and also nmap. A simple print statement after the socket connection is successful will also confirmed that the server port is in fact using the specified port number, which is 65432.
Sorry for the inconvenience, and again much appreciation to Ottomeister for first pointing this out.

Multicast from tcp replay seen by wireshark but not by application

I have got a multicast packet capture I'm playing with tcpreplay:
sysctl net.ipv4.conf.all.rp_filter=0
sysctl net.ipv4.conf.eth0.rp_filter=0
tcpreplay -i eth0 --loop=100 new.pcap
I watch the traffic on eth0 with wireshark and I can see the packets I'm interested in (let's say 224.0.23.60:4937).
But the following python app cannot find the packets:
import socket
import struct
MCAST_GRP = '224.0.23.60'
MCAST_PORT = 4937
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind((MCAST_GRP, MCAST_PORT)) # use MCAST_GRP instead of '' to listen only
# to MCAST_GRP, not all groups on MCAST_PORT
mreq = struct.pack("4sl", socket.inet_aton(MCAST_GRP), socket.INADDR_ANY)
sock.setsockopt(socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, mreq)
while True:
print '#'
print sock.recv(64)
netstat -g is giving the following output:
lo 1 all-systems.mcast.net
eth0 1 224.0.23.60
Am I missing something here ?
[Edit] I should precise that the ip src in my packet capture is not in the same network subdomain (ip src: 192.168.1.10) whereas my ip is something like 146.186.197.164.

After reading carefully the documentation (http://tcpreplay.synfin.net/wiki/FAQ), it seems that tcpreplay sends the packets between the TCP/IP stack and the ethernet device driver, therefore the TCP/IP stack of the host system never sees the packets.
I ended up using a debian Os with virtual box configured with the host only adapter and use tcpreplay in that machine.

Now, it is mentioned clearly on the FAQ page.
https://tcpreplay.appneta.com/wiki/faq.html#can-i-send-packets-on-the-same-computer-running-tcpreplay
Q: Can I send packets on the same computer running tcpreplay?
Generally speaking no. When tcpreplay sends packets, it injects them
between the TCP/IP stack of the system and the device driver of the
network card. The result is the TCP/IP stack system running tcpreplay
never sees the packets.
One suggestion that has been made is using something like VMWare,
Parallels or Xen. Running tcpreplay in the virtual machine (guest)
would allow packets to be seen by the host operating system.

Python service discovery: Advertise a service across a local network

I have a "server" python script running on one of the local network machines, which waits for clients to connect, and passes them some work to do. The server and client code have both been written, and are working as expected...
The problem is, this server might be running from any machine in the local network, so I can't hard code the address in the script... I immediately wondered if I can make a machine advertise about its existence, and clients can respond to that. Is that doable in Python with the standard library? I really don't have time to download twisted or tornado and learn about them, unfortunately, so I need something simple.
I tried to think more about it, and realized I can have a single static IP machine where servers register/unregister from and clients can look for servers from there. Kind of like a torrent tracker, I think. This'll have to do if I can't do the service advertising approach easily.

An easy way to do service announcement/discovery on the local network is by broadcasting UDP packets.
Constants:
PORT = 50000
MAGIC = "fna349fn" #to make sure we don't confuse or get confused by other programs
Announcement:
from time import sleep
from socket import socket, AF_INET, SOCK_DGRAM, SOL_SOCKET, SO_BROADCAST, gethostbyname, gethostname
s = socket(AF_INET, SOCK_DGRAM) #create UDP socket
s.bind(('', 0))
s.setsockopt(SOL_SOCKET, SO_BROADCAST, 1) #this is a broadcast socket
my_ip= gethostbyname(gethostname()) #get our IP. Be careful if you have multiple network interfaces or IPs
while 1:
data = MAGIC+my_ip
s.sendto(data, ('<broadcast>', PORT))
print "sent service announcement"
sleep(5)
Discovery:
from socket import socket, AF_INET, SOCK_DGRAM
s = socket(AF_INET, SOCK_DGRAM) #create UDP socket
s.bind(('', PORT))
while 1:
data, addr = s.recvfrom(1024) #wait for a packet
if data.startswith(MAGIC):
print "got service announcement from", data[len(MAGIC):]
This code was adapted from the demo on python.org

Sharing UDP broadcast reception -- python example

I have a UDP broadcast of some data. I'm able to open the following client in python 2.6.1, under OSX 10.6.8, and it works. I can catch the data, all is well.
But: this code "consumes" the port, in that I can't open another one, the 2nd attempt to bind fails... and I must allow for more than one listener. Here's the code that opens the port:
import select, socket
port = 58083 # port msg is broadcast upon
# Create listening port
# ---------------------
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
try:
s.bind(('', port))
except:
print 'failure to bind'
s.close()
raise
s.setblocking(0)
...Since UDP is intended for broadcast to multiple clients (among other things), I assume I'm doing something wrong. I just can't figure out what.
I found an example on activestate that suggested:
s.bind(('<broadcast>',port))
...but that simply fails every time. Binding to 0.0.0.0 works, but also suffers from the "one client" problem. Binding to the local IP (e.g. 192.168.1.100) doesn't work at all. Removing the bind doesn't work at all.
Anyone?

If you need multiple processes to listen on 58083, you need to set SO_REUSEADDR on the socket before socket.bind()
import select, socket
port = 58083 # port msg is broadcast upon
# Create listening port
# ---------------------
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) # Allow other processes to
# bind to port
try:
s.bind(('0.0.0.0', port))
except:
print 'failure to bind'
s.close()
raise
s.setblocking(0)
After adding an infinite loop at the bottom, and running this twice on my linux server (once as root, and the other as unpriv user), I see:
root#tsunami# lsof | grep 58083
python 25908 root 3u IPv4 284835 0t0 UDP *:58083
python 25945 mpenning 3u IPv4 284850 0t0 UDP *:58083
root#tsunami#