Develop Reference

How do I gracefully close a socket with a persistent HTTP connection?

I'm writing a very simple client in Python that fetches an HTML page from the WWW. This is the code I've come up with so far:
import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect(("www.mywebsite.com", 80))
sock.send(b"GET / HTTP/1.1\r\nHost:www.mywebsite.com\r\n\r\n")
while True:
chunk = sock.recv(1024) # (1)
if len(chunk) == 0:
break
print(chunk)
sock.close()
The problem is: being an HTTP/1.1 connection persistent by default, the code gets stuck in # (1) waiting for more data from the server once the transmission is over.
I know I can solve this by a) adding the Connection: close request header, or by b) setting a timeout to the socket. A non-blocking socket here would not help, as the select() syscall would still hang (unless I set a timeout on it, but that's just another form of case b)).
So is there another way to do it, while keeping the connection persistent?

As has already been said in the comments, there's a lot to consider if you're trying to write an all-singing, all-dancing HTTP processor. However, if you're just practising with sockets then consider this.
Let's assume that you know how the response will end. For example, if we do essentially what you're doing in your code to the main Google page, we know that the response will end with '\r\n\r\n'. So, what we can do is just read 1 byte at a time and look out for that terminating sequence.
This code will NOT give you the full Google main page because, as you will see, the response is chunked - and that's a whole new ball game.
Having said all of that, you may find this instructive:
import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
sock.connect(('www.google.com', 80))
sock.send(b'GET / HTTP/1.1\r\nHost:www.google.com\r\n\r\n')
end = [b'\r', b'\n', b'\r', b'\n']
d = []
while d[-len(end):] != end:
d.append(sock.recv(1))
print(''.join(b.decode() for b in d))
finally:
sock.close()

Reliable UDP implementation using sequence numbers, deadlocking

I am trying to implement a reliable UDP messaging scheme between a single client and a server.
In my current code I can send incrementing numbers by 1 to the server if he only uses the receive command. if the server tries replying with the received data using the send command, it seems to work for 1-3 messages back and forth, and then I enter a deadlock. I do not understand where the source of the deadlock comes from. Below is my implementation of send and receive.
Both the client and server start with their self.seqnumber set to 0 and both sockets are set to timeout after 1 second. Both the client and server share these methods as they belong to a class both the client and server import.
def sendCommand(self, command):
self.s.sendto((self.seqnumber).to_bytes(8, "little") + command, self.address)
try:
data, self.address = self.s.recvfrom(1200)
if int.from_bytes(data[:8], 'little') == self.seqnumber:
self.seqnumber += 1
return 0
else:
return self.sendCommand(command)
except:
return self.sendCommand(command)
def getCommand(self):
while(1):
try:
data, self.address = self.s.recvfrom(1200)
if int.from_bytes(data[:8], 'little') == self.seqnumber:
self.s.sendto(data, self.address)
self.seqnumber += 1
break
elif int.from_bytes(data[:8], 'little') < self.seqnumber:
self.s.sendto(data, self.address)
else:
continue
except:
continue
return data[8:]
The code running on the server (commInf is the class in which the get and send command are defined):
while (1):
command = self.commInf.getCommand()
print(command.decode())
self.commInf.sendCommand(command)
and the code running on the client:
for i in range(100):
self.commInf.sendCommand(f"{i}".encode())
command = self.commInf.getCommand()
print(command.decode())
I expect the output to allow me to reliably send messages and return them using the sendCommand(with the received data) since the returned data from getCommand does not include its sequence number and is just the raw data.

Python + ZMQ: Operation cannot be accomplished in current state

I am trying to get a python program to communicate with another python program via zeromq by using the request-reply pattern. The client program should send a request to the server program which replies.
I have two servers such that when one server fails the other takes over. Communication works perfect when the first server works, however, when the first server fails and when I make a request to the second server, I see the error:
zmp.error.ZMQError: Operation cannot be accomplished in current state
Code of the server 1:
# Run the server
while True:
# Define the socket using the "Context"
sock = context.socket(zmq.REP)
sock.bind("tcp://127.0.0.1:5677")
data = sock.recv().decode("utf-8")
res = "Recvd"
sock.send(res.encode('utf-8'))
Code of the server 2:
# Run the server
while True:
# Define the socket using the "Context"
sock = context.socket(zmq.REP)
sock.bind("tcp://127.0.0.1:5877")
data = sock.recv().decode("utf-8")
res = "Recvd"
sock.send(res.encode('utf-8'))
Code of client:
# ZeroMQ Context For distributed Message amogst processes
context = zmq.Context()
sock_1 = context.socket(zmq.REQ)
sock_2 = context.socket(zmq.REQ)
sock_1.connect("tcp://127.0.0.1:5677")
sock_2.connect("tcp://127.0.0.1:5877")
try:
sock_1.send(data.encode('utf-8'), zmq.NOBLOCK)
socks_1.setsockopt(zmq.RCVTIMEO, 1000)
socks_1.setsockopt(zmq.LINGER, 0)
data = socks_1.recv().decode('utf-8') #receive data from the main node
except:
try:
#when server one fails
sock_2.send(data.encode('utf-8'), zmq.NOBLOCK)
socks_2.setsockopt(zmq.RCVTIMEO, 1000)
socks_2.setsockopt(zmq.LINGER, 0)
data = socks_2.recv().decode('utf-8')
except Exception as e:
print(str(e))
What is the problem with this approach?
How can I resolve this?

Q: How can I resolve this?A: Avoid the known risk of REQ/REP deadlocking!
While the ZeroMQ is a powerful framework, understanding its internal composition is necessary for robust and reliable distributed systems design and prototyping.
After a closer look, using a common REQ/REP Formal Communication Pattern may leave ( and does leave ) counter-parties in a mutual dead-lock: where one is expecting the other to do a step, which will be never accomplished, and there is no way to escape from the deadlocked state.
For more illustrated details and FSA-schematic diagram, see this post
Next, a fail-over system has to survive any collisions of its own components. Thus, one has to design well the distributed system state-signalling and avoid as many dependencies on element-FSA-design/stepping/blocking as possible, otherwise, the fail-safe behaviour remains just an illusion.
Always handle resources with care, do not consider components of the ZeroMQ smart-signalling/messaging as any kind of "expendable disposables", doing so might be tolerated in scholar examples, not in production system environments. You still have to pay the costs ( time, resources allocations / de-allocations / garbage-collection(s) ). As noted in comments, never let resources creation/allocation without a due control. while True: .socket(); .bind(); .send(); is brutally wrong in principle and deteriorating the rest of the design.

On server side, "receive" and "send" pair is critical. I was facing a simiar issue, while socket.send was missed.
def zmq_listen():
global counter
message = socket_.recv().decode("utf-8")
logger.info(f"[{counter}] Message: {message}")
request = json.loads(message)
request["msg_id"] = f"m{counter}"
ack = {"msg_id": request["msg_id"]}
socket_.send(json.dumps(ack).encode("utf-8"))
return request

Implement the lazy pirate pattern. Create a new socket from your context when an error is caught, before trying to send the message again.
The pretty good brute force solution is to close and reopen the REQ
socket after an error
Here is a python example.
#
# Author: Daniel Lundin <dln(at)eintr(dot)org>
#
from __future__ import print_function
import zmq
REQUEST_TIMEOUT = 2500
REQUEST_RETRIES = 3
SERVER_ENDPOINT = "tcp://localhost:5555"
context = zmq.Context(1)
print("I: Connecting to server…")
client = context.socket(zmq.REQ)
client.connect(SERVER_ENDPOINT)
poll = zmq.Poller()
poll.register(client, zmq.POLLIN)
sequence = 0
retries_left = REQUEST_RETRIES
while retries_left:
sequence += 1
request = str(sequence).encode()
print("I: Sending (%s)" % request)
client.send(request)
expect_reply = True
while expect_reply:
socks = dict(poll.poll(REQUEST_TIMEOUT))
if socks.get(client) == zmq.POLLIN:
reply = client.recv()
if not reply:
break
if int(reply) == sequence:
print("I: Server replied OK (%s)" % reply)
retries_left = REQUEST_RETRIES
expect_reply = False
else:
print("E: Malformed reply from server: %s" % reply)
else:
print("W: No response from server, retrying…")
# Socket is confused. Close and remove it.
client.setsockopt(zmq.LINGER, 0)
client.close()
poll.unregister(client)
retries_left -= 1
if retries_left == 0:
print("E: Server seems to be offline, abandoning")
break
print("I: Reconnecting and resending (%s)" % request)
# Create new connection
client = context.socket(zmq.REQ)
client.connect(SERVER_ENDPOINT)
poll.register(client, zmq.POLLIN)
client.send(request)
context.term()

I am creating a Twitch-focused IRC bot in Python, however it is getting the responses slowly. What am I doing wrong?

As the title says, although this is also the first time I have used Python to really do anything big. I'm not all that used to the language yet, so this is probably my missing something. The code is fairly short and is as followed, with username and private pass removed:
import re
import socket
import sys
import time
import string
HOST = "irc.twitch.tv"
PORT = 6667
NICK = ""
PASS = ""
CHAN = ""
RATE = (20/30) # messages per second
CHAT_MSG=re.compile(r"^:\w+!\w+#\w+\.tmi\.twitch\.tv PRIVMSG #\w+ :")
def chat(sock, msg):
sock.send("PRIVMSG #{} :{}".format(cfg.CHAN, msg))
public = socket.socket()
public.connect((HOST, PORT))
public.send("PASS {}\r\n".format(PASS).encode("utf-8"))
public.send("NICK {}\r\n".format(NICK).encode("utf-8"))
public.send("JOIN {}\r\n".format(CHAN).encode("utf-8"))
private = socket.socket()
private.connect((HOST, PORT))
private.send("PASS {}\r\n".format(PASS).encode("utf-8"))
private.send("NICK {}\r\n".format(NICK).encode("utf-8"))
private.send("CAP REQ :twitch.tv/tags twitch.tv/commands {}\r\n".format(CHAN).encode("utf-8"))
while True:
channelResponse = public.recv(1024).decode("utf-8")
privateResponse = private.recv(1024).decode("utf-8")
if privateResponse == "PING :tmi.twitch.tv\r\n":
private.send("PONG :tmi.twitch.tv\r\n".encode("utf-8"))
else:
privateResponseUsername = re.search(r"\w+", privateResponse).group(0) # return the entire match
privateResponseMessage = CHAT_MSG.sub("", privateResponse)
print(privateResponseUsername + ": " + privateResponseMessage)
if channelResponse == "PING :tmi.twitch.tv\r\n":
public.send("PONG :tmi.twitch.tv\r\n".encode("utf-8"))
else:
username = re.search(r"\w+", channelResponse).group(0) # return the entire match
message = CHAT_MSG.sub("", channelResponse)
print(username + ": " + message)
time.sleep(1 / RATE)
One thing to mention is that I was following a basic template style, however it did not cover implementing whispers into the bot - so I'm having to guess by doing research on how to do that, and it seems to be that the most recommended way is two connections, one for public, one for private.

As you've structured your code, you can't get anything from the private socket until you've gotten something from the public socket. If IRC didn't send PING messages occasionally, this would work even worse.
The way to handle this is to use select, and give it your two sockets. Right as soon as one has stuff that can be read, select will return, and indicate which socket has bytes available for reading.
This answer has some general code. You might want to modify it to look something like:
while True:
# this will block until at least one socket is ready
ready_socks,_,_ = select.select([private, public], [], [])
if private in ready_socks:
privateResponse += private.recv()
if public in ready_socks:
channelResponse += public.recv()
# check privateResponse buffer, do stuff
# check channelResponse buffer, do stuff
There are a few other things you should keep in mind:
The network doesn't have to deliver entire IRC messages at the same time, nor does it have to deliver a single one at a time. You could get "PI", "NG :t", "mi.twitch.tv", "\r\n" as separate messages. So you should accumulate bytes in a buffer, and then when you've got at least one entire message, process it, and remove it from the buffer.
UTF-8 characters can span multiple bytes, and might be split up by the network. Don't decode UTF-8 until you're sure you've got an entire message to work with.

Timers and retransmissions in a UDP file transfer program (using low-level sockets)

I'm trying to implement a UDP client program that receives a file from the server by sending multiple read requests and receiving small pieces of the file in return (1400 bytes, to prevent fragmentation). Each time I send a request, I set up a select() call with a timeout of 1 second. If I receive a response within that time, the client goes on to send the request for the next piece of file. Otherwise, the client resends the request for the same piece of file.
The problem with this sequential design is that the client waits for EACH request to be responded to before it sends the next one. If there is even a small ratio of packet losses, this would increase the time required for the sending of even a moderately large file to pretty unacceptable levels.
I would like to design the client to be able to send all of the read requests without waiting for responses, while simultaneously running a timer and retransmission loop that resends the individual requests that do not receive responses. However, I'm pretty new to coding in general, and can't seem to figure out how to do that. Do I need to open up several sockets at a time and run different loops on each? Or is there a more straightforward way of doing it?
My current code is (forgive the messiness):
def read_service_loop(self):
"""Loop governing the timing, checking, and retransmission or processing of read service. """
#Increment start_position each time packet sent, send a read request packet for each new position.
#Expect to receive a read_response packet for each time read request sent.
recv_data = None
print("Sending request to server to read and receive file...")
start_position = 0
while(self.eof == False):
print("Reading from byte " + str(start_position))
num_retransmits = 0
#Loop for retransmissions of the same start position
while(num_retransmits < 60):
num_retransmits = num_retransmits + 1
self.send_read_request(start_position)
input_socket = [self.client_socket]
inputready,outputready,exceptready = select.select(input_socket,[],[], 1)
if (inputready == []):
continue
else:
recv_data = self.client_socket.recv(self.buffer_)
bit_signature = recv_data[0:4]
response_type = recv_data[4:8]
recv_payload = recv_data[8:]
if bit_signature != "\x00\x00\x00\r":
self.recv_invalid_response(recv_data, "bit_signature")
continue
else:
if response_type == "\x00\x00\x00\x02":
#Packet is valid, proceed to recv_read_response to append this bit of file received into local_filename
self.file_append = open(self.local_filename, 'r+b')
self.recv_read_response(recv_payload)
break
else:
self.recv_invalid_response(recv_data, "response_type")
continue
start_position = start_position + self.NUM_BYTES_TO_READ
if (num_retransmits >= 60):
print ("Exceeded number of retransmissions allowed. Exiting program.")
sys.exit()
return

what you want to implement is called "sliding window", like TCP does. It will be complicated, because you need to take round-trip time into account. It is still evolving, see how TCP implementations of different operating systems have different performances. Perhaps you can find some library that already implements it.
Any reason not to use TCP?