How can I make a fifo between two python processes, that allow dropping of lines if the reader is not able to handle the input?
If the reader tries to read or readline faster then the writer writes, it should block.
If the reader cannot work as fast as the writer writes, the writer should not block. Lines should not be buffered (except one line at a time) and only the last line written should be received by the reader on its next readline attempt.
Is this possible with a named fifo, or is there any other simple way for achiving this?
The following code uses a named FIFO to allow communication between two scripts.
If the reader tries to read faster than the writer, it blocks.
If the reader cannot keep up with the writer, the writer does not block.
Operations are buffer oriented. Line oriented operations are not currently implemented.
This code should be considered a proof-of-concept. The delays and buffer sizes are arbitrary.
Code
import argparse
import errno
import os
from select import select
import time
class OneFifo(object):
def __init__(self, name):
self.name = name
def __enter__(self):
if os.path.exists(self.name):
os.unlink(self.name)
os.mkfifo(self.name)
return self
def __exit__(self, exc_type, exc_value, exc_traceback):
if os.path.exists(self.name):
os.unlink(self.name)
def write(self, data):
print "Waiting for client to open FIFO..."
try:
server_file = os.open(self.name, os.O_WRONLY | os.O_NONBLOCK)
except OSError as exc:
if exc.errno == errno.ENXIO:
server_file = None
else:
raise
if server_file is not None:
print "Writing line to FIFO..."
try:
os.write(server_file, data)
print "Done."
except OSError as exc:
if exc.errno == errno.EPIPE:
pass
else:
raise
os.close(server_file)
def read_nonblocking(self):
result = None
try:
client_file = os.open(self.name, os.O_RDONLY | os.O_NONBLOCK)
except OSError as exc:
if exc.errno == errno.ENOENT:
client_file = None
else:
raise
if client_file is not None:
try:
rlist = [client_file]
wlist = []
xlist = []
rlist, wlist, xlist = select(rlist, wlist, xlist, 0.01)
if client_file in rlist:
result = os.read(client_file, 1024)
except OSError as exc:
if exc.errno == errno.EAGAIN or exc.errno == errno.EWOULDBLOCK:
result = None
else:
raise
os.close(client_file)
return result
def read(self):
try:
with open(self.name, 'r') as client_file:
result = client_file.read()
except OSError as exc:
if exc.errno == errno.ENOENT:
result = None
else:
raise
if not len(result):
result = None
return result
def parse_argument():
parser = argparse.ArgumentParser()
parser.add_argument('-c', '--client', action='store_true',
help='Set this flag for the client')
parser.add_argument('-n', '--non-blocking', action='store_true',
help='Set this flag to read without blocking')
result = parser.parse_args()
return result
if __name__ == '__main__':
args = parse_argument()
if not args.client:
with OneFifo('known_name') as one_fifo:
while True:
one_fifo.write('one line')
time.sleep(0.1)
else:
one_fifo = OneFifo('known_name')
while True:
if args.non_blocking:
result = one_fifo.read_nonblocking()
else:
result = one_fifo.read()
if result is not None:
print result
The server checks if the client has opened the FIFO. If the client has opened the FIFO, the server writes a line. Otherwise, the server continues running. I have implemented a non-blocking read because the blocking read causes a problem: If the server restarts, most of the time the client stays blocked and never recovers. With a non-blocking client, a server restart is more easily tolerated.
Output
[user#machine:~] python onefifo.py
Waiting for client to open FIFO...
Waiting for client to open FIFO...
Writing line to FIFO...
Done.
Waiting for client to open FIFO...
Writing line to FIFO...
Done.
[user#machine:~] python onefifo.py -c
one line
one line
Notes
On startup, if the server detects that the FIFO already exists, it removes it. This is the easiest way to notify clients that the server has restarted. This notification is usually ignored by the blocking version of the client.
Well, that's not actually a FIFO (queue) as far as I am aware - it's a single variable. I suppose it might be implementable if you set up a queue or pipe with a maximum size of 1, but it seems that it would work better to use a Lock on a single object in one of the processes, which the other process references via a proxy object. The reader would set it to None whenever it reads, and the writer would overwrite the contents every time it writes.
You can get those to the other processes by passing the proxy of the object, and a proxy of the lock, as an argument to all relevant processes. To get it slightly more conveniently, you can use a Manager, which provides a single object with proxy that you can pass in, which contains and provides proxies for whatever other objects (including locks) you want to put in it. This answer provides a useful example of proper use of a Manager to pass objects into a new process.
Related
I my application, i have below requests:
1. There has one thread will regularly record some logs in file. The log file will be rollovered in certain interval. for keeping the log files small.
2. There has another thread also will regularly to process these log files. ex: Move the log files to other place, parse the log's content to generate some log reports.
But, there has a condition is the second thread can not process the log file that's using to record the log. in code side, the pseudocode similars like below:
#code in second thread to process the log files
for logFile in os.listdir(logFolder):
if not file_is_open(logFile) or file_is_use(logFile):
ProcessLogFile(logFile) # move log file to other place, and generate log report....
So, how do i check is a file is already open or is used by other process?
I did some research in internet. And have some results:
try:
myfile = open(filename, "r+") # or "a+", whatever you need
except IOError:
print "Could not open file! Please close Excel!"
I tried this code, but it doesn't work, no matter i use "r+" or "a+" flag
try:
os.remove(filename) # try to remove it directly
except OSError as e:
if e.errno == errno.ENOENT: # file doesn't exist
break
This code can work, but it can not reach my request, since i don't want to delete the file to check if it is open.
An issue with trying to find out if a file is being used by another process is the possibility of a race condition. You could check a file, decide that it is not in use, then just before you open it another process (or thread) leaps in and grabs it (or even deletes it).
Ok, let's say you decide to live with that possibility and hope it does not occur. To check files in use by other processes is operating system dependant.
On Linux it is fairly easy, just iterate through the PIDs in /proc. Here is a generator that iterates over files in use for a specific PID:
def iterate_fds(pid):
dir = '/proc/'+str(pid)+'/fd'
if not os.access(dir,os.R_OK|os.X_OK): return
for fds in os.listdir(dir):
for fd in fds:
full_name = os.path.join(dir, fd)
try:
file = os.readlink(full_name)
if file == '/dev/null' or \
re.match(r'pipe:\[\d+\]',file) or \
re.match(r'socket:\[\d+\]',file):
file = None
except OSError as err:
if err.errno == 2:
file = None
else:
raise(err)
yield (fd,file)
On Windows it is not quite so straightforward, the APIs are not published. There is a sysinternals tool (handle.exe) that can be used, but I recommend the PyPi module psutil, which is portable (i.e., it runs on Linux as well, and probably on other OS):
import psutil
for proc in psutil.process_iter():
try:
# this returns the list of opened files by the current process
flist = proc.open_files()
if flist:
print(proc.pid,proc.name)
for nt in flist:
print("\t",nt.path)
# This catches a race condition where a process ends
# before we can examine its files
except psutil.NoSuchProcess as err:
print("****",err)
I like Daniel's answer, but for Windows users, I realized that it's safer and simpler to rename the file to the name it already has. That solves the problems brought up in the comments to his answer. Here's the code:
import os
f = 'C:/test.xlsx'
if os.path.exists(f):
try:
os.rename(f, f)
print 'Access on file "' + f +'" is available!'
except OSError as e:
print 'Access-error on file "' + f + '"! \n' + str(e)
You can check if a file has a handle on it using the next function (remember to pass the full path to that file):
import psutil
def has_handle(fpath):
for proc in psutil.process_iter():
try:
for item in proc.open_files():
if fpath == item.path:
return True
except Exception:
pass
return False
I know I'm late to the party but I also had this problem and I used the lsof command to solve it (which I think is new from the approaches mentioned above). With lsof we can basically check for the processes that are using this particular file.
Here is how I did it:
from subprocess import check_output,Popen, PIPE
try:
lsout=Popen(['lsof',filename],stdout=PIPE, shell=False)
check_output(["grep",filename], stdin=lsout.stdout, shell=False)
except:
#check_output will throw an exception here if it won't find any process using that file
just write your log processing code in the except part and you are good to go.
Instead on using os.remove() you may use the following workaround on Windows:
import os
file = "D:\\temp\\test.pdf"
if os.path.exists(file):
try:
os.rename(file,file+"_")
print "Access on file \"" + str(file) +"\" is available!"
os.rename(file+"_",file)
except OSError as e:
message = "Access-error on file \"" + str(file) + "\"!!! \n" + str(e)
print message
You can use inotify to watch for activity in file system. You can watch for file close events, indicating that a roll-over has happened. You should also add additional condition on file-size. Make sure you filter out file close events from the second thread.
A slightly more polished version of one of the answers from above.
from pathlib import Path
def is_file_in_use(file_path):
path = Path(file_path)
if not path.exists():
raise FileNotFoundError
try:
path.rename(path)
except PermissionError:
return True
else:
return False
On Windows, you can also directly retrieve the information by leveraging on the NTDLL/KERNEL32 Windows API. The following code returns a list of PIDs, in case the file is still opened/used by a process (including your own, if you have an open handle on the file):
import ctypes
from ctypes import wintypes
path = r"C:\temp\test.txt"
# -----------------------------------------------------------------------------
# generic strings and constants
# -----------------------------------------------------------------------------
ntdll = ctypes.WinDLL('ntdll')
kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
NTSTATUS = wintypes.LONG
INVALID_HANDLE_VALUE = wintypes.HANDLE(-1).value
FILE_READ_ATTRIBUTES = 0x80
FILE_SHARE_READ = 1
OPEN_EXISTING = 3
FILE_FLAG_BACKUP_SEMANTICS = 0x02000000
FILE_INFORMATION_CLASS = wintypes.ULONG
FileProcessIdsUsingFileInformation = 47
LPSECURITY_ATTRIBUTES = wintypes.LPVOID
ULONG_PTR = wintypes.WPARAM
# -----------------------------------------------------------------------------
# create handle on concerned file with dwDesiredAccess == FILE_READ_ATTRIBUTES
# -----------------------------------------------------------------------------
kernel32.CreateFileW.restype = wintypes.HANDLE
kernel32.CreateFileW.argtypes = (
wintypes.LPCWSTR, # In lpFileName
wintypes.DWORD, # In dwDesiredAccess
wintypes.DWORD, # In dwShareMode
LPSECURITY_ATTRIBUTES, # In_opt lpSecurityAttributes
wintypes.DWORD, # In dwCreationDisposition
wintypes.DWORD, # In dwFlagsAndAttributes
wintypes.HANDLE) # In_opt hTemplateFile
hFile = kernel32.CreateFileW(
path, FILE_READ_ATTRIBUTES, FILE_SHARE_READ, None, OPEN_EXISTING,
FILE_FLAG_BACKUP_SEMANTICS, None)
if hFile == INVALID_HANDLE_VALUE:
raise ctypes.WinError(ctypes.get_last_error())
# -----------------------------------------------------------------------------
# prepare data types for system call
# -----------------------------------------------------------------------------
class IO_STATUS_BLOCK(ctypes.Structure):
class _STATUS(ctypes.Union):
_fields_ = (('Status', NTSTATUS),
('Pointer', wintypes.LPVOID))
_anonymous_ = '_Status',
_fields_ = (('_Status', _STATUS),
('Information', ULONG_PTR))
iosb = IO_STATUS_BLOCK()
class FILE_PROCESS_IDS_USING_FILE_INFORMATION(ctypes.Structure):
_fields_ = (('NumberOfProcessIdsInList', wintypes.LARGE_INTEGER),
('ProcessIdList', wintypes.LARGE_INTEGER * 64))
info = FILE_PROCESS_IDS_USING_FILE_INFORMATION()
PIO_STATUS_BLOCK = ctypes.POINTER(IO_STATUS_BLOCK)
ntdll.NtQueryInformationFile.restype = NTSTATUS
ntdll.NtQueryInformationFile.argtypes = (
wintypes.HANDLE, # In FileHandle
PIO_STATUS_BLOCK, # Out IoStatusBlock
wintypes.LPVOID, # Out FileInformation
wintypes.ULONG, # In Length
FILE_INFORMATION_CLASS) # In FileInformationClass
# -----------------------------------------------------------------------------
# system call to retrieve list of PIDs currently using the file
# -----------------------------------------------------------------------------
status = ntdll.NtQueryInformationFile(hFile, ctypes.byref(iosb),
ctypes.byref(info),
ctypes.sizeof(info),
FileProcessIdsUsingFileInformation)
pidList = info.ProcessIdList[0:info.NumberOfProcessIdsInList]
print(pidList)
I provided one solution. please see the following code.
def isFileinUsed(ifile):
widlcard = "/proc/*/fd/*"
lfds = glob.glob(widlcard)
for fds in lfds:
try:
file = os.readlink(fds)
if file == ifile:
return True
except OSError as err:
if err.errno == 2:
file = None
else:
raise(err)
return False
You can use this function to check if a file is in used.
Note:
This solution only can be used for Linux system.
I have a Python program that reads some parameters from an Arduino and stores it in a database. The serial port is set up and used like this:
ser = serial.Serial(port=port, baudrate=9600)
ser.write('*')
while 1 :
ser.write('*')
out = ''
# Let's wait one second before reading output (let's give device time to answer).
time.sleep(1)
while ser.inWaiting() > 0:
out += ser.read(1)
if out != '':
etc ... handling data
(The Arduino is set up so when it receives a star, it sends back a data string.) I would like to rewrite this as a daemon, so I am using the python-daemon library. In the init-part, I just define the port name, and then:
def run(self):
self.ser = serial.Serial(port=self.port,baudrate=9600)
while True:
self.ser.write('*')
out = ''
# Let's wait one second before reading output (give device time to answer).
time.sleep(1)
while self.ser.inWaiting() > 0:
out += self.ser.read(1)
if out != '':
etc ...
Everything is equal, except that I am now doing the serial handling within an App-object. The first version runs fine, when I try to run the latter, I get
File "storedaemon.py", line 89, in run
while self.ser.inWaiting() > 0:
File "/usr/lib/python2.7/dist-packages/serial/serialposix.py", line 435, in inWaiting
s = fcntl.ioctl(self.fd, TIOCINQ, TIOCM_zero_str)
IOError: [Errno 9] Bad file descriptor
I am not able to see what has changed - except that I have tossed the code inside a new object. I have tried both to do the initialisation in init and in run, but I end up with the same result.
(The complete scripts are available at hhv3.sickel.net/b/storedata.py and hhv3.sickel.net/b/storedaemon.py.)
During the daemonization of your app, all file handlers are closed except stdin, stderr and stdout. This includes the connection to /dev/log, which then fails with the fd error (so it looks like this has nothing to do with the serial fd, but instead with the handler's socket).
You need either to add this FD to the exclusion list:
class App():
def __init__(self):
...
self.files_preserve = [handler.socket]
...
Or alternatively, set up the handler after the daemon process forked:
class App():
def run(self):
handler = logging.handlers.SysLogHandler(address = '/dev/log')
my_logger.addHandler(handler)
my_logger.debug(appname+': Starting up storedata')
...
Both version ran fine during my tests.
I create a FIFO, and periodically open it in read-only and non-blockingly mode from a.py:
os.mkfifo(cs_cmd_fifo_file, 0777)
io = os.open(fifo, os.O_RDONLY | os.O_NONBLOCK)
buffer = os.read(io, BUFFER_SIZE)
From b.py, open the fifo for writing:
out = open(fifo, 'w')
out.write('sth')
Then a.py will raise an error:
buffer = os.read(io, BUFFER_SIZE)
OSError: [Errno 11] Resource temporarily unavailable
Anyone know what's wrong?
According to the manpage of read(2):
EAGAIN or EWOULDBLOCK
The file descriptor fd refers to a socket and has been marked
nonblocking (O_NONBLOCK), and the read would block.
POSIX.1-2001 allows either error to be returned for this case,
and does not require these constants to have the same value, so
a portable application should check for both possibilities.
So what you're getting is that there is no data available for reading. It is safe to handle the error like this:
try:
buffer = os.read(io, BUFFER_SIZE)
except OSError as err:
if err.errno == errno.EAGAIN or err.errno == errno.EWOULDBLOCK:
buffer = None
else:
raise # something else has happened -- better reraise
if buffer is None:
# nothing was received -- do something else
else:
# buffer contains some received data -- do something with it
Make sure you have the errno module imported: import errno.
out = open(fifo, 'w')
Who will close it for you?
Replace your open+write by this:
with open(fifo, 'w') as fp:
fp.write('sth')
UPD:
Ok, than just make this:
out = os.open(fifo, os.O_NONBLOCK | os.O_WRONLY)
os.write(out, 'tetet')
filename = 'tempfile'
def tail(filename):
fd = open(filename)
while True:
line = fd.readline()
if not line:
continue
else:
if filename != 'uh':
yield line
else:
print 'Returning f to close the file'
yield fd
try:
genObj = tail(filename)
valfromgen= genObj.next()
while valfromgen:
print valfromgen
valfromgen= genObj.next()
except:
traceback.print_exc()
try:
fd_Got_Back = genObj.send('uh')
fd_Got_Back.close()
except:
traceback.print_exc()
Intention of the code: I have opened the file in the generator function only and not outside it, but, I want to close that file outside the generator function by using 'send' probably.
What I am trying to do: Replicating tail -f from unix.
How I am trying to do:
Open a tempfile in read mode.
If the tempfile has 1 new line written in it (which I'll keep on writing manually and saving tempfile using notepad), yield the newly written line.
Problem:
The problem is that I'm trying to check how I can close the opened tempfile from this python code if I press Ctrl + C (i.e. SIGTERM) when this Python code runs in command prompt. In order to emulate this, I have opened the tempfile in the tail function, and whenever there is an exception (which will be raised by the system when I press Ctrl + C), the control should go in the 1st except. Then, from here, I'm trying to send a value uh to the generator function tail, so that it should yield the file descriptor of the opened file which I can use to close the opened tempfile.
PS: I expect a solution where I have opened the file in the generator function only and not outside it.
I think You're misunderstanding how "send" works. Send simply causes a generator to yield that value on its next iteration. It does not change the value of the original parameters. You can then use that yielded value for some purpose. So you could make your code:
filename = 'tempfile'
def tail(filename):
fd = open(filename)
while True:
line = fd.readline()
if not line:
continue
else:
x = (yield line)
if (x == 'uh'):
print 'Returning f to close the file'
yield fd
try:
genObj = tail(filename)
valfromgen= genObj.next()
while valfromgen:
print valfromgen
valfromgen= genObj.next()
except:
traceback.print_exc()
try:
genObj.send('uh').close()
except:
traceback.print_exc()
I have figured out the problem where I was stuck and I have come up with this solution:-
When I press Ctrl + C (on Windows), the KeyboardInterrupt actually happens in fd.readline(). So, I just placed a try...except there, so that the generator function yields the file descriptor whenever Ctrl + C is hit. If there is no KeyBoardInterrupt , then, just print a newly read line from tempfile
This file descriptor is checked using isinstance() in the main body, and if it is found to be a file, then, I'm closing the file as well as the generator
PS: (this KeyboardInterrupt might vary on Linux..probably SigTerm will be raised, but, please check. So, in order to make the code generic, just remove KeyBoard Interrupt and use just normal except)
import sys, traceback
filename = 'tempfile'
def tail(filename):
fd = open(filename)
while True:
try:
line = fd.readline()
except KeyboardInterrupt:
print 'keyboard interrupt here'
yield fd
if not line:
continue
else:
yield line
try:
genObj = tail(filename)
valfromgen= genObj.next()
while valfromgen:
if isinstance(valfromgen, file):
print 'Closing this file now as `tail` yielded a file descriptor'
valfromgen.close()
genObj.close()
break
print 'Yielded line: ', valfromgen
valfromgen= genObj.next()
print 'Just in order to check that things are in order, the following line will raise StopIteration. If it raises, it means we are good.'
print genObj.next()
except:
traceback.print_exc()
Short version (if you can answer the short version it does the job for me, the rest is mainly for the benefit of other people with a similar task):
In python in Windows, I want to create 2 file objects, attached to the same file (it doesn't have to be an actual file on the hard-drive), one for reading and one for writing, such that if the reading end tries to read it will never get EOF (it will just block until something is written). I think in linux os.mkfifo() would do the job, but in Windows it doesn't exist. What can be done? (I must use file-objects).
Some extra details:
I have a python module (not written by me) that plays a certain game through stdin and stdout (using raw_input() and print). I also have a Windows executable playing the same game, through stdin and stdout as well. I want to make them play one against the other, and log all their communication.
Here's the code I can write (the get_fifo() function is not implemented, because that's what I don't know to do it Windows):
class Pusher(Thread):
def __init__(self, source, dest, p1, name):
Thread.__init__(self)
self.source = source
self.dest = dest
self.name = name
self.p1 = p1
def run(self):
while (self.p1.poll()==None) and\
(not self.source.closed) and (not self.source.closed):
line = self.source.readline()
logging.info('%s: %s' % (self.name, line[:-1]))
self.dest.write(line)
self.dest.flush()
exe_to_pythonmodule_reader, exe_to_pythonmodule_writer =\
get_fifo()
pythonmodule_to_exe_reader, pythonmodule_to_exe_writer =\
get_fifo()
p1 = subprocess.Popen(exe, shell=False, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
old_stdin = sys.stdin
old_stdout = sys.stdout
sys.stdin = exe_to_pythonmodule_reader
sys.stdout = pythonmodule_to_exe_writer
push1 = Pusher(p1.stdout, exe_to_pythonmodule_writer, p1, '1')
push2 = Pusher(pythonmodule_to_exe_reader, p1.stdin, p1, '2')
push1.start()
push2.start()
ret = pythonmodule.play()
sys.stdin = old_stdin
sys.stdout = old_stdout
Following the two answers above, I accidentally bumped into the answer. os.pipe() does the job. Thank you for your answers.
I'm posting the complete code in case someone else is looking for this:
import subprocess
from threading import Thread
import time
import sys
import logging
import tempfile
import os
import game_playing_module
class Pusher(Thread):
def __init__(self, source, dest, proc, name):
Thread.__init__(self)
self.source = source
self.dest = dest
self.name = name
self.proc = proc
def run(self):
while (self.proc.poll()==None) and\
(not self.source.closed) and (not self.dest.closed):
line = self.source.readline()
logging.info('%s: %s' % (self.name, line[:-1]))
self.dest.write(line)
self.dest.flush()
def get_reader_writer():
fd_read, fd_write = os.pipe()
return os.fdopen(fd_read, 'r'), os.fdopen(fd_write, 'w')
def connect(exe):
logging.basicConfig(level=logging.DEBUG,\
format='%(message)s',\
filename=LOG_FILE_NAME,
filemode='w')
program_to_grader_reader, program_to_grader_writer =\
get_reader_writer()
grader_to_program_reader, grader_to_program_writer =\
get_reader_writer()
p1 = subprocess.Popen(exe, shell=False, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
old_stdin = sys.stdin
old_stdout = sys.stdout
sys.stdin = program_to_grader_reader
sys.stdout = grader_to_program_writer
push1 = Pusher(p1.stdout, program_to_grader_writer, p1, '1')
push2 = Pusher(grader_to_program_reader, p1.stdin, p1, '2')
push1.start()
push2.start()
game_playing_module.play()
sys.stdin = old_stdin
sys.stdout = old_stdout
fil = file(LOG_FILE, 'r')
data = fil.read()
fil.close()
return data
if __name__=='__main__':
if len(sys.argv) != 2:
print 'Usage: connect.py exe'
print sys.argv
exit()
print sys.argv
print connect(sys.argv[1])
On Windows, you are looking at (Named or Anonymous) Pipes.
A pipe is a section of shared memory that processes use for communication. The process that creates a pipe is the pipe server. A process that connects to a pipe is a pipe client. One process writes information to the pipe, then the other process reads the information from the pipe.
To work with Windows Pipes, you can use Python for Windows extensions (pywin32), or the Ctypes module. A special utility module, win32pipe, provides an interface to the win32 pipe API's. It includes implementations of the popen[234]() convenience functions.
See how-to-use-win32-apis-with-python and similar SO questions (not specific to Pipes, but points to useful info).
For a cross-platform solution, I'd recommend building the file-like object on top of a socket on localhost (127.0.0.1) -- that's what IDLE does by default to solve a problem that's quite similar to yours.
os.pipe() returns an anonymous pipe, or a named pipe on Windows, which is very lightweight and efficient.
TCP sockets (as suggested by user1495323) are more heavyweight: you can see them with netstat for example, and each one requires a port number, and the number of available ports is limited to 64k per peer (e.g. 64k from localhost to localhost).
On the other hand, named pipes (on Windows) are limited because:
You can't use select() for nonblocking I/O on Windows, because they're not sockets.
There's no apparent way to read() with a timeout, and
Even making them non-blocking is difficult.
And sockets can be wrapped in Python-compatible filehandles using makefile(), which allows them to be used to redirect stdout or stderr. This makes this an attractive option for some use cases, such as sending stdout from one thread to another.
A socket can be constructed with an automatically-assigned port number like this (based on the excellent Python socket HOWTO):
with closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as input_socket:
# Avoid socket exhaustion by setting SO_REUSEADDR <https://stackoverflow.com/a/12362623/648162>:
input_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
# localhost doesn't work if the definition is missing from the hosts file,
# and 127.0.0.1 only works with IPv4 loopback, but socket.gethostname()
# should always work:
input_socket.bind((socket.gethostname(), 0))
random_port_number = input_socket.getsockname()[1]
input_socket.listen(1)
# Do something with input_socket, for example pass it to another thread.
output_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# close() should not strictly be necessary here, but since connect() could fail, it avoids leaking fds
# in that case. "If a file descriptor is given, it is closed when the returned I/O object is closed".
with output_socket:
output_socket.connect((socket.gethostname(), random_port_number))
The user of input_socket (e.g. another thread) can then do:
with input_socket:
while True:
readables, _, _ = select.select([input_socket], [], [input_socket], 1.0)
if len(readables) > 0:
input_conn, addr = self.input_socket.accept()
break
with input_conn:
while True:
readables, _, errored = select.select([input_conn], [], [input_conn], 1.0)
if len(errored) > 0:
print("connection errored, stopping")
break
if len(readables) > 0:
read_data = input_conn.recv(1024)
if len(read_data) == 0:
print("connection closed, stopping")
break
else:
print(f"read data: {read_data!r}")