I'm working on a camera endpoint for an IoT application and I want to start and stop a flask app on command to show a webpage with streaming camera data. I have all the individual pieces working but it's failing when I put them together.
If I start a flask process after starting another thread it complains that address is already in use. Here's a stripped down example:
#!/usr/bin/env python3
import os
import threading
from flask import Flask
import socket
import time
from multiprocessing import Process
app = Flask(__name__)
def main():
start_thread(udp_listener)
start_process(runflask)
while True:
time.sleep(1)
def start_thread(function, arguments=()):
th = threading.Thread(target=function, args=arguments)
th.daemon = True
th.start()
def start_process(function):
server = Process(target=function)
server.start()
while True:
time.sleep(60)
server.terminate()
server.join()
def udp_listener():
while True:
with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as s:
s.bind(('', 9000))
print('udp listening on port 9000')
while True:
data, server = s.recvfrom(4069)
print('received:')
def runflask():
app.run(host='0.0.0.0', port=8001, debug=True)
if __name__ == "__main__":
main()
Comment out start_thread() and just run start_process() and it's ok.
Comment out start_process() and just run start_thread() and it's ok.
Run both and get an address already in use error even though they're listening on different ports.
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "./webcam_test.py", line 35, in udp_listener
s.bind(('', 9000))
OSError: [Errno 98] Address already in use
I want to run the flask app in a way that will let me start and stop it on command. Should I be doing it a different way?
I was able to resolve this by using a ServerThread as specified here: How to stop flask application without using ctrl-c
It's important to note that calling this ServerThread automatically blocks so if you want an interruptible thread, you need to instantiate this as the child of another thread. My design already spawned threads for each camera mode so now I have a thread calling a thread calling a thread :).
I still don't know why flask, when run as a process, would complain about unrelated listen sockets but now I have a workaround.
Don't how relevant this still is, but:
From a quick glance I don't see any place where the socket you are binding (s.bind(('', 9000))) is being closed. Since it's on a loop it tries to bind a socket that's in use again, hence the error? It should be stopped before binded again:
s.shutdown(1)
s.close()
If this is the case, the first solution I suggest is to close the socket properly, and not have it started in the loop (why do you bind the socket in loop in the first place?)
If this alone does not help, allow the socket to reuse the address:
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
Python: Binding Socket: "Address already in use"
Related
Hypothesis:
thread....start() blocks until start completes.
Question:
Is hypothesis True or False?
Start http web server then open browser has the following code.
import sys
import time
import threading
import webbrowser
from http.server import HTTPServer, SimpleHTTPRequestHandler
ip = "127.0.0.1"
port = 3600
url = f"http://{ip}:{port}"
def start_server():
server_address = (ip, port)
httpd = HTTPServer(server_address, SimpleHTTPRequestHandler)
httpd.serve_forever()
threading.Thread(target=start_server).start()
webbrowser.open_new(url)
while True: # make a blocker to prevent the application finish of execute:
try:
time.sleep(1)
except KeyboardInterrupt:
sys.exit(0)
This works fine. However, the following also works.
import sys
import time
import threading
import webbrowser
from http.server import HTTPServer, SimpleHTTPRequestHandler
ip = "127.0.0.1"
port = 3600
url = f"http://{ip}:{port}"
def start_server():
server_address = (ip, port)
httpd = HTTPServer(server_address, SimpleHTTPRequestHandler)
httpd.serve_forever()
threading.Thread(target=start_server).start()
webbrowser.open_new(url)
Hypothesis:
thread....start() actually blocks until start completes. So,webbrowser.open_new(url) does not execute until start completes. Thus making the following unnecessary.
while True: # make a blocker to prevent the application finish of execute:
try:
time.sleep(1)
except KeyboardInterrupt:
sys.exit(0)
I have not been able to prove or disprove the Hypothesis after extensive searching.
There is no blocking when calling Thread.start() in the way you suggest.
The call is blocking in the sense that a call is placed that initalizes the new-thread internal state, and a system call is made to start the actual OS Thread - but that should take less than 1ms. The function that is the target of the thread is only called on the new thread, and the main thread will continue to run, regardless of what takes place inside that function.
If you want your program not to end, there is no need to resort to a complicated pausing loop like the one you setup - just place a call to threading.join() instead. This will block until all other threads end running, and only them the threading calling join() will proceed.
I am working with PyZMQ and I have what seems to be a rather peculiar issue. I have two classes that are wrapping sockets for communication, MZLSubscriber and MZLRequester. There is a class that contains both of them, MZLLink. For each of these, I also have tests MZLSubscriberTest, MZLRequesterTest, and MZLinkTest. The tests for the subscriber and requester work as they should, but MZLinkTest does not receive any subscriber messages.
Below is what seems to be the relative code, which are the constructors for the 3 classes as well as run() for MZLSubscriber and the tests for MZLink and MZLSubscriber.
MZLink Constructor:
# Host information
self.host = host
self.requestPort = requestPort
self.subscriberPort = subscriberPort
# Set up zmq context
self.zmq_context = zmq.Context()
# Set up subscriber and replier
self.subscriber = MZLSubscriber(self.zmq_context, self.host, self.subscriberPort)
self.requester = MZLRequester(self.zmq_context, self.host, self.requestPort)
# Start subscriber
self.subscriber.start()
MZLink Test:
# Constants
HOST = "localhost"
REQ_PORT = 5555
SUB_PORT = 5556
# Create Link
link = MIDASZMQLink(HOST, REQ_PORT, SUB_PORT)
link.close()
MZLRequester Constructor:
# Initialize class member variables
self.zmq_context = zmq_context
self.host = host
self.port = port
# Set up reply socket
self.socket = self.zmq_context.socket(zmq.REQ)
# Connect socket
self.socket.connect("tcp://{0}:{1}".format(self.host, self.port))
MZLSubscriber Constructor:
# Initialize parent process
Process.__init__(self)
# Store zmq context and connection host/port
self.zmq_context = zmq_context
self.host = host
self.port = port
# Sockets. Don't set them up here because sockets are not thread safe
self.socket = None
# Queue to store data in
# TODO: Make queue not overflow if events come in too quickly
self.queue = Queue()
MZLSubscriber.run():
# Parent call
Process.run(self)
# Set up subscriber socket in this thread
self.socket = self.zmq_context.socket(zmq.SUB)
self.socket.setsockopt_string(zmq.SUBSCRIBE, unicode())
# Connect socket
self.socket.connect("tcp://{0}:{1}".format(self.host, self.port))
# While the thread is alive, poll for data to put into queue
# Calling MZLSubscriber.stop() will automatically change this
while self.is_alive():
datum = self.socket.recv()
self.queue.put(datum)
# Disconnect and close socket.
#FIXME: Doesn't get here because terminate() immediately stops the process
self.socket.disconnect("tcp://{0}:{1}".format(self.host, self.port))
self.socket.close()
MZLSubscriber Test:
# Host information
HOST = "localhost"
SUBSCRIBER_PORT = "5556"
# Set up zmq context
zmq_context = zmq.Context()
# Set up subscriber
subscriber = MZLSubscriber(zmq_context, HOST, SUBSCRIBER_PORT)
# Start subscriber
subscriber.start()
# Stop and join subscriber
subscriber.close()
subscriber.join()
The subscriber thread seems to block at datum = self.socket.recv(), which makes me think it could be some issue with the socket creation. However, it does seem to work when only working with the subscriber. The requester seems to work in both cases. In addition, everything goes smoothly by just commenting out the two lines dealing with requester.
I apologize for the wall of code, but I can't even narrow what code the issue is coming from at this point. When I do, I'll remove the irrelevant code. The test code that deals with the incoming data has been removed.
As a bit of clarification, I am using Python 2.7 with PyZMQ 14.3.1.
UPDATE: It seems that running MZLSubscriber in the main thread rather than creating another Process results in the expected result, so it seems that this could be some sort of thread safety. To my knowledge, zmq contexts are thread-safe, but sockets are not. I thought this wouldn't cause an issue because I'm explicitly making sure there is a socket for each thread.
UPDATE 2: If the calls setting up the socket in MZLSubscriber are moved from run() to __init__, the socket seems to receive a small portion of the published message, but does have an error:
Process MZLSubscriber-1:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/Users/user/Repos/midas-client/client/midasclient/mzlSubscriber.py", line 45, in run
datum = self.socket.recv()
File "socket.pyx", line 628, in zmq.backend.cython.socket.Socket.recv (zmq/backend/cython/socket.c:5616)
File "socket.pyx", line 662, in zmq.backend.cython.socket.Socket.recv (zmq/backend/cython/socket.c:5436)
File "socket.pyx", line 139, in zmq.backend.cython.socket._recv_copy (zmq/backend/cython/socket.c:1771)
File "checkrc.pxd", line 21, in zmq.backend.cython.checkrc._check_rc (zmq/backend/cython/socket.c:6032)
ZMQError: Interrupted system call
I have gotten a workaround to this by creating a new zmq.Context in MZLSubscriber.run(), although I feel that this shouldn't be necessary if zmq contexts are thread-safe.
It seems that my issue was using multiple zmq contexts over different processes. While the PyZMQ documentation states that the zmq context is thread-safe, I can only assume it meant Python threads rather than processes. This is quite confusing as in C, zmq contexts are thread safe despite running in a way similar to the Python multiprocessing.Process.
The issue was solved by creating a zmq context for each Process.
Help to build python ForkingTCPServer. This server returns an error when client connects to the server. When I use ThreadingTCPServer instead of ForkingTCPServer it works fine. socketFileIO module contains functions for pickle to/from socket:
http://code.activestate.com/recipes/577667-pickle-tofrom-socket/
server.py:
import SocketServer, time
from socketFileIO import write, read
class MyClientHandler(SocketServer.BaseRequestHandler):
def handle(self):
print self.client_address
data=read(self.request)
print 'Client:', data
time.sleep(10) # for testing
write(self.request,data)
self.request.close()
HOST, PORT = '', 50007
server = SocketServer.ForkingTCPServer((HOST, PORT), MyClientHandler)
server.serve_forever()
client.py:
import socket
from socketFileIO import write, read
HOST = '127.0.0.1'
PORT = 50007
data=["A","B","C","End"]
for x in data:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
write(s,x)
y=read(s)
print 'Server:', y
s.close()
Error:
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 1469)
Traceback (most recent call last):
File "c:\Python26\lib\SocketServer.py", line 283, in _handle_request_noblock
self.process_request(request, client_address)
File "c:\Python26\lib\SocketServer.py", line 525, in process_request
pid = os.fork()
AttributeError: 'module' object has no attribute 'fork'
----------------------------------------
I have a several other questions:
How to shutdown the server by sending data "End" from client ?
How to restrict the number of connections (for example no more then 5) ?
Your fork question was already answered in comments.
For "how to shut down the server": in a similar application (but much older Python version and using xmlrpc), I used a loop invoking one request at a time until a "stop" variable was set. This appears to be supported directly in Python 2.7 now: in your threaded server, call shutdown() on the server instance, from one of the request-handler threads. (Not sure if this is available in 2.6, I de-installed my old 2.6 version a couple of months ago.) I.e.:
class MyClientHandler(SocketServer.BaseRequestHandler):
def handle(self):
...
# if we're supposed to shut down the server, do that
if somecond:
self.server.shutdown()
For limiting the number of connections: there's nothing built-in that I can see, but if you redefined the process_request handler you could count the number of active threads and wait for one to finish if there are "too many" outstanding (or work it however else you like). See the ThreadingMixIn class in SocketServer.py and compare it to the max_children code in ForkingMixIn.
I'm playing around with sockets in python, just for the purpose of learning about them. However I am really annoyed with the following problem:
import socket
soc = socket.socket(socket.AF_INET)
soc.bind(('localhost',8000))
soc.listen(0)
client = soc.accept()
While the socket is waiting for a connection, pressing ctrl-c does not quit the application.
How can I quit the application?
A similar issue was addressed in these two questions, but there the accept method was called from a separate thread and the problem was how to make ctrl-c kill that thread. Here the accept method is called from the main thread.
Edit: I am running python 3.3.0 on Win7 64 bit.
You should use CTRL + Break. That should kill it.
I couldn't find a way to kill the application using ctrl-c or any other way except for killing it through the task manager, so I wrote a workaround:
import socket,os
from threading import Thread
class socketListener(Thread):
def run(self):
soc = socket.socket(socket.AF_INET)
soc.bind(('localhost',8000))
soc.listen(0)
client = soc.accept()
pid = os.getpid()
sl = socketListener()
sl.start()
input('Socket is listening, press any key to abort...')
os.kill(pid,9)
This runs the script in a separate thread, while waiting for a keystroke in the main thread. Once the user presses a key, the entire application is killed.
"serversocket" module provides the standard solution. I tested Control-C on Windows, it worked.
This is the link, serversocket example
The Control-C handling is even mentioned in the comment of the code
# Activate the server; this will keep running until you
# interrupt the program with Ctrl-C
Here is the complete code from the above link:
import socketserver
class MyTCPHandler(socketserver.BaseRequestHandler):
def handle(self):
# self.request is the TCP socket connected to the client
self.data = self.request.recv(1024).strip()
print("{} wrote:".format(self.client_address[0]))
print(self.data)
# just send back the same data, but upper-cased
self.request.sendall(self.data.upper())
if __name__ == "__main__":
HOST, PORT = "localhost", 9999
with socketserver.TCPServer((HOST, PORT), MyTCPHandler) as server:
# Activate the server; this will keep running until you
# interrupt the program with Ctrl-C
server.serve_forever()
If we wanted to re-invent the wheel, we would do a select() or poll() on the listener socket, with a timeout 0.5 seconds.
To save time of the other people searching for this topic, if you laptop keyboard does not have a break button, please try
Ctrl + Fn + F6
or
Ctrl + F6
After being myself confronted to the same problem I found a little workaround, it might not be the cleanest way but at least it works for me :
import socket
from select import select
#create an INET, STREAMing socket
serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
#bind the socket to localhost
serversocket.bind(('localhost', 8000))
While 1:
serversocket.listen(5)
ready, _, _ = select([serversocket], [], [], 1) #Timeout set to 1 seconds
if ready:
(clientsocket, address) = serversocket.accept()
#Do something with client
else:
#Do nothing, just loop again
By using select you will wait a change on the socket fd until the end of the timeout. As I said this might not be the cleanest way but Ctrl-c will be catch at the end of the timeout.
Portability alert: On Unix, select works both with the sockets and files. On Windows, select works with sockets only.
I'm teaching myself Python networking, and I recalled that back when I was teaching myself threading, I came across this page, so I copied the scripts, updated them for Python 3.1.1 and ran them. They worked perfectly.
Then I made a few modifications. My goal is to do something simple:
The client pickles an integer and sends it to the server.
The server receives the pickled integer, unpickles it, doubles it, then pickles it and sends it back to the client.
The client receives the pickled (and doubled) integer, unpickles it, and outputs it.
Here's the server:
import pickle
import socket
import threading
class ClientThread(threading.Thread):
def __init__(self, channel, details):
self.channel = channel
self.details = details
threading.Thread.__init__ ( self )
def run(self):
print('Received connection:', self.details[0])
request = self.channel.recv(1024)
response = pickle.dumps(pickle.loads(request) * 2)
self.channel.send(response)
self.channel.close()
print('Closed connection:', self.details [ 0 ])
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(('', 2727))
server.listen(5)
while True:
channel, details = server.accept()
ClientThread(channel, details).start()
And here is the client:
import pickle
import socket
import threading
class ConnectionThread(threading.Thread):
def run(self):
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(('localhost', 2727))
for x in range(10):
client.send(pickle.dumps(x))
print('Sent:',str(x))
print('Received:',repr(pickle.loads(client.recv(1024))))
client.close()
for x in range(5):
ConnectionThread().start()
The server runs fine, and when I run the client it successfully connects and starts sending integers and receiving them back doubled as expected. However, very quickly it exceptions out:
Exception in thread Thread-2:
Traceback (most recent call last):
File "C:\Python30\lib\threading.py", line 507, in _bootstrap_inner
self.run()
File "C:\Users\Imagist\Desktop\server\client.py", line 13, in run
print('Received:',repr(pickle.loads(client.recv(1024))))
socket.error: [Errno 10053] An established connection was aborted by the softwar
e in your host machine
The server continues to run and receives connections just fine; only the client crashes. What's causing this?
EDIT: I got the client working with the following code:
import pickle
import socket
import threading
class ConnectionThread(threading.Thread):
def run(self):
for x in range(10):
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(('localhost', 2727))
client.send(pickle.dumps(x))
print('Sent:',str(x))
print('Received:',repr(pickle.loads(client.recv(1024))))
client.close()
for x in range(5):
ConnectionThread().start()
However, I still don't understand what's going on. Isn't this just opening and closing the socket a bunch of times? Shouldn't there be time limitations to that (you shouldn't be able to open a socket so soon after closing it)?
Your client is now correct - you want to open the socket send the data, receive the reply and then close the socket.
The error original error was caused by the server closing the socket after it sent the first response which caused the client to receive a connection closed message when it tried to send the second message on the same connection.
However, I still don't understand
what's going on. Isn't this just
opening and closing the socket a bunch
of times?
Yes. This is acceptable, if not the highest performance way of doing things.
Shouldn't there be time
limitations to that (you shouldn't be
able to open a socket so soon after
closing it)?
You can open a client socket as quickly as you like as every time you open a socket you will get a new local port number, meaning that the connections won't interfere. In the server code above, it will start a new thread for each incoming connection.
There are 4 parts to every IP connection (source_address, source_port, destination_address, destination_port) and this quad (as it is known) must change for ever connection. Everything except source_port is fixed for a client socket so that is what the OS changes for you.
Opening server sockets is more troublesome - if you want to open a new server socket quickly, your
server.bind(('', 2727))
Above then you need to read up on SO_REUSEADDR.