Develop Reference

How to use threading with a list and assign workers?

I am using Python Requests to check if a proxy is working or not. I have them loaded inside a .txt file in the format of IP:PORT. I have then set up threading, using the Threading module, and it works. One problem - It runs over the same IP:PORT on every thread. Once all threads have done the first line, it moves onto the second. I want to make each thread try a different IP:PORT until all of them have been checked.
I have searched around but can not find what I am looking for. I have actually been stuck with this for a couple of weeks.
import requests
import threading
def ip():
with open('ip_found.txt', 'r') as i:
iport = [x.strip().split(':', 1) for x in i]
postIt = { 'ip' : ip, 'port' : port }
r1 = requests.post(URL"",data=postIt)
for _ in range(2):
for ip, port in iport:
t1 = threading.thread(target=ip)
t1.start()
t2 = threading.thread(target=ip)
t2.start()

I see the problem here and I also see another potential problem.
One way of doing this is to read the IP:PORTs first and store them into a queue. That way, your threads don't overlap on the same IP:Port bc each thread would take from the first. (Make sure to use a mutex/lock)
The other issue that I'm seeing is that you're not keeping track of your threads. All I see is you creating two threads, starting them, then forgetting them once the for loop iterates. This code might not really be a problem as pinging an IP:Port probably doesn't take a lot of time for a single thread, but if your function starts to bloat, you might have issues.

Trying to create a discovery script using Python Multithreadinig

I'm trying to create a discovery script, which will use multithreading to ping multiple IP addresses at once.
import ipaddress
import sh
from threading import Thread
from Queue import Queue
user_input = raw_input("")
network = ipaddress.ip_network(unicode(user_input))
def pingit(x):
for i in x.hosts():
try:
sh.ping(i, "-c 1")
print i, "is active"
except sh.ErrorReturnCode_1:
print "no response from", i
queue_to_work = Queue(maxsize=0)
number_of_workers = 30
for i in range(number_of_workers):
workers = Thread(target=pingit(network),args=(queue_to_work,))
workers.getDaemon(True)
workers.start()
When I run this script, I get the ping responses, but it is not fast. I believe the multithreading is not kicking in.
Could someone please tell me where I'm going wrong?
Many thanks.

You are doing it completely wrong.
import ipaddress
import sh
from threading import Thread
user_input = raw_input("")
network = ipaddress.ip_network(unicode(user_input))
def pingit(x):
for i in x.hosts():
try:
sh.ping(i, "-c 1")
print i, "is active"
except sh.ErrorReturnCode_1:
print "no response from", i
workers = Thread(target=pingit,args=(network,))
workers.start()
This is how you start a thread. Writing pingit(network) will actually run the function, and pass its result into Thread, while you want to pass the function itself. You should pass function pingit and the argument network separately. Note this creates a thread that practically run pingit(network).
Now, if you want to use multiple threads, you can do so in a loop. But you also have to create separate sets of data to feed into the threads. Consider you have a list of hosts, e.g. ['A', 'B', 'C', 'D'], and you want to ping them from two threads. You have to create two threads, that call pingit(['A', 'B']) and pingit(['C', 'D']).
As a side note, don't use ip_network to find the ip addresses, use ip_address. You can ping an ip address, but not a network. Of course if you want to ping all ip addresses in the network, ip_network is fine.
You may want to somehow split the user input into multiple ip addresses and separate the list into sublists for your threads. This is pretty easy. Now you can write a for to create threads, and feed each sublist into arguments of the thread. This creates threads that actually run with different parameters.

I would like to share my thoughts on this.
Since I guess this is something you would like to run in the background, I would suggest you use a Queue instead of a Thread.
This will offer you multiple advantages:
You can add multiple functionalities into the queue
If something happens, the queue will just continue, and catch the error for you. You can even add some logging to it in case something goes wrong.
The queue runs as a daemon, with every item in the queue as it's own process
Systems like RabbitMQ or Redis are build for this specific kind of task.
It is relatively easy to setup
I have created a simple script for you that you might be able to use:
import subprocess
from celery import Celery
app = Celery()
#app.task
def check_host(ip, port=80, timeout=1):
output = subprocess.Popen(["ping", "-c", "1", ip], stdout=subprocess.PIPE).communicate()[0]
if "1 packets received" in output.decode("utf-8"):
return "{ip} connected successfully".format_map(vars())
else:
return "{ip} was unable to connect".format_map(vars())
def pingit(ip="8.8.8.8"):
check_host.delay(ip)
What this does is the following.
You first import Celery, this will make you able to connect to Celery that runs in the background.
You create an app which is in instance of the Celery class
You use this app to create a task. Inside this you put task all the actions you want to perform async.
You call the delay() method on the task
Now task will run on the background, and all other tasks will be put in the queue to run async for you.
So you can just put everything in a loop, and the Queue will handle it for you.
The information about Celery: http://docs.celeryproject.org/en/latest/getting-started/first-steps-with-celery.html
And a great tutorial to get everything setup I found on YouTube: https://www.youtube.com/watch?v=68QWZU_gCDA
I hope this can help you a bit further

Threading parameters

Again a question from me.. having some issues again. Hope to find someone who's a lot smarter and knows this.. :D
I'm now having the issue with threading that when opening threading urls in a range of (1,1000), I would love to see actually all the different urls. Only when i run the code i get a lot of double variables (probably because the crawls go that fast). Anyway this is my code: I try to see at which Thread it is, but I get doubles.
import threading
import urllib2
import time
import collections
results2 = []
def crawl():
var_Number = thread.getName().split("-")[1]
try:
data = urllib2.urlopen("http://www.waarmaarraar.nl").read()
results2.append(var_Number)
except:
crawl()
threads = []
for n in xrange(1, 1000):
thread = threading.Thread(target=crawl)
thread.start()
threads.append(thread)
# to wait until all three functions are finished
print "Waiting..."
for thread in threads:
thread.join()
print "Complete."
# print results (All numbers, should be 1/1000)
results2.sort()
print results2
# print doubles (should be [])
print [x for x, y in collections.Counter(results2).items() if y > 1]
However, if I add time.sleep(0.1) directly under the xrange line, those doubles will not occur. Although this does slow my programm down a lot. Anyone knows a better way to fix this?

There is a recursive call to crawl() in the exception handler. The same thread runs the function several times if there is an error. Thus results2 may contain the same var_Number several times. If you add time.sleep(.1) (a pause); your script consumes less resources e.g., number of open fds, running threads and the request to the remote server is more likely to succeed.
Also default thread names may repeat. If a thread exited; another thread may have the same name e.g., if the implementation uses .ident attribute to generate a name.
Notes:
use pep-8 naming conventions. You could use pep8, pyflakes, epylint command-line tools to check your code automatically
you don't need 1000 threads to fetch 1000 urls (see my comment to your previous question)
it is not nice to generate requests without a pause to the same site.

According to the documentation on Thread.getName() it is a correct behavior.
If you want an unique name for each of your thread you have to set it using the name attribute.
Based on what you expect in the end, replacing
for n in xrange(1, 1000):
thread = threading.Thread(target=crawl)
thread.start()
threads.append(thread)
with
for n in xrange(1, 1000):
thread = threading.Thread(target=crawl)
thread.name = n
thread.start()
threads.append(thread)
and var_Number = thread.getName().split("-")[1] with var_Number = thread.name should help you.
EDIT
After some testing a user-custom name can be reused by another thread, so the only way to pass n will be to use the args or kwargs of threading.Thread().
This behavior make sense, if we need to use some sort of data in a Thread, pass it correctly don't try to put it where it don't belong.

Multi-threaded web scraping in Python/PySide/PyQt

I'm building a web scraper of a kind. Basically, what the soft would do is:
User (me) inputs some data (IDs) - IDs are complex, so not just numbers
Based on those IDs, the script visits http://localhost/ID
What is the best way to accomplish this? So I'm looking upwards of 20-30 concurrent connections to do it.
I was thinking, would a simple loop be the solution? This loop would start QThreads (it's a Qt app), so they would run concurrently.
The problem I am seeing with the loop however is how to instruct it to use only those IDs not used before i.e. in the iteration/thread that had been executed just before it was? Would I need some sort of a "delegator" function which will keep track of what IDs had been used and delegate the unused ones to the QThreads?
Now I've written some code but I am not sure if it is correct:
class GUI(QObject):
def __init__(self):
print "GUI CLASS INITIALIZED!!!"
self.worker = Worker()
for i in xrange(300):
QThreadPool().globalInstance().start(self.worker)
class Worker(QRunnable):
def run(self):
print "Hello world from thread", QThread.currentThread()
Now I'm not sure if these achieve really what I want. Is this actually running in separate threads? I'm asking because currentThread() is the same every time this is executed, so it doesn't look that way.
Basically, my question comes down to how do I execute several same QThreads concurrently?
Thanks in advance for the answer!

As Dikei says, Qt is red herring here. Focus on just using Python threads as it will keep your code much simpler.
In the code below we have a set, job_queue, containing the jobs to be executed. We also have a function, worker_thread which takes a job from the passed in queue and executes. Here it just sleeps for a random period of time. The key thing here is that set.pop is thread safe.
We create an array of thread objects, workers, and call start on each as we create it. From the Python documentation threading.Thread.start runs the given callable in a separate thread of control. Lastly we go through each worker thread and block until it has exited.
import threading
import random
import time
pool_size = 5
job_queue = set(range(100))
def worker_thread(queue):
while True:
try:
job = queue.pop()
except KeyError:
break
print "Processing %i..." % (job, )
time.sleep(random.random())
print "Thread exiting."
workers = []
for thread in range(pool_size):
workers.append(threading.Thread(target=worker_thread, args=(job_queue, )))
workers[-1].start()
for worker in workers:
worker.join()
print "All threads exited"

right way to run some code with timeout in Python

I looked online and found some SO discussing and ActiveState recipes for running some code with a timeout. It looks there are some common approaches:
Use thread that run the code, and join it with timeout. If timeout elapsed - kill the thread. This is not directly supported in Python (used private _Thread__stop function) so it is bad practice
Use signal.SIGALRM - but this approach not working on Windows!
Use subprocess with timeout - but this is too heavy - what if I want to start interruptible task often, I don't want fire process for each!
So, what is the right way? I'm not asking about workarounds (eg use Twisted and async IO), but actual way to solve actual problem - I have some function and I want to run it only with some timeout. If timeout elapsed, I want control back. And I want it to work on Linux and Windows.

A completely general solution to this really, honestly does not exist. You have to use the right solution for a given domain.
If you want timeouts for code you fully control, you have to write it to cooperate. Such code has to be able to break up into little chunks in some way, as in an event-driven system. You can also do this by threading if you can ensure nothing will hold a lock too long, but handling locks right is actually pretty hard.
If you want timeouts because you're afraid code is out of control (for example, if you're afraid the user will ask your calculator to compute 9**(9**9)), you need to run it in another process. This is the only easy way to sufficiently isolate it. Running it in your event system or even a different thread will not be enough. It is also possible to break things up into little chunks similar to the other solution, but requires very careful handling and usually isn't worth it; in any event, that doesn't allow you to do the same exact thing as just running the Python code.

What you might be looking for is the multiprocessing module. If subprocess is too heavy, then this may not suit your needs either.
import time
import multiprocessing
def do_this_other_thing_that_may_take_too_long(duration):
time.sleep(duration)
return 'done after sleeping {0} seconds.'.format(duration)
pool = multiprocessing.Pool(1)
print 'starting....'
res = pool.apply_async(do_this_other_thing_that_may_take_too_long, [8])
for timeout in range(1, 10):
try:
print '{0}: {1}'.format(duration, res.get(timeout))
except multiprocessing.TimeoutError:
print '{0}: timed out'.format(duration)
print 'end'

If it's network related you could try:
import socket
socket.setdefaulttimeout(number)

I found this with eventlet library:
http://eventlet.net/doc/modules/timeout.html
from eventlet.timeout import Timeout
timeout = Timeout(seconds, exception)
try:
... # execution here is limited by timeout
finally:
timeout.cancel()

For "normal" Python code, that doesn't linger prolongued times in C extensions or I/O waits, you can achieve your goal by setting a trace function with sys.settrace() that aborts the running code when the timeout is reached.
Whether that is sufficient or not depends on how co-operating or malicious the code you run is. If it's well-behaved, a tracing function is sufficient.

An other way is to use faulthandler:
import time
import faulthandler
faulthandler.enable()
try:
faulthandler.dump_tracebacks_later(3)
time.sleep(10)
finally:
faulthandler.cancel_dump_tracebacks_later()
N.B: The faulthandler module is part of stdlib in python3.3.

If you're running code that you expect to die after a set time, then you should write it properly so that there aren't any negative effects on shutdown, no matter if its a thread or a subprocess. A command pattern with undo would be useful here.
So, it really depends on what the thread is doing when you kill it. If its just crunching numbers who cares if you kill it. If its interacting with the filesystem and you kill it , then maybe you should really rethink your strategy.
What is supported in Python when it comes to threads? Daemon threads and joins. Why does python let the main thread exit if you've joined a daemon while its still active? Because its understood that someone using daemon threads will (hopefully) write the code in a way that it wont matter when that thread dies. Giving a timeout to a join and then letting main die, and thus taking any daemon threads with it, is perfectly acceptable in this context.

I've solved that in that way:
For me is worked great (in windows and not heavy at all) I'am hope it was useful for someone)
import threading
import time
class LongFunctionInside(object):
lock_state = threading.Lock()
working = False
def long_function(self, timeout):
self.working = True
timeout_work = threading.Thread(name="thread_name", target=self.work_time, args=(timeout,))
timeout_work.setDaemon(True)
timeout_work.start()
while True: # endless/long work
time.sleep(0.1) # in this rate the CPU is almost not used
if not self.working: # if state is working == true still working
break
self.set_state(True)
def work_time(self, sleep_time): # thread function that just sleeping specified time,
# in wake up it asking if function still working if it does set the secured variable work to false
time.sleep(sleep_time)
if self.working:
self.set_state(False)
def set_state(self, state): # secured state change
while True:
self.lock_state.acquire()
try:
self.working = state
break
finally:
self.lock_state.release()
lw = LongFunctionInside()
lw.long_function(10)
The main idea is to create a thread that will just sleep in parallel to "long work" and in wake up (after timeout) change the secured variable state, the long function checking the secured variable during its work.
I'm pretty new in Python programming, so if that solution has a fundamental errors, like resources, timing, deadlocks problems , please response)).

solving with the 'with' construct and merging solution from -
Timeout function if it takes too long to finish
this thread which work better.
import threading, time
class Exception_TIMEOUT(Exception):
pass
class linwintimeout:
def __init__(self, f, seconds=1.0, error_message='Timeout'):
self.seconds = seconds
self.thread = threading.Thread(target=f)
self.thread.daemon = True
self.error_message = error_message
def handle_timeout(self):
raise Exception_TIMEOUT(self.error_message)
def __enter__(self):
try:
self.thread.start()
self.thread.join(self.seconds)
except Exception, te:
raise te
def __exit__(self, type, value, traceback):
if self.thread.is_alive():
return self.handle_timeout()
def function():
while True:
print "keep printing ...", time.sleep(1)
try:
with linwintimeout(function, seconds=5.0, error_message='exceeded timeout of %s seconds' % 5.0):
pass
except Exception_TIMEOUT, e:
print " attention !! execeeded timeout, giving up ... %s " % e