I'm trying to create a discovery script, which will use multithreading to ping multiple IP addresses at once.
import ipaddress
import sh
from threading import Thread
from Queue import Queue
user_input = raw_input("")
network = ipaddress.ip_network(unicode(user_input))
def pingit(x):
for i in x.hosts():
try:
sh.ping(i, "-c 1")
print i, "is active"
except sh.ErrorReturnCode_1:
print "no response from", i
queue_to_work = Queue(maxsize=0)
number_of_workers = 30
for i in range(number_of_workers):
workers = Thread(target=pingit(network),args=(queue_to_work,))
workers.getDaemon(True)
workers.start()
When I run this script, I get the ping responses, but it is not fast. I believe the multithreading is not kicking in.
Could someone please tell me where I'm going wrong?
Many thanks.
You are doing it completely wrong.
import ipaddress
import sh
from threading import Thread
user_input = raw_input("")
network = ipaddress.ip_network(unicode(user_input))
def pingit(x):
for i in x.hosts():
try:
sh.ping(i, "-c 1")
print i, "is active"
except sh.ErrorReturnCode_1:
print "no response from", i
workers = Thread(target=pingit,args=(network,))
workers.start()
This is how you start a thread. Writing pingit(network) will actually run the function, and pass its result into Thread, while you want to pass the function itself. You should pass function pingit and the argument network separately. Note this creates a thread that practically run pingit(network).
Now, if you want to use multiple threads, you can do so in a loop. But you also have to create separate sets of data to feed into the threads. Consider you have a list of hosts, e.g. ['A', 'B', 'C', 'D'], and you want to ping them from two threads. You have to create two threads, that call pingit(['A', 'B']) and pingit(['C', 'D']).
As a side note, don't use ip_network to find the ip addresses, use ip_address. You can ping an ip address, but not a network. Of course if you want to ping all ip addresses in the network, ip_network is fine.
You may want to somehow split the user input into multiple ip addresses and separate the list into sublists for your threads. This is pretty easy. Now you can write a for to create threads, and feed each sublist into arguments of the thread. This creates threads that actually run with different parameters.
I would like to share my thoughts on this.
Since I guess this is something you would like to run in the background, I would suggest you use a Queue instead of a Thread.
This will offer you multiple advantages:
You can add multiple functionalities into the queue
If something happens, the queue will just continue, and catch the error for you. You can even add some logging to it in case something goes wrong.
The queue runs as a daemon, with every item in the queue as it's own process
Systems like RabbitMQ or Redis are build for this specific kind of task.
It is relatively easy to setup
I have created a simple script for you that you might be able to use:
import subprocess
from celery import Celery
app = Celery()
#app.task
def check_host(ip, port=80, timeout=1):
output = subprocess.Popen(["ping", "-c", "1", ip], stdout=subprocess.PIPE).communicate()[0]
if "1 packets received" in output.decode("utf-8"):
return "{ip} connected successfully".format_map(vars())
else:
return "{ip} was unable to connect".format_map(vars())
def pingit(ip="8.8.8.8"):
check_host.delay(ip)
What this does is the following.
You first import Celery, this will make you able to connect to Celery that runs in the background.
You create an app which is in instance of the Celery class
You use this app to create a task. Inside this you put task all the actions you want to perform async.
You call the delay() method on the task
Now task will run on the background, and all other tasks will be put in the queue to run async for you.
So you can just put everything in a loop, and the Queue will handle it for you.
The information about Celery: http://docs.celeryproject.org/en/latest/getting-started/first-steps-with-celery.html
And a great tutorial to get everything setup I found on YouTube: https://www.youtube.com/watch?v=68QWZU_gCDA
I hope this can help you a bit further
Related
I’m currently working on a project and I need to fetch data from several switches by sending SSH requests as follow:
Switch 1 -> 100 requests
Switch 2 -> 500 requests
Switch 3 -> 1000 requests
…
Switch 70 -> 250 requests
So several requests (5500 in total) spread over 70 switches.
Today, I am using a json file built like this:
{
"ip_address1":
[
{"command":"command1"},
{"command":"command2"},
...
{"command":"command100"}
],
"ip_address2":
[
{"command":"command1"},
{"command":"command2"},
...
{"command":"command100"}
],
…
"ip_address70":
[
{"command":"command1"},
{"command":"command2"},
...
{"command":"command100"}
],
}
Each command is a CLI command to a switch which I’m connecting on by SSH.
Today, I’m using Python with multi threading with 8 workers because I have only 4 CPUs.
The total of the script make 1 hour to proceed so it’s too long.
Is there a way to drastically speed up this process please?
A friend told me about Golang channels and go routines but I’m not sure if it’s interesting to move from Python to Go if there’s no difference about the time.
Can you please give me some advices?
Thank you very much,
Python offers a pretty straight forward multiprocessing library. Especially for a straight forward task like yours I would stick to the language I am the most comfortable with.
In python you would basically generate a list from your list of commands and ip addresses.
Using an example straight from the documentation: https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing.pool
With the pool.map function from the multiprocessing module, you can pass each element from your list to a function, where you can pass your commands to the servers. You might want to have another look at the different mapping functions provided for the pool module.
from multiprocessing import Pool, TimeoutError
import os
def execute_ssh(address_command_mapping):
# add your logic to pass the commands to the corresponding IP address
return
if __name__ == '__main__':
# assuming your ip addresses are stored in a json file
with open("ip_addresses.json", "r") as file:
ip_addresses = json.load(file)
# transforming the address dictionary to a list of dictionaries
address_list = [{ip: commands} for ip, commands in ip_addresses.items()]
# start 4 worker processes
with Pool(processes=4) as pool:
# pool.map passes each element to the 'execute_ssh' function
pool.map(execute_ssh, address_list)
Thank you Leon,
Is the pool.map function working the same way as the thread pool executor module?
Here is what I’m using:
from concurrent.futures import ThreadPoolExecutor
def task(n):
// sending command
def main():
print("Starting ThreadPoolExecutor")
with ThreadPoolExecutor(max_workers=3) as executor:
for element in mylist:
executor.submit(task, (element))
print("All tasks complete")
if __name__ == '__main__':
main()
So is it working the same way?
Thank you
Is there any way to make multiple calls from an xmlrpc client to different xmlrpc servers at a time.
My Server code looks like this: (I'll have this code runnning in two machines, server1 & server2)
class TestMethods(object):
def printHello(self):
while(1):
time.sleep(10)
print "HELLO FROM SERVER"
return True
class ServerThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.server = SimpleXMLRPCServer(("x.x.x.x", 8000))
self.server.register_instance(TestMethods())
def run(self):
self.server.serve_forever()
server = ServerThread()
server.start()
My Client code looks like this:
import xmlrpclib
client1 = xmlrpclib.ServerProxy("http://x.x.x.x:8080") # registering with server 1
client2 = xmlrpclib.ServerProxy("http:/x.x.x.x:8080") # registering with server 2
ret1 = client1.printHello()
ret2 = client2.printHello()
Now, on the 10th second I'll get a response from server1 and on the 20th second I'll get a response from server2 which is unfortunately not what I want.
I'm trying to make calls to two machines at a time so that I get the response back from those two machines at a time.
PLease help me out, THanks in advance.
There a a few different ways to do this.
python multiprocessing
Is the built-in python module for running stuff in parallel. The docs are fairly clear. The easiest & most extensible way using this method is with a 'Pool' of workers that you can add as many to as you want.
from multiprocessing import Pool
import xmlrpclib
def hello_client(url):
client = xmlrpclib.ServerProxy(url)
return client.printHello()
p = Pool(processes=10) # maximum number of requests at once.
servers_list = ['http://x.x.x.x:8080', 'http://x.x.x.x:8080']
# you can add as many other servers into that list as you want!
results = p.map(hello_client, servers_list)
print results
twisted python
twisted python is an amazing clever system for writing all kinds of multithreaded / parallel / multiprocess stuff. The documentation is a bit confusing.
Tornado
Another non-blocking python framework. Also very cool. Here's an answer about XMLRPC, python, and tornado.
gevent
A 'magic' way of allowing blocking tasks in python to happen in the background. Very very cool. And here's a question about how to use XMLRPC in python with gevent.
I'm trying to create a program in python where I have to run two classes at the same time. So what I'm trying to do is that I just create a thread for every instance of the class. Programming is pretty new for me, so this is what we call a "user error" ;)
Here is my code:
import threading
import sys
servers = [
["Server 01", "192.168.0.1", 12345, "password"],
["Server 02", "192.168.0.1", 12346, "password"],
]
def main():
for row in servers:
serverName = row[0]
serverAddress = row[1]
rconPort = row[2]
rconPass = row[3]
th = threading.Thread(target = rcon.RconProtocol(serverAddress, rconPort, rconPass, serverName))
th.start()
if __name__ == '__main__':
main()
The code works great for the first server in servers. But then it stops after th.start(). So it doesn't matter what ever I do after th.start(), it doesn't run. RconProtocol doesn't do anything special except listening to a socket. And here is where I think the problem is. I have an infinite loop in the RconProtocol class (while True).
I tried to do a similar function (I did another called two too):
def one():
while True:
print "one"
and just specified it as a target for my thread, and I got the same issue. It works without the while-loop (just print "one").
So I'm a little bit curious why this is happening and what I do wrong. I usually see myself as a good google person, but I couldn't find any similar cases. My head doesn't like examples where you have to use time, or random examples. I think I have to see more practical examples.
By the way, is it "wrong" to place a while-loop in a class like this? Should it be outside in some kind of way? Should I maybe create threads in my RconProtocol class instead?
Hope this is enough information.
Thanks in advance!
/Daniel
You are calling rcon.RconProtocol in main thrad. Replcae following line
th = threading.Thread(target=rcon.RconProtocol(serverAddress, rconPort, rconPass, serverName))
with
th = threading.Thread(target=rcon.RconProtocol, args=(serverAddress, rconPort, rconPass, serverName))
In addition, it is a bad idea to use a while loop as the high level control in a thread. Python threads are good for blocking situations where you spend most of your time waiting for something and then react. In your case you should use a threading.Queue to send commands to your thread to be forwarded to your server. This way your thread will be blocked waiting on the Queue or transmitting a message to your server.
I have a list of dynamically generated processes (command line libraries with arguments) which I need to run.
I know that some of them are dependent on each other. I already have some objects which contain this information. For example, standalone_exec_item contains process_data.process_id and also dependent_on_process_ids (which is a list of process ids.)
Currently I am thinking of using the multiprocessing library to run the list of processes asynchronously sort of like this:
from multiprocessing import Process
import subprocess
def execute_standalone_exec_items(standalone_exec_items):
standalones = []
def run_command(command):
output = subprocess.check_output(shlex.split(command))
return output
for standalone_exec_item in standalone_exec_items:
standalone_command = generate_command(standalone_exec_item.process_data)
standalone = Process(
target=run_command,
args=(standalone_command,)
)
standalones.append(standalone)
for standalone in standalones:
standalone.start()
while True:
flag = True
for standalone in standalones:
if standalone.is_alive():
flag = False
if flag:
break
However I want to know if there's a nicer way of waiting for the asynchronous processes to run before running the dependent processes. Can I use callbacks? I've heard about Twisted's deferred, can I use this?
What's the best practice?
Edit:
Is it correct that Popen is non-blocking and I don't need to use multiprocessing? Or do I need to use fcntl()?
I would use a message queue, where some process publishing message(s) which the to be called process will subscribe.
I'm building a web scraper of a kind. Basically, what the soft would do is:
User (me) inputs some data (IDs) - IDs are complex, so not just numbers
Based on those IDs, the script visits http://localhost/ID
What is the best way to accomplish this? So I'm looking upwards of 20-30 concurrent connections to do it.
I was thinking, would a simple loop be the solution? This loop would start QThreads (it's a Qt app), so they would run concurrently.
The problem I am seeing with the loop however is how to instruct it to use only those IDs not used before i.e. in the iteration/thread that had been executed just before it was? Would I need some sort of a "delegator" function which will keep track of what IDs had been used and delegate the unused ones to the QThreads?
Now I've written some code but I am not sure if it is correct:
class GUI(QObject):
def __init__(self):
print "GUI CLASS INITIALIZED!!!"
self.worker = Worker()
for i in xrange(300):
QThreadPool().globalInstance().start(self.worker)
class Worker(QRunnable):
def run(self):
print "Hello world from thread", QThread.currentThread()
Now I'm not sure if these achieve really what I want. Is this actually running in separate threads? I'm asking because currentThread() is the same every time this is executed, so it doesn't look that way.
Basically, my question comes down to how do I execute several same QThreads concurrently?
Thanks in advance for the answer!
As Dikei says, Qt is red herring here. Focus on just using Python threads as it will keep your code much simpler.
In the code below we have a set, job_queue, containing the jobs to be executed. We also have a function, worker_thread which takes a job from the passed in queue and executes. Here it just sleeps for a random period of time. The key thing here is that set.pop is thread safe.
We create an array of thread objects, workers, and call start on each as we create it. From the Python documentation threading.Thread.start runs the given callable in a separate thread of control. Lastly we go through each worker thread and block until it has exited.
import threading
import random
import time
pool_size = 5
job_queue = set(range(100))
def worker_thread(queue):
while True:
try:
job = queue.pop()
except KeyError:
break
print "Processing %i..." % (job, )
time.sleep(random.random())
print "Thread exiting."
workers = []
for thread in range(pool_size):
workers.append(threading.Thread(target=worker_thread, args=(job_queue, )))
workers[-1].start()
for worker in workers:
worker.join()
print "All threads exited"