I’m currently working on a project and I need to fetch data from several switches by sending SSH requests as follow:
Switch 1 -> 100 requests
Switch 2 -> 500 requests
Switch 3 -> 1000 requests
…
Switch 70 -> 250 requests
So several requests (5500 in total) spread over 70 switches.
Today, I am using a json file built like this:
{
"ip_address1":
[
{"command":"command1"},
{"command":"command2"},
...
{"command":"command100"}
],
"ip_address2":
[
{"command":"command1"},
{"command":"command2"},
...
{"command":"command100"}
],
…
"ip_address70":
[
{"command":"command1"},
{"command":"command2"},
...
{"command":"command100"}
],
}
Each command is a CLI command to a switch which I’m connecting on by SSH.
Today, I’m using Python with multi threading with 8 workers because I have only 4 CPUs.
The total of the script make 1 hour to proceed so it’s too long.
Is there a way to drastically speed up this process please?
A friend told me about Golang channels and go routines but I’m not sure if it’s interesting to move from Python to Go if there’s no difference about the time.
Can you please give me some advices?
Thank you very much,
Python offers a pretty straight forward multiprocessing library. Especially for a straight forward task like yours I would stick to the language I am the most comfortable with.
In python you would basically generate a list from your list of commands and ip addresses.
Using an example straight from the documentation: https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing.pool
With the pool.map function from the multiprocessing module, you can pass each element from your list to a function, where you can pass your commands to the servers. You might want to have another look at the different mapping functions provided for the pool module.
from multiprocessing import Pool, TimeoutError
import os
def execute_ssh(address_command_mapping):
# add your logic to pass the commands to the corresponding IP address
return
if __name__ == '__main__':
# assuming your ip addresses are stored in a json file
with open("ip_addresses.json", "r") as file:
ip_addresses = json.load(file)
# transforming the address dictionary to a list of dictionaries
address_list = [{ip: commands} for ip, commands in ip_addresses.items()]
# start 4 worker processes
with Pool(processes=4) as pool:
# pool.map passes each element to the 'execute_ssh' function
pool.map(execute_ssh, address_list)
Thank you Leon,
Is the pool.map function working the same way as the thread pool executor module?
Here is what I’m using:
from concurrent.futures import ThreadPoolExecutor
def task(n):
// sending command
def main():
print("Starting ThreadPoolExecutor")
with ThreadPoolExecutor(max_workers=3) as executor:
for element in mylist:
executor.submit(task, (element))
print("All tasks complete")
if __name__ == '__main__':
main()
So is it working the same way?
Thank you
Related
I have a question about running function parallel in Python.
I have tried using multi processing to reduce time sending and receiving data from API but when I execute code below, it tend to crash my IDE.
def network_request_function(value)
#this function sends requests using value.
for i in list:
p1 = Process(target=network_request_function, args=(i,))
p1.start()
Can you provide a way to fix my code?
Or are there better alternatives?
You should specify what platform this is running on what your IDE is. Also, if all network_request_function is doing is making a network request and awaiting a reply which gets no further processing requiring intensive CPU, then this seems like it should be using multithreading instead of multiprocessing and a multithreading pool where the number of concurrent threads can be limited in case the length of your input list is very large and where it is simpler to get a return value from network_request_function that you might be interested in. And you should not use a name, such as list, that happens to be the name of a built-in function or class for naming a variable.
For example:
def network_request_function(value):
#this function sends requests using value and returns the reply
return reply
if __name__ == '__main__': # Required if we switch to multiprocessing
# To use multiprocessing:
#from multiprocessing.pool import Pool as Executor
# To use multithreading:
from multiprocessing.pool import ThreadPool as Executor
# inputs is our list of value arguments used with network_request_function:
inputs = []; # This variable is set somewhere
# May need to be a smaller number if we are using multiprocessing and
# depending on the platform:
MAX_POOL_SIZE = 200
pool_size = min(len(inputs), MAX_POOL_SIZE)
with Executor(pool_size) as pool:
# Get list of all replies:
replies = pool.map(network_request_function, inputs)
I am mining data from a website through Data Scraping in Python. I am using request package for sending the parameters.
Here is the code snippet in Python:
for param in paramList:
data = get_url_data(param)
def get_url_data(param):
post_data = get_post_data(param)
headers = {}
headers["Content-Type"] = "text/xml; charset=UTF-8"
headers["Content-Length"] = len(post_data)
headers["Connection"] = 'Keep-Alive'
headers["Cache-Control"] = 'no-cache'
page = requests.post(url, data=post_data, headers=headers, timeout=10)
data = parse_page(page.content)
return data
The variable paramList is a list of more than 1000 elements and the endpoint url remains the same. I was wondering if there is a better and more faster way to do this ?
Thanks
As there is a significant amount of networking I/O involved, threading should improve the overall performance significantly.
You can try using a ThreadPool and should test and tweak the number of threads to a one that is best suitable for the situation and shows the overall highest performance .
from multiprocessing.pool import ThreadPool
# Remove 'for param in paramList' iteration
def get_url_data(param):
# Rest of code here
if __name__ == '__main__':
pool = ThreadPool(15)
pool.map(get_url_data, paramList) # Will split the load between the threads nicely
pool.close()
I need to make 1000 post request to same domain, I was wondering if
there is a better and more faster way to do this ?
It depends, if it's a static asset or a servlet which you know what it does, if the same parameters will return the same reponse each time you can implement LRU or some other caching mechanism, if not, 1K of POST requests to some servlet doesn't matter even if they have the same domain.
There is an answer with using multiprocessing whith ThreadPool interface, which actually uses the main process with 15 threads, does it runs on 15 cores machine ? because a core can only run one thread each time (except hyper ones, does it run on 8 hyper-cores?)
ThreadPool interface inside library which has a trivial name, multiprocessing, because python has also threading module, this is confusing as f#ck, lets benchmark some lower level code:
import psutil
from multiprocessing.pool import ThreadPool
from time import sleep
def get_url_data(param):
print(param) # just for convenience
sleep(1) # lets assume it will take one second each time
if __name__ == '__main__':
paramList = [i for i in range(100)] # 100 urls
pool = ThreadPool(psutil.cpu_count()) # each core can run one thread (hyper.. not now)
pool.map(get_url_data, paramList) # splitting the jobs
pool.close()
The code above will use the main process with 4 threads in my case because my laptop has 4 CPUs, benchmark result:
$ time python3_5_2 test.py
real 0m28.127s
user 0m0.080s
sys 0m0.012s
Lets try spawning processes w/ multiprocessing
import psutil
import multiprocessing
from time import sleep
import numpy
def get_url_data(urls):
for url in urls:
print(url)
sleep(1) # lets assume it will take one second each time
if __name__ == "__main__":
jobs = []
# Split URLs into chunks as number of CPUs
chunks = numpy.array_split(range(100), psutil.cpu_count())
# Pass each chunk into process
for url_chunk in chunks:
jobs.append(multiprocessing.Process(target=get_url_data, args=(url_chunk, )))
# Start the processes
for j in jobs:
j.start()
# Ensure all of the processes have finished
for j in jobs:
j.join()
Benchmark result: less 3 seconds
$ time python3_5_2 test2.py
real 0m25.208s
user 0m0.260s
sys 0m0.216
If you will execute ps -aux | grep "test.py" you will see 5 processes because one is the main which manage the others.
There are some drawbacks:
You did not explain in depth what your code is doing, but if you doing some work which needs to be synchronized you need to know multiprocessing is NOT thread safe.
Spawning extra processes introduces I/O overhead as data is having to be shuffled around between processors.
Assuming the data is restricted to each process, it is possible to gain significant speedup, be aware of Amdahl's Law.
If you will reveal what your code does afterwards ( save it into file ? database ? stdout ? ) it will be easier to give better answer/direction, few ideas comes up to my mind like immutable infrastructure with Bash or Java to handle synchronization or is it a memory-bound issue and you need an objects pool to process the JSON responses.. might even be a job for fault tolerance Elixir)
I'm trying to create a discovery script, which will use multithreading to ping multiple IP addresses at once.
import ipaddress
import sh
from threading import Thread
from Queue import Queue
user_input = raw_input("")
network = ipaddress.ip_network(unicode(user_input))
def pingit(x):
for i in x.hosts():
try:
sh.ping(i, "-c 1")
print i, "is active"
except sh.ErrorReturnCode_1:
print "no response from", i
queue_to_work = Queue(maxsize=0)
number_of_workers = 30
for i in range(number_of_workers):
workers = Thread(target=pingit(network),args=(queue_to_work,))
workers.getDaemon(True)
workers.start()
When I run this script, I get the ping responses, but it is not fast. I believe the multithreading is not kicking in.
Could someone please tell me where I'm going wrong?
Many thanks.
You are doing it completely wrong.
import ipaddress
import sh
from threading import Thread
user_input = raw_input("")
network = ipaddress.ip_network(unicode(user_input))
def pingit(x):
for i in x.hosts():
try:
sh.ping(i, "-c 1")
print i, "is active"
except sh.ErrorReturnCode_1:
print "no response from", i
workers = Thread(target=pingit,args=(network,))
workers.start()
This is how you start a thread. Writing pingit(network) will actually run the function, and pass its result into Thread, while you want to pass the function itself. You should pass function pingit and the argument network separately. Note this creates a thread that practically run pingit(network).
Now, if you want to use multiple threads, you can do so in a loop. But you also have to create separate sets of data to feed into the threads. Consider you have a list of hosts, e.g. ['A', 'B', 'C', 'D'], and you want to ping them from two threads. You have to create two threads, that call pingit(['A', 'B']) and pingit(['C', 'D']).
As a side note, don't use ip_network to find the ip addresses, use ip_address. You can ping an ip address, but not a network. Of course if you want to ping all ip addresses in the network, ip_network is fine.
You may want to somehow split the user input into multiple ip addresses and separate the list into sublists for your threads. This is pretty easy. Now you can write a for to create threads, and feed each sublist into arguments of the thread. This creates threads that actually run with different parameters.
I would like to share my thoughts on this.
Since I guess this is something you would like to run in the background, I would suggest you use a Queue instead of a Thread.
This will offer you multiple advantages:
You can add multiple functionalities into the queue
If something happens, the queue will just continue, and catch the error for you. You can even add some logging to it in case something goes wrong.
The queue runs as a daemon, with every item in the queue as it's own process
Systems like RabbitMQ or Redis are build for this specific kind of task.
It is relatively easy to setup
I have created a simple script for you that you might be able to use:
import subprocess
from celery import Celery
app = Celery()
#app.task
def check_host(ip, port=80, timeout=1):
output = subprocess.Popen(["ping", "-c", "1", ip], stdout=subprocess.PIPE).communicate()[0]
if "1 packets received" in output.decode("utf-8"):
return "{ip} connected successfully".format_map(vars())
else:
return "{ip} was unable to connect".format_map(vars())
def pingit(ip="8.8.8.8"):
check_host.delay(ip)
What this does is the following.
You first import Celery, this will make you able to connect to Celery that runs in the background.
You create an app which is in instance of the Celery class
You use this app to create a task. Inside this you put task all the actions you want to perform async.
You call the delay() method on the task
Now task will run on the background, and all other tasks will be put in the queue to run async for you.
So you can just put everything in a loop, and the Queue will handle it for you.
The information about Celery: http://docs.celeryproject.org/en/latest/getting-started/first-steps-with-celery.html
And a great tutorial to get everything setup I found on YouTube: https://www.youtube.com/watch?v=68QWZU_gCDA
I hope this can help you a bit further
I want to test a large number of IPs to look for open DNS resolvers. I'm trying to find the most efficient way to parallelize this. At the moment I'm trying to accomplish this with twisted. I want to have 10 or 20 parallel threads sending a query to avoid blocking trough timeouts.
Twisted has a DNSDatagramProtocol that seems suitable but I just can't figure out how to put it together with the twisted "reactor" and "threads" facilities to make it run efficiently.
I read a lot of the twisted documentation but I'm still not sure what would be the best way to do it.
Could someone give an example how this can be accomplished?
Here's a quick example demonstrating the Twisted Names API:
from sys import argv
from itertools import cycle
from pprint import pprint
from twisted.names import client
from twisted.internet.task import react
from twisted.internet.defer import gatherResults, inlineCallbacks
def query(reactor, server, name):
# Create a new resolver that uses the given DNS server
resolver = client.Resolver(
resolv="/dev/null", servers=[(server, 53)], reactor=reactor)
# Use it to do an A request for the name
return resolver.lookupAddress(name)
#inlineCallbacks
def main(reactor, *names):
# Here's some random DNS servers to which to issue requests.
servers = ["4.2.2.1", "8.8.8.8"]
# Handy trick to cycle through those servers forever
next_server = cycle(servers).next
# Issue queries for all the names given, alternating between servers.
results = []
for n in names:
results.append(query(reactor, next_server(), n))
# Wait for all the results
results = yield gatherResults(results)
# And report them
pprint(zip(names, results))
if __name__ == '__main__':
# Run the main program with the reactor going and pass names
# from the command line arguments to be resolved
react(main, argv[1:])
Try gevent, spawn many greenlets to do a DNS resolution. Also gevent has a nice DNS resolution API : http://www.gevent.org/gevent.dns.html
They have even an example:
https://github.com/gevent/gevent/blob/master/examples/dns_mass_resolve.py
I would love to have this programm improve a lot in speed. It reads +- 12000 pages in 10 minutes. I was wondering if there is something what would help a lot to the speed? I hope you guys know some tips. I am supposed to read +- millions of pages... so that would take way too long :( Here is my code:
from eventlet.green import urllib2
import httplib
import time
import eventlet
# Create the URLS in groups of 400 (+- max for eventlet)
def web_CreateURLS():
print str(str(time.asctime( time.localtime(time.time()) )).split(" ")[3])
for var_indexURLS in xrange(0, 2000000, 400):
var_URLS = []
for var_indexCRAWL in xrange(var_indexURLS, var_indexURLS+400):
var_URLS.append("http://www.nu.nl")
web_ScanURLS(var_URLS)
# Return the HTML Source per URL
def web_ReturnHTML(url):
try:
return [urllib2.urlopen(url[0]).read(), url[1]]
except urllib2.URLError:
time.sleep(10)
print "UrlError"
web_ReturnHTML(url)
# Analyse the HTML Source
def web_ScanURLS(var_URLS):
pool = eventlet.GreenPool()
try:
for var_HTML in pool.imap(web_ReturnHTML, var_URLS):
# do something etc..
except TypeError: pass
web_CreateURLS()
I like using greenlets.. but I often benefit from using multiple processes spread over lots of systems.. or just one single system letting the OS take care of all the checks and balances of running multiple processes.
Check out ZeroMQ at http://zeromq.org/ for some good examples on how to make a dispatcher with a TON of listeners that do whatever the dispatcher says. Alternatively check out execnet for a method of quickly getting started with executing remote or local tasks in parallel.
I also use http://spread.org/ a lot and have LOTS of systems listening to a common spread daemon.. it's a very useful message bus where results can be pooled back to and dispatched from a single thread pretty easily.
And then of course there is always redis pub/sub or sync. :)
"Share the load"