I'm looking to implement multithreading or multiprocessing of request objects.
My code is below:
def validate(testurl):
json_d = {"task_id": "user_uid","data": {"document1":testurl}}
response = requests.post("https://example.net.com/document",headers=headers,json=json_d)
my_data1 = response.text
with open("testurl.txt","a+") as file:
file.write(my_data1)
my_data = json.loads(my_data1)
result = {'bool_value':my_data['data']}
return result
Is there a way to multithread or multiprocess a Pandas apply() function for more than 5000 urls? For example:
df['res'] = df['testurl'].apply(validate)
Should I be using this below?
from joblib import parallel, delayed
you can use swifter or dask for doing this. you can refer to https://gdcoder.com/speed-up-pandas-apply-function-using-dask-or-swifter-tutorial/
df['res'] = df['testurl'].swifter.apply(lambda x: validate(x))
Related
I've just started working with the multiprocessing Python library. I would like to make many API calls (get) using requests. I have a Pandas dataframe in which each row has the arguments I will be using to process the requests.get.
Here is an example of the dataframe I want to starmap to.
import pandas as pd
d = {
"companyId": ['1000','1005'],
"headers": [{'Authorization': 'Bearer token1'},{'Authorization': 'Bearer token1'}],
"employeeId": ['1500','1500'],
"date": ['2022-01-01','2022-01-02']
}
df = pd.DataFrame(d)
df.head()
Code to make request:
import multiprocessing as mp
def get_data(df: pd.DataFrame):
query: dict = {
'companyId': df['companyId'].astype(str),
'driverId': df['employeeId'].astype(str),
'day': df['date'].astype(str)
}
resp = requests.get(url=df['url'], headers=df['headers'], params=query)
return resp
if __name__ == "__main__":
with mp.Pool(mp.cpu_count()) as p:
res = list(p.starmap(get_data, zip(df.itertuples())))
print(res)
p.close()
p.join()
However, I receive some errors I am trying to understand. Ultimately, I want to map the api function to each row of my pandas dataframe in a parallel fashion. I would prefer to just use the multiprocessing library but do not necessarily need to use Pandas here if there is a simpler and more native solution.
I have this part of code in my application.
What I want is to iterate over each row in my data frame (pandas) and modify column to function result.
I tried to implement it with multiprocessing, but I'm to see if there is any faster and easier to implement way to do it.
Is there any simple way to run this part in parallel?
def _format(data: pd.DataFrame, context: pd.DataFrame)
data['context'] = data.apply(lambda row: get_context_value(context, row), axis=1)
The data frame I work with is not to large (10,000 - 100,000) and the function to evaluate the value to assign to the column take around 250ms - 500ms for one row. But the whole process for the size of the data frame takes to much.
Thanks
I have a project which it is done there: https://github.com/mjafari98/dm-classification/blob/main/inference.py
import pandas as pd
from functools import partial
from multiprocessing import Pool
import numpy as np
def parallelize(data, func, num_of_processes=8):
data_split = np.array_split(data, num_of_processes)
pool = Pool(num_of_processes)
data = pd.concat(pool.map(func, data_split))
pool.close()
pool.join()
return data
def run_on_subset(func, data_subset):
return data_subset.apply(func, axis=1)
def parallelize_on_rows(data, func, num_of_processes=8):
return parallelize(data, partial(run_on_subset, func), num_of_processes)
def a_function(row):
...do something ...
return row
df = ...somedf...
new_df = parallelize_on_rows(df, a_function)
datasets = {}
datasets['df1'] = df1
datasets['df2'] = df2
datasets['df3'] = df3
datasets['df4'] = df4
def prepare_dataframe(dataframe):
return dataframe.apply(lambda x: x.astype(str).str.lower().str.replace('[^\w\s]', ''))
for key, value in datasets.items():
datasets[key] = prepare_dataframe(value)
I need to prepare the data in some dataframes for further analysis. I would like to parallelize the for loop that updates the dictionary with a prepared dataframe. This code will eventually run on a machine with dozens of cores and thousands of dataframes. On my local machine I do not appear to be using more than a single core in the prepare_dataframe function.
I have looked at Numba and Joblib but I cannot find a way to work with dictionary values in either library.
Any insight would be very much appreciated!
You can use the multiprocessing library. You can read about its basics here.
Here is the code that does what you need:
from multiprocessing import Pool
def prepare_dataframe(dataframe):
# do whatever you want here
# changes made here are *not* global
# return a modified version of what you want
return dataframe
def worker(dict_item):
key,value = dict_item
return (key,prepare_dataframe(value))
def parallelize(data, func):
data_list = list(data.items())
pool = Pool()
data = dict(pool.map(func, data_list))
pool.close()
pool.join()
return data
datasets = parallelize(datasets,worker)
We have a dataset which has approx 1.5MM rows. I would like to process that in parallel. The main function of that code is to lookup master information and enrich the 1.5MM rows. The master is a two column dataset with roughly 25000 rows. However i am unable to make the multi-process work and test its scalability properly. Can some one please help. The cut-down version of the code is as follows
import pandas
from multiprocessing import Pool
def work(data):
mylist =[]
#Business Logic
return mylist.append(data)
if __name__ == '__main__':
data_df = pandas.read_csv('D:\\retail\\customer_sales_parallel.csv',header='infer')
print('Source Data :', data_df)
agents = 2
chunksize = 2
with Pool(processes=agents) as pool:
result = pool.map(func=work, iterable= data_df, chunksize=20)
pool.close()
pool.join()
print('Result :', result)
Method work will have the business logic and i would like to pass partitioned data_df into work to enable parallel processing. The sample data is as follows
CUSTOMER_ID,PRODUCT_ID,SALE_QTY
641996,115089,2
1078894,78144,1
1078894,121664,1
1078894,26467,1
457347,59359,2
1006860,36329,2
1006860,65237,2
1006860,121189,2
825486,78151,2
825486,78151,2
123445,115089,4
Ideally i would like to process 6 rows in each partition.
Please help.
Thanks and Regards
Bala
First, work is returning the output of mylist.append(data), which is None. I assume (and if not, I suggest) you want to return a processed Dataframe.
To distribute the load, you could use numpy.array_split to split the large Dataframe into a list of 6-row Dataframes, which are then processed by work.
import pandas
import math
import numpy as np
from multiprocessing import Pool
def work(data):
#Business Logic
return data # Return it as a Dataframe
if __name__ == '__main__':
data_df = pandas.read_csv('D:\\retail\\customer_sales_parallel.csv',header='infer')
print('Source Data :', data_df)
agents = 2
rows_per_workload = 6
num_loads = math.ceil(data_df.shape[0]/float(rows_per_workload))
split_df = np.array_split(data_df, num_loads) # A list of Dataframes
with Pool(processes=agents) as pool:
result = pool.map(func=work, iterable=split_df)
result = pandas.concat(result) # Stitch them back together
pool.close()
pool.join()pool = Pool(processes=agents)
print('Result :', result)
My best recommendation is for you to use the chunksize parameter in read_csv (Docs) and iterate over. This way you wont crash your ram trying to load everything plus if you want you can for example use threads to speed up the process.
for i,chunk in enumerate(pd.read_csv('bigfile.csv', chunksize=500000)):
Im not sure if this answer your specific question but i hope it helps.
I am reading in hundreds of HDF files and processing the data of each HDF seperately. However, this takes an awful amount of time, since it is working on one HDF file at a time. I just stumbled upon http://docs.python.org/library/multiprocessing.html and am now wondering how I can speed things up using multiprocessing.
So far, I came up with this:
import numpy as np
from multiprocessing import Pool
def myhdf(date):
ii = dates.index(date)
year = date[0:4]
month = date[4:6]
day = date[6:8]
rootdir = 'data/mydata/'
filename = 'no2track'+year+month+day
records = read_my_hdf(rootdir,filename)
if records.size:
results[ii] = np.mean(records)
dates = ['20080105','20080106','20080107','20080108','20080109']
results = np.zeros(len(dates))
pool = Pool(len(dates))
pool.map(myhdf,dates)
However, this is obviously not correct. Can you follow my chain of thought what I want to do? What do I need to change?
Try joblib for a friendlier multiprocessing wrapper:
from joblib import Parallel, delayed
def myhdf(date):
# do work
return np.mean(records)
results = Parallel(n_jobs=-1)(delayed(myhdf)(d) for d in dates)
The Pool classes map function is like the standard python libraries map function, you're guaranteed to get your results back in the order that you put them in. Knowing that, the only other trick is that you need to return results in a consistant manner, and the filter them afterwards.
import numpy as np
from multiprocessing import Pool
def myhdf(date):
year = date[0:4]
month = date[4:6]
day = date[6:8]
rootdir = 'data/mydata/'
filename = 'no2track'+year+month+day
records = read_my_hdf(rootdir,filename)
if records.size:
return np.mean(records)
dates = ['20080105','20080106','20080107','20080108','20080109']
pool = Pool(len(dates))
results = pool.map(myhdf,dates)
results = [ result for result in results if result ]
results = np.array(results)
If you really do want results as soon as they are available you can use imap_unordered