python ThreadPoolExecutor memory leak issues

python ThreadPoolExecutor memory leak issues - python

I'm trying to debug a memory leak in my application, and I think I managed to reduce it to this minimal example:
from typing import Deque
import gc
import os
import psutil
from concurrent.futures import ThreadPoolExecutor
process = psutil.Process(os.getpid()) # we'll use this to track memory usage
executor = ThreadPoolExecutor() # generic executor
n = 100 # number of tasks to run concurrently at any time
prev = 0 # previous maximum memory usage
m = 10_000
def do(): # an arbitrary function that returns a large value
return list(range(m))
# initialize a deque that can track all our running futures
futures = Deque(executor.submit(do) for _ in range(n))
i = 0 # future counter
while futures:
f = futures.popleft() # pop one of the futures (removing its reference from the deque)
result = f.result()
del result, f # delete the future (the future is definitely removed now)
# check the memory usage and print a message if it is a new record
new_mem = process.memory_info().rss
if new_mem > prev:
print(f"{i}: {new_mem:,} bytes, {len(futures)} tasks pending")
prev = new_mem
# enqueue a new task
futures.append(executor.submit(do))
gc.collect()
i+=1
Essentially what we do here is keep a rotating queue of tasks that return a large value, ensuring that there are no more than n tasks running concurrently at any time.
We would expect, after about max_workers tasks have completed, that top memory usage would reach a plateau since none of the data from the futures is being preserved.
However, on python 3.7.12, we see that memory usage increases continually and plateaus sometimes even on the thousandth task!
This outcome is preserved even when we change do() to return a numpy array, when we limit the max_workers to a smaller number, and when we reduce n.
EDIT: I added a table measuring the pear memory usage (on my machine) of a process running this code by iteration count, for different values of m
| peak memory usage after N iterations | m=1,000 | m=5,000 | m=10,000 |
|--------------------------------------|------------|------------|------------|
| 500 | 16,961,536 | 37,662,720 | 61,689,856 |
| 1,000 | 17,408,000 | 38,981,632 | 62,132,224 |
| 1,500 | 17,760,256 | 39,374,848 | 62,164,992 |
| 2,000 | 18,063,360 | 40,177,664 | 62,832,640 |
| 2,500 | 18,219,008 | 40,845,312 | 62,832,640 |
| 3,000 | 18,374,656 | 40,931,328 | 63,156,224 |
| 3,500 | 18,493,440 | 41,492,480 | 63,328,256 |
| 4,000 | 18,604,032 | 41,586,688 | 63,328,256 |
| 4,500 | 18,673,664 | 41,586,688 | 63,328,256 |
| 5,000 | 18,763,776 | 41,586,688 | 63,328,256 |
| 5,500 | 18,800,640 | 41,586,688 | 63,328,256 |
| 6,000 | 18,853,888 | 41,615,360 | 63,328,256 |
| 6,500 | 18,898,944 | 41,660,416 | 63,328,256 |
| 7,000 | 18,931,712 | 41,795,584 | 63,328,256 |
| 7,500 | 18,952,192 | 41,930,752 | 63,328,256 |
| 8,000 | 18,952,192 | 42,061,824 | 63,328,256 |
| 8,500 | 18,993,152 | 42,377,216 | 63,361,024 |
| 9,000 | 19,046,400 | 42,463,232 | 63,528,960 |
| 9,500 | 19,075,072 | 42,463,232 | 63,528,960 |
| 10,000 | 19,075,072 | 42,463,232 | 63,528,960 |
| 10,500 | 19,083,264 | 42,463,232 | 63,533,056 |
| 11,000 | 19,083,264 | 42,463,232 | 63,664,128 |
| 11,500 | 19,128,320 | 42,553,344 | 63,664,128 |
| 12,000 | 19,185,664 | 42,553,344 | 63,664,128 |
| 12,500 | 19,185,664 | 42,553,344 | 63,664,128 |
| 13,000 | 19,185,664 | 42,553,344 | 63,664,128 |
| 13,500 | 19,193,856 | 42,553,344 | 64,167,936 |
| 14,000 | 19,193,856 | 42,594,304 | 64,167,936 |
| 14,500 | 19,193,856 | 42,594,304 | 64,167,936 |
| 15,000 | 19,202,048 | 42,594,304 | 64,249,856 |
| 15,500 | 19,202,048 | 42,594,304 | 64,249,856 |
| 16,000 | 19,222,528 | 42,594,304 | 64,462,848 |
| 16,500 | 19,353,600 | 42,594,304 | 64,499,712 |
| 17,000 | 19,361,792 | 42,635,264 | 64,499,712 |
| 17,500 | 19,415,040 | 42,725,376 | 64,499,712 |
| 18,000 | 19,423,232 | 42,766,336 | 64,499,712 |
| 18,500 | 19,431,424 | 42,811,392 | 64,499,712 |
| 19,000 | 19,476,480 | 42,811,392 | 64,499,712 |
| 19,500 | 19,537,920 | 42,811,392 | 64,520,192 |
| 20,000 | 19,607,552 | 42,856,448 | 65,044,480 |
| 20,500 | 19,607,552 | 42,946,560 | 65,044,480 |
| 21,000 | 19,636,224 | 43,110,400 | 65,044,480 |
| 21,500 | 19,644,416 | 43,110,400 | 65,044,480 |
| 22,000 | 19,652,608 | 43,110,400 | 65,044,480 |
| 22,500 | 19,722,240 | 43,110,400 | 65,044,480 |
| 23,000 | 19,746,816 | 43,110,400 | 65,044,480 |
| 23,500 | 19,746,816 | 43,110,400 | 65,044,480 |
| 24,000 | 19,746,816 | 43,110,400 | 65,044,480 |
| 24,500 | 19,775,488 | 43,110,400 | 65,044,480 |
| 25,000 | 19,800,064 | 43,110,400 | 65,044,480 |
| 25,500 | 19,820,544 | 43,110,400 | 65,044,480 |
| 26,000 | 19,886,080 | 43,110,400 | 65,044,480 |
| 26,500 | 19,886,080 | 43,118,592 | 65,044,480 |
| 27,000 | 19,894,272 | 43,118,592 | 65,044,480 |
| 27,500 | 19,939,328 | 43,208,704 | 65,044,480 |
| 28,000 | 19,947,520 | 43,208,704 | 65,044,480 |
| 28,500 | 20,058,112 | 43,208,704 | 65,044,480 |
| 29,000 | 20,086,784 | 43,208,704 | 65,044,480 |
| 29,500 | 20,086,784 | 43,208,704 | 65,044,480 |
| 30,000 | 20,086,784 | 43,208,704 | 65,044,480 |

Related

How to classify the image data from the folders?

It is our folder tree.
dataset/
|
|
|-----train/
| |
| |--dog/
| | |
| | |--German Shepherd/
| | | |
| | | |-- image1.png
| | | |-- image2.png
| | | |-- ...
| | |
| | |--Bulldog/
| | |--Labrador Retriever/
| | |--Poodle/
| | |--Chihuahua/
| | |--...
| |
| |--cat/
| | |
| | |--Siamese cat/
| | |--British Shorthair/
| | |--Maine Coon/
| | |--Persian cat/
| | |--Ragdoll/
| | |--...
|
|-----test/
|
|--...
how can be classify it with tensorflow-python ?
I am not using any pre-trained model then how can be classify it ?

Updating values onto an ASCII art without changing the format [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 months ago.
Improve this question
I am exploring ways on bash as well as on Python to print out values onto an ASCII art for improving the readability.
The one difficulty is to update values without changing the format of the art.
The ascii art looks something like this:
========================================================
| | | | | |
| | | | | |
| CPU | | GPU | | HDD |
| | | | | |
| | | | | |
| ${CPU_W}W | | | | |
| ${CPU_Freq}MHz| | | |Avail Mem${size}G|
| | |${GPU_W}W | | Used Mem${size}G|
| | |${GPU}Mhz | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
========================================================
So far, I was able to prevent the format from changing on Bash. But doing this, is not allowing me to change the values.
cat << "EOF"
========================================================
| | | | | |
| | | | | |
| CPU | | GPU | | HDD |
| | | | | |
| | | | | |
| ${CPU_W}W | | | | |
| ${CPU_Freq}MHz| | | |Avail Mem${size}G|
| | |${GPU_W}W | | Used Mem${size}G|
| | |${GPU}Mhz | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
========================================================
EOF
Without cat << "EOF" ...ascii art.... EOF, I can update the values but the format keeps changing .
Is there anyway to keep the same format even with the values changing? Thanks in advance.

I'm not exactly sure what you're trying to achieve... so?
f = """
========================================================
| | | | | |
| | | | | |
| CPU | | GPU | | HDD |
| | | | | |
| | | | | |
|{cpuW:>15}| | | | |
|{cpuF:>15}| | | |Avail Mem{ramA:>8}|
| | |{gpuW:>10}| | Used Mem{ramU:>8}|
| | |{gpuF:>10}| | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
========================================================
"""
def main():
cpuW = 12.2
cpuF = 2354
gpuW = 15.2
gpuF = 789
ramA = 16.1
ramU = 12.2
d = {
'cpuW': f'{cpuW:.1f} W ',
'cpuF': f'{cpuF:d} MHz ',
'gpuW': f'{gpuW:.1f} W ',
'gpuF': f'{gpuF:d} MHz ',
'ramA': f' {ramA:.1f} GB',
'ramU': f' {ramU:.1f} GB',
}
out = f.format_map(d)
print(out)
if __name__ == '__main__':
main()
Output is:
========================================================
| | | | | |
| | | | | |
| CPU | | GPU | | HDD |
| | | | | |
| | | | | |
| 12.2 W | | | | |
| 2354 MHz | | | |Avail Mem 16.1 GB|
| | | 15.2 W | | Used Mem 12.2 GB|
| | | 789 MHz | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
========================================================

In python, you can use the format and str.ljust method.
format(value[, format_spec])
Convert a value to a “formatted” representation, as controlled by format_spec. The interpretation of format_spec will depend on the type of the value argument; however, there is a standard formatting syntax that is used by most built-in types: Format Specification Mini-Language.
It depends on what you want the notation to be (e.g how many zeros, ecc...).
here is an example:
>>> characters = 10
>>> format(32,".2E").zfill(characters)
'003.20E+01'
>>> #first number is the minimum number of characters
>>> format(32,"{0}.2E".format(characters))
' 3.20E+01'
or, with fstrings:
>>> f"{32:.2E}"
'3.20E+01'
Example on how to use it in ascii art:
>>> def create_ascii_art(CPU_W,CPU_freq,GPU_W,
... GPU,Free_Mem,Used_Mem):
... CPU_W,CPU_freq,GPU_W,GPU,Free_Mem,Used_Mem = map(float,(CPU_W,CPU_freq,GPU_W,GPU,Free_Mem,Used_Mem))
... return f'''
...========================================================
...| | | | | |
...| | | | | |
...| CPU | | GPU | | HDD |
...| | | | | |
...| | | | | |
...|{CPU_W:12.2E}W | | | | |
...|{CPU_freq:12.2E}MHz| | | |Avail Mem{Free_Mem:7.3}G|
...| | |{GPU_W:8.2E}W | | Used Mem{Used_Mem:7.3}G|
...| | |{GPU:6.0E}Mhz | | |
...| | | | | |
...| | | | | |
...| | | | | |
...| | | | | |
...========================================================'''
>>> print(create_ascii_art(1,1,1,1,1,1))
output:
========================================================
| | | | | |
| | | | | |
| CPU | | GPU | | HDD |
| | | | | |
| | | | | |
| 1.00E+00W | | | | |
| 1.00E+00MHz| | | |Avail Mem 1.0G|
| | |1.00E+00W | | Used Mem 1.0G|
| | | 1E+00Mhz | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
========================================================

The "format" does not change; the display width of the strings in the here document depends on the values of the variables. You'll want to make sure the values are padded to the expected width (or truncated, if necessary).
A slightly tortured way to accomplish this is to pad the actual values:
#!/bin/bash
:
printf -v CPU_W '%8s' "$CPU_W"
printf -v CPU_Freq '%11s' "$CPU_Freq"
printf -v GPU_W '%8s' "$GPU_W"
printf -v GPU '%6s' "$GPU"
printf -v size '%7s' "$size"
cat <<EOF
========================================================
| | | | | |
| | | | | |
| CPU | | GPU | | HDD |
| | | | | |
| | | | | |
| ${CPU_W}W | | | | |
| ${CPU_Freq}MHz| | | |Avail Mem${size}G|
| | |${GPU_W}W | | Used Mem${size}G|
| | |${GPU}Mhz | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
========================================================
EOF
Probably a better solution altogether is to use printf directly to format the output. (Personally, I would perceive a notable reduction in my blood pressure if you removed at least half of the repetitive "ASCII art", which would somewhat tidy up the necessary code, too.)
We can't know what values these variables contain; if they are integers, maybe experiment with i instead of s for the printf format code, or correspondingly f or perhaps g for floating-point values.
If you need truncation, try %8.8 instead of %8, etc.
An altogether nicer approach is to use a library for producing this format. I can't recommend any particular one for Bash; for Python, look at tabulate.

Cursor Insert return out of memory errors

I am a newbie of MySQL. I am using Python Connect to insert over 350,000 rows into a table running on a MySQL 8 database. My Python code look like this.
cursor = cnx.cursor(buffered=True)
stmt = "INSERT INTO ......"
cursor.executemany(stmt, data)
cnx.commit()
cursor.close()
which returns the following errors:
[ERROR] [MY-010934] [Server] Out of memory; check if mysqld or some
other process uses all available memory; if not, you may have to use
'ulimit' to allow mysqld to use more memory or you can add more swap
space
If I reduce the inserted rows such as inserting 200,000 rows only, the error disappears. I think there must be some size limit on the MySQL settings, but I don't know which one to change. I tried to manually increase the innodb_buffer_pool_size to 500MB as many answers said, but the error continues. What should I do? I printed out my system variable about size and it's listed below.
| binlog_cache_size | 32768 |
| binlog_row_event_max_size | 8192 |
| binlog_stmt_cache_size | 32768 |
| binlog_transaction_dependency_history_size | 25000 |
| bulk_insert_buffer_size | 8388608 |
| delayed_queue_size | 1000 |
| histogram_generation_max_mem_size | 20000000 |
| host_cache_size | 279 |
| innodb_buffer_pool_chunk_size | 134217728 |
| innodb_buffer_pool_size | 134217728 |
| innodb_change_buffer_max_size | 25 |
| innodb_doublewrite_batch_size | 0 |
| innodb_ft_cache_size | 8000000 |
| innodb_ft_max_token_size | 84 |
| innodb_ft_min_token_size | 3 |
| innodb_ft_total_cache_size | 640000000 |
| innodb_log_buffer_size | 16777216 |
| innodb_log_file_size | 50331648 |
| innodb_log_write_ahead_size | 8192 |
| innodb_max_undo_log_size | 1073741824 |
| innodb_online_alter_log_max_size | 134217728 |
| innodb_page_size | 16384 |
| innodb_purge_batch_size | 300 |
| innodb_sort_buffer_size | 1048576 |
| innodb_sync_array_size | 1 |
| join_buffer_size | 262144 |
| key_buffer_size | 8388608 |
| key_cache_block_size | 1024 |
| large_page_size | 0 |
| max_binlog_cache_size | 18446744073709547520 |
| max_binlog_size | 1073741824 |
| max_binlog_stmt_cache_size | 18446744073709547520 |
| max_heap_table_size | 16777216 |
| max_join_size | 18446744073709551615 |
| max_relay_log_size | 0 |
| myisam_data_pointer_size | 6 |
| myisam_max_sort_file_size | 9223372036853727232 |
| myisam_mmap_size | 18446744073709551615 |
| myisam_sort_buffer_size | 8388608 |
| ngram_token_size | 2 |
| optimizer_trace_max_mem_size | 1048576 |
| parser_max_mem_size | 18446744073709551615 |
| performance_schema_accounts_size | -1 |
| performance_schema_digests_size | 10000 |
| performance_schema_error_size | 4890 |
| performance_schema_events_stages_history_long_size | 10000 |
| performance_schema_events_stages_history_size | 10 |
| performance_schema_events_statements_history_long_size | 10000 |
| performance_schema_events_statements_history_size | 10 |
| performance_schema_events_transactions_history_long_size | 10000 |
| performance_schema_events_transactions_history_size | 10 |
| performance_schema_events_waits_history_long_size | 10000 |
| performance_schema_events_waits_history_size | 10 |
| performance_schema_hosts_size | -1 |
| performance_schema_session_connect_attrs_size | 512 |
| performance_schema_setup_actors_size | -1 |
| performance_schema_setup_objects_size | -1 |
| performance_schema_users_size | -1 |
| preload_buffer_size | 32768 |
| profiling_history_size | 15 |
| query_alloc_block_size | 8192 |
| query_prealloc_size | 8192 |
| range_alloc_block_size | 4096 |
| range_optimizer_max_mem_size | 8388608 |
| read_buffer_size | 131072 |
| read_rnd_buffer_size | 262144 |
| rpl_read_size | 8192 |
| select_into_buffer_size | 131072 |
| slave_pending_jobs_size_max | 134217728 |
| sort_buffer_size | 262144 |
| thread_cache_size | 9 |
| tmp_table_size | 16777216 |
| transaction_alloc_block_size | 8192 |
| transaction_prealloc_size | 4096 |

Reducing innodb_buffer_pool_size rather than increasing it could solve memory over-usage.
My machine hardware has 1.7 G memory, MySQL use about 23% memory on startup.
When setting innodb_buffer_pool_size = 100M, after inserting 400,000 rows of data using Python MySQL Connector, MySQL memory usage climbs to 30% and doesn't go down even if the program ended.
However, if I set innodb_buffer_pool_size = 50M, the memory still climbs to a 30% level when the Python program running, but soon goes down if the program ended.
Pure experimental finding, maybe some veteran could explain the back reasoning.

As #BoarGules suggested, this solution proved to be a savior for me in mysql out of memory problem:
https://stackoverflow.com/a/7137270/7977550
MySQL 5.7, 6gb ram free , use mysql.connector and "values" in chunks(values) is 2799617 item list.
def chunks(data, rows=10000):
""" Divides the data into 10000 rows each """
for i in range(0, len(data), rows):
yield data[i:i+rows]
query = "INSERT INTO table VALUES (%s);"
div_values = chunks(values) # divide into 10000 rows each
for values in div_values:
cursor.executemany(query, list(zip(values)))

Web scraping: how to iterate over all the pages from a url and store the result in a table

I am not able to resolve this loop. I want to iterate over all pages from the URL and put all the names in a table. There are 102 pages. Thank you in advance.
[from bs4 import BeautifulSoup
import re
import requests
url = "https://www.prokerala.com/kids/baby-names/hindu/"
r = requests.get(url)
soup = BeautifulSoup(r.content, "html.parser")
next_page = soup.find('a', {'class': 'next'}).get('href')
l=(f'{next_page}')
new_url=url+l
link=soup.find('span', {'class' : \['a name-details girl-names', 'a name-details boy-names'\] })
print("href='>%s'>%s</a>" % (link.get("href"), link.text))
while True:
r = requests.get(new_url)
soup = BeautifulSoup(r.content, "html.parser")
next_page = soup.find('a', {'class': 'next'}).get('href')
link=soup.find('span', {'class' : \['a name-details girl-names', 'a name-details boy-names'\] })
print("href='>%s'>%s</a>" % (link.get("href"), link.text))
l=(f'{next_page}')
new_url=url+l
if BeautifulSoup(r.content, "html.parser")is True:
break][1]
[1]: https://i.stack.imgur.com/SgM4I.png

from bs4 import BeautifulSoup
import requests
import re
from prettytable import PrettyTable
def main(url):
with requests.Session() as req:
p = PrettyTable()
p.field_names = ['Name', 'Url']
for item in range(1, 3):
r = req.get(url.format(item))
soup = BeautifulSoup(r.content, 'html.parser')
for item in soup.findAll("span", class_=re.compile("a name")):
p.add_row([item.a.text, f'{url[:25]}{item.a.get("href")}'])
print(p)
main("https://www.prokerala.com/kids/baby-names/hindu/page-{}.html")
Output:
+---------------+--------------------------------------------------------------------+
| Name | Url |
+---------------+--------------------------------------------------------------------+
| Aabha | https://www.prokerala.com/kids/baby-names/aabha-13597.html |
| Aabheer | https://www.prokerala.com/kids/baby-names/aabheer-2.html |
| Aachal | https://www.prokerala.com/kids/baby-names/aachal-20815.html |
| Aadarsh | https://www.prokerala.com/kids/baby-names/aadarsh-3.html |
| Aadarshini | https://www.prokerala.com/kids/baby-names/aadarshini-20997.html |
| Aadav | https://www.prokerala.com/kids/baby-names/aadav-21329.html |
| Aadavan | https://www.prokerala.com/kids/baby-names/aadavan-19505.html |
| Aadesh | https://www.prokerala.com/kids/baby-names/aadesh-4.html |
| Aadhan | https://www.prokerala.com/kids/baby-names/aadhan-5.html |
| Aadharsha | https://www.prokerala.com/kids/baby-names/aadharsha-20812.html |
| Aadhik | https://www.prokerala.com/kids/baby-names/aadhik-19726.html |
| Aadhikesav | https://www.prokerala.com/kids/baby-names/aadhikesav-21003.html |
| Aadhinath | https://www.prokerala.com/kids/baby-names/aadhinath-21392.html |
| Aadhira | https://www.prokerala.com/kids/baby-names/aadhira-6.html |
| Aadhira | https://www.prokerala.com/kids/baby-names/aadhira-20603.html |
| Aadhith | https://www.prokerala.com/kids/baby-names/aadhith-21626.html |
| Aadhriti | https://www.prokerala.com/kids/baby-names/aadhriti-20601.html |
| Aadhunik | https://www.prokerala.com/kids/baby-names/aadhunik-21198.html |
| Aadhvitha | https://www.prokerala.com/kids/baby-names/aadhvitha-20571.html |
| Aadhya | https://www.prokerala.com/kids/baby-names/aadhya-21327.html |
| Aadi | https://www.prokerala.com/kids/baby-names/aadi-7.html |
| Aadidev | https://www.prokerala.com/kids/baby-names/aadidev-8.html |
| Aadilakshmi | https://www.prokerala.com/kids/baby-names/aadilakshmi-20995.html |
| Aadimoolan | https://www.prokerala.com/kids/baby-names/aadimoolan-20920.html |
| Aadinadh | https://www.prokerala.com/kids/baby-names/aadinadh-20614.html |
| Aadinath | https://www.prokerala.com/kids/baby-names/aadinath-10.html |
| Aadinathan | https://www.prokerala.com/kids/baby-names/aadinathan-24838.html |
| Aadipta | https://www.prokerala.com/kids/baby-names/aadipta-20922.html |
| Aadir | https://www.prokerala.com/kids/baby-names/aadir-11.html |
| Aadish | https://www.prokerala.com/kids/baby-names/aadish-20945.html |
| Aadishree | https://www.prokerala.com/kids/baby-names/aadishree-21389.html |
| Aadit | https://www.prokerala.com/kids/baby-names/aadit-12.html |
| Aaditey | https://www.prokerala.com/kids/baby-names/aaditey-20921.html |
| Aaditeya | https://www.prokerala.com/kids/baby-names/aaditeya-13.html |
| Aaditya | https://www.prokerala.com/kids/baby-names/aaditya-22619.html |
| Aadrika | https://www.prokerala.com/kids/baby-names/aadrika-20925.html |
| Aadvay | https://www.prokerala.com/kids/baby-names/aadvay-22288.html |
| Aadvik | https://www.prokerala.com/kids/baby-names/aadvik-19508.html |
| Aadya | https://www.prokerala.com/kids/baby-names/aadya-19557.html |
| Aadyasha | https://www.prokerala.com/kids/baby-names/aadyasha-20570.html |
| Aafreen | https://www.prokerala.com/kids/baby-names/aafreen-19244.html |
| Aagam | https://www.prokerala.com/kids/baby-names/aagam-16.html |
| Aaghosh | https://www.prokerala.com/kids/baby-names/aaghosh-22289.html |
| Aagney | https://www.prokerala.com/kids/baby-names/aagney-17.html |
| Aagneya | https://www.prokerala.com/kids/baby-names/aagneya-18.html |
| Aahan | https://www.prokerala.com/kids/baby-names/aahan-19.html |
| Aahana | https://www.prokerala.com/kids/baby-names/aahana-21194.html |
| Aahish | https://www.prokerala.com/kids/baby-names/aahish-22290.html |
| Aahlaad | https://www.prokerala.com/kids/baby-names/aahlaad-21.html |
| Aahva | https://www.prokerala.com/kids/baby-names/aahva-22.html |
| Aahvan | https://www.prokerala.com/kids/baby-names/aahvan-23857.html |
| Aakaar | https://www.prokerala.com/kids/baby-names/aakaar-23.html |
| Aakaash | https://www.prokerala.com/kids/baby-names/aakaash-24.html |
| Aakanksha | https://www.prokerala.com/kids/baby-names/aakanksha-19223.html |
| Aakar | https://www.prokerala.com/kids/baby-names/aakar-25.html |
| Aakarshan | https://www.prokerala.com/kids/baby-names/aakarshan-26.html |
| Aakash | https://www.prokerala.com/kids/baby-names/aakash-19650.html |
| Aalap | https://www.prokerala.com/kids/baby-names/aalap-27.html |
| Aalok | https://www.prokerala.com/kids/baby-names/aalok-28.html |
| Aamil | https://www.prokerala.com/kids/baby-names/aamil-17109.html |
| Aamod | https://www.prokerala.com/kids/baby-names/aamod-30.html |
| Aanan | https://www.prokerala.com/kids/baby-names/aanan-23674.html |
| Aanand | https://www.prokerala.com/kids/baby-names/aanand-31.html |
| Aanandhi | https://www.prokerala.com/kids/baby-names/aanandhi-19971.html |
| Aanandswarup | https://www.prokerala.com/kids/baby-names/aanandswarup-32.html |
| Aananthan | https://www.prokerala.com/kids/baby-names/aananthan-26697.html |
| Aanav | https://www.prokerala.com/kids/baby-names/aanav-19506.html |
| Aanchal | https://www.prokerala.com/kids/baby-names/aanchal-26403.html |
| Aandaleeb | https://www.prokerala.com/kids/baby-names/aandaleeb-33.html |
| Aanjaneya | https://www.prokerala.com/kids/baby-names/aanjaneya-34.html |
| Aanjay | https://www.prokerala.com/kids/baby-names/aanjay-23679.html |
| Aapt | https://www.prokerala.com/kids/baby-names/aapt-35.html |
| Aaraadhak | https://www.prokerala.com/kids/baby-names/aaraadhak-37.html |
| Aaradhya | https://www.prokerala.com/kids/baby-names/aaradhya-19272.html |
| Aaranyan | https://www.prokerala.com/kids/baby-names/aaranyan-23680.html |
| Aarathi | https://www.prokerala.com/kids/baby-names/aarathi-24839.html |
| Aarav | https://www.prokerala.com/kids/baby-names/aarav-38.html |
| Aardik | https://www.prokerala.com/kids/baby-names/aardik-26698.html |
| Aarhant | https://www.prokerala.com/kids/baby-names/aarhant-39.html |
| Aarin | https://www.prokerala.com/kids/baby-names/aarin-19496.html |
| Aarit | https://www.prokerala.com/kids/baby-names/aarit-41.html |
| Aariv | https://www.prokerala.com/kids/baby-names/aariv-19507.html |
| Aarna | https://www.prokerala.com/kids/baby-names/aarna-24840.html |
| Aarnav | https://www.prokerala.com/kids/baby-names/aarnav-44.html |
| Aarodeep | https://www.prokerala.com/kids/baby-names/aarodeep-26700.html |
| Aaroha | https://www.prokerala.com/kids/baby-names/aaroha-21497.html |
| Aaromal | https://www.prokerala.com/kids/baby-names/aaromal-26701.html |
| Aarpit | https://www.prokerala.com/kids/baby-names/aarpit-46.html |
| Aarsh | https://www.prokerala.com/kids/baby-names/aarsh-47.html |
| Aarshabh | https://www.prokerala.com/kids/baby-names/aarshabh-23864.html |
| Aarth | https://www.prokerala.com/kids/baby-names/aarth-48.html |
| Aarthi | https://www.prokerala.com/kids/baby-names/aarthi-24320.html |
| Aarti | https://www.prokerala.com/kids/baby-names/aarti-24228.html |
| Aarul | https://www.prokerala.com/kids/baby-names/aarul-19501.html |
| Aarunya | https://www.prokerala.com/kids/baby-names/aarunya-21502.html |
| Aarush | https://www.prokerala.com/kids/baby-names/aarush-49.html |
| Aarushi | https://www.prokerala.com/kids/baby-names/aarushi-50.html |
| Aarya | https://www.prokerala.com/kids/baby-names/aarya-19111.html |
| Aaryah | https://www.prokerala.com/kids/baby-names/aaryah-13617.html |
| Aaryan | https://www.prokerala.com/kids/baby-names/aaryan-51.html |
| Aaryana | https://www.prokerala.com/kids/baby-names/aaryana-13618.html |
| Aaryanna | https://www.prokerala.com/kids/baby-names/aaryanna-13619.html |
| Aashadhar | https://www.prokerala.com/kids/baby-names/aashadhar-21503.html |
| Aashank | https://www.prokerala.com/kids/baby-names/aashank-53.html |
| Aashay | https://www.prokerala.com/kids/baby-names/aashay-54.html |
| Aashi | https://www.prokerala.com/kids/baby-names/aashi-24841.html |
| Aashirya | https://www.prokerala.com/kids/baby-names/aashirya-19779.html |
| Aashish | https://www.prokerala.com/kids/baby-names/aashish-55.html |
| Aashman | https://www.prokerala.com/kids/baby-names/aashman-56.html |
| Aashna | https://www.prokerala.com/kids/baby-names/aashna-19991.html |
| Aashutosh | https://www.prokerala.com/kids/baby-names/aashutosh-57.html |
| Aastha | https://www.prokerala.com/kids/baby-names/aastha-19567.html |
| Aastik | https://www.prokerala.com/kids/baby-names/aastik-60.html |
| Aathira | https://www.prokerala.com/kids/baby-names/aathira-24842.html |
| Aatish | https://www.prokerala.com/kids/baby-names/aatish-62.html |
| Aatmaj | https://www.prokerala.com/kids/baby-names/aatmaj-63.html |
| Aatreya | https://www.prokerala.com/kids/baby-names/aatreya-64.html |
| Aayan | https://www.prokerala.com/kids/baby-names/aayan-65.html |
| Aayu | https://www.prokerala.com/kids/baby-names/aayu-66.html |
| Aayush | https://www.prokerala.com/kids/baby-names/aayush-24843.html |
| Aayushi | https://www.prokerala.com/kids/baby-names/aayushi-26998.html |
| Aayushmaan | https://www.prokerala.com/kids/baby-names/aayushmaan-67.html |
| Abadhya | https://www.prokerala.com/kids/baby-names/abadhya-19727.html |
| Abeesh | https://www.prokerala.com/kids/baby-names/abeesh-19724.html |
| Abeesht | https://www.prokerala.com/kids/baby-names/abeesht-20923.html |
| Aben | https://www.prokerala.com/kids/baby-names/aben-19257.html |
| Abha | https://www.prokerala.com/kids/baby-names/abha-7340.html |
| Abhaiveer | https://www.prokerala.com/kids/baby-names/abhaiveer-23649.html |
| Abhanja | https://www.prokerala.com/kids/baby-names/abhanja-23670.html |
| Abhas | https://www.prokerala.com/kids/baby-names/abhas-19723.html |
| Abhat | https://www.prokerala.com/kids/baby-names/abhat-19722.html |
| Abhati | https://www.prokerala.com/kids/baby-names/abhati-18373.html |
| Abhay | https://www.prokerala.com/kids/baby-names/abhay-154.html |
| Abhaya | https://www.prokerala.com/kids/baby-names/abhaya-7341.html |
| Abhayan | https://www.prokerala.com/kids/baby-names/abhayan-22297.html |
| Abhayananda | https://www.prokerala.com/kids/baby-names/abhayananda-18027.html |
| Abhayaprada | https://www.prokerala.com/kids/baby-names/abhayaprada-18028.html |
| Abhayd | https://www.prokerala.com/kids/baby-names/abhayd-22295.html |
| Abhayd | https://www.prokerala.com/kids/baby-names/abhayd-22296.html |
| Abhaysimha | https://www.prokerala.com/kids/baby-names/abhaysimha-23650.html |
| Abheek | https://www.prokerala.com/kids/baby-names/abheek-155.html |
| Abhey | https://www.prokerala.com/kids/baby-names/abhey-22292.html |
| Abhi | https://www.prokerala.com/kids/baby-names/abhi-156.html |
| Abhibhava | https://www.prokerala.com/kids/baby-names/abhibhava-18029.html |
| Abhicandra | https://www.prokerala.com/kids/baby-names/abhicandra-18030.html |
| Abhidha | https://www.prokerala.com/kids/baby-names/abhidha-18374.html |
| Abhidhya | https://www.prokerala.com/kids/baby-names/abhidhya-18375.html |
| Abhidi | https://www.prokerala.com/kids/baby-names/abhidi-18031.html |
| Abhigita | https://www.prokerala.com/kids/baby-names/abhigita-23855.html |
| Abhigna | https://www.prokerala.com/kids/baby-names/abhigna-24844.html |
| Abhigyaan | https://www.prokerala.com/kids/baby-names/abhigyaan-23671.html |
| Abhihita | https://www.prokerala.com/kids/baby-names/abhihita-18032.html |
| Abhijaat | https://www.prokerala.com/kids/baby-names/abhijaat-157.html |
| Abhijat | https://www.prokerala.com/kids/baby-names/abhijat-18787.html |
| Abhijay | https://www.prokerala.com/kids/baby-names/abhijay-158.html |
| Abhijaya | https://www.prokerala.com/kids/baby-names/abhijaya-18033.html |
| Abhijeet | https://www.prokerala.com/kids/baby-names/abhijeet-17122.html |
| Abhiji | https://www.prokerala.com/kids/baby-names/abhiji-25946.html |
| Abhijit | https://www.prokerala.com/kids/baby-names/abhijit-159.html |
| Abhijith | https://www.prokerala.com/kids/baby-names/abhijith-23839.html |
| Abhijun | https://www.prokerala.com/kids/baby-names/abhijun-160.html |
| Abhijvala | https://www.prokerala.com/kids/baby-names/abhijvala-18034.html |
| Abhik | https://www.prokerala.com/kids/baby-names/abhik-161.html |
| Abhikama | https://www.prokerala.com/kids/baby-names/abhikama-19719.html |
| Abhikansh | https://www.prokerala.com/kids/baby-names/abhikansh-19720.html |
| Abhilasa | https://www.prokerala.com/kids/baby-names/abhilasa-18376.html |
| Abhilash | https://www.prokerala.com/kids/baby-names/abhilash-162.html |
| Abhilasha | https://www.prokerala.com/kids/baby-names/abhilasha-7342.html |
| Abhim | https://www.prokerala.com/kids/baby-names/abhim-19721.html |
| Abhimand | https://www.prokerala.com/kids/baby-names/abhimand-18035.html |
| Abhimani | https://www.prokerala.com/kids/baby-names/abhimani-18036.html |
| Abhimanyu | https://www.prokerala.com/kids/baby-names/abhimanyu-163.html |
| Abhimanyusuta | https://www.prokerala.com/kids/baby-names/abhimanyusuta-18037.html |
| Abhimoda | https://www.prokerala.com/kids/baby-names/abhimoda-18038.html |
| Abhin | https://www.prokerala.com/kids/baby-names/abhin-164.html |
| Abhinabhas | https://www.prokerala.com/kids/baby-names/abhinabhas-18039.html |
| Abhinand | https://www.prokerala.com/kids/baby-names/abhinand-23672.html |
| Abhinanda | https://www.prokerala.com/kids/baby-names/abhinanda-18040.html |
| Abhinandan | https://www.prokerala.com/kids/baby-names/abhinandan-165.html |
| Abhinandana | https://www.prokerala.com/kids/baby-names/abhinandana-18788.html |
| Abhinatha | https://www.prokerala.com/kids/baby-names/abhinatha-18041.html |
| Abhinav | https://www.prokerala.com/kids/baby-names/abhinav-166.html |
| Abhinava | https://www.prokerala.com/kids/baby-names/abhinava-18042.html |
| Abhinavin | https://www.prokerala.com/kids/baby-names/abhinavin-19729.html |
| Abhinay | https://www.prokerala.com/kids/baby-names/abhinay-168.html |
| Abhinaya | https://www.prokerala.com/kids/baby-names/abhinaya-23651.html |
| Abhineet | https://www.prokerala.com/kids/baby-names/abhineet-169.html |
| Abhinit | https://www.prokerala.com/kids/baby-names/abhinit-170.html |
| Abhinithi | https://www.prokerala.com/kids/baby-names/abhinithi-18377.html |
| Abhinivesh | https://www.prokerala.com/kids/baby-names/abhinivesh-171.html |
| Abhiprithi | https://www.prokerala.com/kids/baby-names/abhiprithi-18378.html |
| Abhir | https://www.prokerala.com/kids/baby-names/abhir-172.html |
| Abhiraam | https://www.prokerala.com/kids/baby-names/abhiraam-173.html |
| Abhiraj | https://www.prokerala.com/kids/baby-names/abhiraj-174.html |
| Abhiraksh | https://www.prokerala.com/kids/baby-names/abhiraksh-19733.html |
| Abhirakshit | https://www.prokerala.com/kids/baby-names/abhirakshit-19732.html |
| Abhiram | https://www.prokerala.com/kids/baby-names/abhiram-19728.html |
| Abhirami | https://www.prokerala.com/kids/baby-names/abhirami-24745.html |
| Abhirath | https://www.prokerala.com/kids/baby-names/abhirath-175.html |
| Abhirathi | https://www.prokerala.com/kids/baby-names/abhirathi-18379.html |
+---------------+--------------------------------------------------------------------+

calculate difference of column values for sets of row indices which are not successive in pandas

Say I have the following table:
+----+---------+--------+---------+---------+---------+---------+-----------+-----------+-----------+----------+-----------+------------+------------+---------+---+
| 1 | 0.72694 | 1.4742 | 0.32396 | 0.98535 | 1 | 0.83592 | 0.0046566 | 0.0039465 | 0.04779 | 0.12795 | 0.016108 | 0.0052323 | 0.00027477 | 1.1756 | 1 |
| 2 | 0.74173 | 1.5257 | 0.36116 | 0.98152 | 0.99825 | 0.79867 | 0.0052423 | 0.0050016 | 0.02416 | 0.090476 | 0.0081195 | 0.002708 | 7.48E-05 | 0.69659 | 1 |
| 3 | 0.76722 | 1.5725 | 0.38998 | 0.97755 | 1 | 0.80812 | 0.0074573 | 0.010121 | 0.011897 | 0.057445 | 0.0032891 | 0.00092068 | 3.79E-05 | 0.44348 | 1 |
| 4 | 0.73797 | 1.4597 | 0.35376 | 0.97566 | 1 | 0.81697 | 0.0068768 | 0.0086068 | 0.01595 | 0.065491 | 0.0042707 | 0.0011544 | 6.63E-05 | 0.58785 | 1 |
| 5 | 0.82301 | 1.7707 | 0.44462 | 0.97698 | 1 | 0.75493 | 0.007428 | 0.010042 | 0.0079379 | 0.045339 | 0.0020514 | 0.00055986 | 2.35E-05 | 0.34214 | 1 |
| 7 | 0.82063 | 1.7529 | 0.44458 | 0.97964 | 0.99649 | 0.7677 | 0.0059279 | 0.0063954 | 0.018375 | 0.080587 | 0.0064523 | 0.0022713 | 4.15E-05 | 0.53904 | 1 |
| 8 | 0.77982 | 1.6215 | 0.39222 | 0.98512 | 0.99825 | 0.80816 | 0.0050987 | 0.0047314 | 0.024875 | 0.089686 | 0.0079794 | 0.0024664 | 0.00014676 | 0.66975 | 1 |
| 9 | 0.83089 | 1.8199 | 0.45693 | 0.9824 | 1 | 0.77106 | 0.0060055 | 0.006564 | 0.0072447 | 0.040616 | 0.0016469 | 0.00038812 | 3.29E-05 | 0.33696 | 1 |
| 11 | 0.7459 | 1.4927 | 0.34116 | 0.98296 | 1 | 0.83088 | 0.0055665 | 0.0056395 | 0.0057679 | 0.036511 | 0.0013313 | 0.00030872 | 3.18E-05 | 0.25026 | 1 |
| 12 | 0.79606 | 1.6934 | 0.43387 | 0.98181 | 1 | 0.76985 | 0.0077992 | 0.011071 | 0.013677 | 0.057832 | 0.0033334 | 0.00081648 | 0.00013855 | 0.49751 | 1 |
+----+---------+--------+---------+---------+---------+---------+-----------+-----------+-----------+----------+-----------+------------+------------+---------+---+
I have two sets of row indices :
set1 = [1,3,5,8,9]
set2 = [2,4,7,10,10]
Note : Here, I have indicated the first row with index value 1. Length of both sets shall always be same.
What I am looking for is a fast and pythonic way to get the difference of column values for corresponding row indices, that is : difference of 1-2,3-4,5-7,8-10,9-10.
For this example, my resultant dataframe is the following:
+---+---------+--------+---------+---------+---------+---------+-----------+-----------+-----------+----------+-----------+------------+------------+---------+---+
| 1 | 0.01479 | 0.0515 | 0.0372 | 0.00383 | 0.00175 | 0.03725 | 0.0005857 | 0.0010551 | 0.02363 | 0.037474 | 0.0079885 | 0.0025243 | 0.00019997 | 0.47901 | 0 |
| 1 | 0.02925 | 0.1128 | 0.03622 | 0.00189 | 0 | 0.00885 | 0.0005805 | 0.0015142 | 0.004053 | 0.008046 | 0.0009816 | 0.00023372 | 0.0000284 | 0.14437 | 0 |
| 3 | 0.04319 | 0.1492 | 0.0524 | 0.00814 | 0.00175 | 0.05323 | 0.0023293 | 0.0053106 | 0.0169371 | 0.044347 | 0.005928 | 0.00190654 | 0.00012326 | 0.32761 | 0 |
| 3 | 0.03483 | 0.1265 | 0.02306 | 0.00059 | 0 | 0.00121 | 0.0017937 | 0.004507 | 0.0064323 | 0.017216 | 0.0016865 | 0.00042836 | 0.00010565 | 0.16055 | 0 |
| 1 | 0.05016 | 0.2007 | 0.09271 | 0.00115 | 0 | 0.06103 | 0.0022327 | 0.0054315 | 0.0079091 | 0.021321 | 0.0020021 | 0.00050776 | 0.00010675 | 0.24725 | 0 |
+---+---------+--------+---------+---------+---------+---------+-----------+-----------+-----------+----------+-----------+------------+------------+---------+---+
My resultant difference values are absolute here.
I cant apply diff(), since the row indices may not be consecutive.
I am currently achieving my aim via looping through sets.
Is there a pandas trick to do this?

Use loc based indexing -
df.loc[set1].values - df.loc[set2].values
Ensure that len(set1) is equal to len(set2). Also, keep in mind setX is a counter-intuitive name for list objects.

You need to select by data reindexing and then subtract:
df = df.reindex(set1) - df.reindex(set2).values
loc or iloc will raise a future warning, since passing list-likes to .loc or [] with any missing label will raise KeyError in the future.

In short, try the following:
df.iloc[::2].values - df.iloc[1::2].values
PS:
Or alternatively, if (like in your question the indices follow no simple rule):
df.iloc[set1].values - df.iloc[set2].values

Develop Reference

Python is a programming language that lets you work quickly and integrate systems more effectively.

python ThreadPoolExecutor memory leak issues - python

Related

How to classify the image data from the folders?

Updating values onto an ASCII art without changing the format [closed]

Cursor Insert return out of memory errors

Web scraping: how to iterate over all the pages from a url and store the result in a table

calculate difference of column values for sets of row indices which are not successive in pandas

Categories

Resources