Updating values onto an ASCII art without changing the format [closed] - python
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 months ago.
Improve this question
I am exploring ways on bash as well as on Python to print out values onto an ASCII art for improving the readability.
The one difficulty is to update values without changing the format of the art.
The ascii art looks something like this:
========================================================
| | | | | |
| | | | | |
| CPU | | GPU | | HDD |
| | | | | |
| | | | | |
| ${CPU_W}W | | | | |
| ${CPU_Freq}MHz| | | |Avail Mem${size}G|
| | |${GPU_W}W | | Used Mem${size}G|
| | |${GPU}Mhz | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
========================================================
So far, I was able to prevent the format from changing on Bash. But doing this, is not allowing me to change the values.
cat << "EOF"
========================================================
| | | | | |
| | | | | |
| CPU | | GPU | | HDD |
| | | | | |
| | | | | |
| ${CPU_W}W | | | | |
| ${CPU_Freq}MHz| | | |Avail Mem${size}G|
| | |${GPU_W}W | | Used Mem${size}G|
| | |${GPU}Mhz | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
========================================================
EOF
Without cat << "EOF" ...ascii art.... EOF, I can update the values but the format keeps changing .
Is there anyway to keep the same format even with the values changing? Thanks in advance.
I'm not exactly sure what you're trying to achieve... so?
f = """
========================================================
| | | | | |
| | | | | |
| CPU | | GPU | | HDD |
| | | | | |
| | | | | |
|{cpuW:>15}| | | | |
|{cpuF:>15}| | | |Avail Mem{ramA:>8}|
| | |{gpuW:>10}| | Used Mem{ramU:>8}|
| | |{gpuF:>10}| | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
========================================================
"""
def main():
cpuW = 12.2
cpuF = 2354
gpuW = 15.2
gpuF = 789
ramA = 16.1
ramU = 12.2
d = {
'cpuW': f'{cpuW:.1f} W ',
'cpuF': f'{cpuF:d} MHz ',
'gpuW': f'{gpuW:.1f} W ',
'gpuF': f'{gpuF:d} MHz ',
'ramA': f' {ramA:.1f} GB',
'ramU': f' {ramU:.1f} GB',
}
out = f.format_map(d)
print(out)
if __name__ == '__main__':
main()
Output is:
========================================================
| | | | | |
| | | | | |
| CPU | | GPU | | HDD |
| | | | | |
| | | | | |
| 12.2 W | | | | |
| 2354 MHz | | | |Avail Mem 16.1 GB|
| | | 15.2 W | | Used Mem 12.2 GB|
| | | 789 MHz | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
========================================================
In python, you can use the format and str.ljust method.
format(value[, format_spec])
Convert a value to a “formatted” representation, as controlled by format_spec. The interpretation of format_spec will depend on the type of the value argument; however, there is a standard formatting syntax that is used by most built-in types: Format Specification Mini-Language.
It depends on what you want the notation to be (e.g how many zeros, ecc...).
here is an example:
>>> characters = 10
>>> format(32,".2E").zfill(characters)
'003.20E+01'
>>> #first number is the minimum number of characters
>>> format(32,"{0}.2E".format(characters))
' 3.20E+01'
or, with fstrings:
>>> f"{32:.2E}"
'3.20E+01'
Example on how to use it in ascii art:
>>> def create_ascii_art(CPU_W,CPU_freq,GPU_W,
... GPU,Free_Mem,Used_Mem):
... CPU_W,CPU_freq,GPU_W,GPU,Free_Mem,Used_Mem = map(float,(CPU_W,CPU_freq,GPU_W,GPU,Free_Mem,Used_Mem))
... return f'''
...========================================================
...| | | | | |
...| | | | | |
...| CPU | | GPU | | HDD |
...| | | | | |
...| | | | | |
...|{CPU_W:12.2E}W | | | | |
...|{CPU_freq:12.2E}MHz| | | |Avail Mem{Free_Mem:7.3}G|
...| | |{GPU_W:8.2E}W | | Used Mem{Used_Mem:7.3}G|
...| | |{GPU:6.0E}Mhz | | |
...| | | | | |
...| | | | | |
...| | | | | |
...| | | | | |
...========================================================'''
>>> print(create_ascii_art(1,1,1,1,1,1))
output:
========================================================
| | | | | |
| | | | | |
| CPU | | GPU | | HDD |
| | | | | |
| | | | | |
| 1.00E+00W | | | | |
| 1.00E+00MHz| | | |Avail Mem 1.0G|
| | |1.00E+00W | | Used Mem 1.0G|
| | | 1E+00Mhz | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
========================================================
The "format" does not change; the display width of the strings in the here document depends on the values of the variables. You'll want to make sure the values are padded to the expected width (or truncated, if necessary).
A slightly tortured way to accomplish this is to pad the actual values:
#!/bin/bash
:
printf -v CPU_W '%8s' "$CPU_W"
printf -v CPU_Freq '%11s' "$CPU_Freq"
printf -v GPU_W '%8s' "$GPU_W"
printf -v GPU '%6s' "$GPU"
printf -v size '%7s' "$size"
cat <<EOF
========================================================
| | | | | |
| | | | | |
| CPU | | GPU | | HDD |
| | | | | |
| | | | | |
| ${CPU_W}W | | | | |
| ${CPU_Freq}MHz| | | |Avail Mem${size}G|
| | |${GPU_W}W | | Used Mem${size}G|
| | |${GPU}Mhz | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
========================================================
EOF
Probably a better solution altogether is to use printf directly to format the output. (Personally, I would perceive a notable reduction in my blood pressure if you removed at least half of the repetitive "ASCII art", which would somewhat tidy up the necessary code, too.)
We can't know what values these variables contain; if they are integers, maybe experiment with i instead of s for the printf format code, or correspondingly f or perhaps g for floating-point values.
If you need truncation, try %8.8 instead of %8, etc.
An altogether nicer approach is to use a library for producing this format. I can't recommend any particular one for Bash; for Python, look at tabulate.
Related
How to classify the image data from the folders?
It is our folder tree. dataset/ | | |-----train/ | | | |--dog/ | | | | | |--German Shepherd/ | | | | | | | |-- image1.png | | | |-- image2.png | | | |-- ... | | | | | |--Bulldog/ | | |--Labrador Retriever/ | | |--Poodle/ | | |--Chihuahua/ | | |--... | | | |--cat/ | | | | | |--Siamese cat/ | | |--British Shorthair/ | | |--Maine Coon/ | | |--Persian cat/ | | |--Ragdoll/ | | |--... | |-----test/ | |--... how can be classify it with tensorflow-python ? I am not using any pre-trained model then how can be classify it ?
python ThreadPoolExecutor memory leak issues
I'm trying to debug a memory leak in my application, and I think I managed to reduce it to this minimal example: from typing import Deque import gc import os import psutil from concurrent.futures import ThreadPoolExecutor process = psutil.Process(os.getpid()) # we'll use this to track memory usage executor = ThreadPoolExecutor() # generic executor n = 100 # number of tasks to run concurrently at any time prev = 0 # previous maximum memory usage m = 10_000 def do(): # an arbitrary function that returns a large value return list(range(m)) # initialize a deque that can track all our running futures futures = Deque(executor.submit(do) for _ in range(n)) i = 0 # future counter while futures: f = futures.popleft() # pop one of the futures (removing its reference from the deque) result = f.result() del result, f # delete the future (the future is definitely removed now) # check the memory usage and print a message if it is a new record new_mem = process.memory_info().rss if new_mem > prev: print(f"{i}: {new_mem:,} bytes, {len(futures)} tasks pending") prev = new_mem # enqueue a new task futures.append(executor.submit(do)) gc.collect() i+=1 Essentially what we do here is keep a rotating queue of tasks that return a large value, ensuring that there are no more than n tasks running concurrently at any time. We would expect, after about max_workers tasks have completed, that top memory usage would reach a plateau since none of the data from the futures is being preserved. However, on python 3.7.12, we see that memory usage increases continually and plateaus sometimes even on the thousandth task! This outcome is preserved even when we change do() to return a numpy array, when we limit the max_workers to a smaller number, and when we reduce n. EDIT: I added a table measuring the pear memory usage (on my machine) of a process running this code by iteration count, for different values of m | peak memory usage after N iterations | m=1,000 | m=5,000 | m=10,000 | |--------------------------------------|------------|------------|------------| | 500 | 16,961,536 | 37,662,720 | 61,689,856 | | 1,000 | 17,408,000 | 38,981,632 | 62,132,224 | | 1,500 | 17,760,256 | 39,374,848 | 62,164,992 | | 2,000 | 18,063,360 | 40,177,664 | 62,832,640 | | 2,500 | 18,219,008 | 40,845,312 | 62,832,640 | | 3,000 | 18,374,656 | 40,931,328 | 63,156,224 | | 3,500 | 18,493,440 | 41,492,480 | 63,328,256 | | 4,000 | 18,604,032 | 41,586,688 | 63,328,256 | | 4,500 | 18,673,664 | 41,586,688 | 63,328,256 | | 5,000 | 18,763,776 | 41,586,688 | 63,328,256 | | 5,500 | 18,800,640 | 41,586,688 | 63,328,256 | | 6,000 | 18,853,888 | 41,615,360 | 63,328,256 | | 6,500 | 18,898,944 | 41,660,416 | 63,328,256 | | 7,000 | 18,931,712 | 41,795,584 | 63,328,256 | | 7,500 | 18,952,192 | 41,930,752 | 63,328,256 | | 8,000 | 18,952,192 | 42,061,824 | 63,328,256 | | 8,500 | 18,993,152 | 42,377,216 | 63,361,024 | | 9,000 | 19,046,400 | 42,463,232 | 63,528,960 | | 9,500 | 19,075,072 | 42,463,232 | 63,528,960 | | 10,000 | 19,075,072 | 42,463,232 | 63,528,960 | | 10,500 | 19,083,264 | 42,463,232 | 63,533,056 | | 11,000 | 19,083,264 | 42,463,232 | 63,664,128 | | 11,500 | 19,128,320 | 42,553,344 | 63,664,128 | | 12,000 | 19,185,664 | 42,553,344 | 63,664,128 | | 12,500 | 19,185,664 | 42,553,344 | 63,664,128 | | 13,000 | 19,185,664 | 42,553,344 | 63,664,128 | | 13,500 | 19,193,856 | 42,553,344 | 64,167,936 | | 14,000 | 19,193,856 | 42,594,304 | 64,167,936 | | 14,500 | 19,193,856 | 42,594,304 | 64,167,936 | | 15,000 | 19,202,048 | 42,594,304 | 64,249,856 | | 15,500 | 19,202,048 | 42,594,304 | 64,249,856 | | 16,000 | 19,222,528 | 42,594,304 | 64,462,848 | | 16,500 | 19,353,600 | 42,594,304 | 64,499,712 | | 17,000 | 19,361,792 | 42,635,264 | 64,499,712 | | 17,500 | 19,415,040 | 42,725,376 | 64,499,712 | | 18,000 | 19,423,232 | 42,766,336 | 64,499,712 | | 18,500 | 19,431,424 | 42,811,392 | 64,499,712 | | 19,000 | 19,476,480 | 42,811,392 | 64,499,712 | | 19,500 | 19,537,920 | 42,811,392 | 64,520,192 | | 20,000 | 19,607,552 | 42,856,448 | 65,044,480 | | 20,500 | 19,607,552 | 42,946,560 | 65,044,480 | | 21,000 | 19,636,224 | 43,110,400 | 65,044,480 | | 21,500 | 19,644,416 | 43,110,400 | 65,044,480 | | 22,000 | 19,652,608 | 43,110,400 | 65,044,480 | | 22,500 | 19,722,240 | 43,110,400 | 65,044,480 | | 23,000 | 19,746,816 | 43,110,400 | 65,044,480 | | 23,500 | 19,746,816 | 43,110,400 | 65,044,480 | | 24,000 | 19,746,816 | 43,110,400 | 65,044,480 | | 24,500 | 19,775,488 | 43,110,400 | 65,044,480 | | 25,000 | 19,800,064 | 43,110,400 | 65,044,480 | | 25,500 | 19,820,544 | 43,110,400 | 65,044,480 | | 26,000 | 19,886,080 | 43,110,400 | 65,044,480 | | 26,500 | 19,886,080 | 43,118,592 | 65,044,480 | | 27,000 | 19,894,272 | 43,118,592 | 65,044,480 | | 27,500 | 19,939,328 | 43,208,704 | 65,044,480 | | 28,000 | 19,947,520 | 43,208,704 | 65,044,480 | | 28,500 | 20,058,112 | 43,208,704 | 65,044,480 | | 29,000 | 20,086,784 | 43,208,704 | 65,044,480 | | 29,500 | 20,086,784 | 43,208,704 | 65,044,480 | | 30,000 | 20,086,784 | 43,208,704 | 65,044,480 |
Control Windows app (HMA VPN) using PyWinAuto
I am trying to write Python code to request my vpn software to connect / disconnect upon certain conditions. I use HMA VPN and am running on Windows 10. Python 3.9.5. The PyWinAuto module would appear to have the required functionality, but I cannot get it to work correctly. I'm sure it's my coding, not the module itself... ;) I'm fairly sure I'm referring to the control incorrectly when I call it with the click() method, but can anyone suggest the correct approach, given the dump_tree that I have included? (I have tried tens of variations based based on the code snippets in the Getting Started guide, but no joy). I'm looking to be able to toggle Button4 in GroupBox2, which is named either "Connect" or "Disconnect", depending on the current state of the VPN. I also want to be able to click Button5, "Change IP Address", but once I figure out one button, the other should be easy. Any thoughts where I am going wrong? My code snippet is as follows: from pywinauto import Desktop, Application Desktop(backend='uia').HMAVPN.dump_tree() app = Application().start('C:\Program Files\Privax\HMA VPN\Vpn.exe', timeout=5) dlg_spec = app.window(title='HMA VPN') dlg_spec.AvastChromiumWindow.Pane2.Document.GroupBox2.Button4.click() The resulting output is as follows: Control Identifiers: Dialog - 'HMA VPN' (L-149, T68, R551, B638) ['HMA VPNDialog', 'HMA VPN', 'Dialog'] child_window(title="HMA VPN", control_type="Window") | | Pane - 'AvastChromiumWindow' (L-149, T68, R551, B638) | ['Pane', 'AvastChromiumWindowPane', 'AvastChromiumWindow', 'Pane0', 'Pane1'] | child_window(title="AvastChromiumWindow", control_type="Pane") | | | | Pane - '' (L-149, T68, R551, B638) | | ['Pane2', '\ue01bPane', '\ue01bPane0', '\ue01bPane1'] | | | | | | Document - ' Original IP 91.90.253.244 New IP 80.255.5.198 Original IP 91.90.253.244 Location Germany, Berlin ' (L-149, T68, R551, B638) | | | ['\ue01b \ue014 \ue007 Original IP \xa0 91.90.253.244 New IP \xa0 80.255.5.198 \ue019 Original IP \xa0 91.90.253.244 \xa0 Location Germany, Berlin \ue005 Document', 'Document', '\ue01b \ue014 \ue007 Original IP \xa0 91.90.253.244 New IP \xa0 80.255.5.198 \ue019 Original IP \xa0 91.90.253.244 \xa0 Location Germany, Berlin \ue005 '] | | | child_window(title=" Original IP 91.90.253.244 New IP 80.255.5.198 Original IP 91.90.253.244 Location Germany, Berlin ", auto_id="7211296", control_type="Document") | | | | | | | | GroupBox - '' (L-149, T68, R551, B104) | | | | ['\ue01bGroupBox', 'GroupBox', '\ue01bGroupBox0', '\ue01bGroupBox1', 'GroupBox0', 'GroupBox1'] | | | | | | | | | | Button - 'Preferences' (L-134, T76, R-114, B96) | | | | | ['PreferencesButton', 'Preferences', 'Button', 'Button0', 'Button1'] | | | | | child_window(title="Preferences", auto_id="hma_title__preferences", control_type="Button") | | | | | | | | | | | | Static - '' (L-134, T76, R-114, B96) | | | | | | ['Static', '\ue01b', '\ue01bStatic', 'Static0', 'Static1'] | | | | | | child_window(title="", control_type="Text") | | | | | | | | | | Button - 'Minimize' (L486, T76, R506, B96) | | | | | ['Minimize', 'Button2', 'MinimizeButton'] | | | | | child_window(title="Minimize", control_type="Button") | | | | | | | | | | | | Static - '' (L486, T76, R506, B96) | | | | | | ['\ue014Static', 'Static2', '\ue014'] | | | | | | child_window(title="", control_type="Text") | | | | | | | | | | Button - 'Close' (L516, T76, R536, B96) | | | | | ['Close', 'Button3', 'CloseButton'] | | | | | child_window(title="Close", control_type="Button") | | | | | | | | | | | | Static - '' (L516, T76, R536, B96) | | | | | | ['Static3', '\ue007', '\ue007Static'] | | | | | | child_window(title="", control_type="Text") | | | | | | | | GroupBox - '' (L-149, T68, R551, B638) | | | | ['\ue01bGroupBox2', 'GroupBox2'] | | | | | | | | | | Image - '' (L111, T120, R291, B300) | | | | | ['\ue01bImage', 'Image'] | | | | | | | | | | Button - 'Disconnect' (L126, T323, R276, B383) | | | | | ['Button4', 'DisconnectButton', 'Disconnect'] | | | | | child_window(title="Disconnect", auto_id="dashboard_switch", control_type="Button") | | | | | | | | | | Static - 'Original IP' (L68, T414, R135, B430) | | | | | ['Original IPStatic', 'Static4', 'Original IP', 'Original IPStatic0', 'Original IPStatic1', 'Original IP0', 'Original IP1'] | | | | | child_window(title="Original IP", control_type="Text") | | | | | | | | | | Static - ' ' (L164, T412, R168, B427) | | | | | ['Static5', '\xa0Static', '\xa0', '\xa0Static0', '\xa0Static1', '\xa00', '\xa01'] | | | | | child_window(title=" ", control_type="Text") | | | | | | | | | | Static - '91.90.253.244' (L184, T414, R266, B429) | | | | | ['Static6', '91.90.253.244Static', '91.90.253.244', '91.90.253.244Static0', '91.90.253.244Static1', '91.90.253.2440', '91.90.253.2441'] | | | | | child_window(title="91.90.253.244", control_type="Text") | | | | | | | | | | Static - 'New IP' (L68, T442, R113, B458) | | | | | ['New IP', 'Static7', 'New IPStatic'] | | | | | child_window(title="New IP", control_type="Text") | | | | | | | | | | Static - ' ' (L164, T440, R168, B455) | | | | | ['Static8', '\xa0Static2', '\xa02'] | | | | | child_window(title=" ", control_type="Text") | | | | | | | | | | Static - '80.255.5.198' (L184, T442, R258, B457) | | | | | ['Static9', '80.255.5.198', '80.255.5.198Static'] | | | | | child_window(title="80.255.5.198", control_type="Text") | | | | | | | | | | Button - 'Change IP Address' (L294, T416, R334, B456) | | | | | ['Change IP AddressButton', 'Button5', 'Change IP Address'] | | | | | child_window(title="Change IP Address", auto_id="ip_change", control_type="Button") | | | | | | | | | | Static - 'Original IP' (L95, T428, R162, B444) | | | | | ['Original IPStatic2', 'Static10', 'Original IP2'] | | | | | child_window(title="Original IP", control_type="Text") | | | | | | | | | | Static - ' ' (L191, T426, R196, B441) | | | | | ['Static11', '\xa0Static3', '\xa03'] | | | | | child_window(title=" ", control_type="Text") | | | | | | | | | | Static - '91.90.253.244' (L211, T428, R293, B443) | | | | | ['Static12', '91.90.253.244Static2', '91.90.253.2442'] | | | | | child_window(title="91.90.253.244", control_type="Text") | | | | | | | | | | Button - 'Location Germany, Berlin' (L31, T489, R371, B569) | | | | | ['Button6', 'Location Germany, BerlinButton', 'Location Germany, Berlin'] | | | | | child_window(title="Location Germany, Berlin", auto_id="dashboard_chooseLocation", control_type="Button") | | | | | | | | | | | | Static - ' ' (L51, T507, R55, B522) | | | | | | ['Static13', '\xa0Static4', '\xa04'] | | | | | | child_window(title=" ", control_type="Text") | | | | | | | | | | | | Static - 'Location' (L111, T511, R164, B526) | | | | | | ['Static14', 'Location', 'LocationStatic'] | | | | | | child_window(title="Location", control_type="Text") | | | | | | | | | | | | Static - 'Germany, Berlin' (L111, T529, R219, B546) | | | | | | ['Static15', 'Germany, Berlin', 'Germany, BerlinStatic'] | | | | | | child_window(title="Germany, Berlin", control_type="Text") | | | | | | | | | | | | Static - '' (L331, T519, R351, B539) | | | | | | ['Static16', '\ue005', '\ue005Static'] | | | | | | child_window(title="", control_type="Text") | | | | | | | | | | Button - 'More' (L420, T107, R528, B137) | | | | | ['More', 'Button7', 'MoreButton'] | | | | | child_window(title="More", control_type="Button") | | | | | | Pane - '' (L-149, T68, R551, B638) | | | ['Pane3', '\ue01bPane2'] Traceback (most recent call last): File "C:\Users\db533\VirtualBox VMs\pycharm-gunabot\venv\lib\site-packages\pywinauto\application.py", line 250, in __resolve_control ctrl = wait_until_passes( File "C:\Users\db533\VirtualBox VMs\pycharm-gunabot\venv\lib\site-packages\pywinauto\timings.py", line 458, in wait_until_passes raise err pywinauto.timings.TimeoutError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "C:\Users\db533\PycharmProjects\GunaBot\hma_control.py", line 19, in <module> dlg_spec.AvastChromiumWindow.Pane2.Document.GroupBox2.Button4.click() File "C:\Users\db533\VirtualBox VMs\pycharm-gunabot\venv\lib\site-packages\pywinauto\application.py", line 379, in __getattribute__ ctrls = self.__resolve_control(self.criteria) File "C:\Users\db533\VirtualBox VMs\pycharm-gunabot\venv\lib\site-packages\pywinauto\application.py", line 261, in __resolve_control raise e.original_exception File "C:\Users\db533\VirtualBox VMs\pycharm-gunabot\venv\lib\site-packages\pywinauto\timings.py", line 436, in wait_until_passes func_val = func(*args, **kwargs) File "C:\Users\db533\VirtualBox VMs\pycharm-gunabot\venv\lib\site-packages\pywinauto\application.py", line 203, in __get_ctrl dialog = self.backend.generic_wrapper_class(findwindows.find_element(**criteria[0])) File "C:\Users\db533\VirtualBox VMs\pycharm-gunabot\venv\lib\site-packages\pywinauto\findwindows.py", line 87, in find_element raise ElementNotFoundError(kwargs) pywinauto.findwindows.ElementNotFoundError: {'title': 'HMA VPN', 'backend': 'win32', 'process': 40580} Process finished with exit code 1
Cracked it! The following code let's me toggle the Connect / Disconnect button and change the IP. from pywinauto import Desktop,Application vpn_app = Application(backend="uia").start('C:\Program Files\Privax\HMA VPN\Vpn.exe') dialog=Desktop(backend="uia").HMA panel0=dialog.Pane # Command to connect / disconnect the VPN: connect_button.click() connect_button=panel0.ConnectButton # Command to change the IP address: changeIP.click() changeIP=panel0.Button5 # Check VPN state: # 0 if disconnected # 1 if connected print(connect_button.get_toggle_state()) # Command to connect / disconnect the VPN: connect_button.click() connect_button=panel0.ConnectButton connect_button.click() # Command to change the IP address: changeIP.click() changeIP=panel0.Button5 changeIP.click() # Check VPN state: # 0 if disconnected # 1 if connected print(connect_button.get_toggle_state())
Web scraping: how to iterate over all the pages from a url and store the result in a table
I am not able to resolve this loop. I want to iterate over all pages from the URL and put all the names in a table. There are 102 pages. Thank you in advance. [from bs4 import BeautifulSoup import re import requests url = "https://www.prokerala.com/kids/baby-names/hindu/" r = requests.get(url) soup = BeautifulSoup(r.content, "html.parser") next_page = soup.find('a', {'class': 'next'}).get('href') l=(f'{next_page}') new_url=url+l link=soup.find('span', {'class' : \['a name-details girl-names', 'a name-details boy-names'\] }) print("href='>%s'>%s</a>" % (link.get("href"), link.text)) while True: r = requests.get(new_url) soup = BeautifulSoup(r.content, "html.parser") next_page = soup.find('a', {'class': 'next'}).get('href') link=soup.find('span', {'class' : \['a name-details girl-names', 'a name-details boy-names'\] }) print("href='>%s'>%s</a>" % (link.get("href"), link.text)) l=(f'{next_page}') new_url=url+l if BeautifulSoup(r.content, "html.parser")is True: break][1] [1]: https://i.stack.imgur.com/SgM4I.png
from bs4 import BeautifulSoup import requests import re from prettytable import PrettyTable def main(url): with requests.Session() as req: p = PrettyTable() p.field_names = ['Name', 'Url'] for item in range(1, 3): r = req.get(url.format(item)) soup = BeautifulSoup(r.content, 'html.parser') for item in soup.findAll("span", class_=re.compile("a name")): p.add_row([item.a.text, f'{url[:25]}{item.a.get("href")}']) print(p) main("https://www.prokerala.com/kids/baby-names/hindu/page-{}.html") Output: +---------------+--------------------------------------------------------------------+ | Name | Url | +---------------+--------------------------------------------------------------------+ | Aabha | https://www.prokerala.com/kids/baby-names/aabha-13597.html | | Aabheer | https://www.prokerala.com/kids/baby-names/aabheer-2.html | | Aachal | https://www.prokerala.com/kids/baby-names/aachal-20815.html | | Aadarsh | https://www.prokerala.com/kids/baby-names/aadarsh-3.html | | Aadarshini | https://www.prokerala.com/kids/baby-names/aadarshini-20997.html | | Aadav | https://www.prokerala.com/kids/baby-names/aadav-21329.html | | Aadavan | https://www.prokerala.com/kids/baby-names/aadavan-19505.html | | Aadesh | https://www.prokerala.com/kids/baby-names/aadesh-4.html | | Aadhan | https://www.prokerala.com/kids/baby-names/aadhan-5.html | | Aadharsha | https://www.prokerala.com/kids/baby-names/aadharsha-20812.html | | Aadhik | https://www.prokerala.com/kids/baby-names/aadhik-19726.html | | Aadhikesav | https://www.prokerala.com/kids/baby-names/aadhikesav-21003.html | | Aadhinath | https://www.prokerala.com/kids/baby-names/aadhinath-21392.html | | Aadhira | https://www.prokerala.com/kids/baby-names/aadhira-6.html | | Aadhira | https://www.prokerala.com/kids/baby-names/aadhira-20603.html | | Aadhith | https://www.prokerala.com/kids/baby-names/aadhith-21626.html | | Aadhriti | https://www.prokerala.com/kids/baby-names/aadhriti-20601.html | | Aadhunik | https://www.prokerala.com/kids/baby-names/aadhunik-21198.html | | Aadhvitha | https://www.prokerala.com/kids/baby-names/aadhvitha-20571.html | | Aadhya | https://www.prokerala.com/kids/baby-names/aadhya-21327.html | | Aadi | https://www.prokerala.com/kids/baby-names/aadi-7.html | | Aadidev | https://www.prokerala.com/kids/baby-names/aadidev-8.html | | Aadilakshmi | https://www.prokerala.com/kids/baby-names/aadilakshmi-20995.html | | Aadimoolan | https://www.prokerala.com/kids/baby-names/aadimoolan-20920.html | | Aadinadh | https://www.prokerala.com/kids/baby-names/aadinadh-20614.html | | Aadinath | https://www.prokerala.com/kids/baby-names/aadinath-10.html | | Aadinathan | https://www.prokerala.com/kids/baby-names/aadinathan-24838.html | | Aadipta | https://www.prokerala.com/kids/baby-names/aadipta-20922.html | | Aadir | https://www.prokerala.com/kids/baby-names/aadir-11.html | | Aadish | https://www.prokerala.com/kids/baby-names/aadish-20945.html | | Aadishree | https://www.prokerala.com/kids/baby-names/aadishree-21389.html | | Aadit | https://www.prokerala.com/kids/baby-names/aadit-12.html | | Aaditey | https://www.prokerala.com/kids/baby-names/aaditey-20921.html | | Aaditeya | https://www.prokerala.com/kids/baby-names/aaditeya-13.html | | Aaditya | https://www.prokerala.com/kids/baby-names/aaditya-22619.html | | Aadrika | https://www.prokerala.com/kids/baby-names/aadrika-20925.html | | Aadvay | https://www.prokerala.com/kids/baby-names/aadvay-22288.html | | Aadvik | https://www.prokerala.com/kids/baby-names/aadvik-19508.html | | Aadya | https://www.prokerala.com/kids/baby-names/aadya-19557.html | | Aadyasha | https://www.prokerala.com/kids/baby-names/aadyasha-20570.html | | Aafreen | https://www.prokerala.com/kids/baby-names/aafreen-19244.html | | Aagam | https://www.prokerala.com/kids/baby-names/aagam-16.html | | Aaghosh | https://www.prokerala.com/kids/baby-names/aaghosh-22289.html | | Aagney | https://www.prokerala.com/kids/baby-names/aagney-17.html | | Aagneya | https://www.prokerala.com/kids/baby-names/aagneya-18.html | | Aahan | https://www.prokerala.com/kids/baby-names/aahan-19.html | | Aahana | https://www.prokerala.com/kids/baby-names/aahana-21194.html | | Aahish | https://www.prokerala.com/kids/baby-names/aahish-22290.html | | Aahlaad | https://www.prokerala.com/kids/baby-names/aahlaad-21.html | | Aahva | https://www.prokerala.com/kids/baby-names/aahva-22.html | | Aahvan | https://www.prokerala.com/kids/baby-names/aahvan-23857.html | | Aakaar | https://www.prokerala.com/kids/baby-names/aakaar-23.html | | Aakaash | https://www.prokerala.com/kids/baby-names/aakaash-24.html | | Aakanksha | https://www.prokerala.com/kids/baby-names/aakanksha-19223.html | | Aakar | https://www.prokerala.com/kids/baby-names/aakar-25.html | | Aakarshan | https://www.prokerala.com/kids/baby-names/aakarshan-26.html | | Aakash | https://www.prokerala.com/kids/baby-names/aakash-19650.html | | Aalap | https://www.prokerala.com/kids/baby-names/aalap-27.html | | Aalok | https://www.prokerala.com/kids/baby-names/aalok-28.html | | Aamil | https://www.prokerala.com/kids/baby-names/aamil-17109.html | | Aamod | https://www.prokerala.com/kids/baby-names/aamod-30.html | | Aanan | https://www.prokerala.com/kids/baby-names/aanan-23674.html | | Aanand | https://www.prokerala.com/kids/baby-names/aanand-31.html | | Aanandhi | https://www.prokerala.com/kids/baby-names/aanandhi-19971.html | | Aanandswarup | https://www.prokerala.com/kids/baby-names/aanandswarup-32.html | | Aananthan | https://www.prokerala.com/kids/baby-names/aananthan-26697.html | | Aanav | https://www.prokerala.com/kids/baby-names/aanav-19506.html | | Aanchal | https://www.prokerala.com/kids/baby-names/aanchal-26403.html | | Aandaleeb | https://www.prokerala.com/kids/baby-names/aandaleeb-33.html | | Aanjaneya | https://www.prokerala.com/kids/baby-names/aanjaneya-34.html | | Aanjay | https://www.prokerala.com/kids/baby-names/aanjay-23679.html | | Aapt | https://www.prokerala.com/kids/baby-names/aapt-35.html | | Aaraadhak | https://www.prokerala.com/kids/baby-names/aaraadhak-37.html | | Aaradhya | https://www.prokerala.com/kids/baby-names/aaradhya-19272.html | | Aaranyan | https://www.prokerala.com/kids/baby-names/aaranyan-23680.html | | Aarathi | https://www.prokerala.com/kids/baby-names/aarathi-24839.html | | Aarav | https://www.prokerala.com/kids/baby-names/aarav-38.html | | Aardik | https://www.prokerala.com/kids/baby-names/aardik-26698.html | | Aarhant | https://www.prokerala.com/kids/baby-names/aarhant-39.html | | Aarin | https://www.prokerala.com/kids/baby-names/aarin-19496.html | | Aarit | https://www.prokerala.com/kids/baby-names/aarit-41.html | | Aariv | https://www.prokerala.com/kids/baby-names/aariv-19507.html | | Aarna | https://www.prokerala.com/kids/baby-names/aarna-24840.html | | Aarnav | https://www.prokerala.com/kids/baby-names/aarnav-44.html | | Aarodeep | https://www.prokerala.com/kids/baby-names/aarodeep-26700.html | | Aaroha | https://www.prokerala.com/kids/baby-names/aaroha-21497.html | | Aaromal | https://www.prokerala.com/kids/baby-names/aaromal-26701.html | | Aarpit | https://www.prokerala.com/kids/baby-names/aarpit-46.html | | Aarsh | https://www.prokerala.com/kids/baby-names/aarsh-47.html | | Aarshabh | https://www.prokerala.com/kids/baby-names/aarshabh-23864.html | | Aarth | https://www.prokerala.com/kids/baby-names/aarth-48.html | | Aarthi | https://www.prokerala.com/kids/baby-names/aarthi-24320.html | | Aarti | https://www.prokerala.com/kids/baby-names/aarti-24228.html | | Aarul | https://www.prokerala.com/kids/baby-names/aarul-19501.html | | Aarunya | https://www.prokerala.com/kids/baby-names/aarunya-21502.html | | Aarush | https://www.prokerala.com/kids/baby-names/aarush-49.html | | Aarushi | https://www.prokerala.com/kids/baby-names/aarushi-50.html | | Aarya | https://www.prokerala.com/kids/baby-names/aarya-19111.html | | Aaryah | https://www.prokerala.com/kids/baby-names/aaryah-13617.html | | Aaryan | https://www.prokerala.com/kids/baby-names/aaryan-51.html | | Aaryana | https://www.prokerala.com/kids/baby-names/aaryana-13618.html | | Aaryanna | https://www.prokerala.com/kids/baby-names/aaryanna-13619.html | | Aashadhar | https://www.prokerala.com/kids/baby-names/aashadhar-21503.html | | Aashank | https://www.prokerala.com/kids/baby-names/aashank-53.html | | Aashay | https://www.prokerala.com/kids/baby-names/aashay-54.html | | Aashi | https://www.prokerala.com/kids/baby-names/aashi-24841.html | | Aashirya | https://www.prokerala.com/kids/baby-names/aashirya-19779.html | | Aashish | https://www.prokerala.com/kids/baby-names/aashish-55.html | | Aashman | https://www.prokerala.com/kids/baby-names/aashman-56.html | | Aashna | https://www.prokerala.com/kids/baby-names/aashna-19991.html | | Aashutosh | https://www.prokerala.com/kids/baby-names/aashutosh-57.html | | Aastha | https://www.prokerala.com/kids/baby-names/aastha-19567.html | | Aastik | https://www.prokerala.com/kids/baby-names/aastik-60.html | | Aathira | https://www.prokerala.com/kids/baby-names/aathira-24842.html | | Aatish | https://www.prokerala.com/kids/baby-names/aatish-62.html | | Aatmaj | https://www.prokerala.com/kids/baby-names/aatmaj-63.html | | Aatreya | https://www.prokerala.com/kids/baby-names/aatreya-64.html | | Aayan | https://www.prokerala.com/kids/baby-names/aayan-65.html | | Aayu | https://www.prokerala.com/kids/baby-names/aayu-66.html | | Aayush | https://www.prokerala.com/kids/baby-names/aayush-24843.html | | Aayushi | https://www.prokerala.com/kids/baby-names/aayushi-26998.html | | Aayushmaan | https://www.prokerala.com/kids/baby-names/aayushmaan-67.html | | Abadhya | https://www.prokerala.com/kids/baby-names/abadhya-19727.html | | Abeesh | https://www.prokerala.com/kids/baby-names/abeesh-19724.html | | Abeesht | https://www.prokerala.com/kids/baby-names/abeesht-20923.html | | Aben | https://www.prokerala.com/kids/baby-names/aben-19257.html | | Abha | https://www.prokerala.com/kids/baby-names/abha-7340.html | | Abhaiveer | https://www.prokerala.com/kids/baby-names/abhaiveer-23649.html | | Abhanja | https://www.prokerala.com/kids/baby-names/abhanja-23670.html | | Abhas | https://www.prokerala.com/kids/baby-names/abhas-19723.html | | Abhat | https://www.prokerala.com/kids/baby-names/abhat-19722.html | | Abhati | https://www.prokerala.com/kids/baby-names/abhati-18373.html | | Abhay | https://www.prokerala.com/kids/baby-names/abhay-154.html | | Abhaya | https://www.prokerala.com/kids/baby-names/abhaya-7341.html | | Abhayan | https://www.prokerala.com/kids/baby-names/abhayan-22297.html | | Abhayananda | https://www.prokerala.com/kids/baby-names/abhayananda-18027.html | | Abhayaprada | https://www.prokerala.com/kids/baby-names/abhayaprada-18028.html | | Abhayd | https://www.prokerala.com/kids/baby-names/abhayd-22295.html | | Abhayd | https://www.prokerala.com/kids/baby-names/abhayd-22296.html | | Abhaysimha | https://www.prokerala.com/kids/baby-names/abhaysimha-23650.html | | Abheek | https://www.prokerala.com/kids/baby-names/abheek-155.html | | Abhey | https://www.prokerala.com/kids/baby-names/abhey-22292.html | | Abhi | https://www.prokerala.com/kids/baby-names/abhi-156.html | | Abhibhava | https://www.prokerala.com/kids/baby-names/abhibhava-18029.html | | Abhicandra | https://www.prokerala.com/kids/baby-names/abhicandra-18030.html | | Abhidha | https://www.prokerala.com/kids/baby-names/abhidha-18374.html | | Abhidhya | https://www.prokerala.com/kids/baby-names/abhidhya-18375.html | | Abhidi | https://www.prokerala.com/kids/baby-names/abhidi-18031.html | | Abhigita | https://www.prokerala.com/kids/baby-names/abhigita-23855.html | | Abhigna | https://www.prokerala.com/kids/baby-names/abhigna-24844.html | | Abhigyaan | https://www.prokerala.com/kids/baby-names/abhigyaan-23671.html | | Abhihita | https://www.prokerala.com/kids/baby-names/abhihita-18032.html | | Abhijaat | https://www.prokerala.com/kids/baby-names/abhijaat-157.html | | Abhijat | https://www.prokerala.com/kids/baby-names/abhijat-18787.html | | Abhijay | https://www.prokerala.com/kids/baby-names/abhijay-158.html | | Abhijaya | https://www.prokerala.com/kids/baby-names/abhijaya-18033.html | | Abhijeet | https://www.prokerala.com/kids/baby-names/abhijeet-17122.html | | Abhiji | https://www.prokerala.com/kids/baby-names/abhiji-25946.html | | Abhijit | https://www.prokerala.com/kids/baby-names/abhijit-159.html | | Abhijith | https://www.prokerala.com/kids/baby-names/abhijith-23839.html | | Abhijun | https://www.prokerala.com/kids/baby-names/abhijun-160.html | | Abhijvala | https://www.prokerala.com/kids/baby-names/abhijvala-18034.html | | Abhik | https://www.prokerala.com/kids/baby-names/abhik-161.html | | Abhikama | https://www.prokerala.com/kids/baby-names/abhikama-19719.html | | Abhikansh | https://www.prokerala.com/kids/baby-names/abhikansh-19720.html | | Abhilasa | https://www.prokerala.com/kids/baby-names/abhilasa-18376.html | | Abhilash | https://www.prokerala.com/kids/baby-names/abhilash-162.html | | Abhilasha | https://www.prokerala.com/kids/baby-names/abhilasha-7342.html | | Abhim | https://www.prokerala.com/kids/baby-names/abhim-19721.html | | Abhimand | https://www.prokerala.com/kids/baby-names/abhimand-18035.html | | Abhimani | https://www.prokerala.com/kids/baby-names/abhimani-18036.html | | Abhimanyu | https://www.prokerala.com/kids/baby-names/abhimanyu-163.html | | Abhimanyusuta | https://www.prokerala.com/kids/baby-names/abhimanyusuta-18037.html | | Abhimoda | https://www.prokerala.com/kids/baby-names/abhimoda-18038.html | | Abhin | https://www.prokerala.com/kids/baby-names/abhin-164.html | | Abhinabhas | https://www.prokerala.com/kids/baby-names/abhinabhas-18039.html | | Abhinand | https://www.prokerala.com/kids/baby-names/abhinand-23672.html | | Abhinanda | https://www.prokerala.com/kids/baby-names/abhinanda-18040.html | | Abhinandan | https://www.prokerala.com/kids/baby-names/abhinandan-165.html | | Abhinandana | https://www.prokerala.com/kids/baby-names/abhinandana-18788.html | | Abhinatha | https://www.prokerala.com/kids/baby-names/abhinatha-18041.html | | Abhinav | https://www.prokerala.com/kids/baby-names/abhinav-166.html | | Abhinava | https://www.prokerala.com/kids/baby-names/abhinava-18042.html | | Abhinavin | https://www.prokerala.com/kids/baby-names/abhinavin-19729.html | | Abhinay | https://www.prokerala.com/kids/baby-names/abhinay-168.html | | Abhinaya | https://www.prokerala.com/kids/baby-names/abhinaya-23651.html | | Abhineet | https://www.prokerala.com/kids/baby-names/abhineet-169.html | | Abhinit | https://www.prokerala.com/kids/baby-names/abhinit-170.html | | Abhinithi | https://www.prokerala.com/kids/baby-names/abhinithi-18377.html | | Abhinivesh | https://www.prokerala.com/kids/baby-names/abhinivesh-171.html | | Abhiprithi | https://www.prokerala.com/kids/baby-names/abhiprithi-18378.html | | Abhir | https://www.prokerala.com/kids/baby-names/abhir-172.html | | Abhiraam | https://www.prokerala.com/kids/baby-names/abhiraam-173.html | | Abhiraj | https://www.prokerala.com/kids/baby-names/abhiraj-174.html | | Abhiraksh | https://www.prokerala.com/kids/baby-names/abhiraksh-19733.html | | Abhirakshit | https://www.prokerala.com/kids/baby-names/abhirakshit-19732.html | | Abhiram | https://www.prokerala.com/kids/baby-names/abhiram-19728.html | | Abhirami | https://www.prokerala.com/kids/baby-names/abhirami-24745.html | | Abhirath | https://www.prokerala.com/kids/baby-names/abhirath-175.html | | Abhirathi | https://www.prokerala.com/kids/baby-names/abhirathi-18379.html | +---------------+--------------------------------------------------------------------+
calculate difference of column values for sets of row indices which are not successive in pandas
Say I have the following table: +----+---------+--------+---------+---------+---------+---------+-----------+-----------+-----------+----------+-----------+------------+------------+---------+---+ | 1 | 0.72694 | 1.4742 | 0.32396 | 0.98535 | 1 | 0.83592 | 0.0046566 | 0.0039465 | 0.04779 | 0.12795 | 0.016108 | 0.0052323 | 0.00027477 | 1.1756 | 1 | | 2 | 0.74173 | 1.5257 | 0.36116 | 0.98152 | 0.99825 | 0.79867 | 0.0052423 | 0.0050016 | 0.02416 | 0.090476 | 0.0081195 | 0.002708 | 7.48E-05 | 0.69659 | 1 | | 3 | 0.76722 | 1.5725 | 0.38998 | 0.97755 | 1 | 0.80812 | 0.0074573 | 0.010121 | 0.011897 | 0.057445 | 0.0032891 | 0.00092068 | 3.79E-05 | 0.44348 | 1 | | 4 | 0.73797 | 1.4597 | 0.35376 | 0.97566 | 1 | 0.81697 | 0.0068768 | 0.0086068 | 0.01595 | 0.065491 | 0.0042707 | 0.0011544 | 6.63E-05 | 0.58785 | 1 | | 5 | 0.82301 | 1.7707 | 0.44462 | 0.97698 | 1 | 0.75493 | 0.007428 | 0.010042 | 0.0079379 | 0.045339 | 0.0020514 | 0.00055986 | 2.35E-05 | 0.34214 | 1 | | 7 | 0.82063 | 1.7529 | 0.44458 | 0.97964 | 0.99649 | 0.7677 | 0.0059279 | 0.0063954 | 0.018375 | 0.080587 | 0.0064523 | 0.0022713 | 4.15E-05 | 0.53904 | 1 | | 8 | 0.77982 | 1.6215 | 0.39222 | 0.98512 | 0.99825 | 0.80816 | 0.0050987 | 0.0047314 | 0.024875 | 0.089686 | 0.0079794 | 0.0024664 | 0.00014676 | 0.66975 | 1 | | 9 | 0.83089 | 1.8199 | 0.45693 | 0.9824 | 1 | 0.77106 | 0.0060055 | 0.006564 | 0.0072447 | 0.040616 | 0.0016469 | 0.00038812 | 3.29E-05 | 0.33696 | 1 | | 11 | 0.7459 | 1.4927 | 0.34116 | 0.98296 | 1 | 0.83088 | 0.0055665 | 0.0056395 | 0.0057679 | 0.036511 | 0.0013313 | 0.00030872 | 3.18E-05 | 0.25026 | 1 | | 12 | 0.79606 | 1.6934 | 0.43387 | 0.98181 | 1 | 0.76985 | 0.0077992 | 0.011071 | 0.013677 | 0.057832 | 0.0033334 | 0.00081648 | 0.00013855 | 0.49751 | 1 | +----+---------+--------+---------+---------+---------+---------+-----------+-----------+-----------+----------+-----------+------------+------------+---------+---+ I have two sets of row indices : set1 = [1,3,5,8,9] set2 = [2,4,7,10,10] Note : Here, I have indicated the first row with index value 1. Length of both sets shall always be same. What I am looking for is a fast and pythonic way to get the difference of column values for corresponding row indices, that is : difference of 1-2,3-4,5-7,8-10,9-10. For this example, my resultant dataframe is the following: +---+---------+--------+---------+---------+---------+---------+-----------+-----------+-----------+----------+-----------+------------+------------+---------+---+ | 1 | 0.01479 | 0.0515 | 0.0372 | 0.00383 | 0.00175 | 0.03725 | 0.0005857 | 0.0010551 | 0.02363 | 0.037474 | 0.0079885 | 0.0025243 | 0.00019997 | 0.47901 | 0 | | 1 | 0.02925 | 0.1128 | 0.03622 | 0.00189 | 0 | 0.00885 | 0.0005805 | 0.0015142 | 0.004053 | 0.008046 | 0.0009816 | 0.00023372 | 0.0000284 | 0.14437 | 0 | | 3 | 0.04319 | 0.1492 | 0.0524 | 0.00814 | 0.00175 | 0.05323 | 0.0023293 | 0.0053106 | 0.0169371 | 0.044347 | 0.005928 | 0.00190654 | 0.00012326 | 0.32761 | 0 | | 3 | 0.03483 | 0.1265 | 0.02306 | 0.00059 | 0 | 0.00121 | 0.0017937 | 0.004507 | 0.0064323 | 0.017216 | 0.0016865 | 0.00042836 | 0.00010565 | 0.16055 | 0 | | 1 | 0.05016 | 0.2007 | 0.09271 | 0.00115 | 0 | 0.06103 | 0.0022327 | 0.0054315 | 0.0079091 | 0.021321 | 0.0020021 | 0.00050776 | 0.00010675 | 0.24725 | 0 | +---+---------+--------+---------+---------+---------+---------+-----------+-----------+-----------+----------+-----------+------------+------------+---------+---+ My resultant difference values are absolute here. I cant apply diff(), since the row indices may not be consecutive. I am currently achieving my aim via looping through sets. Is there a pandas trick to do this?
Use loc based indexing - df.loc[set1].values - df.loc[set2].values Ensure that len(set1) is equal to len(set2). Also, keep in mind setX is a counter-intuitive name for list objects.
You need to select by data reindexing and then subtract: df = df.reindex(set1) - df.reindex(set2).values loc or iloc will raise a future warning, since passing list-likes to .loc or [] with any missing label will raise KeyError in the future.
In short, try the following: df.iloc[::2].values - df.iloc[1::2].values PS: Or alternatively, if (like in your question the indices follow no simple rule): df.iloc[set1].values - df.iloc[set2].values