Develop Reference

How do I parse out specific dataframes when using pandas to web scrape data from a page with multiple dataframes?

I'm working on a project that will scrape data off of https://www.pro-football-reference.com/years/2021/opp.htm. When you visit this webpage you'll see that there are multiple tables. I can obtain the first table with no issues when I run the following code:
import pandas as pd
year=2021
defense_url = 'https://www.pro-football-reference.com/years/{}/opp.htm'.format(year)
df5 = pd.read_html(defense_url, header=1)[0]
df5.head()
However, when I attempt to obtain data from the other tables by changing the index I get a table without a header or an error. For example, df5 = pd.read_html(defense_url, header=1)[1] will create a dataframe without a header a (as shown in the image below):
Additionally, df5 = pd.read_html(defense_url, header=1)[2] generates an IndexError (as shown below):
---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
Input In [46], in <cell line: 1>()
----> 1 df5 = pd.read_html(defense_url, header=1)[2]
2 df5.head()
IndexError: list index out of range
Does anyone know what i could possibly be doing wrong here?

The reason for the index error is because the html response only returns 2 <table> tags. So when you try to get the dataframe at index position [2] (the 3rd table), its not in the list of dataframes.
Those other tables are actually there in the html response but as comments. So there is 2 ways to get that:
Use BeautifulSoups function of pulling out the comments and parse those.
Simply remove/replace the html string that denotes commented html.
I'll code out both for you below:
1. bs4 Comments:
import requests
from bs4 import BeautifulSoup, Comment
import pandas as pd
year=2021
defense_url = 'https://www.pro-football-reference.com/years/{}/opp.htm'.format(year)
df5 = pd.read_html(defense_url, header=1)
result = requests.get(defense_url).text
data = BeautifulSoup(result, 'html.parser')
comments = data.find_all(string=lambda text: isinstance(text, Comment))
for each in comments:
if 'table' in str(each):
try:
level = 0
if isinstance(pd.read_html(str(each))[0].columns, pd.MultiIndex):
level = 1
df5.append(pd.read_html(str(each), header=level)[0])
except:
continue
2: remove/replace the html string that denotes commented html:
import requests
import pandas as pd
year=2021
defense_url = 'https://www.pro-football-reference.com/years/{}/opp.htm'.format(year)
# Get the data
response = requests.get(defense_url)
html = response.text.replace('<!--', '').replace('-->', '')
df5 = pd.read_html(html)
Only difference between the 2 is the second option you'll need to do one more step to find which dataframes are multi-indexed and deal with those if needed.
Output from option 1:
print(df5[2])
Rk Tm G Cmp ... Sk% NY/A ANY/A EXP
0 1.0 Buffalo Bills 17.0 297.0 ... 7.3 4.80 3.8 20.04
1 2.0 New England Patriots 17.0 319.0 ... 6.3 5.50 4.5 -0.86
2 3.0 Chicago Bears 17.0 314.0 ... 9.3 6.20 6.7 -78.80
3 4.0 Carolina Panthers 17.0 337.0 ... 7.0 5.90 6.1 -47.59
4 5.0 Cleveland Browns 17.0 367.0 ... 6.9 5.60 5.5 -51.01
5 6.0 San Francisco 49ers 17.0 372.0 ... 8.1 5.90 6.1 -91.78
6 7.0 Arizona Cardinals 17.0 367.0 ... 6.8 6.10 6.1 -49.69
7 8.0 Denver Broncos 17.0 341.0 ... 6.0 6.10 5.9 -43.39
8 9.0 Pittsburgh Steelers 17.0 355.0 ... 8.9 5.90 5.7 -44.60
9 10.0 Green Bay Packers 17.0 379.0 ... 6.1 5.80 5.5 -33.40
10 11.0 Philadelphia Eagles 17.0 409.0 ... 4.7 6.10 6.1 -81.44
11 12.0 Los Angeles Chargers 17.0 357.0 ... 5.9 6.30 6.4 -89.74
12 13.0 Las Vegas Raiders 17.0 400.0 ... 5.5 5.90 6.4 -159.12
13 14.0 New Orleans Saints 17.0 369.0 ... 7.2 6.00 5.3 -1.74
14 15.0 New York Giants 17.0 402.0 ... 5.3 6.00 5.7 -56.19
15 16.0 Miami Dolphins 17.0 373.0 ... 7.3 5.90 5.6 -26.14
16 17.0 Jacksonville Jaguars 17.0 377.0 ... 5.6 6.70 7.0 -137.16
17 18.0 Atlanta Falcons 17.0 391.0 ... 3.0 6.60 6.8 -119.11
18 19.0 Indianapolis Colts 17.0 390.0 ... 5.2 6.30 6.0 -73.90
19 20.0 Dallas Cowboys 17.0 364.0 ... 6.3 6.20 5.1 13.82
20 21.0 Tampa Bay Buccaneers 17.0 445.0 ... 6.5 5.60 5.3 -30.09
21 22.0 Los Angeles Rams 17.0 416.0 ... 7.4 6.10 5.3 -41.65
22 23.0 Houston Texans 17.0 363.0 ... 5.5 7.10 6.7 -131.46
23 24.0 Detroit Lions 17.0 359.0 ... 5.2 7.20 7.5 -161.97
24 25.0 Tennessee Titans 17.0 395.0 ... 6.4 6.20 5.9 -59.34
25 26.0 Cincinnati Bengals 17.0 420.0 ... 6.3 6.30 6.2 -67.23
26 27.0 Kansas City Chiefs 17.0 401.0 ... 4.8 6.70 6.5 -110.25
27 28.0 Minnesota Vikings 17.0 401.0 ... 7.5 6.40 6.1 -57.45
28 29.0 Washington Football Team 17.0 400.0 ... 6.0 6.80 7.1 -142.08
29 30.0 New York Jets 17.0 401.0 ... 5.3 7.10 7.5 -198.18
30 31.0 Seattle Seahawks 17.0 443.0 ... 4.9 6.50 6.5 -126.34
31 32.0 Baltimore Ravens 17.0 397.0 ... 5.2 7.20 7.6 -166.26
32 NaN Avg Team NaN 378.8 ... 6.2 6.22 6.1 -76.40
33 NaN League Total NaN 12121.0 ... 6.2 6.22 6.1 NaN
34 NaN Avg Tm/G NaN 22.3 ... 6.2 6.22 6.1 NaN
[35 rows x 25 columns]

Scraping table returning repeated values

I'm trying to build a simple web scraper. I am trying to scrape a table, but I'm not sure why the output is: School, 20-5, 33.2 26 times over.
Here is my code:
from bs4 import BeautifulSoup
import requests
url = 'https://www.maxpreps.com/rankings/basketball/1/state/michigan.htm'
page = requests.get(url)
soup = BeautifulSoup(page.content, 'html.parser')
teams = soup.find_all('tr')
for team in teams:
teamname = soup.find('th', class_ = "school").text
record = soup.find('td', class_= "overall dw").text
rating = soup.find('td', class_ = "rating sorted dw").text
print(teamname, record, rating)

Notice that you're never using the Tag that team refers to. Inside the for loop, all of the calls to soup.find() should be calls to team.find():
for team in teams[1:]:
teamname = team.find('th', class_ = "school").text
record = team.find('td', class_= "overall dw").text
rating = team.find('td', class_ = "rating sorted dw").text
print(teamname, record, rating)
This outputs:
St. Mary's Prep (Orchard Lake) 20-5 33.2
University of Detroit Jesuit (Detroit) 16-7 30.0
Williamston 25-0 29.3
Ferndale 21-3 28.9
Catholic Central (Grand Rapids) 25-1 28.4
King (Detroit) 18-3 27.4
De La Salle Collegiate (Warren) 18-7 27.2
Catholic Central (Novi) 16-9 26.6
Brother Rice (Bloomfield Hills) 15-7 26.5
Unity Christian (Hudsonville) 21-1 26.4
Hamtramck 21-4 26.3
Grand Blanc 20-5 25.9
East Lansing 18-5 25.0
Muskegon 20-3 24.8
Northview (Grand Rapids) 25-1 24.6
Cass Tech (Detroit) 21-4 24.3
North Farmington (Farmington Hills) 18-4 24.2
Beecher (Flint) 23-2 24.0
Okemos 19-5 23.9
Benton Harbor 23-3 23.2
Rockford 19-3 22.9
Grand Haven 17-4 21.9
Hartland 19-4 21.0
Marshall 20-3 21.0
Freeland 24-0 21.0
We use [1:] to skip the table header, slicing off the first element in the teams list.

Let pandas parse that table for you (it uses BeautifulSoup under the hoop).
import pandas as pd
url = 'https://www.maxpreps.com/rankings/basketball/1/state/michigan.htm'
df = pd.read_html(url)[0]
Output:
print(df)
# School Ovr. Rating Str. +/-
0 1 St. Mary's Prep (Orchard Lake) 20-5 33.2 23.0 NaN
1 2 University of Detroit Jesuit (Detroit) 16-7 30.0 24.1 NaN
2 3 Williamston 25-0 29.3 10.9 NaN
3 4 Ferndale 21-3 28.9 16.5 NaN
4 5 Catholic Central (Grand Rapids) 25-1 28.4 11.4 NaN
5 6 King (Detroit) 18-3 27.4 15.2 NaN
6 7 De La Salle Collegiate (Warren) 18-7 27.2 19.6 2.0
7 8 Catholic Central (Novi) 16-9 26.6 22.6 -1.0
8 9 Brother Rice (Bloomfield Hills) 15-7 26.5 21.0 -1.0
9 10 Unity Christian (Hudsonville) 21-1 26.4 10.4 NaN
10 11 Hamtramck 21-4 26.3 14.5 2.0
11 12 Grand Blanc 20-5 25.9 15.3 -1.0
12 13 East Lansing 18-5 25.0 15.6 1.0
13 14 Muskegon 20-3 24.8 11.4 1.0
14 15 Northview (Grand Rapids) 25-1 24.6 8.2 1.0
15 16 Cass Tech (Detroit) 21-4 24.3 11.8 -4.0
16 17 North Farmington (Farmington Hills) 18-4 24.2 13.1 NaN
17 18 Beecher (Flint) 23-2 24.0 8.6 2.0
18 19 Okemos 19-5 23.9 13.7 -1.0
19 20 Benton Harbor 23-3 23.2 9.9 -1.0
20 21 Rockford 19-3 22.9 11.6 NaN
21 22 Grand Haven 17-4 21.9 11.3 NaN
22 23 Hartland 19-4 21.0 10.4 1.0
23 24 Marshall 20-3 21.0 8.6 -1.0
24 25 Freeland 24-0 21.0 2.7 4.0

Beautiful Soup not finding specific table by ID

I am trying to parse a basketball reference player page to extract one of the tables from the page and work with the data from it. For some reason, though, beautiful soup cannot find the table in the page. I have tried to search for other tables in the page and it has successfully found them but for some reason will not find this specific one.
I have the following line which takes a link to the page of the specific player I am searching for and gets the BeautifulSoup version of it:
page_soup = BeautifulSoup(bball_ref_page.content, 'lxml')
I then search for the table with the following line:
table = page_soup.find('table', attrs={'id': 'per_poss'})
Whenever I try to print(table) it just comes out as None.
I have also tried searching for the contents by doing:
table = page_soup.find(attrs={'id': 'per_poss'})
same result of None
I have also tried searching for all tables in the page_soup and it returns a list of a bunch of tables not including the one I am looking for
I have tried changing the parse in the page_soup assignment to html.parser and the result remains the same. I have also tried printing the contents of page_soup and can find the table in their:
<div class="table_container current" id="div_per_poss">
<table class="stats_table sortable row_summable" id="per_poss" data-cols-to-freeze="1,3"> <caption>Per 100 Poss Table</caption> <colgroup><col>....
Any ideas what might be causing this to happen?

The page is storing the <table> data inside the HTML comment <!-- --> so normally BeautifulSoup doesn't see it. To load it as pandas dataframe you can use next example:
import requests
import pandas as pd
from bs4 import BeautifulSoup, Comment
url = "https://www.basketball-reference.com/players/j/jordami01.html"
soup = BeautifulSoup(requests.get(url).content, "lxml")
soup = BeautifulSoup("\n".join(soup.find_all(text=Comment)), "lxml")
df = pd.read_html(str(soup.select_one("table#per_poss")))[0]
print(df.to_markdown())
Prints:
Season
Age
Tm
Lg
Pos
G
GS
MP
FG
FGA
FG%
3P
3PA
3P%
2P
2PA
2P%
FT
FTA
FT%
ORB
DRB
TRB
AST
STL
BLK
TOV
PF
PTS
Unnamed: 29
ORtg
DRtg
0
1984-85
21
CHI
NBA
SG
82
82
3144
12.9
25
0.515
0.1
0.8
0.173
12.7
24.2
0.526
9.7
11.5
0.845
2.6
5.6
8.2
7.4
3
1.1
4.5
4.4
35.5
nan
118
107
1
1985-86
22
CHI
NBA
SG
18
7
451
16
35
0.457
0.3
1.9
0.167
15.7
33.1
0.474
11.2
13.3
0.84
2.5
4.4
6.8
5.7
3.9
2.2
4.8
4.9
43.5
nan
109
107
2
1986-87
23
CHI
NBA
SG
82
82
3281
16.8
34.8
0.482
0.2
1
0.182
16.6
33.8
0.491
12.7
14.8
0.857
2.5
4
6.6
5.8
3.6
1.9
4.2
3.6
46.4
nan
117
104
3
1987-88
24
CHI
NBA
SG
82
82
3311
16.2
30.3
0.535
0.1
0.8
0.132
16.1
29.5
0.546
11
13.1
0.841
2.1
4.7
6.8
7.4
3.9
2
3.8
4.1
43.6
nan
123
101
4
1988-89
25
CHI
NBA
SG
81
81
3255
14.7
27.3
0.538
0.4
1.5
0.276
14.3
25.8
0.553
10.2
12.1
0.85
2.3
7.6
9.9
9.9
3.6
1
4.4
3.8
40
nan
123
103
5
1989-90
26
CHI
NBA
SG
82
82
3197
16
30.5
0.526
1.4
3.8
0.376
14.6
26.7
0.548
9.2
10.8
0.848
2.2
6.6
8.8
8.1
3.5
0.8
3.8
3.7
42.7
nan
123
106
6
1990-91
27
CHI
NBA
SG
82
82
3034
16.4
30.4
0.539
0.5
1.5
0.312
15.9
28.9
0.551
9.4
11.1
0.851
2
6.2
8.1
7.5
3.7
1.4
3.3
3.8
42.7
nan
125
102
7
1991-92
28
CHI
NBA
SG
80
80
3102
15.5
29.8
0.519
0.4
1.6
0.27
15
28.2
0.533
8
9.7
0.832
1.5
6.9
8.4
8
3
1.2
3.3
3.3
39.4
nan
121
102
8
1992-93
29
CHI
NBA
SG
78
78
3067
16.8
33.9
0.495
1.4
3.9
0.352
15.4
30
0.514
8.1
9.6
0.837
2.3
6.5
8.8
7.2
3.7
1
3.5
3.2
43
nan
119
102
9
1994-95
31
CHI
NBA
SG
17
17
668
13
31.5
0.411
1.2
2.5
0.5
11.7
29
0.403
8.5
10.6
0.801
2
7.2
9.1
7
2.3
1
2.7
3.7
35.7
nan
109
103
10
1995-96
32
CHI
NBA
SG
82
82
3090
15.6
31.5
0.495
1.9
4.4
0.427
13.7
27.1
0.506
9.3
11.2
0.834
2.5
6.7
9.3
6
3.1
0.7
3.4
3.3
42.5
nan
124
100
11
1996-97
33
CHI
NBA
SG
82
82
3106
15.8
32.5
0.486
1.9
5.1
0.374
13.9
27.4
0.507
8.2
9.9
0.833
1.9
6.3
8.3
6
2.4
0.8
2.9
2.7
41.8
nan
121
102
12
1997-98
34
CHI
NBA
SG
82
82
3181
14.9
32.1
0.465
0.5
2.1
0.238
14.4
30
0.482
9.6
12.2
0.784
2.2
5.8
8.1
4.8
2.4
0.8
3.1
2.6
40
nan
114
100
13
2001-02
38
WAS
NBA
SF
60
53
2093
14.3
34.4
0.416
0.3
1.4
0.189
14
33
0.426
6.8
8.6
0.79
1.3
7.5
8.8
8
2.2
0.7
4.2
3.1
35.7
nan
99
105
14
2002-03
39
WAS
NBA
SF
82
67
3031
12.2
27.4
0.445
0.3
1
0.291
11.9
26.4
0.45
4.8
5.8
0.821
1.3
7.7
8.9
5.6
2.2
0.7
3.1
3.1
29.5
nan
101
103
15
Career
nan
nan
NBA
nan
1072
1039
41011
15.3
30.7
0.497
0.7
2.2
0.327
14.5
28.5
0.51
9.2
11
0.835
2.1
6.3
8.3
7
3.1
1.1
3.7
3.5
40.4
nan
118
103
16
nan
nan
nan
nan
nan
nan
nan
nan
nan
nan
nan
nan
nan
nan
nan
nan
nan
nan
nan
nan
nan
nan
nan
nan
nan
nan
nan
nan
nan
nan
nan
nan
17
13 seasons
nan
CHI
NBA
nan
930
919
35887
15.5
30.8
0.505
0.8
2.4
0.332
14.8
28.4
0.52
9.6
11.5
0.838
2.2
6.1
8.3
7.1
3.3
1.2
3.7
3.5
41.5
nan
120
103
18
2 seasons
nan
WAS
NBA
nan
142
120
5124
13.1
30.3
0.431
0.3
1.1
0.241
12.8
29.1
0.439
5.6
7
0.805
1.3
7.6
8.9
6.6
2.2
0.7
3.6
3.1
32
nan
100
104
To iterate the rows of dataframe, you can use df.iterrows() for example:
for index, row in df.iterrows():
print(row["Season"], row["Age"])
Prints:
1984-85 21.0
1985-86 22.0
1986-87 23.0
1987-88 24.0
1988-89 25.0
...

Unable to extract Tables

Beginner here. I'm having issues while trying to extract data from the second (Team Statistics) and third (Team Analytics 5-on-5) Table on this page:
https://www.hockey-reference.com/leagues/NHL_2021.html
I'm using this code:
import pandas as pd
url = 'https://www.hockey-reference.com/leagues/NHL_2021.html'
html = requests.get(url).content
df_list = pd.read_html(html)
df = df_list[1]
print(df)
and
url = 'https://www.hockey-reference.com/leagues/NHL_2021.html'
html = requests.get(url).content
df_list = pd.read_html(html)
df = df_list[2]
print(df)
to get the right tables.
But for some kind of reason I will always get this error message:
IndexError: list index out of range
I could extract the first table by using the same code with df = df_list[0], that will work, but it is useless to me. I really need the 2nd an 3rd Table, and I just don't know why it doesn't work.
Pretty sure that's easy to answer for most of you.
Thanx in advance!

You get this error because the read_html() method returns a list of 1 element and that element is at position 0
instead of
df = df_list[1]
use this
df = df_list[0]
You get combined table of all teams from your mentioned site so if you want to extract the table of 2nd and 3rd team use loc[] accessor:-
east_division=df.loc[9:17]
north_division=df.loc[18:25]

Use the URL directly in pandas.read_html
df = pd.read_html('https://www.hockey-reference.com/leagues/NHL_2021.html')

The tables are in fact there in the html (within the comments). Use BeautifulSoup to pull out the comments and parse those tables as well. The code below will pull all (both commented and uncommented tables). and put it into a list. Just a matter of pulling out the table by index that you want, in this case indices 1 and 2.
import requests
from bs4 import BeautifulSoup, Comment
import pandas as pd
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36'}
url = "https://www.hockey-reference.com/leagues/NHL_2021.html"
# Gets all uncommented tables
tables = pd.read_html(url, header=1)
# Get the html source
response = requests.get(url, headers=headers)
# Creat soup object form html
soup = BeautifulSoup(response.content, 'html.parser')
# Get the comments in html
comments = soup.find_all(string=lambda text: isinstance(text, Comment))
# Iterate thorugh each comment and parse the table if found
# # Append the table to the tables list
for each in comments:
if 'table' in str(each):
try:
tables.append(pd.read_html(each, header=1)[0])
tables = tables[tables['Rk'].ne('Rk')]
tables = tables.rename(columns={'Unnamed: 1':'Team'})
except:
continue
Output:
for table in tables[1:3]:
print(table)
Rk Unnamed: 1 AvAge GP W ... S S% SA SV% SO
0 1.0 New York Islanders 29.0 28 18 ... 841 9.8 767 0.920 5
1 2.0 Tampa Bay Lightning 28.3 26 19 ... 798 12.2 725 0.919 3
2 3.0 Florida Panthers 28.1 27 18 ... 918 10.0 840 0.910 0
3 4.0 Toronto Maple Leafs 28.9 29 19 ... 883 11.2 828 0.909 2
4 5.0 Carolina Hurricanes 27.2 26 19 ... 816 10.9 759 0.912 3
5 6.0 Washington Capitals 30.4 27 17 ... 768 12.0 808 0.895 0
6 7.0 Vegas Golden Knights 29.1 25 18 ... 752 11.0 691 0.920 4
7 8.0 Edmonton Oilers 28.4 30 18 ... 945 10.6 938 0.907 2
8 9.0 Winnipeg Jets 28.0 27 17 ... 795 11.4 856 0.910 1
9 10.0 Pittsburgh Penguins 28.1 27 17 ... 779 11.0 784 0.899 1
10 11.0 Chicago Blackhawks 27.2 29 14 ... 863 10.1 997 0.910 2
11 12.0 Minnesota Wild 28.8 25 16 ... 764 10.3 723 0.913 2
12 13.0 St. Louis Blues 28.2 28 14 ... 836 10.4 835 0.892 0
13 14.0 Boston Bruins 28.8 25 14 ... 772 8.8 665 0.913 2
14 15.0 Colorado Avalanche 26.8 25 15 ... 846 8.7 622 0.905 4
15 16.0 Montreal Canadiens 28.8 27 12 ... 890 9.7 782 0.909 0
16 17.0 Philadelphia Flyers 27.5 25 13 ... 699 11.7 753 0.892 3
17 18.0 Calgary Flames 28.0 28 13 ... 838 8.9 845 0.904 3
18 19.0 Los Angeles Kings 27.7 26 11 ... 748 10.3 814 0.910 2
19 20.0 Vancouver Canucks 27.7 31 13 ... 951 8.8 1035 0.903 1
20 21.0 Columbus Blue Jackets 27.0 29 11 ... 839 9.3 902 0.895 1
21 22.0 Arizona Coyotes 28.5 27 12 ... 689 9.7 851 0.907 1
22 23.0 San Jose Sharks 29.3 25 11 ... 749 9.5 800 0.890 1
23 24.0 New York Rangers 25.7 26 11 ... 773 9.2 746 0.906 2
24 25.0 Nashville Predators 28.9 28 11 ... 880 7.4 837 0.885 1
25 26.0 Anaheim Ducks 28.4 29 8 ... 804 7.7 852 0.891 3
26 27.0 Dallas Stars 28.3 23 8 ... 657 10.2 626 0.904 3
27 28.0 Detroit Red Wings 29.4 28 8 ... 785 8.0 870 0.891 0
28 29.0 Ottawa Senators 26.4 30 9 ... 942 8.2 960 0.874 0
29 30.0 New Jersey Devils 26.2 24 8 ... 708 8.5 741 0.896 2
30 31.0 Buffalo Sabres 27.4 26 6 ... 728 7.7 804 0.893 0
31 NaN League Average 28.1 27 13 ... 808 9.8 808 0.902 2
[32 rows x 32 columns]
Rk Unnamed: 1 S% SV% ... HDGF HDC% HDGA HDCO%
0 1 New York Islanders 8.3 0.931 ... 11 12.2 11 11.8
1 2 Tampa Bay Lightning 8.7 0.933 ... 11 14.9 6 6.3
2 3 Florida Panthers 7.9 0.926 ... 15 14.4 12 17.6
3 4 Toronto Maple Leafs 8.8 0.933 ... 16 13.4 8 11.1
4 5 Carolina Hurricanes 7.5 0.932 ... 12 12.8 7 9.3
5 6 Washington Capitals 9.8 0.919 ... 10 10.9 5 7.8
6 7 Vegas Golden Knights 9.3 0.927 ... 20 15.9 11 14.5
7 8 Edmonton Oilers 8.2 0.920 ... 9 11.3 13 9.8
8 9 Winnipeg Jets 8.5 0.926 ... 15 15.0 8 7.8
9 10 Pittsburgh Penguins 8.8 0.922 ... 10 14.5 15 13.5
10 11 Chicago Blackhawks 7.3 0.925 ... 10 10.5 14 15.1
11 12 Minnesota Wild 9.9 0.930 ... 16 14.2 8 11.9
12 13 St. Louis Blues 8.4 0.914 ... 15 18.1 15 15.8
13 14 Boston Bruins 6.6 0.922 ... 5 7.4 11 12.2
14 15 Colorado Avalanche 6.7 0.916 ... 8 8.1 8 13.3
15 16 Montreal Canadiens 7.8 0.935 ... 15 12.0 8 11.3
16 17 Philadelphia Flyers 10.1 0.907 ... 18 15.9 9 12.9
17 18 Calgary Flames 7.6 0.929 ... 6 6.9 8 9.2
18 19 Los Angeles Kings 7.5 0.925 ... 11 13.1 8 9.8
19 20 Vancouver Canucks 7.3 0.919 ... 17 13.2 20 17.4
20 21 Columbus Blue Jackets 8.1 0.918 ... 5 9.6 15 13.6
21 22 Arizona Coyotes 7.7 0.924 ... 11 14.7 14 12.8
22 23 San Jose Sharks 8.1 0.909 ... 12 14.6 16 14.0
23 24 New York Rangers 7.8 0.921 ... 17 14.0 8 12.7
24 25 Nashville Predators 5.7 0.918 ... 5 10.6 11 13.4
25 26 Anaheim Ducks 7.4 0.909 ... 12 13.3 25 16.8
26 27 Dallas Stars 7.4 0.929 ... 11 13.3 5 12.8
27 28 Detroit Red Wings 7.5 0.923 ... 13 15.3 12 16.7
28 29 Ottawa Senators 7.1 0.894 ... 7 8.6 20 14.3
29 30 New Jersey Devils 7.2 0.923 ... 10 14.3 12 13.2
30 31 Buffalo Sabres 5.8 0.911 ... 6 8.2 16 14.0

Webscraping in BeautifulSoup is returning an empty list

I am trying to web scrape a table from basketball reference and it returns an empty list. I was hoping someone could help me debug or explain why. The page has many tables but it is the Miscellaneous Stats section in particular. Thanks in advance!
from bs4 import BeautifulSoup
import requests
import time
import pandas as pd
import matplotlib as plt
import numpy as np
url = 'https://www.basketball-reference.com/leagues/NBA_2020.html#all_misc_stats'
res = requests.get(url)
soup = BeautifulSoup(res.content,'lxml')
soup.find('div', {'id':'div_misc_stats'})

Your implementation isn't wrong for parsing the soup, its just that the particular element you're looking for requires javascript to render. You're probably better off looking for some other source of the data if you can find it.
If you really need THIS data, then you may wish to looking rendering the page first (see this for some inspiration)
From my cursory analysis, it also seems there isn't an external network call made to get the data before rendering it, so it may be elsewhere embeded in the page, as xml/json/etc, although I didn't find it in my search. May be worth checking that before investing in a more compute expensive approach if this is not a one time thing you'll need to scrape.

The data is inside HTML comment <!-- ... -->. You can use this script to load it inside a DataFrame:
import requests
import pandas as pd
from bs4 import BeautifulSoup, Comment
url = 'https://www.basketball-reference.com/leagues/NBA_2020.html'
soup = BeautifulSoup(requests.get(url).content, 'html.parser')
table = soup.select_one('h2:contains("Miscellaneous Stats")').find_next(text=lambda t: isinstance(t, Comment))
df = pd.read_html(str(table))[0].droplevel(0, axis=1)
print(df)
Prints:
Rk Team Age W L PW PL MOV SOS SRS ORtg DRtg ... TS% eFG% TOV% ORB% FT/FGA eFG% TOV% DRB% FT/FGA Arena Attend. Attend./G
0 1.0 Milwaukee Bucks* 29.2 53.0 12.0 52 13 11.29 -0.85 10.44 112.6 101.9 ... 0.583 0.553 12.8 20.7 0.196 0.486 12.2 81.7 0.172 Fiserv Forum 549036 17711
1 2.0 Los Angeles Lakers* 29.6 49.0 14.0 45 18 7.41 0.34 7.75 113.0 105.6 ... 0.577 0.548 13.2 24.6 0.196 0.509 13.8 78.4 0.202 STAPLES Center 588907 18997
2 3.0 Los Angeles Clippers* 27.4 44.0 20.0 44 20 6.52 0.22 6.74 113.6 107.2 ... 0.574 0.532 12.7 24.0 0.232 0.503 12.3 77.3 0.210 STAPLES Center 610176 19068
3 4.0 Toronto Raptors* 26.6 46.0 18.0 44 20 6.45 -0.57 5.88 111.6 105.2 ... 0.574 0.536 12.8 21.6 0.205 0.502 14.6 76.1 0.200 Scotiabank Arena 633456 19796
4 5.0 Dallas Mavericks 26.2 40.0 27.0 45 22 6.04 -0.21 5.84 116.7 110.6 ... 0.581 0.548 11.3 23.5 0.198 0.519 10.9 77.4 0.172 American Airlines Center 682096 20062
5 6.0 Boston Celtics* 25.3 43.0 21.0 44 20 6.17 -0.48 5.69 112.9 106.8 ... 0.567 0.529 12.0 23.9 0.204 0.510 13.6 77.5 0.212 TD Garden 610864 19090
6 7.0 Houston Rockets* 29.1 40.0 24.0 39 25 3.75 0.03 3.78 113.8 110.2 ... 0.578 0.539 12.6 22.4 0.226 0.528 13.5 75.6 0.194 Toyota Center 578458 18077
7 8.0 Utah Jazz* 27.5 41.0 23.0 38 26 3.17 0.03 3.20 112.6 109.4 ... 0.587 0.552 13.6 21.2 0.208 0.514 10.9 79.0 0.180 Vivint Smart Home Arena 567486 18306
8 9.0 Denver Nuggets* 25.6 43.0 22.0 39 26 2.95 0.06 3.02 112.5 109.5 ... 0.564 0.532 12.3 24.7 0.178 0.526 13.0 77.0 0.194 Pepsi Center 633153 19186
9 10.0 Oklahoma City Thunder* 25.6 40.0 24.0 37 27 2.45 0.34 2.79 111.6 109.1 ... 0.577 0.534 12.3 19.2 0.233 0.520 12.4 76.8 0.164 Chesapeake Energy Arena 600699 18203
10 11.0 Miami Heat* 25.9 41.0 24.0 39 26 3.23 -0.65 2.58 112.7 109.4 ... 0.587 0.549 13.5 20.5 0.231 0.522 12.3 79.7 0.208 AmericanAirlines Arena 629771 19680
11 12.0 Philadelphia 76ers* 26.4 39.0 26.0 37 28 2.22 0.01 2.22 110.4 108.2 ... 0.562 0.530 12.7 23.7 0.189 0.522 12.7 80.4 0.211 Wells Fargo Center 639491 20629
12 13.0 Indiana Pacers* 25.6 39.0 26.0 37 28 1.94 -0.33 1.61 110.3 108.3 ... 0.565 0.533 11.9 20.3 0.170 0.513 12.8 77.1 0.193 Bankers Life Fieldhouse 529002 16531
13 14.0 New Orleans Pelicans 25.4 28.0 36.0 30 34 -0.83 1.13 0.30 110.8 111.6 ... 0.567 0.538 13.7 24.3 0.183 0.531 12.3 78.1 0.207 Smoothie King Center 528172 16505
14 15.0 Orlando Magic 26.0 30.0 35.0 30 35 -0.97 0.12 -0.85 108.0 109.0 ... 0.540 0.503 11.4 22.4 0.191 0.535 13.5 79.0 0.170 Amway Center 529870 17093
15 16.0 Memphis Grizzlies 24.0 32.0 33.0 30 35 -1.08 0.02 -1.05 109.4 110.4 ... 0.561 0.530 13.2 23.2 0.178 0.520 12.6 77.6 0.213 FedEx Forum 523297 15857
16 17.0 Phoenix Suns 24.7 26.0 39.0 30 35 -1.37 0.32 -1.05 110.5 111.8 ... 0.572 0.528 13.3 22.2 0.226 0.543 14.0 78.3 0.221 Talking Stick Resort Arena 550633 15606
17 18.0 Portland Trail Blazers 27.5 29.0 37.0 30 36 -1.61 0.49 -1.11 112.5 114.1 ... 0.566 0.530 11.5 22.0 0.191 0.523 11.0 75.0 0.204 Moda Center 628303 19634
18 19.0 Brooklyn Nets 26.5 30.0 34.0 31 33 -0.64 -0.54 -1.18 108.1 108.7 ... 0.550 0.515 13.4 23.5 0.199 0.507 10.9 77.8 0.181 Barclays Center 524907 16403
19 20.0 San Antonio Spurs 27.9 27.0 36.0 28 35 -1.76 0.57 -1.21 111.9 113.7 ... 0.569 0.529 11.0 19.5 0.206 0.542 11.5 79.2 0.194 AT&T Center 550515 18351
20 21.0 Sacramento Kings 27.1 28.0 36.0 28 36 -1.92 0.48 -1.44 109.7 111.6 ... 0.563 0.531 13.0 21.8 0.178 0.540 13.6 78.5 0.222 Golden 1 Center 520663 16796
21 22.0 Minnesota Timberwolves 24.8 19.0 45.0 24 40 -4.30 0.51 -3.78 108.1 112.2 ... 0.551 0.514 13.0 22.1 0.209 0.541 13.2 77.2 0.218 Target Center 482112 15066
22 23.0 Chicago Bulls 24.4 22.0 43.0 26 39 -3.08 -0.73 -3.81 106.7 109.8 ... 0.547 0.515 13.7 22.8 0.175 0.546 16.3 75.6 0.239 United Center 639352 18804
23 24.0 Detroit Pistons 25.9 20.0 46.0 26 40 -3.56 -0.66 -4.22 109.0 112.7 ... 0.561 0.529 13.8 22.6 0.194 0.541 12.7 75.9 0.186 Little Caesars Arena 509469 15294
24 25.0 Washington Wizards 25.4 24.0 40.0 24 40 -4.05 -0.81 -4.86 111.9 115.8 ... 0.568 0.528 12.1 22.0 0.214 0.560 14.0 74.9 0.230 Capital One Arena 532702 16647
25 26.0 New York Knicks 24.5 21.0 45.0 20 46 -6.45 -0.09 -6.55 106.5 113.0 ... 0.531 0.501 12.6 25.8 0.182 0.541 12.4 78.3 0.224 Madison Square Garden (IV) 620789 18812
26 27.0 Charlotte Hornets 24.3 23.0 42.0 19 46 -6.75 -0.12 -6.88 106.3 113.3 ... 0.539 0.504 13.3 23.9 0.188 0.546 13.1 74.4 0.159 Spectrum Center 478591 15428
27 28.0 Cleveland Cavaliers 25.0 19.0 46.0 18 47 -7.89 0.33 -7.55 107.5 115.4 ... 0.553 0.522 14.6 24.6 0.172 0.560 11.7 77.4 0.164 Quicken Loans Arena 643008 17861
28 29.0 Atlanta Hawks 24.1 20.0 47.0 18 49 -7.97 0.40 -7.57 107.2 114.8 ... 0.554 0.515 13.8 21.6 0.204 0.543 12.7 74.9 0.233 State Farm Arena 545453 16043
29 30.0 Golden State Warriors 24.4 15.0 50.0 16 49 -8.71 0.79 -7.92 105.2 113.8 ... 0.540 0.497 13.2 21.5 0.212 0.553 13.7 76.4 0.193 Chase Center 614176 18064
30 NaN League Average 26.2 NaN NaN 32 32 0.00 0.00 0.00 110.4 110.4 ... 0.564 0.528 12.8 22.6 0.199 0.528 12.8 77.4 0.199 NaN 575820 17788
[31 rows x 28 columns]

this website that you want to scrape, is a dynamic website, because of this you can't access to all the data at the first you request to the website, you need to wait for some seconds for rendering javascript and then access to all of the website data, for this solution you can use selenium. read the documentation and download the driver chrome or firefox then use it, I wrote the code that you can access to that table :
from selenium import webdriver
import pandas as pd
import os
import time
chromedriver = "driver/chromedriver"
os.environ["webdriver.chrome.driver"] = chromedriver
driver = webdriver.Chrome(chromedriver)
url = 'https://www.basketball-reference.com/leagues/NBA_2020.html#all_misc_stats'
driver.get(url)
time.sleep(15)
soruce = driver.page_source
tables = pd.read_html(soruce)
for table in tables:
try:
if 'Arena' in table.columns[25][1]:
print(table)
except:
pass
print:
Rk Team Age ... Arena Attend. Attend./G
0 1.0 Milwaukee Bucks* 29.2 ... Fiserv Forum 549036 17711
1 2.0 Los Angeles Lakers* 29.6 ... STAPLES Center 588907 18997
2 3.0 Los Angeles Clippers* 27.4 ... STAPLES Center 610176 19068
3 4.0 Toronto Raptors* 26.6 ... Scotiabank Arena 633456 19796
4 5.0 Dallas Mavericks 26.2 ... American Airlines Center 682096 20062
5 6.0 Boston Celtics* 25.3 ... TD Garden 610864 19090
6 7.0 Houston Rockets* 29.1 ... Toyota Center 578458 18077
7 8.0 Utah Jazz* 27.5 ... Vivint Smart Home Arena 567486 18306
8 9.0 Denver Nuggets* 25.6 ... Pepsi Center 633153 19186
9 10.0 Oklahoma City Thunder* 25.6 ... Chesapeake Energy Arena 600699 18203
10 11.0 Miami Heat* 25.9 ... AmericanAirlines Arena 629771 19680
11 12.0 Philadelphia 76ers* 26.4 ... Wells Fargo Center 639491 20629
12 13.0 Indiana Pacers* 25.6 ... Bankers Life Fieldhouse 529002 16531
13 14.0 New Orleans Pelicans 25.4 ... Smoothie King Center 528172 16505
14 15.0 Orlando Magic 26.0 ... Amway Center 529870 17093
15 16.0 Memphis Grizzlies 24.0 ... FedEx Forum 523297 15857
16 17.0 Phoenix Suns 24.7 ... Talking Stick Resort Arena 550633 15606
17 18.0 Portland Trail Blazers 27.5 ... Moda Center 628303 19634
18 19.0 Brooklyn Nets 26.5 ... Barclays Center 524907 16403
19 20.0 San Antonio Spurs 27.9 ... AT&T Center 550515 18351
20 21.0 Sacramento Kings 27.1 ... Golden 1 Center 520663 16796
21 22.0 Minnesota Timberwolves 24.8 ... Target Center 482112 15066
22 23.0 Chicago Bulls 24.4 ... United Center 639352 18804
23 24.0 Detroit Pistons 25.9 ... Little Caesars Arena 509469 15294
24 25.0 Washington Wizards 25.4 ... Capital One Arena 532702 16647
25 26.0 New York Knicks 24.5 ... Madison Square Garden (IV) 620789 18812
26 27.0 Charlotte Hornets 24.3 ... Spectrum Center 478591 15428
27 28.0 Cleveland Cavaliers 25.0 ... Quicken Loans Arena 643008 17861
28 29.0 Atlanta Hawks 24.1 ... State Farm Arena 545453 16043
29 30.0 Golden State Warriors 24.4 ... Chase Center 614176 18064
30 NaN League Average 26.2 ... NaN 575820 17788
[31 rows x 28 columns]