How to read a webpage table using requests-html?

Problem Description:

I am new to python and am trying to parse a table from the given website into a PANDAS DATAFRAME.

I am using modules requests-html, requests, and beautifulSoup.

Here is the website, I would like to gather the table from:
https://www.aamc.org/data-reports/workforce/interactive-data/active-physicians-largest-specialties-2019

MWE

import pandas as pd
from urllib.request import Request, urlopen
from bs4 import BeautifulSoup

url = 'https://www.aamc.org/data-reports/workforce/interactive-data/active-physicians-largest-specialties-2019'

req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})
page = urlopen(req).read()

soup = BeautifulSoup(page, 'html.parser')

# soup.find_all('table')
pages = soup.find('div', {'class': 'data-table-wrapper'})
df = pd.read_html(pages) # PROBLEM: somehow this table has no data
df.head()

Another attempt:

import requests_html

sess = requests_html.HTMLSession()
res = sess.get(url)
page = res.html
import requests_html

sess = requests_html.HTMLSession()
res = sess.get(url)
page_html = res.html

df = pd.read_html(page_html.raw_html)
df # This gives dataframe, but has no Values

The screenshot is given below:

Solution – 1

The data you see on the page is embedded inside <script> in form of JavaScript. You can use selenium or parse the data manually from the page. I’m using js2py module to decode the data:

import re
import js2py
import requests
import pandas as pd


url = "https://www.aamc.org/data-reports/workforce/interactive-data/active-physicians-largest-specialties-2019"
html_doc = requests.get(url).text

data = re.search(r"(?s)$scope.schools = (.*?);", html_doc).group(1)
data = [{k: v.strip() for k, v in d.items()} for d in js2py.eval_js(data)]

columns = {
    "specialty": "Specialty",
    "one": "Total Active Physicians",
    "two": "Patient Care",
    "three": "Teaching",
    "four": "Research",
    "five": "Other",
}

df = pd.DataFrame(data).rename(columns=columns)
print(df[list(columns.values())].to_markdown(index=False))

Prints: