I try to scrap some data from a js web site but even with selenium it still can't reach it.
from discord.ext import commands
from selenium.webdriver.chrome.options import Options
from selenium import webdriver
from selenium.webdriver.common.by import By
options = Options()
options.add_argument("--headless")
options.add_argument("-no-sandbox")
options.add_argument("-disable-dev-shm-usage")
driver = webdriver.Chrome('chromedriver', options=options)
driver.get('http://mc164.boxtoplay.com:65248')
print(driver.page_source.encode("utf-8"))
The output:
b'<html lang="en"><head>\n\n\t<title>Minecraft Dynamic Map</title>\n\n\t<meta charset="utf-8">\n\t<meta name="keywords" content="minecraft, map, dynamic">\n\t<meta name="description" content="Minecraft Dynamic Map">\n\t<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=no">\n\t<!-- These 2 lines make us fullscreen on apple mobile products - remove if you don\'t like that -->\n\t<meta name="apple-mobile-web-app-capable" content="yes">\n\t<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent">\t\n\n\t<link rel="icon" href="images/dynmap.ico" type="image/ico">\n\n\t<script type="text/javascript" src="js/jquery-3.5.1.js?_=3.3.2-696"></script>\n\t<link rel="stylesheet" type="text/css" href="css/leaflet.css?_=3.3.2-696">\n\t<script type="text/javascript" src="js/leaflet.js?_=3.3.2-696"></script>\n <!-- FOr source debug on leaflet \t<script type="text/javascript" src="js/leaflet-src.js?_=3.3.2-696"></script> -->\n\t<script type="text/javascript" src="js/custommarker.js?_=3.3.2-696"></script>\n\n\t<script type="text/javascript" src="js/dynmaputils.js?_=3.3.2-696"></script>\n\t<script type="text/javascript" src="js/sidebarutils.js?_=3.3.2-696"></script>\n\n\t<!--<link rel="stylesheet" type="text/css" href="css/embedded.css" media="screen" />-->\n\t<link rel="stylesheet" type="text/css" href="css/standalone.css?_=3.3.2-696" media="screen">\n\t<link rel="stylesheet" type="text/css" href="css/dynmap_style.css?_=3.3.2-696" media="screen">\n\t<!-- <link rel="stylesheet" type="text/css" href="css/override.css" media="screen" /> -->\n\n\t<script type="text/javascript" src="version.js?_=3.3.2-696"></script>\n\t<script type="text/javascript" src="js/jquery.json.js?_=3.3.2-696"></script>\n\t<script type="text/javascript" src="js/jquery.mousewheel.js?_=3.3.2-696"></script>\n\t<script type="text/javascript" src="js/minecraft.js?_=3.3.2-696"></script>\n\t<script type="text/javascript" src="js/map.js?_=3.3.2-696"></script>\n\t<script type="text/javascript" src="js/hdmap.js?_=3.3.2-696"></script>\n\t<script type="text/javascript" src="standalone/config.js?_=3.3.2-696"></script>\n\n\t<script type="text/javascript">\n\t\t\t$(document).ready(function() {\n\t\t\t\twindow.dynmap = new DynMap($.extend({\n\t\t\t\t\tcontainer: $(\'#mcmap\')\n\t\t\t\t}, config));\n\t\t\t});\n\t</script>\n\n</head>\n<body>\n<noscript>\n For full functionality of this site it is necessary to enable JavaScript.\n Here are the \n instructions how to enable JavaScript in your web browser.\n</noscript>\n\n\t<div id="mcmap"></div>\n\n</body></html>'
I think, javascript is not executed while in headless mode. You can try without headleass mode.
if the app is developed using javascript. you can try any javascript framework to capture the data.
While working on finding how data is being processed on the webpage. I was figuring out this site investorscout.co/investors.
I tried looking at the Network tab to see how they are rendering the data from backend onto the page. I have also looked into WS but no luck.
I am confused as to how the site is able to display the data while none of the requests in the Network tab shows that.
I aim to fetch the data using requests and bs4.
Sending a GET request to the page https://investorscout.co/investors returns a response with multiple references to external JavaScript code in it. This is what is being loaded on the page - dynamic content based on JavaScript functions.
I would suggest an implementation involving selenium instead as you would not be able to scrape content on the page otherwise.
HTML code of page for reference:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta
name="viewport"
content="width=device-width,initial-scale=1,shrink-to-fit=no"
/>
<meta name="theme-color" content="#000000" />
<link rel="manifest" href="/manifest.json" />
<link rel="”shortcut" icon” href="”/favicon.ico”" />
<title>Investor Scout</title>
<script>
!function(n,u){n._rwq=u,n[u]=n[u]||function(){(n[u].q=n[u].q||[]).push(arguments)}}(window,"rewardful")
</script>
<script
async
src="https://r.wdfl.co/rw.js"
data-rewardful="76b542"
></script>
<script type="text/javascript">
var _iub=_iub||[];_iub.csConfiguration={consentOnContinuedBrowsing:!1,ccpaAcknowledgeOnDisplay:!0,whitelabel:!1,lang:"en",siteId:2020596,enableCcpa:!0,countryDetection:!0,cookiePolicyId:26558236,banner:{acceptButtonDisplay:!0,customizeButtonDisplay:!0,acceptButtonColor:"#0073CE",acceptButtonCaptionColor:"white",customizeButtonColor:"#DADADA",customizeButtonCaptionColor:"#4D4D4D",rejectButtonColor:"#0073CE",rejectButtonCaptionColor:"white",position:"float-top-center",textColor:"black",backgroundColor:"white"}}
</script>
<script
type="text/javascript"
src="//cdn.iubenda.com/cs/ccpa/stub.js"
></script>
<script
type="text/javascript"
src="//cdn.iubenda.com/cs/iubenda_cs.js"
charset="UTF-8"
async
></script>
<script
defer="defer"
src="https://use.fontawesome.com/releases/v5.3.1/js/all.js"
></script>
<script type="text/javascript">
window.__lo_site_id=176375,function(){var t=document.createElement("script");t.type="text/javascript",t.async=!0,t.src="https://d10lpsik1i8c69.cloudfront.net/w.js";var e=document.getElementsByTagName("script")[0];e.parentNode.insertBefore(t,e)}()
</script>
<script type="text/javascript">
window.$crisp=[],window.CRISP_WEBSITE_ID="95efad36-fefd-4cf1-ae4b-a3bb5a61360c",d=document,s=d.createElement("script"),s.src="https://client.crisp.chat/l.js",s.async=1,d.getElementsByTagName("head")[0].appendChild(s)
</script>
<link href="/static/css/main.fc05b0f9.chunk.css" rel="stylesheet" />
</head>
<body>
<noscript>You need to enable JavaScript to run this app.</noscript>
<div id="root"></div>
<script>
!function(e){function t(t){for(var n,i,l=t[0],f=t[1],a=t[2],p=0,s=[];p<l.length;p++)i=l[p],Object.prototype.hasOwnProperty.call(o,i)&&o[i]&&s.push(o[i][0]),o[i]=0;for(n in f)Object.prototype.hasOwnProperty.call(f,n)&&(e[n]=f[n]);for(c&&c(t);s.length;)s.shift()();return u.push.apply(u,a||[]),r()}function r(){for(var e,t=0;t<u.length;t++){for(var r=u[t],n=!0,l=1;l<r.length;l++){var f=r[l];0!==o[f]&&(n=!1)}n&&(u.splice(t--,1),e=i(i.s=r[0]))}return e}var n={},o={1:0},u=[];function i(t){if(n[t])return n[t].exports;var r=n[t]={i:t,l:!1,exports:{}};return e[t].call(r.exports,r,r.exports,i),r.l=!0,r.exports}i.m=e,i.c=n,i.d=function(e,t,r){i.o(e,t)||Object.defineProperty(e,t,{enumerable:!0,get:r})},i.r=function(e){"undefined"!=typeof Symbol&&Symbol.toStringTag&&Object.defineProperty(e,Symbol.toStringTag,{value:"Module"}),Object.defineProperty(e,"__esModule",{value:!0})},i.t=function(e,t){if(1&t&&(e=i(e)),8&t)return e;if(4&t&&"object"==typeof e&&e&&e.__esModule)return e;var r=Object.create(null);if(i.r(r),Object.defineProperty(r,"default",{enumerable:!0,value:e}),2&t&&"string"!=typeof e)for(var n in e)i.d(r,n,function(t){return e[t]}.bind(null,n));return r},i.n=function(e){var t=e&&e.__esModule?function(){return e.default}:function(){return e};return i.d(t,"a",t),t},i.o=function(e,t){return Object.prototype.hasOwnProperty.call(e,t)},i.p="/";var l=this["webpackJsonpinvestor-scout"]=this["webpackJsonpinvestor-scout"]||[],f=l.push.bind(l);l.push=t,l=l.slice();for(var a=0;a<l.length;a++)t(l[a]);var c=f;r()}([])
</script>
<script src="/static/js/2.540fc93a.chunk.js"></script>
<script src="/static/js/main.9ac00620.chunk.js"></script>
</body>
</html>
Hello World,
New in Python, I am trying to webscrape a javascript page : https://search.gleif.org/#/search/
Please find below the result from my code (using request)
<!DOCTYPE html>
<html>
<head><meta charset="utf-8"/>
<meta content="width=device-width,initial-scale=1" name="viewport"/>
<title>LEI Search 2.0</title>
<link href="/static/icons/favicon.ico" rel="shortcut icon" type="image/x-icon"/>
<link href="https://fonts.googleapis.com/css?family=Open+Sans:200,300,400,600,700,900&subset=cyrillic,cyrillic-ext,greek,greek-ext,latin-ext,vietnamese" rel="stylesheet"/>
<link href="/static/css/main.045139db483277222eb714c1ff8c54f2.css" rel="stylesheet"/></head>
<body>
<div id="app"></div>
<script src="/static/js/manifest.2ae2e69a05c33dfc65f8.js" type="text/javascript"></script>
<script src="/static/js/vendor.6bd9028998d5ca3bb72f.js" type="text/javascript"></script>
<script src="/static/js/main.5da23c5198041f0ec5af.js" type="text/javascript"></script>
</body>
</html>
The question:
Instead of retrieving the above script:
"src="/static/js/manifest.2ae2e69a05c33dfc65f8.js" type="text/javascript""
I would like to have the content of the table in order to store it.
Table that I want to scrape
Following code is written using PySelenium.
import time
from selenium import webdriver
country = []
legal_name = []
lei = []
driver = webdriver.Chrome()
driver.implicitly_wait(5)
for i in range(1,30395):
driver.get('https://search.gleif.org/#/search/fulltextFilterId=LEIREC_FULLTEXT¤tPage='+str(i)+'&perPage=50&expertMode=false#results-section')
time.sleep(5)
country += [i.get_attribute('innerHTML') for i in driver.find_elements_by_xpath('//*[#class="table-cell country"]/a')]
legal_name += [i.get_attribute('innerHTML') for i in driver.find_elements_by_xpath('//*[#class="table-cell legal-name"]/a')]
lei += [i.get_attribute('innerHTML') for i in driver.find_elements_by_xpath('//*[#class="table-cell lei"]/a')]
Logging in (Change this with the respective elements.)
driver.find_element_by_id("UserName").send_keys("xxxx")
driver.find_element_by_name("Password").send_keys("yyyy")
driver.find_element_by_class("loginButton").click()
Get page content
print(driver.page_source)
I am trying to write a Python script to scrape data from this webpage. I am trying to scrape the data from the second table ('class': 'char-pico-table') and am using this script to do so:
def getPICO(url):
r = requests.get(url)
print (r.content)
However, this prints this:
b'<!DOCTYPE html>\n<html class="view">\n <head>\n <title>RobotReviewer: Automating evidence synthesis</title>\n <meta charset="utf-8">\n <meta name="viewport" content="width=device-width, initial-scale=1.0">\n <meta name="google" content="notranslate">\n\n <link rel="stylesheet" type="text/css" href="//maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css">\n <link rel="stylesheet" type="text/css" href="/css/main.css">\n <link rel="stylesheet alternative prefetch" type=text/css href="/css/report.css">\n\n <!-- Preload examples -->\n <link rel="prefetch" href="/report_view/Tvg0-pHV2QBsYpJxE2KW-/html">\n <link rel="prefetch" href="/report_view/_fzGUEvWAeRsqYSmNQbBq/html">\n <link rel="prefetch" href="/report_view/HBkzX1I3Uz_kZEQYeqXJf/html">\n\n <!-- / Preload examples -->\n\n\n <script src="/scripts/modernizr.js"></script>\n <script src="/scripts/spa/scripts/vendor/pdfjs/pdf.js"></script>\n <script src="/scripts/spa/scripts/vendor/compatibility.js"></script>\n <script data-main="/scripts/main" src="/scripts/require.js"></script>\n\n <script>\n PDFJS.disableWebGL = false;\n CSRF_TOKEN = "1508009356##6a03b1bf519972b27a0d871ae4823eb3a3366c0c";\n </script>\n </head>\n\n <body>\n <nav id="top-bar" class="top-bar" data-topbar role="navigation">\n <div>\n <ul class="title-area">\n <li class="name">\n <h1><img src="/img/logo.svg" width="190px"></h1>\n </li>\n </ul>\n\n <section class="top-bar-section">\n <ul class="right">\n <li>About</li>\n </ul>\n </section>\n </div>\n </nav>\n\n <div id="breadcrumbs"></div>\n\n <main id="main"></main>\n\n\n </body>\n</html>'
which is not the output that I see when I view the page in my browser - it contains none of the data that I wish to scrape. Why is this not the case?
When viewing the page in a web browser it looks like this:
Expected Output
Based on the comment from #Shahin, I wrote the following code, which gave me the data in a JSON format from which I was easily able to extract the data.
result = json.loads(requests.get('https://robot-reviewer.vortext.systems/report_view/'+id+'/json').content)
This is my first foray into Selenium. Apologies in advance if this is a stupid/trivial question.
I am trying to scrape information from a webpage. With Python/Selenium I am able to log on to the site and get to the page with the information I need. After the page I need is displayed, I am issuing
time.sleep(20)
html_source = driver.page_source
print html_source
The "source" that gets printed is different from both the
right click and select view page source and
right click and select This Frame, View Frame source
The required information is in the View Frame source. All of this is in Firefox.
What do I need to do to get to the Frame Source? There is no frame name in the Frame Source.
Additional information below:
When I right click and select view page source I get the below:
<!DOCTYPE html><html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>xxxxxxx Portal</title>
<base href="https://website.org/page/">
<link rel="shortcut icon" href="images/logos/xxxxxxx.ico">
<meta http-equiv="Pragma" content="no-cache">
<meta http-equiv="Expires" content="-1"><script type="text/javascript" src="https://website.org/page/security/csrf.js"> </script><script type="text/javascript" src="https://website.org/page/security/csrf/execute.js"> </script><script>
function pushFocus()
{
frameDetail.focus();
}
function addInProgressPanel(doc)
{
var d = doc.createElement('div');
d.id="inProgressPane";
d.className="freezeOn";
var tbl = doc.createElement("table");
var row = tbl.insertRow(-1);
var oi = doc.createElement("img");
oi.src= 'https://website.org/page/'+ "images/actions/loading2.gif";
var td = doc.createElement("td");
td.className="detailFormField";
td.bgcolor="red";
td.appendChild(oi);
row.appendChild(td);
td = doc.createElement("td");
td.className="inProcessing";
td.appendChild(doc.createTextNode("Your Request is Being Processed ..."));
row.appendChild(td);
d.appendChild(tbl);
doc.body.appendChild(d);
return d;
}
function inProgressScreen(type)
{
var ws = frames["frameDetail"];
if(!ws) return true;
var ips = ws.document.getElementById("inProgressPane");
if(ips)
{
if(type) ips.className = 'freezeOn';
else ips.className = 'freezeOff';
}else if(type)
ips = addInProgressPanel(ws.document);
}
</script></head>
<frameset id="main" framespacing="0" frameborder="0">
<frame id="frameDetail" name="frameDetail" scrolling="auto" marginwidth="0" marginheight="0" src="portal/portal.xsl?x=portal.PortalOutline&lang=en&mode=notices">
</frameset>
</html>
When I right click and select This Frame, View Frame source I get
<!DOCTYPE html><html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<base href="https://website.org/xxxxxx/">
<meta http-equiv="Content-Language" content="en-us">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta http-equiv="Pragma" content="no-cache">
<meta http-equiv="Expires" content="-1">
<title>xxxxxxxx Portal</title>
<link rel="stylesheet" type="text/css" href="styles/portal/menu.css">
<link rel="stylesheet" type="text/css" href="styles/portal/header.css">
<link rel="stylesheet" type="text/css" href="styles/portal/footer.css">
<link rel="stylesheet" type="text/css" href="styles/portal/jquery-ui-1.8.7.portal.css">
<link rel="stylesheet" type="text/css" href="styles/portal/fg.menu.css">
<link rel="stylesheet" type="text/css" href="styles/portal/portal.css">
<link rel="stylesheet" type="text/css" href="styles/icons.css">
<link rel="stylesheet" type="text/css" href="styles/portal/notifications.css"><script type="text/javascript" src="https://website.org/xxxxxxxx/security/csrf.js"> </script><script type="text/javascript" src="https://website.org/xxxxxxxx/security/csrf/execute.js"> </script><script src="scripts/widgets/common.js"></script><script src="scripts/controller.js"></script><script src="scripts/portal.js"></script><script src="scripts/jquery/jquery-1.7.2.min.js"></script><script type="text/javascript" src="https://website.org/xxxxxxxx/security/csrf/jquery.js"> </script><script src="scripts/jquery/jquery-ui-1.8.16.min.js"></script><script src="scripts/jquery/fg.menu.js"></script><script src="portal/lang/datePickerLanguage.jsp?lang=en"></script><script src="portal/portal.js"></script><script src="portal/portalNoShim.js"></script><script>
Lots more code here. Did not paste as it was too long. There is no frame name other than the reference to iSessionFrame below:
</script><script language="javascript" src="portal/grades.js"></script></div>
</div>
</div>
<div id="footer">
<table id="language"><select id="locale" style="width:175px"></select></table>
</div>
</div><iframe id="iSessionFrame" name="iSessionFrame" width="0" height="0" src="https://website.org/xxxxxx/white.jsp" style="visibility:hidden;"></iframe></body>
</html>
Q: What do I need to do to get to the Frame Source?
A: First you must switch to the wanted frame using the switch_to command and then you should use .page_source to get the html source.
Obs.: take a look at Selenium Docs, more specifically at Moving between windows and frames.
Code:
driver.switch_to_frame(driver.find_element_by_tag_name("frameDetail"))
driver.page_source
You could try to switch to the frame using its ID :
driver.switch_to_frame(driver.find_element_by_id("iSessionFrame"))
driver.page_source