Access Denied Using Selenium and adding User Agent - python

I am trying to access a website and I keep getting the "access denied" message. I have googled and searched all over this, and everything points to using a "User Agent". I have added my user agent and it is not working. Here is my code:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
webpage = str('https://www.kroger.com/account/')
options = Options()
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Firefox/78.0'
options.add_argument('user-agent={0}'.format(user_agent))
driver = webdriver.Chrome('/Path/chromedriver', options=options)
driver.get(webpage)
create = driver.find_element_by_xpath('/html/body/div[1]/div[2]/div[2]/div/main/section/section/section/button[2]')
create.click()
When I click on the create account page (I am using this to demonstrate the error I am getting) it takes me to this page:
And after refreshing it gives me the error.
The web page will load, but as soon as I do anything (Sometimes even manually tabbing between the boxes) it will kick me off and take me to the Access Denied page. Any way to resolve this?
EDIT: I have added code to click the "Create Account" button so to show the error that I am getting, and I have also added a photo of the page it sends me to before hitting the Error page.

It's not that clear in which circumstances you are facing access denied page. However I have executed your usecase and here are the observations:
Code Block:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-logging"])
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get('https://www.kroger.com/account/')
print(driver.page_source)
Browser Snapshot:
ConsoleOutput:
<html lang="en-us" data-react-helmet="lang" class="hydrated"><head>
<meta charset="utf-8"><style data-styles="">kds-tooltippable{visibility:hidden}.hydrated{visibility:inherit}</style>
<meta name="viewport" content="width=device-width,initial-scale=1">
<meta name="google-site-verification" content="mLDjWodVihPJXvMAL0-8hcbuNludulWFtLJ5FFFMbyk">
<meta name="apple-itunes-app" content="app-id=403901186">
<meta name="theme-color" content="#0067b1">
<iframe src="javascript:void(0)" title="" style="width: 0px; height: 0px; border: 0px; display: none;"></iframe><script src="https://apis.google.com/_/scs/apps-static/_/js/k=oz.gapi.en_US.hc3rLxj9u8o.O/m=auth2/rt=j/sv=1/d=1/ed=1/am=wQE/rs=AGLTcCMtAagp6kGxB19Nep_bTJunj37kww/cb=gapi.loaded_0" async=""></script><script type="text/javascript" src="https://www.kroger.com/resources/0f06f8547303cb204a2ba5ee8d0c2be4f278e07179439"></script><script type="text/javascript" src="/ruxitagentjs_ICA27SVfghjqrtux_10197200717183318.js" data-dtconfig="rid=RID_-461653321|rpid=-319708356|domain=kroger.com|reportUrl=/rb_7571065c-f052-471e-a3d7-f99d529548bb|app=81222ad3b2deb1ef|agentId=215b1e64d6441901|ssc=1|featureHash=ICA27SVfghjqrtux|vcv=1|rdnt=0|uxrgce=1|cuc=49xey1j6|md=mdcc1=cabTest,mdcc3=bdocument.referrer,mdcc4=bs.visitorID,mdcc6=bs.transactionID,mdcc7=cs_ecid,mdcc8=adiv[data-qa^e^dqsubmit-error^dq] .kds-Message-content,mdcc9=bs_dtm.pageName,mdcc10=cStoreCode,mdcc11=cStoreZipCode,mdcc12=cStoreLocalName,mdcc13=dutm_medium,mdcc14=dutm_campaign,mdcc15=dutm_content,mdcc16=dutm_source,mdcc17=bkrgrData.payload.metaData.campaignID,mdcc18=bsearchCID,mdcc19=ali[data-qa^e^dqCartEstimatedTotal-subTotal^dq],mdcc20=bnavigator.userAgent,mdcc21=cloggedin|lastModification=1597688640250|dtVersion=10197200717183318|tp=500,50,0,1|uxdcw=1500|agentUri=/ruxitagentjs_ICA27SVfghjqrtux_10197200717183318.js"></script><link rel="search" type="application/opensearchdescription+xml" href="/osd.xml" title="Kroger">
<link rel="manifest" href="/site.webmanifest">
<link rel="apple-touch-icon" href="/apple-touch-icon.png">
<title>Kroger</title>
<script src="/sa/kroger-header.d2aa6e624b99b8e4993b.js" defer=""></script>
<script src="/sa/#kroger/account-sign-in.5570149badf101ae09f5.js" defer=""></script>
<script src="/sa/coupons~main.e313a51a37dbad8980b1.js" defer=""></script>
<script src="/sa/products~main.6fd57a24319d5b8ad376.js" defer=""></script>
<script src="/sa/redux~main.b54055dbff5d2dbea98b.js" defer=""></script>
<script src="/sa/internal~main.b6c4b86585460ad2d826.js" defer=""></script>
<script src="/sa/kds~main.a4e3dbc91309d0b1dbb5.js" defer=""></script>
<script src="/sa/time~main.689d89c867b93785cd58.js" defer=""></script>
<script src="/sa/react~main.0f8c529ae5985d95333e.js" defer=""></script>
<script src="/sa/compat~main.a1504007c3b3afabc8e0.js" defer=""></script>
<script src="/sa/common~main.f64c9b672d7d0a00c2d7.js" defer=""></script>
<script src="/sa/vendors~main.725b80732ad8d3325d46.js" defer=""></script>
<script src="/sa/main.240039c3d849b8bd33bc.js" defer=""></script>
<link data-react-helmet="true" rel="canonical" href="https://www.kroger.com/signin">
<link rel="stylesheet" href="/sa/vendors~main.d3cc9575af.css">
<link rel="stylesheet" href="/sa/internal~main.00555b7772.css">
<link rel="stylesheet" href="/sa/products~main.a1bfd3c28a.css">
<link rel="stylesheet" href="/sa/coupons~main.c36bbd64b9.css">
<link rel="stylesheet" href="/sa/kroger-header.75a650a0c2.css">
.
.
.
<div id="ZN_dnk7EnVUuZidS97"></div>
<noscript><img src="https://www.kroger.com/akam/11/pixel_29e0b938?a=dD0zZGYzZWUxMjAzZDM3ZmRlYjA3YjExYjRkM2Y2MDlmOWJlOGUxNDY1JmpzPW9mZg==" style="visibility: hidden; position: absolute; left: -999px; top: -999px;" /></noscript><script type="text/javascript">var _cf = _cf || []; _cf.push(['_setFsp', true]); _cf.push(['_setBm', true]); _cf.push(['_setAu', '/resources/0f06f85473rn244317954ff2256514de']); </script><script type="text/javascript" src="/resources/0f06f85473rn244317954ff2256514de"></script>
<div id="kds-Portal-toast" class="kds-Portal pointer-events-none undefined"><div class="kds-ToastGroup"></div></div><iframe sandbox="allow-scripts allow-same-origin" title="Adobe ID Syncing iFrame" id="destination_publishing_iframe_kroger_0" name="destination_publishing_iframe_kroger_0_name" src="https://kroger.demdex.net/dest5.html?d_nsid=0#https%3A%2F%2Fwww.kroger.com" style="display: none; width: 0px; height: 0px;"></iframe></body></html>

Related

Can't scraping a javascript page even with selenium

I try to scrap some data from a js web site but even with selenium it still can't reach it.
from discord.ext import commands
from selenium.webdriver.chrome.options import Options
from selenium import webdriver
from selenium.webdriver.common.by import By
options = Options()
options.add_argument("--headless")
options.add_argument("-no-sandbox")
options.add_argument("-disable-dev-shm-usage")
driver = webdriver.Chrome('chromedriver', options=options)
driver.get('http://mc164.boxtoplay.com:65248')
print(driver.page_source.encode("utf-8"))
The output:
b'<html lang="en"><head>\n\n\t<title>Minecraft Dynamic Map</title>\n\n\t<meta charset="utf-8">\n\t<meta name="keywords" content="minecraft, map, dynamic">\n\t<meta name="description" content="Minecraft Dynamic Map">\n\t<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=no">\n\t<!-- These 2 lines make us fullscreen on apple mobile products - remove if you don\'t like that -->\n\t<meta name="apple-mobile-web-app-capable" content="yes">\n\t<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent">\t\n\n\t<link rel="icon" href="images/dynmap.ico" type="image/ico">\n\n\t<script type="text/javascript" src="js/jquery-3.5.1.js?_=3.3.2-696"></script>\n\t<link rel="stylesheet" type="text/css" href="css/leaflet.css?_=3.3.2-696">\n\t<script type="text/javascript" src="js/leaflet.js?_=3.3.2-696"></script>\n <!-- FOr source debug on leaflet \t<script type="text/javascript" src="js/leaflet-src.js?_=3.3.2-696"></script> -->\n\t<script type="text/javascript" src="js/custommarker.js?_=3.3.2-696"></script>\n\n\t<script type="text/javascript" src="js/dynmaputils.js?_=3.3.2-696"></script>\n\t<script type="text/javascript" src="js/sidebarutils.js?_=3.3.2-696"></script>\n\n\t<!--<link rel="stylesheet" type="text/css" href="css/embedded.css" media="screen" />-->\n\t<link rel="stylesheet" type="text/css" href="css/standalone.css?_=3.3.2-696" media="screen">\n\t<link rel="stylesheet" type="text/css" href="css/dynmap_style.css?_=3.3.2-696" media="screen">\n\t<!-- <link rel="stylesheet" type="text/css" href="css/override.css" media="screen" /> -->\n\n\t<script type="text/javascript" src="version.js?_=3.3.2-696"></script>\n\t<script type="text/javascript" src="js/jquery.json.js?_=3.3.2-696"></script>\n\t<script type="text/javascript" src="js/jquery.mousewheel.js?_=3.3.2-696"></script>\n\t<script type="text/javascript" src="js/minecraft.js?_=3.3.2-696"></script>\n\t<script type="text/javascript" src="js/map.js?_=3.3.2-696"></script>\n\t<script type="text/javascript" src="js/hdmap.js?_=3.3.2-696"></script>\n\t<script type="text/javascript" src="standalone/config.js?_=3.3.2-696"></script>\n\n\t<script type="text/javascript">\n\t\t\t$(document).ready(function() {\n\t\t\t\twindow.dynmap = new DynMap($.extend({\n\t\t\t\t\tcontainer: $(\'#mcmap\')\n\t\t\t\t}, config));\n\t\t\t});\n\t</script>\n\n</head>\n<body>\n<noscript>\n For full functionality of this site it is necessary to enable JavaScript.\n Here are the \n instructions how to enable JavaScript in your web browser.\n</noscript>\n\n\t<div id="mcmap"></div>\n\n</body></html>'
I think, javascript is not executed while in headless mode. You can try without headleass mode.
if the app is developed using javascript. you can try any javascript framework to capture the data.

Access Denied You don't have permission to access "site" on this server using ChromeDriver and Chrome through Selenium Python

I've been using selenium with chrome and I've been trying to access a website but I keep receiving this error message:
Access Denied You don't have permission to access "website" on this server.
The website works fine on my regular chrome browser, but when opened through selenium I receive this error message.
I've tried the regular troubleshooting for the error but it didn't help, I've tried installing firefox but that resulted in another error: "Message: Unable to find a matching set of capabilities"
browser = webdriver.Chrome('C:\\Users\Ferdinand\Downloads\chromedriver_win32\chromedriver.exe')
browser.get('https://www.vitacost.com/MyAccount/Login.aspx')
I'm completely stuck here, I tried it another computer and still the same result.
If your trying has Headless option, I think it might be related to user-agent.
Because some web site block the user-agent with Headless.
So, go to https://www.whatsmyua.info/ to find your user-agent which works in your regular chrome browser and try again with this.
ChromeOptions options = new ChromeOptions();
String userAget = "YOUR USER AGENT";
options.addArguments(String.format("user-agent=%s", userAgent));
This is a java code but I believe there is an exactly same way in python.
A bit unclear in which circumstances you are facing the error ..."Access Denied You don't have permission to access "website" on this server.... However using Selenium and ChromeDriver to access the website you can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get('https://www.vitacost.com/MyAccount/Login.aspx')
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "h1.h1")))
print(driver.page_source)
driver.quit()
Console Output:
<html class="desktop js flexbox flexboxlegacy no-touch" lang="en" style=""><head class="at-element-marker"><title>
Login - Vitacost
</title>
<meta charset="UTF-8">
<meta name="robots" content="noindex,nofollow">
<meta name="googlebot" content="noindex,nofollow">
<meta http-equiv="expires" content="0">
<meta name="description" content="Buy discount vitamins, nutritional supplements, low carb, weight loss supplements and other health products from Vitacost. Vitacost sells top brand vitamins at wholesale cost. Save 33% - 75% on every nutritional product we carry. Why are you paying retail prices?">
<meta name="keywords" content="Vitacost, vitamin, supplement, buy vitamin, discount vitamin, health supplement, vita, vitamin store, wholesale vitamin, vitacost.com, wholesale vitamin supplement, vitacost vitamin">
<link rel="shortcut icon" href="/favicon.ico">
<link rel="canonical" href="https://www.vitacost.com/myaccount/login.aspx">
<link rel="P3Pv1" href="http: //www.vitacost.com/w3c/p3p.xml">
<link href="/res/(19.12.2)/Styles/vitacostCore.min.css" rel="styleSheet" type="text/css">
<link href="/res/(19.12.2)/Styles/vitacostProduct.min.css" rel="styleSheet" type="text/css">
<link href="/res/(19.12.2)/Styles/vitacostAux.min.css" rel="styleSheet" type="text/css">
<link href="/res/(19.12.2)/Styles/vitacostShared.min.css" rel="styleSheet" type="text/css">
<link href="https://www.vitacost.com/styles/modules.css?ref050819" rel="styleSheet" type="text/css">
<link href="/res/(19.12.2)/Styles/print.css" rel="styleSheet" media="print" type="text/css">
<iframe src="javascript:void(0)" title="" style="width: 0px; height: 0px; border: 0px; display: none;"></iframe><script src="https://js-agent.newrelic.com/nr-1158.min.js"></script><script async="true" type="text/javascript" src="https://sslwidget.criteo.com/event?a=3439&v=5.5.0&p0=e%3Dexd%26ci%3D%26site_type%3Dd&p1=e%3Dvh&p2=e%3Ddis&adce=1&tld=vitacost.com&dtycbr=29175" data-owner="criteo-tag"></script><script type="text/javascript" async="" src="https://www.googleadservices.com/pagead/conversion_async.js"></script><script id="socialannex-s13-core" async="" src="//cdn.socialannex.com/s13/v2/s13-main.min.js"></script><script type="text/javascript" async="" src="https://offer.slgnt.us/e146f0b2dfa340dca911e2ac94675415/51bb5e46657842.js"></script><script src="https://static.criteo.net/js/ld/ld.js"></script><script async="" src="//dp.shoprunner.com/2.9.0/sp.js"></script><script id="socialannex-s13" async="" src="//cdn.socialannex.com/partner/2209221/s13.js"></script><script type="text/javascript" async="" src="https://tracker.marinsm.com/tracker/async/15b71u0950.js"></script><script src="https://cltgtstor001.blob.core.windows.net/eddb5e6a-4e95-4420-9313-e6876dc0a04b/eddb5e6a4e9544209313e6876dc0a04b_1.js" type="text/javascript"></script><script src="//bat.bing.com/bat.js" async=""></script><script src="https://connect.facebook.net/signals/config/967518976648812?v=2.9.15&r=stable" async=""></script><script async="" src="//connect.facebook.net/en_US/fbevents.js"></script><script>
// akam-sw.js install script version 1.3.3
"serviceWorker"in navigator&&"find"in[]&&function(){var e=new Promise(function(e){"complete"===document.readyState||!1?e():(window.addEventListener("load",function(){e()}),setTimeout(function(){"complete"!==document.readyState&&e()},1e4))}),n=window.akamServiceWorkerInvoked,r="1.3.3";if(n)aka3pmLog("akam-setup already invoked");else{window.akamServiceWorkerInvoked=!0,window.aka3pmLog=function(){window.akamServiceWorkerDebug&&console.log.apply(console,arguments)};function o(e){(window.BOOMR_mq=window.BOOMR_mq||[]).push(["addVar",{"sm.sw.s":e,"sm.sw.v":r}])}var i="/akam-sw.js",a=new Map;navigator.serviceWorker.addEventListener("message",function(e){var n,r,o=e.data;if(o.isAka3pm)if(o.command){var i=(n=o.command,(r=a.get(n))&&r.length>0?r.shift():null);i&&i(e.data.response)}else if(o.commandToClient)switch(o.commandToClient){case"enableDebug":window.akamServiceWorkerDebug||(window.akamServiceWorkerDebug=!0,aka3pmLog("Setup script debug enabled via service worker message"),p());break;case"boomerangMQ":o.payload&&(window.BOOMR_mq=window.BOOMR_mq||[]).push(o.payload)}aka3pmLog("akam-sw message: "+JSON.stringify(e.data))});var t=function(e){return new Promise(function(n){var r,o;r=e.command,o=n,a.has(r)||a.set(r,[]),a.get(r).push(o),navigator.serviceWorker.controller&&(e.isAka3pm=!0,navigator.serviceWorker.controller.postMessage(e))})},c=function(e){return t({command:"navTiming",navTiming:e})},s=null,m={},d=function(){var e=i;return s&&(e+="?othersw="+encodeURIComponent(s)),function(e,n){return new Promise(function(r,i){aka3pmLog("Registering service worker with URL: "+e),navigator.serviceWorker.register(e,n).then(function(e){aka3pmLog("ServiceWorker registration successful with scope: ",e.scope),r(e),o(1)}).catch(function(e){aka3pmLog("ServiceWorker registration failed: ",e),o(0),i(e)})})}(e,m)},g=ServiceWorkerContainer.prototype.register;if(ServiceWorkerContainer.prototype.register=function(n,r){return n.includes(i)?g.call(this,n,r):(aka3pmLog("Overriding registration of service worker for: "+n),s=new URL(n,window.location.href),m=r,navigator.serviceWorker.controller?new Promise(function(n,r){var o=navigator.serviceWorker.controller.scriptURL;if(o.includes(i)){var a=encodeURIComponent(s);o.includes(a)?(aka3pmLog("Cancelling registration as we already integrate other SW: "+s),navigator.serviceWorker.getRegistration().then(function(e){n(e)})):e.then(function(){aka3pmLog("Unregistering existing 3pm service worker"),navigator.serviceWorker.getRegistration().then(function(e){e.unregister().then(function(){return d()}).then(function(e){n(e)}).catch(function(e){r(e)})})})}else aka3pmLog("Cancelling registration as we already have akam-sw.js installed"),navigator.serviceWorker.getRegistration().then(function(e){n(e)})}):g.call(this,n,r))},navigator.serviceWorker.controller){var u=navigator.serviceWorker.controller.scriptURL;u.includes("/akam-sw.js")||u.includes("/akam-sw-preprod.js")||u.includes("/threepm-sw.js")||(aka3pmLog("Detected existing service worker. Removing and re-adding inside akam-sw.js"),s=new URL(u,window.location.href),e.then(function(){navigator.serviceWorker.getRegistration().then(function(e){m={scope:e.scope},e.unregister(),d()})}))}else e.then(function(){window.akamServiceWorkerPreprod&&(i="/akam-sw-preprod.js"),d()});if(window.performance){var w=window.performance.timing,l=w.responseEnd-w.responseStart;c(l)}e.then(function(){t({command:"pageLoad"})});var k=!1;function p(){window.akamServiceWorkerDebug&&!k&&(k=!0,aka3pmLog("Initializing debug functions at window scope"),window.aka3pmInjectSwPolicy=function(e){return t({command:"updatePolicy",policy:e})},window.aka3pmDisableInjectedPolicy=function(){return t({command:"disableInjectedPolicy"})},window.aka3pmDeleteInjectedPolicy=function(){return t({command:"deleteInjectedPolicy"})},window.aka3pmGetStateAsync=function(){return t({command:"getState"})},window.aka3pmDumpState=function(){aka3pmGetStateAsync().then(function(e){aka3pmLog(JSON.stringify(e,null,"\t"))})},window.aka3pmInjectTiming=function(e){return c(e)},window.aka3pmUpdatePolicyFromNetwork=function(){return t({command:"pullPolicyFromNetwork"})})}p()}}();</script>
<script type="text/javascript">window.NREUM||(NREUM={});NREUM.info = {"beacon":"bam.nr-data.net","errorBeacon":"bam.nr-data.net","licenseKey":"d5fad1d432","applicationID":"38996487,13782876","transactionName":"ZFRVZEJSVxUDURJYDV0edmNgHFQfA1EFXhddRRhcX1RQCExTFUEa","queueTime":0,"applicationTime":310,"agent":"","atts":""}</script><script type="text/javascript">(window.NREUM||(NREUM={})).loader_config={licenseKey:"d5fad1d432",applicationID:"38996487"};window.NREUM||(NREUM={}),__nr_require=function(n,e,t){function r(t){if(!e[t]){var i=e[t]={exports:{}};n[t][0].call(i.exports,function(e){var i=n[t][1][e];return r(i||e)},i,i.exports)}return e[t].exports}if("function"==typeof __nr_require)return __nr_require;for(var i=0;i<t.length;i++)r(t[i]);return r}({1:[function(n,e,t){function r(){}function i(n,e,t){return function(){return o(n,[u.now()].concat(f(arguments)),e?null:this,t),e?void 0:this}}var o=n("handle"),a=n(4),f=n(5),c=n("ee").get("tracer"),u=n("loader"),s=NREUM;"undefined"==typeof window.newrelic&&(newrelic=s);var p=["setPageViewName","setCustomAttribute","setErrorHandler","finished","addToTrace","inlineHit","addRelease"],d="api-",l=d+"ixn-";a(p,function(n,e){s[e]=i(d+e,!0,"api")}),s.addPageAction=i(d+"addPageAction",!0),s.setCurrentRouteName=i(d+"routeName",!0),e.exports=newrelic,s.interaction=function(){return(new r).get()};var m=r.prototype={createTracer:function(n,e){var t={},r=this,i="function"==typeof e;return o(l+"tracer",[u.now(),n,t],r),function(){if(c.emit((i?"":"no-")+"fn-start",[u.now(),r,i],t),i)try{return e.apply(this,arguments)}catch(n){throw c.emit("fn-err",[arguments,this,n],t),n}finally{c.emit("fn-end",[u.now()],t)}}}};a("actionText,setName,setAttribute,save,ignore,onEnd,getContext,end,get".split(","),function(n,e){m[e]=i(l+e)}),newrelic.noticeError=function(n,e){"string"==typeof n&&(n=new Error(n)),o("err",[n,u.now(),!1,e])}},{}],2:[function(n,e,t){function r(n,e){var t=n.getEntries();t.forEach(function(n){"first-paint"===n.name?a("timing",["fp",Math.floor(n.startTime)]):"first-contentful-paint"===n.name&&a("timing",["fcp",Math.floor(n.startTime)])})}function i(n){if(n instanceof c&&!s){var e,t=Math.round(n.timeStamp);e=t>1e12?Date.now()-t:f.now()-t,s=!0,a("timing",["fi",t,{type:n.type,fid:e}])}}if(!("init"in NREUM&&"page_view_timing"in NREUM.init&&"enabled"in NREUM.init.page_view_timing&&NREUM.init.page_view_timing.enabled===!1)){var o,a=n("handle"),f=n("loader"),c=NREUM.o.EV;if("PerformanceObserver"in window&&"function"==typeof window.PerformanceObserver){o=new PerformanceObserver(r);try{o.observe({entryTypes:["paint"]})}catch(u){}}if("addEventListener"in document){var s=!1,p=["click","keydown","mousedown","pointerdown","touchstart"];p.forEach(function(n){document.addEventListener(n,i,!1)})}}},{}],3:[function(n,e,t){function r(n,e){if(!i)return!1;if(n!==i)return!1;if(!e)return!0;if(!o)return!1;for(var t=o.split("."),r=e.split("."),a=0;a<r.length;a++)if(r[a]!==t[a])return!1;return!0}var i=null,o=null,a=/Version\/(\S+)\s+Safari/;if(navigator.userAgent){var f=navigator.userAgent,c=f.match(a);c&&f.indexOf("Chrome")===-1&&f.indexOf("Chromium")===-1&&(i="Safari",o=c[1])}e.exports={agent:i,version:o,match:r}},{}],4:[function(n,e,t){function r(n,e){var t=[],r="",o=0;for(r in n)i.call(n,r)&&(t[o]=e(r,n[r]),o+=1);return t}var i=Object.prototype.hasOwnProperty;e.exports=r},{}],5:[function(n,e,t){function r(n,e,t){e||(e=0),"undefined"==typeof t&&(t=n?n.length:0);for(var r=-1,i=t-e||0,o=Array(i<0?0:i);++r<i;)o[r]=n[e+r];return o}e.exports=r},{}],6:[function(n,e,t){e.exports={exists:"undefined"!=typeof window.performance&&window.performance.timing&&"undefined"!=typeof window.performance.timing.navigationStart}},{}],ee:[function(n,e,t){function r(){}function i(n){function e(n){return n&&n instanceof r?n:n?c(n,f,o):o()}function t(t,r,i,o){if(!d.aborted||o){n&&n(t,r,i);for(var a=e(i),f=v(t),c=f.length,u=0;u<c;u++)f[u].apply(a,r);var p=s[y[t]];return p&&p.push([b,t,r,a]),a}}function l(n,e){h[n]=v(n).concat(e)}function m(n,e){var t=h[n];if(t)for(var r=0;r<t.length;r++)t[r]===e&&t.splice(r,1)}function v(n){return h[n]||[]}function g(n){return p[n]=p[n]||i(t)}function w(n,e){u(n,function(n,t){e=e||"feature",y[t]=e,e in s||(s[e]=[])})}var h={},y={},b={on:l,addEventListener:l,removeEventListener:m,emit:t,get:g,listeners:v,context:e,buffer:w,abort:a,aborted:!1};return b}function o(){return new r}function a(){(s.api||s.feature)&&(d.aborted=!0,s=d.backlog={})}var f="nr#context",c=n("gos"),u=n(4),s={},p={},d=e.exports=i();d.backlog=s},{}],gos:[function(n,e,t){function r(n,e,t){if(i.call(n,e))return n[e];var r=t();if(Object.defineProperty&&Object.keys)try{return Object.defineProperty(n,e,{value:r,writable:!0,enumerable:!1}),r}catch(o){}return n[e]=r,r}var i=Object.prototype.hasOwnProperty;e.exports=r},{}],handle:[function(n,e,t){function r(n,e,t,r){i.buffer([n],r),i.emit(n,e,t)}var i=n("ee").get("handle");e.exports=r,r.ee=i},{}],id:[function(n,e,t){function r(n){var e=typeof n;return!n||"object"!==e&&"function"!==e?-1:n===window?0:a(n,o,function(){return i++})}var i=1,o="nr#id",a=n("gos");e.exports=r},{}],loader:[function(n,e,t){function r(){if(!x++){var n=E.info=NREUM.info,e=l.getElementsByTagName("script")[0];if(setTimeout(s.abort,3e4),!(n&&n.licenseKey&&n.applicationID&&e))return s.abort();u(y,function(e,t){n[e]||(n[e]=t)}),c("mark",["onload",a()+E.offset],null,"api");var t=l.createElement("script");t.src="https://"+n.agent,e.parentNode.insertBefore(t,e)}}function i(){"complete"===l.readyState&&o()}function o(){c("mark",["domContent",a()+E.offset],null,"api")}function a(){return O.exists&&performance.now?Math.round(performance.now()):(f=Math.max((new Date).getTime(),f))-E.offset}var f=(new Date).getTime(),c=n("handle"),u=n(4),s=n("ee"),p=n(3),d=window,l=d.document,m="addEventListener",v="attachEvent",g=d.XMLHttpRequest,w=g&&g.prototype;NREUM.o={ST:setTimeout,SI:d.setImmediate,CT:clearTimeout,XHR:g,REQ:d.Request,EV:d.Event,PR:d.Promise,MO:d.MutationObserver};var h=""+location,y={beacon:"bam.nr-data.net",errorBeacon:"bam.nr-data.net",agent:"js-agent.newrelic.com/nr-1158.min.js"},b=g&&w&&w[m]&&!/CriOS/.test(navigator.userAgent),E=e.exports={offset:f,now:a,origin:h,features:{},xhrWrappable:b,userAgent:p};n(1),n(2),l[m]?(l[m]("DOMContentLoaded",o,!1),d[m]("load",r,!1)):(l[v]("onreadystatechange",i),d[v]("onload",r)),c("mark",["firstbyte",f],null,"api");var x=0,O=n(6)},{}],"wrap-function":[function(n,e,t){function r(n){return!(n&&n instanceof Function&&n.apply&&!n[a])}var i=n("ee"),o=n(5),a="nr#original",f=Object.prototype.hasOwnProperty,c=!1;e.exports=function(n,e){function t(n,e,t,i){function nrWrapper(){var r,a,f,c;try{a=this,r=o(arguments),f="function"==typeof t?t(r,a):t||{}}catch(u){d([u,"",[r,a,i],f])}s(e+"start",[r,a,i],f);try{return c=n.apply(a,r)}catch(p){throw s(e+"err",[r,a,p],f),p}finally{s(e+"end",[r,a,c],f)}}return r(n)?n:(e||(e=""),nrWrapper[a]=n,p(n,nrWrapper),nrWrapper)}function u(n,e,i,o){i||(i="");var a,f,c,u="-"===i.charAt(0);for(c=0;c<e.length;c++)f=e[c],a=n[f],r(a)||(n[f]=t(a,u?f+i:i,o,f))}function s(t,r,i){if(!c||e){var o=c;c=!0;try{n.emit(t,r,i,e)}catch(a){d([a,t,r,i])}c=o}}function p(n,e){if(Object.defineProperty&&Object.keys)try{var t=Object.keys(n);return t.forEach(function(t){Object.defineProperty(e,t,{get:function(){return n[t]},set:function(e){return n[t]=e,e}})}),e}catch(r){d([r])}for(var i in n)f.call(n,i)&&(e[i]=n[i]);return e}function d(e){try{n.emit("internal-error",e)}catch(t){}}return n||(n=i),t.inPlace=u,t.flag=a,t}},{}]},{},["loader"]);</script><script language="JavaScript" src="/res/(19.12.2)/javascripts/vitacost.min.js" type="text/javascript"></script>
<script language="JavaScript" src="/res/(19.12.2)/javascripts/vitacost.custom.min.js" type="text/javascript"></script>
<script language="JavaScript" src="/res/(19.12.2)/JavaScripts/gzchange.js" type="text/javascript"></script>
<script type="text/javascript" language="javascript"><!--
var SPCExperience = true;
var PPDS = '';
var PPDNS = '';
var searchV2Enabled = false;
var ESCM = '';
var ESCMoc = '';
var ESCMos = '';
var vPageType = 'other';
var vSLTS = '2019-12-26 14:17:10 -08:00';
var vDSLT = '0';
var vSUTCTS = '2019-12-26 22:17:10';
var vSvr = 'LVDCWEB07';
var vCustomerID = '';
var vCustomerSegmentID = '';
var vProgramIDs = [];
var vLocale = 'US';
var vIsSRcheckout_enabled = 'True';
var vIsProductionEnvironment = '1';
$jq(document).ready(function(){new vcmc('MyCartHover','TopNav');});
var bumblebeeUserId='MmU4MmQwZGI1ZTFhMTQ2ZmRkY2Q1ZjRlZWQ1ZGExNzQ=';
var s_account='vitacost-global';
var s_linkInternalFiltersAppend='';
// --></script>
<script language="JavaScript" src="/res/(19.12.2)/JavaScripts/shoprunner_init.js" type="text/javascript"></script>
<script language="JavaScript" src="//assets.adobedtm.com/3c42c63b493c1af35d8caf2270583ef8ca5bdce4/satelliteLib-ef4882563ec768305944db752a2f61c31a748022.js" type="text/javascript"></script><link href="https://content.shoprunner.com/VITACOST.css" type="text/css" rel="stylesheet"><script src="https://content.shoprunner.com/VITACOST.js" type="text/javascript"></script><script src="https://assets.adobedtm.com/3c42c63b493c1af35d8caf2270583ef8ca5bdce4/mbox-contents-e848cae1f237cd42fb137c1796f98b33eafc3c2a.js"></script><script src="https://assets.adobedtm.com/3c42c63b493c1af35d8caf2270583ef8ca5bdce4/s-code-contents-71524c2997e1e185948c91ac4b0e3d25a9ad8106.js"></script><style id="at-mbox-default-style">.mboxDefault {visibility:hidden;}</style><style>.at-element-marker {visibility:visible;}</style><!-- Original Facebook Script -->
<script>
!function(f,b,e,v,n,t,s){if(f.fbq)return;n=f.fbq=function(){n.callMethod?
n.callMethod.apply(n,arguments):n.queue.push(arguments)};if(!f._fbq)f._fbq=n;
n.push=n;n.loaded=!0;n.version='2.0';n.queue=[];t=b.createElement(e);t.async=!0;
t.src=v;s=b.getElementsByTagName(e)[0];s.parentNode.insertBefore(t,s)}(window,
document,'script','//connect.facebook.net/en_US/fbevents.js');
fbq('init', '967518976648812');
fbq('track', "PageView");
</script>
<noscript>
<img height="1" width="1" style="display:none"
src="https://www.facebook.com/tr?id=967518976648812&ev=PageView&noscript=1"/>
</noscript><script>
(function(w,d,t,r,u){var f,n,i;w[u]=w[u]||[],f=function(){var o={ti:"4031096"};o.q=w[u],w[u]=new UET(o),w[u].push("pageLoad")},n=d.createElement(t),n.src=r,n.async=1,n.onload=n.onreadystatechange=function(){var s=this.readyState;s&&s!=="loaded"&&s!=="complete"||(f(),n.onload=n.onreadystatechange=null)},i=d.getElementsByTagName(t)[0],i.parentNode.insertBefore(n,i)})(window,document,"script","//bat.bing.com/bat.js","uetq");
</script>
<script>
if (window.vSKU == null || "undefined") {
window.vSKU = window.vSKUs;
}
console.log("bing_: " + window.vSKU);
window.uetq = window.uetq || [];
window.uetq.push(
'event', '',
{
'ecomm_prodid': window.vSKU,
'ecomm_pagetype': window.vPageType
}
);
</script>
<script>(window.BOOMR_mq=window.BOOMR_mq||[]).push(["addVar",{"rua.upush":"true","rua.cpush":"true","rua.upre":"true","rua.cpre":"true","rua.uprl":"false","rua.cprl":"false","rua.cprf":"false","rua.trans":"SJ-eeb5cac3-8acc-430a-ab8a-ea7c0b375f1e","rua.cook":"false","rua.ims":"false","rua.ufprl":"false","rua.cfprl":"true"}]);</script>
<script>!function(){function o(n,i){if(n&&i)for(var r in i)i.hasOwnProperty(r)&&(void 0===n[r]?n[r]=i[r]:n[r].constructor===Object&&i[r].constructor===Object?o(n[r],i[r]):n[r]=i[r])}try{var n=decodeURIComponent("");if(n.length>0&&window.JSON&&"function"==typeof window.JSON.parse){var i=JSON.parse(n);void 0!==window.BOOMR_config?o(window.BOOMR_config,i):window.BOOMR_config=i}}catch(r){window.console&&"function"==typeof window.console.error&&console.error("mPulse: Could not parse configuration",r)}}();</script>
<script>!function(e){var a="https://s.go-mpulse.net/boomerang/",t="addEventListener";if("False"=="True")e.BOOMR_config=e.BOOMR_config||{},e.BOOMR_config.PageParams=e.BOOMR_config.PageParams||{},e.BOOMR_config.PageParams.pci=!0,a="https://s2.go-mpulse.net/boomerang/";if(window.BOOMR_API_key="AZH8B-WP34K-EW2CF-6XVMH-NX5FB",function(){function n(a){e.BOOMR_onload=a&&a.timeStamp||(new Date).getTime()}if(!e.BOOMR||!e.BOOMR.version&&!e.BOOMR.snippetExecuted){e.BOOMR=e.BOOMR||{},e.BOOMR.snippetExecuted=!0;var i,o,r,_=document.createElement("iframe");if(e[t])e[t]("load",n,!1);else if(e.attachEvent)e.attachEvent("onload",n);_.src="javascript:void(0)",_.title="",_.role="presentation",(_.frameElement||_).style.cssText="width:0;height:0;border:0;display:none;",r=document.getElementsByTagName("script")[0],r.parentNode.insertBefore(_,r);try{o=_.contentWindow.document}catch(O){i=document.domain,_.src="javascript:var d=document.open();d.domain='"+i+"';void(0);",o=_.contentWindow.document}o.open()._l=function(){var e=this.createElement("script");if(i)this.domain=i;e.id="boomr-if-as",e.src=a+"AZH8B-WP34K-EW2CF-6XVMH-NX5FB",BOOMR_lstart=(new Date).getTime(),this.body.appendChild(e)},o.write("<bo"+'dy onload="document._l();">'),o.close()}}(),"".length>0)if(e&&"performance"in e&&e.performance&&"function"==typeof e.performance.setResourceTimingBufferSize)e.performance.setResourceTimingBufferSize();!function(){if(BOOMR=e.BOOMR||{},BOOMR.plugins=BOOMR.plugins||{},!BOOMR.plugins.AK){var a="true"=="true"?1:0,t="",n="nlavsoliomt4gxqfgfsq-f-cd9fb8aee-clientnsv4-s.akamaihd.net",i={"ak.v":25,"ak.cp":"322881","ak.ai":parseInt("198589",10),"ak.ol":"0","ak.cr":67,"ak.ipv":4,"ak.proto":"h2","ak.rid":"74e5043f","ak.r":36234,"ak.a2":a,"ak.m":"a","ak.n":"essl","ak.bpcip":"106.193.89.0","ak.cport":26067,"ak.gh":"104.115.39.191","ak.quicv":"","ak.tlsv":"tls1.3","ak.0rtt":"","ak.csrc":"-","ak.acc":"","ak.t":"1577398629"};if(""!==t)i["ak.ruds"]=t;var o={i:!1,av:function(a){var t="http.initiator";if(a&&(!a[t]||"spa_hard"===a[t]))i["ak.feo"]=void 0!==e.aFeoApplied?1:0,BOOMR.addVar(i)},rv:function(){var e=["ak.bpcip","ak.cport","ak.cr","ak.csrc","ak.gh","ak.ipv","ak.m","ak.n","ak.ol","ak.proto","ak.quicv","ak.tlsv","ak.0rtt","ak.r","ak.acc","ak.t"];BOOMR.removeVar(e)}};BOOMR.plugins.AK={akVars:i,akDNSPreFetchDomain:n,init:function(){if(!o.i){var e=BOOMR.subscribe;e("before_beacon",o.av,null,null),e("onbeacon",o.rv,null,null),o.i=!0}return this},is_complete:function(){return!0}}}}()}(window);</script><script>bazadebezolkohpepadr="1326392439"</script><script type="text/javascript" src="https://www.vitacost.com/akam/11/4f0f2637" defer=""></script><script language="javascript" type="text/javascript" src="https://mpsnare.iesnare.com/script/logo.js"></script><script src="https://assets.adobedtm.com/3c42c63b493c1af35d8caf2270583ef8ca5bdce4/scripts/satellite-592dd26e64746d5067001727.js"></script><script src="https://assets.adobedtm.com/3c42c63b493c1af35d8caf2270583ef8ca5bdce4/scripts/satellite-59cd065664746d6b7a00613f.js"></script><script src="https://assets.adobedtm.com/3c42c63b493c1af35d8caf2270583ef8ca5bdce4/scripts/satellite-5b4fab4564746d104700041e.js"></script><script src="https://ssl.google-analytics.com/ga.js"></script><script type="text/javascript">
/*mboxHighlight+ (1of2) v1 ==> Response Plugin*/
window.ttMETA=(typeof(window.ttMETA)!='undefined')?window.ttMETA:[];window.ttMETA.push({'mbox':'globalMbox','campaign':'','experience':'','offer':''});window.ttMBX=function(x){var mbxList=[];for(i=0;i<ttMETA.length;i++){if(ttMETA[i].mbox==x.getName()){mbxList.push(ttMETA[i])}}return mbxList[x.getId()]}
</script><iframe src="https://content.shoprunner.com/srsec/sra.html?partner=VITACOST" id="sr_tracking_connector" name="sr_tracking_connector" title="content" aria-hidden="true" class=" sr_iframe_hide" style="display: none !important; height: 0 !important; width: 0 !important;"></iframe><iframe src="https://content.shoprunner.com/sso.html?purl=https://www.vitacost.com/MyAccount/Login.aspx" title="shoprunner_secure_sso" aria-hidden="true" class=" sr_iframe_hide" style="display: none !important; height: 0 !important; width: 0 !important;"></iframe><script async="" src="https://static.hotjar.com/c/hotjar-1141144.js?sv=6"></script><script type="text/javascript" async="" src="//cltgtstor001.blob.core.windows.net/eddb5e6a-4e95-4420-9313-e6876dc0a04b/eddb5e6a4e9544209313e6876dc0a04b_2_1568798940.js"></script><script async="" src="https://script.hotjar.com/modules.297b225e0b92ebb96f25.js" charset="utf-8"></script><style type="text/css">iframe#_hjRemoteVarsFrame {display: none !important; width: 1px !important; height: 1px !important; opacity: 0 !important; pointer-events: none !important;}</style><script src="https://googleads.g.doubleclick.net/pagead/viewthroughconversion/1022050475/?random=1577397021494&cv=9&fst=1577397021494&num=1&bg=ffffff&guid=ON&resp=GooglemKTybQhCsO&eid=376635470&u_h=768&u_w=1366&u_ah=728&u_aw=1366&u_cd=24&u_his=2&u_tz=330&u_java=false&u_nplug=3&u_nmime=4&gtm=2oac61&sendb=1&ig=1&data=event%3Dgtag.config&frm=0&url=https%3A%2F%2Fwww.vitacost.com%2FMyAccount%2FLogin.aspx&tiba=Login%20-%20Vitacost&hn=www.googleadservices.com&async=1&rfmt=3&fmt=4"></script><script type="text/javascript" src="https://api.socialannex.com/s13/v2/service/get_config.php?siteid=2209221&callback=s13callback_1577397021692" async=""></script><script type="text/javascript" src="https://api.socialannex.com/s13/v2/service/get_button_small_html.php?siteid=2209221&divid=show_provider_small&callback=s13callback_1577397023318" async=""></script></head>
<body style="" class="email-modal-overlay-noScroll">

Python | Selenium cannot detect button

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
import os
Game_Pin = input('Enter your PIN: ')
NickNAME = input('Enter your nickname: ')
driver = webdriver.Chrome(executable_path=r"C:\WebDriver\bin\chromedriver.exe")
def Enter_Press(driver):
driver.find_element_by_xpath("//*[contains(text(), 'Enter')]").click()
def OK_GO(driver):
driver.find_element_by_xpath("//*[contains(text(), 'OK, go!')]").click()
def Kahoot_Spammer(Game_Pin, NickNAME, driver):
driver.get('https://kahoot.it/')
driver.maximize_window() #For maximizing window
driver.implicitly_wait(2) #gives an implicit wait for 2 seconds
game_pin = driver.find_element_by_xpath("//*[#id='inputSession']")
game_pin.send_keys(Game_Pin)
Enter_Press(driver)
driver.implicitly_wait(2)
Name = driver.find_element_by_xpath("//*[#id='username']")
Name.send_keys(NickNAME)
OK_GO(driver)
Kahoot_Spammer(Game_Pin, NickNAME, driver)
This is the code. Its supposed to open a chrome browser and navigate to the Kahoot.it website. Then take what information you gave it and put it in for you. It works for the first part of entering a game but once it gets to create your nickname it cannot detect the OK, go! button.
driver.find_element_by_xpath("//*[contains(text(), 'OK, go!')]").click()
I've inspected the button but cannot seem to find what to put within the code above. Any ideas?
Here is the source code.
<!doctype html>
<!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7" lang="en"> <![endif]-->
<!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8" lang="en"> <![endif]-->
<!--[if IE 8]> <html class="no-js lt-ie9" lang="en"> <![endif]-->
<!--[if IE 9]> <html class="no-js lt-ie10" lang="en"> <![endif]-->
<!--[if gt IE 9]><!--> <html class="no-js" lang="en"> <!--<![endif]-->
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
<title>Kahoot!</title>
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1.0, minimum-scale=1.0"/>
<meta name="viewport" content="initial-scale=1, maximum-scale=1.0, minimum-scale=1.0" media="(device-height: 568px)"/>
<meta name="apple-mobile-web-app-capable" content="yes" />
<meta name="apple-itunes-app" content="app-id=1131203560">
<meta name="description" content="Join a game of kahoot here. Kahoot! is a free game-based learning platform that makes it fun to learn – any subject, in any language, on any device, for all ages!">
<meta name="keywords" content="education, platform, smart phone, tablet, mobile, social, inclusive, HTML5, classroom, engagement, play, game, fun, quiz, multi-player, pedagogy, learning model, learn, gamification." />
<link rel="shortcut icon" href="/shared/theme/kahoot/img/icons/favicon.ico">
<link rel="apple-touch-icon-precomposed" sizes="144x144" href="/shared/theme/kahoot/img/icons/touch_icon_144.png">
<link rel="apple-touch-icon-precomposed" sizes="114x114" href="/shared/theme/kahoot/img/icons/touch_icon_114.png">
<link rel="apple-touch-icon-precomposed" sizes="72x72" href="/shared/theme/kahoot/img/icons/touch_icon_72.png">
<link rel="apple-touch-icon-precomposed" href="/shared/theme/kahoot/img/icons/touch_icon_57.png">
<link rel="stylesheet" type="text/css" href="/shared/css/cloak.css">
<div style="height: 0; width: 0; position: absolute; visibility: hidden">
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><defs><filter x="-2.2%" y="-2.3%" width="104.4%" height="104.8%" filterUnits="objectBoundingBox" id="a"><feOffset dy="1" in="SourceAlpha" result="shadowOffsetOuter1"/><symbol id="logo-shapes" viewBox="0 0 24 24"><ellipse cx="5.506" cy="18.966" rx="4.953" ry="4.953"/><path d="M12.005 5.902L17.873.033l5.869 5.869-5.869 5.868zm1.443 8.899h8.849v8.849h-8.849zm-2.584-4.977H.146l5.36-8.555z"/></symbol></svg>
</div>
<script src="https://tap-nexus.appspot.com/js/sdk/kahunaAPI_min.js"></script>
<script type="text/javascript">
(function(e,t){var n=e.amplitude||{_q:[],_iq:{}};var r=t.createElement("script");r.type="text/javascript";
r.async=true;r.src="https://d24n15hnbwhuhn.cloudfront.net/libs/amplitude-3.4.0-min.gz.js";
r.onload=function(){e.amplitude.runQueuedFunctions()};var i=t.getElementsByTagName("script")[0];
i.parentNode.insertBefore(r,i);function s(e,t){e.prototype[t]=function(){this._q.push([t].concat(Array.prototype.slice.call(arguments,0)));
return this}}var o=function(){this._q=[];return this};var a=["add","append","clearAll","prepend","set","setOnce","unset"];
for(var u=0;u<a.length;u++){s(o,a[u])}n.Identify=o;var c=function(){this._q=[];return this;
};var p=["setProductId","setQuantity","setPrice","setRevenueType","setEventProperties"];
for(var l=0;l<p.length;l++){s(c,p[l])}n.Revenue=c;var d=["init","logEvent","logRevenue","setUserId","setUserProperties","setOptOut","setVersionName","setDomain","setDeviceId","setGlobalUserProperties","identify","clearUserProperties","setGroup","logRevenueV2","regenerateDeviceId","logEventWithTimestamp","logEventWithGroups"];
function v(e){function t(t){e[t]=function(){e._q.push([t].concat(Array.prototype.slice.call(arguments,0)));
}}for(var n=0;n<d.length;n++){t(d[n])}}v(n);n.getInstance=function(e){e=(!e||e.length===0?"$default_instance":e).toLowerCase();
if(!n._iq.hasOwnProperty(e)){n._iq[e]={_q:[]};v(n._iq[e])}return n._iq[e]};e.amplitude=n;
})(window,document);
</script>
<base href="/">
<script type="text/javascript">
document.write('<scri'+'pt ');
document.write('type="text/javascript" ');
document.write('src="'+'/shared/theme/config.js');
document.write("?"+new Date().getTime()+'">');
document.write('</scri'+'pt>');
</script>
</head>
<body snitch ios7-viewport-fix>
<noscript>
<h1>Kahoot! needs JavaScript to work</h1>
<p>
To use Kahoot!, you need to have JavaScript enabled in your browser. To enable JavaScript, please do the following:
</p>
<ul>
<li>Follow these instructions.</li>
<li>Make sure you have the latest browser.</li>
<li>Turn off or disable the NoScript extension, if you have it.</li>
<li>Contact your IT administrator to allow access to Kahoot! in your security preferences.</li>
</ul>
<p>If you continue to have problems, please let us know by contacting Kahoot! support.</p>
</noscript>
<div id="debug-info" debug-info="dev,test" debug-timestamp></div>
<dev-mode></dev-mode>
<div class="loader" loader></div>
<iframe
id="gameBlockIframe"
style="display:none;"
class="game-block-iframe"
sandbox="allow-scripts allow-same-origin"
scrolling="no">
</iframe>
<div id="mainView" ng-cloak ng-view>
<h1>Join in a Kahoot! here</h1>
<p>To learn more about Kahoot! visit kahoot.com</p>
</div>
<div ng-cloak alerts></div>
<script type="text/javascript" src="/js/bootstrap.js"></script>
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
function gup( name, url ) {
if (!url) url = location.href;
name = name.replace(/[\[]/,"\\\[").replace(/[\]]/,"\\\]");
var regexS = "[\\?&]"+name+"=([^&#]*)";
var regex = new RegExp( regexS );
var results = regex.exec( url );
return results == null ? null : results[1];
}
var clientId = gup('gaId', window.location.search);
if (clientId) {
ga('create', 'UA-35308575-1', 'auto', {'allowLinker': true, 'clientId':gup('gaId', window.location.search)});
ga('create', 'UA-35308575-4', 'auto', {'name': 'legacy', 'clientId':gup('gaId', window.location.search)});
var platform = gup('platform', window.location.search);
if (typeof platform === 'string' && platform == 'iOS') {
window.ga('set', 'appName', 'Kahoot');
window.ga('set', 'appId', 'no.mobitroll.kahoot.controller');
}
if (typeof platform === 'string' && platform == 'Android') {
window.ga('set', 'appName', 'Kahoot');
window.ga('set', 'appId', 'no.mobitroll.kahoot.android');
}
} else {
ga('create', 'UA-35308575-1', 'auto', {'allowLinker': true});
ga('create', 'UA-35308575-4', 'auto', {'name': 'legacy'});
}
ga('send', 'pageview');
ga('legacy.send', 'pageview');
</script>
</body>
</html>
implicitly_wait needs to only be declared once when driver is initialized. To explicitly wait a specified number of seconds, you may use time.sleep(), although the more practical solution is just dynamically wait for the element to be present/clickable using Selenium's WebDriverWait.

Web Scraping - sessionStorage distil_referrer

I am trying to find a good way to get into a website, but I have found a big issue.
What is returning is the header of the html file, where there are those lines of code:
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml"><head>
<meta name="ROBOTS" content="NOINDEX, NOFOLLOW" />
<meta http-equiv="cache-control" content="max-age=0" />
<meta http-equiv="cache-control" content="no-cache" />
<meta http-equiv="expires" content="0" />
<meta http-equiv="expires" content="Tue, 01 Jan 1980 1:00:00 GMT" />
<meta http-equiv="pragma" content="no-cache" />
<meta http-equiv="refresh" content="10; url=/distil_r_blocked.html? Ref=/de/company/TI&distil_RID=F180AC0A-ECA2-11E6-80AC- E678F530D985&distil_TID=20170206193206" />
<script type="text/javascript">
(function(window){
try {
if (typeof sessionStorage !== 'undefined'){
sessionStorage.setItem('distil_referrer', document.referrer);
}
} catch (e){}
})(window);
</script>
<script type="text/javascript" src="/ga625684.js" defer="defer"></script><style type="text/css">#d__fFH{position:absolute;top:-5000px;left:-5000px}#d__fF{font-family:serif;font-size:200px;visibility:hidden}#svevtdraftsrybvfradvq{display:none!important}</style></head>
<body>
<div id="distil_ident_block"> </div>
<div id="d__fFH" style="position: absolute; top: -5000px; left: -5000px;"><object id="d_dlg" classid="clsid:3050f819-98b5-11cf-bb82-00aa00bdce0b" width="0px" height="0px"></object><span id="d__fF" style="font-family: ZWAdobeF,serif; font-size: 72px; visibility: hidden;">mmmmmmmmlli</span></div></body></html>
</script>
Is there a workaround to actually get the webpage?
For now i am trying to use Selenium without results
The code is the following:
from selenium import webdriver
fp = webdriver.FirefoxProfile()
driver = webdriver.Firefox(firefox_profile=fp)
driver.get("https://www.moneyhouse.ch/de/company/TI")
print driver.page_source

Browserless web scraping of Vaadin based website in Python

I'm new with web scraping and I encountered a problem.
I tried to extract the list of the states from this site, 'https://www.iso.org/obp/ui/#iso:code:3166:JP', by using Python, selenium and PhantomJS but I failed with the output as below.
<!DOCTYPE html><html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=11;chrome=1">
<style type="text/css">html, body {height:100%;margin:0;}</style>
<link rel="shortcut icon" type="image/vnd.microsoft.icon" href="./../VAADIN/themes/obp/favicon.ico">
<link rel="icon" type="image/vnd.microsoft.icon" href="./../VAADIN/themes/obp/favicon.ico">
<link rel="stylesheet" type="text/css" href="./../VAADIN/themes/obp/styles.css"><script type="text/javascript" src="./../VAADIN/widgetsets/org.iso.obp.ui.widgetset.applicationWidgetset/org.iso.obp.ui.widgetset.applicationWidgetset.nocache.js?1444641834593"></script><script src="https://www.iso.org/obp/VAADIN/widgetsets/org.iso.obp.ui.widgetset.applicationWidgetset/913365F3A38F531CF0D09D8744F3A155.cache.js"></script></head>
<body scroll="auto" class=" v-generated-body">
<div id="obpui-105541713" class=" v-app obp">
<div class=" v-app-loading"></div>
<noscript>
You have to enable javascript in your browser to use an application built with Vaadin.
</noscript>
</div>
<script type="text/javascript" src="./../VAADIN/vaadinBootstrap.js"></script>
<script type="text/javascript">//<![CDATA[
if (!window.vaadin) alert("Failed to load the bootstrap javascript: ./../VAADIN/vaadinBootstrap.js");
vaadin.initApplication("obpui-105541713",{"heartbeatInterval":300,"versionInfo":{"vaadinVersion":"7.3.10"},"vaadinDir":"./../VAADIN/","authErrMsg":{"message":"Take note of any unsaved data, and <u>click here<\/u> or press ESC to continue.","caption":"Authentication problem"},"widgetset":"org.iso.obp.ui.widgetset.applicationWidgetset","theme":"obp","comErrMsg":{"message":"Take note of any unsaved data, and <u>click here<\/u> or press ESC to continue.","caption":"Communication problem"},"serviceUrl":".","standalone":true,"sessExpMsg":{"message":"Take note of any unsaved data, and <u>click here<\/u> or press ESC key to continue.","caption":"Session Expired"}});
//]]></script>
</body></html>
My code in Python is here.
from selenium import webdriver
target_url = 'https://www.iso.org/obp/ui/#iso:code:3166:JP'
driver = webdriver.PhantomJS()
driver.get( target_url)
print driver.page_source
Is there any solution for this?

Categories

Resources