Python Requests Get not pulling webpage - python
I have a list of UPC code and I am trying to write a script to pull information about them form https://www.barcodelookup.com but the request is returning only the html tags but none of the information I want.
Here is a sample of my code:
import requests
from bs4 import BeautifulSoup
page = requests.get('https://www.barcodelookup.com/075610166101')
soup = BeautifulSoup(page.text, 'html.parser')
bsoup = soup.prettify()
with open('output1.html', 'w') as file:
file.write(str(bsoup))
with open('output.html', 'w')as file:
file.write(str(page.text))
sample outpout.html1:
<!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]> <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]> <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!-->
<html class="no-js" lang="en-US">
<!--<![endif]-->
<head>
<title>
Attention Required! | Cloudflare
</title>
<meta id="captcha-bypass" name="captcha-bypass"/>
<meta charset="utf-8"/>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type"/>
<meta content="IE=Edge,chrome=1" http-equiv="X-UA-Compatible"/>
<meta content="noindex, nofollow" name="robots"/>
<meta content="width=device-width,initial-scale=1" name="viewport"/>
<link href="/cdn-cgi/styles/cf.errors.css" id="cf_styles-css" media="screen,projection" rel="stylesheet" type="text/css"/>
<!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" type="text/css" media="screen,projection" /><![endif]-->
<style type="text/css">
body{margin:0;padding:0}
</style>
<!--[if gte IE 10]><!-->
<script src="/cdn-cgi/scripts/zepto.min.js" type="text/javascript">
</script>
<!--<![endif]-->
<!--[if gte IE 10]><!-->
<script src="/cdn-cgi/scripts/cf.common.js" type="text/javascript">
</script>
<!--<![endif]-->
<style type="text/css">
#cf-wrapper #spinner {width:69px; margin: auto;}
#cf-wrapper #cf-please-wait{text-align:center}
.attribution {margin-top: 32px;}
.bubbles { background-color: #f58220; width:20px; height: 20px; margin:2px; border-radius:100%; display:inline-block; }
#cf-wrapper #challenge-form { padding-top:25px; padding-bottom:25px; }
#cf-hcaptcha-container { text-align:center;}
</style>
</head>
<body>
<div id="cf-wrapper">
<div class="cf-alert cf-alert-error cf-cookie-error" data-translate="enable_cookies" id="cookie-alert">
Please enable cookies.
</div>
<div class="cf-error-details-wrapper" id="cf-error-details">
<div class="cf-wrapper cf-header cf-error-overview">
<h1 data-translate="challenge_headline">
One more step
</h1>
<h2 class="cf-subheadline">
<span data-translate="complete_sec_check">
Please complete the security check to access
</span>
www.barcodelookup.com
</h2>
</div>
sample outpou1.html:
<div class="cf-section cf-highlight cf-captcha-container">
<div class="cf-wrapper">
<div class="cf-columns two">
<div class="cf-column">
<div class="cf-highlight-inverse cf-form-stacked">
<form action="/075610166101?__cf_chl_captcha_tk__=10080e641441171d59b24657ed37a7381be4a368-1595778921-0-AS91JaY_1ozqjwuL0cLJj39tDQ8tO-5t6vMnZ4LFD6V9L_k_jFw1qb6NW_KOPGyf53pazgUHKpjsBF0oCu3pWy-n1rks1eGTzPNdPJvDUgly5EfmCU2hfkPgF0u9Mmb0jAt0uNra1wy-xDgG87ZgWd3KvYSj1Jre0DtwvkXITbLAaAdSg5UeBhw4DDEuCxFILAwhLTU3YHEm9F1CbC7cqA-U05kTDiOIBnZngHGBrnOWB9LYl6asezmwfpuzNZTovixMVE8BBKVfIf1gJjllYh7626I1abfYw38uuoIy0viPuN_CtjB8JoBbs2qrix4gXW6PGu9EA5ZPhBw-IQ8csPLN-a0WFRqB3Il-Hz6M6z9Wdb-OHUKOjX37n_fBuQarqU34cgbG4CNpD_7cdn_NUrlJ6xsRZiFV13V2q4zBS4XpPwabA_unBIjziYgIiB-y9hwndtV08bMXxtoSqtNxxev3fNnL_cQ" class="challenge-form" enctype="application/x-www-form-urlencoded" id="challenge-form" method="POST">
<input name="r" type="hidden" value="33260f1c9e17bb57e0d89a1d21e050da58f9c0a0-1595778921-0-Ad2sk2X3qN2WwWLekQkZpeJCOg0H0bI9CHDtAranzrOjQHfchnqyW9dHD3S6CpbKRRrV/9pFNY+jLG7XUks78zi0PsNBHSNwDV4ad2liittfYU5X73GgFmyN3COYAQomUPoPxw+YPyMTRPrR0P6qFUh92fhmLMbivztY8iwFFTppCHO1Kx8Ax+4orJWgb31sJpRrtuasqpgFs9qCAhBgBKzue/BginjozYpNbGDlrdjnWnh+b+SxL+HWxzkFLwxoIWDJ6dMHaZSp/zvBptO5cgBTpPupAYNvcB2O3YGapY0UefpxmhXntG50yXyrQmobqrh4rjuyXgDup3HO8ETKUwnZ37f4NN0LuYA2k9nveVh0j9hqy/P09wbQE8AChLs2/u2uqpTcGyPSpbTOyNo1FjfD+BpE6KqQsL8l9hOtHuHviayTngoqOrOMW6"/>
<input name="cf_captcha_kind" type="hidden" value="h"/>
<input name="vc" type="hidden" value=""/>
<script async="" data-ray="5b8f4df52fe3741d"
I am trying to post both output file to show the returned information but the system won't let me.
Websites usually put some security mechanisms in order to avoid getting scraped. The most basic check is serving content based on a user-agent so if a requesting client is not sharing any user-agent information it will be considered as an unsupported browser or some bot/script. So, just adding a user-agent header parameter (mimicking Google Chrome) is allowing us to get content from this site.
Here is your updated script:
import requests
from bs4 import BeautifulSoup
headers = {
'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36'
}
page = requests.get('https://www.barcodelookup.com/075610166101', headers=headers)
soup = BeautifulSoup(page.text, 'html.parser')
bsoup = soup.prettify()
with open('output1.html', 'w') as file:
file.write(str(bsoup))
with open('output.html', 'w')as file:
file.write(str(page.text))
Related
can't able to login whalewisdom website using BeautifulSoup
I'm trying to login whalewisdom website for last two week but I'm not able to log in, I was tried many libraries like scrapy, selenium, beautifulsoup, etc...
from requests import Session
from bs4 import BeautifulSoup as bs
with Session() as s:
login_url = s.get("https://whalewisdom.com/login")
bs_content = bs(login_url.content, "lxml")
authenticity_token = bs_content.find("input", {"name":"authenticity_token"})["value"]
login_data = {
"authenticity_token": authenticity_token,
"login": "info#example.com",
"password": "***********",
"commit": "Log+In",
}
s.post("https://whalewisdom.com/session", data=login_data)
html_data = bs(s.get("https://whalewisdom.com/dashboard").content, "html.parser")
print(html_data)
enter image description here
Here the outputenter image description here:
<!DOCTYPE html>
<html lang="en">
<head>[enter image description here][1]
<meta charset="utf-8"/>
<title>WhaleWisdom Dashboard</title>
<meta content="IE=edge" http-equiv="X-UA-Compatible"/>
<meta content="width=device-width,initial-scale=1.0" name="viewport"/>
<meta content="WhaleWisdom tracks 13F, Schedule 13D, and 13G EDGAR filings by hedge funds. Hedge Fund Whale Backtesting and search tools" name="description"/>
<link href="https://d27mjrcvcy56qq.cloudfront.net/images/apple-touch-icon-76x76.png" rel="apple-touch-icon" sizes="76x76"/>
<link href="https://d27mjrcvcy56qq.cloudfront.net/images/favicon-32x32.png" rel="icon" sizes="32x32" type="image/png"/>
<link href="https://d27mjrcvcy56qq.cloudfront.net/images/favicon-96x96.png" rel="icon" sizes="96x96" type="image/png"/>
<link href="https://d27mjrcvcy56qq.cloudfront.net/images/favicon-16x16.png" rel="icon" sizes="16x16" type="image/png"/>
<meta content="r4hQnHlN2H-GtcIb06YHl49VSipApmfQQWIOvZzfnAU" name="google-site-verification">
<link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400,500,700|Material+Icons" rel="stylesheet" type="text/css"/>
<link href="https://cdn.jsdelivr.net/npm/font-awesome#4.7.0/css/font-awesome.min.css" rel="stylesheet"/>
<link href="https://d27mjrcvcy56qq.cloudfront.net/packs/css/whalewisdom-24fbc382.css" media="screen" rel="stylesheet">
<meta content="authenticity_token" name="csrf-param">
<meta content="XMAu/LK+dKi/zt/XSTvxIJ8jKl2x8Rx47/ZnAiN6MQCcZmSSlUrOLMeURRr54eCfEWHY8oyS8c6GYxLoIMomNQ==" name="csrf-token">
</meta></meta></link></meta></head>
<body>
<noscript>
<strong>We're sorry but the WhaleWisdom Dashboard doesn't work properly without JavaScript enabled. Please enable it to continue.</strong>
</noscript>
<div id="app"></div>
<script src="https://d27mjrcvcy56qq.cloudfront.net/packs/js/whalewisdom-4b32da19479fdebf5332.js"></script>
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','https://www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-11651599-1', 'auto');
ga('send', 'pageview');
</script>
<script async="" charset="utf-8" src="//ads.investingchannel.com/adtags/WhaleWisdom/quotepages/970x91.js" type="text/javascript"></script>
</body>
</html>
You can see that in the HTML output, at line 23, there is an error type message stating that the WhaleWisdom dashboard doesn't work properly without JavaScript.
<!DOCTYPE html>
<html lang="en">
<head>[enter image description here][1]
<meta charset="utf-8"/>
<title>WhaleWisdom Dashboard</title>
<meta content="IE=edge" http-equiv="X-UA-Compatible"/>
<meta content="width=device-width,initial-scale=1.0" name="viewport"/>
<meta content="WhaleWisdom tracks 13F, Schedule 13D, and 13G EDGAR filings by hedge funds. Hedge Fund Whale Backtesting and search tools" name="description"/>
<link href="https://d27mjrcvcy56qq.cloudfront.net/images/apple-touch-icon-76x76.png" rel="apple-touch-icon" sizes="76x76"/>
<link href="https://d27mjrcvcy56qq.cloudfront.net/images/favicon-32x32.png" rel="icon" sizes="32x32" type="image/png"/>
<link href="https://d27mjrcvcy56qq.cloudfront.net/images/favicon-96x96.png" rel="icon" sizes="96x96" type="image/png"/>
<link href="https://d27mjrcvcy56qq.cloudfront.net/images/favicon-16x16.png" rel="icon" sizes="16x16" type="image/png"/>
<meta content="r4hQnHlN2H-GtcIb06YHl49VSipApmfQQWIOvZzfnAU" name="google-site-verification">
<link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400,500,700|Material+Icons" rel="stylesheet" type="text/css"/>
<link href="https://cdn.jsdelivr.net/npm/font-awesome#4.7.0/css/font-awesome.min.css" rel="stylesheet"/>
<link href="https://d27mjrcvcy56qq.cloudfront.net/packs/css/whalewisdom-24fbc382.css" media="screen" rel="stylesheet">
<meta content="authenticity_token" name="csrf-param">
<meta content="XMAu/LK+dKi/zt/XSTvxIJ8jKl2x8Rx47/ZnAiN6MQCcZmSSlUrOLMeURRr54eCfEWHY8oyS8c6GYxLoIMomNQ==" name="csrf-token">
</meta></meta></link></meta></head>
<body>
----
<noscript>
<strong>We're sorry but the WhaleWisdom Dashboard doesn't work properly without JavaScript enabled. Please enable it to continue.</strong>**
</noscript>
----
<div id="app"></div>
<script src="https://d27mjrcvcy56qq.cloudfront.net/packs/js/whalewisdom-4b32da19479fdebf5332.js"></script>
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','https://www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-11651599-1', 'auto');
ga('send', 'pageview');
</script>
<script async="" charset="utf-8" src="//ads.investingchannel.com/adtags/WhaleWisdom/quotepages/970x91.js" type="text/javascript"></script>
</body>
</html>
I think because of this it is not working. I also can't test it right now because I don't use WhaleWisdom.
Web scrape an embedded table in google tag manager's datalayer?
I am trying to web scrape table from OECD Data. I am learning to use Python and BeautifulSoup in Jupyter to achieve this.
The closest similar post that I've found is How to web scrape tables embedded in websites using Python. In my case, the scraped object has a nested JSON object that doesn't match with what I inspect from the page.
Scraped object
<!DOCTYPE html>
<html lang="en"><head>
<script>
dataLayer = [{
'siteName':'data',
'siteEnvironment':'live',
'pageLanguage':'en',
'pageTopic':'',
'pageSubTopic':'',
'pageCategory':'oecdChart',
'indicatorCode':'FBORNPOP'
}];
</script>
<title>Foreign-born population, Total, % of population, 2013 – 2018</title>
<meta content="IE=edge" http-equiv="X-UA-Compatible"/>
<meta content="noindex" name="robots"/>
<link href="//data.oecd.org/media/dataportal/stylesheets/main.min.css" rel="stylesheet" type="text/css"/>
<link href="//data.oecd.org/media/oecdorg/styleassets/css/responsive/bootstrap.min.css" rel="stylesheet" type="text/css"/>
<meta content="summary_large_image" name="twitter:card"/>
<meta content="#OECD" name="twitter:site"/>
<meta content="#OECD" name="twitter:creator"/>
<meta content="Interactive charts by the OECD" name="twitter:title"/>
<meta content="Foreign-born population, Total, % of population, 2013 – 2018" name="twitter:description"/>
<meta content="https://data.oecd.org/custom/share/txqy.png" name="twitter:image:src"/>
<meta content="website" property="og:type"/>
<meta content="OECD Data" property="og:site_name"/>
<meta content="https://data.oecd.org/chart/5Rmq" property="og:url"/>
<meta content="Interactive charts by the OECD" property="og:title"/>
<meta content="Foreign-born population, Total, % of population, 2013 – 2018" property="og:description"/>
<meta content="https://data.oecd.org/custom/share/txqy.png" property="og:image"/>
<meta content="en_GB" property="og:locale"/>
<meta content="fr_FR" property="og:locale:alternate"/>
<meta content="embedded" itemprop="context"/>
</head>
<body>
<!-- New Google Tag Manager -->
<noscript><iframe height="0" src="https://www.googletagmanager.com/ns.html?id=GTM-P5JSM4P" style="display:none;visibility:hidden" width="0"></iframe></noscript>
<script>
(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':
new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],
j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src=
'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);
})(window, document, 'script', 'dataLayer', 'GTM-P5JSM4P');
</script>
<!-- End New Google Tag Manager -->
<!-- Google Tag Manager -->
<noscript>
<iframe height="0" src="//www.googletagmanager.com/ns.html?id=GTM-WZ3XJR" style="display: none; visibility: hidden" width="0"></iframe>
</noscript>
<script>
(function(w, d, s, l, i) {
w[l] = w[l] || [];
w[l].push({
'gtm.start':
new Date().getTime(),
event: 'gtm.js'
});
var f = d.getElementsByTagName(s)[0],
j = d.createElement(s),
dl = l != 'dataLayer' ? '&l=' + l : '';
j.async = true;
j.src =
'//www.googletagmanager.com/gtm.js?id=' + i + dl;
f.parentNode.insertBefore(j, f);
})(window, document, 'script', 'dataLayer', 'GTM-WZ3XJR');
</script>
<!-- End Google Tag Manager -->
<div class="ddp-root">
<div class="embedded-chart">
<div class="ddp-chart" data-baseline-comparison="" data-compact-header="false" data-data='{"header":{"id":"6bda3584-cfa5-46cb-afd3-d2512d0d52f8","test":false,"prepared":"2020-03-01T09:16:26.3887278Z","sender":{"id":"OECD","name":"Organisation for Economic Co-operation and Development"},"links":[{"href":"https://stats.oecd.org:443/sdmx-json/data/DP_LIVE/.FBORNPOP.TOT.PC_POP.A/OECD?json-lang=en&dimensionAtObservation=allDimensions&startPeriod=2013&endPeriod=2018","rel":"request"}]},"dataSets":[{"action":"Information","observations":{"0:0:0:0:0:0":[27.54,0,null],"0:0:0:0:0:1":[27.814,0,null],"0:0:0:0:0:2":[28.077,0,null],"0:0:0:0:0:3":[28.65,0,null],"0:0:0:0:0:4":[29.2,0,null],"0:0:0:0:0:5":[29.641,0,null],"1:0:0:0:0:0":[15.676,0,null],"1:0:0:0:0:1":[15.816,0,null],"1:0:0:0:0:2":[15.807,0,null],"1:0:0:0:0:3":[16.281,0,null],"1:0:0:0:0:4":[16.566,0,null],"1:0:0:0:0:5":[16.809,0,null],"2:0:0:0:0:0":[19.952,0,null],"2:0:0:0:0:1":[20.108,0,null],"2:0:0:0:0:2":[20.275,0,null],"2:0:0:0:0:3":[20.779,0,null],"3:0:0:0:0:0":[8.115,0,null],"3:0:0:0:0:1":[8.43,0,null],"3:0:0:0:0:2":[8.838,0,null],"3:0:0:0:0:3":[9.463,0,null],"3:0:0:0:0:4":[9.952,0,null],"3:0:0:0:0:5":[10.282,0,null],"4:0:0:0:0:0":[5.235,0,null],"4:0:0:0:0:1":[5.553,0,null],"4:0:0:0:0:2":[5.85,0,null],"4:0:0:0:0:3":[6.127,0,null],"4:0:0:0:0:4":[6.473,0,null],"4:0:0:0:0:5":[6.726,0,null],"5:0:0:0:0:0":[12.539,0,null],"5:0:0:0:0:1":[12.976,0,null],"5:0:0:0:0:2":[13.45,0,null],"5:0:0:0:0:3":[13.982,0,null],"5:0:0:0:0:4":[15.513,0,null],"5:0:0:0:0:5":[16.006,0,null],"6:0:0:0:0:0":[4.274,0,null],"6:0:0:0:0:1":[4.527,0,null],"6:0:0:0:0:2":[4.831,0,null],"6:0:0:0:0:3":[5.171,0,null],"6:0:0:0:0:4":[5.289,0,null],"6:0:0:0:0:5":[5.534,0,null],"7:0:0:0:0:0":[41.495,0,null],"7:0:0:0:0:1":[42.707,0,null],"7:0:0:0:0:2":[43.887,0,null],"7:0:0:0:0:3":[45.258,0,null],"7:0:0:0:0:4":[46.404,0,null],"7:0:0:0:0:5":[47.563,0,null],"8:0:0:0:0:0":[0.787,0,null],"8:0:0:0:0:1":[0.79,0,null],"8:0:0:0:0:2":[0.74,0,null],"8:0:0:0:0:3":[0.79,0,null],"9:0:0:0:0:0":[11.468,0,null],"9:0:0:0:0:1":[11.581,0,null],"9:0:0:0:0:2":[11.795,0,null],"9:0:0:0:0:3":[12.106,0,null],"9:0:0:0:0:4":[12.545,0,null],"9:0:0:0:0:5":[12.97,0,null],"10:0:0:0:0:0":[13.059,0,null],"10:0:0:0:0:1":[13.685,0,null],"10:0:0:0:0:2":[14.236,0,null],"10:0:0:0:0:3":[14.701,0,null],"10:0:0:0:0:4":[15.075,0,null],"10:0:0:0:0:5":[15.362,0,null],"11:0:0:0:0:0":[8.628,0,null],"11:0:0:0:0:1":[8.455,0,null],"11:0:0:0:0:2":[8.555,0,null],"11:0:0:0:0:3":[8.613,0,null],"12:0:0:0:0:0":[15.308,0,null],"12:0:0:0:0:1":[15.804,0,null],"12:0:0:0:0:2":[16.397,0,null],"12:0:0:0:0:3":[17.039,0,null],"12:0:0:0:0:4":[18.006,0,null],"12:0:0:0:0:5":[18.803,0,null],"13:0:0:0:0:0":[27.325,0,null],"13:0:0:0:0:1":[27.883,0,null],"13:0:0:0:0:2":[28.376,0,null],"13:0:0:0:0:3":[28.761,0,null],"13:0:0:0:0:4":[29.259,0,null],"13:0:0:0:0:5":[29.483,0,null],"14:0:0:0:0:0":[12.29,0,null],"14:0:0:0:0:1":[12.535,0,null],"14:0:0:0:0:2":[13.107,0,null],"14:0:0:0:0:3":[13.662,0,null],"14:0:0:0:0:4":[14.157,0,null],"14:0:0:0:0:5":[13.794,0,null],"15:0:0:0:0:0":[12.846,0,null],"15:0:0:0:0:1":[12.942,0,null],"15:0:0:0:0:2":[13.174,0,null],"15:0:0:0:0:3":[13.436,0,null],"15:0:0:0:0:4":[13.481,0,null],"15:0:0:0:0:5":[13.626,0,null],"16:0:0:0:0:0":[2.364,0,null],"16:0:0:0:0:1":[2.486,0,null],"16:0:0:0:0:2":[2.593,0,null],"17:0:0:0:0:0":[6.603,0,null],"17:0:0:0:0:1":[0.0,0,null],"17:0:0:0:0:2":[6.641,0,null],"17:0:0:0:0:4":[5.811,0,null],"18:0:0:0:0:0":[16.039,0,null],"18:0:0:0:0:1":[16.132,0,null],"18:0:0:0:0:2":[16.376,0,null],"18:0:0:0:0:3":[17.148,0,null],"19:0:0:0:0:0":[22.435,0,null],"19:0:0:0:0:1":[23.36,0,null],"19:0:0:0:0:2":[24.478,0,null],"19:0:0:0:0:3":[25.077,0,null],"20:0:0:0:0:0":[15.031,0,null],"20:0:0:0:0:1":[14.934,0,null],"20:0:0:0:0:2":[14.831,0,null],"20:0:0:0:0:3":[14.767,0,null],"20:0:0:0:0:4":[14.701,0,null],"20:0:0:0:0:5":[15.014,0,null],"21:0:0:0:0:0":[11.89,0,null],"21:0:0:0:0:1":[12.032,0,null],"21:0:0:0:0:2":[12.186,0,null],"21:0:0:0:0:3":[12.276,0,null],"21:0:0:0:0:4":[12.476,0,null],"21:0:0:0:0:5":[12.486,0,null],"22:0:0:0:0:0":[9.529,0,null],"22:0:0:0:0:1":[9.596,0,null],"22:0:0:0:0:2":[9.708,0,null],"22:0:0:0:0:3":[9.94,0,null],"22:0:0:0:0:4":[10.199,0,null],"22:0:0:0:0:5":[10.415,0,null],"23:0:0:0:0:0":[14.515,0,null],"23:0:0:0:0:1":[16.023,0,null],"23:0:0:0:0:2":[16.504,0,null],"23:0:0:0:0:3":[16.378,0,null],"23:0:0:0:0:4":[16.844,0,null],"23:0:0:0:0:5":[17.593,0,null],"24:0:0:0:0:0":[16.081,0,null],"24:0:0:0:0:1":[16.61,0,null],"24:0:0:0:0:2":[17.375,0,null],"24:0:0:0:0:3":[18.305,0,null],"24:0:0:0:0:4":[18.96,0,null],"24:0:0:0:0:5":[19.392,0,null],"25:0:0:0:0:0":[7.063,0,null],"25:0:0:0:0:1":[7.161,0,null],"25:0:0:0:0:2":[7.3,0,null],"25:0:0:0:0:3":[7.523,0,null],"26:0:0:0:0:0":[13.292,0,null],"26:0:0:0:0:1":[12.88,0,null],"26:0:0:0:0:2":[12.773,0,null],"26:0:0:0:0:3":[12.769,0,null],"26:0:0:0:0:4":[12.997,0,null],"26:0:0:0:0:5":[13.365,0,null],"27:0:0:0:0:0":[0.0,0,null],"27:0:0:0:0:1":[1.883,0,null],"27:0:0:0:0:2":[2.024,0,null],"27:0:0:0:0:3":[2.235,0,null],"27:0:0:0:0:4":[2.383,0,null],"27:0:0:0:0:5":[2.781,0,null],"28:0:0:0:0:0":[0.0,0,null],"28:0:0:0:0:1":[0.0,0,null],"28:0:0:0:0:2":[0.0,0,null],"29:0:0:0:0:0":[10.891,0,null],"29:0:0:0:0:1":[11.374,0,null],"29:0:0:0:0:2":[11.906,0,null],"29:0:0:0:0:3":[12.639,0,null],"29:0:0:0:0:4":[13.884,0,null],"29:0:0:0:0:5":[16.162,0,null],"30:0:0:0:0:0":[3.186,0,null],"30:0:0:0:0:1":[3.225,0,null],"30:0:0:0:0:2":[3.273,0,null],"30:0:0:0:0:3":[3.336,0,null],"30:0:0:0:0:4":[3.418,0,null],"30:0:0:0:0:5":[3.492,0,null],"31:0:0:0:0:0":[23.472,0,null],"31:0:0:0:0:1":[22.936,0,null],"31:0:0:0:0:2":[22.532,0,null],"31:0:0:0:0:3":[22.187,0,null],"31:0:0:0:0:4":[21.78,0,null],"32:0:0:0:0:0":[13.879,0,null],"32:0:0:0:0:1":[13.629,0,null],"32:0:0:0:0:2":[13.47,0,null],"32:0:0:0:0:3":[13.138,0,null],"32:0:0:0:0:4":[12.898,0,null],"32:0:0:0:0:5":[12.749,0,null],"33:0:0:0:0:1":[4.807,0,null],"33:0:0:0:0:2":[9.251,0,null],"33:0:0:0:0:3":[4.463,0,null],"33:0:0:0:0:4":[4.404,0,null],"33:0:0:0:0:5":[4.554,0,null]}}],"structure":{"links":[{"href":"https://stats.oecd.org/sdmx-json/dataflow/DP_LIVE/OECD","rel":"dataflow"}],"name":"OECD Data Live dataset","description":"OECD Data Live dataset","dimensions":{"observation":[{"keyPosition":0,"id":"LOCATION","name":"Country","values":[{"id":"AUS","name":"Australia"},{"id":"BEL","name":"Belgium"},{"id":"CAN","name":"Canada"},{"id":"DNK","name":"Denmark"},{"id":"FIN","name":"Finland"},{"id":"DEU","name":"Germany"},{"id":"HUN","name":"Hungary"},{"id":"LUX","name":"Luxembourg"},{"id":"MEX","name":"Mexico"},{"id":"NLD","name":"Netherlands"},{"id":"NOR","name":"Norway"},{"id":"PRT","name":"Portugal"},{"id":"SWE","name":"Sweden"},{"id":"CHE","name":"Switzerland"},{"id":"GBR","name":"United Kingdom"},{"id":"USA","name":"United States"},{"id":"CHL","name":"Chile"},{"id":"GRC","name":"Greece"},{"id":"IRL","name":"Ireland"},{"id":"NZL","name":"New Zealand"},{"id":"EST","name":"Estonia"},{"id":"FRA","name":"France"},{"id":"ITA","name":"Italy"},{"id":"SVN","name":"Slovenia"},{"id":"AUT","name":"Austria"},{"id":"CZE","name":"Czech Republic"},{"id":"ESP","name":"Spain"},{"id":"TUR","name":"Turkey"},{"id":"POL","name":"Poland"},{"id":"ISL","name":"Iceland"},{"id":"SVK","name":"Slovak Republic"},{"id":"ISR","name":"Israel"},{"id":"LVA","name":"Latvia"},{"id":"LTU","name":"Lithuania"}],"role":"REF_AREA"},{"keyPosition":1,"id":"INDICATOR","name":"Indicator","values":[{"id":"FBORNPOP","name":"Foreign-born population"}]},{"keyPosition":2,"id":"SUBJECT","name":"Subject","values":[{"id":"TOT","name":"Total"}]},{"keyPosition":3,"id":"MEASURE","name":"Measure","values":[{"id":"PC_POP","name":"% of population"}]},{"keyPosition":4,"id":"FREQUENCY","name":"Frequency","values":[{"id":"A","name":"Annual"}],"role":"FREQ"},{"id":"TIME_PERIOD","name":"Time","values":[{"id":"2013","name":"2013"},{"id":"2014","name":"2014"},{"id":"2015","name":"2015"},{"id":"2016","name":"2016"},{"id":"2017","name":"2017"},{"id":"2018","name":"2018"}],"role":"TIME_PERIOD"}]},"attributes":{"dataSet":[],"series":[],"observation":[{"id":"TIME_FORMAT","name":"Time Format","values":[{"id":"P1Y","name":"Annual"}]},{"id":"OBS_STATUS","name":"Observation Status","values":[]}]},"annotations":[{"title":"Copyright OECD - All rights reserved","uri":"","text":""},{"title":"Terms and Conditions","uri":"http://www.oecd.org/termsandconditions/","text":""},{"title":"Privacy Policy","uri":"http://www.oecd.org/privacy/","text":""},{"title":"MyOECD","uri":"https://www.oecd.org/login","text":""},{"title":"Contact Us","uri":"http://www.oecd.org/contact/","text":""}]}}' data-editable="false" data-embedded="true" data-end-time="2018" data-external-footer-id="embedded-chart-footer" data-fixed-height="true" data-frequency="A" data-highlighted-locations="" data-indicator="FBORNPOP" data-language="en" data-locations="" data-measure="PC_POP" data-referrer="//data.oecd.org/migration/foreign-born-population.htm" data-show-baseline="False" data-show-non-baseline="True" data-source="OECD International Migration Statistics: International migration database" data-start-time="2013" data-stored-configuration="impose" data-subject="TOT" data-table-note="Information on data for Israel: http://oe.cd/israel-disclaimer" data-type="table" data-use-latest-data="False">
</div>
<footer class="embedded-chart-footer" id="embedded-chart-footer">
<div class="copyright-tooltip info-tooltip" tabindex="0">
<i class="icon-ddp-copyright"></i>
<p class="info-tooltip-content">
© OECD
<span class="separator">·</span>
Terms & Conditions
</p>
</div>
<p class="explore">Compare countries on <a class="chart-open-referrer" href="//data.oecd.org/migration/foreign-born-population.htm" target="_top">data.oecd.org</a></p>
<p class="logo"><a class="chart-open-referrer" href="//data.oecd.org/migration/foreign-born-population.htm" target="_top" title="OECD Data">OECD Data</a></p>
</footer>
</div>
</div>
<script src="//data.oecd.org/media/dataportal/scripts/ddp-chart-config.js"></script>
<script src="//data.oecd.org/media/dataportal/scripts/jquery.min.js"></script>
<script src="//data.oecd.org/media/dataportal/scripts/application.js"></script>
</body></html>
Am I scraping a cached webpage?
Edited
See below for the code I use and above for the response.
from bs4 import BeautifulSoup
from requests_html import HTMLSession
session = HTMLSession()
oecd_url = "https://data.oecd.org/chart/5Rmq"
response = session.get(oecd_url)
soup = BeautifulSoup(response.html.html,"html5lib")
soup
it has a googletaskmanager, it is used for analytics and ecomerce services, it deploys all items to your page as an api. thats why you couldnt find the elements that you choose <noscript> <iframe height="0" src="//www.googletagmanager.com/ns.html?id=GTM-WZ3XJR" style="display: none; visibility: hidden" width="0"></iframe> </noscript>
Python | Selenium cannot detect button
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
import os
Game_Pin = input('Enter your PIN: ')
NickNAME = input('Enter your nickname: ')
driver = webdriver.Chrome(executable_path=r"C:\WebDriver\bin\chromedriver.exe")
def Enter_Press(driver):
driver.find_element_by_xpath("//*[contains(text(), 'Enter')]").click()
def OK_GO(driver):
driver.find_element_by_xpath("//*[contains(text(), 'OK, go!')]").click()
def Kahoot_Spammer(Game_Pin, NickNAME, driver):
driver.get('https://kahoot.it/')
driver.maximize_window() #For maximizing window
driver.implicitly_wait(2) #gives an implicit wait for 2 seconds
game_pin = driver.find_element_by_xpath("//*[#id='inputSession']")
game_pin.send_keys(Game_Pin)
Enter_Press(driver)
driver.implicitly_wait(2)
Name = driver.find_element_by_xpath("//*[#id='username']")
Name.send_keys(NickNAME)
OK_GO(driver)
Kahoot_Spammer(Game_Pin, NickNAME, driver)
This is the code. Its supposed to open a chrome browser and navigate to the Kahoot.it website. Then take what information you gave it and put it in for you. It works for the first part of entering a game but once it gets to create your nickname it cannot detect the OK, go! button.
driver.find_element_by_xpath("//*[contains(text(), 'OK, go!')]").click()
I've inspected the button but cannot seem to find what to put within the code above. Any ideas?
Here is the source code.
<!doctype html>
<!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7" lang="en"> <![endif]-->
<!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8" lang="en"> <![endif]-->
<!--[if IE 8]> <html class="no-js lt-ie9" lang="en"> <![endif]-->
<!--[if IE 9]> <html class="no-js lt-ie10" lang="en"> <![endif]-->
<!--[if gt IE 9]><!--> <html class="no-js" lang="en"> <!--<![endif]-->
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
<title>Kahoot!</title>
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1.0, minimum-scale=1.0"/>
<meta name="viewport" content="initial-scale=1, maximum-scale=1.0, minimum-scale=1.0" media="(device-height: 568px)"/>
<meta name="apple-mobile-web-app-capable" content="yes" />
<meta name="apple-itunes-app" content="app-id=1131203560">
<meta name="description" content="Join a game of kahoot here. Kahoot! is a free game-based learning platform that makes it fun to learn – any subject, in any language, on any device, for all ages!">
<meta name="keywords" content="education, platform, smart phone, tablet, mobile, social, inclusive, HTML5, classroom, engagement, play, game, fun, quiz, multi-player, pedagogy, learning model, learn, gamification." />
<link rel="shortcut icon" href="/shared/theme/kahoot/img/icons/favicon.ico">
<link rel="apple-touch-icon-precomposed" sizes="144x144" href="/shared/theme/kahoot/img/icons/touch_icon_144.png">
<link rel="apple-touch-icon-precomposed" sizes="114x114" href="/shared/theme/kahoot/img/icons/touch_icon_114.png">
<link rel="apple-touch-icon-precomposed" sizes="72x72" href="/shared/theme/kahoot/img/icons/touch_icon_72.png">
<link rel="apple-touch-icon-precomposed" href="/shared/theme/kahoot/img/icons/touch_icon_57.png">
<link rel="stylesheet" type="text/css" href="/shared/css/cloak.css">
<div style="height: 0; width: 0; position: absolute; visibility: hidden">
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><defs><filter x="-2.2%" y="-2.3%" width="104.4%" height="104.8%" filterUnits="objectBoundingBox" id="a"><feOffset dy="1" in="SourceAlpha" result="shadowOffsetOuter1"/><symbol id="logo-shapes" viewBox="0 0 24 24"><ellipse cx="5.506" cy="18.966" rx="4.953" ry="4.953"/><path d="M12.005 5.902L17.873.033l5.869 5.869-5.869 5.868zm1.443 8.899h8.849v8.849h-8.849zm-2.584-4.977H.146l5.36-8.555z"/></symbol></svg>
</div>
<script src="https://tap-nexus.appspot.com/js/sdk/kahunaAPI_min.js"></script>
<script type="text/javascript">
(function(e,t){var n=e.amplitude||{_q:[],_iq:{}};var r=t.createElement("script");r.type="text/javascript";
r.async=true;r.src="https://d24n15hnbwhuhn.cloudfront.net/libs/amplitude-3.4.0-min.gz.js";
r.onload=function(){e.amplitude.runQueuedFunctions()};var i=t.getElementsByTagName("script")[0];
i.parentNode.insertBefore(r,i);function s(e,t){e.prototype[t]=function(){this._q.push([t].concat(Array.prototype.slice.call(arguments,0)));
return this}}var o=function(){this._q=[];return this};var a=["add","append","clearAll","prepend","set","setOnce","unset"];
for(var u=0;u<a.length;u++){s(o,a[u])}n.Identify=o;var c=function(){this._q=[];return this;
};var p=["setProductId","setQuantity","setPrice","setRevenueType","setEventProperties"];
for(var l=0;l<p.length;l++){s(c,p[l])}n.Revenue=c;var d=["init","logEvent","logRevenue","setUserId","setUserProperties","setOptOut","setVersionName","setDomain","setDeviceId","setGlobalUserProperties","identify","clearUserProperties","setGroup","logRevenueV2","regenerateDeviceId","logEventWithTimestamp","logEventWithGroups"];
function v(e){function t(t){e[t]=function(){e._q.push([t].concat(Array.prototype.slice.call(arguments,0)));
}}for(var n=0;n<d.length;n++){t(d[n])}}v(n);n.getInstance=function(e){e=(!e||e.length===0?"$default_instance":e).toLowerCase();
if(!n._iq.hasOwnProperty(e)){n._iq[e]={_q:[]};v(n._iq[e])}return n._iq[e]};e.amplitude=n;
})(window,document);
</script>
<base href="/">
<script type="text/javascript">
document.write('<scri'+'pt ');
document.write('type="text/javascript" ');
document.write('src="'+'/shared/theme/config.js');
document.write("?"+new Date().getTime()+'">');
document.write('</scri'+'pt>');
</script>
</head>
<body snitch ios7-viewport-fix>
<noscript>
<h1>Kahoot! needs JavaScript to work</h1>
<p>
To use Kahoot!, you need to have JavaScript enabled in your browser. To enable JavaScript, please do the following:
</p>
<ul>
<li>Follow these instructions.</li>
<li>Make sure you have the latest browser.</li>
<li>Turn off or disable the NoScript extension, if you have it.</li>
<li>Contact your IT administrator to allow access to Kahoot! in your security preferences.</li>
</ul>
<p>If you continue to have problems, please let us know by contacting Kahoot! support.</p>
</noscript>
<div id="debug-info" debug-info="dev,test" debug-timestamp></div>
<dev-mode></dev-mode>
<div class="loader" loader></div>
<iframe
id="gameBlockIframe"
style="display:none;"
class="game-block-iframe"
sandbox="allow-scripts allow-same-origin"
scrolling="no">
</iframe>
<div id="mainView" ng-cloak ng-view>
<h1>Join in a Kahoot! here</h1>
<p>To learn more about Kahoot! visit kahoot.com</p>
</div>
<div ng-cloak alerts></div>
<script type="text/javascript" src="/js/bootstrap.js"></script>
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
function gup( name, url ) {
if (!url) url = location.href;
name = name.replace(/[\[]/,"\\\[").replace(/[\]]/,"\\\]");
var regexS = "[\\?&]"+name+"=([^&#]*)";
var regex = new RegExp( regexS );
var results = regex.exec( url );
return results == null ? null : results[1];
}
var clientId = gup('gaId', window.location.search);
if (clientId) {
ga('create', 'UA-35308575-1', 'auto', {'allowLinker': true, 'clientId':gup('gaId', window.location.search)});
ga('create', 'UA-35308575-4', 'auto', {'name': 'legacy', 'clientId':gup('gaId', window.location.search)});
var platform = gup('platform', window.location.search);
if (typeof platform === 'string' && platform == 'iOS') {
window.ga('set', 'appName', 'Kahoot');
window.ga('set', 'appId', 'no.mobitroll.kahoot.controller');
}
if (typeof platform === 'string' && platform == 'Android') {
window.ga('set', 'appName', 'Kahoot');
window.ga('set', 'appId', 'no.mobitroll.kahoot.android');
}
} else {
ga('create', 'UA-35308575-1', 'auto', {'allowLinker': true});
ga('create', 'UA-35308575-4', 'auto', {'name': 'legacy'});
}
ga('send', 'pageview');
ga('legacy.send', 'pageview');
</script>
</body>
</html>
implicitly_wait needs to only be declared once when driver is initialized. To explicitly wait a specified number of seconds, you may use time.sleep(), although the more practical solution is just dynamically wait for the element to be present/clickable using Selenium's WebDriverWait.
Web Scraping - sessionStorage distil_referrer
I am trying to find a good way to get into a website, but I have found a big issue.
What is returning is the header of the html file, where there are those lines of code:
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml"><head>
<meta name="ROBOTS" content="NOINDEX, NOFOLLOW" />
<meta http-equiv="cache-control" content="max-age=0" />
<meta http-equiv="cache-control" content="no-cache" />
<meta http-equiv="expires" content="0" />
<meta http-equiv="expires" content="Tue, 01 Jan 1980 1:00:00 GMT" />
<meta http-equiv="pragma" content="no-cache" />
<meta http-equiv="refresh" content="10; url=/distil_r_blocked.html? Ref=/de/company/TI&distil_RID=F180AC0A-ECA2-11E6-80AC- E678F530D985&distil_TID=20170206193206" />
<script type="text/javascript">
(function(window){
try {
if (typeof sessionStorage !== 'undefined'){
sessionStorage.setItem('distil_referrer', document.referrer);
}
} catch (e){}
})(window);
</script>
<script type="text/javascript" src="/ga625684.js" defer="defer"></script><style type="text/css">#d__fFH{position:absolute;top:-5000px;left:-5000px}#d__fF{font-family:serif;font-size:200px;visibility:hidden}#svevtdraftsrybvfradvq{display:none!important}</style></head>
<body>
<div id="distil_ident_block"> </div>
<div id="d__fFH" style="position: absolute; top: -5000px; left: -5000px;"><object id="d_dlg" classid="clsid:3050f819-98b5-11cf-bb82-00aa00bdce0b" width="0px" height="0px"></object><span id="d__fF" style="font-family: ZWAdobeF,serif; font-size: 72px; visibility: hidden;">mmmmmmmmlli</span></div></body></html>
</script>
Is there a workaround to actually get the webpage?
For now i am trying to use Selenium without results
The code is the following:
from selenium import webdriver
fp = webdriver.FirefoxProfile()
driver = webdriver.Firefox(firefox_profile=fp)
driver.get("https://www.moneyhouse.ch/de/company/TI")
print driver.page_source
Customizing GET Response Using Python Requests Module
I'm a newbie to Python here, so bear with me...
Trying to experiment making a simple oAuth call to Instagram API. After you register your application, you get your client ID, client secret, etc, the first step in the oAuth process is to direct the user to this authorization URL:
https://api.instagram.com/oauth/authorize/?client_id=CLIENT-ID&redirect_uri=REDIRECT-URI&response_type=code
When I load this URL in a browser with my client ID and Redirect URL, the following URL appears in the browser (for example):
http://instagrram.geometryfletch.com/home.html?code=956237827314ee22092384984938
My question is, how can I replicate what happens in the browser using the Requests module?
when I try the following:
>>> import requests
>>> b = requests.get('https://api.instagram.com/oauth/authorize/?client_id=c918883453360349850498&redirect_uri=http://instagrram.myredirect.com/home.html&response_type=code')
>>> b.text
What I get back is this "garbbled" response (I know it's not really grabbled, Requests is doing what I tell it and returning something appropriate):
u'<!DOCTYPE html>\n<!--[if lt IE 7]> <html lang="en" class="no-js lt-ie9 lt-ie8 lt-ie7 not-logged-in "> <![endif]-->\n<!--[if IE 7]> <html lang="en" class="no-js lt-ie9 lt-ie8 not-logged-in "> <![endif]-->\n<!--[if IE 8]> <html lang="en" class="no-js lt-ie9 not-logged-in "> <![endif]-->\n<!--[if gt IE 8]><!-->
<html lang="en" class="no-js not-logged-in "> <!--<![endif]-->\n
<head>\n
<meta charset="utf-8">
\n
<meta http-equiv="X-UA-Compatible" content="IE=edge">
\n\n <title>Log in — Instagram</title>\n\n
<script type="text/javascript">\
n
WebFontConfig = {
\
n
custom: {\n
families: [\'proxima-nova:n4,n7\'],\n urls: [\'//instagramstatic-a.akamaihd.net/bluebar/660508e/cache/styles/fonts.css\']\n }\n };\n</script>
\n
<script src="//instagramstatic-a.akamaihd.net/bluebar/660508e/scripts/webfont.js" type="text/javascript"
async></script>
\n\n \n \n
<meta name="robots" content="noimageindex">
\n \n
<meta name="apple-mobile-web-app-capable" content="yes">
\n
<meta name="apple-mobile-web-app-status-bar-style" content="black">
\n\n\n \n
<meta id="viewport" name="viewport"
content="width=device-width, user-scalable=no, initial-scale=1, minimum-scale=1, maximum-scale=1">
\n\n\n
<script type="text/javascript">\
n(function () {\n
var docElement = document.documentElement;\n
var classRE = new RegExp(\'(^|\\\\s)no-js(\\\\s|$)\');\n var className = docElement.className;\n docElement.className = className.replace(classRE, \'$1js$2\');\n })();\n </script>
\n\n \n\n \n \n \n
<link rel="Shortcut Icon" type="image/x-icon"
href="//instagramstatic-a.akamaihd.net/bluebar/660508e/images/ico/favicon.ico">
\n \n \n
<link rel="apple-touch-icon-precomposed"
href="//instagramstatic-a.akamaihd.net/bluebar/660508e/images/ico/apple-touch-icon-precomposed.png">
\n
<link rel="apple-touch-icon-precomposed" sizes="72x72"
href="//instagramstatic-a.akamaihd.net/bluebar/660508e/images/ico/apple-touch-icon-72x72-precomposed.png">
\n
<link rel="apple-touch-icon-precomposed" sizes="114x114"
href="//instagramstatic-a.akamaihd.net/bluebar/660508e/images/ico/apple-touch-icon-114x114-precomposed.png">
\n
<link rel="apple-touch-icon-precomposed" sizes="144x144"
href="//instagramstatic-a.akamaihd.net/bluebar/660508e/images/ico/apple-touch-icon-144x144-precomposed.png">
\n \n \n
<link href="//instagramstatic-a.akamaihd.net/bluebar/660508e/cache/styles/distillery/dialog-main.css"
type="text/css" rel="stylesheet"></link>
\n
<!--[if lt IE 9]>\n <style>\n .dialog-outer {\n min-height: 0;\n }\n </style>\n
<![endif]-->\n\n \n
<script src="//instagramstatic-a.akamaihd.net/bluebar/660508e/scripts/jquery.js" type="text/javascript"></script>
\n
<script src="//instagramstatic-a.akamaihd.net/bluebar/660508e/scripts/bluebar.js" type="text/javascript"></script>
\n
<script type="text/javascript">\
n
$(document).ready(function () {\n
$("#id_username").focus();\n
setTimeout(function () {\n
document.getElementById(\'viewport\').setAttribute(\'content\', \'width=\'+ window.innerWidth + \', user-scalable=no\');\n }, 5);\n });\n</script>
\n\n\n
</head>
\n
<body class="p-dialog oauth-login">\n \n \n
<div class="root">\n \n
<section class="dialog-outer">\n
<div class="dialog">\n
<header>\n <h1 class="logo">Instagram</h1>\n \n</header>
\n
<div class="dialog-main">\n \n\n\n\n\n\n\n
<form method="POST" id="login-form" class="adjacent"
action="/accounts/login/?force_classic_login=&next=/oauth/authorize/?client_id=c91888345336494ab7ea7046427ca23e%26redirect_uri=http://instagrram.geometryfletch.com/home.html%26response_type=code">
\n <input type="hidden" name="csrfmiddlewaretoken" ..........
but how can I get Requests to return just simply the code:code=956237827314ee22092384984938 as when you load the URL into a browser?
For production purposes, you should not re-implement oauth. Please have a look at https://pypi.python.org/pypi/oauthlib which is an established library for performing the oauth authentication logic. If you want to stick with requests, then there also is https://github.com/requests/requests-oauthlib. Other than that, regarding your question My question is, how can I replicate what happens in the browser using the Requests module? This is not trivial. First, use curl or a browser plugin for debugging/reconstructing the protocol flow. The second step then is to rebuild the same flow using requests. Example: When accessing the first URL you mentioned in your question via GET, the server responds with a 302 redirection whose target is given in the Location field of the response header. The response also sets a cookie via the Set-Cookie header field. All of this is important.