I am trying to upload images to ImageBam by filling its web-forms and requesting POST.
I don't know too much about urllib2, httplib, multipart stuff. I am trying to use MECHANIZE module
But I think it shouldn't be too complex because it is just a web form, I will fill it and post it.
The page, where upload forms are:
http://www.imagebam.com/basic-upload
The form I am trying to fill:
<form name='form' id='form' enctype="multipart/form-data" method="post" action="/sys/upload/save">
<table align="center">
<tr>
<td>
01: <input type="file" name="file[]" size="30"><br>
02: <input type="file" name="file[]" size="30"><br>
03: <input type="file" name="file[]" size="30"><br>
04: <input type="file" name="file[]" size="30"><br>
05: <input type="file" name="file[]" size="30"><br>
also I saw a guy created an app using python;
http://sourceforge.net/projects/pymguploader/files/pymguploader/2011-12-24/
I want to write something like that, but much more basic of course.
anyway, here is my problem;
when I execute these;
import mechanize
a=mechanize.Browser()
a.open("http://www.imagebam.com/basic-upload")
forms=mechanize.ParseResponse(response)
a.select_form(nr=0)
dosya=open("file path...","r")
everything works fine I think.
also
print a
gives this output:
<Browser visiting http://www.imagebam.com/basic-upload
selected form:
<form POST http://www.imagebam.com/sys/upload/save multipart/form-data
<FileControl(file[]=<No files added>)>
<FileControl(file[]=<No files added>)>
<FileControl(file[]=<No files added>)>
<FileControl(file[]=<No files added>)>
<SelectControl(content_type=[*x, 1, 0])>
<SelectControl(thumb_size=[*100, 150, 180, 250, 300, 350])>
<SelectControl(thumb_aspect_ratio=[crop, *resize])>
<SelectControl(thumb_file_type=[gif, *jpg])>
<CheckboxControl(thumb_info=[1])>
<CheckboxControl(gallery_options=[*1])>>
>
but when I
a["file[]"]=dosya
the error is;
Traceback (most recent call last):
File "<pyshell#46>", line 1, in <module>
a["file[]"]=dosya
File "build\bdist.win32\egg\mechanize\_form.py", line 2780, in __setitem__
control = self.find_control(name)
File "build\bdist.win32\egg\mechanize\_form.py", line 3101, in find_control
return self._find_control(name, type, kind, id, label, predicate, nr)
File "build\bdist.win32\egg\mechanize\_form.py", line 3183, in _find_control
raise AmbiguityError("more than one control matching "+description)
AmbiguityError: more than one control matching name 'file[]'
How can I solve this problem?
SOLVED
Solution:
a.add_file(dosya,"filename",nr=0)
that automatically searches type=file inputs and adds my file to first one(nr=0 provides it)
New Problem
After I sending POST data (or I think it sends)
This page comes as a response;
<?xml version='1.0' encoding='utf-8'?><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML+RDFa 1.0//EN" "http://www.w3.org/MarkUp/DTD/xhtml-rdfa-1.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:og="http://opengraphprotocol.org/schema/" xmlns:fb="http://developers.facebook.com/schema/" xmlns:dc="http://purl.org/dc/elements/1.1/" xml:lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8" />
<meta name="author" content="ImageBam.com" />
<meta name="description" content="Free Image Hosting and photo sharing. Create an online album with bulk upload tools and share with family and friends." />
<meta name="keywords" content="image hosting, free image hosting, photo sharing, upload photo, free photo gallery, photo host, image gallery" />
<meta name="robots" content="follow" />
<meta name="revisit-after" content="1 days" />
<meta property="fb:admins" content="3433880" />
<link rel="stylesheet" href="http://www.imagebam.com/style.css" type="text/css" />
<link rel="shortcut icon" href="http://www.imagebam.com/favicon.ico" />
<title>Fast, Free Image Hosting - ImageBam</title>
<script type="text/javascript" src="http://www.imagebam.com/JS/imagebam.js"></script>
<script type="text/javascript" src="http://www.imagebam.com/JS/pt.js"></script>
</head>
<body>
<!-- IMAGEBAM HEADER -->
<div class="scrollme">
<div class="abody">
<!-- everything -->
<div class="banner cursor" style="float:left;" onclick='top.location="http://www.imagebam.com"'></div>
<div style="float:right; text-align:right; border:0px solid #f2f2f2; border-top:none; padding-top: 5px; padding-left:3px; padding-right:10px;">
</div>
<div style="clear:left;"></div>
<div class="dtab">
<ul>
<li class="inactive">Multi-Upload</li>
<li class="inactive">Zip-Upload</li>
<li class="inactive">Basic Upload</li>
<li class="inactive">Learn More</li>
<li class="inactive">FAQ</li>
<li class="inactive">Register</li>
<li class="inactive">Login</li>
<li class="inactive">Premium</li>
</ul>
</div><br />
<!-- Google Code for Imagebam Uploaded Image Conversion Page -->
<script type="text/javascript">
/* <![CDATA[ */
function img404(ID,fsrc){
document.getElementById('thumb_404_info').style.display="block";
document.getElementById("img_"+ID).style.display = "none";
document.getElementById("alt_"+ID).style.display = "inline";
setTimeout("reloadImg("+ID+",'"+fsrc+"')", 500);
}
function reloadImg(ID,fsrc){
mrand = Math.random();
document.getElementById("img_"+ID).style.display = "inline";
document.getElementById("alt_"+ID).style.display = "none";
document.getElementById("img_"+ID).src = fsrc+"?"+mrand;
}
/* ]]> */
</script>
<div class="box_wait" style="text-align:center; display:none;" id="thumb_404_info">Thumbnails that are being processed in the background might not load right away.<br /></div>
<div style="text-align:center; margin-bottom:5px;">
<b>NEW!</b> VideoBam.com (HD Video Hosting)
</div>
<fieldset><legend><img src="/img/icons/photos.png" alt="" style="vertical-align:middle; line-height:16px; height:16px; padding-right:5px;" /> All at Once</legend>
<table style="width:100%;"><tr>
<td>
<b>BB-Code</b><br />
<textarea onclick="this.select();" style="width:300px; height:200px;"></textarea>
</td>
<td>
<b>HTML-Code</b><br />
<textarea onclick="this.select();" style="width:300px; height:200px;"></textarea>
</td>
</tr>
</table>
</fieldset>
<!--
<fieldset><legend style='color:green;'><img src='/img/icons/new.png' alt='' style='vertical-align:middle; line-height:16px; height:16px; padding-right:5px;'> NEW! ImageBam Remote Upload Widget</legend>
<b>Webmasters / Mods!</b><br> Allow your users to upload images to ImageBam <b>without leaving your website or forum!</b><br> Add our new ImageBam Remote Upload Widget to you website!<br>
Please spread the word! Thank you!
</fieldset>
-->
<div style="text-align:center; margin-bottom:5px;">
<b>NEW!</b> VideoBam.com (HD Video Hosting)
</div>
<fieldset><legend><img src="/img/icons/delete.png" alt="" style="vertical-align:middle; line-height:16px; height:16px; padding-right:5px;" /> All Removal Links</legend>
Do not share the links below. You can use them to delete the photos you have uploaded.<br />
<textarea onclick="this.select()" style="width:600px; height:200px;"></textarea>
</fieldset>
<!-- Google Code for Imagebam Uploaded Image Conversion Page -->
<script type="text/javascript">
/* <![CDATA[ */
var google_conversion_id = 1068053810;
var google_conversion_language = "en_US";
var google_conversion_format = "3";
var google_conversion_color = "ffffff";
var google_conversion_label = "6tqpCPa-chCy6qT9Aw";
var google_conversion_value = 0;
/* ]]> */
</script>
<script type="text/javascript" src="http://www.googleadservices.com/pagead/conversion.js">
</script>
<noscript>
<div style="display:inline;">
<img height="1" width="1" style="border-style:none;" alt="" src="http://www.googleadservices.com/pagead/conversion/1068053810/?label=6tqpCPa-chCy6qT9Aw&guid=ON&script=0" />
</div>
</noscript>
</div>
<div class="footer">
<a class="footera" href="http://www.imagebam.com/">ImageBam</a> | <a class="footera" href="/remote-upload-widget">Remote Upload Widget</a> | <a class="footera" href="http://www.imagebam.com/screengrab_upload_firefox_extension">screengrab tool</a> | <a class="footera" href="http://www.imagebam.com/terms-of-service">terms of service</a> | <a class="footera" href="http://www.imagebam.com/frequently-asked-questions">help</a> | <a class="footera" href="http://support.imagebam.com" target="_blank">support forums</a> | <a class="footera" href="http://code.google.com/p/imagebam-api/">API for developers</a> | <a class="footera" href="http://www.imagebam.com/report-abuse">report abuse</a>
<div style="height:35px; overflow:hidden;">
<div id="google_translate_element" style="margin-top:9px;"></div><script type="text/javascript">
function googleTranslateElementInit() {
new google.translate.TranslateElement({
pageLanguage: 'en'
}, 'google_translate_element');
}
</script><script src="http://translate.google.com/translate_a/element.js?cb=googleTranslateElementInit" type="text/javascript"></script>
</div>
<div style="text-align:center; color:#999; margin-top:10px;">
<table style="margin:auto;"><tr><td><img src="http://1.imagebam.com/static/img/tux.png" alt="tux" /></td><td>Powered by dedicated Linux servers. Flixya Entertainment, LLC © 2010</td></tr></table>
</div>
</div>
</div>
<div id="updater_index"></div>
<script type="text/javascript">
</script>
<script src="http://www.google-analytics.com/urchin.js" type="text/javascript">
</script>
<script type="text/javascript">
_uacct = "UA-2424497-2";
urchinTracker();
</script>
</body>
</html>
Normally, it is the after-uploading page that comes out with image's links etc.
But I think there is a dynamic process, because the links were not prepared when I got the page.
Am I missing something? because even if you dont fill the inputs on form, if you submit() it, it redirects you to that after-uploading page..
Use select_control() also with nr=0 to select the first file.
Related
I am new to selenium. I have managed to login in to our work practice management system. So the basic setup is fine.
I am then faced with this:
I need to drop down the Work dropdown and select a premade report (All Tasks For Export):
I have tried a lot of stuff...... CSS Selector, Class, ID
But I always get error: Message: no such element: Unable to locate element:
Code:
driver = webdriver.Chrome()
driver.get('https://xxxxx.senta.co/a/i/a')
WebDriverWait(driver,20).until(EC.element_to_be_clickable((By.CSS_SELECTOR,"input[name='email']"))).send_keys("xxx#xxx.com")
WebDriverWait(driver,20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[name='password']"))).send_keys("xxxxxx")
driver.find_element(By.CSS_SELECTOR, 'button#submit').click()
dropdown = Select(driver.find_element(By.CSS_SELECTOR,'major#navjobs'))
But maybe I am selecting the wrong element entirely. I will post the HTML below. Maybe I understand the Selenium but not the HTML!! Thanks in advance.
And then the HTML for the elements in the list look like this:
OK here is the HTML of the page. Not sure it's going to help much!
<!DOCTYPE html><html lang="en" ng-app="senta" se-file-drop="onFileSelect($files)" ng-controller="BodyCtrl" ng-class="{ selectfile:selectfile }"> <head> <base href="/"> <meta charset="utf-8"> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" /> <meta name="HandheldFriendly" content="true" /> <meta name="robots" content="noindex"> <link id="favicon" rel='icon' type='image/ico' sizes='32x32' href='https://dsik6juztonps.cloudfront.net/client/public/images/favicon.ico'> <title ng-bind="$root.notiftitle + $root.title + $root.appTitle">Loading...</title> <link href="https://dsik6juztonps.cloudfront.net/client/public/dist/lib/20211115/lib.css" rel="stylesheet" /> <link href="https://dsik6juztonps.cloudfront.net/client/public/dist/_m258abe0ebbf4d5628de49b9a35dff42e/style.css" rel="stylesheet" /> <script src="https://dsik6juztonps.cloudfront.net/client/public/dist/lib/20211115/prelib.min.js"></script> <script src="https://dsik6juztonps.cloudfront.net/client/public/dist/lib/20211115/momentjs/en-gb.js"></script> </head> <body id="{{$root.bodyId}}" class="droptarget {{userClass}} {{skinClass}}" ng-class="{ 'on-scrolled':notAtTop, 'preheader-on':preheaderOn, 'preheader2-on':preheader2On, 'preheadertimer-on':preheaderTimerOn }"> <div ng-if="$root.user.loggedin" ng-include="'https://dsik6juztonps.cloudfront.net/client/public/dist/_m258abe0ebbf4d5628de49b9a35dff42e/html/en-gb/header.html'" ng-controller="NavBarCtrl"></div> <div class="dropindic"> <div class="lightbox"></div> <div class="centred"> <p class="text" style="">Drop your files here to upload into Senta</p><i class="fa fa-file"></i> <p class="selectbutton">Alternatively: <input type="file" id="selectfile"> <button type="button" class="btn btn-primary pseudoselect" ng-click="selectFile()">Select file</button> <button type="button" class="btn btn-normal" ng-click="cancelSelectFile()">Cancel</button> </p> </div> </div> <div ng-if="deepheader" class="deepheader"></div> <div class="container"> <div ui-view> <div class="positioner"> <div class="notifier"> <span class="spinning"><span class="spinner"><i class="fa fa-spin fa-refresh"></i></span></span> <span class="msg">Loading...</span> </div> </div> </div> <div id="react-root"></div> </div> <div ng-if="$root.expressionfooter" ng-include="'https://dsik6juztonps.cloudfront.net/client/public/dist/_m258abe0ebbf4d5628de49b9a35dff42e/html/en-gb/settings/expression/footer.html'" ng-controller="ExpressionTesterCtrl" ></div> <div ng-if="$root.previewfooter" ng-include="'https://dsik6juztonps.cloudfront.net/client/public/dist/_m258abe0ebbf4d5628de49b9a35dff42e/html/en-gb/modal/preview-footer.html'"></div> <script src="https://dsik6juztonps.cloudfront.net/client/public/dist/lib/20211115/postlib.min.js"></script> <script src="https://dsik6juztonps.cloudfront.net/client/public/dist/_m258abe0ebbf4d5628de49b9a35dff42e/app.min.en-gb.js"></script> <script src="https://www.gstatic.com/charts/loader.js"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/pdf.js/1.9.638/pdf.min.js"></script> </body><script src="https://dsik6juztonps.cloudfront.net/react/static/js/en-gb/main.eda07c95.js"></script></html>
It's look like you want find element By.CSS_SELECTOR like you are using By.XPATH.
For example, if you want to find <ul> which contains here on your screenshot
you can use:
driver.find_element(By.CLASS_NAME, 'dropdown-menu')
driver.find_element(By.CSS_SELECTOR, '.dropdown-menu')
driver.find_element(By.XPATH, '//ul[#class='dropdown-menu']')
driver.find_element(By.XPATH, '//ul[#id='work-dropdown'][#class='dropdown-menu'])
but i can't understand what you are looking for...
UPDATE:
try to find all <li> with ng-repeat='viewt in viewst track by viewt._id'
driver.find_elements(By.XPATH, '//li[#ng-repeat="viewt in viewst track by viewt._id"]')[here_is_index(even 0 idk)]
and choose needed by indexing. It's really hard to help you without html code...
I have my jupyter hub and single-user server running. I am using Jupyter notebook REST APIs to get all available notebooks. I am using http://127.0.0.1:8000/user/username/api/contents . With this I am able to see all available notebooks on mu UI.
However when I try to do the same from my python script I am getting 200 response code but unable to extract any content as its redirecting to login page where I need to enter username and password for my user. I also tried to pass these credentials through requests package. Something like this
import requests
api_url = 'http://127.0.0.1:8000/user/username/api/contents'
result = requests.get(api_url, auth=('username', 'password'))
print result
print result.json()
The outpus is:
<!DOCTYPE HTML>
<html>
<head>
<meta charset="utf-8">
<title>Jupyter Hub</title>
<meta http-equiv="X-UA-Compatible" content="chrome=1">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<link rel="stylesheet" href="/hub/static/css/style.min.css?v=d96e0760e0c2b7356ce89635b646c350" type="text/css"/>
<script src="/hub/static/components/requirejs/require.js?v=6da8be361b9ee26c5e721e76c6d4afce" type="text/javascript" charset="utf-8"></script>
<script>
require.config({
urlArgs: "v=('20170907062813',)",
baseUrl: '/hub/static/js',
paths: {
components: '../components',
jquery: '../components/jquery/jquery.min',
bootstrap: '../components/bootstrap/js/bootstrap.min',
moment: "../components/moment/moment",
},
shim: {
bootstrap: {
deps: ["jquery"],
exports: "bootstrap"
},
}
});
</script>
<script type="text/javascript">
window.jhdata = {
base_url: "/hub/",
prefix: "/",
}
</script>
</head>
<body>
<noscript>
<div id='noscript'>
Jupyter Hub requires JavaScript.<br>
Please enable it to proceed.
</div>
</noscript>
<div id="header" class="navbar navbar-static-top">
<div class="container">
<span id="jupyterhub-logo" class="pull-left"><img src='/hub/logo' alt='JupyterHub' class='jpy-logo' title='Home'/></span>
</div>
</div>
<div id="login-main" class="container">
<form action="/hub/login?next=%2Fhub%2Fuser%2Fpnda%2Fapi%2Fcontents" method="post" role="form">
<div class="auth-form-header">
Sign in
</div>
<div class='auth-form-body'>
<p id='insecure-login-warning' class='hidden'>
Warning: JupyterHub seems to be served over an unsecured HTTP connection.
We strongly recommend enabling HTTPS for JupyterHub.
</p>
<label for="username_input">Username:</label>
<input
id="username_input"
type="username"
autocapitalize="off"
autocorrect="off"
class="form-control"
name="username"
val=""
tabindex="1"
autofocus="autofocus"
/>
<label for='password_input'>Password:</label>
<input
type="password"
class="form-control"
name="password"
id="password_input"
tabindex="2"
/>
<input
type="submit"
id="login_submit"
class='btn btn-jupyter'
value='Sign In'
tabindex="3"
/>
</div>
</form>
</div>
<div class="modal fade" id="error-dialog" tabindex="-1" role="dialog" aria-labelledby="error-label" aria-hidden="true">
<div class="modal-dialog">
<div class="modal-content">
<div class="modal-header">
<button type="button" class="close" data-dismiss="modal"><span aria-hidden="true">×</span><span class="sr-only">Close</span></button>
<h4 class="modal-title" id="error-label">Error</h4>
</div>
<div class="modal-body">
<div class="ajax-error">
The error
</div>
</div>
<div class="modal-footer">
<button type="button" class="btn btn-default" data-dismiss="modal">Cancel</button>
<button type="button" class="btn btn-primary" data-dismiss="modal" data-dismiss="modal">OK</button>
</div>
</div>
</div>
</div>
<script>
if (window.location.protocol === "http:") {
// unhide http warning
var warning = document.getElementById('insecure-login-warning');
warning.className = warning.className.replace(/\bhidden\b/, '');
}
</script>
</body>
</html>
Traceback (most recent call last):
File "test.py", line 31, in <module>
check_running()
File "test.py", line 28, in check_running
contents = r.json()
File "/usr/lib/python2.7/dist-packages/requests/models.py", line 740, in json
return json.loads(self.content.decode(encoding), **kwargs)
File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
raise ValueError("No JSON object could be decoded")
I am fairly new to Jupyter. My I missing out on something. How else can I authenticate my user prior calling any API.
Saw this in jupyerhub documentation - Note that the API token authorizes JupyterHub REST API requests. The same token does not authorize access to the Jupyter Notebook REST API provided by notebook servers managed by JupyterHub. A different token is used to access the Jupyter Notebook API.
So this is the exactly what I am looking for. How can I create this different token so as to access jupyter notebook APIs thorugh hub.
Jupyter Notebooks require an access token (this should show when you start up jupyter notebook). In my case, something like this, then worked:
import requests
#Note that the url doesn't have a user or username attribute
api_url = 'http://127.0.0.1:8888/api/contents'
payload = {'token': 'xxxxxx'}
result = requests.get(api_url
#, auth=('username', 'password')
,params = payload)
I have one phone number with a request url which works (speaks the text and then forwards the call) but then I have an app (a browser outbound call) going to the same request url and only the text to speech works, and the forwarding of the call gives an error and hangs up. Why doesn't it forward like the number's request url? I checked and the capability token renders fine...
views.py
def once_connected_view(request):
response = twiml.Response()
response.say("Please wait while we connect your call.", voice='alice')
response.dial("xxx-xx-xxx-xxxx")
return HttpResponse(str(response))
def home_view(request):
capability = TwilioCapability(account_sid, auth_token)
capability.allow_client_outgoing(application_sid)
token = capability.generate()
query_set = Model.objects.all()
return render(request, "base.html", {"query_set":query_set, "token":token})
urls.py
urlpatterns = [
url(r'^$', views.home_view, name="home"),
url(r'^once_connected/', views.once_connected_view, name="once_connected"),
]
number request url
http://xx.xx.com/once_connected/ http GET
app request url
http://xx.xx.com/once_connected/ http GET
main site url
https://xx.xx.com/
base.html
<!doctype html>
<head>
<script type="text/javascript" src="https://static.twilio.com/libs/twiliojs/1.2/twilio.min.js"></script>
<script type="text/javascript"src="https://ajax.googleapis.com/ajax/libs/jquery/1.7.2/jquery.min.js"></script>
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css" integrity="sha384-1q8mTJOASx8j1Au+a5WDVnPi2lkFfwwEAa8hDDdjZlpLegxhjVME1fgjWPGmkzs7" crossorigin="anonymous">
<meta name="viewport" content="user-scalable=no, initial-scale=1, maximum-scale=1, minimum-scale=1, width=320, height=device-height, target-densitydpi=medium-dpi" />
{% load staticfiles %}
<link rel="stylesheet" href="{% static 'cars/responsive.css' %}">
</head>
<body>
<div class="container" id="wrapper">
<div class="video-background-container">
<video preload="auto" autoplay="" loop="" muted="" class="video-background hidden-xs hidden-sm">
<source type="video/mp4" src="omitted">
</video>
</div>
<div class="grid-overlay text-center">
<nav class="navbar navbar-default navbar-fixed-top" style="background:none;">
<div class="container">
<div class="navbar-header">
<a class="navbar-brand" id="logo" href="#">Brand Name</a>
</div>
<div id="navbar" class="navbar-collapse collapse">
<ul class="nav navbar-nav navbar-right">
<li class="replace-call">Contact Us</li>
</ul>
</div>
</div>
</nav>
...
<script src="https://code.jquery.com/jquery-2.2.0.min.js" type="text/javascript"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/js/bootstrap.min.js" integrity="sha384-0mSbJDEHialfmuBBQP6A4Qrprq5OVfW37PRR3j5ELqxss1yVqOtnepnHVP9aJ7xS" crossorigin="anonymous"></script>
<script type="text/javascript">
/* Create the Client with a Capability Token */
Twilio.Device.setup("{{ token }}");
/* Let us know when the client is ready. */
Twilio.Device.ready(function (device) {
$(".replace-call").html("<a href='#' onclick='call()'>Call From Browser</a>");
});
/* Report any errors on the screen */
Twilio.Device.error(function (error) {
$(".replace-call").html('Contact Us');
});
Twilio.Device.connect(function (conn) {
$(".replace-call").html("<a href='#' onclick='hangup()'>End Call</a>");
});
/* Connect to Twilio when we call this function. */
function call() {
Twilio.Device.connect();
}
function hangup() {
Twilio.Device.disconnectAll();
$(".replace-call").html('Contact Us');
}
</script>
</body>
</html>
I am hosted on pythonanywhere.
After only minimal hair loss, I have made it work by adding a callerID attribute to the dial verb.
def once_connected_view(request):
response = twiml.Response()
response.say("Please wait while we connect your call.", voice='alice')
response.dial("xxx-xx-xxx-xxxx", callerId="+xxxxxxxxxx") # here
return HttpResponse(str(response))
I am trying to get through a proxy server through python to extract some information from the website, so far I this piece of code, but it doesnt seem to be working
import requests
import BeautifulSoup
URL = 'http://proxy.library.upenn.edu/login?url=http://clients1.ibisworld.com/'
session = requests.session()
# This is the form data that the page sends when logging in
login_data = {
'pennkey': "****",
'password': "****",
'submit': 'login',
}
# Authenticate
r = session.post(URL, data=login_data)
doc = BeautifulSoup.BeautifulSoup(r.content)
print doc
edit: this is what this prints:
Gorkems-MacBook-Pro:desktop gorkemyurtseven$ python extract.py
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<head>
<meta name="HandheldFriendly" content="True" />
<meta name="viewport" content="width=device-width, height=device-height, user-scalable=yes, minimum-scale=.5" />
<title>University of Pennsylvania Libraries Proxy Service - Login</title>
<link href="/public/proxysm.css" media="print, screen" rel="stylesheet" type="text/css" />
<script language="javascript">
function validate(){
var isgoldcard = document.authenticate.pass.value;
var isgoldcardRegxp = /00000/;
if (isgoldcardRegxp.test(isgoldcard) == true)
alert("Authentication is by PennKey only.");
}
</script>
<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-982196-4']);
_gaq.push(['_trackPageview']);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
<!--[if IE]>
<style>
table, form .limitwidth {width: 252px;}
.holdsubmit {width: 143px;}
</style>
<![endif]-->
</head>
<body onload="document.authenticate.user.focus();">
<div id="logostripe">
<div><img src="/public/librarieslogologin.gif" border="0" alt="Penn Libraries Home" /></div>
</div>
<h1>Libraries Proxy Service</h1>
<div id="holder">
<form name="authenticate" action="https://proxy.library.upenn.edu/login" method="post" autocomplete="off">
<div class="limitwidth">
<input type="hidden" name="url" value="http://clients1.ibisworld.com/" />
<script type="text/javascript">
var t = location.search;
t = t.substr(t.indexOf('proxySessionID')+15,t.indexOf('&')-16);
document.cookie="proxySessionID="+escape(t)+"; path=/; domain=.library.upenn.edu";
</script>
<table align="center" cellspacing="0" cellpadding="2" border="0">
<tr>
<td class="holdlabels"><label for="user">PennKey:</label></td>
<td><input type="text" name="user" /></td>
</tr>
<tr>
<td><label for="password">Password:</label></td>
<td><input type="password" name="pass" onblur="validate(); return false;" /></td>
</tr>
<tr>
<td></td>
<td class="holdsubmit">
<div><input type="submit" value="Login" /></div>
</td>
</tr>
</table>
</div>
</form>
<ul class="moreinfo">
<li><a class="menuitem" href="http://www.upenn.edu/computing/pennkey">PennKey information</a></li>
</ul>
<div class="notices">
The Library Proxy Service allows you to use
domain-restricted resources & services by authenticating yourself as Penn Faculty,
Student, or Staff.
</div>
<div class="alert">
Please note limitations on the use of restricted online resources.
<br /><br />
PennKey holders must be current faculty, student, or staff, have valid University PennCommunity credentials and abide by stated Restrictions On Use.
<br /><br />
In addition, users agree to the University's Appropriate Use Policy.
</div>
</div><!-- close holder -->
</body>
</html>
Here's a solution that works for me (also using Penn's proxy server):
import requests
from bs4 import BeautifulSoup
proxies = {'https': 'https://proxy.library.upenn.edu'}
auth = requests.auth.HTTPProxyAuth('[username]', '[password]')
r = requests.get('http://www.example.com/', proxies=proxies, auth=auth)
print BeautifulSoup(r.content)
The first key is that the proxy server is https, and not http (this took me far too long to figure out). Next, you must use the requests.auth.HTTPProxyAuth method for authenticating with the server. Once, you set these two vars, however, you should be able to navigate wherever you need.
I posted a question yesterday :
Redirect python script to another python script for validation of login credentials
after i was done with that bit, i had another problem related to cookies. I sent the cookies in the header from the python script code:
#Create Cookie
C= Cookie.SimpleCookie()
#take the value of usernameLogin into the variable username
username= form.getvalue('usernameLogin')
#Set-Cookie header with the usernameLogin key
C['usernameLogin'] = username
this code is in the previous python script validate.py
i want to send the cookies to the next script page1.py
this is my code for page1.py :
import cgi
import cgitb
import sqlite3
import Cookie
import os
user_name=""
user_id=""
useridDb=""
resultid=""
resultname=""
idUser=""
if os.environ.has_key("HTTP_COOKIE"):
C= Cookie.SimpleCookie(os.environ.get("HTTP_COOKIE",""))
if C.has_key("usernameLogin"):
user_name= C['usernameLogin'].value
print user_name
form= cgi.FieldStorage()
cgitb.enable()
#Open connection
conn= sqlite3.connect("manager.db")
page1head= """
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<title>Excursions</title>
<link rel='stylesheet' type='text/css' href='/page1.css/' />
<script lang="javascript" type="text/javascript" src="/suggestions.js/"> </script>
</head><body>
<div id="container">
<div id="header">
<h1>Field Note Manager</h1>
<p class="description">Observe...Record...Save!</p>
</div>
<!-- Content Section -->
<div id="wrapper">
<!-- Main Content Section -->
<div id="content">
<h2>Excursions</h2>
<table CELLPADDING="10" ><tr><th> <strong> Location </strong> </th><th> <strong> Date </strong> </th> <th> Time </th></tr>"""
page2head="""</table>
</div>
</div>
<!-- Logout Section -->
<div id="navigation">
<input type="hidden" name="greeting"/>
<form action="http://localhost:8000/cgi-bin/logout.py">
<p><input type="submit" name="logoutBtn" value="Logout" /> </p>
</form>
</div>
<!-- Extra Section for Queries -->
<div id="extra">
<h2>Quick Links</h2>
<dl> <dd><a href="http://localhost:8000/cgi-bin/query.py"/>Query the Database</a> </dd></dl>
<dl> <dd><a href="http://localhost:8000/cgi-bin/addFieldNote.py"/>Add Field Note</a> </dd></dl>
</div>
<!-- Footer -->
<div id="footer">
<p>Copyright 42578647, 2012</p>
</div>
</div>
"""
page1foot= """
</body>
</html>
"""
print "Content_type: text/html\n\n"
print page1head
#print excursion details
cur=conn.cursor()
resultid= cur.execute("SELECT userid FROM login WHERE username=?",[user_name])
cur.fetchone()
for data in resultid:
idUser= int(data)
resultname= cur.execute("""SELECT location,excurDate,excurTime FROM excursion WHERE user=?""",[idUser])
cur.fetchall()
for record in resultname:
print"<tr><td>",record[0],"</td><td>",record[1],"</td><td>",record[2],"</td></tr>"
print page2head
print page1foot
It prints the page, but no queries are generated. Is it because the cookie has not been retrieved and parsed or is there some problem with the SELECT statement?
I found out the answer. The cookies had not been set before the header in the previous validate.py