InstaMapper is a GPS tracking service that updates the device's position more frequently when the device is being tracked live on the InstaMapper webpage. I'd like to have this happen all the time so I thought I'd write a python script to login to my account and access the page periodically.
import urllib2, urllib, cookielib
cj = cookielib.LWPCookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
urllib2.install_opener(opener)
params = urllib.urlencode(dict(username_hb='user', password_hb='hunter2'))
opener.open('http://www.instamapper.com/fe?action=login', params)
if not 'id' in [cookie.name for cookie in cj]:
raise ValueError, "Login failed"
# try secured page
resp = opener.open('http://www.instamapper.com/fe?page=track&device_key=abc')
print resp.read()
resp.close()
The ValueError is raised each time. If I remove this and read the response, the page thinks I have disabled cookies and blocks access to that page. Why isn't cj grabbing the InstaMapper cookie?
Are there better ways to make the tracking service think I'm viewing my account constantly?
action=login is part of the parameters, and should be treated accordingly:
params = urllib.urlencode(dict(action='login', username_hb='user', password_hb='hunter2'))
opener.open('http://www.instamapper.com/fe', params)
(Also, this particular username/password combination is invalid, I assume, that you actually use a valid username and password in your actual code, otherwise the login fails corretly.)
Have you looked at whether there is a cookie specifically designed to foil your attempts? I suggest using Wireshark or other inspector to see if there is a cookie that changes (via javascript, etc) when you manually log in.
(Ethical note: You may be violating the terms of service and incurring much more cost to the company than you are paying for. I used to run a service like this and every additional/unplanned location update was between $0.01 - $0.05 but I'm sure its come down.)
Related
Trying to log into the website kiphideways.com using Requesrs andI am having trouble logging in.
Without needing an account or password, is there any way to tell if I am missing anything from the payload?
LOGIN_URL = 'https://www.kiphideaways.com/login'
URL = 'https://www.kiphideaways.com/my-kip/account/'
I set the following for payload
payload = {'log':"myemail", 'pwd':"mypass"}
I then go do
with requests.Session() as s:
p = s.post(LOGIN_URL, data=payload)
r = s.get(URL)
I can’t log in as the my account page is not populated with my information.
Is there anything wrong with my payload?
By analyzing the POST request in Chrome tools, I see that the complete payload when trying to login through the website forms is:
log=test&pwd=test&rememberme=forever&wp-submit=Log+In&redirect_to=https%3A%2F%2Fwww.kiphideaways.com%2Fmy-kip%2F&mepr_process_login_form=true&mepr_is_login_page=true
Besides it, there are some cookies caught from the browser session. If you want to do the request externally, you should provide all of that.
Though, I can't replicate what happens when the account is good because the account creation seems to be paid :/
I'm trying to get contest data from the url: "https://www.draftkings.com/contest/gamecenter/32947401"
If you go to this URL and aren't logged in, it'll just re-direct you to the lobby. If you're logged in, it'll actually show you the contest results.
Here's some things I tried:
-First, I used Chrome's Dev networking tools to watch requests while I manually logged in
-I then tried copying the cookie that I thought contained the authentication info, it was of the form:
'ajs_anonymous_id=%123123123123123, mlc=true; optimizelyEndUserId'
-I then stored that cookie as an Evironment variable and ran this code:
HEADERS= {'cookie': os.environ['MY_COOKIE'] }
requests.get(draft_kings_url, headers= HEADERS)
No luck, this just gave me the lobby.
I then tried request's built in:
HTTPBasicAuth
HTTPDigestAuth
No luck here either.
I'm no python expert by far, and I've pretty much exhausted what I know and the search results I've found. Any ideas?
The tool that you want is selenium. Something along the lines of:
from selenium import webdriver
browser = webdriver.Firefox()
browser.get(r"https://www.draftkings.com/contest/gamecenter/32947401" )
username = browser.find_element_by_id("user")
username.send_keys("username")
password = browser.find_element_by_id("password")
password.send_keys("top_secret")
login = selenium.find_element_by_name("login")
login.click()
Use fiddler to see the exact request they are making when you try to log in. Then use Session class in requests package.
import requests
session = requests.Session()
session.get('YOUR_URL_LOGIN_PAGE')
this will save all the cookies from your url in your session variable (Like when you use a browser).
Then make a post request to the login url with appropriate data.
You dont have to manually pass cookie data as it is auto generated when you first visit a website. However you can set some header explicitly like UserAgent etc by:
session.headers.update({'header_name':'header_value'})
HTTPBasicAuth & HTTPDigestAuth might not work based on the website.
I am writing a script to automatically scrape information from my companies directory website using mechanize. However, the interpreter returns _response.httperror_seek_wrapper: HTTP Error 401: Authorization Required onbr.open(url) when I run my script.
This is the portion of my code where the interpreter runs into the error.
from sys import path
path.append("./mechanize/mechanize")
import _mechanize
from base64 import b64encode
def login (url, username, password):
b64login = b64encode('%s:%s' % (username, password))
br = _mechanize.Browser()
br.set_handle_robots(False)
br.addheaders.append(('Authorization','Basic %s' % b64login))
br.open(url)
r = br.response()
print r.read()
The site I am trying to access is an internal site within my companies network, and it uses a GlobalSign Certificate for authentication on company-issued computers.
I am sure the authentication information I am inputting is correct, and I have looked everywhere for a solution. Any hints on how to resolve this? Thanks!
It looks like your authentication methods don't match up. You state that your company uses GlobalSign certificates but your code is using Basic authentication. They are NOT equal!!
From a brief look at the Mechanize documentation (limited as it is), you don't implement authentication by manually adding headers. It has it's own add_password method for handling authentication.
Also, as a general HTTP authentication policy, you should NOT use preemptive authentication by adding the authentication headers yourself. You should set up your code with the necessary authentication (based on your library's documentation) and let it handle the authentication negotiation.
I have been googling for this problem for a week now.
The thing I want to achive is the following:
Send a POST request to the URL including the correct credentials.
Save the session (not cookie since my website is not using cookies at the moment)
With the saved session open a session protected URL and grab the contents.
I have seen alot of topics on this with cookies but not with sessions, I tried sessions with requests but seems to fail everytime.
You want to use a URL opener. Here's a sample of how I've managed to do it. If you just want a default opener, use opener=urllib.request.build_opener(), otherwise use the custom opener. This worked when I had to log into a website and keep a session, using URL as your URL, user as user, password as password, all changed as appropriate.
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(http.cookiejar.CookieJar()))
pData=urllib.parse.urlencode({"identity":user,"password":password})
req=urllib.request.Request(URL,pData.encode('utf-8'))
opener.open(req)
req=urllib.request.Request(url)
response= opener.open(req)
I've been googling around for quite some time now and can't seem to get this to work. A lot of my searches have pointed me to finding similar problems but they all seem to be related to cookie grabbing/storing. I think I've set that up properly, but when I try to open the 'hidden' page, it keeps bringing me back to the login page saying my session has expired.
import urllib, urllib2, cookielib, webbrowser
username = 'userhere'
password = 'passwordhere'
url = 'http://example.com'
webbrowser.open(url, new=1, autoraise=1)
cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
login_data = urllib.urlencode({'username' : username, 'j_password' : password})
opener.open('http://example.com', login_data)
resp = opener.open('http://example.com/afterlogin')
print resp
webbrowser.open(url, new=1, autoraise=1)
First off, when doing cookie-based authentication, you need to have a CookieJar to store your cookies in, much in the same way that your browser stores its cookies a place where it can find them again.
After opening a login-page through python, and saving the cookie from a successful login, you should use the MozillaCookieJar to pass the python created cookies to a format a firefox browser can parse. Firefox 3.x no longer uses the cookie format that MozillaCookieJar produces, and I have not been able to find viable alternatives.
If all you need to do is to retrieve specific (in advance known format formatted) data, then I suggest you keep all your HTTP interactions within python. It is much easier, and you don't have to rely on specific browsers being available. If it is absolutely necessary to show stuff in a browser, you could render the so-called 'hidden' page through urllib2 (which incidentally integrates very nicely with cookielib), save the html to a temporary file and pass this to the webbrowser.open which will then render that specific page. Further redirects are not possible.
I've generally used the mechanize library to handle stuff like this. That doesn't answer your question about why your existing code isn't working, but it's something else to play with.
The provided code calls:
opener.open('http://example.com', login_data)
but throws away the response. I would look at this response to see if it says "Bad password" or "I only accept IE" or similar.