I have programmed an application in Python and implemented an auto update mechanism which just retrieves a text file from a cloud server and then checks the version number.
This works fine, but some subsidiaries have their proxies configured in a way that the cloud server can only be accessed through a proxy server.
Now, retrieving something from the web while using a proxy server is generally not a big deal.
I could just use something like this:
import requests
url = 'https://www.cloudserver.com/versionfile'
proxy = 'http://user:pass#proxyserver:port'
proxies = {'http': proxy, 'https': proxy}
requests.get(url, proxies=proxies)
This works wonderfully. The problem is that I don't want my customers to enter username, password and proxyserver. Ok, I could get the username with getpass.getuser(), but not the password.
Another option that sounded promising was pypac:
>>> from pypac import PACSession
>>> session = PACSession()
>>> session.get('http://example.org')
<Response [407]>
Alas, it answers with 407 - Proxy Authentication Required.
There are professional programs out there which just magically use the system proxy settings including username and password (or maybe a hashed version or a ticket of some form) and never have to ask the user about anything. It just works, e.g. Firefox seems to do it that way.
Is it possible to extract or reuse the system settings to access the web without asking the user for the credentials in Python?
Related
I'm trying to send requests to an HTTPS website behind corporate proxy that requires authentication.
The below code works:
import requests
import urllib
requests.get('https://google.com',
proxies={
'http': 'http://myusername:mypassword#10.20.30.40:8080',
'https': 'http://myusername:mypassword#10.20.30.40:8080'
},
verify=False)
but I want to avoid hardcoding the username and password in the script file, especially that we have to reset our passwords every 60 days. I need it to automatically authenticate as the logged-in Windows user, same behavior as a browser.
I looked online and learned that this is not possible through requests (1, 2, 3, 4) and I would have to resort to something like pycurl or px but all examples online had username and password provided explicitly. There was also this solution utilizing win32com.client but I have no idea how to use this in place of requests.
I am using below python code. But it keep throwing wrong user name or password. Looks like credential are not parsed correctly. But i know credential are correct since it works when i use CURL in DOS command prompt.
import requests as re
import json
re.packages.urllib3.disable_warnings()
url = 'https://nwsppl300p:9090/nwrestapi/v3/global/clients/?q=hostname:BMCJCA001T.corpads.local'
auth = ('cormama.remote\jamal', 'Inventigation100get$pump')
r = re.get(url, auth=auth,verify=False)
print (r.content)
Getting message
b'{"message":"Unauthorized access: The username or password is incorrect","status":{"code":401,"codeClass":"Client Error","reasonPhrase":"Unauthorized"},"timestamp":"2022-06-17T15:00:14-04:00","userAgentRequest":{"headers":[{"name":"Accept","value":"*/*"},{"name":"Accept-Language"},{"name":"Content-Type"}],"method":"GET","query":"q=hostname:BMCJCA001T.corpads.local","url":"https://nwsppl300p:9090/nwrestapi/v3/global/clients/"},"version":"19.5.0.5.Build.154"}'
It seems to me that you are either providing the wrong creds, or perhaps in the wrong format.
Are you able to access your site in a browser using those credentials?
Do you know how to use Fiddler Classic?
You can use Fiddler to capture the call (turn ON HTTPS encryption) when using the browser and capture that call to understand the format needed. note: if you leave fiddler running when debugging; it is a proxy and may interfere with VScode if you are using that to debug...you can use the following to get proxy certs:
os.environ['CURL_CA_BUNDLE'] = ''
The example below requires that I POST a json with my creds in order get my auth token. Your site may be different, but you can use this to figure out what it needs specifically.
in the example shown:
userName = {"email":"someEmail", "password":"somepass"}
auth = re.post(url, json=userName)
I'm not sure how else to describe this. I'm trying to log into a website using the requests library with Python but it doesn't seem to be capturing all cookies from when I login and subsequent requests to the site go back to the login page.
The code I'm using is as follows: (with redactions)
with requests.Session() as s:
r = s.post('https://www.website.co.uk/login', data={
'amember_login': 'username',
'amember_password': 'password'
})
Looking at the developer tools in Chrome. I see the following:
After checking r.cookies it seems only that PHPSESSID was captured there's no sign of the amember_nr cookie.
The value in PyCharm only shows:
{RequestsCookieJar: 1}<RequestsCookieJar[<Cookie PHPSESSID=kjlb0a33jm65o1sjh25ahb23j4 for .website.co.uk/>]>
Why does this code fail to save 'amember_nr' and is there any way to retrieve it?
SOLUTION:
It appears the only way I can get this code to work properly is using Selenium, selecting the elements on the page and automating the typing/clicking. The following code produces the desired result.
from seleniumrequests import Chrome
driver = Chrome()
driver.get('http://www.website.co.uk')
username = driver.find_element_by_xpath("//input[#name='amember_login']")
password = driver.find_element_by_xpath("//input[#name='amember_pass']")
username.send_keys("username")
password.send_keys("password")
driver.find_element_by_xpath("//input[#type='submit']").click() # Page is logged in and all relevant cookies saved.
You can try this:
with requests.Session() as s:
s.get('https://www.website.co.uk/login')
r = s.post('https://www.website.co.uk/login', data={
'amember_login': 'username',
'amember_password': 'password'
})
The get request will set the required cookies.
FYI I would use something like BurpSuite to capture ALL the data being sent to the server and sort out what headers etc are required ... sometimes people/servers to referrer checking, set cookies via JAVA or wonky scripting, even seen java obfuscation and blocking of agent tags not in whitelist etc... it's likely something the headers that the server is missing to give you the cookie.
Also you can have Python use burp as a proxy so you can see exactly what gets sent to the server and the response.
https://github.com/freeload101/Python/blob/master/CS_HIDE/CS_HIDE.py (proxy support )
At work, I sit behind a proxy. When I connect to the company WiFi and open up a browser, a pop-up box usually appears asking for my company credentials before it will let me navigate to any internal/external site.
I am using the Python Requests package to automate pulling data from an external site but am encountering a 401 error that is related to not having authenticated first. This happens when I don't authenticate first using the browser. If I authenticate first with the browser and then use Python requests then everything is fine and I'm able to navigate to any site.
My questions is how do I perform the work authentication part using Python? I want to be able to automate this process so that I can set a cron job that grabs data from an external source every night.
I've tried providing an blank URL:
import requests
response = requests.get('')
But requests.get() requires a properly structure URL. I want to be able to emulate as if I've opened up a browser and capturing the pop-up that asks for authentication. This does not rely on any URL being used.
From the requests documentation
import requests
proxies = {
"http": "http://10.10.1.10:3128",
"https": "http://10.10.1.10:1080",
}
requests.get("http://example.org", proxies=proxies)
I am writing a script to automatically scrape information from my companies directory website using mechanize. However, the interpreter returns _response.httperror_seek_wrapper: HTTP Error 401: Authorization Required onbr.open(url) when I run my script.
This is the portion of my code where the interpreter runs into the error.
from sys import path
path.append("./mechanize/mechanize")
import _mechanize
from base64 import b64encode
def login (url, username, password):
b64login = b64encode('%s:%s' % (username, password))
br = _mechanize.Browser()
br.set_handle_robots(False)
br.addheaders.append(('Authorization','Basic %s' % b64login))
br.open(url)
r = br.response()
print r.read()
The site I am trying to access is an internal site within my companies network, and it uses a GlobalSign Certificate for authentication on company-issued computers.
I am sure the authentication information I am inputting is correct, and I have looked everywhere for a solution. Any hints on how to resolve this? Thanks!
It looks like your authentication methods don't match up. You state that your company uses GlobalSign certificates but your code is using Basic authentication. They are NOT equal!!
From a brief look at the Mechanize documentation (limited as it is), you don't implement authentication by manually adding headers. It has it's own add_password method for handling authentication.
Also, as a general HTTP authentication policy, you should NOT use preemptive authentication by adding the authentication headers yourself. You should set up your code with the necessary authentication (based on your library's documentation) and let it handle the authentication negotiation.