I want to use python to log into a website which uses Microsoft Forefront, and retrieve the content of an internal webpage for processing.
I am not new to python but I have not used any URL libraries.
I checked the following posts:
How can I log into a website using python?
How can I login to a website with Python?
How to use Python to login to a webpage and retrieve cookies for later usage?
Logging in to websites with python
I have also tried a couple of modules such as requests. Still I am unable to understand how this should be done, Is it enough to enter username/password? Or should I somehow use the cookies to authenticate? Any sample code would really be appreciated.
This is the code I have so far:
import requests
NAME = 'XXX'
PASSWORD = 'XXX'
URL = 'https://intra.xxx.se/CookieAuth.dll?GetLogon?curl=Z2F&reason=0&formdir=3'
def main():
# Start a session so we can have persistant cookies
session = requests.session()
# This is the form data that the page sends when logging in
login_data = {
'username': NAME,
'password': PASSWORD,
'SubmitCreds': 'login',
}
# Authenticate
r = session.post(URL, data=login_data)
# Try accessing a page that requires you to be logged in
r = session.get('https://intra.xxx.se/?t=1-2')
print r
main()
but the above code results in the following exception, on the session.post-line:
raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='intra.xxx.se', port=443): Max retries exceeded with url: /CookieAuth.dll?GetLogon?curl=Z2F&reason=0&formdir=3 (Caused by <class 'socket.error'>: [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond)
UPDATE:
I noticed that I was providing wrong username/password.
Once that was updated I get a HTTP-200 response with the above code, but when I try to access any internal site I get a HTTP 401 response. Why Is this happening? What is wrong with the above code? Should I be using the cookies somehow?
TMG can be notoriously fussy about what types of connections it blocks. The next step is to find out why TMG is blocking your connection attempts.
If you have access to the TMG server, log in to it, start the TMG management user-interface (I can't remember what it is called) and have a look at the logs for failed requests coming from your IP address. Hopefully it should tell you why the connection was denied.
It seems you are attempting to connect to it over an intranet. One way I've seen it block connections is if it receives them from an address it considers to be on its 'internal' network. (TMG has two network interfaces as it is intended to be used between two networks: an internal network, whose resources it protects from threats, and an external network, where threats may come from.) If it receives on its external network interface a request that appears to have come from the internal network, it assumes the IP address has been spoofed and blocks the connection. However, I can't be sure that this is the case as I don't know what this TMG server's internal network is set up as nor whether your machine's IP address is on this internal network.
Related
I am using Python soap API client Zeep and here is the code that I have written:
from zeep import Client
def myapi(request):
client = Client("https://siteURL.asmx?wsdl")
key = client.service.LogOnUser('myusername', 'mypassord')
print(key)
it is giving me an error as: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
While I am trying below command the URL works well and shows all the services it has
python -mzeep https://siteURL.asmx?wsdl
Please help to understand what is the reason above code is not working.
PS: I couldn't share site URL which I am trying to connect to.
Additional Info: The site/page is accessible only through intranet and I am testing locally from intranet itself.
Traceback error:
Exception Type: ConnectionError at /music/mypersonalapi/
Exception Value: HTTPSConnectionPool(host='URL I have hidden', port=81):
Max retries exceeded with url: /ABC/XYZ/Logon.asmx
(Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x0546E770>:
Failed to establish a new connection:
[WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond',))
Please note: I have removed URL and Host information from my traceback due to confidentiality
What this does:
python -mzeep https://site/url.asmx?wsdl
is:
c = Client("https://site/url.asmx?wsdl")
c.wsdl.dump()
both alternatives are using port 443 since that is the default https port.
From your traceback we see
Exception Value: HTTPSConnectionPool(host='URL I have hidden', port=81):
which would have been similar to
python -mzeep https://site:81/url.asmx?wsdl
I.e. the command line and your code are not connecting to the same address (also note that port values less than 1024 requires system level permissions to use -- in case you're writing/controlling the service too).
The last line does say "..failed because the connected party did not properly respond after a period of time..", but that is not the underlying reason. In line 3 you can read
Max retries exceeded with url: /ABC/XYZ/Logon.asmx
in other words, you've tried (and failed) to log on too many times and the server is probably doubling the time it uses to respond every time you try (a well known mitigation strategy for "things" that fail to login multiple times -- i.e. look like an attack). The extended delay is most likely causing the error message you see at the bottom.
You'll need to wait a while, or reset your account for the service, and if the service is yours then perhaps turn off this feature during development?
Maybe this can help. I had the same connexion problem (Max retries exceeded...). I solved it by increasing the transport timeout.
client = Client(wsdl=wsdl, transport=Transport(session=session, timeout=120))
I've been running an instance of Django on Windows R2 2012 for over a year and I've come to a road block. Yesterday something happened, I don't know what it could be. The same two errors keep popping up at different times though when trying to send an email:
[WinError 10060] A connection attempt failed because the connected
party did not properly respond after a period of time, or established
connection failed because connected host has failed to respond
and
socket.gaierror: [Errno 11001] getaddrinfo failed
Users are able to connect to the IP address of the server and the port Django is running on 192.168.1.5:8000, but they cannot send emails anymore. Though a percentage do go through as described here, but very few.
Things I've tried
1) This solution
import socket
socket.getaddrinfo('localhost', 8000)
Since I'm doing python manage.py runserver 192.168.1.5:8000, I added that IP and nothing.
2) I went into the Firewall settings and made sure that the ports were all good. The SMTP one that is declared in the setting.py file in my Django project and 25. All of them, inbound and out.
3) I tried sending things on my local machine and it does work. I used other programs that do not use Django to send emails and they do process on all other machines except the Server. So I know it's not my email server.
4) I changed the email config to use my Gmail account and it does process on all other machines except for the server. So it has to be the environment.
5) Editing http_proxy environment variables
The problem, in my case, was that some install at some point defined
an
environment variable http_proxy on my machine when I had no proxy.
Removing the http_proxy environment variable fixed the problem.
As described here
and in my Django project in the wsgi.y file:
os.environ['http_proxy'] = "http://192.168.1.5:8080"
os.environ['https_proxy'] = "http://192.168.1.5:8080"
6) Given this answer here (can someone please explain how I would do it to a django email function), I've also tried this method of wrapping it from solutions here
import smtplib
import socks
#socks.setdefaultproxy(TYPE, ADDR, PORT)
socks.setdefaultproxy(socks.SOCKS5, '192.168.1.5', 8080)
socks.wrapmodule(smtplib)
smtpserver = 'smtp.live.com'
AUTHREQUIRED = 1
smtpuser = 'example#hotmail.fr'
smtppass = 'mypassword'
RECIPIENTS = 'mailto#gmail.com'
SENDER = 'example#hotmail.fr'
mssg = "test message"
s = mssg
server = smtplib.SMTP(smtpserver,587)
server.ehlo()
server.starttls()
server.ehlo()
server.login(smtpuser,smtppass)
server.set_debuglevel(1)
server.sendmail(SENDER, [RECIPIENTS], s)
server.quit()
Though I wouldn't like to use such a method as I'd prefer using Django's built in email service.
Since you have not changed the code and errors you shared shows that it's a network related problem.
It's most probably a DNS issue. In your settings.py you have specified the EMAIL_HOST, which is i believe a hostname. You need to check you server's DNS server.
You are mentioning about checking your firewall settings but what you are doing wrong is not checking the actual connection.
To address the problem you can use couple of command line utilities like telnet or nslookup. You can check if you can resolve a hostname:
nslookup smptp.mail_host.com
This command will fail most probably.
I would like to point what you did wrong in your steps:
1) You have tried to get your services getaddrinfo in which you needed to put your smtp servers hostname, which would result with the same error. Sockets are the very primitive part of the connection on the application layer, you don't really need to dig in to that.
2) Checking firewall settings is OK.
3) This is a good step which shows that there is a problem with your servers network connection.
4) That is another evidence :)
5) You have got it wrong, if you have a proxy server on your network to connect external networks, than you use this settings. But you have configured it wrong. You should not set your projects url as proxy server.
6) This is another deep level coding. You should not use such low level script, which will cause you numerious problems, which would have been handled in high level modules.
I focused my answer on the strange fact that you can get around the problem using a SOCKS5 proxy. (I believe you. There was no time to ask you for details.) You verified that your example solution by SOCKS5 works for you. Django uses the same smtplib and you can easily wrap it the same way by this code added to wsgi.py.
import smtplib
import socks # it is the package SocksiPy or PySocks
socks.setdefaultproxy(socks.SOCKS5, '192.168.1.5', 8080)
socks.wrapmodule(smtplib)
Http(s) proxy (paragraph 5)) is not related because it does not affect SMTP and other protocols except http(s) because "SOCKS operates at a lower level than HTTP proxying".
I have an API written in Flask and am testing the endpoints with nosetests using requests to send a request to the API. During the tests, I randomly get an error
ConnectionError: HTTPConnectionPool(host='localhost', port=5555): Max retries exceeded with url: /api (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fe4e794fd50>: Failed to establish a new connection: [Errno 111] Connection refused',))
This error only seems to happen when running tests and randomly affects anywhere between none and all of the tests. All of my tests are being run from one subclass of unittests.TestCase:
class WebServerTests(unittest.TestCase):
# Args to run web server with
server_args = {'port': WEB_SERVER_PORT, 'debug': True}
# Process to run web server
server_process = multiprocessing.Process(
target=les.web_server.run_server, kwargs=server_args)
#classmethod
def setup_class(cls):
"""
Set up testing
"""
# Start server
cls.server_process.start()
#classmethod
def teardown_class(cls):
"""
Clean up after testing
"""
# Kill server
cls.server_process.terminate()
cls.server_process.join()
def test_api_info(self):
"""
Tests /api route that gives information about API
"""
# Test to make sure the web service returns the expected output, which at
# the moment is just the version of the API
url = get_endpoint_url('api')
response = requests.get(url)
assert response.status_code == 200, 'Status Code: {:d}'.format(
response.status_code)
assert response.json() == {
'version': module.__version__}, 'Response: {:s}'.format(response.json())
Everything is happening on localhost and the server is listening on 127.0.0.1. My guess would be that too many requests are being sent to the server and some are being refused, but I'm not seeing anything like that in the debug logs. I had also thought that it may be an issue with the server process not being up before the requests were being made, but the issue persists with a sleep after starting the server process. Another attempt was to let requests attempt retrying the connection by setting requests.adapters.DEFAULT_RETRIES. That didn't work either.
I've tried running the tests on two machines both normally and in docker containers and the issue seems to occur regardless of the platform on which they are run.
Any ideas of what may be causing this and what could be done to fix it?
It turns out that my problem was indeed an issue with the server not having enough time to start up, so the tests would be running before it could respond to tests. I thought I had tried to fix this with a sleep, but had accidentally placed it after creating the process instead of after starting the process. In the end, changing
cls.server_process.start()
to
cls.server_process.start()
time.sleep(1)
fixed the issue.
I would be very grateful if anyone could help me with this issue I am having.
I’m trying to use the request lib to download a .gz file from the internet. I have successfully used the lib before to get xml data that is parsed to the browser, but the .gz version is not working.
Once the URL_To_Gzip link is clicked in my browser, the .gz file automatically starts to download the file. --> so the url is ok, but just points directly to the file.
I’m trying to code this in python 2.7 so I can then process the file and data it contains, but I get an error message that I am struggling to resolve.
Error Message:
HTTPSConnectionPool(host=HOST_URL_TO_GZip, port=443): Max retries exceeded with url: URL_TO_GZip.gz (Caused by : [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond)
import requests
data = requests.get(url_to_gzip,proxies = {"http":proxy_url}) # Does not work data = #Does not work
data = requests.get(url_to_gzip,proxies = {"http":proxy_url}, stream = True) # Does not work
The information on Errno 10060 suggests the error is related to my proxy, as a connection can not be established. --> But I have successfully used these to get the xml data in a similar version.
Thanks,
Ravi
EDIT
The URL_TO_GZip.gz file is via a https:// whereas the xml file that works ok is via a http:// which I think is the cause of my problem and why it works for one file but not another.
For anyone else that comes across this issue, I needed to add a auth = (username, password) keyword to access the HTTPS site auth keyword.
I am currently using Python + Mechanize for retrieving pages from a local server. As you can see the code uses "localhost" as a proxy. The proxy is an instance of the Fiddler2 debug proxy. This works exactly as expected. This indicates that my machine can reach the test_box.
import time
import mechanize
url = r'http://test_box.test_domain.com:8000/helloWorldTest.html'
browser = mechanize.Browser();
browser.set_proxies({"http": "127.0.0.1:8888"})
browser.add_password(url, "test", "test1234")
start_timer = time.time()
resp = browser.open(url)
resp.read()
latency = time.time() - start_timer
However when I remove the browser.set_proxies statement it stops to work. I get an error <"urlopen error [Errno 10061] No connection could be made because the target machine actively refused it>". The point is that I can access the test_box from my machine with any browser. This also indicates that test_box can be reached from my machine.
My suspicion is that this has something to do with Mechanize trying to guess the proper proxy settings. That is: my Browsers are configured to go to a web proxy for any domain but test_domain.com. So I suspect that mechanize tries to use the web proxy while it should actually not use the proxy.
How can I tell mechanize to NOT guess any proxy settings and instead force it to try to connect directly to the test_box?
Argh, found it out myself. The docstring says:
"To avoid all use of proxies, pass an empty proxies dict."
This fixed the issue.