I have an API written in Flask and am testing the endpoints with nosetests using requests to send a request to the API. During the tests, I randomly get an error
ConnectionError: HTTPConnectionPool(host='localhost', port=5555): Max retries exceeded with url: /api (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fe4e794fd50>: Failed to establish a new connection: [Errno 111] Connection refused',))
This error only seems to happen when running tests and randomly affects anywhere between none and all of the tests. All of my tests are being run from one subclass of unittests.TestCase:
class WebServerTests(unittest.TestCase):
# Args to run web server with
server_args = {'port': WEB_SERVER_PORT, 'debug': True}
# Process to run web server
server_process = multiprocessing.Process(
target=les.web_server.run_server, kwargs=server_args)
#classmethod
def setup_class(cls):
"""
Set up testing
"""
# Start server
cls.server_process.start()
#classmethod
def teardown_class(cls):
"""
Clean up after testing
"""
# Kill server
cls.server_process.terminate()
cls.server_process.join()
def test_api_info(self):
"""
Tests /api route that gives information about API
"""
# Test to make sure the web service returns the expected output, which at
# the moment is just the version of the API
url = get_endpoint_url('api')
response = requests.get(url)
assert response.status_code == 200, 'Status Code: {:d}'.format(
response.status_code)
assert response.json() == {
'version': module.__version__}, 'Response: {:s}'.format(response.json())
Everything is happening on localhost and the server is listening on 127.0.0.1. My guess would be that too many requests are being sent to the server and some are being refused, but I'm not seeing anything like that in the debug logs. I had also thought that it may be an issue with the server process not being up before the requests were being made, but the issue persists with a sleep after starting the server process. Another attempt was to let requests attempt retrying the connection by setting requests.adapters.DEFAULT_RETRIES. That didn't work either.
I've tried running the tests on two machines both normally and in docker containers and the issue seems to occur regardless of the platform on which they are run.
Any ideas of what may be causing this and what could be done to fix it?
It turns out that my problem was indeed an issue with the server not having enough time to start up, so the tests would be running before it could respond to tests. I thought I had tried to fix this with a sleep, but had accidentally placed it after creating the process instead of after starting the process. In the end, changing
cls.server_process.start()
to
cls.server_process.start()
time.sleep(1)
fixed the issue.
Related
I have a web process and an api process (a "worker" process similar to what is described in the docs and these other docs). However, I don't understand how the networking between different processes works.
Both processes are listening on 0.0.0.0, and the worker process is bound to port 5000. However, when I try to make a request from the web process, I get a ConnectionRefusedError:
ConnectionError: HTTPConnectionPool(host='0.0.0.0', port=5000): Max retries exceeded with url: /tokenize?sentence=hello (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd307019dc0>: Failed to establish a new connection: [Errno 111] Connection refused'))
Am I supposed to figure out the IP of the other process? Am I doing something wrong here?
Procfile:
api: python app.py
web: voila UI.ipynb --port $PORT --Voila.ip=0.0.0.0
app.py:
from flask import Flask, request
from ie_utils import tokenize
app = Flask(__name__)
#app.route("/")
def home():
return {
"message": "Hello world!",
"version": "0.1",
}
if __name__ == "__main__":
import os
port = 5000
app.run(host="0.0.0.0", port=port)
Relevant code in UI.ipynb:
import requests
requests.get("http://0.0.0.0:5000/")
Full source code: https://github.com/astrojuanlu/ie-titanic-utils-a/tree/test-procfile
I don't understand how the networking between different processes works
Unless you are using Private Spaces (part of Heroku's enterprise offering), it doesn't:
The Common Runtime provides strong isolation by firewalling all dynos off from one another. The only traffic that can reach a dyno is web requests forwarded from the router to web processes listening on the port number specified in the $PORT environment variable. Worker and one-off dynos cannot receive inbound requests.
I'm not totally clear what you're trying to do here, but you won't be able to do it by making an HTTP request to localhost. You might have better luck running this as two separate apps—in that case your API could be its own web process and your Voilà app could make requests to the API app's hostname.
Side note: even on a system where this is permitted your request should not go to 0.0.0.0. That isn't a real IP address.
Telling Flask to bind to 0.0.0.0 really means that it should listen on all available IP addresses. When you want to make a request you should use one of the real hostnames or IP addresses that your system is using, e.g. localhost or 127.0.0.1 or one of its public IP addresses or hostnames.
I am using Python soap API client Zeep and here is the code that I have written:
from zeep import Client
def myapi(request):
client = Client("https://siteURL.asmx?wsdl")
key = client.service.LogOnUser('myusername', 'mypassord')
print(key)
it is giving me an error as: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
While I am trying below command the URL works well and shows all the services it has
python -mzeep https://siteURL.asmx?wsdl
Please help to understand what is the reason above code is not working.
PS: I couldn't share site URL which I am trying to connect to.
Additional Info: The site/page is accessible only through intranet and I am testing locally from intranet itself.
Traceback error:
Exception Type: ConnectionError at /music/mypersonalapi/
Exception Value: HTTPSConnectionPool(host='URL I have hidden', port=81):
Max retries exceeded with url: /ABC/XYZ/Logon.asmx
(Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x0546E770>:
Failed to establish a new connection:
[WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond',))
Please note: I have removed URL and Host information from my traceback due to confidentiality
What this does:
python -mzeep https://site/url.asmx?wsdl
is:
c = Client("https://site/url.asmx?wsdl")
c.wsdl.dump()
both alternatives are using port 443 since that is the default https port.
From your traceback we see
Exception Value: HTTPSConnectionPool(host='URL I have hidden', port=81):
which would have been similar to
python -mzeep https://site:81/url.asmx?wsdl
I.e. the command line and your code are not connecting to the same address (also note that port values less than 1024 requires system level permissions to use -- in case you're writing/controlling the service too).
The last line does say "..failed because the connected party did not properly respond after a period of time..", but that is not the underlying reason. In line 3 you can read
Max retries exceeded with url: /ABC/XYZ/Logon.asmx
in other words, you've tried (and failed) to log on too many times and the server is probably doubling the time it uses to respond every time you try (a well known mitigation strategy for "things" that fail to login multiple times -- i.e. look like an attack). The extended delay is most likely causing the error message you see at the bottom.
You'll need to wait a while, or reset your account for the service, and if the service is yours then perhaps turn off this feature during development?
Maybe this can help. I had the same connexion problem (Max retries exceeded...). I solved it by increasing the transport timeout.
client = Client(wsdl=wsdl, transport=Transport(session=session, timeout=120))
How can I determine when a flask server is ready to receive traffic?
For example, I have server A, inside a dockerfile, that sends a request to an already running server B, announcing its existence upon the container being run.
However once the already running server (B) recieves that request, it tries to send data to one of server A's routes, receiving the error: Failed to establish a new connection: [Errno 111]
Is there a flask or external resource that can send a request in the server when it is ready to receive resulting traffic?
I just pull the server state by sending a simple GET request:
#app.route('/srv_status', methods=['GET'])
def get_srv_status():
return 'OK', 200 # just means we're on air
If it replied, that means the server is ready to process the requests.
Additionally, it can serve for monitoring purposes.
PS: it's been a while after your question. It would be interesting to learn how you've solved the problem.
In a small test project, I'm trying to test a remote proxy server. We have a client that runs on the local machine that talks to the proxy server. I've setup my python code to talk over the client like so:
proxy_handler = urllib.request.ProxyHandler({'http': 'http://localhost:' + self.proxy_port})
opener = urllib.request.build_opener(proxy_handler)
urllib.request.install_opener(opener)
And then I have code that opens the url and reads N bytes at a time:
try:
video_stream = urllib.request.urlopen(self.url, timeout=10)
while something_is_true:
data = video_stream.read(bucket_size)
except URLError:
# handle errors
except Exception:
# handle errors
The problem I'm having is that I can start the test and have it run for several minutes. During that time, I can turn off the remote proxy and the code continues to read from the video_stream connection. No exceptions are thrown.
If I run the code with the remote proxy off from the start, it fails on urlopen(), which I'd expect. But I'd also expect the read() to fail if I stop the remote proxy during the run.
I want to use python to log into a website which uses Microsoft Forefront, and retrieve the content of an internal webpage for processing.
I am not new to python but I have not used any URL libraries.
I checked the following posts:
How can I log into a website using python?
How can I login to a website with Python?
How to use Python to login to a webpage and retrieve cookies for later usage?
Logging in to websites with python
I have also tried a couple of modules such as requests. Still I am unable to understand how this should be done, Is it enough to enter username/password? Or should I somehow use the cookies to authenticate? Any sample code would really be appreciated.
This is the code I have so far:
import requests
NAME = 'XXX'
PASSWORD = 'XXX'
URL = 'https://intra.xxx.se/CookieAuth.dll?GetLogon?curl=Z2F&reason=0&formdir=3'
def main():
# Start a session so we can have persistant cookies
session = requests.session()
# This is the form data that the page sends when logging in
login_data = {
'username': NAME,
'password': PASSWORD,
'SubmitCreds': 'login',
}
# Authenticate
r = session.post(URL, data=login_data)
# Try accessing a page that requires you to be logged in
r = session.get('https://intra.xxx.se/?t=1-2')
print r
main()
but the above code results in the following exception, on the session.post-line:
raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='intra.xxx.se', port=443): Max retries exceeded with url: /CookieAuth.dll?GetLogon?curl=Z2F&reason=0&formdir=3 (Caused by <class 'socket.error'>: [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond)
UPDATE:
I noticed that I was providing wrong username/password.
Once that was updated I get a HTTP-200 response with the above code, but when I try to access any internal site I get a HTTP 401 response. Why Is this happening? What is wrong with the above code? Should I be using the cookies somehow?
TMG can be notoriously fussy about what types of connections it blocks. The next step is to find out why TMG is blocking your connection attempts.
If you have access to the TMG server, log in to it, start the TMG management user-interface (I can't remember what it is called) and have a look at the logs for failed requests coming from your IP address. Hopefully it should tell you why the connection was denied.
It seems you are attempting to connect to it over an intranet. One way I've seen it block connections is if it receives them from an address it considers to be on its 'internal' network. (TMG has two network interfaces as it is intended to be used between two networks: an internal network, whose resources it protects from threats, and an external network, where threats may come from.) If it receives on its external network interface a request that appears to have come from the internal network, it assumes the IP address has been spoofed and blocks the connection. However, I can't be sure that this is the case as I don't know what this TMG server's internal network is set up as nor whether your machine's IP address is on this internal network.