Develop Reference

django.db.utils.InterfaceError: connection already closed failures when updating to Django 3.0

I am updating a medium-sized project to Django 3.0 and I am encountering several errors in my tests after doing nothing more than bumping the Django version from 2.3.
The whole test suite has been running correctly for years and I couldn't find any relevant change in the changelog that may point to the cause of this issue. Apparently a single test fail is triggering every remaining test in the same TestCase class to fail with the following exception:
Traceback (most recent call last):
File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/db/backends/base/base.py", line 238, in _cursor
return self._prepare_cursor(self.create_cursor(name))
File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/utils/asyncio.py", line 24, in inner
return func(*args, **kwargs)
File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/db/backends/postgresql/base.py", line 231, in create_cursor
cursor = self.connection.cursor()
psycopg2.InterfaceError: connection already closed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/federicobond/code/forks/core/apps/participants/tests/test_views.py", line 40, in setUp
self.client.force_login(self.user)
File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/test/client.py", line 602, in force_login
self._login(user, backend)
File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/test/client.py", line 611, in _login
if self.session:
File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/test/client.py", line 461, in session
session.save()
File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/contrib/sessions/backends/db.py", line 81, in save
return self.create()
File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/contrib/sessions/backends/db.py", line 51, in create
self._session_key = self._get_new_session_key()
File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/contrib/sessions/backends/base.py", line 162, in _get_new_session_key
if not self.exists(session_key):
File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/contrib/sessions/backends/db.py", line 47, in exists
return self.model.objects.filter(session_key=session_key).exists()
File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/db/models/query.py", line 777, in exists
return self.query.has_results(using=self.db)
File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/db/models/sql/query.py", line 534, in has_results
return compiler.has_results()
File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1107, in has_results
return bool(self.execute_sql(SINGLE))
File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1135, in execute_sql
cursor = self.connection.cursor()
File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/utils/asyncio.py", line 24, in inner
return func(*args, **kwargs)
File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/db/backends/base/base.py", line 260, in cursor
return self._cursor()
File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/db/backends/base/base.py", line 238, in _cursor
return self._prepare_cursor(self.create_cursor(name))
File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/db/utils.py", line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/db/backends/base/base.py", line 238, in _cursor
return self._prepare_cursor(self.create_cursor(name))
File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/utils/asyncio.py", line 24, in inner
return func(*args, **kwargs)
File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/db/backends/postgresql/base.py", line 231, in create_cursor
cursor = self.connection.cursor()
django.db.utils.InterfaceError: connection already closed
I am out of ideas as to what could be going on here.

I ran into this as well. It appears to be a bug in pytest-django. Here's the relevant issue. There's an open PR to resolve it. If it's a big enough inconvenience you can use the branch in that PR or pin your dependencies to an earlier version.

We were hitting the same issue, and upgrading from Django 3.0.2 to Django-3.0.4 resolved it. There are several DB related fixes in those two versions, but I don't know which one solved our problem.

Just a forward note, it's near impossible to provide more information than already is in the stacktrace. However, you can investigate:
Check when the connection get closed and by which test (run them individually via script for example).
For tests that fail, check the code for deprecated parts of Django (searching for things that were removed/deprecated between 2.3 and 3.0).
Run a linter to see if someone changed a private variable inside of Django framework as workaround.
Check the transactions of postgres.
Then once you have which part of the code has the error narrow it down by creating smaller failing tests.

I had the same issue using pytest.
Downgrading from 5.4.1 to 5.3.5 fixed it.

All these issues occur because of the incompatibility of the other packages with django 3.0
when i ran into this error i updated my requirement.txt file manually and then intalled all the requirements using pip in the same env.

pyppeteer.errors.BrowserError: Failed to connect to browser port

I have problems when using requests-html package on Python 3.6.5, Ubuntu 16.04(x64). To be more specific, the last line of
from requests_html import HTMLSession
session = HTMLSession()
r = session.get('http://python-requests.org/')
r.html.render()
produces the following error:
Traceback (most recent call last): File "", line 1, in
File
"/home/candy/.conda/envs/candy_env/lib/python3.6/site-packages/requests_html.py",
line 572, in render
self.session.browser # Automatycally create a event loop and browser
File
"/home/candy/.conda/envs/candy_env/lib/python3.6/site-packages/requests_html.py",
line 680, in browser
self._browser = self.loop.run_until_complete(pyppeteer.launch(headless=True,
args=['--no-sandbox']))
File
"/home/candy/.conda/envs/candy_env/lib/python3.6/asyncio/base_events.py",
line 468, in run_until_complete
return future.result()
File
"/home/candy/.conda/envs/candy_env/lib/python3.6/site-packages/pyppeteer/launcher.py",
line 243, in launch
return await Launcher(options, **kwargs).launch()
File
"/home/candy/.conda/envs/candy_env/lib/python3.6/site-packages/pyppeteer/launcher.py",
line 160, in launch
self.browserWSEndpoint = self._get_ws_endpoint()
File
"/home/candy/.conda/envs/candy_env/lib/python3.6/site-packages/pyppeteer/launcher.py",
line 178, in _get_ws_endpoint
raise BrowserError(f'Failed to connect to browser port: {url}')
pyppeteer.errors.BrowserError: Failed to connect to browser port:
http://127.0.0.1:43623/json/version
However, the same code works well without errors on another Windows 10 platform, with the same Python requirements configured.
I have checked whether a Chrome has been downloaded successfully on my computer and the result is yes! So I think that's not where the problem is.
(candy_env)
candy#botwriter01:~/.pyppeteer/local-chromium/543305/chrome-linux$ ls
chrome
chrome_sandbox libclearkeycdm.so locales nacl_helper_bootstrap natives_blob.bin resources.pak
xdg-mime chrome_100_percent.pak chrome-wrapper libEGL.so
MEIPreload nacl_helper_nonsfi product_logo_48.png swiftshader
xdg-settings chrome_200_percent.pak icudtl.dat libGLESv2.so
nacl_helper nacl_irt_x86_64.nexe resources
v8_context_snapshot.bin
I have already searched the guidebook of requests-html for answers but got nothing found. I want the command r.html.render() to work correctly, what can I do now?

Python script suddenly throwing timeout exceptions

I have a Python script that downloads product feeds from multiple affiliates in different ways. This didn't give me any problems until last Wednesday, when it started throwing all kinds of timeout exceptions from different locations.
Examples: Here I connect with a FTP service:
ftp = FTP(host=self.host)
threw:
Exception in thread Thread-7:
Traceback (most recent call last):
File "C:\Python27\Lib\threading.py", line 810, in __bootstrap_inner
self.run()
File "C:\Python27\Lib\threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "C:\Users\Administrator\Documents\Crawler\src\Crawlers\LDLC.py", line 23, in main
ftp = FTP(host=self.host)
File "C:\Python27\Lib\ftplib.py", line 120, in __init__
self.connect(host)
File "C:\Python27\Lib\ftplib.py", line 138, in connect
self.welcome = self.getresp()
File "C:\Python27\Lib\ftplib.py", line 215, in getresp
resp = self.getmultiline()
File "C:\Python27\Lib\ftplib.py", line 201, in getmultiline
line = self.getline()
File "C:\Python27\Lib\ftplib.py", line 186, in getline
line = self.file.readline(self.maxline + 1)
File "C:\Python27\Lib\socket.py", line 476, in readline
data = self._sock.recv(self._rbufsize)
timeout: timed out
Or downloading an XML File :
xmlFile = urllib.URLopener()
xmlFile.retrieve(url, self.feedPath + affiliate + "/" + website + '.' + fileType)
xmlFile.close()
throws:
File "C:\Users\Administrator\Documents\Crawler\src\Crawlers\FeedCrawler.py", line 106, in save
xmlFile.retrieve(url, self.feedPath + affiliate + "/" + website + '.' + fileType)
File "C:\Python27\Lib\urllib.py", line 240, in retrieve
fp = self.open(url, data)
File "C:\Python27\Lib\urllib.py", line 208, in open
return getattr(self, name)(url)
File "C:\Python27\Lib\urllib.py", line 346, in open_http
errcode, errmsg, headers = h.getreply()
File "C:\Python27\Lib\httplib.py", line 1139, in getreply
response = self._conn.getresponse()
File "C:\Python27\Lib\httplib.py", line 1067, in getresponse
response.begin()
File "C:\Python27\Lib\httplib.py", line 409, in begin
version, status, reason = self._read_status()
File "C:\Python27\Lib\httplib.py", line 365, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File "C:\Python27\Lib\socket.py", line 476, in readline
data = self._sock.recv(self._rbufsize)
IOError: [Errno socket error] timed out
These are just two examples but there are other methods, like authenticate or other API specific methods where my script throws these timeout errors. It never showed this behavior until Wednesday. Also, it starts throwing them at random times. Sometimes at the beginning of the crawl, sometimes later on. My script has this behavior on both my server and my local machine. I've been struggling with it for two days now but can't seem to figure it out.
This is what I know might have caused this:
On Wednesday one affiliate script broke down with the following error:
URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>
I didn't change anything to my script but suddenly it stopped crawling that affiliate and threw that error all the time where I tried to authenticate. I looked it up and found that is was due to an OpenSSL error (where did that come from). I fixed it by adding the following before the authenticate method:
if hasattr(ssl, '_create_unverified_context'):
ssl._create_default_https_context = ssl._create_unverified_context
Little did I know, this was just the start of my problems... At that same time, I changed from Python 2.7.8 to Python 2.7.9. It seems that this is the moment that everything broke down and started throwing timeouts.
I tried changing my script in endless ways but nothing worked and like I said, it's not just one method that throws it. Also I switched back to Python 2.7.8, but this didn't do the trick either. Basically everything that makes a request to an external source can throw an error.
Final note: My script is multi threaded. It downloads product feeds from different affiliates at the same time. It used to run 10 threads per affiliate without a problem. Now I tried lowering it to 3 per affiliate, but it still throws these errors. Setting it to 1 is no option because that will take ages. I don't think that's the problem anyway because it used to work fine.
What could be wrong?

Can't run PhantomJS in python via Selenium

I have been trying to run PhantomJS via selenium for past 3 days and have had no success.
So far i have tried installing PhantomJS via npm, building it from source, installing via apt-get and downloading prebuilt executable and placing it in /usr/bin/phantomjs.
Every time I was able to run this example script loadspeed.js :
var page = require('webpage').create(),
system = require('system'),
t, address;
if (system.args.length === 1) {
console.log('Usage: loadspeed.js <some URL>');
phantom.exit();
}
t = Date.now();
address = system.args[1];
page.open(address, function (status) {
if (status !== 'success') {
console.log('FAIL to load the address');
} else {
t = Date.now() - t;
console.log('Loading time ' + t + ' msec');
}
phantom.exit();
});
and run it with 'phantomjs test.js http://google.com' and it worked just as it should.
but running PhantomJS via selenium in this small python script produces errors:
from selenium import webdriver
browser = webdriver.PhantomJS()
browser.get('http://seleniumhq.org')
python test.py
Traceback (most recent call last):
File "test.py", line 4, in <module>
browser.get('http://seleniumhq.org/')
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 176, in get
self.execute(Command.GET, {'url': url})
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 162, in execute
response = self.command_executor.execute(driver_command, params)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 350, in execute
return self._request(url, method=command_info[0], data=data)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 382, in _request
resp = self._conn.getresponse()
File "/usr/lib/python2.7/httplib.py", line 1045, in getresponse
response.begin()
File "/usr/lib/python2.7/httplib.py", line 409, in begin
version, status, reason = self._read_status()
File "/usr/lib/python2.7/httplib.py", line 373, in _read_status
raise BadStatusLine(line)
httplib.BadStatusLine: ''
Replacing second LOC with browser = webdriver.Firefox() works fine.
I am on Ubuntu 13.10 desktop and same error occurs on Ubuntu 13.04 aswell.
Python: 2.7
PhantomJS: 1.9.2
What am I doing wrong here?

There seems to be some issue introduced in newer Selenium, see
http://code.google.com/p/selenium/issues/detail?id=6690
I got a bit further using by using
pip install selenium==2.37
Avoids the stack trace above. Still having problems with driver.save_screenshot('foo.png') resulting in an empty file though.

Selenium in Python

I've been using urllib2 to access webpages, but it doesn't support javascript, so I took a look at Selenium, but I'm quite confused even having read its docs.
I downloaded Selenium IDE add-on for firefox and I tried some simple things.
from selenium import selenium
import unittest, time, re
class test(unittest.TestCase):
def setUp(self):
self.verificationErrors = []
self.selenium = selenium("localhost", 4444, "*chrome", "http://www.wikipedia.org/")
self.selenium.start()
def test_test(self):
sel = self.selenium
sel.open("/")
sel.type("searchInput", "pacific ocean")
sel.click("go")
sel.wait_for_page_to_load("30000")
def tearDown(self):
self.selenium.stop()
self.assertEqual([], self.verificationErrors)
if __name__ == "__main__":
unittest.main()
I just access wikipedia.org and type pacific ocean in the search field, but when I try to compile it, it gives me a lot of errors.

If running the script results in a [Errno 111] Connection refused error such as this:
% test.py
E
======================================================================
ERROR: test_test (__main__.test)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/unutbu/pybin/test.py", line 11, in setUp
self.selenium.start()
File "/data1/unutbu/pybin/selenium.py", line 189, in start
result = self.get_string("getNewBrowserSession", [self.browserStartCommand, self.browserURL, self.extensionJs])
File "/data1/unutbu/pybin/selenium.py", line 219, in get_string
result = self.do_command(verb, args)
File "/data1/unutbu/pybin/selenium.py", line 207, in do_command
conn.request("POST", "/selenium-server/driver/", body, headers)
File "/usr/lib/python2.6/httplib.py", line 898, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python2.6/httplib.py", line 935, in _send_request
self.endheaders()
File "/usr/lib/python2.6/httplib.py", line 892, in endheaders
self._send_output()
File "/usr/lib/python2.6/httplib.py", line 764, in _send_output
self.send(msg)
File "/usr/lib/python2.6/httplib.py", line 723, in send
self.connect()
File "/usr/lib/python2.6/httplib.py", line 704, in connect
self.timeout)
File "/usr/lib/python2.6/socket.py", line 514, in create_connection
raise error, msg
error: [Errno 111] Connection refused
----------------------------------------------------------------------
Ran 1 test in 0.063s
FAILED (errors=1)
then the solution is most likely that you need get the selenium server running first.
In the download for SeleniumRC you will find a file called selenium-server.jar (as of a few months ago, that file was located at SeleniumRC/selenium-server-1.0.3/selenium-server.jar).
On Linux, you could run the selenium server in the background with the command
java -jar /path/to/selenium-server.jar 2>/dev/null 1>&2 &
You will find more complete instructions on how to set up the server here.

I would suggest you to use a webdriver, you can find it here: http://code.google.com/p/selenium/downloads/list. If you want to write tests as a coder (and not with the use of your mouse), that thing would work better then the RC version you're trying to use, at least because it would not ask you for an SeleniumRC Jar Instance. You would simply have a binary of a browser or use those ones that are already installed on your system, for example, Firefox.

I faced with this issue in my project and found that problem was in few webdriver.get calls with very small time interval between them. My fix was not to put delay, just remove unneeded calls and error disappears.
Hope, it can helps for somebody.