apache - Django stops working with RuntimeError: populate() isn't reentrant

Question

I've been developing a Django web application deployed on an Apache server with WSGI, and everything has been going smoothly. Today, I made some minor changes to my app's admin.py in an attempt to customize the build-in Django Admin interface, and initially made a syntax error (an unclosed parenthesis). This meant that when I touched wsgi.py and loaded the code (I have WSGI running in daemon mode on my virtual host), my website was replaced with an Internal Server Error because WSGI stopped when it hit the syntax error.

So I fixed the syntax error, checked that I didn't have any more with manage.py check, and touched wsgi.py to redeploy. But my website still displays an Internal Server Error! Checking the Apache logs, this is what I see:

[Sun Nov 23 13:52:46 2014] [info] mod_wsgi (pid=19093): Create interpreter 'quotes.cs.cornell.edu|'.
[Sun Nov 23 13:52:46 2014] [info] mod_wsgi (pid=19093): Adding '/extra/www/html/quotes/quotes_django' to path.
[Sun Nov 23 13:52:46 2014] [info] mod_wsgi (pid=19093): Adding '/opt/rh/python27/root/usr/lib64/python2.7/site-
packages/' to path.
[Sun Nov 23 13:52:46 2014] [info] [client 128.84.33.19] mod_wsgi (pid=19093, process='quotes.cs.cornell.edu',
  application='quotes.cs.cornell.edu|'): Loading WSGI script '/extra/www/html/quotes/quotes_django/quotes_django/
wsgi.py'.
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19] mod_wsgi (pid=19093): Target WSGI script '/extra/www/html/
quotes/quotes_django/quotes_django/wsgi.py' cannot be loaded as Python module.
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19] mod_wsgi (pid=19093): Exception occurred processing WSGI
script '/extra/www/html/quotes/quotes_django/quotes_django/wsgi.py'.
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19] Traceback (most recent call last):
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]   File "/extra/www/html/quotes/quotes_django/
quotes_django/wsgi.py", line 14, in <module>
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]     application = get_wsgi_application()
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]   File "/opt/rh/python27/root/usr/lib64/python2.7/site-
packages/django/core/wsgi.py", line 14, in get_wsgi_application
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]     django.setup()
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]   File "/opt/rh/python27/root/usr/lib64/python2.7/site-
packages/django/__init__.py", line 21, in setup
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]     apps.populate(settings.INSTALLED_APPS)
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]   File "/opt/rh/python27/root/usr/lib64/python2.7/site-
packages/django/apps/registry.py", line 115, in populate
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]     app_config.ready()
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]   File "/opt/rh/python27/root/usr/lib64/python2.7/site-
packages/django/contrib/admin/apps.py", line 22, in ready
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]     self.module.autodiscover()
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]   File "/opt/rh/python27/root/usr/lib64/python2.7/site-
packages/django/contrib/admin/__init__.py", line 23, in autodiscover
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]     autodiscover_modules('admin', register_to=site)
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]   File "/opt/rh/python27/root/usr/lib64/python2.7/site-
packages/django/utils/module_loading.py", line 74, in autodiscover_modules
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]     import_module('%s.%s' % (app_config.name,         
module_to_search))
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]   File "/usr/lib64/python2.7/importlib/__init__.py", line 
37, in import_module
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]     __import__(name)
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]   File "/extra/www/html/quotes/quotes_django/quotespage/
admin.py", line 25
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]     approve_quotes.short_description = "Approve selected
quotes"
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]                  ^
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19] SyntaxError: invalid syntax
[Sun Nov 23 13:53:36 2014] [info] [client 128.84.33.19] mod_wsgi (pid=19093, process='quotes.cs.cornell.edu',
  application='quotes.cs.cornell.edu|'): Loading WSGI script '/extra/www/html/quotes/quotes_django/quotes_django/
wsgi.py'.
[Sun Nov 23 13:53:36 2014] [error] [client 128.84.33.19] mod_wsgi (pid=19093): Target WSGI script '/extra/www/html/
quotes/quotes_django/quotes_django/wsgi.py' cannot be loaded as Python module.
[Sun Nov 23 13:53:36 2014] [error] [client 128.84.33.19] mod_wsgi (pid=19093): Exception occurred processing WSGI
script '/extra/www/html/quotes/quotes_django/quotes_django/wsgi.py'.
[Sun Nov 23 13:53:36 2014] [error] [client 128.84.33.19] Traceback (most recent call last):
[Sun Nov 23 13:53:36 2014] [error] [client 128.84.33.19]   File "/extra/www/html/quotes/quotes_django/         
quotes_django/wsgi.py", line 14, in <module>
[Sun Nov 23 13:53:36 2014] [error] [client 128.84.33.19]     application = get_wsgi_application()
[Sun Nov 23 13:53:36 2014] [error] [client 128.84.33.19]   File "/opt/rh/python27/root/usr/lib64/python2.7/site-
packages/django/core/wsgi.py", line 14, in get_wsgi_application
[Sun Nov 23 13:53:36 2014] [error] [client 128.84.33.19]     django.setup()
[Sun Nov 23 13:53:36 2014] [error] [client 128.84.33.19]   File "/opt/rh/python27/root/usr/lib64/python2.7/site-
packages/django/__init__.py", line 21, in setup
[Sun Nov 23 13:53:36 2014] [error] [client 128.84.33.19]     apps.populate(settings.INSTALLED_APPS)
[Sun Nov 23 13:53:36 2014] [error] [client 128.84.33.19]   File "/opt/rh/python27/root/usr/lib64/python2.7/site-
packages/django/apps/registry.py", line 78, in populate
[Sun Nov 23 13:53:36 2014] [error] [client 128.84.33.19]     raise RuntimeError("populate() isn't reentrant")
[Sun Nov 23 13:53:36 2014] [error] [client 128.84.33.19] RuntimeError: populate() isn't reentrant

The first series of errors shows WSGI failing due to the syntax error in my admin.py. However, the second series of errors seems to show an error internal to Django:

RuntimeError: populate() isn't reentrant

thrown from the populate method of registry.py.

Googling this error message returns surprisingly little information, none of it from Django documentation. Apparently, it can sometimes happen if you name an app twice in your settings.py, but I'm not doing that. More importantly, I haven't changed settings.py since the point where the website was working fine -- the only thing I changed was admin.py.

I tried reverting all the changes I made, so all my Python code is back in the state it was when the website was working -- and I still get the populate() isn't reentrant error when I try to make WSGI reload the code!

I've also tried commenting-out different apps in the INSTALLED_APPS section of settings.py, and even with only 'django.contrib.staticfiles' enabled the error still happens. Weirdly, I still get the error even if I comment out all the apps -- Django throws the error even when it isn't loading any apps!

Does anyone know what's going on here? Or any better way for me to debug this error, since the traceback in the Apache log is pretty unhelpful?

Notes: I'm using Django 1.7, Apache 2.2, and Python 2.7.

Answer 1

This is caused by a bug in your Django settings somewhere. Unfortunately, Django's hiding the bug behind this generic and un-useful error message.

To reveal the true problem, open django/apps/registry.py and around line 80, replace:

raise RuntimeError("populate() isn't reentrant")

with:

self.app_configs = {}

This will allow Django to continue loading, and reveal the actual error.

I've encountered this error for several different causes. Once was because I had a bad import in one of my app's admin.py.