Django giving 500 instead of 404 for unknown URLs when debug is false

Django = 2.1.x

Python = 3.7.x

If Debug is True - it returns a 404.

If Debug is False - it gives a 500 error.

My project.urls file looks like this:

urlpatterns = [
    path("admin/", admin.site.urls),
    path("", app1.views.log_in, name="log_in"),
    path("log_in/", app1.views.log_in, name="log_in"),
    path("logout/", app1.views.log_out, name="logout"),
    path("launcher/", app1.views.launcher, name="launcher"),
    path("app2/", include("app2.urls")),
    path("app3/", include("app3.urls")),
]

My directory structure looks like this:

Project_directory
    static_directory
        ...js files and css files and such...
    templates_directory
        400.html
        403.html
        404.html
        500.html
        base.html (all apps extend this page, which works great)
    project_directory
        urls.py
        settings.py
        ...other files...
    app1_directory
        views.py
        models.py
        templates_directory
            app1
                ...template files...
        ...other app1 files/directories...
    app2_directory
        ...app2 directories and files...
    app3_directory
        ...app3 directories and files...

When I python manage.py runserver and I hit a URL I know doesn't exist (like http://project/randomtrash.php) it gives an appropriate 404 if DEBUG = True

If DEBUG = False then hitting that same URL will give a 500 and the 500.html displays.

Important parts of my settings.py look like this:

# These two settings are only for testing purposes and are different
# In production
DEBUG = False
ALLOWED_HOSTS = ["*"]

ROOT_URLCONF = "project.urls"

TEMPLATES = [
    {
        "BACKEND": "django.template.backends.django.DjangoTemplates",
        # DIRS lets the apps extend base.html
        "DIRS": [os.path.join(BASE_DIR, "templates")],
        "APP_DIRS": True,
        "OPTIONS": {
            "context_processors": [
                "django.template.context_processors.debug",
                "django.template.context_processors.request",
                "django.contrib.auth.context_processors.auth",
                "django.contrib.messages.context_processors.messages",
                "project.context_processors.app_context",
                "project.context_processors.registrations",
            ]
        },
    }
]

STATIC_URL = "/static/"

STATIC_ROOT = os.path.join(os.path.dirname(BASE_DIR), "static")

SESSION_EXPIRE_AT_BROWSER_CLOSE = True

if DEBUG:
    STATICFILES_DIRS = [os.path.join(BASE_DIR, "static")]