django: Use valid identifiers for application labels

While I was locally playing to change swh-web database backend from SQLite to PostgreSQL, I stumbled across that error while trying to load a JSON dump of the webapp database into the new Postgres one.

17:25 $ django-admin loaddata datadump.json --settings=swh.web.settings.development
[08/Jan/2021 16:26:07] [DEBUG] urllib3.util.retry.from_int:332 - Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
[08/Jan/2021 16:26:07] [DEBUG] urllib3.util.retry.from_int:332 - Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
Traceback (most recent call last):
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/django/core/serializers/json.py", line 69, in Deserializer
    yield from PythonDeserializer(objects, **options)
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/django/core/serializers/python.py", line 92, in Deserializer
    Model = _get_model(d["model"])
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/django/core/serializers/python.py", line 154, in _get_model
    return apps.get_model(model_identifier)
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/django/apps/registry.py", line 206, in get_model
    app_label, model_name = app_label.split(".")
ValueError: too many values to unpack (expected 2)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/anlambert/.virtualenvs/swh/bin/django-admin", line 8, in <module>
    sys.exit(execute_from_command_line())
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/django/core/management/commands/loaddata.py", line 72, in handle
    self.loaddata(fixture_labels)
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/django/core/management/commands/loaddata.py", line 114, in loaddata
    self.load_label(fixture_label)
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/django/core/management/commands/loaddata.py", line 172, in load_label
    for obj in objects:
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/django/core/serializers/json.py", line 73, in Deserializer
    raise DeserializationError() from exc
django.core.serializers.base.DeserializationError: Problem installing fixture '/home/anlambert/swh/swh-environment/swh-web/datadump.json':

This is due to the fact that swh-web declares the following labels for some custom django applications: swh.web.common, swh.web.auth.

But Django application label must be a valid Python identifier and thus must not contain dot characters.

As a result, having invalid identifiers will make database data importing from JSON fails.

This was not clearly indicated in Django documentation and application label validity check has been added a couple of weeks ago in upstream Django source code. https://code.djangoproject.com/ticket/32285

That diff renames the invalid django application labels and ensure migrations can still be applied.

In order to apply the renaming changes to the swh-web instance in production, the migration command will have to be the following:

django-admin migrate --settings=swh.web.settings.production --fake

This will simulate the migrations but without any effects apart filling the django migrations table with the new application labels.

Calling migrate without the fake option will raise errors otherwise as the changes have already been applied to database, see below.

17:38 $ django-admin migrate --settings=swh.web.settings.development
[08/Jan/2021 16:38:12] [DEBUG] urllib3.util.retry.from_int:332 - Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
[08/Jan/2021 16:38:12] [DEBUG] urllib3.util.retry.from_int:332 - Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions, swh_web_auth, swh_web_common
Running migrations:
  Applying swh_web_auth.0001_initial...Traceback (most recent call last):
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/django/db/backends/utils.py", line 82, in _execute
    return self.cursor.execute(sql)
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/django/db/backends/sqlite3/base.py", line 381, in execute
    return Database.Cursor.execute(self, query)
sqlite3.OperationalError: table "oidc_user_offline_tokens" already exists

This could be implemented the following way in debian/postinst:

#!/bin/bash

set -e

if [ -f /var/lib/swh/web.sqlite3 ]; then
    echo "Software Heritage production db detected; running migrations"
    if [ -d /usr/lib/python3/dist-packages/swh.web-0.0.279.egg-info/ ]
    then
        # use fake migrations after django app labels renaming in v0.0.279
        django-admin migrate --settings=swh.web.settings.production --fake
    else
        django-admin migrate --settings=swh.web.settings.production
    fi
fi

Related to swh/infra/sysadm-environment#2945 (closed)

---

Migrated from D4830 (view on Phabricator)