Web API: add back persistent and shared backend for rate limit counters
Right now, the new archive.softwareheritage.org running django has been deployed with the rate limit backend as an in-memory one (default is LocMemCache).
This will pose a problem regarding the rate limit being shared between all gunicorn workers (which should not).
Prior to this deployment, we used redis (when using Flask_Limiter). We should be running an equivalent system to avoid such shortcomings.
According to djangorestframework/django, we could use memcached.
Discussion:
14:18:34 ardumont | what do i do with the redis instance that flask limiter used?
14:49:02 zack | where are the live counters stored in the new implementation?
14:49:08 zack | anyway, if it's no longer needed, it should go away
14:50:29 ardumont | i don't think it's stored anymore (NDA: I meant not persistent :)
14:50:32 ardumont | and not in redis anyway
14:50:39 ardumont | anlambert: do you confirm?
14:50:48 ardumont | other than that i'm ready to deploy
14:50:54 ardumont | i fixed the tiny errors in the manifest
14:52:18 olasd | it uses django's caching mechanism
14:55:29 olasd | so, by default, a local memory cache
14:58:31 olasd | which means that the rate limit would be spread out across workers
14:58:57 olasd | which is fine for a first deployment but needs to be fixed eventually
15:00:07 ardumont | you mean the different gunicorn workers?
15:00:10 olasd | yes
15:00:47 olasd | each gunicorn worker would get its own separate counter
15:01:20 ardumont | ack, thx (i was reading the doc, for info http://www.django-rest-framework.org/api-guide/throttling/#setting-up-the-cache)
15:01:31 ardumont | (not for you olasd :)
15:02:58 zack | so, maybe, before throwing away redis, check if it is still a viable backend?
15:04:01 olasd | stretch and up have python3-django-redis
15:04:04 olasd | not jessie
15:04:22 ardumont | ah, nice to know
15:04:51 ardumont | i won't throw away redis yet :)
15:04:52 olasd | but I'm not married to redis
15:06:04 ardumont | zack: it's not clear if it is a blocker to deploy (as olasd said, we can fix it later, as in next week for example)
15:06:18 zack | i didn't say it was a blocker :)
15:06:26 ardumont | ok, thx for clarifying :)
15:07:14 olasd | rate limiting is the only reason for redis to be deployed on our infra, we can just as well replace it with memcached if that's what django prefers
15:07:24 olasd | (flask-limiter also supported memcached)
15:07:37 ardumont | yes, and memcached is in the default list i refer to
15:08:07 ardumont | (well, i'd need to check for the version though, we're still in 1.7 in oldstable :)
15:08:19 ardumont | and the link is 1.11
15:08:45 ardumont | (it is present in 1.7 :)Migrated from T754 (view on Phabricator)