Django + Redis (master/slave) = Pseudo distributed event based caching
This article focuses on the following topology to address usingRedis as the cache backend for Django where sets and deletes aresent to a master Redis server along with a local or dedicated Redisserver for a set of or single cluster member.
- 'Cluster Slave 1' <- 'Cluster Master 1'
- 'Cluster Slave 2' <- 'Cluster Master 1'
- 'Cluster Slave 3' <- 'Cluster Master 1'
- 'Django Node 1' uses 'Cluster Slave 1' as 'default' and Cluster Master 1 as 'master'
- 'Django Node 2' uses 'Cluster Slave 2' as 'default' and Cluster Master 1 as 'master'
- 'Django Node 3' uses 'Cluster Slave 3' as 'default' and Cluster Master 1 as 'master'
All updates from 'Cluster Master 1' are sent to all slaves as quickly as possible, not atomic, not verified. This requires you to expect slave cache keys to timeout appropriately as a backup.
You should note that using this method does not necessarilyaugment the standard of a memory cache.
- Store/Fetch/Invalidate Keys
- Keys can be deleted at any time
- The application is ultimately responsible for understanding cache behaviour
- Caches in a cluster may hold different values for a key
Number 4 is really only ever seen in a replicated or isolated environment where the backend is a database slave, a database in a master/master relationship, redis, or when using many different local caches per cluster. In the setup I am testing there is a single local slave at '10.10.0.41' and a single master at '10.10.0.40'. The redis servers are already configured, here is the Django cache configuraiton.
CACHES = {'default': {'BACKEND': 'redis_cache.RedisCache', 'LOCATION': '10.10.0.41:6379',},'master': {'BACKEND': 'redis_cache.RedisCache', 'LOCATION': '10.10.0.40:6379',},}
And lets do a little testing:
>>> from django.core.cache import get_cache>>> slave = get_cache('default')>>> master = get_cache('master')>>> master.set('hi','there')True>>> slave.get('hi')'there'
Huzzah.. it replicated!
>>> master.delete('hi')>>> slave.get('hi')
Huzzah.. it deleted!
>>> master.set('hi','there')True>>> slave.set('hi','pony')True>>> master.get('hi')'there'>>> slave.get('hi')'pony'
Huzzah.. it can be different!
>>> master.delete('hi')>>> master.get('hi')>>> slave.get('hi')
Huzzah.. even though there were differences I can delete the key!
What this means to me is that I can use the master for all changes and isolate all lookups to the slave server and if I'm feeling frisky I can even update the slave server. Keep in mind that with Redis and the Django-Redis-Cache module you can specify a different namespace or rather database ID making the following possible.
CACHES = {'default': { 'BACKEND': 'redis_cache.RedisCache', 'LOCATION': '10.10.0.41:6379', 'OPTIONS': { 'DB': 1, }, },'master': { 'BACKEND': 'redis_cache.RedisCache', 'LOCATION': '10.10.0.40:6379', 'OPTIONS': { 'DB': 1, }, },'local': { 'BACKEND': 'redis_cache.RedisCache', 'LOCATION': '10.10.0.41:6379', 'OPTIONS': { 'DB': 2, }, }, }
Huzzah.
The idea behind all of this is to allow the master to fail while the slave continues operating. In a hash based distribution that is possible, however whenever a cache member leaves there is no persistance for what was once cached without hacking the distributor to deal with n+1 replication of events. Once a hash member is alive again all keys that were reallocated to a different member are invalidated simply because the distributor won't think to look there any more unless the same member fails again.. at which point you are seeing old data that you couldn't possible expire using 'delete' since you didn't know it existed. This makes deleting data in a hash based distribution very clumsy.
The above scenario works best if you set a key on the slave first and then send it to the master. This allows the Django cluster node to have immediate access to data that it is attempting to persist into memory between request which is assumed to be the most appropriate node (given an intelligent entry distribution for the request/requestor pairs). Now the master can fail and come back up without too much of a problem. You can take it one step further and store a local set of keys (command queues would be a lot of trouble, just use keys) for master updates that never made it for when it comes back online and needs to be updated. Redis can be set up using heartbeat and DRBD to mitigate the down time of a Redis master as well as supports master/master mode by way of slaving eachother. Redis can store the database to disk which is great. You can write a quick program to replay the master cache to a new slave if you're using dynamic nodes since redis supports key lookups. You can even have the slave store values only in memory while the master uses the disk as well. Lots of options.
