[Python] Azure Cache for Redis randomly throws WRONGPASS

I have a Django website running as an Azure Web App, which is uzing Azure Cache for Redis.

We use a Managed Identity to connect to Azure Cache for Redis so we do not need to use access keys.

The application itself is running in gunicorn so as multiple threads.

There is a health check, being called by Azure Web App platform itself, for which I have this middleware:

import logging
import os
import signal

from django.conf import settings
from django.core.cache import cache
from django.db import connection
from django.http import HttpResponse
from django_redis import get_redis_connection
from tenacity import retry, stop_after_attempt, wait_random_exponential

from .azure_helper import RedisCredentials, get_db_password, get_redis_credentials

logger = logging.getLogger("pulumi_django_azure.health_check")


class HealthCheckMiddleware:
def __init__(self, get_response):
self.get_response = get_response

def _self_heal(self):
logger.warning("Self-healing by gracefully restarting Gunicorn.")

master_pid = os.getppid()

logger.debug("Master PID: %d", master_pid)

# https://docs.gunicorn.org/en/latest/signals.html

# Reload a new master with new workers,
# since the application is preloaded this is the only safe way for now.
os.kill(master_pid, signal.SIGUSR2)

# Gracefully shutdown the current workers
os.kill(master_pid, signal.SIGTERM)

def _test_redis_connection(self) -> bool:
try:
cache.set("health_check", "test")
return True
except Exception:
return False

@retry(stop=stop_after_attempt(3), wait=wait_random_exponential(multiplier=0.5, max=5))
def _update_redis_credentials(self, redis_credentials: RedisCredentials):
logger.debug("Updating Redis credentials with password: %s", redis_credentials.password)

# Re-authenticate the Redis connection
redis_connection = get_redis_connection("default")
redis_connection.execute_command("AUTH", redis_credentials.username, redis_credentials.password)

settings.CACHES["default"]["OPTIONS"]["PASSWORD"] = redis_credentials.password

def __call__(self, request):
if request.path == settings.HEALTH_CHECK_PATH:
# Update the database credentials if needed
if settings.AZURE_DB_PASSWORD:
try:
current_db_password = settings.DATABASES["default"]["PASSWORD"]
new_db_password = get_db_password()

if new_db_password != current_db_password:
logger.debug("Database password has changed, updating credentials")
settings.DATABASES["default"]["PASSWORD"] = new_db_password

# Close existing connections to force reconnect with new password
connection.close()
else:
logger.debug("Database password unchanged, keeping existing credentials")
except Exception as e:
logger.error("Failed to update database credentials: %s", str(e))
self._self_heal()
return HttpResponse(status=503)

# Update the Redis credentials if needed
if settings.AZURE_REDIS_CREDENTIALS:
try:
current_redis_password = settings.CACHES["default"]["OPTIONS"]["PASSWORD"]
redis_credentials = get_redis_credentials()

if redis_credentials.password != current_redis_password:
logger.debug("Redis password has changed, updating credentials")

self._update_redis_credentials(redis_credentials)
elif not self._test_redis_connection():
logger.debug("Redis connection check failed, updating credentials")
self._update_redis_credentials(redis_credentials)
else:
logger.debug("Redis password unchanged and connection check passed, keeping existing credentials")
except Exception as e:
logger.error("Failed to update Redis credentials: %s", str(e))
self._self_heal()
return HttpResponse(status=503)

try:
# Test the database connection
connection.ensure_connection()
logger.debug("Database connection check passed")

# Test the Redis connection
cache.set("health_check", "test")
logger.debug("Redis connection check passed")

return HttpResponse("OK")

except Exception as e:
logger.error("Health check failed with unexpected error: %s", str(e))
self._self_heal()
return HttpResponse(status=503)

return self.get_response(request)


So essentially, we retrieve an auth token from the Azure management API, and if that password has changed OR the connection test fails, we perform an AUTH command.

Now, randomly we see that our self-healing mechanisms kicks in (which we added because of all the issues we had and we want to keep the site running while we figure this out). After adding more logging, as can be seen in the above code, we see in the logs:

2025-08-12 13:56:54,647 DEBUG [p:1293] [t:123417039960960] Updating Redis credentials with password: eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6IkpZaEFjVFBNWl9MWDZEQmxPV1E3SG4wTmVYRSIsImtpZCI6IkpZaEFjVFBNWl9MWDZEQmxPV1E3SG4wTmVYRSJ9.eyJhdWQiOiJodHRwczovL3JlZGlzLmF6dXJlLmNvbSIsImlzcyI6Imh0dHBzOi8vc3RzLndpbmRvd3MubmV0L2UyOThmYWJhLTdjNGEtNGQzZS1iZGQ2LTg3ZDA5OGI0ZjdlMi8iLCJpYXQiOjE3NTQ5OTk0NTksIm5iZiI6MTc1NDk5OTQ1OSwiZXhwIjoxNzU1MDg2MTU5LCJhaW8iOiJrMlJnWU5nLzYvY2JXN3ZGWmgwdERvOC9zbmJ3QVFBPSIsImFwcGlkIjoiN2YxY2UzOWItZmVmOS00NDAwLTg0NDMtYzhjMGVlMDc2ZGJlIiwiYXBwaWRhY3IiOiIyIiwiaWRwIjoiaHR0cHM6Ly9zdHMud2luZG93cy5uZXQvZTI5OGZhYmEtN2M0YS00ZDNlLWJkZDYtODdkMDk4YjRmN2UyLyIsImlkdHlwIjoiYXBwIiwib2lkIjoiZTlmYWE3ZjUtZWQ4MC00MjhmLWFkNzMtMjM4MzdmNTFmZTVlIiwicmgiOiIxLkFRd0F1dnFZNGtwOFBrMjkxb2ZRbUxUMzRydGZ5cXprdHdsQWdmRTM0NF9XYlhqZEFBQU1BQS4iLCJzdWIiOiJlOWZhYTdmNS1lZDgwLTQyOGYtYWQ3My0yMzgzN2Y1MWZlNWUiLCJ0aWQiOiJlMjk4ZmFiYS03YzRhLTRkM2UtYmRkNi04N2QwOThiNGY3ZTIiLCJ1dGkiOiIwdDltQThYeVJVeVpzWWozOTEwYUFBIiwidmVyIjoiMS4wIiwieG1zX2Z0ZCI6InIwdGUwZVZ6TURXV0xJaThJcm5LMXpmRWJ3bDNqdEpnYTk4RzlQZm1CblVCYzNkbFpHVnVZeTFrYzIxeiIsInhtc19pZHJlbCI6IjggNyIsInhtc19yZCI6IjAuNDJMbFlCSmk1QkVTNGVBVUVtQkFBMEJSRGlHQjNmR24xanpaenVuUS1OSDhjZi0xM0FvQSJ9.nQTOjqYjyyfjKU5b4Y9fSWMYrkggCRRCeQAF4dhCTcuw-vUOpbLbBeRAJ1nM987L6qlOW03guLfENyGlPCdGEpCwDZrlo0HA13i35qeEklxKUUvieBu_rSz9lDBPrRLiPWDKANzgLZ9JuDj6ZpdH7RqVazK1Ygn-aIZ-J1iYPzgY-XCA0vMcCyHTwDDlNsOAq7uuvQwdDu9cUzbQk5bUwE7tWiyvr6cbzImuROAIAv6ZmxwfxrTBgUV7KcEwLP_vmMFipgvjqJDiydHtq7rERPSq0ixVspbZZCM0wsW56XZ5WjfXW2lfBnqRdTOeFhMTtJAQBDgOgijsIfRSup902Q
2025-08-12 13:56:54,852 DEBUG [p:1293] [t:123417039960960] Database connection check passed
2025-08-12 13:56:54,856 DEBUG [p:1293] [t:123417039960960] Redis connection check passed
2025-08-12 13:57:54,562 DEBUG [p:1294] [t:123417039960960] Database password unchanged, keeping existing credentials
2025-08-12 13:57:54,565 DEBUG [p:1294] [t:123417039960960] Redis password unchanged and connection check passed, keeping existing credentials
2025-08-12 13:57:54,565 DEBUG [p:1294] [t:123417039960960] Database connection check passed
2025-08-12 13:57:54,570 DEBUG [p:1294] [t:123417039960960] Redis connection check passed
2025-08-12 13:58:54,760 DEBUG [p:1297] [t:123417039960960] Database password has changed, updating credentials
2025-08-12 13:58:54,788 DEBUG [p:1297] [t:123417039960960] Redis password has changed, updating credentials
2025-08-12 13:58:54,795 DEBUG [p:1297] [t:123417039960960] Updating Redis credentials with password: eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6IkpZaEFjVFBNWl9MWDZEQmxPV1E3SG4wTmVYRSIsImtpZCI6IkpZaEFjVFBNWl9MWDZEQmxPV1E3SG4wTmVYRSJ9.eyJhdWQiOiJodHRwczovL3JlZGlzLmF6dXJlLmNvbSIsImlzcyI6Imh0dHBzOi8vc3RzLndpbmRvd3MubmV0L2UyOThmYWJhLTdjNGEtNGQzZS1iZGQ2LTg3ZDA5OGI0ZjdlMi8iLCJpYXQiOjE3NTQ5OTk0NTksIm5iZiI6MTc1NDk5OTQ1OSwiZXhwIjoxNzU1MDg2MTU5LCJhaW8iOiJrMlJnWU5nLzYvY2JXN3ZGWmgwdERvOC9zbmJ3QVFBPSIsImFwcGlkIjoiN2YxY2UzOWItZmVmOS00NDAwLTg0NDMtYzhjMGVlMDc2ZGJlIiwiYXBwaWRhY3IiOiIyIiwiaWRwIjoiaHR0cHM6Ly9zdHMud2luZG93cy5uZXQvZTI5OGZhYmEtN2M0YS00ZDNlLWJkZDYtODdkMDk4YjRmN2UyLyIsImlkdHlwIjoiYXBwIiwib2lkIjoiZTlmYWE3ZjUtZWQ4MC00MjhmLWFkNzMtMjM4MzdmNTFmZTVlIiwicmgiOiIxLkFRd0F1dnFZNGtwOFBrMjkxb2ZRbUxUMzRydGZ5cXprdHdsQWdmRTM0NF9XYlhqZEFBQU1BQS4iLCJzdWIiOiJlOWZhYTdmNS1lZDgwLTQyOGYtYWQ3My0yMzgzN2Y1MWZlNWUiLCJ0aWQiOiJlMjk4ZmFiYS03YzRhLTRkM2UtYmRkNi04N2QwOThiNGY3ZTIiLCJ1dGkiOiIwdDltQThYeVJVeVpzWWozOTEwYUFBIiwidmVyIjoiMS4wIiwieG1zX2Z0ZCI6InIwdGUwZVZ6TURXV0xJaThJcm5LMXpmRWJ3bDNqdEpnYTk4RzlQZm1CblVCYzNkbFpHVnVZeTFrYzIxeiIsInhtc19pZHJlbCI6IjggNyIsInhtc19yZCI6IjAuNDJMbFlCSmk1QkVTNGVBVUVtQkFBMEJSRGlHQjNmR24xanpaenVuUS1OSDhjZi0xM0FvQSJ9.nQTOjqYjyyfjKU5b4Y9fSWMYrkggCRRCeQAF4dhCTcuw-vUOpbLbBeRAJ1nM987L6qlOW03guLfENyGlPCdGEpCwDZrlo0HA13i35qeEklxKUUvieBu_rSz9lDBPrRLiPWDKANzgLZ9JuDj6ZpdH7RqVazK1Ygn-aIZ-J1iYPzgY-XCA0vMcCyHTwDDlNsOAq7uuvQwdDu9cUzbQk5bUwE7tWiyvr6cbzImuROAIAv6ZmxwfxrTBgUV7KcEwLP_vmMFipgvjqJDiydHtq7rERPSq0ixVspbZZCM0wsW56XZ5WjfXW2lfBnqRdTOeFhMTtJAQBDgOgijsIfRSup902Q
2025-08-12 13:58:55,099 DEBUG [p:1297] [t:123417039960960] Database connection check passed
2025-08-12 13:58:55,102 DEBUG [p:1297] [t:123417039960960] Redis connection check passed
2025-08-12 13:59:54,606 DEBUG [p:1292] [t:123417039960960] Database password has changed, updating credentials
2025-08-12 13:59:54,644 DEBUG [p:1292] [t:123417039960960] Redis password has changed, updating credentials
2025-08-12 13:59:54,644 DEBUG [p:1292] [t:123417039960960] Updating Redis credentials with password: eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6IkpZaEFjVFBNWl9MWDZEQmxPV1E3SG4wTmVYRSIsImtpZCI6IkpZaEFjVFBNWl9MWDZEQmxPV1E3SG4wTmVYRSJ9.eyJhdWQiOiJodHRwczovL3JlZGlzLmF6dXJlLmNvbSIsImlzcyI6Imh0dHBzOi8vc3RzLndpbmRvd3MubmV0L2UyOThmYWJhLTdjNGEtNGQzZS1iZGQ2LTg3ZDA5OGI0ZjdlMi8iLCJpYXQiOjE3NTQ5OTk0NTksIm5iZiI6MTc1NDk5OTQ1OSwiZXhwIjoxNzU1MDg2MTU5LCJhaW8iOiJrMlJnWU5nLzYvY2JXN3ZGWmgwdERvOC9zbmJ3QVFBPSIsImFwcGlkIjoiN2YxY2UzOWItZmVmOS00NDAwLTg0NDMtYzhjMGVlMDc2ZGJlIiwiYXBwaWRhY3IiOiIyIiwiaWRwIjoiaHR0cHM6Ly9zdHMud2luZG93cy5uZXQvZTI5OGZhYmEtN2M0YS00ZDNlLWJkZDYtODdkMDk4YjRmN2UyLyIsImlkdHlwIjoiYXBwIiwib2lkIjoiZTlmYWE3ZjUtZWQ4MC00MjhmLWFkNzMtMjM4MzdmNTFmZTVlIiwicmgiOiIxLkFRd0F1dnFZNGtwOFBrMjkxb2ZRbUxUMzRydGZ5cXprdHdsQWdmRTM0NF9XYlhqZEFBQU1BQS4iLCJzdWIiOiJlOWZhYTdmNS1lZDgwLTQyOGYtYWQ3My0yMzgzN2Y1MWZlNWUiLCJ0aWQiOiJlMjk4ZmFiYS03YzRhLTRkM2UtYmRkNi04N2QwOThiNGY3ZTIiLCJ1dGkiOiIwdDltQThYeVJVeVpzWWozOTEwYUFBIiwidmVyIjoiMS4wIiwieG1zX2Z0ZCI6InIwdGUwZVZ6TURXV0xJaThJcm5LMXpmRWJ3bDNqdEpnYTk4RzlQZm1CblVCYzNkbFpHVnVZeTFrYzIxeiIsInhtc19pZHJlbCI6IjggNyIsInhtc19yZCI6IjAuNDJMbFlCSmk1QkVTNGVBVUVtQkFBMEJSRGlHQjNmR24xanpaenVuUS1OSDhjZi0xM0FvQSJ9.nQTOjqYjyyfjKU5b4Y9fSWMYrkggCRRCeQAF4dhCTcuw-vUOpbLbBeRAJ1nM987L6qlOW03guLfENyGlPCdGEpCwDZrlo0HA13i35qeEklxKUUvieBu_rSz9lDBPrRLiPWDKANzgLZ9JuDj6ZpdH7RqVazK1Ygn-aIZ-J1iYPzgY-XCA0vMcCyHTwDDlNsOAq7uuvQwdDu9cUzbQk5bUwE7tWiyvr6cbzImuROAIAv6ZmxwfxrTBgUV7KcEwLP_vmMFipgvjqJDiydHtq7rERPSq0ixVspbZZCM0wsW56XZ5WjfXW2lfBnqRdTOeFhMTtJAQBDgOgijsIfRSup902Q
2025-08-12 13:59:55,021 DEBUG [p:1292] [t:123417039960960] Updating Redis credentials with password: eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6IkpZaEFjVFBNWl9MWDZEQmxPV1E3SG4wTmVYRSIsImtpZCI6IkpZaEFjVFBNWl9MWDZEQmxPV1E3SG4wTmVYRSJ9.eyJhdWQiOiJodHRwczovL3JlZGlzLmF6dXJlLmNvbSIsImlzcyI6Imh0dHBzOi8vc3RzLndpbmRvd3MubmV0L2UyOThmYWJhLTdjNGEtNGQzZS1iZGQ2LTg3ZDA5OGI0ZjdlMi8iLCJpYXQiOjE3NTQ5OTk0NTksIm5iZiI6MTc1NDk5OTQ1OSwiZXhwIjoxNzU1MDg2MTU5LCJhaW8iOiJrMlJnWU5nLzYvY2JXN3ZGWmgwdERvOC9zbmJ3QVFBPSIsImFwcGlkIjoiN2YxY2UzOWItZmVmOS00NDAwLTg0NDMtYzhjMGVlMDc2ZGJlIiwiYXBwaWRhY3IiOiIyIiwiaWRwIjoiaHR0cHM6Ly9zdHMud2luZG93cy5uZXQvZTI5OGZhYmEtN2M0YS00ZDNlLWJkZDYtODdkMDk4YjRmN2UyLyIsImlkdHlwIjoiYXBwIiwib2lkIjoiZTlmYWE3ZjUtZWQ4MC00MjhmLWFkNzMtMjM4MzdmNTFmZTVlIiwicmgiOiIxLkFRd0F1dnFZNGtwOFBrMjkxb2ZRbUxUMzRydGZ5cXprdHdsQWdmRTM0NF9XYlhqZEFBQU1BQS4iLCJzdWIiOiJlOWZhYTdmNS1lZDgwLTQyOGYtYWQ3My0yMzgzN2Y1MWZlNWUiLCJ0aWQiOiJlMjk4ZmFiYS03YzRhLTRkM2UtYmRkNi04N2QwOThiNGY3ZTIiLCJ1dGkiOiIwdDltQThYeVJVeVpzWWozOTEwYUFBIiwidmVyIjoiMS4wIiwieG1zX2Z0ZCI6InIwdGUwZVZ6TURXV0xJaThJcm5LMXpmRWJ3bDNqdEpnYTk4RzlQZm1CblVCYzNkbFpHVnVZeTFrYzIxeiIsInhtc19pZHJlbCI6IjggNyIsInhtc19yZCI6IjAuNDJMbFlCSmk1QkVTNGVBVUVtQkFBMEJSRGlHQjNmR24xanpaenVuUS1OSDhjZi0xM0FvQSJ9.nQTOjqYjyyfjKU5b4Y9fSWMYrkggCRRCeQAF4dhCTcuw-vUOpbLbBeRAJ1nM987L6qlOW03guLfENyGlPCdGEpCwDZrlo0HA13i35qeEklxKUUvieBu_rSz9lDBPrRLiPWDKANzgLZ9JuDj6ZpdH7RqVazK1Ygn-aIZ-J1iYPzgY-XCA0vMcCyHTwDDlNsOAq7uuvQwdDu9cUzbQk5bUwE7tWiyvr6cbzImuROAIAv6ZmxwfxrTBgUV7KcEwLP_vmMFipgvjqJDiydHtq7rERPSq0ixVspbZZCM0wsW56XZ5WjfXW2lfBnqRdTOeFhMTtJAQBDgOgijsIfRSup902Q
2025-08-12 13:59:55,775 DEBUG [p:1292] [t:123417039960960] Updating Redis credentials with password: eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6IkpZaEFjVFBNWl9MWDZEQmxPV1E3SG4wTmVYRSIsImtpZCI6IkpZaEFjVFBNWl9MWDZEQmxPV1E3SG4wTmVYRSJ9.eyJhdWQiOiJodHRwczovL3JlZGlzLmF6dXJlLmNvbSIsImlzcyI6Imh0dHBzOi8vc3RzLndpbmRvd3MubmV0L2UyOThmYWJhLTdjNGEtNGQzZS1iZGQ2LTg3ZDA5OGI0ZjdlMi8iLCJpYXQiOjE3NTQ5OTk0NTksIm5iZiI6MTc1NDk5OTQ1OSwiZXhwIjoxNzU1MDg2MTU5LCJhaW8iOiJrMlJnWU5nLzYvY2JXN3ZGWmgwdERvOC9zbmJ3QVFBPSIsImFwcGlkIjoiN2YxY2UzOWItZmVmOS00NDAwLTg0NDMtYzhjMGVlMDc2ZGJlIiwiYXBwaWRhY3IiOiIyIiwiaWRwIjoiaHR0cHM6Ly9zdHMud2luZG93cy5uZXQvZTI5OGZhYmEtN2M0YS00ZDNlLWJkZDYtODdkMDk4YjRmN2UyLyIsImlkdHlwIjoiYXBwIiwib2lkIjoiZTlmYWE3ZjUtZWQ4MC00MjhmLWFkNzMtMjM4MzdmNTFmZTVlIiwicmgiOiIxLkFRd0F1dnFZNGtwOFBrMjkxb2ZRbUxUMzRydGZ5cXprdHdsQWdmRTM0NF9XYlhqZEFBQU1BQS4iLCJzdWIiOiJlOWZhYTdmNS1lZDgwLTQyOGYtYWQ3My0yMzgzN2Y1MWZlNWUiLCJ0aWQiOiJlMjk4ZmFiYS03YzRhLTRkM2UtYmRkNi04N2QwOThiNGY3ZTIiLCJ1dGkiOiIwdDltQThYeVJVeVpzWWozOTEwYUFBIiwidmVyIjoiMS4wIiwieG1zX2Z0ZCI6InIwdGUwZVZ6TURXV0xJaThJcm5LMXpmRWJ3bDNqdEpnYTk4RzlQZm1CblVCYzNkbFpHVnVZeTFrYzIxeiIsInhtc19pZHJlbCI6IjggNyIsInhtc19yZCI6IjAuNDJMbFlCSmk1QkVTNGVBVUVtQkFBMEJSRGlHQjNmR24xanpaenVuUS1OSDhjZi0xM0FvQSJ9.nQTOjqYjyyfjKU5b4Y9fSWMYrkggCRRCeQAF4dhCTcuw-vUOpbLbBeRAJ1nM987L6qlOW03guLfENyGlPCdGEpCwDZrlo0HA13i35qeEklxKUUvieBu_rSz9lDBPrRLiPWDKANzgLZ9JuDj6ZpdH7RqVazK1Ygn-aIZ-J1iYPzgY-XCA0vMcCyHTwDDlNsOAq7uuvQwdDu9cUzbQk5bUwE7tWiyvr6cbzImuROAIAv6ZmxwfxrTBgUV7KcEwLP_vmMFipgvjqJDiydHtq7rERPSq0ixVspbZZCM0wsW56XZ5WjfXW2lfBnqRdTOeFhMTtJAQBDgOgijsIfRSup902Q
2025-08-12 13:59:55,959 ERROR [p:1292] [t:123417039960960] Failed to update Redis credentials: RetryError[<Future at 0x703f38c02d70 state=finished raised ResponseError>]
2025-08-12 13:59:55,970 WARNING [p:1292] [t:123417039960960] Self-healing by gracefully restarting Gunicorn.


So, the health check arrives every minute on a different process (is random), in that process the logic kicks in that sees that the password has changed; but then the AUTH fails. It is not visible in these logs anymore but the error from Redis is WRONGPASS.

So the credentials we freshly obtained from Azure, are not valid in P1292 in this example, while it was valid 3 minutes earlier on PID 1293.

I have enabled full logs on Azure Cache hoping to spot something, but awaiting my findings of that, hoping to find an answer here.

Continue reading...