Fixing GCP IAM Permission Issues After Outages

  • 10/4/2020
  • One-minute read

Following a Google Cloud IAM disruption, long-running automation began failing with:

ServiceException: 401 Anonymous caller does not have storage.buckets.list access to project <project>

The service account credentials were unchanged and still granted roles/storage.admin.

Resolution Steps

  1. Re-authenticate: Run gcloud auth activate-service-account with the existing key to confirm it loads correctly.
  2. List Permissions: gcloud projects get-iam-policy to ensure the service account retains the necessary roles.
  3. Force Permission Propagation: Creating a new bucket (gsutil mb ...) forced IAM caches to refresh, after which gsutil ls showed the original bucket again.
  4. Purge Cached Tokens: If issues persist, delete ~/.config/gcloud cached tokens and re-authenticate.

Preventive Actions

  • Monitor Google Cloud status feeds and set up alerting for IAM outages.
  • Design scripts to retry with exponential backoff and surface IAM errors clearly.
  • Where possible, avoid scripting against implicit storage.buckets.list; fetch known buckets directly by name.

Disclaimer

Behaviour observed during one outage; future incidents may differ. Always engage Google Cloud support for production-impacting IAM issues and avoid relying on undocumented refresh side effects.

gcpcloudsecuritytroubleshooting