Fixing GCP IAM Permission Issues After Outages
Following a Google Cloud IAM disruption, long-running automation began failing with:
ServiceException: 401 Anonymous caller does not have storage.buckets.list access to project <project>
The service account credentials were unchanged and still granted roles/storage.admin.
Resolution Steps
- Re-authenticate: Run
gcloud auth activate-service-accountwith the existing key to confirm it loads correctly. - List Permissions:
gcloud projects get-iam-policyto ensure the service account retains the necessary roles. - Force Permission Propagation: Creating a new bucket (
gsutil mb ...) forced IAM caches to refresh, after whichgsutil lsshowed the original bucket again. - Purge Cached Tokens: If issues persist, delete
~/.config/gcloudcached tokens and re-authenticate.
Preventive Actions
- Monitor Google Cloud status feeds and set up alerting for IAM outages.
- Design scripts to retry with exponential backoff and surface IAM errors clearly.
- Where possible, avoid scripting against implicit
storage.buckets.list; fetch known buckets directly by name.
Disclaimer
Behaviour observed during one outage; future incidents may differ. Always engage Google Cloud support for production-impacting IAM issues and avoid relying on undocumented refresh side effects.