New App Push Notifications Outage
Incident Report for Expo
Postmortem

Overview

A code change was made to the Push Notification API affecting the logic for loading push credentials. Apps that were already sending push notifications were not affected. No manual action needs to be taken by any affected accounts; all push credentials created before, during, or after the incident are operational.

Timeline

  • Friday, February 28, 2020 @ 3:30pm: An erroneous code change was pushed that changed the logic for loading stored push credentials. When sending a push notification, we look for push credentials in a read-through cache with a fallback to a database. The logic was accidentally changed to not fall back to the database when locating push credentials, meaning that any credentials not already in the cache were reported to not exist even though they were really in the database. This meant that all push credentials that were previously in the cache remained operational and unaffected, and only new credentials that were not yet in the cache before the incident started reported as not found.
  • Sunday, March 1 @ 10:52pm: Issue identified.
  • Sunday, March 1 @ 11:38pm: Issue fixed.

Follow-up and Remediation

  • Identify and audit similar codepaths where bugs of this nature can occur during code refactoring.
  • Improve test coverage in area containing bug and surrounding bug.
Posted Mar 02, 2020 - 12:34 PST

Resolved
Posted Mar 02, 2020 - 09:39 PST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Mar 01, 2020 - 23:40 PST
Identified
The issue has been identified and a fix is being implemented.
Posted Mar 01, 2020 - 23:11 PST
Investigating
We are currently investigating this issue.
Posted Mar 01, 2020 - 17:16 PST
This incident affected: Push Notifications Broker.