A code change was made to the Push Notification API affecting the logic for loading push credentials. Apps that were already sending push notifications were not affected. No manual action needs to be taken by any affected accounts; all push credentials created before, during, or after the incident are operational.
Timeline
Friday, February 28, 2020 @ 3:30pm: An erroneous code change was pushed that changed the logic for loading stored push credentials. When sending a push notification, we look for push credentials in a read-through cache with a fallback to a database. The logic was accidentally changed to not fall back to the database when locating push credentials, meaning that any credentials not already in the cache were reported to not exist even though they were really in the database. This meant that all push credentials that were previously in the cache remained operational and unaffected, and only new credentials that were not yet in the cache before the incident started reported as not found.
Sunday, March 1 @ 10:52pm: Issue identified.
Sunday, March 1 @ 11:38pm: Issue fixed.
Follow-up and Remediation
Identify and audit similar codepaths where bugs of this nature can occur during code refactoring.
Improve test coverage in area containing bug and surrounding bug.