Sporadic iOS push notification outage
Incident Report for Expo
Resolved
From late 2019-12-31 PST, one instance of the iOS push notification service was using a different version of an internal protocol. Messages sent to this one instance would fail to be sent and result in "InvalidCredentials" push receipts with the message, "The Apple Push Notification service key for your account could not be loaded, likely due to a malformed p8 file. Expo needs a valid PEM key in order to deliver notifications to APNs." The underlying error's message looks like, "Failed to generate token: error:0909006C:PEM routines:get_name:no start line" for projects that use modern APNs P8 keys.

Symptoms: A portion of notifications would fail and create push receipts with the error information described above. Servers that implement retry logic (re-sending a notification when the push receipt indicates a failure) would likely have succeeded eventually but obviously this isn't intentional behavior of the Expo push notification service (it's good to implement retry logic on your server and even better if it doesn't need to run usually).

Resolution (Jan 2, 4PM PST): The instance with the problematic code was stopped and shortly after, some Expo developers saw notifications succeeding as expected again.
Posted Jan 02, 2020 - 16:00 PST