Heroku sync integrations are failing
Incident Report for Doppler
Postmortem

Root Cause

Doppler syncs secrets to Heroku via a Heroku OAuth application. This application is created in the Heroku dashboard and must be owned by a single Heroku user account.

Doppler’s previous Heroku OAuth application was owned by a specific Doppler employee’s Heroku account without access to any additional resources. During a routine external account audit, this account was mistakenly identified as unused and manually deleted by our security team. This irrecoverably deleted Doppler’s existing Heroku OAuth application, thereby breaking any existing syncs and requiring the creation of a new OAuth application in a new account.

Resolution

Because users had authorized our previous Heroku OAuth application to their Heroku account(s), users need to authorize the new Heroku OAuth application. This involves reconnecting the integration from the Doppler dashboard. Once the integration is reconnected, Doppler will re-enable all associated syncs that have been disabled and perform a fresh sync.

Note that the previous OAuth application was deleted and therefore no action is required to remove its access to your Heroku account.

Next Steps

Internally, we’re reorganizing how shared accounts used for critical functionality are stored in 1Password. This new 1Password organization should help prevent this kind of accidental deletion in the future. We avoid shared accounts whenever possible, but this isn’t always feasible given third party implementations.

We'll also be adding our individual integrations to our status page. This will allow customers to more easily see which integrations, if any, are currently experiencing issues.

Posted Jan 05, 2023 - 13:40 PST

Resolved
This issue is now resolved. Additional action is required to re-enable existing Heroku syncs. All workplaces will need to reconnect their Heroku sync integrations from the [workplace Settings page](https://dashboard.doppler.com/workplace/settings). Once integrations are reconnected, all syncs will automatically be triggered to sync any pending changes in Doppler.

A postmortem will be available soon with more details regarding what happened.
Posted Jan 05, 2023 - 10:23 PST
Update
This incident may persist for up to 24 hours. In the meantime, if you have an urgent secret change that needs to be made, please update the secret in your Doppler dashboard and then manually update the secret in Heroku directly via `heroku config:set`. Updating the secret in Doppler will ensure that the value in Heroku is not overwritten once the incident is resolved.
Posted Jan 04, 2023 - 17:06 PST
Identified
New and existing Heroku sync integrations are currently not functioning. We have identified the issue and are working on resolving it. Secrets previously synced to Heroku apps will remain available but new secret changes in Doppler will not be synced until this is resolved.
Posted Jan 04, 2023 - 16:48 PST
This incident affected: Integrations (Heroku).