Drip Campaign Issues
Incident Report for sendwithus
Postmortem

Summary

Due to an issue during a database migration on 2019-02-14, Drip Campaign activations failed to activate during a 27 hour period from 2019-02-14 12:50 PST to 2019-02-15 15:36 PST. Any Drip Campaign emails scheduled to send before the incident and any deactivations made with the API or automation triggers continued to work as normal.

We apologize for any disruption this may have caused for our customers and their business. We have already notified all account administrators for effected accounts.

What happened

Our infrastructure team executed a planned data migration of the database responsible for Drip Campaigns. During this time all new Drip Campaign activations were to be made redundant across both the old and new datasets. An error in our code to configure this redundancy caused loss of data during this migration period. This ultimately prevented any new Drip Campaigns from properly activating.

What was affected

During the incident window:
- Drip Campaign stopped activating via the API
- Drip Campaign stopped activating via automation triggers
- No new Drip Campaigns could be created

Who was affected

Any accounts that activated a Drip Campaign during the period above would have been affected. This includes Drip Campaign activations through the API and through Automation Triggers. If this was the case, your account administrator An email has been sent to your Sendwithus administrators if your account was affected.

What Sendwithus is doing to fix it

While do mistakes happen, it's most important to us that they're caught early so they can be swiftly corrected. To mitigate the likelihood of this happening in the future we will be:
- Adjusting our alert thresholds for this particular issue.
- Adding more thorough migration testing processes

Posted 5 months ago. Feb 26, 2019 - 15:42 PST

Resolved
This incident has been resolved.
Posted 5 months ago. Feb 15, 2019 - 16:22 PST
Monitoring
We've identified the issue, deployed a solution, and are now monitoring the results.
Posted 5 months ago. Feb 15, 2019 - 15:54 PST
Investigating
Some customers are experiencing issues with activating drip campaigns and running legacy segmentation. The Sendwithus engineering team is currently investigating. API triggered emails are still sending normally.
Posted 5 months ago. Feb 15, 2019 - 15:15 PST
This incident affected: API and Email Delivery.