Between 18:33 July 13th and 11:20 July 14th 2023 (PDT), inline attachments from a small number of emails were incorrectly attached to an unknown number of emails sent by other accounts.
Question | Answer |
---|---|
What was the impact | Incorrect inline attachments were incorrectly attached to a maximum of 17% of overall emails sent by Sendwithus |
Who was impacted | A subset of Sendwithus users sending emails between July 13th 6:33 pm PDT and July 14th 11:20 am PDT may have been impacted |
Question | Answer |
---|---|
When was the incident detected | July 14th 2023, 10:21 am PDT |
How was the incident detected | An impacted user reached out to the Sendwithus Customer Support team and the incident was quickly escalated |
Times are in Pacific Time
Jul 13, 2023
09:44 A new feature related to sending inline attachments was released
18:33 The first detection of a send using the feature since the change
Jul 14, 2023
10:21 A user reported some of their recipients were receiving unexpected inline attachments on their emails
10:45 Customer Support notifies on-call engineers and the incident is escalated
11:00 Status page updated
11:10 Resolved the issue by reverting to the previous version of Sendwithus
12:40 Underlying issue identified
13:00 Engineering team began an investigation to determine total user impact
17:00 First emails to personally reach out to impacted users sent
Problem: A user has reported that some of their emails contain an unexpected attachment. Why?
Reason: Recent changes were made to our systems to support sending an arbitrary number of inline attachments. This change introduced a bug which resulted in a variable being reused for subsequent emails sent by the same process. Why?
Reason: An optional parameter in a function definition had a default reference-type value. The issue was not identified before deploying to production. Why?
Reason: Our current testing, linting, and code review processes were not sufficient to identify the problem.