Summary

Between 18:33 July 13th and 11:20 July 14th 2023 (PDT), inline attachments from a small number of emails were incorrectly attached to an unknown number of emails sent by other accounts.

Impact & Metrics

Question	Answer
What was the impact	Incorrect inline attachments were incorrectly attached to a maximum of 17% of overall emails sent by Sendwithus
Who was impacted	A subset of Sendwithus users sending emails between July 13th 6:33 pm PDT and July 14th 11:20 am PDT may have been impacted

Detection & Response‌

Question	Answer
When was the incident detected	July 14th 2023, 10:21 am PDT
How was the incident detected	An impacted user reached out to the Sendwithus Customer Support team and the incident was quickly escalated

Timeline

Times are in Pacific Time

Jul 13, 2023

09:44 A new feature related to sending inline attachments was released

18:33 The first detection of a send using the feature since the change

Jul 14, 2023

10:21 A user reported some of their recipients were receiving unexpected inline attachments on their emails

10:45 Customer Support notifies on-call engineers and the incident is escalated

11:00   Status page updated

11:10 Resolved the issue by reverting to the previous version of Sendwithus

12:40 Underlying issue identified

13:00 Engineering team began an investigation to determine total user impact

17:00 First emails to personally reach out to impacted users sent

Root Cause Analysis

Problem: A user has reported that some of their emails contain an unexpected attachment. Why?

Reason: Recent changes were made to our systems to support sending an arbitrary number of inline attachments. This change introduced a bug which resulted in a variable being reused for subsequent emails sent by the same process. Why?

Reason: An optional parameter in a function definition had a default reference-type value. The issue was not identified before deploying to production. Why?

Reason: Our current testing, linting, and code review processes were not sufficient to identify the problem.

What can be improved

• Changes are being made to our testing and linting processes to catch this category of bug in our CI pipeline
• The transient nature of inline attachments and a lack of thread/process level identification in our archived transaction logs has made it difficult to identify exactly which customers and emails were affected. Due to this only the maximum number of potential impacted emails can be identified
• Increased controls to improve cross-tenant isolation of information

Corrective actions

• The problematic change was quickly identified and reverted after we became aware of the issue
• Changes have since been made to the codebase to reduce the probability of similar issues manifesting elsewhere