RCA: Email delivery failure

What was the issue?

On Saturday, May 4th, 2019, emails sent to Gmail and G Suite domains were blocked. All product email services such as candidate invite, password reset, verification emails, etc. were impacted and a total of 3640 emails failed during the issue.

Why did it happen?

Our emails for Gmail and G Suite domains started failing due to a new restriction defined on the sender’s address. Our emails had the sender's address in the format: '<name>' <email ID>, for example, 'HackerEarth Support' <support@hackerearth.com> instead of <email ID>, for example, support@hackerearth.com.

Emails sent to domains hosted by other providers like Yahoo and Hotmail were working as intended.

Actions taken toward a resolution

We updated our sender’s address with our email service provider, SendGrid. A small batch of emails was tested, and the concern was resolved on May 6th, 2019 at 9:30 AM UTC. This resolution was only applicable to new emails triggering from the platform to new users.

Impacted users with Gmail and G Suite domains were still affected as the failed emails were categorized as bounces by SendGrid. The concern was reported to SendGrid to clear the bounce list and the issue was completely resolved on May 7th, 2019 at 15:05 UTC.

All emails that failed during the impacted time were sent again within the resolution time.

If you have any questions, please reach out to us on support@hackerearth.com.