How I created a custom email automation system for my SaaS
Last updated
Antonio Ufano
If you're building a SaaS, soon enough you're going to need an email automation system. Email is the main marketing and sales platform that you can use to increase growth and user retention for your product. In one of my previous articles, I explained a few tactics to increase user rentention, and one of them was about sending email reminders.
These reminders are usually sent when a user didn't complete certain action inside a timeframe in our app or when the user didn't reach a specific point of the customer journey. To send this reminders, you can use an email provider like Mailchimp, ActiveCampaign or SendGrid. The latest describes their email-automation system pretty well:
Automated email, also referred to as triggered email or behavior-driven email, is any message automatically sent from your email service provider (ESP) in direct response to an individual user’s specific actions made (or not made) on your website or web app.
Why I created a custom email automation system
First of all, building a custom email automation system can get very tricky. You have to deal with user events, schedule emails, unschedule them, control deliverabily etc... It's not a simple system and at the same time, it's a crucial part of your project as it'll help you keep your users engaged and grow.
If you're alredy paying a monthly subscription for MailChimp, SendGrid or any other email provider that includes an automation system, you can probably use their built in automations. In my case, I'm trying to keep the costs of running my project as low as possible so those are not an option for me at the moment. On top of that, I don't like the idea of my project to be so dependant on another system and also the prices usually increase a lot with the number of subscribers 🤑
In addition, although some user events trigger emails inmediately (for example, send a welcome email upon sign up) there are a few cases that require very specific dates depending on the data entered by the user. I'm not sure if I'd be able to achive all my required scenarios by using an email automation system from a provider so that's the main reason I decided to build my own.
Overview of my custom email automation system
There are four key components in my custom email automation system:
-
Email provider: Obviously! We need to actually send emails so we need a provider with a good deliverability and, if possible, cheap 🤑 The one that I'm currently using is MailerSend because it has the best free tier I've found, 12000 emails/month and after that it charges you $1 per 1000 emails.
-
User Events: These are triggered from the web application when a user completes certain actions. I've defined a limited number of user events (like signup, login, createTaks, etc) and they're stored in the application database along with the user id that triggered it.
-
Email schedule: I store the emails pending to be sent in a database collection. To add and remove documents from this collection, I have two cron jobs.
-
Cron jobs: I have two jobs to take care of the user events and the email schedule:
- processEvents: this job reads events from the database and add or remove emails from the email schedule.
- sendMails: this job reads pending emails from the schedule and sends the emails to the users.
How to create user events
I created an API that exposes and endpoint to which I can send a POST request with the event name. I'm using Strapi to create the API and, as it offers authentication controls and request body validations out of the box, I had little to code so that's awesome.
You can read more about Strapi in the article I wrote a few months ago: Building APIs FAST with Strapi, an overview
As the endpoint requires the user to be authenticated, using the JWT data from the request header I can obtain the user id which I'll include in the event stored in the database. This is an exmaple of a user event in the database:
{
"_id": ObjectId("yyyyyzzzzzzyyyyyyzzzzzz"),
"eventName": "signup",
"hasEmails": true,
"isProcessed": false,
"createdAt": ISODate("2021-05-27T12:28:45.777Z"),
"updatedAt": ISODate("2021-05-27T12:30:00.103Z"),
"__v": 0,
"user": ObjectId("xxxxxxkkkkkkxxxxxxkkkkk")
}
As you can see, I'm also including two bool flags:
- isProcessed : indicates is the event has already been processed by the processEvents job.
- hasEmails : indicates if this type of user event adds or removes emails from the schedule.
Job to process user events
As mentioned earlier, I'm using Strapi and it provides a super simple way to create cron jobs as well.
The job to process user events runs every five minutes and queries the user events database collection for events with flags isProcessed == false and hasEmails == true. I have a query limit as well to avoid processing lots of events in case there is a spike in user's activity.
"*/5 * * * *": async () => {
try {
const userEvents = await strapi.services[`user-events`].find({
hasEmails: true,
isProcessed: false,
_limit: 20,
});
if (!userEvents.length) {
strapi.log.info("There are no userEvents to process");
return;
}
strapi.log.info(`There are ${userEvents.length} userEvents to process`);
for (const evt of userEvents) {
strapi.log.info(`processing eventt ${evt.eventName}`);
// utility function to actually process the events
if (evt.hasEmails) await utils.processEventWithMails(evt);
strapi.log.info(`Attempting to update event processed flag`);
await strapi.services[`user-events`].update(
{ id: evt.id },
{ isProcessed: true }
);
strapi.log.info(`Event processed flag updated`);
}
} catch (error) {
strapi.log.error(error);
strapi.log.info(`Attempting to update event processed flag`);
await strapi.services[`user-events`].update(
{ id: evt.id },
{ isProcessed: true }
);
strapi.log.info(`Event processed flag updated`);
}
},
I created an object that indicates which emails have to be added and deleted from the schedule by each event:
const eventsMailSchedule = {
signup: {
add: [
{ template: 'welcome', delayInDays: 0 },
{ template: 'createGoalsHabits1', delayInDays: 1 },
{ template: 'createGoalsHabits3', delayInDays: 3 },
{ template: 'createGoalsHabits7', delayInDays: 7 },
],
delete: ['earlyAccess3', 'earlyAccess7'],
},
finishOnboard: {
add: [],
delete: [],
},
createFirstGH: {
add: [
{ template: 'createFirstWeek1', delayInDays: 1 },
{ template: 'createFirstWeek3', delayInDays: 3 },
{ template: 'createFirstWeek7', delayInDays: 7 },
],
delete: ['createGoalsHabits1', 'createGoalsHabits3', 'createGoalsHabits7'],
},
// More events...
}
The name of the events sent from the web application, matches with the keys from the object above so that I can just do eventsMailSchedule[EVENT_NAME] to obtain the emails to add and delete from the schedule in each case.
As you can see, for the emails that will be added to the schedule I've also included the delay in days, as some emails will be sent multiple days after the event was triggered 😉
The utility functions processEventWithMails() and scheduleMailsForEvent() are the ones that actually handles the work:
- processEventWithMails: retrieves the user information for each event and pass it to scheduleMailsForEvent()
- scheduleMailsForEvent: deletes and adds event to the email schedule based on the event name.
/**
* receives event object
*/
const processEventWithMails = async (event) => {
// retrieves the user information
const [user] = await strapi
.query('user', 'users-permissions')
.find({ id: event.user })
if (!user) {
strapi.log.info(`user not found :S`)
return
}
strapi.log.debug(
`Scheduling emails for event ${event.eventName} for user ${user.name}`
)
await scheduleMailsForEvent(event.eventName, user.name, user.email)
return
}
/**
* receives event name, user name and user email
*/
const scheduleMailsForEvent = async (eventName, name, email) => {
let added = 0
let deleted = 0
try {
strapi.log.info(`Checking mails to add and delete...`)
if (!eventsMailSchedule[`${eventName}`]) {
strapi.log.error('Invalid Event name')
return
}
if (eventsMailSchedule[eventName].delete.length > 0) {
strapi.log.debug(
`Deleting ${eventsMailSchedule[eventName].delete.length} pending mails from schedule`
)
for (const del of eventsMailSchedule[eventName].delete) {
const mailsToDelete = await strapi.services.email.find({
error: false,
email: email,
template: del,
})
console.log('mailsToDelete :>> ', mailsToDelete)
if (mailsToDelete.length) {
strapi.log.info(
`There are ${mailsToDelete.length} ${del} mails to un-schedule`
)
for (const pending of mailsToDelete) {
await strapi.services.email.delete({ id: pending.id })
deleted += 1
strapi.log.info(`Email ${pending.id} deleted!`)
}
} else {
strapi.log.info('There are NO mails to un-schedule')
}
}
}
if (eventsMailSchedule[eventName].add.length > 0) {
strapi.log.debug(
`There are ${eventsMailSchedule[eventName].add.length} emails to add to the schedule`
)
strapi.log.debug('Adding pending mails to schedule')
for (const newMail of eventsMailSchedule[eventName].add) {
let newEmailDoc = {
name: name,
email: email,
}
newEmailDoc.template = newMail.template
newEmailDoc.sendDate = format(
addDays(new Date(), newMail.delayInDays),
'yyy-MM-dd'
)
console.log('newEmailDoc :>> ', newEmailDoc)
const mailCreated = await strapi.services.email.create(newEmailDoc)
added += 1
strapi.log.info(`Email ${mailCreated.id} added!`)
}
} else {
strapi.log.info('There are NO mails to schedule')
}
strapi.log.info(
`🏁 Finished scheduling mails: TOTAL ${added} added and ${deleted} deleted`
)
return
} catch (error) {
strapi.log.error(error)
}
}
For each email to be added, I insert a new record in the schedule (which is just another collection in the database) including the template to use, the date in which the email has to be sent and the name and email address of the user that will receive the email.
And finally, here is an example of an email document from the schedule:
{
"_id": ObjectId("xxxxyyyyxxxxyyyyy"),
"template": "createFirstWeek1",
"error": false,
"sendDate": "2021-06-02",
"email": "cbuljujtwjtcpvthse@twzhhq.com",
"name": "tester0712",
"createdAt": ISODate("2021-05-26T10:12:00.055Z"),
"updatedAt": ISODate("2021-06-02T12:44:02.373Z"),
"__v": 0,
"errorMessage": ""
}
If there are emails to be deleted from the schedule, I'll just query the collection by template and user email address and remove the results 😉
Job to send emails
First I created all the HTML email templates that will be sent. Then I created another object to indicate which email template will be sent for each email and the subject to be used:
// HTML templates file locations
const MAIL_TEMPLATE_EARLYACCESS = 'config/templates/earlyAccess.html'
const MAIL_TEMPLATE_EARLYACCESS_3 = 'config/templates/earlyAccess3.html'
const MAIL_TEMPLATE_EARLYACCESS_7 = 'config/templates/earlyAccess7.html'
const MAIL_TEMPLATE_WELCOME = 'config/templates/welcome.html'
// more templates....
// read template files and cast them to strings
const earlyAccessTemplate = fs
.readFileSync(MAIL_TEMPLATE_EARLYACCESS)
.toString()
const earlyAccess3Template = fs
.readFileSync(MAIL_TEMPLATE_EARLYACCESS_3)
.toString()
const earlyAccess7Template = fs
.readFileSync(MAIL_TEMPLATE_EARLYACCESS_7)
.toString()
const welcomeTemplate = fs.readFileSync(MAIL_TEMPLATE_WELCOME).toString()
// more templates ....
const emails = {
earlyAccess: {
// contains the HTML as a string
template: earlyAccessTemplate,
subject: `🗓 Your early access invite for theLIFEBOARD`,
},
earlyAccess3: {
template: earlyAccess3Template,
subject: `💨 Start working on your goals an habits!`,
},
earlyAccess7: {
template: earlyAccess7Template,
subject: `📈 Plan your weeks and be more productive!`,
},
welcome: {
template: welcomeTemplate,
subject: `🤙 A few tips to use theLIFEBOARD`,
},
// more emails...
}
With that in place, the only thing left is a job to actually send the emails. This job runs every 20 minutes and queries the email schedule collection and filter by sendDate and error == false. I've also limited this query to avoid going over my email provider free email tier 🤑 For each email retrieved from the email schedule collection, it'll send the email using the template and subject from the object declared above.
Once the email is sent, the email will be deleted from the schedule. If there is an issue sending the email, the error flag will be updated to true and an error message will be saved so I can review it later.
Here's the code of the job:
// Runs every 20 minutes to send different types of emails in batches of 10
"*/20 * * * *": async () => {
let pendingEmails;
const today = format(toDate(new Date()), "yyy-MM-dd");
try {
pendingEmails = await strapi.services.email.find({
error: false,
sendDate: today,
_limit: 10,
});
} catch (error) {
strapi.log.error("Unable to query for pending emails");
return;
}
if (!pendingEmails.length) {
strapi.log.info("There are NO pending emails to send");
return;
}
strapi.log.info(
`There are ${pendingEmails.length} emails to send ${today}`
);
// Uses files loaded at the beginning
let selectedTemplate;
for (const pending of pendingEmails) {
strapi.log.info(
`Attempting to send email of type ${pending.template} to ${pending.name}`
);
try {
if (!emails[pending.template]) {
throw new Error("Invalid template");
}
selectedTemplate = emails[pending.template].template;
strapi.log.info(`Sending email template ${pending.template}`);
} catch (error) {
strapi.log.error(`Error processing mail ${pending.id}`);
await strapi.services.email.update(
{ id: pending.id },
{ error: true, errorMessage: "Invalid template" }
);
strapi.log.info(`Email ${pending.id} updated to error!`);
return;
}
try {
await strapi.plugins["email"].services.email.send({
to: pending.email,
from: "theLIFEBOARD team<mymail@thelifeboard.app",
// from: "theLIFEBOARD team",
subject: emails[pending.template].subject,
// replace variables from templates
html: selectedTemplate
.replace(/{{{username}}}/g, pending.name)
.replace(/{{{useremail}}}/g, pending.email),
});
} catch (error) {
strapi.log.error(`Error processing mail ${pending.id}`);
await strapi.services.email.update(
{ id: pending.id },
{ error: true, errorMessage: error.message }
);
strapi.log.info(`Email ${pending.id} updated to error!`);
return;
}
strapi.log.info(`Email ${pending.template} sent to ${pending.name}`);
await strapi.services.email.delete({ id: pending.id });
strapi.log.info(`Email ${pending.id} - ${pending.template} deleted!`);
}
strapi.log.info("All done");
},
Conclusion
I'm still testing this email automation system but so far, it's working pretty good. The thing that I really like about this email automation system is that I can extend it easily. If I decide to create new a new event that triggers emails, I just have to create the HTML template and add the event information to the objects used in the cron jobs.
In addition, using an SMTP client to send the emails allows me to migrate to another provider if I need to by just changing the provider details in a config file. That's something I detailed in a previous article about how to send emails with any provider with Strapi.
Of course there is room for improvement. I still have to make sure that the systems works well under load and I could move the objects that contain the event names, templates and subjects to the database as well. I'll have to improve error handling but I'll leave that for a future update.
Should you create your own automation email system? Probably not 😅 There are tons of providers that offer automations but in my case, I wanted to keep my project costs as low as possible and, to be honest, I'm always looking for fun things to code and this was a nice challenge. If you decide to build it, feel free to reach me on Twitter and let me know how it goes or if you need any help.
Hope you find this useful.
Happy coding!
If you enjoyed this article consider sharing it on social media or buying me a coffee ✌️
Oh! and don't forget to follow me on Twitter where I share tons of dev tips 🤙
Other articles that might help you
my projects
Apart from writing articles in this blog, I spent most of my time working on my personal projects.
theLIFEBOARD.app
theLIFEBOARD is a weekly planner that helps people achieve their goals, create new habits and avoid burnout. It encourages you to plan and review each week so you can easily identify ways to improve your productivity while keeping track of your progress.
Sign up
SolidityTips.com
I'm very interested in blockchain, smart contracts and all the possiblilities chains like Ethereum can bring to the web. SolidityTips is a blog in which I share everything I learn about Solidity and Web3 development.
Check it out if you want to learn Solidity
Quicktalks.io
Quicktalks is a place where indie hackers, makers, creators and entrepreneurs share their knowledge, ideas, lessons learned, failures and tactics they use to build successfull online products and businesses. It'll contain recorded short interviews with indie makers.
Message me to be part of it