How I created a custom email automation system for my SaaS

Last updated

Antonio Ufano avatar

Antonio Ufano

If you're building a SaaS, soon enough you're going to need an email automation system. Email is the main marketing and sales platform that you can use to increase growth and user retention for your product. In one of my previous articles, I explained a few tactics to increase user rentention, and one of them was about sending email reminders.

These reminders are usually sent when a user didn't complete certain action inside a timeframe in our app or when the user didn't reach a specific point of the customer journey. To send this reminders, you can use an email provider like Mailchimp, ActiveCampaign or SendGrid. The latest describes their email-automation system pretty well:

Automated email, also referred to as triggered email or behavior-driven email, is any message automatically sent from your email service provider (ESP) in direct response to an individual user’s specific actions made (or not made) on your website or web app.

Why I created a custom email automation system

First of all, building a custom email automation system can get very tricky. You have to deal with user events, schedule emails, unschedule them, control deliverabily etc... It's not a simple system and at the same time, it's a crucial part of your project as it'll help you keep your users engaged and grow.

If you're alredy paying a monthly subscription for MailChimp, SendGrid or any other email provider that includes an automation system, you can probably used their built in automations. In my case, I'm trying to keep the costs of running my project as low as possible so those are not an option for me at the moment. On top of that, I don't like the idea of my project to be so dependant on another system and also the prices usually increase a lot with the number of subscribers πŸ€‘

In addition, although some user events trigger emails inmediately (for example, send a welcome email upon sign up) there are a few cases that require very specific dates depending on the data entered by the user. I'm not sure if I'd be able to achive all my required scenarios by using an email automation system from a provider so that's the main reason I decided to build my own.

Overview of my custom email automation system

There are four key components in my custom email automation system:

  • Email provider: Obviously! We need to actually send emails so we need a provider with a good deliverability and, if possible, cheap πŸ€‘ The one that I'm currently using is MailerSend because it has the best free tier I've found, 12000 emails/month and after that it charges you $1 per 1000 emails.

  • User Events: These are triggered from the web application when a user completes certain actions. I've defined a limited number of user events (like signup, login, createTaks, etc) and they're stored in the application database along with the user id that triggered it.

  • Email schedule: I store the emails pending to be sent in a database collection. To add and remove documents from this collection, I have two cron jobs.

  • Cron jobs: I have two jobs to take care of the user events and the email schedule:

    • processEvents: this job reads events from the database and add or remove emails from the email schedule.
    • sendMails: this job reads pending emails from the schedule and sends the emails to the users.

How to create user events

I created an API that exposes and endpoint to which I can send a POST request with the event name. I'm using Strapi to create the API and, as it offers authentication controls and request body validations out of the box, I had little to code so that's awesome.

You can read more about Strapi in the article I wrote a few months ago: Building APIs FAST with Strapi, an overview

As the endpoint requires the user to be authenticated, using the JWT data from the request header I can obtain the user id which I'll include in the event stored in the database. This is an exmaple of a user event in the database:

{
  "_id": ObjectId("yyyyyzzzzzzyyyyyyzzzzzz"),
  "eventName": "signup",
  "hasEmails": true,
  "isProcessed": false,
  "createdAt": ISODate("2021-05-27T12:28:45.777Z"),
  "updatedAt": ISODate("2021-05-27T12:30:00.103Z"),
  "__v": 0,
  "user": ObjectId("xxxxxxkkkkkkxxxxxxkkkkk")
}

As you can see, I'm also including two bool flags:

  • isProcessed : indicates is the event has already been processed by the processEvents job.
  • hasEmails : indicates if this type of user event adds or removes emails from the schedule.

Job to process user events

As mentioned earlier, I'm using Strapi and it provides a super simple way to create cron jobs as well.

The job to process user events runs every five minutes and queries the user events database collection for events with flags isProcessed == false and hasEmails == true. I have a query limit as well to avoid processing lots of events in case there is a spike in user's activity.

  "*/5 * * * *": async () => {
    try {
      const userEvents = await strapi.services[`user-events`].find({
        hasEmails: true,
        isProcessed: false,
        _limit: 20,
      });

      if (!userEvents.length) {
        strapi.log.info("There are no userEvents to process");
        return;
      }
      strapi.log.info(`There are ${userEvents.length} userEvents to process`);
      for (const evt of userEvents) {
        strapi.log.info(`processing eventt ${evt.eventName}`);
        // utility function to actually process the events
        if (evt.hasEmails) await utils.processEventWithMails(evt);

        strapi.log.info(`Attempting to update event processed flag`);
        await strapi.services[`user-events`].update(
          { id: evt.id },
          { isProcessed: true }
        );
        strapi.log.info(`Event processed flag updated`);
      }
    } catch (error) {
      strapi.log.error(error);
      strapi.log.info(`Attempting to update event processed flag`);
      await strapi.services[`user-events`].update(
        { id: evt.id },
        { isProcessed: true }
      );
      strapi.log.info(`Event processed flag updated`);
    }
  },

I created an object that indicates which emails have to be added and deleted from the schedule by each event:

const eventsMailSchedule = {
  signup: {
    add: [
      { template: 'welcome', delayInDays: 0 },
      { template: 'createGoalsHabits1', delayInDays: 1 },
      { template: 'createGoalsHabits3', delayInDays: 3 },
      { template: 'createGoalsHabits7', delayInDays: 7 },
    ],
    delete: ['earlyAccess3', 'earlyAccess7'],
  },
  finishOnboard: {
    add: [],
    delete: [],
  },
  createFirstGH: {
    add: [
      { template: 'createFirstWeek1', delayInDays: 1 },
      { template: 'createFirstWeek3', delayInDays: 3 },
      { template: 'createFirstWeek7', delayInDays: 7 },
    ],
    delete: ['createGoalsHabits1', 'createGoalsHabits3', 'createGoalsHabits7'],
  },
  // More events...
}

The name of the events sent from the web application, matches with the keys from the object above so that I can just do eventsMailSchedule[EVENT_NAME] to obtain the emails to add and delete from the schedule in each case.

As you can see, for the emails that will be added to the schedule I've also included the delay in days, as some emails will be sent multiple days after the event was triggered πŸ˜‰

The utility functions processEventWithMails() and scheduleMailsForEvent() are the ones that actually handles the work:

  • processEventWithMails: retrieves the user information for each event and pass it to scheduleMailsForEvent()
  • scheduleMailsForEvent: deletes and adds event to the email schedule based on the event name.
/**
 * receives event object
 */
const processEventWithMails = async (event) => {
  // retrieves the user information
  const [user] = await strapi
    .query('user', 'users-permissions')
    .find({ id: event.user })
  if (!user) {
    strapi.log.info(`user not found :S`)
    return
  }
  strapi.log.debug(
    `Scheduling emails for event ${event.eventName} for user ${user.name}`
  )
  await scheduleMailsForEvent(event.eventName, user.name, user.email)

  return
}

/**
 * receives event name, user name and user email
 */
const scheduleMailsForEvent = async (eventName, name, email) => {
  let added = 0
  let deleted = 0
  try {
    strapi.log.info(`Checking mails to add and delete...`)

    if (!eventsMailSchedule[`${eventName}`]) {
      strapi.log.error('Invalid Event name')
      return
    }

    if (eventsMailSchedule[eventName].delete.length > 0) {
      strapi.log.debug(
        `Deleting ${eventsMailSchedule[eventName].delete.length} pending mails from schedule`
      )
      for (const del of eventsMailSchedule[eventName].delete) {
        const mailsToDelete = await strapi.services.email.find({
          error: false,
          email: email,
          template: del,
        })
        console.log('mailsToDelete :>> ', mailsToDelete)
        if (mailsToDelete.length) {
          strapi.log.info(
            `There are ${mailsToDelete.length} ${del} mails to un-schedule`
          )
          for (const pending of mailsToDelete) {
            await strapi.services.email.delete({ id: pending.id })
            deleted += 1
            strapi.log.info(`Email ${pending.id} deleted!`)
          }
        } else {
          strapi.log.info('There are NO mails to un-schedule')
        }
      }
    }

    if (eventsMailSchedule[eventName].add.length > 0) {
      strapi.log.debug(
        `There are ${eventsMailSchedule[eventName].add.length} emails to add to the schedule`
      )
      strapi.log.debug('Adding pending mails to schedule')
      for (const newMail of eventsMailSchedule[eventName].add) {
        let newEmailDoc = {
          name: name,
          email: email,
        }

        newEmailDoc.template = newMail.template
        newEmailDoc.sendDate = format(
          addDays(new Date(), newMail.delayInDays),
          'yyy-MM-dd'
        )
        console.log('newEmailDoc :>> ', newEmailDoc)
        const mailCreated = await strapi.services.email.create(newEmailDoc)
        added += 1
        strapi.log.info(`Email ${mailCreated.id} added!`)
      }
    } else {
      strapi.log.info('There are NO mails to schedule')
    }

    strapi.log.info(
      `🏁 Finished scheduling mails: TOTAL ${added} added and ${deleted} deleted`
    )
    return
  } catch (error) {
    strapi.log.error(error)
  }
}

For each email to be added, I insert a new record in the schedule (which is just another collection in the database) including the template to use, the date in which the email has to be sent and the name and email address of the user that will receive the email.

And finally, here is an example of an email document from the schedule:

{
  "_id": ObjectId("xxxxyyyyxxxxyyyyy"),
  "template": "createFirstWeek1",
  "error": false,
  "sendDate": "2021-06-02",
  "email": "cbuljujtwjtcpvthse@twzhhq.com",
  "name": "tester0712",
  "createdAt": ISODate("2021-05-26T10:12:00.055Z"),
  "updatedAt": ISODate("2021-06-02T12:44:02.373Z"),
  "__v": 0,
  "errorMessage": ""
}

If there are emails to be deleted from the schedule, I'll just query the collection by template and user email address and remove the results πŸ˜‰

Job to send emails

First I created all the HTML email templates that will be sent. Then I created another object to indicate which email template will be sent for each email and the subject to be used:

// HTML templates file locations
const MAIL_TEMPLATE_EARLYACCESS = 'config/templates/earlyAccess.html'
const MAIL_TEMPLATE_EARLYACCESS_3 = 'config/templates/earlyAccess3.html'
const MAIL_TEMPLATE_EARLYACCESS_7 = 'config/templates/earlyAccess7.html'
const MAIL_TEMPLATE_WELCOME = 'config/templates/welcome.html'
// more templates....

// read template files and cast them to strings
const earlyAccessTemplate = fs
  .readFileSync(MAIL_TEMPLATE_EARLYACCESS)
  .toString()
const earlyAccess3Template = fs
  .readFileSync(MAIL_TEMPLATE_EARLYACCESS_3)
  .toString()
const earlyAccess7Template = fs
  .readFileSync(MAIL_TEMPLATE_EARLYACCESS_7)
  .toString()
const welcomeTemplate = fs.readFileSync(MAIL_TEMPLATE_WELCOME).toString()
// more templates ....

const emails = {
  earlyAccess: {
    // contains the HTML as a string
    template: earlyAccessTemplate,
    subject: `πŸ—“ Your early access invite for theLIFEBOARD`,
  },
  earlyAccess3: {
    template: earlyAccess3Template,
    subject: `πŸ’¨ Start working on your goals an habits!`,
  },
  earlyAccess7: {
    template: earlyAccess7Template,
    subject: `πŸ“ˆ Plan your weeks and be more productive!`,
  },
  welcome: {
    template: welcomeTemplate,
    subject: `πŸ€™ A few tips to use theLIFEBOARD`,
  },
  // more emails...
}

With that in place, the only thing left is a job to actually send the emails. This job runs every 20 minutes and queries the email schedule collection and filter by sendDate and error == false. I've also limited this query to avoid going over my email provider free email tier πŸ€‘ For each email retrieved from the email schedule collection, it'll send the email using the template and subject from the object declared above.

Once the email is sent, the email will be deleted from the schedule. If there is an issue sending the email, the error flag will be updated to true and an error message will be saved so I can review it later.

Here's the code of the job:

// Runs every 20 minutes to send different types of emails in batches of 10
  "*/20 * * * *": async () => {

    let pendingEmails;
    const today = format(toDate(new Date()), "yyy-MM-dd");

    try {
      pendingEmails = await strapi.services.email.find({
        error: false,
        sendDate: today,
        _limit: 10,
      });
    } catch (error) {
      strapi.log.error("Unable to query for pending emails");
      return;
    }

    if (!pendingEmails.length) {
      strapi.log.info("There are NO pending emails to send");
      return;
    }
    strapi.log.info(
      `There are ${pendingEmails.length} emails to send ${today}`
    );

    // Uses files loaded at the beginning
    let selectedTemplate;

    for (const pending of pendingEmails) {
      strapi.log.info(
        `Attempting to send email of type ${pending.template} to ${pending.name}`
      );
      try {
        if (!emails[pending.template]) {
          throw new Error("Invalid template");
        }
        selectedTemplate = emails[pending.template].template;
        strapi.log.info(`Sending email template ${pending.template}`);
      } catch (error) {
        strapi.log.error(`Error processing mail ${pending.id}`);
        await strapi.services.email.update(
          { id: pending.id },
          { error: true, errorMessage: "Invalid template" }
        );
        strapi.log.info(`Email ${pending.id} updated to error!`);
        return;
      }
      try {
        await strapi.plugins["email"].services.email.send({
          to: pending.email,
          from: "theLIFEBOARD team<mymail@thelifeboard.app",
          // from: "theLIFEBOARD team",

          subject: emails[pending.template].subject,
          // replace variables from templates
          html: selectedTemplate
            .replace(/{{{username}}}/g, pending.name)
            .replace(/{{{useremail}}}/g, pending.email),
        });
      } catch (error) {
        strapi.log.error(`Error processing mail ${pending.id}`);
        await strapi.services.email.update(
          { id: pending.id },
          { error: true, errorMessage: error.message }
        );
        strapi.log.info(`Email ${pending.id} updated to error!`);
        return;
      }
      strapi.log.info(`Email ${pending.template} sent to ${pending.name}`);
      await strapi.services.email.delete({ id: pending.id });

      strapi.log.info(`Email ${pending.id} - ${pending.template} deleted!`);


    }
    strapi.log.info("All done");
  },

Conclusion

I'm still testing this email automation system but so far, it's working pretty good. The thing that I really like about this email automation system is that I can extend it easily. If I decide to create new a new event that triggers emails, I just have to create the HTML template and add the event information to the objects used in the cron jobs.

In addition, using an SMTP client to send the emails allows me to migrate to another provider if I need to by just changing the provider details in a config file. That's something I details in a previous article about how to send emails with any provider with Strapi.

Of course there is room for improvement. I still have to make sure that the systems works well under load and I could move the objects that contain the event names, templates and subjects to the database as well. I'll have to improve error handling but I'll leave that for a future update.

Should you create your own automation email system? Probably not πŸ˜… There are tons of providers that offer automations but in my case, I wanted to keep my project costs as low as possible and, to be honest, I'm always looking for fun things to code and this was a nice challenge. If you decide to build it, feel free to reach me on Twitter and let me know how it goes or if you need any help.

Hope you find useful.

Happy coding!

If you enjoyed this article consider sharing it on social media or buying me a coffee ✌️

Oh! and don't forget to follow me on Twitter where I share tons of dev tips πŸ€™

Other articles that might help you

my projects

Apart from writing articles in this blog, I spent most of my time working on my personal projects.

lifeboard.app logo

theLIFEBOARD.app

theLIFEBOARD is a weekly planner that helps people achieve their goals, create new habits and avoid burnout. It encourages you to plan and review each week so you can easily identify ways to improve your productivity while keeping track of your progress.

Request early access
quicktalks.io logo

Quicktalks.io

Quicktalks is a place where indie hackers, makers, creators and entrepreneurs share their knowledge, ideas, lessons learned, failures and tactics they use to build successfull online products and businesses. It'll contain recorded short interviews with indie makers.

Message me to be part of it