Background services

Some work belongs to no request. When a customer reserves seats and never pays, those seats stay blocked. Someone has to release them, every minute, whether anyone visits the site or not. And confirmation emails shouldn't keep the checkout waiting for the mail server.

Background services run inside serve, next to your HTTP servers. They start together and stop together.

Scheduled tasks

#[scheduled_with_tick_timer] runs its #[process] method on an interval. The interval is a Duration constant, imported by its path, and the behavior decides what happens when a tick runs late:

use std::time::Duration;

pub const RELEASE_INTERVAL: Duration = Duration::from_secs(60);

To find the expired reservations fast, orders get an index by status and reservation time:

#[model(table = "orders")]
#[index(name = "orders_by_customer", fields = [customer, id])]
#[index(name = "orders_by_concert", fields = [concert, id])]
#[index(name = "orders_by_status", fields = [status, reserved_until])]
#[has_many(name = "tickets", model = Ticket, key = order)]
#[derive(Clone)]
pub struct Order {

The task deletes every reserved order whose time ran out, in one go. Their tickets go with them, thanks to OnDelete::Cascade, and the seats are free again:

#[scheduled_with_tick_timer(interval = RELEASE_INTERVAL, behavior = MissedTickBehavior::Skip)]
pub struct ReleaseExpiredReservations {
    clock: Arc<Clock>,
    database: Arc<Database>,
}

impl ReleaseExpiredReservations {
    #[constructor]
    pub fn create(clock: Arc<Clock>, database: Arc<Database>) -> anyhow::Result<Self> {
        Ok(Self { clock, database })
    }

    #[process]
    pub async fn run(&self) -> anyhow::Result<()> {
        let released = Order::query()
            .status
            .eq(OrderStatus::Reserved)
            .reserved_until
            .below(self.clock.now())
            .delete(self.database.as_ref())
            .await?;

        if released > 0 {
            println!("Released {released} expired reservations.");
        }

        Ok(())
    }
}

The first tick runs as soon as serve starts. Add the time feature to tokio for MissedTickBehavior.

An outbox for emails

The checkout should confirm every order by email, but the mail server can be slow, or down. So the checkout doesn't send anything itself. It writes the email into an outbox table, in the same transaction as the order, and a background service sends it. Either both the order and its email exist, or neither does:

#[model(table = "outgoing_emails")]
#[index(name = "outgoing_emails_due", fields = [send_after, id])]
pub struct OutgoingEmail {
    #[column(primary_key, default = ColumnDefault::UuidV7)]
    pub id: Uuid,
    #[column]
    pub recipient: String,
    #[column]
    pub subject: String,
    #[column]
    pub body: String,
    #[column]
    pub send_after: DateTime<Utc>,
}

        let order_url = addressed(routes.public.show_order(order.id.to_string()))?.url();

        OutgoingEmail::create(OutgoingEmailDraft {
            recipient: customer.email,
            subject: format!("Your tickets for {}", concert.artist),
            body: format!("Pay within 15 minutes to keep your seats: {order_url}"),
            send_after: now,
        })
        .run(&transaction)
        .await?;
        transaction.commit().await?;
        self.outbox.wake();

The PlaceOrder route passes its &Routes to reserve, so the email can link to the order page. After the commit, the repository wakes the Outbox, a singleton around a Notify:

#[singleton]
pub struct Outbox {
    filled: Notify,
}

impl Outbox {
    #[constructor]
    pub fn create() -> anyhow::Result<Self> {
        Ok(Self {
            filled: Notify::new(),
        })
    }

    pub fn wake(&self) {
        self.filled.notify_one();
    }

    pub async fn woken(&self) {
        self.filled.notified().await;
    }
}

Services

A #[service] runs until it's asked to stop. The MailSender sends whatever is due, then waits until it's woken, until the next poll, or until serve stops:

    #[process]
    pub async fn run(&self, cancellation_token: CancellationToken) -> anyhow::Result<()> {
        loop {
            self.send_due().await?;

            tokio::select! {
                () = cancellation_token.cancelled() => return Ok(()),
                () = self.outbox.woken() => {}
                () = tokio::time::sleep(OUTBOX_POLL_INTERVAL) => {}
            }
        }
    }

    async fn send_due(&self) -> anyhow::Result<()> {
        let database = self.database.as_ref();
        let due = OutgoingEmail::query()
            .send_after
            .at_most(self.clock.now())
            .ascending()
            .limit::<20>()
            .fetch(database)
            .await?
            .records;

        for email in due {
            let Removal::Removed(claimed) = OutgoingEmail::query()
                .id
                .eq(email.id)
                .delete(database)
                .await?
            else {
                continue;
            };
            let sent = self
                .mailer
                .send(&Email {
                    to: claimed.recipient.clone(),
                    subject: claimed.subject.clone(),
                    body: claimed.body.clone(),
                })
                .await;

            if let Err(error) = sent {
                eprintln!(
                    "Couldn't email {}, retrying in a minute: {error}",
                    claimed.recipient
                );

                OutgoingEmail {
                    send_after: self.clock.now() + TimeDelta::minutes(1),
                    ..claimed
                }
                .insert()
                .run(database)
                .await?;
            }
        }

        Ok(())
    }

Deleting an email by its primary key answers with Removal::Removed, holding the deleted row, or Removal::Missing when someone else got there first. That's how several instances of the box office share one outbox: whoever removes an email sends it, and only once. A failed send puts the email back, one minute later.

The MailSender reaches the Mailer, so serve now asks for the mail settings from Configuration:

$ box_office serve --mail-api-url https://mail.example.com/send --mail-sender tickets@example.com ...

Several instances

Every instance of the box office runs every background service:

• Releasing expired reservations is the same delete on every instance, so running it twice is harmless.

• The outbox hands each email to exactly one instance.

• The Outbox only wakes the instance that took the order. The others pick up emails on their next poll, at most 30 seconds later.

Failures and shutdown

When a service stops or a tick fails, serve shuts everything down, so your process manager can restart it cleanly. That's why a service keeps running until its token is cancelled, and returns errors only for problems it can't recover from.

On Ctrl+C or SIGTERM, every service is cancelled and given time to finish.