Batching jobs via fan-out

Sometimes you need to reliably trigger hundreds or thousands of jobs at once, making sure that every job succeeds independently. For example, you might have a cron which triggers importing thousands of products, or a user might require 100 different OpenAPI calls at once.

You might start by running everything in a single function, or within steps. However, as the number of jobs grows this becomes slower, more difficult to observe, and more prone to temporary failures. Instead of doing everything in a single job you should fan-out by sending events to trigger new jobs.

How to implement this pattern

In order to manage thousands of jobs at once you'll need to fan-out to run functions in parallel. The fan-out pattern works as follows:

• The job trigger runs as a single function (eg. a cron for triggering product imports)
• The function loads the data needed for all future jobs (eg. loading all product IDs)
• It sends new events to trigger jobs in parallel for each job needed.

When you schedule each job independently by sending events, each job gets its own retries and runs in parallel. This makes your system faster, more observable, and more resilient to temporary failures.

How to implement with Inngest

Inngest supports scheduled functions and event-triggered functions. Combining the two enables you to fan-out functions to run in parallel. We'll define two these two functions:

typescript
import { inngest } from "./client";
const inngest = new Inngest({ id: "scheduling-backend" });
// A scheduled function uses the current time to find notifications to send
const slackCron = inngest.createFunction(
  { id: "slack-notification-cron" },
  { cron: "0 9,12 * * MON,FRI" },
  async () => {
    const notifications = await getNotificationsToRun();
    const events = notifications.map((notification) => ({
      name: "app/notification.dispatched",
      data: { notification },
    }));
    // Send an array of events to Inngest, triggering many jobs in parallel.
    await inngest.send(events);
    return `${notifications.length} notifications dispatched`;
  }
);
// A function runs for every app/notification.dispatched event to
// post the notification to Slack
const postSlackNotification = inngest.createFunction(
  { id: "send-slack-notification" },
  { event: "app/notification.dispatched" },
  async ({ event }) => {
    const reportData = getAccountReportData(event.data.notification.accountId);
    await app.client.chat.postMessage({
      channel: event.data.notification.slackChannelId,
      blocks: generateReportSlackBlocks(reportData),
      // ...
    });
  }
);

This is the system — both functions can even be defined in the same file to keep things simple and maintainable. This approach works well for systems that have commonly scheduled times, but for more flexible systems that are scheduled as one-off, non-repeated tasks you should review Patterns: Running at specific times.

Additional Resources

• Patterns: Running at specific times
• Patterns: Cancelling scheduled functions
• Docs: Writing Functions