Background Jobs

Some work should not happen while a user waits: sending a batch of email, regenerating images, calling a slow third-party API. A job is that work, queued and run later.

Availability

Define a job

import { defineJob } from "nextly";

export const sendDigest = defineJob({
  slug: "send-digest",
  handler: async (input, context) => {
    // `input` is the payload the job was enqueued with. `context` is what the
    // runner tells the handler about the run it is in.
  },
});

defineJob only builds the definition. Register it in your config, or nothing will run it:

import { defineConfig } from "nextly";

import { sendDigest } from "./jobs/send-digest";

export default defineConfig({
  jobs: [sendDigest],
});

jobs takes untyped definitions, so a job written with a type argument, defineJob<{ ... }>, is a type error there and fails next build. Leave the type argument off and check input at the top of the handler, as the samples below do. The payload comes from whoever queued the job, so checking it is worth doing anyway.

Retries and backoff

A failed job is retried with exponential backoff, starting at 1 second and capped at 1 hour, with jitter so a batch of simultaneous failures does not retry in lockstep.

attemptCount counts the attempt that just failed, so a job with a budget of three attempts gives up when the third fails — not after a fourth. That boundary is the usual bug in retry code, so it is worth stating rather than leaving to be discovered.

Who a job runs as

A job is not run with the authority of whoever enqueued it. It resolves a run-as user, and that identity is what access control sees while the job runs, so a job cannot quietly do more than the user it runs as is allowed to do.

Running the queue

Queuing a job records it. Nothing runs it until something calls the jobs route, so a Nextly installation with no scheduler has a queue that fills and never drains.

The scaffolded apps mount Nextly's handlers at src/app/admin/api/[[...params]]/route.ts, so the URL is:

POST /admin/api/jobs/run

If you mounted the handlers somewhere else, use that path instead.

Authenticating the caller

The route is not open. A scheduler has no session, so it authenticates with a shared secret: set NEXTLY_DRAIN_SECRET (or CRON_SECRET) to a value of at least 32 characters, and send it as a bearer token.

POST /admin/api/jobs/run
Authorization: Bearer $NEXTLY_DRAIN_SECRET

The two variables fail differently when the value is too short, and knowing which one you set decides where to look:

CRON_SECRET is Vercel Cron's convention, which is why it is accepted at all and why it is tolerated rather than rejected.

A signed-in human can pull the trigger instead, with the manage-background-jobs permission and a POST. GET always requires the secret, because a permission check alone would let a browser be steered into firing it.

A pass is bounded. It claims at most 10 queued jobs, and it stops starting new ones after 20 seconds. That 20 seconds is a budget for starting work, not a ceiling on the request: the check happens before each claim, so a handler already running is awaited to completion and the call can overrun. Size any serverless timeout for your slowest handler, not for 20 seconds.

The queue is durable, so a pass that stops early loses no work: the next call continues from the same table.

Writing a handler that can run twice

Every handler is told the id of the job it is running, and that id is the same on every attempt at it. It is the key to hand anything the database cannot take back:

import { defineJob } from "nextly";

export const chargeInvoice = defineJob({
  slug: "charge-invoice",
  async handler(input, { jobId }) {
    const invoiceId = typeof input?.invoiceId === "string" ? input.invoiceId : null;
    const cents = typeof input?.cents === "number" ? input.cents : null;
    if (invoiceId === null || cents === null) {
      throw new Error("charge-invoice needs { invoiceId: string, cents: number }");
    }

    // The provider de-duplicates on this key, so a second run of the same job
    // returns the first charge instead of making a second one.
    const response = await fetch("https://api.payments.example/charges", {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        "Idempotency-Key": jobId,
      },
      body: JSON.stringify({ invoiceId, amountCents: cents }),
    });

    // `fetch` resolves for a 429 or a 500. Returning here would finish the job
    // as done and the charge would never be made. Throw, so the runner records
    // a failed attempt and the backoff policy retries it.
    if (!response.ok) {
      throw new Error(
        `charge for ${invoiceId} failed: ${String(response.status)}`
      );
    }
  },
});

Where to put the key depends on the provider. Stripe and Adyen read an Idempotency-Key request header; Square takes an idempotency_key field in the request body. Check your provider's own documentation rather than assuming the header. Where the side effect is a row rather than a call, the same idea is an upsert against a unique column holding the key, so the second attempt collides instead of inserting.

One key per side effect, not one key per job. A handler that charges a card and then posts to a ledger must not send the same jobId to both if they share a uniqueness namespace, or the second is rejected as a duplicate of the first. Derive one key per operation:

import { defineJob } from "nextly";

export const settleInvoice = defineJob({
  slug: "settle-invoice",
  async handler(input, { jobId }) {
    const invoiceId = typeof input?.invoiceId === "string" ? input.invoiceId : null;
    if (invoiceId === null) throw new Error("settle-invoice needs { invoiceId: string }");

    await charge(`${jobId}:charge`, invoiceId);
    await postToLedger(`${jobId}:ledger`, invoiceId);
  },
});

declare function charge(key: string, invoiceId: string): Promise<void>;
declare function postToLedger(key: string, invoiceId: string): Promise<void>;

Deriving the key from the input instead works only while the input is unique, and that is a property of whoever queued the job rather than of the queue. The same payload queued twice is two jobs with two ids, and both are meant to happen.

The context also carries attempt, counting from 1. It is written to the row before the handler starts, so a handler that dies part-way still leaves the count advanced and the next run is numbered higher. Use it for behaviour that should change on a retry, such as logging more loudly, or giving up on an optimisation that failed last time.

It still does not tell you whether a side effect landed. attempt > 1 says an earlier run began, not what it finished, and nothing inside this process can know what a provider did with a request it never answered. That is the question jobId answers by asking the provider.

The lease

The lease is renewed for you every third of its length while a handler is alive, so ordinary long work does not expire it. What it bounds is a runner that stops renewing: a process that stalls, is paused, or loses its connection.

The built-in route uses the fixed 30 second default and takes no configuration. A project running its own pass can call runJobsPass directly and pass leaseMs, which is worth raising where an event loop pause or a slow database can plausibly exceed 30 seconds, because the renewal timer cannot fire during a stall either.

Retention

Finished jobs are pruned on a schedule, so the table holds recent history rather than everything that has ever run.

Related

• Webhooks — telling another system that something changed
• Releases — scheduling one depends on this queue draining
• Audit log