tabshift/payloadcms
1
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

feat: crons for all bin scripts, new jobs:handle-schedules script, more reliable job system crons (#13564)

Browse Source 
## New jobs:handle-schedules bin script

Similarly to `payload jobs:run`, this PR adds a new
`jobs:handle-schedules` bin script which only handles scheduling.

## Allows jobs:run bin script to handle scheduling

Similarly to how [payload
autoRun](https://payloadcms.com/docs/jobs-queue/queues#cron-jobs)
handles both running and scheduling jobs by default, you can now set the
`payload jobs:run` bin script to also handle scheduling. This is opt-in:

```sh
pnpm payload jobs:run --cron "*/5 * * * *" --queue myQueue --handle-schedules # This will both schedule jobs according to the configuration and run them
```

## Cron schedules for all bin scripts

Previously, only the `payload jobs:run` bin script accepted a cron flag.
The `payload jobs:handle-schedules` would have required the same logic
to also handle a cron flag.

Instead of opting for this duplicative logic, I'm now handling cron
logic before we determine which script to run. This means: it's simpler
and requires less duplicative code.

**This allows all other bin scripts (including custom ones) to use the
`--cron` flag**, enabling cool use-cases like scheduling your own custom
scripts - no additional config required!

Example:

```sh
pnpm payload run ./myScript.ts --cron "0 * * * *"
```

Video Example:


https://github.com/user-attachments/assets/4ded738d-2ef9-43ea-8136-f47f913a7ba8

## More reliable job system crons

When using autorun or `--cron`, if one cron run takes longer than the
cron interval, the second cron would run before the first one finishes.

This can be especially dangerous when running jobs using a bin script,
potentially causing race conditions, as the first cron run will take
longer due to payload initialization overhead (only for first cron run,
consecutive ones use cached payload). Now, consecutive cron runs will
wait for the first one to finish by using the `{ protect: true }`
property of Croner.

This change will affect both autorun and bin scripts.

## Cleanup

- Centralized payload instance cleanup (payload.destroy()) for all bin
scripts
- The `getPayload` function arguments were not properly typed. Arguments
like `disableOnInit: true` are already supported, but the type did not
reflect that. This simplifies the type and makes it more accurate.

## Fixes

- `allQueues` argument for `payload jobs:run` was not respected


---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
  - https://app.asana.com/0/0/1211124797199077
This commit is contained in:
Alessio Gravili
2025-09-15 10:15:50 -07:00
committed by GitHub
parent b34e5eadf4
commit cdeb828971
5 changed files with 156 additions and 70 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
docs/configuration/overview.mdx
View File

@@ -319,3 +319,13 @@ Now you can run the command using:
```sh
pnpm payload seed
```
## Running bin scripts on a schedule
Every bin script supports being run on a schedule using cron syntax. Simply pass the `--cron` flag followed by the cron expression when running the script. Example:
```sh
pnpm payload run ./myScript.ts --cron "0 * * * *"
```
This will use the `run` bin script to execute the specified script on the defined schedule.

14
docs/jobs-queue/queues.mdx
View File

@@ -173,25 +173,31 @@ const results = await payload.jobs.runByID({
Finally, you can process jobs via the bin script that comes with Payload out of the box. By default, this script will run jobs from the `default` queue, with a limit of 10 jobs per invocation:
```sh
npx payload jobs:run
pnpm payload jobs:run
```
You can override the default queue and limit by passing the `--queue` and `--limit` flags:
```sh
npx payload jobs:run --queue myQueue --limit 15
pnpm payload jobs:run --queue myQueue --limit 15
```
If you want to run all jobs from all queues, you can pass the `--all-queues` flag:
```sh
npx payload jobs:run --all-queues
pnpm payload jobs:run --all-queues
```
In addition, the bin script allows you to pass a `--cron` flag to the `jobs:run` command to run the jobs on a scheduled, cron basis:
```sh
npx payload jobs:run --cron "*/5 * * * *"
pnpm payload jobs:run --cron "*/5 * * * *"
```
You can also pass `--handle-schedules` flag to the `jobs:run` command to make it schedule jobs according to configured schedules:
```sh
pnpm payload jobs:run --cron "*/5 * * * *" --queue myQueue --handle-schedules # This will both schedule jobs according to the configuration and run them
```
## Processing Order

14
docs/jobs-queue/schedules.mdx
View File

@@ -35,6 +35,20 @@ Something needs to actually trigger the scheduling of jobs (execute the scheduli
You can disable this behavior by setting `disableScheduling: true` in your `autorun` configuration, or by passing `disableScheduling=true` to the `/api/payload-jobs/run` endpoint. This is useful if you want to handle scheduling manually, for example, by using a cron job or a serverless function that calls the `/api/payload-jobs/handle-schedules` endpoint or the `payload.jobs.handleSchedules()` local API method.
### Bin Scripts
Payload provides a set of bin scripts that can be used to handle schedules. If you're already using the `jobs:run` bin script, you can set it to also handle schedules by passing the `--handle-schedules` flag:
```sh
pnpm payload jobs:run --cron "*/5 * * * *" --queue myQueue --handle-schedules # This will both schedule jobs according to the configuration and run them
```
If you only want to handle schedules, you can use the dedicated `jobs:handle-schedules` bin script:
```sh
pnpm payload jobs:handle-schedules --cron "*/5 * * * *" --queue myQueue # or --all-queues
```
## Defining schedules on Tasks or Workflows
Schedules are defined using the `schedule` property:

114
packages/payload/src/bin/index.ts
View File

@@ -1,4 +1,3 @@
// @ts-strict-ignore
/* eslint-disable no-console */
import { Cron } from 'croner'
import minimist from 'minimist'
@@ -6,7 +5,7 @@ import { pathToFileURL } from 'node:url'
import path from 'path'
import { findConfig } from '../config/find.js'
import payload, { getPayload } from '../index.js'
import { getPayload, type Payload } from '../index.js'
import { generateImportMap } from './generateImportMap/index.js'
import { generateTypes } from './generateTypes.js'
import { info } from './info.js'
@@ -20,19 +19,61 @@ const availableScripts = [
'generate:types',
'info',
'jobs:run',
'jobs:handle-schedules',
'run',
...migrateCommands,
] as const
export const bin = async () => {
loadEnv()
process.env.DISABLE_PAYLOAD_HMR = 'true'
const args = minimist(process.argv.slice(2))
const script = (typeof args._[0] === 'string' ? args._[0] : '').toLowerCase()
if (args.cron) {
new Cron(
args.cron,
async () => {
// If the bin script initializes payload (getPayload), this will only happen once, as getPayload
// caches the payload instance on the module scope => no need to manually cache and manage getPayload initialization
// outside the Cron here.
await runBinScript({ args, script })
},
{
// Do not run consecutive crons if previous crons still ongoing
protect: true,
},
)
process.stdin.resume() // Keep the process alive
return
} else {
const { payload } = await runBinScript({ args, script })
if (payload) {
await payload.destroy() // close database connections after running jobs so process can exit cleanly
}
process.exit(0)
}
}
async function runBinScript({
args,
script,
}: {
args: minimist.ParsedArgs
script: string
}): Promise<{
/**
* Scripts can return a payload instance if it exists. The bin script runner can then safely
* shut off the instance, depending on if it's running in a cron job or not.
*/
payload?: Payload
}> {
if (script === 'info') {
await info()
return
return {}
}
if (script === 'run') {
@@ -58,7 +99,7 @@ export const bin = async () => {
// Restore original process.argv
process.argv = originalArgv
}
return
return {}
}
const configPath = findConfig()
@@ -91,65 +132,70 @@ export const bin = async () => {
console.error(err)
}
return
return {}
}
if (script.startsWith('migrate')) {
return migrate({ config, parsedArgs: args }).then(() => process.exit(0))
await migrate({ config, parsedArgs: args })
return {}
}
if (script === 'generate:types') {
return generateTypes(config)
await generateTypes(config)
return {}
}
if (script === 'generate:importmap') {
return generateImportMap(config)
await generateImportMap(config)
return {}
}
if (script === 'jobs:run') {
const payload = await getPayload({ config }) // Do not setup crons here - this bin script can set up its own crons
const limit = args.limit ? parseInt(args.limit, 10) : undefined
const queue = args.queue ? args.queue : undefined
const allQueues = !!args.allQueues
const allQueues = !!args['all-queues']
const handleSchedules = !!args['handle-schedules']
if (args.cron) {
new Cron(args.cron, async () => {
await payload.jobs.run({
if (handleSchedules) {
await payload.jobs.handleSchedules({
allQueues,
limit,
queue,
})
})
process.stdin.resume() // Keep the process alive
return
} else {
await payload.jobs.run({
allQueues,
limit,
queue,
})
await payload.destroy() // close database connections after running jobs so process can exit cleanly
process.exit(0)
}
await payload.jobs.run({
allQueues,
limit,
queue,
})
return { payload }
}
if (script === 'jobs:handle-schedules') {
const payload = await getPayload({ config }) // Do not setup crons here - this bin script can set up its own crons
const queue = args.queue ? args.queue : undefined
const allQueues = !!args['all-queues']
await payload.jobs.handleSchedules({
allQueues,
queue,
})
return { payload }
}
if (script === 'generate:db-schema') {
// Barebones instance to access database adapter, without connecting to the DB
await payload.init({
config,
disableDBConnect: true,
disableOnInit: true,
})
const payload = await getPayload({ config, disableDBConnect: true, disableOnInit: true }) // Do not setup crons here
if (typeof payload.db.generateSchema !== 'function') {
payload.logger.error({
msg: `${payload.db.packageName} does not support database schema generation`,
})
await payload.destroy()
process.exit(1)
}
@@ -158,7 +204,7 @@ export const bin = async () => {
prettify: args.prettify === 'false' ? false : true,
})
process.exit(0)
return { payload }
}
console.error(script ? `Unknown command: "${script}"` : 'Please provide a command to run')

22
packages/payload/src/index.ts
View File

@@ -637,7 +637,9 @@ export class BasePayload {
await Promise.all(
cronJobs.map((cronConfig) => {
const jobAutorunCron = new Cron(cronConfig.cron ?? DEFAULT_CRON, async () => {
const jobAutorunCron = new Cron(
cronConfig.cron ?? DEFAULT_CRON,
async () => {
if (
_internal_jobSystemGlobals.shouldAutoSchedule &&
!cronConfig.disableScheduling &&
@@ -668,7 +670,12 @@ export class BasePayload {
queue: cronConfig.queue,
silent: cronConfig.silent,
})
})
},
{
// Do not run consecutive crons if previous crons still ongoing
protect: true,
},
)
this.crons.push(jobAutorunCron)
}),
@@ -942,6 +949,7 @@ export const reload = async (
config: SanitizedConfig,
payload: Payload,
skipImportMapGeneration?: boolean,
options?: InitOptions,
): Promise<void> => {
if (typeof payload.db.destroy === 'function') {
// Only destroy db, as we then later only call payload.db.init and not payload.init
@@ -991,9 +999,11 @@ export const reload = async (
})
}
await payload.db.init?.()
if (payload.db?.init) {
await payload.db.init()
}
if (payload.db.connect) {
if (!options?.disableDBConnect && payload.db.connect) {
await payload.db.connect({ hotReload: true })
}
@@ -1037,7 +1047,7 @@ export const getPayload = async (
* @default 'default'
*/
key?: string
} & Pick<InitOptions, 'config' | 'cron' | 'disableOnInit' | 'importMap'>,
} & InitOptions,
): Promise<Payload> => {
if (!options?.config) {
throw new Error('Error: the payload config is required for getPayload to work.')
@@ -1080,7 +1090,7 @@ export const getPayload = async (
// will reach `if (cached.reload instanceof Promise) {` which then waits for the first reload to finish.
cached.reload = new Promise((res) => (resolve = res))
const config = await options.config
await reload(config, cached.payload, !options.importMap)
await reload(config, cached.payload, !options.importMap, options)
resolve()
}