tabshift/payloadcms
1
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

chore: docs improvements (#9142)

Browse Source 
Improvements to `select` docs and `jobs-queue` docs
This commit is contained in:
James Mikrut
2024-11-12 12:00:21 -05:00
committed by GitHub
parent 97cffa51f8
commit 432741bca3
6 changed files with 516 additions and 434 deletions
Download Patch File Download Diff File Expand all files Collapse all files

46
docs/jobs-queue/jobs.mdx Normal file
View File

@@ -0,0 +1,46 @@
---
title: Jobs
label: Jobs
order: 40
desc: A Job is a set of work that is offloaded from your APIs and will be processed at a later date.
keywords: jobs queue, application framework, typescript, node, react, nextjs
---
Now that we have covered Tasks and Workflows, we can tie them together with a concept called a Job.
<Banner type="default">
Whereas you define Workflows and Tasks, which control your business logic, a <strong>Job</strong> is an individual instance of either a Task or a Workflow which contains many tasks.
</Banner>
For example, let's say we have a Workflow or Task that describes the logic to sync information from Payload to a third-party system. This is how you'd declare how to sync that info, but it wouldn't do anything on its own. In order to run that task or workflow, you'd create a Job that references the corresponding Task or Workflow.
Jobs are stored in the Payload database in the `payload-jobs` collection, and you can decide to keep a running list of all jobs, or configure Payload to delete the job when it has been successfully executed.
#### Queuing a new job
In order to queue a job, you can use the `payload.jobs.queue` function.
Here's how you'd queue a new Job, which will run a `createPostAndUpdate` workflow:
```ts
const createdJob = await payload.jobs.queue({
// Pass the name of the workflow
workflow: 'createPostAndUpdate',
// The input type will be automatically typed
// according to the input you've defined for this workflow
input: {
title: 'my title',
},
})
```
In addition to being able to queue new Jobs based on Workflows, you can also queue a job for a single Task:
```ts
const createdJob = await payload.jobs.queue({
task: 'createPost',
input: {
title: 'my title',
},
})
```

465
docs/jobs-queue/overview.mdx
View File

@@ -6,452 +6,65 @@ desc: Payload provides all you need to run job queues, which are helpful to offl
keywords: jobs queue, application framework, typescript, node, react, nextjs
---
Payload's Jobs Queue gives you a simple, yet powerful way to offload large or future tasks to separate compute resources.
Payload's Jobs Queue gives you a simple, yet powerful way to offload large or future tasks to separate compute resources which is a very powerful feature of many application frameworks.
For example, when building applications with Payload, you might run into a case where you need to perform some complex logic in a Payload [Hook](/docs/hooks/overview) but you don't want that hook to "block" or slow down the response returned from the Payload API.
### Example use cases
Instead of running long or expensive logic in a Hook, you can instead create a Job and add it to a Queue. It can then be picked up by a separate worker which periodically checks the queue for new jobs, and then executes each job accordingly. This way, your Payload API responses can remain as fast as possible, and you can still perform logic as necessary without blocking or affecting your users' experience.
**Non-blocking workloads**
Jobs are also handy for delegating certain actions to take place in the future, such as scheduling a post to be published at a later date. In this example, you could create a Job that will automatically publish a post at a certain time.
You might need to perform some complex, slow-running logic in a Payload [Hook](/docs/hooks/overview) but you don't want that hook to "block" or slow down the response returned from the Payload API. Instead of running this logic directly in a hook, which would block your API response from returning until the expensive work is completed, you can queue a new Job and let it run at a later date.
#### How it works
Examples:
There are a few concepts that you should become familiarized with before using Payload's Jobs Queue - [Tasks](#tasks), [Workflows](#workflows), [Jobs](#jobs), and finally [Queues](#queues).
- Create vector embeddings from your documents, and keep them in sync as your documents change
- Send data to a third-party API on document change
- Trigger emails based on customer actions
## Tasks
**Scheduled actions**
<Banner type="default">
A <strong>"Task"</strong> is a function definition that performs business logic and whose input and output are both strongly typed.
</Banner>
If you need to schedule an action to be run or processed at a certain date in the future, you can queue a job with the `waitUntil` property set. This will make it so the job is not "picked up" until that `waitUntil` date has passed.
You can register Tasks on the Payload config, and then create Jobs or Workflows that use them. Think of Tasks like tidy, isolated "functions that do one specific thing".
Examples:
Payload Tasks can be configured to automatically retried if they fail, which makes them valuable for "durable" workflows like AI applications where LLMs can return non-deterministic results, and might need to be retried.
- Process scheduled posts, where the scheduled date is at a time set in the future
- Unpublish posts at a given time
- Send a reminder email to a customer after X days of signing up for a trial
Tasks can either be defined within the `jobs.tasks` array in your payload config, or they can be defined inline within a workflow.
**Periodic sync or similar scheduled action**
### Defining tasks in the config
Some applications may need to perform a regularly scheduled operation of some type. Jobs are perfect for this because you can execute their logic using `cron`, scheduled nightly, every twelve hours, or some similar time period.
Simply add a task to the `jobs.tasks` array in your Payload config. A task consists of the following fields:
Examples:
| Option | Description |
| --------------------------- | -------------------------------------------------------------------------------- |
| `slug` | Define a slug-based name for this job. This slug needs to be unique among both tasks and workflows.|
| `handler` | The function that should be responsible for running the job. You can either pass a string-based path to the job function file, or the job function itself. If you are using large dependencies within your job, you might prefer to pass the string path because that will avoid bundling large dependencies in your Next.js app. |
| `inputSchema` | Define the input field schema - payload will generate a type for this schema. |
| `interfaceName` | You can use interfaceName to change the name of the interface that is generated for this task. By default, this is "Task" + the capitalized task slug. |
| `outputSchema` | Define the output field schema - payload will generate a type for this schema. |
| `label` | Define a human-friendly label for this task. |
| `onFail` | Function to be executed if the task fails. |
| `onSuccess` | Function to be executed if the task succeeds. |
| `retries` | Specify the number of times that this step should be retried if it fails. |
- You'd like to send emails to all customers on a regular, scheduled basis
- Periodically trigger a rebuild of your frontend at night
- Sync resources to or from a third-party API during non-peak times
The logic for the Task is defined in the `handler` - which can be defined as a function, or a path to a function. The `handler` will run once a worker picks picks up a Job that includes this task.
**Offloading complex operations**
It should return an object with an `output` key, which should contain the output of the task as you've defined.
You may run into the need to perform computationally expensive functions which might slow down your main Payload API server(s). The Jobs Queue allows you to offload these tasks a separate compute resource rather than slowing down the server(s) that run your Payload APIs. With Payload Task definitions, you can even keep large dependencies out of your main Next.js bundle by dynamically importing them only when they are used. This keeps your Next.js + Payload compilation fast and ensures large dependencies do not get bundled into your Payload production build.
Example:
Examples:
```ts
export default buildConfig({
// ...
jobs: {
tasks: [
{
// Configure this task to automatically retry
// up to two times
retries: 2,
- You need to create (and then keep in sync) vector embeddings of your documents as they change, but you use an open source model to generate embeddings
- You have a PDF generator that needs to dynamically build and send PDF versions of documents to customers
- You need to use a headless browser to perform some type of logic
- You need to perform a series of actions, each of which depends on a prior action and should be run in as "durable" of a fashion as possible
// This is a unique identifier for the task
### How it works
slug: 'createPost',
There are a few concepts that you should become familiarized with before using Payload's Jobs Queue. We recommend learning what each of these does in order to fully understand how to leverage the power of Payload's Jobs Queue.
// These are the arguments that your Task will accept
inputSchema: [
{
name: 'title',
type: 'text',
required: true,
},
],
1. [Tasks](/docs/beta/jobs-queue/tasks)
1. [Workflows](/docs/beta/jobs-queue/workflows)
1. [Jobs](/docs/beta/jobs-queue/jobs)
1. [Queues](/docs/beta/jobs-queue/queues)
// These are the properties that the function should output
outputSchema: [
{
name: 'postID',
type: 'text',
required: true,
},
],
All of these pieces work together in order to allow you to offload long-running, expensive, or future scheduled work from your main APIs.
// This is the function that is run when the task is invoked
handler: async ({ input, job, req }) => {
const newPost = await req.payload.create({
collection: 'post',
req,
data: {
title: input.title,
},
})
return {
output: {
postID: newPost.id,
},
}
},
} as TaskConfig<'createPost'>,
]
}
})
```
Here's a quick overview:
In addition to defining handlers as functions directly provided to your Payload config, you can also pass an _absolute path_ to where the handler is defined. If your task has large dependencies, and you are planning on executing your jobs in a separate process that has access to the filesystem, this could be a handy way to make sure that your Payload + Next.js app remains quick to compile and has minimal dependencies.
In general, this is an advanced use case. Here's how this would look:
`payload.config.ts:`
```ts
import { fileURLToPath } from 'node:url'
import path from 'path'
const filename = fileURLToPath(import.meta.url)
const dirname = path.dirname(filename)
export default buildConfig({
jobs: {
tasks: [
{
// ...
// The #createPostHandler is a named export within the `createPost.ts` file
handler: path.resolve(dirname, 'src/tasks/createPost.ts') + '#createPostHandler',
}
]
}
})
```
Then, the `createPost` file itself:
`src/tasks/createPost.ts:`
```ts
import type { TaskHandler } from 'payload'
export const createPostHandler: TaskHandler<'createPost'> = async ({ input, job, req }) => {
const newPost = await req.payload.create({
collection: 'post',
req,
data: {
title: input.title,
},
})
return {
output: {
postID: newPost.id,
},
}
}
```
## Workflows
<Banner type="default">
A <strong>"Workflow"</strong> is an optional way to <em>combine multiple tasks together</em> in a way that can be gracefully retried from the point of failure.
</Banner>
They're most helpful when you have multiple tasks in a row, and you want to configure each task to be able to be retried if they fail.
If a task within a workflow fails, the Workflow will automatically "pick back up" on the task where it failed and **not re-execute any prior tasks that have already been executed**.
#### Defining a workflow
The most important aspect of a Workflow is the `handler`, where you can declare when and how the tasks should run by simply calling the task function, which you can find under the `tasks` object. If any task within the workflow, fails, the entire `handler` function will re-run.
However, importantly, tasks that have successfully been completed will simply re-return the cached and saved output without running again. The Workflow will pick back up where it failed and only task from the failure point onward will be re-executed.
To define a JS-based workflow, simply add a workflow to the `jobs.wokflows` array in your Payload config. A workflow consists of the following fields:
| Option | Description |
| --------------------------- | -------------------------------------------------------------------------------- |
| `slug` | Define a slug-based name for this workflow. This slug needs to be unique among both tasks and workflows.|
| `handler` | The function that should be responsible for running the workflow. You can either pass a string-based path to the workflow function file, or workflow job function itself. If you are using large dependencies within your workflow, you might prefer to pass the string path because that will avoid bundling large dependencies in your Next.js app. |
| `inputSchema` | Define the input field schema - payload will generate a type for this schema. |
| `interfaceName` | You can use interfaceName to change the name of the interface that is generated for this workflow. By default, this is "Workflow" + the capitalized workflow slug. |
| `label` | Define a human-friendly label for this workflow. |
| `queue` | Optionally, define the queue name that this workflow should be tied to. Defaults to "default". |
Example:
```ts
export default buildConfig({
// ...
jobs: {
tasks: [
// ...
]
workflows: [
{
slug: 'createPostAndUpdate',
// The arguments that the workflow will accept
inputSchema: [
{
name: 'title',
type: 'text',
required: true,
},
],
// The handler that defines the "control flow" of the workflow
// Notice how it calls `tasks.[task name]` to execute tasks
handler: async ({ job, tasks }) => {
// This workflow first runs a task called `createPost`
// You need to pass a unique ID for this task invocation as the first argument
// that will always be the same if this workflow fails
// and is re-executed in the future.
// Do not use IDs that change between invocations, like Date.now(), Math.random() or uuid()
const output = await tasks.createPost('1',{
input: {
title: job.input.title,
},
})
// Once the prior task completes, it will run a task
// called `updatePost`
await tasks.updatePost('2', {
input: {
post: job.taskStatus.createPost['1'].output.postID, // or output.postID
title: job.input.title + '2',
},
})
},
} as WorkflowConfig<'updatePost'>
]
}
})
```
#### Running tasks inline
In the above example, our workflow was executing tasks that we already had defined in our Payload config. But, you can also run tasks without predefining them.
To do this, you can use the `inlineTask` function.
The drawbacks of this approach are that tasks cannot be re-used across workflows as easily, and the **task data stored in the job** will not be typed. In the following example, the inline task data will be stored on the job under `job.taskStatus.inline['2']` but completely untyped, as types for dynamic tasks like these cannot be generated beforehand.
Example:
```ts
export default buildConfig({
// ...
jobs: {
tasks: [
// ...
]
workflows: [
{
slug: 'createPostAndUpdate',
inputSchema: [
{
name: 'title',
type: 'text',
required: true,
},
],
handler: async ({ job, tasks, inlineTask }) => {
// Here, we run a predefined task.
// The `createPost` handler arguments and return type
// are both strongly typed
const output = await tasks.createPost('1', {
input: {
title: job.input.title,
},
})
// Here, this task is not defined in the Payload config
// and is "inline". Its output will be stored on the Job in the database
// however its arguments will be untyped.
const { newPost } = await inlineTask('2', {
task: async ({ req }) => {
const newPost = await req.payload.update({
collection: 'post',
id: '2',
req,
retries: 3,
data: {
title: 'updated!',
},
})
return {
output: {
newPost
},
}
}
})
},
} as WorkflowConfig<'updatePost'>
]
}
})
```
## Jobs
Now that we have covered Tasks and Workflows, we can tie them together with a concept called a Job.
<Banner type="default">
Whereas you define Workflows and Tasks, which control your business logic, a <strong>Job</strong> is an individual instance of either a Task or a Workflow which contains many tasks.
</Banner>
For example, let's say we have a Workflow or Task that describes the logic to sync information from Payload to a third-party system. This is how you'd declare how to sync that info, but it wouldn't do anything on its own. In order to run that task or workflow, you'd create a Job that references the corresponding Task or Workflow.
Jobs are stored in the Payload database in the `payload-jobs` collection, and you can decide to keep a running list of all jobs, or configure Payload to delete the job when it has been successfully executed.
#### Queuing a new job
In order to queue a job, you can use the `payload.jobs.queue` function.
Here's how you'd queue a new Job, which will run a `createPostAndUpdate` workflow:
```ts
const createdJob = await payload.jobs.queue({
// Pass the name of the workflow
workflow: 'createPostAndUpdate',
// The input type will be automatically typed
// according to the input you've defined for this workflow
input: {
title: 'my title',
},
})
```
In addition to being able to queue new Jobs based on Workflows, you can also queue a job for a single Task:
```ts
const createdJob = await payload.jobs.queue({
task: 'createPost',
input: {
title: 'my title',
},
})
```
## Queues
Now let's talk about how to _run these jobs_. Right now, all we've covered is how to queue up jobs to run, but so far, we aren't actually running any jobs. This is the final piece of the puzzle.
<Banner type="default">
A <strong>Queue</strong> is a list of jobs that should be executed in order of when they were added.
</Banner>
When you go to run jobs, Payload will query for any jobs that are added to the queue and then run them. By default, all queued jobs are added to the `default` queue.
**But, imagine if you wanted to have some jobs that run nightly, and other jobs which should run every five minutes.**
By specifying the `queue` name when you queue a new job using `payload.jobs.queue()`, you can queue certain jobs with `queue: 'nightly'`, and other jobs can be left as the default queue.
Then, you could configure two different runner strategies:
1. A `cron` that runs nightly, querying for jobs added to the `nightly` queue
2. Another that runs any jobs that were added to the `default` queue every ~5 minutes or so
## Executing jobs
As mentioned above, you can queue jobs, but the jobs won't run unless a worker picks up your jobs and runs them. This can be done in two ways:
#### Endpoint
You can execute jobs by making a fetch request to the `/api/payload-jobs/run` endpoint:
```ts
// Here, we're saying we want to run only 100 jobs for this invocation
// and we want to pull jobs from the `nightly` queue:
await fetch('/api/payload-jobs/run?limit=100&queue=nightly', {
method: 'GET',
headers: {
'Authorization': `Bearer ${token}`,
},
});
```
This endpoint is automatically mounted for you and is helpful in conjunction with serverless platforms like Vercel, where you might want to use Vercel Cron to invoke a serverless function that executes your jobs.
**Vercel Cron Example**
If you're deploying on Vercel, you can add a `vercel.json` file in the root of your project that configures Vercel Cron to invoke the `run` endpoint on a cron schedule.
Here's an example of what this file will look like:
```json
{
"crons": [
{
"path": "/api/payload-jobs/run",
"schedule": "*/5 * * * *"
}
]
}
```
The configuration above schedules the endpoint `/api/payload-jobs/run` to be invoked every 5 minutes.
The last step will be to secure your `run` endpoint so that only the proper users can invoke the runner.
To do this, you can set an environment variable on your Vercel project called `CRON_SECRET`, which should be a random string—ideally 16 characters or longer.
Then, you can modify the `access` function for running jobs by ensuring that only Vercel can invoke your runner.
```ts
export default buildConfig({
// Other configurations...
jobs: {
access: {
run: ({ req }: { req: PayloadRequest }): boolean => {
// Allow logged in users to execute this endpoint (default)
if (req.user) return true
// If there is no logged in user, then check
// for the Vercel Cron secret to be present as an
// Authorization header:
const authHeader = req.headers.get('authorization');
return authHeader === `Bearer ${process.env.CRON_SECRET}`;
},
},
// Other job configurations...
}
})
```
This works because Vercel automatically makes the `CRON_SECRET` environment variable available to the endpoint as the `Authorization` header when triggered by the Vercel Cron, ensuring that the jobs can be run securely.
After the project is deployed to Vercel, the Vercel Cron job will automatically trigger the `/api/payload-jobs/run` endpoint in the specified schedule, running the queued jobs in the background.
#### Local API
If you want to process jobs programmatically from your server-side code, you can use the Local API:
```ts
const results = await payload.jobs.run()
// You can customize the queue name and limit by passing them as arguments:
await payload.jobs.run({ queue: 'nightly', limit: 100 })
```
#### Bin script
Finally, you can process jobs via the bin script that comes with Payload out of the box.
```sh
npx payload jobs:run --queue default --limit 10
```
In addition, the bin script allows you to pass a `--cron` flag to the `jobs:run` command to run the jobs on a scheduled, cron basis:
```sh
npx payload jobs:run --cron "*/5 * * * *"
```
- A Task is a specific function that performs business logic
- Workflows are groupings of specific tasks which should be run in-order, and can be retried from a specific point of failure
- A Job is an instance of a single task or workflow which will be executed
- A Queue is a way to segment your jobs into different "groups" - for example, some to run nightly, and others to run every 10 minutes

120
docs/jobs-queue/queues.mdx Normal file
View File

@@ -0,0 +1,120 @@
---
title: Queues
label: Queues
order: 50
desc: A Queue is a specific group of jobs which can be executed in the order that they were added.
keywords: jobs queue, application framework, typescript, node, react, nextjs
---
Queues are the final aspect of Payload's Jobs Queue and deal with how to _run your jobs_. Up to this point, all we've covered is how to queue up jobs to run, but so far, we aren't actually running any jobs.
<Banner type="default">
A <strong>Queue</strong> is a grouping of jobs that should be executed in order of when they were added.
</Banner>
When you go to run jobs, Payload will query for any jobs that are added to the queue and then run them. By default, all queued jobs are added to the `default` queue.
**But, imagine if you wanted to have some jobs that run nightly, and other jobs which should run every five minutes.**
By specifying the `queue` name when you queue a new job using `payload.jobs.queue()`, you can queue certain jobs with `queue: 'nightly'`, and other jobs can be left as the default queue.
Then, you could configure two different runner strategies:
1. A `cron` that runs nightly, querying for jobs added to the `nightly` queue
2. Another that runs any jobs that were added to the `default` queue every ~5 minutes or so
## Executing jobs
As mentioned above, you can queue jobs, but the jobs won't run unless a worker picks up your jobs and runs them. This can be done in two ways:
#### Endpoint
You can execute jobs by making a fetch request to the `/api/payload-jobs/run` endpoint:
```ts
// Here, we're saying we want to run only 100 jobs for this invocation
// and we want to pull jobs from the `nightly` queue:
await fetch('/api/payload-jobs/run?limit=100&queue=nightly', {
method: 'GET',
headers: {
'Authorization': `Bearer ${token}`,
},
});
```
This endpoint is automatically mounted for you and is helpful in conjunction with serverless platforms like Vercel, where you might want to use Vercel Cron to invoke a serverless function that executes your jobs.
**Vercel Cron Example**
If you're deploying on Vercel, you can add a `vercel.json` file in the root of your project that configures Vercel Cron to invoke the `run` endpoint on a cron schedule.
Here's an example of what this file will look like:
```json
{
"crons": [
{
"path": "/api/payload-jobs/run",
"schedule": "*/5 * * * *"
}
]
}
```
The configuration above schedules the endpoint `/api/payload-jobs/run` to be invoked every 5 minutes.
The last step will be to secure your `run` endpoint so that only the proper users can invoke the runner.
To do this, you can set an environment variable on your Vercel project called `CRON_SECRET`, which should be a random string—ideally 16 characters or longer.
Then, you can modify the `access` function for running jobs by ensuring that only Vercel can invoke your runner.
```ts
export default buildConfig({
// Other configurations...
jobs: {
access: {
run: ({ req }: { req: PayloadRequest }): boolean => {
// Allow logged in users to execute this endpoint (default)
if (req.user) return true
// If there is no logged in user, then check
// for the Vercel Cron secret to be present as an
// Authorization header:
const authHeader = req.headers.get('authorization');
return authHeader === `Bearer ${process.env.CRON_SECRET}`;
},
},
// Other job configurations...
}
})
```
This works because Vercel automatically makes the `CRON_SECRET` environment variable available to the endpoint as the `Authorization` header when triggered by the Vercel Cron, ensuring that the jobs can be run securely.
After the project is deployed to Vercel, the Vercel Cron job will automatically trigger the `/api/payload-jobs/run` endpoint in the specified schedule, running the queued jobs in the background.
#### Local API
If you want to process jobs programmatically from your server-side code, you can use the Local API:
```ts
const results = await payload.jobs.run()
// You can customize the queue name and limit by passing them as arguments:
await payload.jobs.run({ queue: 'nightly', limit: 100 })
```
#### Bin script
Finally, you can process jobs via the bin script that comes with Payload out of the box.
```sh
npx payload jobs:run --queue default --limit 10
```
In addition, the bin script allows you to pass a `--cron` flag to the `jobs:run` command to run the jobs on a scheduled, cron basis:
```sh
npx payload jobs:run --cron "*/5 * * * *"
```

141
docs/jobs-queue/tasks.mdx Normal file
View File

@@ -0,0 +1,141 @@
---
title: Tasks
label: Tasks
order: 20
desc: A Task is a distinct function declaration that can be run within Payload's Jobs Queue.
keywords: jobs queue, application framework, typescript, node, react, nextjs
---
<Banner type="default">
A <strong>"Task"</strong> is a function definition that performs business logic and whose input and output are both strongly typed.
</Banner>
You can register Tasks on the Payload config, and then create [Jobs](/docs/beta/jobs-queue/jobs) or [Workflows](/docs/beta/jobs-queue/workflows) that use them. Think of Tasks like tidy, isolated "functions that do one specific thing".
Payload Tasks can be configured to automatically retried if they fail, which makes them valuable for "durable" workflows like AI applications where LLMs can return non-deterministic results, and might need to be retried.
Tasks can either be defined within the `jobs.tasks` array in your payload config, or they can be defined inline within a workflow.
### Defining tasks in the config
Simply add a task to the `jobs.tasks` array in your Payload config. A task consists of the following fields:
| Option | Description |
| --------------------------- | -------------------------------------------------------------------------------- |
| `slug` | Define a slug-based name for this job. This slug needs to be unique among both tasks and workflows.|
| `handler` | The function that should be responsible for running the job. You can either pass a string-based path to the job function file, or the job function itself. If you are using large dependencies within your job, you might prefer to pass the string path because that will avoid bundling large dependencies in your Next.js app. |
| `inputSchema` | Define the input field schema - payload will generate a type for this schema. |
| `interfaceName` | You can use interfaceName to change the name of the interface that is generated for this task. By default, this is "Task" + the capitalized task slug. |
| `outputSchema` | Define the output field schema - payload will generate a type for this schema. |
| `label` | Define a human-friendly label for this task. |
| `onFail` | Function to be executed if the task fails. |
| `onSuccess` | Function to be executed if the task succeeds. |
| `retries` | Specify the number of times that this step should be retried if it fails. |
The logic for the Task is defined in the `handler` - which can be defined as a function, or a path to a function. The `handler` will run once a worker picks picks up a Job that includes this task.
It should return an object with an `output` key, which should contain the output of the task as you've defined.
Example:
```ts
export default buildConfig({
// ...
jobs: {
tasks: [
{
// Configure this task to automatically retry
// up to two times
retries: 2,
// This is a unique identifier for the task
slug: 'createPost',
// These are the arguments that your Task will accept
inputSchema: [
{
name: 'title',
type: 'text',
required: true,
},
],
// These are the properties that the function should output
outputSchema: [
{
name: 'postID',
type: 'text',
required: true,
},
],
// This is the function that is run when the task is invoked
handler: async ({ input, job, req }) => {
const newPost = await req.payload.create({
collection: 'post',
req,
data: {
title: input.title,
},
})
return {
output: {
postID: newPost.id,
},
}
},
} as TaskConfig<'createPost'>,
]
}
})
```
In addition to defining handlers as functions directly provided to your Payload config, you can also pass an _absolute path_ to where the handler is defined. If your task has large dependencies, and you are planning on executing your jobs in a separate process that has access to the filesystem, this could be a handy way to make sure that your Payload + Next.js app remains quick to compile and has minimal dependencies.
In general, this is an advanced use case. Here's how this would look:
`payload.config.ts:`
```ts
import { fileURLToPath } from 'node:url'
import path from 'path'
const filename = fileURLToPath(import.meta.url)
const dirname = path.dirname(filename)
export default buildConfig({
jobs: {
tasks: [
{
// ...
// The #createPostHandler is a named export within the `createPost.ts` file
handler: path.resolve(dirname, 'src/tasks/createPost.ts') + '#createPostHandler',
}
]
}
})
```
Then, the `createPost` file itself:
`src/tasks/createPost.ts:`
```ts
import type { TaskHandler } from 'payload'
export const createPostHandler: TaskHandler<'createPost'> = async ({ input, job, req }) => {
const newPost = await req.payload.create({
collection: 'post',
req,
data: {
title: input.title,
},
})
return {
output: {
postID: newPost.id,
},
}
}
```

150
docs/jobs-queue/workflows.mdx Normal file
View File

@@ -0,0 +1,150 @@
---
title: Workflows
label: Workflows
order: 30
desc: A Task is a distinct function declaration that can be run within Payload's Jobs Queue.
keywords: jobs queue, application framework, typescript, node, react, nextjs
---
<Banner type="default">
A <strong>"Workflow"</strong> is an optional way to <em>combine multiple tasks together</em> in a way that can be gracefully retried from the point of failure.
</Banner>
They're most helpful when you have multiple tasks in a row, and you want to configure each task to be able to be retried if they fail.
If a task within a workflow fails, the Workflow will automatically "pick back up" on the task where it failed and **not re-execute any prior tasks that have already been executed**.
#### Defining a workflow
The most important aspect of a Workflow is the `handler`, where you can declare when and how the tasks should run by simply calling the `runTask` function. If any task within the workflow, fails, the entire `handler` function will re-run.
However, importantly, tasks that have successfully been completed will simply re-return the cached and saved output without running again. The Workflow will pick back up where it failed and only task from the failure point onward will be re-executed.
To define a JS-based workflow, simply add a workflow to the `jobs.wokflows` array in your Payload config. A workflow consists of the following fields:
| Option | Description |
| --------------------------- | -------------------------------------------------------------------------------- |
| `slug` | Define a slug-based name for this workflow. This slug needs to be unique among both tasks and workflows.|
| `handler` | The function that should be responsible for running the workflow. You can either pass a string-based path to the workflow function file, or workflow job function itself. If you are using large dependencies within your workflow, you might prefer to pass the string path because that will avoid bundling large dependencies in your Next.js app. |
| `inputSchema` | Define the input field schema - payload will generate a type for this schema. |
| `interfaceName` | You can use interfaceName to change the name of the interface that is generated for this workflow. By default, this is "Workflow" + the capitalized workflow slug. |
| `label` | Define a human-friendly label for this workflow. |
| `queue` | Optionally, define the queue name that this workflow should be tied to. Defaults to "default". |
Example:
```ts
export default buildConfig({
// ...
jobs: {
tasks: [
// ...
]
workflows: [
{
slug: 'createPostAndUpdate',
// The arguments that the workflow will accept
inputSchema: [
{
name: 'title',
type: 'text',
required: true,
},
],
// The handler that defines the "control flow" of the workflow
// Notice how it uses the `tasks` argument to execute your predefined tasks.
// These are strongly typed!
handler: async ({ job, tasks }) => {
// This workflow first runs a task called `createPost`.
// You need to define a unique ID for this task invocation
// that will always be the same if this workflow fails
// and is re-executed in the future. Here, we hard-code it to '1'
const output = await tasks.createPost('1', {
input: {
title: job.input.title,
},
})
// Once the prior task completes, it will run a task
// called `updatePost`
await tasks.updatePost('2', {
input: {
post: job.taskStatus.createPost['1'].output.postID, // or output.postID
title: job.input.title + '2',
},
})
},
} as WorkflowConfig<'updatePost'>
]
}
})
```
#### Running tasks inline
In the above example, our workflow was executing tasks that we already had defined in our Payload config. But, you can also run tasks without predefining them.
To do this, you can use the `inlineTask` function.
The drawbacks of this approach are that tasks cannot be re-used across workflows as easily, and the **task data stored in the job** will not be typed. In the following example, the inline task data will be stored on the job under `job.taskStatus.inline['2']` but completely untyped, as types for dynamic tasks like these cannot be generated beforehand.
Example:
```ts
export default buildConfig({
// ...
jobs: {
tasks: [
// ...
]
workflows: [
{
slug: 'createPostAndUpdate',
inputSchema: [
{
name: 'title',
type: 'text',
required: true,
},
],
handler: async ({ job, tasks, inlineTask }) => {
// Here, we run a predefined task.
// The `createPost` handler arguments and return type
// are both strongly typed
const output = await tasks.createPost('1', {
input: {
title: job.input.title,
},
})
// Here, this task is not defined in the Payload config
// and is "inline". Its output will be stored on the Job in the database
// however its arguments will be untyped.
const { newPost } = await inlineTask('2', {
task: async ({ req }) => {
const newPost = await req.payload.update({
collection: 'post',
id: '2',
req,
retries: 3,
data: {
title: 'updated!',
},
})
return {
output: {
newPost
},
}
},
})
},
} as WorkflowConfig<'updatePost'>
]
}
})
```

28
docs/queries/select.mdx
View File

@@ -6,11 +6,13 @@ desc: Payload select determines which fields are selected to the result.
keywords: query, documents, pagination, documentation, Content Management System, cms, headless, javascript, node, react, nextjs
---
You may not need the full data from your Local API / REST queries, but only some specific fields. The select fields API can help you to optimize those cases.
By default, Payload's APIs will return _all fields_ for a given collection or global. But, you may not need all of that data for all of your queries. Sometimes, you might want just a few fields from the response, which can speed up the Payload API and reduce the amount of JSON that is sent to you from the API.
This is where Payload's `select` feature comes in. Here, you can define exactly which fields you'd like to retrieve from the API.
## Local API
To specify select in the [Local API](../local-api/overview), you can use the `select` option in your query:
To specify `select` in the [Local API](../local-api/overview), you can use the `select` option in your query:
```ts
// Include mode
@@ -51,7 +53,7 @@ const getPosts = async () => {
<Banner type="warning">
<strong>Important:</strong>
To perform querying with `select` efficiently, it works on the database level. Because of that, your `beforeRead` and `afterRead` hooks may not receive the full `doc`.
To perform querying with `select` efficiently, Payload implements your `select` query on the database level. Because of that, your `beforeRead` and `afterRead` hooks may not receive the full `doc`.
</Banner>
@@ -100,11 +102,17 @@ const getPosts = async () => {
</Banner>
## `defaultPopulate` collection config property
## defaultPopulate collection config property
The `defaultPopulate` property allows you specify which fields to select when populating the collection from another document.
This is especially useful for links where only the `slug` is needed instead of the entire document.
With this feature, you can dramatically reduce the amount of JSON that is populated from [Relationship](/docs/beta/fields/relationship) or [Upload](/docs/beta/fields/upload) fields.
For example, in your content model, you might have a `Link` field which links out to a different page. When you go to retrieve these links, you really only need the `slug` of the page.
Loading all of the page content, its related links, and everything else is going to be overkill and will bog down your Payload APIs. Instead, you can define the `defaultPopulate` property on your `Pages` collection, so that when Payload "populates" a related Page, it only selects the `slug` field and therefore returns significantly less JSON:
```ts
import type { CollectionConfig } from 'payload'
@@ -129,17 +137,20 @@ export const Pages: CollectionConfig<'pages'> = {
}
```
## `populate`
## populate
You can override `defaultPopulate` with the `populate` property in the Local and REST API
Setting `defaultPopulate` will enforce that each time Payload performs a "population" of a related document, only the fields specified will be queried and returned. However, you can override `defaultPopulate` with the `populate` property in the Local and REST API:
**Local API:**
Local API:
```ts
const getPosts = async () => {
const posts = await payload.find({
collection: 'posts',
populate: {
// Select only `text` from populated docs in the "pages" collection
// Now, no matter what the `defaultPopulate` is set to on the "pages" collection,
// it will be overridden, and the `text` field will be returned instead.
pages: {
text: true,
}, // highlight-line
@@ -150,7 +161,8 @@ const getPosts = async () => {
}
```
REST API:
**REST API:**
```ts
fetch('https://localhost:3000/api/posts?populate[pages][text]=true') // highlight-line
.then((res) => res.json())