Currently, attempting to run tasks in parallel will result in DB errors.
## Solution
The problem was caused due to inefficient db update calls. After each
task completes, we need to update the log array in the payload-jobs
collection. On postgres, that's a different table.
Currently, the update works the following way:
1. Nuke the table
2. Re-insert every single row, including the new one
This will throw db errors if multiple processes start doing that.
Additionally, due to conflicts, new log rows may be lost.
This PR makes use of the the [new db $push operation
](https://github.com/payloadcms/payload/pull/13453) we recently added to
atomically push a new log row to the database in a single round-trip.
This not only reduces the amount of db round trips (=> faster job queue
system) but allows multiple tasks to perform this db operation in
parallel, without conflicts.
## Problem
**Example:**
```ts
export const fastParallelTaskWorkflow: WorkflowConfig<'fastParallelTask'> = {
slug: 'fastParallelTask',
handler: async ({nlineTask }) => {
const taskFunctions = []
for (let i = 0; i < 20; i++) {
const idx = i + 1
taskFunctions.push(async () => {
return await inlineTask(`parallel task ${idx}`, {
input: {
test: idx,
},
task: () => {
return {
output: {
taskID: idx.toString(),
},
}
},
})
})
}
await Promise.all(taskFunctions.map((f) => f()))
},
}
```
On SQLite, this would throw the following error:
```bash
Caught error Error: UNIQUE constraint failed: payload_jobs_log.id
at Object.next (/Users/alessio/Documents/GitHub/payload/node_modules/.pnpm/libsql@0.4.7/node_modules/libsql/index.js:335:20)
at Statement.all (/Users/alessio/Documents/GitHub/payload/node_modules/.pnpm/libsql@0.4.7/node_modules/libsql/index.js:360:16)
at executeStmt (/Users/alessio/Documents/GitHub/payload/node_modules/.pnpm/@libsql+client@0.14.0_bufferutil@4.0.8_utf-8-validate@6.0.5/node_modules/@libsql/client/lib-cjs/sqlite3.js:285:34)
at Sqlite3Client.execute (/Users/alessio/Documents/GitHub/payload/node_modules/.pnpm/@libsql+client@0.14.0_bufferutil@4.0.8_utf-8-validate@6.0.5/node_modules/@libsql/client/lib-cjs/sqlite3.js:101:16)
at /Users/alessio/Documents/GitHub/payload/node_modules/.pnpm/drizzle-orm@0.44.2_@libsql+client@0.14.0_bufferutil@4.0.8_utf-8-validate@6.0.5__@opentelemetr_asjmtflojkxlnxrshoh4fj5f6u/node_modules/src/libsql/session.ts:288:58
at LibSQLPreparedQuery.queryWithCache (/Users/alessio/Documents/GitHub/payload/node_modules/.pnpm/drizzle-orm@0.44.2_@libsql+client@0.14.0_bufferutil@4.0.8_utf-8-validate@6.0.5__@opentelemetr_asjmtflojkxlnxrshoh4fj5f6u/node_modules/src/sqlite-core/session.ts:79:18)
at LibSQLPreparedQuery.values (/Users/alessio/Documents/GitHub/payload/node_modules/.pnpm/drizzle-orm@0.44.2_@libsql+client@0.14.0_bufferutil@4.0.8_utf-8-validate@6.0.5__@opentelemetr_asjmtflojkxlnxrshoh4fj5f6u/node_modules/src/libsql/session.ts:286:21)
at LibSQLPreparedQuery.all (/Users/alessio/Documents/GitHub/payload/node_modules/.pnpm/drizzle-orm@0.44.2_@libsql+client@0.14.0_bufferutil@4.0.8_utf-8-validate@6.0.5__@opentelemetr_asjmtflojkxlnxrshoh4fj5f6u/node_modules/src/libsql/session.ts:214:27)
at QueryPromise.all (/Users/alessio/Documents/GitHub/payload/node_modules/.pnpm/drizzle-orm@0.44.2_@libsql+client@0.14.0_bufferutil@4.0.8_utf-8-validate@6.0.5__@opentelemetr_asjmtflojkxlnxrshoh4fj5f6u/node_modules/src/sqlite-core/query-builders/insert.ts:402:26)
at QueryPromise.execute (/Users/alessio/Documents/GitHub/payload/node_modules/.pnpm/drizzle-orm@0.44.2_@libsql+client@0.14.0_bufferutil@4.0.8_utf-8-validate@6.0.5__@opentelemetr_asjmtflojkxlnxrshoh4fj5f6u/node_modules/src/sqlite-core/query-builders/insert.ts:414:40)
at QueryPromise.then (/Users/alessio/Documents/GitHub/payload/node_modules/.pnpm/drizzle-orm@0.44.2_@libsql+client@0.14.0_bufferutil@4.0.8_utf-8-validate@6.0.5__@opentelemetr_asjmtflojkxlnxrshoh4fj5f6u/node_modules/src/query-promise.ts:31:15) {
rawCode: 1555,
code: 'SQLITE_CONSTRAINT_PRIMARYKEY',
libsqlError: true
}
```
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1211001438499053
This PR adds **atomic** `$push` **support for array fields**. It makes
it possible to safely append new items to arrays, which is especially
useful when running tasks in parallel (like job queues) where multiple
processes might update the same record at the same time. By handling
pushes atomically, we avoid race conditions and keep data consistent -
especially on postgres, where the current implementation would nuke the
entire array table before re-inserting every single array item.
The feature works for both localized and unlocalized arrays, and
supports pushing either single or multiple items at once.
This PR is a requirement for reliably running parallel tasks in the job
queue - see https://github.com/payloadcms/payload/pull/13452.
Alongside documenting `$push`, this PR also adds documentation for
`$inc`.
## Changes to updatedAt behavior
https://github.com/payloadcms/payload/pull/13335 allows us to override
the updatedAt property instead of the db always setting it to the
current date.
However, we are not able to skip updating the updatedAt property
completely. This means, usage of $push results in 2 postgres db calls:
1. set updatedAt in main row
2. append array row in arrays table
This PR changes the behavior to only automatically set updatedAt if it's
undefined. If you explicitly set it to `null`, this now allows you to
skip the db adapter automatically setting updatedAt.
=> This allows us to use $push in just one single db call
## Usage Examples
### Pushing a single item to an array
```ts
const post = (await payload.db.updateOne({
data: {
array: {
$push: {
text: 'some text 2',
id: new mongoose.Types.ObjectId().toHexString(),
},
},
},
collection: 'posts',
id: post.id,
}))
```
### Pushing a single item to a localized array
```ts
const post = (await payload.db.updateOne({
data: {
arrayLocalized: {
$push: {
en: {
text: 'some text 2',
id: new mongoose.Types.ObjectId().toHexString(),
},
es: {
text: 'some text 2 es',
id: new mongoose.Types.ObjectId().toHexString(),
},
},
},
},
collection: 'posts',
id: post.id,
}))
```
### Pushing multiple items to an array
```ts
const post = (await payload.db.updateOne({
data: {
array: {
$push: [
{
text: 'some text 2',
id: new mongoose.Types.ObjectId().toHexString(),
},
{
text: 'some text 3',
id: new mongoose.Types.ObjectId().toHexString(),
},
],
},
},
collection: 'posts',
id: post.id,
}))
```
### Pushing multiple items to a localized array
```ts
const post = (await payload.db.updateOne({
data: {
arrayLocalized: {
$push: {
en: {
text: 'some text 2',
id: new mongoose.Types.ObjectId().toHexString(),
},
es: [
{
text: 'some text 2 es',
id: new mongoose.Types.ObjectId().toHexString(),
},
{
text: 'some text 3 es',
id: new mongoose.Types.ObjectId().toHexString(),
},
],
},
},
},
collection: 'posts',
id: post.id,
}))
```
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1211110462564647
Previously, when manually setting `createdAt` or `updatedAt` in a
`payload.db.*` or `payload.*` operation, the value may have been
ignored. In some cases it was _impossible_ to change the `updatedAt`
value, even when using direct db adapter calls. On top of that, this
behavior sometimes differed between db adapters. For example, mongodb
did accept `updatedAt` when calling `payload.db.updateVersion` -
postgres ignored it.
This PR changes this behavior to consistently respect `createdAt` and
`updatedAt` values for `payload.db.*` operations.
For `payload.*` operations, this also works with the following
exception:
- update operations do no respect `updatedAt`, as updates are commonly
performed by spreading the old data, e.g. `payload.update({ data:
{...oldData} })` - in these cases, we usually still want the `updatedAt`
to be updated. If you need to get around this, you can use the
`payload.db.updateOne` operation instead.
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1210919646303994
Previously, a single run of the simplest job queue workflow (1 single
task, no db calls by user code in the task - we're just testing db
system overhead) would result in **22 db roundtrips** on drizzle. This
PR reduces it to **17 db roundtrips** by doing the following:
- Modifies db.updateJobs to use the new optimized upsertRow function if
the update is simple
- Do not unnecessarily pass the job log to the final job update when the
workflow completes => allows using the optimized upsertRow function, as
only the main table is involved
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1210888186878606
Previously, `db.deleteMany` on postgres resulted in 2 roundtrips to the
database (find + delete with ids). This PR passes the where query
directly to the `deleteWhere` function, resulting in only one roundtrip
to the database (delete with where).
If the where query queries other tables (=> joins required), this falls
back to find + delete with ids. However, this is also more optimized
than before, as we now pass `select: { id: true }` to the findMany
query.
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1210871676349299
### What?
This PR introduces complete trash (soft-delete) support. When a
collection is configured with `trash: true`, documents can now be
soft-deleted and restored via both the API and the admin panel.
```
import type { CollectionConfig } from 'payload'
const Posts: CollectionConfig = {
slug: 'posts',
trash: true, // <-- New collection config prop @default false
fields: [
{
name: 'title',
type: 'text',
},
// other fields...
],
}
```
### Why
Soft deletes allow developers and admins to safely remove documents
without losing data immediately. This enables workflows like reversible
deletions, trash views, and auditing—while preserving compatibility with
drafts, autosave, and version history.
### How?
#### Backend
- Adds new `trash: true` config option to collections.
- When enabled:
- A `deletedAt` timestamp is conditionally injected into the schema.
- Soft deletion is performed by setting `deletedAt` instead of removing
the document from the database.
- Extends all relevant API operations (`find`, `findByID`, `update`,
`delete`, `versions`, etc.) to support a new `trash` param:
- `trash: false` → excludes trashed documents (default)
- `trash: true` → includes both trashed and non-trashed documents
- To query **only trashed** documents: use `trash: true` with a `where`
clause like `{ deletedAt: { exists: true } }`
- Enforces delete access control before allowing a soft delete via
update or updateByID.
- Disables version restoring on trashed documents (must be restored
first).
#### Admin Panel
- Adds a dedicated **Trash view**: `/collections/:collectionSlug/trash`
- Default delete action now soft-deletes documents when `trash: true` is
set.
- **Delete confirmation modal** includes a checkbox to permanently
delete instead.
- Trashed documents:
- Displays UI banner for better clarity of trashed document edit view vs
non-trashed document edit view
- Render in a read-only edit view
- Still allow access to **Preview**, **API**, and **Versions** tabs
- Updated Status component:
- Displays “Previously published” or “Previously a draft” for trashed
documents.
- Disables status-changing actions when documents are in trash.
- Adds new **Restore** bulk action to clear the `deletedAt` timestamp.
- New `Restore` and `Permanently Delete` buttons for
single-trashed-document restore and permanent deletion.
- **Restore confirmation modal** includes a checkbox to restore as
`published`, defaults to `draft`.
- Adds **Empty Trash** and **Delete permanently** bulk actions.
#### Notes
- This feature is completely opt-in. Collections without trash: true
behave exactly as before.
https://github.com/user-attachments/assets/00b83f8a-0442-441e-a89e-d5dc1f49dd37
~~Sometimes, drizzle is adding the same join to the joins array twice
(`addJoinTable`), despite the table being the same. This is due to a bug
in `getNameFromDrizzleTable` where it would sometimes return a UUID
instead of the table name.~~
~~This PR changes it to read from the drizzle:BaseName symbol instead,
which is correctly returning the table name in my testing. It falls back
to `getTableName`, which uses drizzle:Name.~~
This for some reason fails the tests. Instead, this PR just uses the
getTableName utility now instead of searching for the symbol manually.
Currently, an optimized DB update (simple data => no
delete-and-create-row) does the following:
1. sql UPDATE
2. sql SELECT
This PR reduces this further to one single DB call for simple
collections:
1. sql UPDATE with RETURNING()
This only works for simple collections that do not have any fields that
need to be fetched from other tables. If a collection has fields like
relationship or blocks, we'll need that separate SELECT call to join in
the other tables.
In 4.0, we can remove all "complex" fields from the jobs collection and
replace them with a JSON field to make use of this optimization
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1210803039809814
Adds a new operation findDistinct that can give you distinct values of a
field for a given collection
Example:
Assume you have a collection posts with multiple documents, and some of
them share the same title:
```js
// Example dataset (some titles appear multiple times)
[
{ title: 'title-1' },
{ title: 'title-2' },
{ title: 'title-1' },
{ title: 'title-3' },
{ title: 'title-2' },
{ title: 'title-4' },
{ title: 'title-5' },
{ title: 'title-6' },
{ title: 'title-7' },
{ title: 'title-8' },
{ title: 'title-9' },
]
```
You can now retrieve all unique title values using findDistinct:
```js
const result = await payload.findDistinct({
collection: 'posts',
field: 'title',
})
console.log(result.values)
// Output:
// [
// 'title-1',
// 'title-2',
// 'title-3',
// 'title-4',
// 'title-5',
// 'title-6',
// 'title-7',
// 'title-8',
// 'title-9'
// ]
```
You can also limit the number of distinct results:
```js
const limitedResult = await payload.findDistinct({
collection: 'posts',
field: 'title',
sortOrder: 'desc',
limit: 3,
})
console.log(limitedResult.values)
// Output:
// [
// 'title-1',
// 'title-2',
// 'title-3'
// ]
```
You can also pass a `where` query to filter the documents.
This is a follow-up to https://github.com/payloadcms/payload/pull/13060.
There are a bunch of other db adapter methods that use `upsertRow` for
updates: `updateGlobal`, `updateGlobalVersion`, `updateJobs`,
`updateMany`, `updateVersion`.
The previous PR had the logic for using the optimized row updating logic
inside the `updateOne` adapter. This PR moves that logic to the original
`upsertRow` function. Benefits:
- all the other db methods will benefit from this massive optimization
as well. This will be especially relevant for optimizing postgres job
queue initial updates - we should be able to close
https://github.com/payloadcms/payload/pull/11865 after another follow-up
PR
- easier to read db adapter methods due to less code.
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1210803039809810
Based on https://github.com/payloadcms/payload/pull/13060 which should
be merged first
This PR adds ability to update number fields atomically, which could be
important with parallel writes. For now we support this only via
`payload.db.updateOne`.
For example:
```js
// increment by 10
const res = await payload.db.updateOne({
data: {
number: {
$inc: 10,
},
},
collection: 'posts',
where: { id: { equals: post.id } },
})
// decrement by 3
const res2 = await payload.db.updateOne({
data: {
number: {
$inc: -3,
},
},
collection: 'posts',
where: { id: { equals: post.id } },
})
```
In case, if `payload.db.updateOne` received simple data, meaning no:
* Arrays / Blocks
* Localized Fields
* `hasMany: true` text / select / number / relationship fields
* relationship fields with `relationTo` as an array
This PR simplifies the logic to a single SQL `set` call. No any extra
(useless) steps with rewriting all the arrays / blocks / localized
tables even if there were no any changes to them. However, it's good to
note that `payload.update` (not `payload.db.updateOne`) as for now
passes all the previous data as well, so this change won't have any
effect unless you're using `payload.db.updateOne` directly (or for our
internal logic that uses it), in the future a separate PR with
optimization for `payload.update` as well may be implemented.
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1210710489889576
Previously, `db.updateOne` calls with `where` queries that lead to no
results would create new rows on drizzle. Essentially, `db.updateOne`
behaved like `db.upsertOne` on drizzle
Adds support for `halfvec` and `sparsevec` and `bit` (binary vector)
column types. This is required for supporting indexing of embeddings >
2000 dimensions on postgres using the pg-vector extension.
Continuation of https://github.com/payloadcms/payload/pull/6245.
This PR allows you to pass `blocksAsJSON: true` to SQL adapters and the
adapter instead of aligning with the SQL preferred relation approach for
blocks will just use a simple JSON column, which can improve performance
with a large amount of blocks.
To try these changes you can install `3.43.0-internal.c5bbc84`.
Currently, we globally enable both DOM and Node.js types. While this
mostly works, it can cause conflicts - particularly with `fetch`. For
example, TypeScript may incorrectly allow browser-only properties (like
`cache`) and reject valid Node.js ones like `dispatcher`.
This PR disables DOM types for server-only packages like payload,
ensuring Node-specific typings are applied. This caught a few instances
of incorrect fetch usage that were previously masked by overlapping DOM
types.
This is not a perfect solution - packages that contain both server and
client code (like richtext-lexical or next) will still suffer from this
issue. However, it's an improvement in cases where we can cleanly
separate server and client types, like for the `payload` package which
is server-only.
## Use-case
This change enables https://github.com/payloadcms/payload/pull/12622 to
explore using node-native fetch + `dispatcher`, instead of `node-fetch`
+ `agent`.
Currently, it will incorrectly report that `dispatcher` is not a valid
property for node-native fetch
Fixes https://github.com/payloadcms/payload/issues/12628
When using sqlite, the error from the db is a bit different than
Postgres.
This PR allows us to extract the fieldName when using sqlite for the
unique constraint error.
Adds support for read replicas
https://orm.drizzle.team/docs/read-replicas that can be used to offload
read-heavy traffic.
To use (both `db-postgres` and `db-vercel-postgres` are supported):
```ts
import { postgresAdapter } from '@payloadcms/db-postgres'
database: postgresAdapter({
pool: {
connectionString: process.env.POSTGRES_URL,
},
readReplicas: [process.env.POSTGRES_REPLICA_URL],
})
```
---------
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
### What?
This PR fixes an issue while using `text` & `number` fields with
`hasMany: true` where the last entry would be unreachable, and thus
undeletable, because the `transformForWrite` function did not track
these rows for deletion. This causes values that should've been deleted
to remain in the edit view form, as well as the db, after a submission.
This PR also properly threads the placeholder value from
`admin.placeholder` to `text` & `number` `hasMany: true` fields.
### Why?
To remove rows from the db when a submission is made where these fields
are empty arrays, and to properly show an appropriate placeholder when
one is set in config.
### How?
Adjusting `transformForWrite` and the `traverseFields` to keep track of
rows for deletion.
Fixes#11781
Before:
[Editing---Post-dbpg-before--Payload.webm](https://github.com/user-attachments/assets/5ba1708a-2672-4b36-ac68-05212f3aa6cb)
After:
[Editing---Post--dbpg-hasmany-after-Payload.webm](https://github.com/user-attachments/assets/1292e998-83ff-49d0-aa86-6199be319937)
Previously, this was possible in MongoDB but not in Postgres/SQLite
(having `null` in an `in` query)
```
const { docs } = await payload.find({
collection: 'posts',
where: { text: { in: ['text-1', 'text-3', null] } },
})
```
This PR fixes that behavior
This PR introduces a few changes to improve turbopack compatibility and
ensure e2e tests pass with turbopack enabled
## Changes to improve turbopack compatibility
- Use correct sideEffects configuration to fix scss issues
- Import scss directly instead of duplicating our scss rules
- Fix some scss rules that are not supported by turbopack
- Bump Next.js and all other dependencies used to build payload
## Changes to get tests to pass
For an unknown reason, flaky tests flake a lot more often in turbopack.
This PR does the following to get them to pass:
- add more `wait`s
- fix actual flakes by ensuring previous operations are properly awaited
## Blocking turbopack bugs
- [X] https://github.com/vercel/next.js/issues/76464
- Fix PR: https://github.com/vercel/next.js/pull/76545
- Once fixed: change `"sideEffectsDisabled":` back to `"sideEffects":`
## Non-blocking turbopack bugs
- [ ] https://github.com/vercel/next.js/issues/76956
## Related PRs
https://github.com/payloadcms/payload/pull/12653https://github.com/payloadcms/payload/pull/12652
