Fixes https://github.com/payloadcms/payload/issues/11975
Previously, this configuration was causing errors in postgres due to
long names, even though `dbName` is used:
```
{
slug: 'aliases',
fields: [
{
name: 'thisIsALongFieldNameThatWillCauseAPostgresErrorEvenThoughWeSetAShorterDBName',
dbName: 'shortname',
type: 'array',
fields: [
{
name: 'nested_field_1',
type: 'array',
dbName: 'short_nested_1',
fields: [],
},
{
name: 'nested_field_2',
type: 'text',
},
],
},
],
},
```
This is because we were generating Drizzle relation name (for arrays)
always based on the field path and internally, drizzle uses this name
for aliasing. Now, if `dbName` is present, we use `_{dbName}` instead
for the relation name.
Fixes https://github.com/payloadcms/payload/issues/11882
Previously, down migration that dropped the `payload_migrations` table
was failing because `migrationTableExists` doesn't check the current
transaction, only in which you can get a `false` value result.
Fixes https://github.com/payloadcms/payload/issues/11901
Previously, when `ValidationError` `errors.path` was referring to a
field with `label` defined as a function, the error message was
generated with `[object Object]`. Now, we call that function instead.
Since the `i18n` argument is required for `StaticLabel`, this PR
introduces so you can pass a partial `req` to `ValidationError` from
which we thread `req.i18n` to the label args.
This replaces usage of our `chainMethods` helper to dynamically chain
queries with [drizzle dynamic query
building](https://orm.drizzle.team/docs/dynamic-query-building).
This is more type-safe, more readable and requires less code
This adds support for running multiple job queue tasks in parallel
within the same workflow while preventing conflicts. Previously, this
would have caused the following issues:
- Job log entries get lost - the final job log is incomplete, despite
all tasks having been executed
- Write conflicts in postgres, leading to unique constraint violation
errors
The solution involves handling job log data updates in a way that avoids
overwriting, and ensuring the final update reflects the latest job log
data. Each job log entry now initializes its own ID, so a given job log
entry’s ID remains the same across multiple, parallel task executions.
## Postgres
In Postgres, we need to enable transactions for the
`payload.db.updateJobs` operation; otherwise, two tasks updating the
same job in parallel can conflict. This happens because Postgres handles
array rows by deleting them all, then re-inserting (rather than
upserting). The rows are stored in a separate table, and the following
scenario can occur:
Op 1: deletes all job log rows
Op 2: deletes all job log rows
Op 1: inserts 200 job log rows
Op 2: insert the same 200 job log rows again => `error: “duplicate key
value violates unique constraint "payload_jobs_log_pkey”`
Because transactions were not used, the rows inserted by Op 1
immediately became visible to Op 2, causing the conflict. Enabling
transactions fixes this. In theory, it can still happen if Op 1 commits
before Op 2 starts inserting (due to the read committed isolation
level), but it should occur far less frequently.
Alongside this change, we should consider inserting the rows using an
upsert (update on conflict), which will get rid of this error
completely. That way, if the insertion of Op 1 is visible to Op 2, Op 2
will simply overwrite it, rather than erroring. Individual job entries
are immutable and job entries cannot be deleted, thus this shouldn't
corrupt any data.
## Mongo
In Mongo, the issue is addressed by ensuring that log row deletions
caused due to different log states in concurrent operations are not
merged back to the client job log, and by making sure the final update
includes all job logs.
There is no duplicate key error in Mongo because the array log resides
in the same document and duplicates are simply upserted. We cannot use
transactions in Mongo, as it appears to lock the document in a way that
prevents reliable parallel updates, leading to:
`MongoServerError: WriteConflict error: this operation conflicted with
another operation. Please retry your operation or multi-document
transaction`
In 3.0, we made the decision to export all types from the main package
export (e.g. `payload/types` => `payload`). This improves type
discoverability by IDEs and simplifies importing types.
This PR does the same for our db adapters, which still have a separate
`/types` subpath export. While those are kept for
backwards-compatibility, we can remove them in 4.0.
Continuation of #11489. This adds a new, optional `updateJobs` db
adapter method that reduces the amount of database calls for the jobs
queue.
## MongoDB
### Previous: running a set of 50 queued jobs
- 1x db.find (= 1x `Model.paginate`)
- 50x db.updateOne (= 50x `Model.findOneAndUpdate`)
### Now: running a set of 50 queued jobs
- 1x db.updateJobs (= 1x `Model.find` and 1x `Model.updateMany`)
**=> 51 db round trips before, 2 db round trips after**
### Previous: upon task completion
- 1x db.find (= 1x `Model.paginate`)
- 1x db.updateOne (= 1x `Model.findOneAndUpdate`)
### Now: upon task completion
- 1x db.updateJobs (= 1x `Model.findOneAndUpdate`)
**=> 2 db round trips before, 1 db round trip after**
## Drizzle (e.g. Postgres)
### running a set of 50 queued jobs
- 1x db.query[tablename].findMany
- 50x db.select
- 50x upsertRow
This is unaffected by this PR and will be addressed in a future PR
This will improve performance when updating a single document in
postgres/drizzle, if the ID is known.
Previously, this resulted in 2 sequential operations:
- `db.select `to fetch the document by the ID
- `upsertRow` to update the document (multiple db operations)
This PR removes the unnecessary `db.select` call, as the document ID is
already known
Previously, if you were querying a collection that has a join field with
`draft: true`, and the join field's collection also has
`versions.drafts: true` our db adapter would still query the original
SQL table / mongodb collection instead of the versions one which isn't
quite right since we respect `draft: true` when populating relationships
Fixes https://github.com/payloadcms/payload/issues/6884
Adds a new flag `acceptIDOnCreate` that allows you to thread your own
`id` to `payload.create` `data`, for example:
```ts
// doc created with id 1
const doc = await payload.create({ collection: 'posts', data: {id: 1, title: "my title"}})
```
```ts
import { Types } from 'mongoose'
const id = new Types.ObjectId().toHexString()
const doc = await payload.create({ collection: 'posts', data: {id, title: "my title"}})
```
### What?
This PR adds ability to define indexes on several fields for collections
(compound indexes).
Example:
```ts
{
indexes: [{ unique: true, fields: ['title', 'group.name'] }]
}
```
### Why?
This can be used to either speed up querying/sorting by 2 or more fields
at the same time or to ensure uniqueness between several fields.
### How?
Implements this logic in database adapters. Additionally, adds a utility
`getFieldByPath`.
This PR adds a new `limit` property to `payload.db.updateMany`. This functionality is required for [migrating our job system to use faster, direct db adapter calls](https://github.com/payloadcms/payload/pull/11489)
This PR adds a new `returning` option to various db adapter methods. Setting it to `false` where the return value is not used will lead to performance gains, as we don't have to do additional db calls to fetch the updated document and then sanitize it.
### What?
For the join field query adds ability to specify `count: true`, example:
```ts
const result = await payload.find({
joins: {
'group.relatedPosts': {
sort: '-title',
count: true,
},
},
collection: "categories",
})
result.group?.relatedPosts?.totalDocs // available
```
### Why?
Can be useful to implement full pagination / show total related
documents count in the UI.
### How?
Implements the logic in database adapters. In MongoDB it's additional
`$lookup` that has `$count` in the pipeline. In SQL, it's additional
subquery with `COUNT(*)`. Preserves the current behavior by default,
since counting introduces overhead.
Additionally, fixes a typescript generation error for join fields.
Before, `docs` and `hasNextPage` were marked as nullable, which is not
true, these fields cannot be `null`.
Additionally, fixes threading of `joinQuery` in
`transform/read/traverseFields` for group / tab fields recursive calls.
Previously, `updateOne` was using `buildFindManyArgs` and `findFirst` just to retrieve the ID of the document to update, which is a huge function that's not necessary to run just to get the document ID.
This PR refactors it to use a simple `db.select` query to retrieve the ID
This feature allows you to specify `collection` for the join field as
array.
This can be useful for example to describe relationship linking like
this:
```ts
{
slug: 'folders',
fields: [
{
type: 'join',
on: 'folder',
collection: ['files', 'documents', 'folders'],
name: 'children',
},
{
type: 'relationship',
relationTo: 'folders',
name: 'folder',
},
],
},
{
slug: 'files',
upload: true,
fields: [
{
type: 'relationship',
relationTo: 'folders',
name: 'folder',
},
],
},
{
slug: 'documents',
fields: [
{
type: 'relationship',
relationTo: 'folders',
name: 'folder',
},
],
},
```
Documents and files can be placed to folders and folders themselves can
be nested to other folders (root folders just have `folder` as `null`).
Output type of `Folder`:
```ts
export interface Folder {
id: string;
children?: {
docs?:
| (
| {
relationTo?: 'files';
value: string | File;
}
| {
relationTo?: 'documents';
value: string | Document;
}
| {
relationTo?: 'folders';
value: string | Folder;
}
)[]
| null;
hasNextPage?: boolean | null;
} | null;
folder?: (string | null) | Folder;
updatedAt: string;
createdAt: string;
}
```
While you could instead have many join fields (for example
`childrenFolders`, `childrenFiles`) etc - this doesn't allow you to
sort/filter and paginate things across many collections, which isn't
trivial. With SQL we use `UNION ALL` query to achieve that.
---------
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
### What?
Previously, in postgres query like:
```ts
const result = await payload.find({
collection: 'blocks',
where: { 'blocks.director.name': { equals: 'Test Director' } },
})
```
where `blocks` is a blocks field, `director` is a relationship field and
`name` is a text field inside `directors`, failed with:
### Why?
The generated query before was a bit wrong.
Before:
```sql
select distinct
"blocks"."id",
"blocks"."created_at",
"blocks"."created_at"
from
"blocks"
left join "directors" "a5ad426a_eda4_4067_af7e_5b294d7f0968" on "a5ad426a_eda4_4067_af7e_5b294d7f0968"."id" = "blocks_blocks_some"."director_id"
left join "blocks_blocks_some" on "blocks"."id" = "blocks_blocks_some"."_parent_id"
where
"a5ad426a_eda4_4067_af7e_5b294d7f0968"."name" = 'Test Director'
order by
"blocks"."created_at" desc
limit
10
```
Notice `left join directors` _before_ join of `blocks_blocks_some`.
`blocks_blocks_some` doesn't exist yet, this PR changes so now we
generate
```sql
select distinct
"blocks"."id",
"blocks"."created_at",
"blocks"."created_at"
from
"blocks"
left join "blocks_blocks_some" on "blocks"."id" = "blocks_blocks_some"."_parent_id"
left join "directors" "a5ad426a_eda4_4067_af7e_5b294d7f0968" on "a5ad426a_eda4_4067_af7e_5b294d7f0968"."id" = "blocks_blocks_some"."director_id"
where
"a5ad426a_eda4_4067_af7e_5b294d7f0968"."name" = 'Test Director'
order by
"blocks"."created_at" desc
limit
10
```
Currently, the join field outputs to its result `hasNextPage: boolean`
and have the `limit` query parameter but lacks `page` which can be
useful. This PR adds it.
The `localized` properly was not stripped out of referenced block fields, if any parent was localized. For normal fields, this is done in sanitizeConfig. As the same referenced block config can be used in both a localized and non-localized config, we are not able to strip it out inside sanitizeConfig by modifying the block config.
Instead, this PR had to bring back tedious logic to handle it everywhere the `field.localized` property is accessed. For backwards-compatibility, we need to keep the existing sanitizeConfig logic. In 4.0, we should remove it to benefit from better test coverage of runtime field.localized handling - for now, this is done for our test suite using the `PAYLOAD_DO_NOT_SANITIZE_LOCALIZED_PROPERTY` flag.
The fix, added in https://github.com/payloadcms/payload/pull/11096
wasn't sufficient enough. It did handle the case when the same query
path / table was joined twice and caused incorrect `totalDocs`, but it
didn't handle the case when `JOIN` returns more than 1 rows, which 2
added new assertions here check.
Now, we use `COUNT(*)` only if we don't have any joined tables. If we
do, instead of using `SELECT (COUNT DISTINCT id)` which as described in
the previous PR is _very slow_ for large tables, we use the following
query:
```sql
SELECT COUNT(1) OVER() as count -- window function, executes for each row only once
FROM users
LEFT JOIN -- ... here additional rows are added
WHERE -- ...
GROUP BY users.id -- this ensures we're counting only users without additional rows from joins.
LIMIT 1 -- Since COUNT(1) OVER() executes and resolves before doing LIMIT, we can safely apply LIMIT 1.
```
If you have multiple blocks that are used in multiple places, this can quickly blow up the size of your Payload Config. This will incur a performance hit, as more data is
1. sent to the client (=> bloated `ClientConfig` and large initial html) and
2. processed on the server (permissions are calculated every single time you navigate to a page - this iterates through all blocks you have defined, even if they're duplicative)
This can be optimized by defining your block **once** in your Payload Config, and just referencing the block slug whenever it's used, instead of passing the entire block config. To do this, the block can be defined in the `blocks` array of the Payload Config. The slug can then be passed to the `blockReferences` array in the Blocks Field - the `blocks` array has to be empty for compatibility reasons.
```ts
import { buildConfig } from 'payload'
import { lexicalEditor, BlocksFeature } from '@payloadcms/richtext-lexical'
// Payload Config
const config = buildConfig({
// Define the block once
blocks: [
{
slug: 'TextBlock',
fields: [
{
name: 'text',
type: 'text',
},
],
},
],
collections: [
{
slug: 'collection1',
fields: [
{
name: 'content',
type: 'blocks',
// Reference the block by slug
blockReferences: ['TextBlock'],
blocks: [], // Required to be empty, for compatibility reasons
},
],
},
{
slug: 'collection2',
fields: [
{
name: 'editor',
type: 'richText',
editor: lexicalEditor({
BlocksFeature({
// Same reference can be reused anywhere, even in the lexical editor, without incurred performance hit
blocks: ['TextBlock'],
})
})
},
],
},
],
})
```
## v4.0 Plans
In 4.0, we will remove the `blockReferences` property, and allow string block references to be passed directly to the blocks `property`. Essentially, we'd remove the `blocks` property and rename `blockReferences` to `blocks`.
The reason we opted to a new property in this PR is to avoid breaking changes. Allowing strings to be passed to the `blocks` property will prevent plugins that iterate through fields / blocks from compiling.
## PR Changes
- Testing: This PR introduces a plugin that automatically converts blocks to block references. This is done in the fields__blocks test suite, to run our existing test suite using block references.
- Block References support: Most changes are similar. Everywhere we iterate through blocks, we have to now do the following:
1. Check if `field.blockReferences` is provided. If so, only iterate through that.
2. Check if the block is an object (= actual block), or string
3. If it's a string, pull the actual block from the Payload Config or from `payload.blocks`.
The exception is config sanitization and block type generations. This PR optimizes them so that each block is only handled once, instead of every time the block is referenced.
## Benchmarks
60 Block fields, each block field having the same 600 Blocks.
### Before:
**Initial HTML:** 195 kB
**Generated types:** takes 11 minutes, 461,209 lines
https://github.com/user-attachments/assets/11d49a4e-5414-4579-8050-e6346e552f56
### After:
**Initial HTML:** 73.6 kB
**Generated types:** takes 2 seconds, 35,810 lines
https://github.com/user-attachments/assets/3eab1a99-6c29-489d-add5-698df67780a3
### After Permissions Optimization (follow-up PR)
Initial HTML: 73.6 kB
https://github.com/user-attachments/assets/a909202e-45a8-4bf6-9a38-8c85813f1312
## Future Plans
1. This PR does not yet deduplicate block references during permissions calculation. We'll optimize that in a separate PR, as this one is already large enough
2. The same optimization can be done to deduplicate fields. One common use-case would be link field groups that may be referenced in multiple entities, outside of blocks. We might explore adding a new `fieldReferences` property, that allows you to reference those same `config.blocks`.
Fixes https://github.com/payloadcms/payload/issues/10810
This was caused by using `COUNT(*)` aggregation instead of
`COUNT(DISTINCT table.id)`. However, we want to use `COUNT(*)` because
`COUNT(DISTINCT table.id)` is slow on large tables. Now we fallback to
`COUNT(DISTINCT table.id)` only when `COUNT(*)` cannot work properly.
Example of a query that leads to incorrect `totalDocs`:
```ts
const res = await payload.find({
collection: 'directors',
limit: 10,
where: {
or: [
{
movies: {
equals: movie2.id,
},
},
{
movies: {
equals: movie1.id,
},
},
{
movies: {
equals: movie1.id,
},
},
],
},
})
```
### What?
If you had multiple operator constraints on a single field, the last one
defined would be the only one used.
Example:
```ts
where: {
id: {
in: [doc2.id],
not_in: [], // <-- only respected this operator constraint
},
}
```
and
```ts
where: {
id: {
not_in: [],
in: [doc2.id], // <-- only respected this operator constraint
},
}
```
They would yield different results.
### Why?
The results were not merged into an `$and` query inside parseParams.
### How?
Merges the results within an `$and` constraint.
Fixes https://github.com/payloadcms/payload/issues/10944
Supersedes https://github.com/payloadcms/payload/pull/11011
Previously, data for globals was inconsistent across database adapters.
In Postgres, globals didn't store correct `createdAt`, `updatedAt`
fields and the `updateGlobal` lacked the `globalType` field. This PR
solves that without introducing schema changes.
Fixes https://github.com/payloadcms/payload/issues/10780
Previously, with enabled versions, nested select `hasMany: true` fields
weren't working with SQL database adapters. This was due to wrongly
passed `parent` to select rows data because we store arrays and blocks
in versions a bit differently, using both, `id` and `_uuid` (which
contains the normal Object ID) columns. And unlike with non versions
`_uuid` column isn't actually applicable here as it's not unique, thus
we need to save blocks/arrays first and then map their ObjectIDs to
generated by the database IDs and use them for select fields `parent`
data
