Adds a new operation findDistinct that can give you distinct values of a
field for a given collection
Example:
Assume you have a collection posts with multiple documents, and some of
them share the same title:
```js
// Example dataset (some titles appear multiple times)
[
{ title: 'title-1' },
{ title: 'title-2' },
{ title: 'title-1' },
{ title: 'title-3' },
{ title: 'title-2' },
{ title: 'title-4' },
{ title: 'title-5' },
{ title: 'title-6' },
{ title: 'title-7' },
{ title: 'title-8' },
{ title: 'title-9' },
]
```
You can now retrieve all unique title values using findDistinct:
```js
const result = await payload.findDistinct({
collection: 'posts',
field: 'title',
})
console.log(result.values)
// Output:
// [
// 'title-1',
// 'title-2',
// 'title-3',
// 'title-4',
// 'title-5',
// 'title-6',
// 'title-7',
// 'title-8',
// 'title-9'
// ]
```
You can also limit the number of distinct results:
```js
const limitedResult = await payload.findDistinct({
collection: 'posts',
field: 'title',
sortOrder: 'desc',
limit: 3,
})
console.log(limitedResult.values)
// Output:
// [
// 'title-1',
// 'title-2',
// 'title-3'
// ]
```
You can also pass a `where` query to filter the documents.
This is a follow-up to https://github.com/payloadcms/payload/pull/13060.
There are a bunch of other db adapter methods that use `upsertRow` for
updates: `updateGlobal`, `updateGlobalVersion`, `updateJobs`,
`updateMany`, `updateVersion`.
The previous PR had the logic for using the optimized row updating logic
inside the `updateOne` adapter. This PR moves that logic to the original
`upsertRow` function. Benefits:
- all the other db methods will benefit from this massive optimization
as well. This will be especially relevant for optimizing postgres job
queue initial updates - we should be able to close
https://github.com/payloadcms/payload/pull/11865 after another follow-up
PR
- easier to read db adapter methods due to less code.
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1210803039809810
Based on https://github.com/payloadcms/payload/pull/13060 which should
be merged first
This PR adds ability to update number fields atomically, which could be
important with parallel writes. For now we support this only via
`payload.db.updateOne`.
For example:
```js
// increment by 10
const res = await payload.db.updateOne({
data: {
number: {
$inc: 10,
},
},
collection: 'posts',
where: { id: { equals: post.id } },
})
// decrement by 3
const res2 = await payload.db.updateOne({
data: {
number: {
$inc: -3,
},
},
collection: 'posts',
where: { id: { equals: post.id } },
})
```
In case, if `payload.db.updateOne` received simple data, meaning no:
* Arrays / Blocks
* Localized Fields
* `hasMany: true` text / select / number / relationship fields
* relationship fields with `relationTo` as an array
This PR simplifies the logic to a single SQL `set` call. No any extra
(useless) steps with rewriting all the arrays / blocks / localized
tables even if there were no any changes to them. However, it's good to
note that `payload.update` (not `payload.db.updateOne`) as for now
passes all the previous data as well, so this change won't have any
effect unless you're using `payload.db.updateOne` directly (or for our
internal logic that uses it), in the future a separate PR with
optimization for `payload.update` as well may be implemented.
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1210710489889576
Previously, `db.updateOne` calls with `where` queries that lead to no
results would create new rows on drizzle. Essentially, `db.updateOne`
behaved like `db.upsertOne` on drizzle
If you (using the MongoDB adapter) delete a block from the payload
config, but still have some data with that block in the DB, you'd
receive in the admin panel an error like:
```
Block with type "cta" was found in block data, but no block with that type is defined in the config for field with schema path pages.blocks
```
Now, we remove those "unknown" blocks at the DB adapter level.
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
We were running scripts as they were without encompassing our logic in a
function for jest's teardown and we were subsequently running
`process.exit(0)` which meant that tests didn't correctly return an
error status code when they failed in CI.
The following tests have been skipped as well:
```
● postgres vector custom column › should add a vector column and query it
● Sort › Local API › Orderable › should not break with existing base 62 digits
● Sort › Local API › Orderable join › should set order by default
● Sort › Local API › Orderable join › should allow setting the order with the local API
● Sort › Local API › Orderable join › should sort join docs in the correct
```
---------
Co-authored-by: Elliot DeNolf <denolfe@gmail.com>
Co-authored-by: Alessio Gravili <alessio@gravili.de>
Fixes an issue when querying deeply new relationship virtual fields with
`draft: true`. Changes the method for `where` sanitization, before it
was done in `validateSearchParam` which didn't work with versions
properly, now there's a separate `sanitizeWhereQuery` function that does
this.
Continuation of https://github.com/payloadcms/payload/pull/6245.
This PR allows you to pass `blocksAsJSON: true` to SQL adapters and the
adapter instead of aligning with the SQL preferred relation approach for
blocks will just use a simple JSON column, which can improve performance
with a large amount of blocks.
To try these changes you can install `3.43.0-internal.c5bbc84`.
Fixes https://github.com/payloadcms/payload/issues/12628
When using sqlite, the error from the db is a bit different than
Postgres.
This PR allows us to extract the fieldName when using sqlite for the
unique constraint error.
This clarifies that jobs.autoRun only *runs* already-queued jobs. It does not queue the jobs for you.
Also adds an e2e test as this functionality had no e2e coverage
Previously, this was possible in MongoDB but not in Postgres/SQLite
(having `null` in an `in` query)
```
const { docs } = await payload.find({
collection: 'posts',
where: { text: { in: ['text-1', 'text-3', null] } },
})
```
This PR fixes that behavior
Previously, `hidden: true` on a virtual field that references a
relationship field didn't work. Now, this field doesn't get calculated
if there's `hidden: true` and no `showHiddenFields` was passed.
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
Continuation of https://github.com/payloadcms/payload/pull/12265.
Currently, using `select` on new relationship virtual fields:
```
const doc = await payload.findByID({
collection: 'virtual-relations',
depth: 0,
id,
select: { postTitle: true },
})
```
doesn't work, because in order to calculate `post.title`, the `post`
field must be selected as well. This PR adds logic that sanitizes the
incoming `select` to include those relationships into `select` (that are
related to selected virtual fields)
---------
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
When doing `payload.db.queryDrafts` with `select` without `version`, or
simply your select looks like:
`select: { version: { nonExistingField: true } }` - the `queryDrafts`
function will crash because it tries to access the `version` field.
This PR adds a fallback.
This PR optimizes the new virtual fields with relationships feature
https://github.com/payloadcms/payload/pull/11805 when the path
references the ID field, for example:
```
{
name: 'postCategoryID',
type: 'number',
virtual: 'post.category.id',
},
```
Previously, we did additional population of `category`, which is
unnecessary as we can always grab the ID from the `category` value
itself. One less querying step.
This PR adds an ability to specify a virtual field in this way
```js
{
slug: 'posts',
fields: [
{
name: 'title',
type: 'text',
required: true,
},
],
},
{
slug: 'virtual-relations',
fields: [
{
name: 'postTitle',
type: 'text',
virtual: 'post.title',
},
{
name: 'post',
type: 'relationship',
relationTo: 'posts',
},
],
},
```
Then, every time you query `virtual-relations`, `postTitle` will be
automatically populated (even if using `depth: 0`) on the db level. This
field also, unlike `virtual: true` is available for querying / sorting /
`useAsTitle`.
Also, the field can be deeply nested to 2 or more relationships, for
example:
```
{
name: 'postCategoryTitle',
type: 'text',
virtual: 'post.category.title',
},
```
Where the current collection has `post` - a relationship to `posts`, the
collection `posts` has `category` that's a relationship to `categories`
and finally `categories` has `title`.
### What?
Converts numbers passed to a text field to avoid the database/drizzle
from converting it incorrectly.
### Why?
If you have a hook that passes a value to another field you can
experience this problem where drizzle converts a number value for a text
field to a floating point number in sqlite for example.
### How?
Adds logic to `transform/write/traverseFields.ts` to cast text field
values to string.
Fixes https://github.com/payloadcms/payload/issues/11975
Previously, this configuration was causing errors in postgres due to
long names, even though `dbName` is used:
```
{
slug: 'aliases',
fields: [
{
name: 'thisIsALongFieldNameThatWillCauseAPostgresErrorEvenThoughWeSetAShorterDBName',
dbName: 'shortname',
type: 'array',
fields: [
{
name: 'nested_field_1',
type: 'array',
dbName: 'short_nested_1',
fields: [],
},
{
name: 'nested_field_2',
type: 'text',
},
],
},
],
},
```
This is because we were generating Drizzle relation name (for arrays)
always based on the field path and internally, drizzle uses this name
for aliasing. Now, if `dbName` is present, we use `_{dbName}` instead
for the relation name.
Fixes https://github.com/payloadcms/payload/issues/6884
Adds a new flag `acceptIDOnCreate` that allows you to thread your own
`id` to `payload.create` `data`, for example:
```ts
// doc created with id 1
const doc = await payload.create({ collection: 'posts', data: {id: 1, title: "my title"}})
```
```ts
import { Types } from 'mongoose'
const id = new Types.ObjectId().toHexString()
const doc = await payload.create({ collection: 'posts', data: {id, title: "my title"}})
```
This change makes so that data that exists in MongoDB but isn't defined
in the Payload config won't be included to `payload.find` /
`payload.db.find` calls. Now we strip all the additional keys.
Consider you have a field named `secretField` that's also `hidden: true`
(or `read: () => false`) that contains some sensitive data. Then you
removed this field from the database and as for now with the MongoDB
adapter this field will be included to the Local API / REST API results
without any consideration, as Payload doesn't know about it anymore.
This also fixes https://github.com/payloadcms/payload/issues/11542 if
you removed / renamed a relationship field from the schema, Payload
won't sanitize ObjectIDs back to strings anymore.
Ideally you should create a migration script that completely removes the
deleted field from the database with `$unset`, but people rarely do
this.
If you still need to keep those fields to the result, this PR allows you
to do this with the new `allowAdditionalKeys: true` flag.
### What?
This PR adds ability to define indexes on several fields for collections
(compound indexes).
Example:
```ts
{
indexes: [{ unique: true, fields: ['title', 'group.name'] }]
}
```
### Why?
This can be used to either speed up querying/sorting by 2 or more fields
at the same time or to ensure uniqueness between several fields.
### How?
Implements this logic in database adapters. Additionally, adds a utility
`getFieldByPath`.
This PR adds a new `limit` property to `payload.db.updateMany`. This functionality is required for [migrating our job system to use faster, direct db adapter calls](https://github.com/payloadcms/payload/pull/11489)
### What?
If you had multiple operator constraints on a single field, the last one
defined would be the only one used.
Example:
```ts
where: {
id: {
in: [doc2.id],
not_in: [], // <-- only respected this operator constraint
},
}
```
and
```ts
where: {
id: {
not_in: [],
in: [doc2.id], // <-- only respected this operator constraint
},
}
```
They would yield different results.
### Why?
The results were not merged into an `$and` query inside parseParams.
### How?
Merges the results within an `$and` constraint.
Fixes https://github.com/payloadcms/payload/issues/10944
Supersedes https://github.com/payloadcms/payload/pull/11011
Previously, data for globals was inconsistent across database adapters.
In Postgres, globals didn't store correct `createdAt`, `updatedAt`
fields and the `updateGlobal` lacked the `globalType` field. This PR
solves that without introducing schema changes.
Field paths within hooks are not correct.
For example, an unnamed tab containing a group field and nested text
field should have the path:
- `myGroupField.myTextField`
However, within hooks that path is formatted as:
- `_index-1.myGroupField.myTextField`
The leading index shown above should not exist, as this field is
considered top-level since it is located within an unnamed tab.
This discrepancy is only evident through the APIs themselves, such as
when creating a request with invalid data and reading the validation
errors in the response. Form state contains proper field paths, which is
ultimately why this issue was never caught. This is because within the
admin panel we merge the API response with the current form state,
obscuring the underlying issue. This becomes especially obvious in
#10580, where we no longer initialize validation errors within form
state until the form has been submitted, and instead rely solely on the
API response for the initial error state.
Here's comprehensive example of how field paths _should_ be formatted:
```
{
// ...
fields: [
{
// path: 'topLevelNamedField'
// schemaPath: 'topLevelNamedField'
// indexPath: ''
name: 'topLevelNamedField',
type: 'text',
},
{
// path: 'array'
// schemaPath: 'array'
// indexPath: ''
name: 'array',
type: 'array',
fields: [
{
// path: 'array.[n].fieldWithinArray'
// schemaPath: 'array.fieldWithinArray'
// indexPath: ''
name: 'fieldWithinArray',
type: 'text',
},
{
// path: 'array.[n].nestedArray'
// schemaPath: 'array.nestedArray'
// indexPath: ''
name: 'nestedArray',
type: 'array',
fields: [
{
// path: 'array.[n].nestedArray.[n].fieldWithinNestedArray'
// schemaPath: 'array.nestedArray.fieldWithinNestedArray'
// indexPath: ''
name: 'fieldWithinNestedArray',
type: 'text',
},
],
},
{
// path: 'array.[n]._index-2'
// schemaPath: 'array._index-2'
// indexPath: '2'
type: 'row',
fields: [
{
// path: 'array.[n].fieldWithinRowWithinArray'
// schemaPath: 'array._index-2.fieldWithinRowWithinArray'
// indexPath: ''
name: 'fieldWithinRowWithinArray',
type: 'text',
},
],
},
],
},
{
// path: '_index-2'
// schemaPath: '_index-2'
// indexPath: '2'
type: 'row',
fields: [
{
// path: 'fieldWithinRow'
// schemaPath: '_index-2.fieldWithinRow'
// indexPath: ''
name: 'fieldWithinRow',
type: 'text',
},
],
},
{
// path: '_index-3'
// schemaPath: '_index-3'
// indexPath: '3'
type: 'tabs',
tabs: [
{
// path: '_index-3-0'
// schemaPath: '_index-3-0'
// indexPath: '3-0'
label: 'Unnamed Tab',
fields: [
{
// path: 'fieldWithinUnnamedTab'
// schemaPath: '_index-3-0.fieldWithinUnnamedTab'
// indexPath: ''
name: 'fieldWithinUnnamedTab',
type: 'text',
},
{
// path: '_index-3-0-1'
// schemaPath: '_index-3-0-1'
// indexPath: '3-0-1'
type: 'tabs',
tabs: [
{
// path: '_index-3-0-1-0'
// schemaPath: '_index-3-0-1-0'
// indexPath: '3-0-1-0'
label: 'Nested Unnamed Tab',
fields: [
{
// path: 'fieldWithinNestedUnnamedTab'
// schemaPath: '_index-3-0-1-0.fieldWithinNestedUnnamedTab'
// indexPath: ''
name: 'fieldWithinNestedUnnamedTab',
type: 'text',
},
],
},
],
},
],
},
{
// path: 'namedTab'
// schemaPath: '_index-3.namedTab'
// indexPath: ''
label: 'Named Tab',
name: 'namedTab',
fields: [
{
// path: 'namedTab.fieldWithinNamedTab'
// schemaPath: '_index-3.namedTab.fieldWithinNamedTab'
// indexPath: ''
name: 'fieldWithinNamedTab',
type: 'text',
},
],
},
],
},
]
}
```
### What?
Previously, field error messages displayed in toast notifications used
the field path to reference fields that failed validation. This
path-based approach was necessary to distinguish between fields that
might share the same name when nested inside arrays, groups, rows, or
collapsible fields.
However, the human readability of these paths was lacking, especially
for unnamed fields like rows and collapsible fields. For example:
- A text field inside a row could display as: `_index-0.text`
- A text field nested within multiple arrays could display as:
`items.0.subArray.0.text`
These outputs are technically correct but not user-friendly.
### Why?
While the previous format was helpful for pinpointing the specific field
that caused the validation error, it could be more user-friendly and
clearer to read. The goal is to maintain the same level of accuracy
while improving the readability for both developers and content editors.
### How?
To improve readability, the following changes were made:
1. Use Field Labels Instead of Field Paths:
- The ValidationError component now uses the label prop from the field
config (if available) instead of the field’s name.
- If a label is provided, it will be used in the error message.
- If no label exists, it will fall back to the field’s name.
2. Remove _index from Paths for Unnamed Fields (In the validationError
component only):
- For unnamed fields like rows and collapsibles, the _index prefix is
now stripped from the output to make it cleaner.
- Instead of `_index-0.text`, it now outputs just `Text`.
3. Reformat the Error Path for Readability:
- The error message format has been improved to be more human-readable,
showing the field hierarchy in a structured way with array indices
converted to 1-based numbers.
#### Example transformation:
##### Before:
The following fields are invalid: `items.0.subArray.0.text`
##### After:
The following fields are invalid: `Items 1 > SubArray 1 > Text`
Based on https://github.com/payloadcms/payload/pull/10154
If the actual database schema is not changed (no new columns, enums,
indexes, tables) - skip calling Drizzle push. This, potentially can
significantly reduce overhead on reloads in development mode especially
when using remote databases.
If for whatever reason you need to preserve the current behavior you can
use `PAYLOAD_FORCE_DRIZZLE_PUSH=true` env flag.
IDs that are supplied directly through the API, such as client-side
generated IDs when adding new blocks and array rows, are overwritten on
create. This is because when adding blocks or array rows on the client,
their IDs are generated first before being sent to the server for
processing. Then when the server receives this data, it incorrectly
overrides them to ensure they are unique when using relational DBs. But
this only needs to happen when no ID was supplied on create, or
specifically when duplicating documents via the `beforeDuplicate` hook.
### What?
Exposes ability to enable
[AUTOINCREMENT](https://www.sqlite.org/autoinc.html) for Primary Keys
which ensures that the same ID cannot be reused from previously deleted
rows.
```ts
sqliteAdapter({
autoIncrement: true
})
```
### Why?
This may be essential for some systems. Enabled `autoIncrement: true`
also for the SQLite Adapter in our tests, which can be useful when
testing whether the doc was deleted or not when you also have other
create operations.
### How?
Uses Drizzle's `autoIncrement` option.
WARNING:
This cannot be enabled in an existing project without a custom
migration, as it completely changes how primary keys are stored in the
database.
This PR allows to have full type safety on `payload.drizzle` with a
single command
```sh
pnpm payload generate:db-schema
```
Which generates TypeScript code with Drizzle declarations based on the
current database schema.
Example of generated file with the website template:
https://gist.github.com/r1tsuu/b8687f211b51d9a3a7e78ba41e8fbf03
Video that shows the power:
https://github.com/user-attachments/assets/3ced958b-ec1d-49f5-9f51-d859d5fae236
We also now proxy drizzle package the same way we do for Lexical so you
don't have to install it (and you shouldn't because you may have version
mismatch).
Instead, you can import from Drizzle like this:
```ts
import {
pgTable,
index,
foreignKey,
integer,
text,
varchar,
jsonb,
boolean,
numeric,
serial,
timestamp,
uniqueIndex,
pgEnum,
} from '@payloadcms/db-postgres/drizzle/pg-core'
import { sql } from '@payloadcms/db-postgres/drizzle'
import { relations } from '@payloadcms/db-postgres/drizzle/relations'
```
Fixes https://github.com/payloadcms/payload/discussions/4318
In the future we can also support types generation for mongoose / raw
mongodb results.
