In case, if `payload.db.updateOne` received simple data, meaning no:
* Arrays / Blocks
* Localized Fields
* `hasMany: true` text / select / number / relationship fields
* relationship fields with `relationTo` as an array
This PR simplifies the logic to a single SQL `set` call. No any extra
(useless) steps with rewriting all the arrays / blocks / localized
tables even if there were no any changes to them. However, it's good to
note that `payload.update` (not `payload.db.updateOne`) as for now
passes all the previous data as well, so this change won't have any
effect unless you're using `payload.db.updateOne` directly (or for our
internal logic that uses it), in the future a separate PR with
optimization for `payload.update` as well may be implemented.
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1210710489889576
Previously, `db.updateOne` calls with `where` queries that lead to no
results would create new rows on drizzle. Essentially, `db.updateOne`
behaved like `db.upsertOne` on drizzle
Adds support for `halfvec` and `sparsevec` and `bit` (binary vector)
column types. This is required for supporting indexing of embeddings >
2000 dimensions on postgres using the pg-vector extension.
If you (using the MongoDB adapter) delete a block from the payload
config, but still have some data with that block in the DB, you'd
receive in the admin panel an error like:
```
Block with type "cta" was found in block data, but no block with that type is defined in the config for field with schema path pages.blocks
```
Now, we remove those "unknown" blocks at the DB adapter level.
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
We were running scripts as they were without encompassing our logic in a
function for jest's teardown and we were subsequently running
`process.exit(0)` which meant that tests didn't correctly return an
error status code when they failed in CI.
The following tests have been skipped as well:
```
● postgres vector custom column › should add a vector column and query it
● Sort › Local API › Orderable › should not break with existing base 62 digits
● Sort › Local API › Orderable join › should set order by default
● Sort › Local API › Orderable join › should allow setting the order with the local API
● Sort › Local API › Orderable join › should sort join docs in the correct
```
---------
Co-authored-by: Elliot DeNolf <denolfe@gmail.com>
Co-authored-by: Alessio Gravili <alessio@gravili.de>
Fixes an issue when querying deeply new relationship virtual fields with
`draft: true`. Changes the method for `where` sanitization, before it
was done in `validateSearchParam` which didn't work with versions
properly, now there's a separate `sanitizeWhereQuery` function that does
this.
Continuation of https://github.com/payloadcms/payload/pull/6245.
This PR allows you to pass `blocksAsJSON: true` to SQL adapters and the
adapter instead of aligning with the SQL preferred relation approach for
blocks will just use a simple JSON column, which can improve performance
with a large amount of blocks.
To try these changes you can install `3.43.0-internal.c5bbc84`.
Fixes https://github.com/payloadcms/payload/issues/12628
When using sqlite, the error from the db is a bit different than
Postgres.
This PR allows us to extract the fieldName when using sqlite for the
unique constraint error.
Adds support for read replicas
https://orm.drizzle.team/docs/read-replicas that can be used to offload
read-heavy traffic.
To use (both `db-postgres` and `db-vercel-postgres` are supported):
```ts
import { postgresAdapter } from '@payloadcms/db-postgres'
database: postgresAdapter({
pool: {
connectionString: process.env.POSTGRES_URL,
},
readReplicas: [process.env.POSTGRES_REPLICA_URL],
})
```
---------
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
This clarifies that jobs.autoRun only *runs* already-queued jobs. It does not queue the jobs for you.
Also adds an e2e test as this functionality had no e2e coverage
Previously, this was possible in MongoDB but not in Postgres/SQLite
(having `null` in an `in` query)
```
const { docs } = await payload.find({
collection: 'posts',
where: { text: { in: ['text-1', 'text-3', null] } },
})
```
This PR fixes that behavior
Fixes#6871
To review this PR, use `pnpm dev lexical` and the auto-created document
in the `lexical fields` collection. Select any input within the blocks
and press `cmd+a`. The selection should contain the entire input.
I made sure that `cmd+a` still works fine inside the editor but outside
of inputs.
I think it's easier to review this PR commit by commit, so I'll explain
it this way:
## Commits
1. [parallelize eslint script (still showing logs results in
serial)](c9ac49c12d):
Previously, `--concurrency 1` was added to the script to make the logs
more readable. However, turborepo has an option specifically for these
use cases: `--log-order=grouped` runs the tasks in parallel but outputs
them serially. As a result, the lint script is now significantly faster.
2. [run pnpm
lint:fix](9c128c276a)
The auto-fix was run, which resolved some eslint errors that were
slipped in due to the use of `no-verify`. Most of these were
`perfectionist` fixes (property ordering) and the removal of unnecessary
assertions. Starting with this PR, this won't happen again in the
future, as we'll be verifying the linter in every PR across the entire
codebase (see commit 7).
3. [fix eslint non-autofixable
errors](700f412a33)
All manual errors have been resolved except for the configuration errors
addressed in commit 5. Most were React compiler violations, which have
been disabled and commented out "TODO" for now. There's also an unused
`use no memo` and a couple of `require` errors.
4. [move react-compiler linter to eslint-config
package](4f7cb4d63a)
To simplify the eslint configuration. My concern was that there would be
a performance regression when used in non-react related packages, but
none was experienced. This is probably because it only runs on .tsx
files.
5. [remove redundant eslint config files and fix
allowDefaultProject](a94347995a)
The main feature introduced by `typescript-eslint` v8 was
`projectService`, which automatically searches each file for the closest
`tsconfig`, greatly simplifying configuration in monorepos
([source](https://typescript-eslint.io/blog/announcing-typescript-eslint-v8#project-service)).
Once I moved `projectService` to `packages/eslint-config`, all the other
configuration files could be easily removed.
I confirmed that pnpm lint still works on individual packages.
The other important change was that the pending eslint errors from
commits 2 and 3 were resolved. That is, some files were giving the
error: "[File] was not found by the project service. Consider either
including it in the tsconfig.json or including it in
allowDefaultProject." Below I copy the explanatory comment I left in the
code:
```ts
// This is necessary because `tsconfig.base.json` defines `"rootDir": "${configDir}/src"`,
// And the following files aren't in src because they aren't transpiled.
// This is typescript-eslint's way of adding files that aren't included in tsconfig.
// See: https://typescript-eslint.io/troubleshooting/typed-linting/#i-get-errors-telling-me--was-not-found-by-the-project-service-consider-either-including-it-in-the-tsconfigjson-or-including-it-in-allowdefaultproject
// The best practice is to have a tsconfig.json that covers ALL files and is used for
// typechecking (with noEmit), and a `tsconfig.build.json` that is used for the build
// (or alternatively, swc, tsup or tsdown). That's what we should ideally do, in which case
// this hardcoded list wouldn't be necessary. Note that these files don't currently go
// through ts, only through eslint.
```
6. [Differentiate errors from warnings in VScode ESLint
Rules](5914d2f48d)
There's no reason to do that. If an eslint rule isn't an error, it
should be disabled or converted to a warning.
7. [Disable skip lint, and lint over the entire repo now that it's
faster](e4b28f1360)
The GitHub action linted only the files that had changed in the PR.
While this seems like a good idea, once exceptions were introduced with
[skip lint], they opened the door to propagating more and more errors.
Often, the linter was skipped, not because someone introduced new
errors, but because they were trying to avoid those that had already
crept in, sometimes accidentally introducing new ones.
On the other hand, `pnpm lint` now runs in parallel (commit 1), so it's
not that slow. Additionally, it runs in parallel with other GitHub
actions like e2e tests, which take much longer, so it can't represent a
bottleneck in CI.
8. [fix lint in next
package](4506595f91)
Small fix missing from commit 5
9. [Merge remote-tracking branch 'origin/main' into
fix-eslint](563d4909c1)
10. [add again eslint.config.js in payload
package](78f6ffcae7)
The comment in the code explains it. Basically, after the merge from
main, the payload package runs out of memory when linting, probably
because it grew in recent PRs. That package will sooner or later
collapse for our tooling, so we may have to split it. It's already too
big.
## Future Actions
- Resolve React compiler violations, as mentioned in commit 3.
- Decouple the `tsconfig` used for typechecking and build across the
entire monorepo (as explained in point 5) to ensure ts coverage even for
files that aren't transpiled (such as scripts).
- Remove the few remaining `eslint.config.js`. I had to leave the
`richtext-lexical` and `next` ones for now. They could be moved to the
root config and scoped to their packages, as we do for example with
`templates/vercel-postgres/**`. However, I couldn't get it to work, I
don't know why.
- Make eslint in the test folder usable. Not only are we not linting
`test` in CI, but now the `pnpm eslint .` command is so large that my
computer freezes. If each suite were its own package, this would be
solved, and dynamic codegen + git hooks to modify tsconfig.base.json
wouldn't be necessary
([related](https://github.com/payloadcms/payload/pull/11984)).
Previously, `hidden: true` on a virtual field that references a
relationship field didn't work. Now, this field doesn't get calculated
if there's `hidden: true` and no `showHiddenFields` was passed.
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
Continuation of https://github.com/payloadcms/payload/pull/12265.
Currently, using `select` on new relationship virtual fields:
```
const doc = await payload.findByID({
collection: 'virtual-relations',
depth: 0,
id,
select: { postTitle: true },
})
```
doesn't work, because in order to calculate `post.title`, the `post`
field must be selected as well. This PR adds logic that sanitizes the
incoming `select` to include those relationships into `select` (that are
related to selected virtual fields)
---------
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
When doing `payload.db.queryDrafts` with `select` without `version`, or
simply your select looks like:
`select: { version: { nonExistingField: true } }` - the `queryDrafts`
function will crash because it tries to access the `version` field.
This PR adds a fallback.
This PR optimizes the new virtual fields with relationships feature
https://github.com/payloadcms/payload/pull/11805 when the path
references the ID field, for example:
```
{
name: 'postCategoryID',
type: 'number',
virtual: 'post.category.id',
},
```
Previously, we did additional population of `category`, which is
unnecessary as we can always grab the ID from the `category` value
itself. One less querying step.
This PR adds an ability to specify a virtual field in this way
```js
{
slug: 'posts',
fields: [
{
name: 'title',
type: 'text',
required: true,
},
],
},
{
slug: 'virtual-relations',
fields: [
{
name: 'postTitle',
type: 'text',
virtual: 'post.title',
},
{
name: 'post',
type: 'relationship',
relationTo: 'posts',
},
],
},
```
Then, every time you query `virtual-relations`, `postTitle` will be
automatically populated (even if using `depth: 0`) on the db level. This
field also, unlike `virtual: true` is available for querying / sorting /
`useAsTitle`.
Also, the field can be deeply nested to 2 or more relationships, for
example:
```
{
name: 'postCategoryTitle',
type: 'text',
virtual: 'post.category.title',
},
```
Where the current collection has `post` - a relationship to `posts`, the
collection `posts` has `category` that's a relationship to `categories`
and finally `categories` has `title`.
### What?
Converts numbers passed to a text field to avoid the database/drizzle
from converting it incorrectly.
### Why?
If you have a hook that passes a value to another field you can
experience this problem where drizzle converts a number value for a text
field to a floating point number in sqlite for example.
### How?
Adds logic to `transform/write/traverseFields.ts` to cast text field
values to string.
Fixes https://github.com/payloadcms/payload/issues/11975
Previously, this configuration was causing errors in postgres due to
long names, even though `dbName` is used:
```
{
slug: 'aliases',
fields: [
{
name: 'thisIsALongFieldNameThatWillCauseAPostgresErrorEvenThoughWeSetAShorterDBName',
dbName: 'shortname',
type: 'array',
fields: [
{
name: 'nested_field_1',
type: 'array',
dbName: 'short_nested_1',
fields: [],
},
{
name: 'nested_field_2',
type: 'text',
},
],
},
],
},
```
This is because we were generating Drizzle relation name (for arrays)
always based on the field path and internally, drizzle uses this name
for aliasing. Now, if `dbName` is present, we use `_{dbName}` instead
for the relation name.
Significantly optimizes the component rendering strategy within the form
state endpoint by precisely rendering only the fields that require it.
This cuts down on server processing and network response sizes when
invoking form state requests **that manipulate array and block rows
which contain server components**, such as rich text fields, custom row
labels, etc. (results listed below).
Here's a breakdown of the issue:
Previously, when manipulating array and block fields, _all_ rows would
render any server components that might exist within them, including
rich text fields. This means that subsequent changes to these fields
would potentially _re-render_ those same components even if they don't
require it.
For example, if you have an array field with a rich text field within
it, adding the first row would cause the rich text field to render,
which is expected. However, when you add a second row, the rich text
field within the first row would render again unnecessarily along with
the new row.
This is especially noticeable for fields with many rows, where every
single row processes its server components and returns RSC data. And
this does not only affect nested rich text fields, but any custom
component defined on the field level, as these are handled in the same
way.
The reason this was necessary in the first place was to ensure that the
server components receive the proper data when they are rendered, such
as the row index and the row's data. Changing one of these rows could
cause the server component to receive the wrong data if it was not
freshly rendered.
While this is still a requirement that rows receive up-to-date props, it
is no longer necessary to render everything.
Here's a breakdown of the actual fix:
This change ensures that only the fields that are actually being
manipulated will be rendered, rather than all rows. The existing rows
will remain in memory on the client, while the newly rendered components
will return from the server. For example, if you add a new row to an
array field, only the new row will render its server components.
To do this, we send the path of the field that is being manipulated to
the server. The server can then use this path to determine for itself
which fields have already been rendered and which ones need required
rendering.
## Results
The following results were gathered by booting up the `form-state` test
suite and seeding 100 array rows, each containing a rich text field. To
invoke a form state request, we navigate to a document within the
"posts" collection, then add a new array row to the list. The result is
then saved to the file system for comparison.
| Test Suite | Collection | Number of Rows | Before | After | Percentage
Change |
|------|------|---------|--------|--------|--------|
| `form-state` | `posts` | 101 | 1.9MB / 266ms | 80KB / 70ms | ~96%
smaller / ~75% faster |
---------
Co-authored-by: James <james@trbl.design>
Co-authored-by: Alessio Gravili <alessio@gravili.de>
Fixes https://github.com/payloadcms/payload/issues/11882
Previously, down migration that dropped the `payload_migrations` table
was failing because `migrationTableExists` doesn't check the current
transaction, only in which you can get a `false` value result.
### What?
Extends our dataloader to add a momiozed payload find function. This way
it will cache the query for the same find request using a cacheKey from
find operation args.
### Why?
This was needed internally for `filterOptions` that exist in an array or
other sitautions where you have the same exact query being made and
awaited many times.
### How?
- Added `find` to payloadDataLoader. Marked `@experimental` in case it
needs to change.
- Created a cache key from the args
- Validate filterOptions changed from `payload.find` to
`payloadDataLoader.find`
- Made `payloadDataLoader` no longer optional on `PayloadRequest`, since
other args are required which are created from createLocalReq (context
for example), I don't see a reason why dataLoader shouldn't be required
also.
Example usage:
```ts
const result = await req.payloadDataLoader.find({
collection,
req,
where,
})
```
Fixes https://github.com/payloadcms/payload/issues/6884
Adds a new flag `acceptIDOnCreate` that allows you to thread your own
`id` to `payload.create` `data`, for example:
```ts
// doc created with id 1
const doc = await payload.create({ collection: 'posts', data: {id: 1, title: "my title"}})
```
```ts
import { Types } from 'mongoose'
const id = new Types.ObjectId().toHexString()
const doc = await payload.create({ collection: 'posts', data: {id, title: "my title"}})
```
This change makes so that data that exists in MongoDB but isn't defined
in the Payload config won't be included to `payload.find` /
`payload.db.find` calls. Now we strip all the additional keys.
Consider you have a field named `secretField` that's also `hidden: true`
(or `read: () => false`) that contains some sensitive data. Then you
removed this field from the database and as for now with the MongoDB
adapter this field will be included to the Local API / REST API results
without any consideration, as Payload doesn't know about it anymore.
This also fixes https://github.com/payloadcms/payload/issues/11542 if
you removed / renamed a relationship field from the schema, Payload
won't sanitize ObjectIDs back to strings anymore.
Ideally you should create a migration script that completely removes the
deleted field from the database with `$unset`, but people rarely do
this.
If you still need to keep those fields to the result, this PR allows you
to do this with the new `allowAdditionalKeys: true` flag.
### What?
This PR adds ability to define indexes on several fields for collections
(compound indexes).
Example:
```ts
{
indexes: [{ unique: true, fields: ['title', 'group.name'] }]
}
```
### Why?
This can be used to either speed up querying/sorting by 2 or more fields
at the same time or to ensure uniqueness between several fields.
### How?
Implements this logic in database adapters. Additionally, adds a utility
`getFieldByPath`.
This PR adds a new `limit` property to `payload.db.updateMany`. This functionality is required for [migrating our job system to use faster, direct db adapter calls](https://github.com/payloadcms/payload/pull/11489)
### What?
If you had multiple operator constraints on a single field, the last one
defined would be the only one used.
Example:
```ts
where: {
id: {
in: [doc2.id],
not_in: [], // <-- only respected this operator constraint
},
}
```
and
```ts
where: {
id: {
not_in: [],
in: [doc2.id], // <-- only respected this operator constraint
},
}
```
They would yield different results.
### Why?
The results were not merged into an `$and` query inside parseParams.
### How?
Merges the results within an `$and` constraint.
Fixes https://github.com/payloadcms/payload/issues/10944
Supersedes https://github.com/payloadcms/payload/pull/11011
Previously, data for globals was inconsistent across database adapters.
In Postgres, globals didn't store correct `createdAt`, `updatedAt`
fields and the `updateGlobal` lacked the `globalType` field. This PR
solves that without introducing schema changes.
Field paths within hooks are not correct.
For example, an unnamed tab containing a group field and nested text
field should have the path:
- `myGroupField.myTextField`
However, within hooks that path is formatted as:
- `_index-1.myGroupField.myTextField`
The leading index shown above should not exist, as this field is
considered top-level since it is located within an unnamed tab.
This discrepancy is only evident through the APIs themselves, such as
when creating a request with invalid data and reading the validation
errors in the response. Form state contains proper field paths, which is
ultimately why this issue was never caught. This is because within the
admin panel we merge the API response with the current form state,
obscuring the underlying issue. This becomes especially obvious in
#10580, where we no longer initialize validation errors within form
state until the form has been submitted, and instead rely solely on the
API response for the initial error state.
Here's comprehensive example of how field paths _should_ be formatted:
```
{
// ...
fields: [
{
// path: 'topLevelNamedField'
// schemaPath: 'topLevelNamedField'
// indexPath: ''
name: 'topLevelNamedField',
type: 'text',
},
{
// path: 'array'
// schemaPath: 'array'
// indexPath: ''
name: 'array',
type: 'array',
fields: [
{
// path: 'array.[n].fieldWithinArray'
// schemaPath: 'array.fieldWithinArray'
// indexPath: ''
name: 'fieldWithinArray',
type: 'text',
},
{
// path: 'array.[n].nestedArray'
// schemaPath: 'array.nestedArray'
// indexPath: ''
name: 'nestedArray',
type: 'array',
fields: [
{
// path: 'array.[n].nestedArray.[n].fieldWithinNestedArray'
// schemaPath: 'array.nestedArray.fieldWithinNestedArray'
// indexPath: ''
name: 'fieldWithinNestedArray',
type: 'text',
},
],
},
{
// path: 'array.[n]._index-2'
// schemaPath: 'array._index-2'
// indexPath: '2'
type: 'row',
fields: [
{
// path: 'array.[n].fieldWithinRowWithinArray'
// schemaPath: 'array._index-2.fieldWithinRowWithinArray'
// indexPath: ''
name: 'fieldWithinRowWithinArray',
type: 'text',
},
],
},
],
},
{
// path: '_index-2'
// schemaPath: '_index-2'
// indexPath: '2'
type: 'row',
fields: [
{
// path: 'fieldWithinRow'
// schemaPath: '_index-2.fieldWithinRow'
// indexPath: ''
name: 'fieldWithinRow',
type: 'text',
},
],
},
{
// path: '_index-3'
// schemaPath: '_index-3'
// indexPath: '3'
type: 'tabs',
tabs: [
{
// path: '_index-3-0'
// schemaPath: '_index-3-0'
// indexPath: '3-0'
label: 'Unnamed Tab',
fields: [
{
// path: 'fieldWithinUnnamedTab'
// schemaPath: '_index-3-0.fieldWithinUnnamedTab'
// indexPath: ''
name: 'fieldWithinUnnamedTab',
type: 'text',
},
{
// path: '_index-3-0-1'
// schemaPath: '_index-3-0-1'
// indexPath: '3-0-1'
type: 'tabs',
tabs: [
{
// path: '_index-3-0-1-0'
// schemaPath: '_index-3-0-1-0'
// indexPath: '3-0-1-0'
label: 'Nested Unnamed Tab',
fields: [
{
// path: 'fieldWithinNestedUnnamedTab'
// schemaPath: '_index-3-0-1-0.fieldWithinNestedUnnamedTab'
// indexPath: ''
name: 'fieldWithinNestedUnnamedTab',
type: 'text',
},
],
},
],
},
],
},
{
// path: 'namedTab'
// schemaPath: '_index-3.namedTab'
// indexPath: ''
label: 'Named Tab',
name: 'namedTab',
fields: [
{
// path: 'namedTab.fieldWithinNamedTab'
// schemaPath: '_index-3.namedTab.fieldWithinNamedTab'
// indexPath: ''
name: 'fieldWithinNamedTab',
type: 'text',
},
],
},
],
},
]
}
```
### What?
Previously, field error messages displayed in toast notifications used
the field path to reference fields that failed validation. This
path-based approach was necessary to distinguish between fields that
might share the same name when nested inside arrays, groups, rows, or
collapsible fields.
However, the human readability of these paths was lacking, especially
for unnamed fields like rows and collapsible fields. For example:
- A text field inside a row could display as: `_index-0.text`
- A text field nested within multiple arrays could display as:
`items.0.subArray.0.text`
These outputs are technically correct but not user-friendly.
### Why?
While the previous format was helpful for pinpointing the specific field
that caused the validation error, it could be more user-friendly and
clearer to read. The goal is to maintain the same level of accuracy
while improving the readability for both developers and content editors.
### How?
To improve readability, the following changes were made:
1. Use Field Labels Instead of Field Paths:
- The ValidationError component now uses the label prop from the field
config (if available) instead of the field’s name.
- If a label is provided, it will be used in the error message.
- If no label exists, it will fall back to the field’s name.
2. Remove _index from Paths for Unnamed Fields (In the validationError
component only):
- For unnamed fields like rows and collapsibles, the _index prefix is
now stripped from the output to make it cleaner.
- Instead of `_index-0.text`, it now outputs just `Text`.
3. Reformat the Error Path for Readability:
- The error message format has been improved to be more human-readable,
showing the field hierarchy in a structured way with array indices
converted to 1-based numbers.
#### Example transformation:
##### Before:
The following fields are invalid: `items.0.subArray.0.text`
##### After:
The following fields are invalid: `Items 1 > SubArray 1 > Text`
