Fixes https://github.com/payloadcms/payload/issues/11888
Previously, if you had `disableLocalStategy: true` and a custom
`password` field, Payload would still control it in `update.ts` by
deleting. Now, we don't do that in this case, unless we have
`disableLocalStetegy.enableFields: true`.
Significantly optimizes the component rendering strategy within the form
state endpoint by precisely rendering only the fields that require it.
This cuts down on server processing and network response sizes when
invoking form state requests **that manipulate array and block rows
which contain server components**, such as rich text fields, custom row
labels, etc. (results listed below).
Here's a breakdown of the issue:
Previously, when manipulating array and block fields, _all_ rows would
render any server components that might exist within them, including
rich text fields. This means that subsequent changes to these fields
would potentially _re-render_ those same components even if they don't
require it.
For example, if you have an array field with a rich text field within
it, adding the first row would cause the rich text field to render,
which is expected. However, when you add a second row, the rich text
field within the first row would render again unnecessarily along with
the new row.
This is especially noticeable for fields with many rows, where every
single row processes its server components and returns RSC data. And
this does not only affect nested rich text fields, but any custom
component defined on the field level, as these are handled in the same
way.
The reason this was necessary in the first place was to ensure that the
server components receive the proper data when they are rendered, such
as the row index and the row's data. Changing one of these rows could
cause the server component to receive the wrong data if it was not
freshly rendered.
While this is still a requirement that rows receive up-to-date props, it
is no longer necessary to render everything.
Here's a breakdown of the actual fix:
This change ensures that only the fields that are actually being
manipulated will be rendered, rather than all rows. The existing rows
will remain in memory on the client, while the newly rendered components
will return from the server. For example, if you add a new row to an
array field, only the new row will render its server components.
To do this, we send the path of the field that is being manipulated to
the server. The server can then use this path to determine for itself
which fields have already been rendered and which ones need required
rendering.
## Results
The following results were gathered by booting up the `form-state` test
suite and seeding 100 array rows, each containing a rich text field. To
invoke a form state request, we navigate to a document within the
"posts" collection, then add a new array row to the list. The result is
then saved to the file system for comparison.
| Test Suite | Collection | Number of Rows | Before | After | Percentage
Change |
|------|------|---------|--------|--------|--------|
| `form-state` | `posts` | 101 | 1.9MB / 266ms | 80KB / 70ms | ~96%
smaller / ~75% faster |
---------
Co-authored-by: James <james@trbl.design>
Co-authored-by: Alessio Gravili <alessio@gravili.de>
### What?
Makes several fields and list item types in query results (e.g. `docs`)
non-nullable.
### Why?
When dealing with code generated from a Payload GraphQL schema, it is
often necessary to use type guards and optional chaining.
For example:
```graphql
type Posts {
docs: [Post]
...
}
```
This implies that the `docs` field itself is nullable and that the array
can contain nulls. In reality, neither of these is true. But because of
the types generated by tools like `graphql-code-generator`, the way to
access `posts` ends up something like this:
```ts
const posts = (query.data.docs ?? []).filter(doc => doc != null);
```
Instead, we would like the schema to be:
```graphql
type Posts {
docs: [Post!]!
...
}
```
### How?
The proposed change involves adding `GraphQLNonNull` where appropriate.
---------
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
Fixes https://github.com/payloadcms/payload/issues/11882
Previously, down migration that dropped the `payload_migrations` table
was failing because `migrationTableExists` doesn't check the current
transaction, only in which you can get a `false` value result.
The same as https://github.com/payloadcms/payload/pull/11763 but also
for GraphQL. The previous fix was working only for the Local API and
REST API due to a different method for querying joins in GraphQL.
Continuation of #11867. When rendering custom fields nested within
arrays or blocks, such as the Lexical rich text editor which is treated
as a custom field, these fields will sometimes disappear when form state
requests are invoked sequentially. This is especially reproducible on
slow networks.
This is different from the previous PR in that this issue is caused by
adding _rows_ back-to-back, whereas the previous issue was caused when
adding a single row followed by a change to another field.
Here's a screen recording demonstrating the issue:
https://github.com/user-attachments/assets/5ecfa9ec-b747-49ed-8618-df282e64519d
The problem is that `requiresRender` is never sent in the form state
request for row 2. This is because the [task
queue](https://github.com/payloadcms/payload/pull/11579) processes tasks
within a single `useEffect`. This forces React to batch the results of
these tasks into a single rendering cycle. So if request 1 sets state
that request 2 relies on, request 2 will never use that state since
they'll execute within the same lifecycle.
Here's a play-by-play of the current behavior:
1. The "add row" event is dispatched
a. This sets `requiresRender: true` in form state
1. A form state request is sent with `requiresRender: true`
1. While that request is processing, another "add row" event is
dispatched
a. This sets `requiresRender: true` in form state
b. This adds a form state request into the queue
1. The initial form state request finishes
a. This sets `requiresRender: false` in form state
1. The next form state request that was queued up in 3b is sent with
`requiresRender: false`
a. THIS IS EXPECTED, BUT SHOULD ACTUALLY BE `true`!!
To fix this this, we need to ensure that the `requiresRender` property
is persisted into the second request instead of overridden. To do this,
we can add a new `serverPropsToIgnore` to form state which is read when
the processing results from the server. So if `requiresRender` exists in
`serverPropsToIgnore`, we do not merge it. This works because we
actually mutate form state in between requests. So request 2 can read
the results from request 1 without going through an additional rendering
cycle.
Here's a play-by-play of the fix:
1. The "add row" event is dispatched
a. This sets `requiresRender: true` in form state
b. This adds a task in the queue to mutate form state with
`requiresRender: true`
1. A form state request is sent with `requiresRender: true`
1. While that request is processing, another "add row" event is
dispatched
a. This sets `requiresRender: true` in form state AND
`serverPropsToIgnore: [ "requiresRender" ]`
c. This adds a form state request into the queue
1. The initial form state request finishes
a. This returns `requiresRender: false` from the form state endpoint BUT
IS IGNORED
1. The next form state request that was queued up in 3c is sent with
`requiresRender: true`
Previously, jobs were executed in FIFO order on MongoDB, and LIFO on
Postgres, with no way to configure this behavior.
This PR makes FIFO the default on both MongoDB and Postgres and
introduces the following new options to configure the processing order
globally or on a queue-by-queue basis:
- a `processingOrder` property to the jobs config
- a `processingOrder` argument to `payload.jobs.run()` to override
what's set in the jobs config
It also adds a new `sequential` option to `payload.jobs.run()`, which
can be useful for debugging.
This adds support for running multiple job queue tasks in parallel
within the same workflow while preventing conflicts. Previously, this
would have caused the following issues:
- Job log entries get lost - the final job log is incomplete, despite
all tasks having been executed
- Write conflicts in postgres, leading to unique constraint violation
errors
The solution involves handling job log data updates in a way that avoids
overwriting, and ensuring the final update reflects the latest job log
data. Each job log entry now initializes its own ID, so a given job log
entry’s ID remains the same across multiple, parallel task executions.
## Postgres
In Postgres, we need to enable transactions for the
`payload.db.updateJobs` operation; otherwise, two tasks updating the
same job in parallel can conflict. This happens because Postgres handles
array rows by deleting them all, then re-inserting (rather than
upserting). The rows are stored in a separate table, and the following
scenario can occur:
Op 1: deletes all job log rows
Op 2: deletes all job log rows
Op 1: inserts 200 job log rows
Op 2: insert the same 200 job log rows again => `error: “duplicate key
value violates unique constraint "payload_jobs_log_pkey”`
Because transactions were not used, the rows inserted by Op 1
immediately became visible to Op 2, causing the conflict. Enabling
transactions fixes this. In theory, it can still happen if Op 1 commits
before Op 2 starts inserting (due to the read committed isolation
level), but it should occur far less frequently.
Alongside this change, we should consider inserting the rows using an
upsert (update on conflict), which will get rid of this error
completely. That way, if the insertion of Op 1 is visible to Op 2, Op 2
will simply overwrite it, rather than erroring. Individual job entries
are immutable and job entries cannot be deleted, thus this shouldn't
corrupt any data.
## Mongo
In Mongo, the issue is addressed by ensuring that log row deletions
caused due to different log states in concurrent operations are not
merged back to the client job log, and by making sure the final update
includes all job logs.
There is no duplicate key error in Mongo because the array log resides
in the same document and duplicates are simply upserted. We cannot use
transactions in Mongo, as it appears to lock the document in a way that
prevents reliable parallel updates, leading to:
`MongoServerError: WriteConflict error: this operation conflicted with
another operation. Please retry your operation or multi-document
transaction`
### What
This PR improves the `getSafeRedirect` utility to improve security
around open redirect handling.
### How
- Normalizes and decodes the redirect path using `decodeURIComponent`
- Catches malformed encodings with a try/catch fallback
- Blocks open redirects
<!--
Thank you for the PR! Please go through the checklist below and make
sure you've completed all the steps.
Please review the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository if you haven't already.
The following items will ensure that your PR is handled as smoothly as
possible:
- PR Title must follow conventional commits format. For example, `feat:
my new feature`, `fix(plugin-seo): my fix`.
- Minimal description explained as if explained to someone not
immediately familiar with the code.
- Provide before/after screenshots or code diffs if applicable.
- Link any related issues/discussions from GitHub or Discord.
- Add review comments if necessary to explain to the reviewer the logic
behind a change
### What?
### Why?
### How?
Fixes #
-->
### What?
This PR aims to deflake the `test/versions/e2e.spec.ts:925:5 › Versions
› Collections with draft validation › - with autosave - shows a prevent
leave alert when form is submitted but invalid` e2e test.
The issue seems to be that the `fill` call followed by a `page.reload`
sometimes conflicts with autosave which may cause the test to flake.
### Why?
To deflake this test in ci.
### How?
Adds a single `waitForAutoSaveToRunAndComplete` function call prior to
the last call to `page.reload`. In my testing, on my local machine,
adding the `waitForAutoSaveToRunAndComplete` function allows the test to
pass every time. Without this, the tests fails on my machine
consistently.
### What?
This PR aims to deflake the indexed fields e2e test in
`test/fields/collections/Indexed/e2e.spec.ts`.
The issue is that this test is setup in a way where sometimes two toasts
will present themselves in the ui. The second toast assertion will fail
with a strict mode violation as the toast locator will resolve to two
elements.
### Why?
To prevent this test from flaking in ci.
### How?
Adding a new `dismissAfterAssertion` flag to the `assertToastErrors`
helper function which dismisses the toasts. This way, the toasts will
not raise the aforementioned error as they will be dismissed from the
ui.
The logic is handled in a separate loop through such that the assertions
occur first. This is done so that dismissing a toast does not surface
errors due to the order of toasts being shown changing.
### What?
Fixed client config caching to properly update when switching languages
in the admin UI.
### Why?
Currently, switching languages doesn't fully update the UI because
client config stays cached with previous language translations.
### How?
Created a language-aware caching system that stores separate configs for
each language and only uses cached config when it matches the active
language.
Before:
```typescript
let cachedClientConfig: ClientConfig | null = global._payload_clientConfig
if (!cachedClientConfig) {
cachedClientConfig = global._payload_clientConfig = null
}
export const getClientConfig = cache(
(args: { config: SanitizedConfig; i18n: I18nClient; importMap: ImportMap }): ClientConfig => {
if (cachedClientConfig && !global._payload_doNotCacheClientConfig) {
return cachedClientConfig
}
// ... create new config ...
}
);
```
After:
```typescript
let cachedClientConfigs: Record<string, ClientConfig> = global._payload_localizedClientConfigs
if (!cachedClientConfigs) {
cachedClientConfigs = global._payload_localizedClientConfigs = {}
}
export const getClientConfig = cache(
(args: { config: SanitizedConfig; i18n: I18nClient; importMap: ImportMap }): ClientConfig => {
const { config, i18n, importMap } = args
const currentLocale = i18n.language
if (!global._payload_doNotCacheClientConfig && cachedClientConfigs[currentLocale]) {
return cachedClientConfigs[currentLocale]
}
// ... create new config with correct translations ...
}
);
```
Also added handling for cache clearing during HMR to ensure
compatibility with the existing system.
Fixes#11406
---------
Co-authored-by: Jacob Fletcher <jacobsfletch@gmail.com>
Lexical nested fields are currently not set-up to handle access control
on the client properly. Despite that, we were passing parent permissions
to `RenderFields`, which causes certain fields to not show up if the
document does not have `create` permission.
In the Cell component for a select field such as our `_status` fields it
will now add a class eg. `selected--published` for the selected option
so it can be easily targeted with CSS.
---------
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
### What?
In the same vein as #11696, this PR optimizes how images are selected
for display in the document edit view. It ensures that only image files
are processed and selects the most appropriate size to minimize
unnecessary downloads and improve performance.
#### Previously:
- Non-image files were being processed unnecessarily, despite not
generating thumbnails.
- Images without a `thumbnailURL` defaulted to their original full size,
even when smaller, optimized versions were available.
#### Now:
- **Only images** are processed for thumbnails, avoiding redundant
requests for non-images.
- **The smallest available image within a target range** (`40px -
180px`) is prioritized for display.
- **If no images fit within this range**, the logic selects:
- The next smallest larger image (if available).
- The **original** image if it is smaller than the next available larger
size.
- The largest **smaller** image if no better fit exists.
### Why?
Prevents unnecessary downloads of non-image files, reduces bandwidth
usage by selecting more efficient image sizes and improves load times
and performance in the edit view.
### How?
- **Filters out non-image files** when determining which assets to
display.
- Uses the same algorithm as in #11696 but turns it into a reusable
function to be used in various areas around the codebase. Namely the
upload field hasOne and hasMany components.
Before (4.5mb transfer):
After (15.9kb transfer):
When selecting query presets from the list drawer, all query presets are
available for selection, even if unrelated to the underlying collection.
When selecting one of these presets, the list view will crash with
client-side exceptions because the columns and filters that are applied
are incompatible.
The fix is to the thread `filterOptions` through the query presets
drawer. This will ensure that only related collections are shown.
When rendering custom fields nested within arrays or blocks, such as the
Lexical rich text editor which is treated as a custom field, these
fields will sometimes disappear when form state requests are invoked
sequentially. This is especially reproducible on slow networks.
This is because form state invocations are placed into a [task
queue](https://github.com/payloadcms/payload/pull/11579) which aborts
the currently running tasks when a new one arrives. By doing this, local
form state is never dispatched, and the second task in the queue becomes
stale.
The fix is to _not_ abort the currently running task. This will trigger
a complete rendering cycle, and when the second task is invoked, local
state will be up to date.
Fixes#11340, #11425, and #11824.
Previously, many test cases in `int/relationships` were wrapped to the
"custom IDs" describe block even though they aren't related to custom
IDs at all. This rearranges them as they should be.
Fixes#11458
Some complex, nested fields were receiving incorrect field paths and
schema paths, leading to a `"Error: No client field found"` error.
This PR ensures field paths are calculated correctly, by matching it to
how they're calculated in payload hooks.
Query Presets allow you to save and share filters, columns, and sort
orders for your collections. This is useful for reusing common or
complex filtering patterns and column configurations across your team.
Query Presets are defined on the fly by the users of your app, rather
than being hard coded into the Payload Config.
Here's a screen recording demonstrating the general workflow as it
relates to the list view. Query Presets are not exclusive to the admin
panel, however, as they could be useful in a number of other contexts
and environments.
https://github.com/user-attachments/assets/1fe1155e-ae78-4f59-9138-af352762a1d5
Each Query Preset is saved as a new record in the database under the
`payload-query-presets` collection. This will effectively make them
CRUDable and allows for an endless number of preset configurations. As
you make changes to filters, columns, limit, etc. you can choose to save
them as a new record and optionally share them with others.
Normal document-level access control will determine who can read,
update, and delete these records. Payload provides a set of sensible
defaults here, such as "only me", "everyone", and "specific users", but
you can also extend your own set of access rules on top of this, such as
"by role", etc. Access control is customizable at the operation-level,
for example you can set this to "everyone" can read, but "only me" can
update.
To enable the Query Presets within a particular collection, set
`enableQueryPresets` on that collection's config.
Here's an example:
```ts
{
// ...
enableQueryPresets: true
}
```
Once enabled, a new set of controls will appear within the list view of
the admin panel. This is where you can select and manage query presets.
General settings for Query Presets are configured under the root
`queryPresets` property. This is where you can customize the labels,
apply custom access control rules, etc.
Here's an example of how you might augment the access control properties
with your own custom rule to achieve RBAC:
```ts
{
// ...
queryPresets: {
constraints: {
read: [
{
label: 'Specific Roles',
value: 'specificRoles',
fields: [roles],
access: ({ req: { user } }) => ({
'access.update.roles': {
in: [user?.roles],
},
}),
},
],
}
}
}
```
Related: #4193 and #3092
---------
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
Previously, if you were querying a collection that has a join field with
`draft: true`, and the join field's collection also has
`versions.drafts: true` our db adapter would still query the original
SQL table / mongodb collection instead of the versions one which isn't
quite right since we respect `draft: true` when populating relationships
**BREAKING CHANGE:**
This bumps the **minimum required Next.js** version from 15.0.0 to
15.2.3. This update is necessary due to a critical security
vulnerability found in earlier Next.js versions, which requires an
exception to our standard semantic versioning process.
Additionally, this bumps all templates to the latest Next.js and Payload
versions.
This is already supported in Postgres / SQLite.
For example:
```
const result = await payload.find({
collection: 'directors',
depth: 0,
sort: '-movies.name', // movies is a relationship field here
})
```
Removes the condition in tests:
```
// no support for sort by relation in mongodb
if (isMongoose(payload)) {
return
}
```
Since the introduction of loading states in nested fields, i.e. array
and block rows, the conditional logic tests would fail periodically
because it wouldn't wait for loading states to resolve before
continuing. This has been increasingly flaky since the introduction of
form state queues.
Previously, our job queue system relied on `payload.*` operations, which
ran very frequently:
- whenever job execution starts, as all jobs need to be set to
`processing: true`
- every single time a task completes or fails, as the job log needs to
be updated
- whenever job execution stops, to mark it as completed and to delete it
(if `deleteJobOnComplete` is set)
This PR replaces these with direct `payload.db.*` calls, which are
significantly faster than payload operations. Given how often the job
queue system communicates with the database, this should be a massive
performance improvement.
## How it affects running hooks
To generate the task status, we previously used an `afterRead` hook.
Since direct db adapter calls no longer execute hooks, this PR
introduces new `updateJob` and `updateJobs` helpers to handle task
status generation outside the normal payload hook lifecycle.
Additionally, a new `runHooks` property has been added to the global job
configuration. While setting this to `true` can be useful if custom
hooks were added to the `payload-jobs` collection config, this will
revert the job system to use normal payload operations.
This should be avoided as it degrades performance. In most cases, the
`onSuccess` or `onFail` properties in the job config will be sufficient
and much faster.
Furthermore, if the `depth` property is set in the global job
configuration, the job queue system will also fall back to the slower,
normal payload operations.
---------
Co-authored-by: Dan Ribbens <DanRibbens@users.noreply.github.com>
Fixes#10019. When bulk editing subfields, such as a field within a
group, changes are not persisted to the database. Not only this, but
nested fields with the same name as another selected field are
controlled by the same input. E.g. typing into one fields changes the
value of both.
The root problem is that field paths are incorrect.
When opening the bulk edit drawer, fields are flattened into options for
the field selector. This is so that fields in a tab, for example, aren't
hidden behind their tab when bulk editing. The problem is that
`RenderFields` is not set up to receive pre-determined field paths. It
attempts to build up its own field paths, but are never correct because
`getFieldPaths` receives the wrong arguments.
The fix is to just render the top-level fields directly, bypassing
`RenderFields` altogether.
Fields with subfields will still recurse through this function, but at
the top-level, fields can be sent directly to `RenderField` (singular)
since their paths have already been already formatted in the flattening
step.
This PR updates the email validation regex to better handle use cases
with hyphens.
Changes:
- Disallows domains starting or ending with a hyphen
(`user@-example.com`, `user@example-.com`).
- Allows domains with consecutive hyphens inside (`user@ex--ample.com`).
- Allows multiple subdomains (`user@sub.domain.example.com`).
- Adds `int test` coverage for multiple domain use case scenarios.
### What?
Extends our dataloader to add a momiozed payload find function. This way
it will cache the query for the same find request using a cacheKey from
find operation args.
### Why?
This was needed internally for `filterOptions` that exist in an array or
other sitautions where you have the same exact query being made and
awaited many times.
### How?
- Added `find` to payloadDataLoader. Marked `@experimental` in case it
needs to change.
- Created a cache key from the args
- Validate filterOptions changed from `payload.find` to
`payloadDataLoader.find`
- Made `payloadDataLoader` no longer optional on `PayloadRequest`, since
other args are required which are created from createLocalReq (context
for example), I don't see a reason why dataLoader shouldn't be required
also.
Example usage:
```ts
const result = await req.payloadDataLoader.find({
collection,
req,
where,
})
```
### What?
- GraphQL was broken because of an error with the enum for the drafts
input which cannot be 'true'.
- Selecting Draft was not doing anything as it wasn't being passed
through to the find arguments.
### Why?
This was causing any graphql calls to error.
### How?
- Changed draft options to Yes/No instead of True/False
- Correctly pass the drafts arg to `draft`
Fixes #
### What?
The import-export preview UI component does not handle localized fields
and crash the UI when they are used. This fixes that issue.
### Why?
We were not properly handling the label translated object notation that
field.label can have.
### How?
Now we call `getTranslation` with the field label to handle language
keyed labels.
Fixes # https://github.com/payloadcms/payload/issues/11668
Consolidates all bulk edit related tests into a single, dedicated suite.
Currently, bulk edit tests are dispersed throughout the Admin > General
and the Versions test suites, which are considerably bloated for their
own purposes. This made them very hard to locate, mentally digest, and
add on new tests. Going forward, many more tests specifically for bulk
edit will need to be written. This gives us a simple, isolated place for
that.
With this change are also a few improvements to the tests themselves to
make them more predictable and efficient.
### What?
This PR ensures that bulk uploads fail if any file is missing, rather
than skipping missing files and proceeding with the upload.
### Why?
This fixes unintended behavior where missing files were skipped,
allowing partial uploads when they shouldn't be allowed.
### How?
- Prevents submission if any file is missing by checking `req.status ===
400`.
- Updates `FileSidebar` to correctly handle cases where a file is
`null`.
Fixes https://github.com/payloadcms/payload/issues/6884
Adds a new flag `acceptIDOnCreate` that allows you to thread your own
`id` to `payload.create` `data`, for example:
```ts
// doc created with id 1
const doc = await payload.create({ collection: 'posts', data: {id: 1, title: "my title"}})
```
```ts
import { Types } from 'mongoose'
const id = new Types.ObjectId().toHexString()
const doc = await payload.create({ collection: 'posts', data: {id, title: "my title"}})
```
Bulk edit can now request a partial form state thanks to #11689. This
means that we only need to build form state (and send it through the
network) for the currently selected fields, as opposed to the entire
field schema.
Not only this, but there is no longer a need to filter out unselected
fields before submitting the form, as the form state will only ever
include the currently selected fields. This is unnecessary processing
and causes an excessive amount of rendering, especially since we were
dispatching actions within a `for` loop to remove each field. React may
have batched these updates, but is bad practice regardless.
Related: stripping unselected fields was also error prone. This is
because the `overrides` function we were using to do this receives
`FormState` (shallow) as an argument, but was being treated as `Data`
(not shallow, what the create and update operations expect).
E.g. `{ myGroup.myTitle: { value: 'myValue' }}` → `{ myGroup: { myTitle:
'myValue' }}`.
This led to the `sanitizeUnselectedFields` function improperly
formatting data sent to the server and would throw an API error upon
submission. This is only evident when sanitizing nested fields. Instead
of converting this data _again_, the select API takes care of this by
ensuring only selected fields exist in form state.
Related: bulk upload was not hitting form state on change. This means
that no field-level validation was occurring on type.
