Follow up to #11375.
When setting `filterOptions` on relationship or upload fields _that are
nested within a named field_, those options won't be applied to the
`Filter` component in the list view.
This is because of how we key the results when resolving `filterOptions`
on the server. Instead of using the field path as expected, we were
using the field name, causing a failed lookup on the front-end. This
also solves an issue where two fields with the same name would override
each other's `filterOptions`, since field names alone are not unique.
Unrelated: this PR also does some general housekeeping to e2e test
helpers.
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1211332845301583
Supports grouping documents by specific fields within the list view.
For example, imagine having a "posts" collection with a "categories"
field. To report on each specific category, you'd traditionally filter
for each category, one at a time. This can be quite inefficient,
especially with large datasets.
Now, you can interact with all categories simultaneously, grouped by
distinct values.
Here is a simple demonstration:
https://github.com/user-attachments/assets/0dcd19d2-e983-47e6-9ea2-cfdd2424d8b5
Enable on any collection by setting the `admin.groupBy` property:
```ts
import type { CollectionConfig } from 'payload'
const MyCollection: CollectionConfig = {
// ...
admin: {
groupBy: true
}
}
```
This is currently marked as beta to gather feedback while we reach full
stability, and to leave room for API changes and other modifications.
Use at your own risk.
Note: when using `groupBy`, bulk editing is done group-by-group. In the
future we may support cross-group bulk editing.
Dependent on #13102 (merged).
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1210774523852467
---------
Co-authored-by: Paul Popus <paul@payloadcms.com>
Some search params within the list view do not properly sync to user
preferences, and visa versa.
For example, when selecting a query preset, the `?preset=123` param is
injected into the URL and saved to preferences, but when reloading the
page without the param, that preset is not reactivated as expected.
### Problem
The reason this wasn't working before is that omitting this param would
also reset prefs. It was designed this way in order to support
client-side resets, e.g. clicking the query presets "x" button.
This pattern would never work, however, because this means that every
time the user navigates to the list view directly, their preference is
cleared, as no param would exist in the query.
Note: this is not an issue with _all_ params, as not all are handled in
the same way.
### Solution
The fix is to use empty values instead, e.g. `?preset=`. When the server
receives this, it knows to clear the pref. If it doesn't exist at all,
it knows to load from prefs. And if it has a value, it saves to prefs.
On the client, we sanitize those empty values back out so they don't
appear in the URL in the end.
This PR also refactors much of the list query context and its respective
provider to be significantly more predictable and easier to work with,
namely:
- The `ListQuery` type now fully aligns with what Payload APIs expect,
e.g. `page` is a number, not a string
- The provider now receives a single `query` prop which matches the
underlying context 1:1
- Propagating the query from the server to the URL is significantly more
predictable
- Any new props that may be supported in the future will automatically
work
- No more reconciling `columns` and `listPreferences.columns`, its just
`query.columns`
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1210827129744922
This clarifies that jobs.autoRun only *runs* already-queued jobs. It does not queue the jobs for you.
Also adds an e2e test as this functionality had no e2e coverage
I noticed a few issues when running e2e tests that will be resolved by
this PR:
- Most important: for some test suites (fields, fields-relationship,
versions, queues, lexical), the database was cleared and seeded
**twice** in between each test run. This is because the onInit function
was running the clear and seed script, when it should only have been
running the seed script. Clearing the database / the snapshot workflow
is being done by the reInit endpoint, which then calls onInit to seed
the actual data.
- The slowest part of `clearAndSeedEverything` is recreating indexes on
mongodb. This PR slightly improves performance here by:
- Skipping this process for the built-in `['payload-migrations',
'payload-preferences', 'payload-locked-documents']` collections
- Previously we were calling both `createIndexes` and `ensureIndexes`.
This was unnecessary - `ensureIndexes` is a deprecated alias of
`createIndexes`. This PR changes it to only call `createIndexes`
- Makes the reinit endpoint accept GET requests instead of POST requests
- this makes it easier to debug right in the browser
- Some typescript fixes
- Adds a `dev:memorydb` script to the package.json. For some reason,
`dev` is super unreliable on mongodb locally when running e2e tests - it
frequently fails during index creation. Using the memorydb fixes this
issue, with the bonus of more closely resembling the CI environment
- Previously, you were unable to run test suites using turbopack +
postgres. This fixes it, by explicitly installing `pg` as devDependency
in our monorepo
- Fixes jest open handles warning
You can now specify exactly who can change the constraints within a
query preset.
For example, you want to ensure that only "admins" are allowed to set a
preset to "everyone".
To do this, you can use the new `queryPresets.filterConstraints`
property. When a user lacks the permission to change a constraint, the
option will either be hidden from them or disabled if it is already set.
```ts
import { buildConfig } from 'payload'
const config = buildConfig({
// ...
queryPresets: {
// ...
filterConstraints: ({ req, options }) =>
!req.user?.roles?.includes('admin')
? options.filter(
(option) =>
(typeof option === 'string' ? option : option.value) !==
'everyone',
)
: options,
},
})
```
The `filterConstraints` functions takes the same arguments as
`reduceOptions` property on select fields introduced in #12487.
In #12322 we prevented against accidental query preset lockout by
throwing a validation error when the user is going to change the preset
in a way that removes their own access to it. This, however, puts the
responsibility on the user to make the corrections and is an unnecessary
step.
For example, the API currently forbids leaving yourself out of the
`users` array when specifying the `specificUsers` constraint, but when
you encounter this error, have to update the field manually and try
again.
To improve the experience, we now automatically inject the requesting
user onto the `users` array when this constraint is selected. This will
guarantee they have access and prevent an accidental lockout while also
avoiding the API error feedback loop.
I think it's easier to review this PR commit by commit, so I'll explain
it this way:
## Commits
1. [parallelize eslint script (still showing logs results in
serial)](c9ac49c12d):
Previously, `--concurrency 1` was added to the script to make the logs
more readable. However, turborepo has an option specifically for these
use cases: `--log-order=grouped` runs the tasks in parallel but outputs
them serially. As a result, the lint script is now significantly faster.
2. [run pnpm
lint:fix](9c128c276a)
The auto-fix was run, which resolved some eslint errors that were
slipped in due to the use of `no-verify`. Most of these were
`perfectionist` fixes (property ordering) and the removal of unnecessary
assertions. Starting with this PR, this won't happen again in the
future, as we'll be verifying the linter in every PR across the entire
codebase (see commit 7).
3. [fix eslint non-autofixable
errors](700f412a33)
All manual errors have been resolved except for the configuration errors
addressed in commit 5. Most were React compiler violations, which have
been disabled and commented out "TODO" for now. There's also an unused
`use no memo` and a couple of `require` errors.
4. [move react-compiler linter to eslint-config
package](4f7cb4d63a)
To simplify the eslint configuration. My concern was that there would be
a performance regression when used in non-react related packages, but
none was experienced. This is probably because it only runs on .tsx
files.
5. [remove redundant eslint config files and fix
allowDefaultProject](a94347995a)
The main feature introduced by `typescript-eslint` v8 was
`projectService`, which automatically searches each file for the closest
`tsconfig`, greatly simplifying configuration in monorepos
([source](https://typescript-eslint.io/blog/announcing-typescript-eslint-v8#project-service)).
Once I moved `projectService` to `packages/eslint-config`, all the other
configuration files could be easily removed.
I confirmed that pnpm lint still works on individual packages.
The other important change was that the pending eslint errors from
commits 2 and 3 were resolved. That is, some files were giving the
error: "[File] was not found by the project service. Consider either
including it in the tsconfig.json or including it in
allowDefaultProject." Below I copy the explanatory comment I left in the
code:
```ts
// This is necessary because `tsconfig.base.json` defines `"rootDir": "${configDir}/src"`,
// And the following files aren't in src because they aren't transpiled.
// This is typescript-eslint's way of adding files that aren't included in tsconfig.
// See: https://typescript-eslint.io/troubleshooting/typed-linting/#i-get-errors-telling-me--was-not-found-by-the-project-service-consider-either-including-it-in-the-tsconfigjson-or-including-it-in-allowdefaultproject
// The best practice is to have a tsconfig.json that covers ALL files and is used for
// typechecking (with noEmit), and a `tsconfig.build.json` that is used for the build
// (or alternatively, swc, tsup or tsdown). That's what we should ideally do, in which case
// this hardcoded list wouldn't be necessary. Note that these files don't currently go
// through ts, only through eslint.
```
6. [Differentiate errors from warnings in VScode ESLint
Rules](5914d2f48d)
There's no reason to do that. If an eslint rule isn't an error, it
should be disabled or converted to a warning.
7. [Disable skip lint, and lint over the entire repo now that it's
faster](e4b28f1360)
The GitHub action linted only the files that had changed in the PR.
While this seems like a good idea, once exceptions were introduced with
[skip lint], they opened the door to propagating more and more errors.
Often, the linter was skipped, not because someone introduced new
errors, but because they were trying to avoid those that had already
crept in, sometimes accidentally introducing new ones.
On the other hand, `pnpm lint` now runs in parallel (commit 1), so it's
not that slow. Additionally, it runs in parallel with other GitHub
actions like e2e tests, which take much longer, so it can't represent a
bottleneck in CI.
8. [fix lint in next
package](4506595f91)
Small fix missing from commit 5
9. [Merge remote-tracking branch 'origin/main' into
fix-eslint](563d4909c1)
10. [add again eslint.config.js in payload
package](78f6ffcae7)
The comment in the code explains it. Basically, after the merge from
main, the payload package runs out of memory when linting, probably
because it grew in recent PRs. That package will sooner or later
collapse for our tooling, so we may have to split it. It's already too
big.
## Future Actions
- Resolve React compiler violations, as mentioned in commit 3.
- Decouple the `tsconfig` used for typechecking and build across the
entire monorepo (as explained in point 5) to ensure ts coverage even for
files that aren't transpiled (such as scripts).
- Remove the few remaining `eslint.config.js`. I had to leave the
`richtext-lexical` and `next` ones for now. They could be moved to the
root config and scoped to their packages, as we do for example with
`templates/vercel-postgres/**`. However, I couldn't get it to work, I
don't know why.
- Make eslint in the test folder usable. Not only are we not linting
`test` in CI, but now the `pnpm eslint .` command is so large that my
computer freezes. If each suite were its own package, this would be
solved, and dynamic codegen + git hooks to modify tsconfig.base.json
wouldn't be necessary
([related](https://github.com/payloadcms/payload/pull/11984)).
Prevents an accidental lockout of query preset documents. An "accidental
lockout" occurs when the user sets access control on a preset and
excludes themselves. This can happen in a variety of scenarios,
including:
- You select `specificUsers` without specifying yourself
- You select `specificRoles` without specifying a role that you are a
part of
- Etc.
#### How it works
To make this happen, we use a custom validation function that executes
access against the user's proposed changes. If those changes happen to
remove access for them, we throw a validation error and prevent that
change from ever taking place. This means that only a user with proper
access can remove another user from the preset. You cannot remove
yourself.
To do this, we create a temporary record in the database that we can
query against. We use transactions to ensure that the temporary record
is not persisted once our work is completed. Since not all Payload
projects have transactions enabled, we flag these temporary records with
the `isTemp` field.
Once created, we query the temp document to determine its permissions.
If any of the operations throw an error, this means the user can no
longer act on them, and we throw a validation error.
#### Alternative Approach
A previous approach that was explored was to add an `owner` field to the
presets collection. This way, the "owner" of the preset would be able to
completely bypass all access control, effectively eliminating the
possibility of a lockout event.
But this doesn't work for other users who may have update access. E.g.
they could still accidentally remove themselves from the read or update
operation, preventing them from accessing that preset after submitting
the form. We need a solution that works for all users, not just the
owner.
Returning a boolean value from a constraint-level access control
function does nothing. For example:
```ts
{
label: 'Noone',
value: 'noone',
access: () => false,
},
```
This is because we were only handling query objects, disregarding any
boolean values. The fix is to check if the query is a boolean, and if
so, format a query object to return.
This adds support for running multiple job queue tasks in parallel
within the same workflow while preventing conflicts. Previously, this
would have caused the following issues:
- Job log entries get lost - the final job log is incomplete, despite
all tasks having been executed
- Write conflicts in postgres, leading to unique constraint violation
errors
The solution involves handling job log data updates in a way that avoids
overwriting, and ensuring the final update reflects the latest job log
data. Each job log entry now initializes its own ID, so a given job log
entry’s ID remains the same across multiple, parallel task executions.
## Postgres
In Postgres, we need to enable transactions for the
`payload.db.updateJobs` operation; otherwise, two tasks updating the
same job in parallel can conflict. This happens because Postgres handles
array rows by deleting them all, then re-inserting (rather than
upserting). The rows are stored in a separate table, and the following
scenario can occur:
Op 1: deletes all job log rows
Op 2: deletes all job log rows
Op 1: inserts 200 job log rows
Op 2: insert the same 200 job log rows again => `error: “duplicate key
value violates unique constraint "payload_jobs_log_pkey”`
Because transactions were not used, the rows inserted by Op 1
immediately became visible to Op 2, causing the conflict. Enabling
transactions fixes this. In theory, it can still happen if Op 1 commits
before Op 2 starts inserting (due to the read committed isolation
level), but it should occur far less frequently.
Alongside this change, we should consider inserting the rows using an
upsert (update on conflict), which will get rid of this error
completely. That way, if the insertion of Op 1 is visible to Op 2, Op 2
will simply overwrite it, rather than erroring. Individual job entries
are immutable and job entries cannot be deleted, thus this shouldn't
corrupt any data.
## Mongo
In Mongo, the issue is addressed by ensuring that log row deletions
caused due to different log states in concurrent operations are not
merged back to the client job log, and by making sure the final update
includes all job logs.
There is no duplicate key error in Mongo because the array log resides
in the same document and duplicates are simply upserted. We cannot use
transactions in Mongo, as it appears to lock the document in a way that
prevents reliable parallel updates, leading to:
`MongoServerError: WriteConflict error: this operation conflicted with
another operation. Please retry your operation or multi-document
transaction`
When selecting query presets from the list drawer, all query presets are
available for selection, even if unrelated to the underlying collection.
When selecting one of these presets, the list view will crash with
client-side exceptions because the columns and filters that are applied
are incompatible.
The fix is to the thread `filterOptions` through the query presets
drawer. This will ensure that only related collections are shown.
Query Presets allow you to save and share filters, columns, and sort
orders for your collections. This is useful for reusing common or
complex filtering patterns and column configurations across your team.
Query Presets are defined on the fly by the users of your app, rather
than being hard coded into the Payload Config.
Here's a screen recording demonstrating the general workflow as it
relates to the list view. Query Presets are not exclusive to the admin
panel, however, as they could be useful in a number of other contexts
and environments.
https://github.com/user-attachments/assets/1fe1155e-ae78-4f59-9138-af352762a1d5
Each Query Preset is saved as a new record in the database under the
`payload-query-presets` collection. This will effectively make them
CRUDable and allows for an endless number of preset configurations. As
you make changes to filters, columns, limit, etc. you can choose to save
them as a new record and optionally share them with others.
Normal document-level access control will determine who can read,
update, and delete these records. Payload provides a set of sensible
defaults here, such as "only me", "everyone", and "specific users", but
you can also extend your own set of access rules on top of this, such as
"by role", etc. Access control is customizable at the operation-level,
for example you can set this to "everyone" can read, but "only me" can
update.
To enable the Query Presets within a particular collection, set
`enableQueryPresets` on that collection's config.
Here's an example:
```ts
{
// ...
enableQueryPresets: true
}
```
Once enabled, a new set of controls will appear within the list view of
the admin panel. This is where you can select and manage query presets.
General settings for Query Presets are configured under the root
`queryPresets` property. This is where you can customize the labels,
apply custom access control rules, etc.
Here's an example of how you might augment the access control properties
with your own custom rule to achieve RBAC:
```ts
{
// ...
queryPresets: {
constraints: {
read: [
{
label: 'Specific Roles',
value: 'specificRoles',
fields: [roles],
access: ({ req: { user } }) => ({
'access.update.roles': {
in: [user?.roles],
},
}),
},
],
}
}
}
```
Related: #4193 and #3092
---------
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
