Previously, `db.updateVersion` had a mistake with using `transform({
operation: 'write' })` instead of `transform({ operation: 'read' })`
which led to improper DB data sanitization (like ObjectID -> string,
Date -> string) when calling `payload.update` with `autosave: true` when
some other autosave draft already exists. This fixes
https://github.com/payloadcms/payload/issues/11542 additionally for this
case.
This change makes so that data that exists in MongoDB but isn't defined
in the Payload config won't be included to `payload.find` /
`payload.db.find` calls. Now we strip all the additional keys.
Consider you have a field named `secretField` that's also `hidden: true`
(or `read: () => false`) that contains some sensitive data. Then you
removed this field from the database and as for now with the MongoDB
adapter this field will be included to the Local API / REST API results
without any consideration, as Payload doesn't know about it anymore.
This also fixes https://github.com/payloadcms/payload/issues/11542 if
you removed / renamed a relationship field from the schema, Payload
won't sanitize ObjectIDs back to strings anymore.
Ideally you should create a migration script that completely removes the
deleted field from the database with `$unset`, but people rarely do
this.
If you still need to keep those fields to the result, this PR allows you
to do this with the new `allowAdditionalKeys: true` flag.
### What?
This PR adds ability to define indexes on several fields for collections
(compound indexes).
Example:
```ts
{
indexes: [{ unique: true, fields: ['title', 'group.name'] }]
}
```
### Why?
This can be used to either speed up querying/sorting by 2 or more fields
at the same time or to ensure uniqueness between several fields.
### How?
Implements this logic in database adapters. Additionally, adds a utility
`getFieldByPath`.
This PR adds a new `limit` property to `payload.db.updateMany`. This functionality is required for [migrating our job system to use faster, direct db adapter calls](https://github.com/payloadcms/payload/pull/11489)
Migrates the `db-mongodb` package to use `strict: true` and
`noUncheckedIndexedAccess: true` TSConfig properties.
This greatly improves code quality and prevents some runtime errors or
gives better error messages.
This PR adds a new `returning` option to various db adapter methods. Setting it to `false` where the return value is not used will lead to performance gains, as we don't have to do additional db calls to fetch the updated document and then sanitize it.
### What?
For the join field query adds ability to specify `count: true`, example:
```ts
const result = await payload.find({
joins: {
'group.relatedPosts': {
sort: '-title',
count: true,
},
},
collection: "categories",
})
result.group?.relatedPosts?.totalDocs // available
```
### Why?
Can be useful to implement full pagination / show total related
documents count in the UI.
### How?
Implements the logic in database adapters. In MongoDB it's additional
`$lookup` that has `$count` in the pipeline. In SQL, it's additional
subquery with `COUNT(*)`. Preserves the current behavior by default,
since counting introduces overhead.
Additionally, fixes a typescript generation error for join fields.
Before, `docs` and `hasNextPage` were marked as nullable, which is not
true, these fields cannot be `null`.
Additionally, fixes threading of `joinQuery` in
`transform/read/traverseFields` for group / tab fields recursive calls.
Previously, `hasNextPage` was working incorrectly with polymorphic joins
(that have an array of `collection`) in MongoDB.
This PR fixes it and adds extra assertions to the polymorphic joins
test.
---------
Co-authored-by: Jarrod Flesch <jarrodmflesch@gmail.com>
Improves performance and optimizes memory usage for mongodb adapter by
cutting down copying of results via `JSON.parse(JSON.stringify())`.
Instead, `transform` does necessary transformations (`ObjectID` ->
`string,` `Date` -> `string`) without any copying
In the `findOne` db operation, we return `null` if the document was not
found.
For single-document delete and update operations, if the document you
wanted to update is not found, the following runtime error is thrown
instead: `Cannot read properties of null (reading '_id')`.
This PR correctly handles these cases and returns `null` from the db
method, just like the `findOne` operation.
This feature allows you to specify `collection` for the join field as
array.
This can be useful for example to describe relationship linking like
this:
```ts
{
slug: 'folders',
fields: [
{
type: 'join',
on: 'folder',
collection: ['files', 'documents', 'folders'],
name: 'children',
},
{
type: 'relationship',
relationTo: 'folders',
name: 'folder',
},
],
},
{
slug: 'files',
upload: true,
fields: [
{
type: 'relationship',
relationTo: 'folders',
name: 'folder',
},
],
},
{
slug: 'documents',
fields: [
{
type: 'relationship',
relationTo: 'folders',
name: 'folder',
},
],
},
```
Documents and files can be placed to folders and folders themselves can
be nested to other folders (root folders just have `folder` as `null`).
Output type of `Folder`:
```ts
export interface Folder {
id: string;
children?: {
docs?:
| (
| {
relationTo?: 'files';
value: string | File;
}
| {
relationTo?: 'documents';
value: string | Document;
}
| {
relationTo?: 'folders';
value: string | Folder;
}
)[]
| null;
hasNextPage?: boolean | null;
} | null;
folder?: (string | null) | Folder;
updatedAt: string;
createdAt: string;
}
```
While you could instead have many join fields (for example
`childrenFolders`, `childrenFiles`) etc - this doesn't allow you to
sort/filter and paginate things across many collections, which isn't
trivial. With SQL we use `UNION ALL` query to achieve that.
---------
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
Currently, the join field outputs to its result `hasNextPage: boolean`
and have the `limit` query parameter but lacks `page` which can be
useful. This PR adds it.
The `localized` properly was not stripped out of referenced block fields, if any parent was localized. For normal fields, this is done in sanitizeConfig. As the same referenced block config can be used in both a localized and non-localized config, we are not able to strip it out inside sanitizeConfig by modifying the block config.
Instead, this PR had to bring back tedious logic to handle it everywhere the `field.localized` property is accessed. For backwards-compatibility, we need to keep the existing sanitizeConfig logic. In 4.0, we should remove it to benefit from better test coverage of runtime field.localized handling - for now, this is done for our test suite using the `PAYLOAD_DO_NOT_SANITIZE_LOCALIZED_PROPERTY` flag.
### What?
This PR removes a pair unnecessary calls to `schema.index` against the
timestamp fields. The issue is when a user sets `indexSortableFields` as
this is what will ultimately pass the predicate which then creates
duplicate indexes.
### Why?
These calls are redundant as `index` is [already
passed](https://github.com/payloadcms/payload/blob/main/packages/db-mongodb/src/models/buildSchema.ts#L69)
to the underlying fields base schema options in the process of
formatting and will already be indexed.
These warnings were surfaced after the bump to mongoose to version 8.9.5
as [in 8.9.3 mongoose began throwing these warnings to indicate
duplicative
indexes](https://github.com/Automattic/mongoose/releases/tag/8.9.3).
### How?
By removing these calls and, as a result, silencing the warnings thrown
by mongoose.
If you have multiple blocks that are used in multiple places, this can quickly blow up the size of your Payload Config. This will incur a performance hit, as more data is
1. sent to the client (=> bloated `ClientConfig` and large initial html) and
2. processed on the server (permissions are calculated every single time you navigate to a page - this iterates through all blocks you have defined, even if they're duplicative)
This can be optimized by defining your block **once** in your Payload Config, and just referencing the block slug whenever it's used, instead of passing the entire block config. To do this, the block can be defined in the `blocks` array of the Payload Config. The slug can then be passed to the `blockReferences` array in the Blocks Field - the `blocks` array has to be empty for compatibility reasons.
```ts
import { buildConfig } from 'payload'
import { lexicalEditor, BlocksFeature } from '@payloadcms/richtext-lexical'
// Payload Config
const config = buildConfig({
// Define the block once
blocks: [
{
slug: 'TextBlock',
fields: [
{
name: 'text',
type: 'text',
},
],
},
],
collections: [
{
slug: 'collection1',
fields: [
{
name: 'content',
type: 'blocks',
// Reference the block by slug
blockReferences: ['TextBlock'],
blocks: [], // Required to be empty, for compatibility reasons
},
],
},
{
slug: 'collection2',
fields: [
{
name: 'editor',
type: 'richText',
editor: lexicalEditor({
BlocksFeature({
// Same reference can be reused anywhere, even in the lexical editor, without incurred performance hit
blocks: ['TextBlock'],
})
})
},
],
},
],
})
```
## v4.0 Plans
In 4.0, we will remove the `blockReferences` property, and allow string block references to be passed directly to the blocks `property`. Essentially, we'd remove the `blocks` property and rename `blockReferences` to `blocks`.
The reason we opted to a new property in this PR is to avoid breaking changes. Allowing strings to be passed to the `blocks` property will prevent plugins that iterate through fields / blocks from compiling.
## PR Changes
- Testing: This PR introduces a plugin that automatically converts blocks to block references. This is done in the fields__blocks test suite, to run our existing test suite using block references.
- Block References support: Most changes are similar. Everywhere we iterate through blocks, we have to now do the following:
1. Check if `field.blockReferences` is provided. If so, only iterate through that.
2. Check if the block is an object (= actual block), or string
3. If it's a string, pull the actual block from the Payload Config or from `payload.blocks`.
The exception is config sanitization and block type generations. This PR optimizes them so that each block is only handled once, instead of every time the block is referenced.
## Benchmarks
60 Block fields, each block field having the same 600 Blocks.
### Before:
**Initial HTML:** 195 kB
**Generated types:** takes 11 minutes, 461,209 lines
https://github.com/user-attachments/assets/11d49a4e-5414-4579-8050-e6346e552f56
### After:
**Initial HTML:** 73.6 kB
**Generated types:** takes 2 seconds, 35,810 lines
https://github.com/user-attachments/assets/3eab1a99-6c29-489d-add5-698df67780a3
### After Permissions Optimization (follow-up PR)
Initial HTML: 73.6 kB
https://github.com/user-attachments/assets/a909202e-45a8-4bf6-9a38-8c85813f1312
## Future Plans
1. This PR does not yet deduplicate block references during permissions calculation. We'll optimize that in a separate PR, as this one is already large enough
2. The same optimization can be done to deduplicate fields. One common use-case would be link field groups that may be referenced in multiple entities, outside of blocks. We might explore adding a new `fieldReferences` property, that allows you to reference those same `config.blocks`.
### What?
If you had multiple operator constraints on a single field, the last one
defined would be the only one used.
Example:
```ts
where: {
id: {
in: [doc2.id],
not_in: [], // <-- only respected this operator constraint
},
}
```
and
```ts
where: {
id: {
not_in: [],
in: [doc2.id], // <-- only respected this operator constraint
},
}
```
They would yield different results.
### Why?
The results were not merged into an `$and` query inside parseParams.
### How?
Merges the results within an `$and` constraint.
Fixes https://github.com/payloadcms/payload/issues/10944
Supersedes https://github.com/payloadcms/payload/pull/11011
Bumps the following dependencies:
- next
- typescript
- http-status
- nodemailer
- Payload & next versions in all templates
- Monorepo only: playwright and dotenv
Removes unused dependencies:
- ts-jest
- jest-environment-jsdom
- resend (we don't use their sdk, we only use their rest API)
- reduces unnecessary shallow copying within operations by removing
unnecessary spreads or .map()'s
- removes unnecessary `deleteMany` call in `deleteUserPreferences` for
auth-enabled collections
- replaces all instances of `validOperators.includes` with
`validOperatorMap[]`. O(n) => O(1)
- optimizes the `sanitizeInternalFields` function. Previously, it was
doing a **lot** of shallow copying
Previously, every error from MongoDB was logged as "Value must be
unique", as well the response code should not be `BAD_REQUEST` but
`INTERNAL_SERVER_ERROR`. `throw error` preserves the original error so
it can be traced.
This PR modifies `tsconfig.base.json` by setting the following
strictness properties to true: `strict`, `noUncheckedIndexedAccess` and
`noImplicitOverride`.
In packages where compilation errors were observed, these settings were
opted out, and TODO comments were added to make it easier to track the
roadmap for converting everything to strict mode.
The following packages now have increased strictness, which prevents new
errors from being accidentally introduced:
- storage-vercel-blob
- storage-s3*
- storage-gcs
- plugin-sentry
- payload-cloud*
- email-resend*
- email-nodemailer*
*These packages already had `strict: true`, but now have
`noUncheckedIndexedAccess` and `noImplicitOverride`.
Note that this only affects the `/packages` folder, but not
`/templates`, `/test` or `/examples` which have a different `tsconfig`.
