If you have multiple blocks that are used in multiple places, this can quickly blow up the size of your Payload Config. This will incur a performance hit, as more data is
1. sent to the client (=> bloated `ClientConfig` and large initial html) and
2. processed on the server (permissions are calculated every single time you navigate to a page - this iterates through all blocks you have defined, even if they're duplicative)
This can be optimized by defining your block **once** in your Payload Config, and just referencing the block slug whenever it's used, instead of passing the entire block config. To do this, the block can be defined in the `blocks` array of the Payload Config. The slug can then be passed to the `blockReferences` array in the Blocks Field - the `blocks` array has to be empty for compatibility reasons.
```ts
import { buildConfig } from 'payload'
import { lexicalEditor, BlocksFeature } from '@payloadcms/richtext-lexical'
// Payload Config
const config = buildConfig({
// Define the block once
blocks: [
{
slug: 'TextBlock',
fields: [
{
name: 'text',
type: 'text',
},
],
},
],
collections: [
{
slug: 'collection1',
fields: [
{
name: 'content',
type: 'blocks',
// Reference the block by slug
blockReferences: ['TextBlock'],
blocks: [], // Required to be empty, for compatibility reasons
},
],
},
{
slug: 'collection2',
fields: [
{
name: 'editor',
type: 'richText',
editor: lexicalEditor({
BlocksFeature({
// Same reference can be reused anywhere, even in the lexical editor, without incurred performance hit
blocks: ['TextBlock'],
})
})
},
],
},
],
})
```
## v4.0 Plans
In 4.0, we will remove the `blockReferences` property, and allow string block references to be passed directly to the blocks `property`. Essentially, we'd remove the `blocks` property and rename `blockReferences` to `blocks`.
The reason we opted to a new property in this PR is to avoid breaking changes. Allowing strings to be passed to the `blocks` property will prevent plugins that iterate through fields / blocks from compiling.
## PR Changes
- Testing: This PR introduces a plugin that automatically converts blocks to block references. This is done in the fields__blocks test suite, to run our existing test suite using block references.
- Block References support: Most changes are similar. Everywhere we iterate through blocks, we have to now do the following:
1. Check if `field.blockReferences` is provided. If so, only iterate through that.
2. Check if the block is an object (= actual block), or string
3. If it's a string, pull the actual block from the Payload Config or from `payload.blocks`.
The exception is config sanitization and block type generations. This PR optimizes them so that each block is only handled once, instead of every time the block is referenced.
## Benchmarks
60 Block fields, each block field having the same 600 Blocks.
### Before:
**Initial HTML:** 195 kB
**Generated types:** takes 11 minutes, 461,209 lines
https://github.com/user-attachments/assets/11d49a4e-5414-4579-8050-e6346e552f56
### After:
**Initial HTML:** 73.6 kB
**Generated types:** takes 2 seconds, 35,810 lines
https://github.com/user-attachments/assets/3eab1a99-6c29-489d-add5-698df67780a3
### After Permissions Optimization (follow-up PR)
Initial HTML: 73.6 kB
https://github.com/user-attachments/assets/a909202e-45a8-4bf6-9a38-8c85813f1312
## Future Plans
1. This PR does not yet deduplicate block references during permissions calculation. We'll optimize that in a separate PR, as this one is already large enough
2. The same optimization can be done to deduplicate fields. One common use-case would be link field groups that may be referenced in multiple entities, outside of blocks. We might explore adding a new `fieldReferences` property, that allows you to reference those same `config.blocks`.
Having the `scripts` dir re-use all packages from the top-level was
getting quite unwieldy. Created new `tools` directory that is part of
the workspace. Packages are exported with the `@tools` package
namespace.
Fixes an issue where if a checkbox field was in the first position of a
collection, and you tried to filter on it via the List view, the page
would crash.
Improves the admin e2e test splitting by grouping them by type with
semantic names as opposed to numerically. This will provide much needed
clarity to exactly _where_ new admin tests should be written and help to
quickly distinguish the areas of failure within the CI overview.
Updates the plugin template and adds it to the monorepo
Includes:
* Integration testing setup
* Adding custom client / server components via a plugin
* The same building setup that we use for our plugins in the monorepo
* `create-payload-app` dynamically configures the project based on the
name:`dev/tsconfig.json`, `src/index.ts`, `dev/payload.config.ts`
For example, from project name: `payload-plugin-cool`
`src/index.ts`:
```ts
export type PayloadPluginCoolConfig = {
/**
* List of collections to add a custom field
*/
collections?: Partial<Record<CollectionSlug, true>>
disabled?: boolean
}
export const payloadPluginCool =
(pluginOptions: PayloadPluginCoolConfig) =>
/// ...
```
`dev/tsconfig.json`:
```json
{
"extends": "../tsconfig.json",
"exclude": [],
"include": [
"**/*.ts",
"**/*.tsx",
"../src/**/*.ts",
"../src/**/*.tsx",
"next.config.mjs",
".next/types/**/*.ts"
],
"compilerOptions": {
"baseUrl": "./",
"paths": {
"@payload-config": [
"./payload.config.ts"
],
"payload-plugin-cool": [
"../src/index.ts"
],
"payload-plugin-cool/client": [
"../src/exports/client.ts"
],
"payload-plugin-cool/rsc": [
"../src/exports/rsc.ts"
]
},
"noEmit": true
}
}
```
`./dev/payload.config.ts`
```
import { payloadPluginCool } from 'payload-plugin-cool'
///
plugins: [
payloadPluginCool({
collections: {
posts: true,
},
}),
],
```
Example of published plugin
https://www.npmjs.com/package/payload-plugin-cool
In PR #9930 we added `overrideAccess: false` to the find operation and
failed to pass the user. This caused
https://github.com/payloadcms/payload/issues/9974 where any access
control causes the edit view to error.
The fix was to pass the user through.
This change also adds Join Field e2e tests to the CI pipeline which was
previously missing and would have caught the error.
As field tests grow in size, they need to be moved out of the greater
fields test spec and into their own standalone files for readability,
maintainability, and speed. This way they we can write field tests in a
more isolated environment, and they can run in parallel in CI.
Adds the missing tests to the `needs:` dependency array for `all-green`
step in CI so that all-green doesn't pass if these tests fail or are in
progress
```
- build-templates
- tests-types
- tests-type-generation
```
As proposed here
https://github.com/payloadcms/payload/pull/9782#issuecomment-2522090135
with additional testing of our types we can be more sure that we don't
break them between updates.
This PR already adds types testing for most Local API methods
6beb921c2e/test/types/types.spec.ts
but new tests for types can be easily added, either to that same file or
you can create `types.spec.ts` in any other test folder.
The new test folder uses `strict: true` to ensure our types do not break
with it.
---------
Co-authored-by: Tom Mrazauskas <tom@mrazauskas.de>
The runner image `ubuntu-latest` image will be switching from Ubuntu
22.04 to Ubuntu 24.04 as specified in
https://github.com/actions/runner-images/issues/10636.
> Rollout will begin on December 5th and will complete on January 17th,
2025.
Breaking changes
Ubuntu 24.04 is ready to be the default version for the "ubuntu-latest"
label in GitHub Actions and Azure DevOps.
This PR moves us to explicitly use `ubuntu-24.04` to ensure
compatibility and to allow explicit upgrades in the future.
We should fix all the flaky tests in the future - in the meantime this
PR will reduce collectively wasted engineer hours, as we now don't have
to manually open the awkward GH actions UI and press the retry button -
often multiple times for each PR.
It may not be enough to simply retry the test:int / test:e2e commands to
get the tests not to flake for the next run, but let's see how this goes
If you had a lot of fields and collections, createClientConfig would be
extremely slow, as it was copying a lot of memory. In my test config
with a lot of fields and collections, it took 4 seconds(!!).
And not only that, it also ran between every single page navigation.
This PR significantly speeds up the createClientConfig function. In my
test config, its execution speed went from 4 seconds to 50 ms.
Additionally, createClientConfig is now properly cached in both dev &
prod. It no longer runs between every single page navigation. Even if
you trigger a full page reload, createClientConfig will be cached and
not run again. Despite that, HMR remains fully-functional.
This will make payload feel noticeably faster for large configs -
especially if it contains a lot of richtext fields, as it was previously
deep-copying the relatively large richText editor configs over and over
again.
## Before - 40 sec navigation speed
https://github.com/user-attachments/assets/fe6b707a-459b-44c6-982a-b277f6cbb73f
## After - 1 sec navigation speed
https://github.com/user-attachments/assets/384fba63-dc32-4396-b3c2-0353fcac6639
## Todo
- [x] Implement ClientSchemaMap and cache it, to remove
createClientField call in our form state endpoint
- [x] Enable schemaMap caching for dev
- [x] Cache lexical clientField generation, or add it to the parent
clientConfig
## Lexical changes
Red: old / removed
Green: new
### Speed up version queries
This PR comes with performance optimizations for fetching versions
before a document is loaded. Not only does it use the new select API to
limit the fields it queries, it also completely skips a database query
if the current document is published.
### Speed up lexical init
Removes a bunch of unnecessary deep copying of lexical objects which
caused higher memory usage and slower load times. Additionally, the
lexical default config sanitization now happens less often.
Changes:
- Migrates `email` example project to `3.0` from `2.0`
- Replaces `inline-css` dependency with `juice` package instead.
- Replaces `Handlebars` dependency with `ejs` package instead.
Reason for replacing packages:
- Both `inline-css` & `Handlebars` had issues with Nextjs and its
Webpack bundling i.e does not support `require.extensions`.
- `ejs` & `juice` do not rely on `require.extensions`.
### What?
Adds full support for the point field to Postgres and Vercel Postgres
adapters through the Postgis extension. Fully the same API as with
MongoDB, including support for `near`, `within` and `intersects`
operators.
Additionally, exposes to adapter args:
*
`tablesFilter`https://orm.drizzle.team/docs/drizzle-kit-push#including-tables-schemas-and-extensions.
* `extensions` list of extensions to create, for example `['vector',
'pg_search']`, `postgis` is created automatically if there's any point
field
### Why?
It's essential to support that field type, especially if the postgres
adapter should be out of beta on 3.0 stable.
### How?
* Bumps `drizzle-orm` to `0.36.1` and `drizzle-kit` to `0.28.0` as we
need this change https://github.com/drizzle-team/drizzle-orm/pull/3141
* Uses its functions to achieve querying functionality, for example the
`near` operator works through `ST_DWithin` or `intersects` through
`ST_Intersects`.
* Removes MongoDB condition from all point field tests, but keeps for
SQLite
Resolves these discussions:
https://github.com/payloadcms/payload/discussions/8996https://github.com/payloadcms/payload/discussions/8644
## Description
Adds a new property to `collection` / `global` configs called
`lockDocuments`.
Set to `true` by default - the lock is automatically triggered when a
user begins editing a document within the Admin Panel and remains in
place until the user exits the editing view or the lock expires due to
inactivity.
Set to `false` to disable document locking entirely - i.e.
`lockDocuments: false`
You can pass an object to this property to configure the `duration` in
seconds, which defines how long the document remains locked without user
interaction. If no edits are made within the specified time (default:
300 seconds), the lock expires, allowing other users to edit / update or
delete the document.
```
lockDocuments: {
duration: 180, // 180 seconds or 3 minutes
}
```
- [x] I have read and understand the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository.
## Type of change
- [x] New feature (non-breaking change which adds functionality)
## Checklist:
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] Existing test suite passes locally with my changes
- [x] I have made corresponding changes to the documentation
This noticeably improves performance in the admin panel, for example
when there are multiple richtext editors on one page (& likely
performance in other areas too, though I mainly tested rich text).
The babel plugin currently only optimizes files with a 'use client'
directive at the top - thus we have to make sure to add use client
wherever possible, even if it's imported by a parent client component.
There's one single component that broke when it was compiled using the
React compiler (it stopped being reactive and failed one of our admin
e2e tests):
150808f608
opting out of it completely fixed that issue
Fixes https://github.com/payloadcms/payload/issues/7366
Bumps the github_actions group with 2 updates:
[pnpm/action-setup](https://github.com/pnpm/action-setup) and
[supercharge/mongodb-github-action](https://github.com/supercharge/mongodb-github-action).
Updates `pnpm/action-setup` from 3 to 4
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/pnpm/action-setup/releases">pnpm/action-setup's
releases</a>.</em></p>
<blockquote>
<h2>v4.0.0</h2>
<p>An error is thrown if one version of pnpm is specified in the
<code>packageManager</code> field of <code>package.json</code> and a
different version is specified in the action's settings <a
href="https://redirect.github.com/pnpm/action-setup/pull/122">#122</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="fe02b34f77"><code>fe02b34</code></a>
docs: bump action-setup version in README</li>
<li><a
href="bee1f099e5"><code>bee1f09</code></a>
feat: throw error when multiple versions specified (<a
href="https://redirect.github.com/pnpm/action-setup/issues/122">#122</a>)</li>
<li><a
href="ce859e384f"><code>ce859e3</code></a>
refactor: replace <code>fs-extra</code> with Node.js built-in fs methods
(<a
href="https://redirect.github.com/pnpm/action-setup/issues/120">#120</a>)</li>
<li><a
href="2ab6dce4f5"><code>2ab6dce</code></a>
docs(README): fix link to LICENSE</li>
<li><a
href="e280758d01"><code>e280758</code></a>
docs(README): update dependency versions (<a
href="https://redirect.github.com/pnpm/action-setup/issues/117">#117</a>)</li>
<li><a
href="129abb77bf"><code>129abb7</code></a>
Bump undici from 5.28.2 to 5.28.3 (<a
href="https://redirect.github.com/pnpm/action-setup/issues/115">#115</a>)</li>
<li>See full diff in <a
href="https://github.com/pnpm/action-setup/compare/v3...v4">compare
view</a></li>
</ul>
</details>
<br />
Updates `supercharge/mongodb-github-action` from 1.10.0 to 1.11.0
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/supercharge/mongodb-github-action/releases">supercharge/mongodb-github-action's
releases</a>.</em></p>
<blockquote>
<p>Release 1.11.0</p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/supercharge/mongodb-github-action/blob/main/CHANGELOG.md">supercharge/mongodb-github-action's
changelog</a>.</em></p>
<blockquote>
<h2><a
href="https://github.com/superchargejs/mongodb-github-action/compare/v1.10.0...v1.11.0">1.11.0</a>
- 2024-05-22</h2>
<h3>Added</h3>
<ul>
<li>added <code>mongodb-container-name</code> input: this option allows
you to define the Docker container name</li>
</ul>
<h3>Fixed</h3>
<ul>
<li>use the <code>mongo</code> command to interact with MongoDB versions
4.x or lower. Previously, we only checked for MongoDB 4 and would use
<code>mongosh</code> for MongoDB 3 (and lower). <a
href="https://redirect.github.com/supercharge/mongodb-github-action/pull/61">Thanks
to Aravind!</a></li>
</ul>
<h3>Updated</h3>
<ul>
<li>bump dependencies</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="5a87bd81f8"><code>5a87bd8</code></a>
prepare changelog for 1.11.0</li>
<li><a
href="7c12fc679c"><code>7c12fc6</code></a>
update readme</li>
<li><a
href="ad73029553"><code>ad73029</code></a>
bump mongoose dependency</li>
<li><a
href="268fb2c93c"><code>268fb2c</code></a>
Merge pull request <a
href="https://redirect.github.com/supercharge/mongodb-github-action/issues/61">#61</a>
from aravindnc/main</li>
<li><a
href="12b898a9c8"><code>12b898a</code></a>
Fix to use mongo client if MongoDB verison is less than or equal to
4.</li>
<li><a
href="b8277548e0"><code>b827754</code></a>
wait 20 seconds</li>
<li><a
href="5f37c5fb42"><code>5f37c5f</code></a>
revert ESLint to 8.x</li>
<li><a
href="fcc7443a6b"><code>fcc7443</code></a>
bump verions</li>
<li><a
href="fde299bc70"><code>fde299b</code></a>
bump deps</li>
<li><a
href="9ceda80ede"><code>9ceda80</code></a>
bump versions of GitHub Actions</li>
<li>Additional commits viewable in <a
href="https://github.com/supercharge/mongodb-github-action/compare/1.10.0...1.11.0">compare
view</a></li>
</ul>
</details>
<br />
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
- regenerates the lockfile
- upgrades pnpm from v8 to v9.7.0 minimum
- ensures playwright does not import payload config. Even after our
importmap revamp that made the payload config server-only / node-safe, I
was getting these `Error: Invariant: AsyncLocalStorage accessed in
runtime where it is not available` errors in combination with pnpm v9
and lockfile regeneration.
This does not happen with pnpm v8, however I'm still blaming playwright
for this, as this does not happen in dev and we've had this specific
error with playwright in the past when we were importing the payload
config. Perhaps it's related to both playwright and the future Next.js
process importing the same config file, and not related to the config
file containing client-side React code.
Making sure playwright doesn't import the config fixed it (it was
importing it through the import map generation). The import map
generation is now run in a separate process, and playwright simply waits
for it
- One positive thing: this pr fixes a bunch of typescript errors with
react-select components. We got those errors because react-select types
are not compatible with react 19. lockfile regeneration fixed that (not
related to pnpm v9) - probably because we were installing mismatching
react versions (I saw both `fb9a90fa48-20240614` and `06d0b89e-20240801`
in our lockfile). I have thus removed the caret for react and react-dom
in our package.json - now it's consistent
