Continuation of https://github.com/payloadcms/payload/pull/13501.
When merging server form state with `acceptValues: true`, like on submit
(not autosave), rows are not deeply merged causing custom row
components, like row labels, to disappear. This is because we never
attach components to the form state response unless it has re-rendered
server-side, so unless we merge these rows with the current state, we
lose them.
Instead of allowing `acceptValues` to override all local changes to
rows, we need to flag any newly added rows with `addedByServer` so they
can bypass the merge strategy. Existing rows would continue to be merged
as expected, and new rows are simply appended to the end.
Discovered here:
https://discord.com/channels/967097582721572934/967097582721572937/1408367321797365840
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1211115023863814
No need to re-fetch doc permissions during autosave. This will save us
from making two additional client-side requests on every autosave
interval, on top of the two existing requests needed to autosave and
refresh form state.
This _does_ mean that the UI will not fully reflect permissions again
until you fully save, or until you navigating back, but that has always
been the behavior anyway (until #13416). Maybe we can find another
solution for this in the future, or otherwise consider this to be
expected behavior.
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1211094073049052
If you have a beforeChange hook that manipulates arrays or blocks by
_adding rows_, the result of that hook will not be reflected in the UI
after save or autosave as you might expect.
For example, this hook that ensures at least one array row is populated:
```ts
{
type: 'array',
hooks: {
beforeChange: [
({ value }) =>
!value?.length
? [
// this is an added/computed row if attempt to save with no rows
]
: value,
],
},
// ...
}
```
When you save without any rows, this hook will have automatically
computed a row for you and saved it to the database. Form state will not
reflect this fact, however, until you refresh or navigate back.
This is for two reasons:
1. When merging server form state, we receive the new fields, but do not
receive the new rows. This is because the `acceptValues` flag only
applies to the `value` property of fields, but should also apply to the
`rows` property on `array` and `blocks` fields too.
2. When creating new form state on the server, the newly added rows are
not being flagged with `addedByServer`, and so never make it into form
state when it is merged in on the client. To do this we need to send the
previous form state to the server and set `renderAllFields` to false in
order receive this property as expected. Fixed by #13524.
Before:
https://github.com/user-attachments/assets/3ab07ef5-3afd-456f-a9a8-737909b75016
After:
https://github.com/user-attachments/assets/27ad1d83-9313-45a9-b44a-db1e64452a99
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1211094073049042
Adds various helpers to make it easier and more standard to manage array
fields within e2e tests. Retrofits existing tests to ensure consistent
interactions across the board, and also organizes existing blocks and
relationship field helpers in the same way.
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1211094073049046
Follow-up to #13416. Supersedes #13434.
When autosave is triggered and the user continues to modify fields,
their changes are overridden by the server's value, i.e. the value at
the time the form state request was made. This makes it almost
impossible to edit fields when using a small autosave interval and/or a
slow network.
This is because autosave is now merged into form state, which by default
uses `acceptValues: true`. This does exactly what it sounds like,
accepts all the values from the server—which may be stale if underlying
changes have been made. We ignore these values for onChange events,
because the user is actively making changes. But during form
submissions, we can accept them because the form is disabled while
processing anyway.
This pattern allows us to render "computed values" from the server, i.e.
a field with an `beforeChange` hook that modifies its value.
Autosave, on the other hand, happens in the background _while the form
is still active_. This means changes may have been made since sending
the request. We still need to accept computed values from the server,
but we need to avoid doing this if the user has active changes since the
time of the request.
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1211027929253429
Fields such as groups and arrays would not always reset errorPaths when
there were no more errors. The server and client state was not being
merged safely and the client state was always persisting when the server
sent back no errorPaths, i.e. itterable fields with fully valid
children. This change ensures errorPaths is defaulted to an empty array
if it is not present on the incoming field.
Likely a regression from
https://github.com/payloadcms/payload/pull/9388.
Adds e2e test.
This clarifies that jobs.autoRun only *runs* already-queued jobs. It does not queue the jobs for you.
Also adds an e2e test as this functionality had no e2e coverage
I think it's easier to review this PR commit by commit, so I'll explain
it this way:
## Commits
1. [parallelize eslint script (still showing logs results in
serial)](c9ac49c12d):
Previously, `--concurrency 1` was added to the script to make the logs
more readable. However, turborepo has an option specifically for these
use cases: `--log-order=grouped` runs the tasks in parallel but outputs
them serially. As a result, the lint script is now significantly faster.
2. [run pnpm
lint:fix](9c128c276a)
The auto-fix was run, which resolved some eslint errors that were
slipped in due to the use of `no-verify`. Most of these were
`perfectionist` fixes (property ordering) and the removal of unnecessary
assertions. Starting with this PR, this won't happen again in the
future, as we'll be verifying the linter in every PR across the entire
codebase (see commit 7).
3. [fix eslint non-autofixable
errors](700f412a33)
All manual errors have been resolved except for the configuration errors
addressed in commit 5. Most were React compiler violations, which have
been disabled and commented out "TODO" for now. There's also an unused
`use no memo` and a couple of `require` errors.
4. [move react-compiler linter to eslint-config
package](4f7cb4d63a)
To simplify the eslint configuration. My concern was that there would be
a performance regression when used in non-react related packages, but
none was experienced. This is probably because it only runs on .tsx
files.
5. [remove redundant eslint config files and fix
allowDefaultProject](a94347995a)
The main feature introduced by `typescript-eslint` v8 was
`projectService`, which automatically searches each file for the closest
`tsconfig`, greatly simplifying configuration in monorepos
([source](https://typescript-eslint.io/blog/announcing-typescript-eslint-v8#project-service)).
Once I moved `projectService` to `packages/eslint-config`, all the other
configuration files could be easily removed.
I confirmed that pnpm lint still works on individual packages.
The other important change was that the pending eslint errors from
commits 2 and 3 were resolved. That is, some files were giving the
error: "[File] was not found by the project service. Consider either
including it in the tsconfig.json or including it in
allowDefaultProject." Below I copy the explanatory comment I left in the
code:
```ts
// This is necessary because `tsconfig.base.json` defines `"rootDir": "${configDir}/src"`,
// And the following files aren't in src because they aren't transpiled.
// This is typescript-eslint's way of adding files that aren't included in tsconfig.
// See: https://typescript-eslint.io/troubleshooting/typed-linting/#i-get-errors-telling-me--was-not-found-by-the-project-service-consider-either-including-it-in-the-tsconfigjson-or-including-it-in-allowdefaultproject
// The best practice is to have a tsconfig.json that covers ALL files and is used for
// typechecking (with noEmit), and a `tsconfig.build.json` that is used for the build
// (or alternatively, swc, tsup or tsdown). That's what we should ideally do, in which case
// this hardcoded list wouldn't be necessary. Note that these files don't currently go
// through ts, only through eslint.
```
6. [Differentiate errors from warnings in VScode ESLint
Rules](5914d2f48d)
There's no reason to do that. If an eslint rule isn't an error, it
should be disabled or converted to a warning.
7. [Disable skip lint, and lint over the entire repo now that it's
faster](e4b28f1360)
The GitHub action linted only the files that had changed in the PR.
While this seems like a good idea, once exceptions were introduced with
[skip lint], they opened the door to propagating more and more errors.
Often, the linter was skipped, not because someone introduced new
errors, but because they were trying to avoid those that had already
crept in, sometimes accidentally introducing new ones.
On the other hand, `pnpm lint` now runs in parallel (commit 1), so it's
not that slow. Additionally, it runs in parallel with other GitHub
actions like e2e tests, which take much longer, so it can't represent a
bottleneck in CI.
8. [fix lint in next
package](4506595f91)
Small fix missing from commit 5
9. [Merge remote-tracking branch 'origin/main' into
fix-eslint](563d4909c1)
10. [add again eslint.config.js in payload
package](78f6ffcae7)
The comment in the code explains it. Basically, after the merge from
main, the payload package runs out of memory when linting, probably
because it grew in recent PRs. That package will sooner or later
collapse for our tooling, so we may have to split it. It's already too
big.
## Future Actions
- Resolve React compiler violations, as mentioned in commit 3.
- Decouple the `tsconfig` used for typechecking and build across the
entire monorepo (as explained in point 5) to ensure ts coverage even for
files that aren't transpiled (such as scripts).
- Remove the few remaining `eslint.config.js`. I had to leave the
`richtext-lexical` and `next` ones for now. They could be moved to the
root config and scoped to their packages, as we do for example with
`templates/vercel-postgres/**`. However, I couldn't get it to work, I
don't know why.
- Make eslint in the test folder usable. Not only are we not linting
`test` in CI, but now the `pnpm eslint .` command is so large that my
computer freezes. If each suite were its own package, this would be
solved, and dynamic codegen + git hooks to modify tsconfig.base.json
wouldn't be necessary
([related](https://github.com/payloadcms/payload/pull/11984)).
When running the Payload admin panel on a machine with a slower CPU,
form state lags significantly and can become nearly unusable or even
crash when interacting with the document's `useAsTitle` field.
Here's an example:
https://github.com/user-attachments/assets/3535fa99-1b31-4cb6-b6a8-5eb9a36b31b7
#### Why this happens
The reason for this is that entire React component trees are
re-rendering on every keystroke of the `useAsTitle` field, twice over.
Here's a breakdown of the flow:
1. First, we dispatch form state events to the form context. Only the
components that are subscribed to form state re-render when this happens
(good).
2. Then, we sync the `useAsTitle` field to the document info provider,
which lives outside the form. Regardless of whether its children need to
be aware of the document title, all components subscribed to the
document info context will re-render (there are many, including the form
itself).
Given how far up the rendering tree the document info provider is, its
rendering footprint, and the rate of speed at which these events are
dispatched, this is resource intensive.
#### What is the fix
The fix is to isolate the document's title into it's own context. This
way only the components that are subscribed to specifically this context
will re-render as the title changes.
Here's the same test with the same CPU throttling, but no lag:
https://github.com/user-attachments/assets/c8ced9b1-b5f0-4789-8d00-a2523d833524
In this case, the `blockType` property is created on the server, but -
prior to this fix - was discarded on the client in
[`fieldReducer.ts`](https://github.com/payloadcms/payload/blob/main/packages/ui/src/forms/Form/fieldReducer.ts#L186-L198)
via
[`mergerServerFormState.ts`](b9832f40e4/packages/ui/src/forms/Form/mergeServerFormState.ts (L29-L31)),
because the field's path neither existed in the client's form state, nor
was it marked as `addedByServer`.
This caused later calls to POST requests to form state to send without
the `blockType` key for block rows, which in turn caused
`addFieldStatePromise.ts` to throw the following error:
```
Block with type "undefined" was found in block data, but no block with that type is defined in the config for field with schema path ${schemaPath}.
```
This prevented the client side form state update from completing, and if
the form state was saved, broke the document.
This is a follow-up to #12131, which treated the symptom, but not the
cause. The original issue seems to have been introduced in
https://github.com/payloadcms/payload/releases/tag/v3.34.0. It's unclear
to me whether this issue is connected to block E2E tests having been
disabled in the same release in
https://github.com/payloadcms/payload/pull/11988.
## How to reproduce
### Collection configuration
```ts
const RICH_TEXT_BLOCK_TYPE = 'richTextBlockType'
const RichTextBlock: Block = {
slug: RICH_TEXT_BLOCK_TYPE,
interfaceName: 'RichTextBlock',
fields: [
{
name: 'richTextBlockField',
label: 'Rich Text Field in Block Field',
type: 'richText',
editor: lexicalEditor({}),
required: true,
},
],
}
const MyCollection: CollectionConfig = {
slug: 'my-collection-slug,
fields: [
{
name: 'arrayField',
label: 'Array Field',
type: 'array',
fields: [
{
name: 'blockField',
type: 'blocks',
blocks: [RichTextBlock],
required: true,
},
],
},
]
}
export default MyCollection
```
### Steps
- Press "Add Array Field"
--> ✅ 1st block with rich text is added
- Press "Add Array Field" a 2nd time
### Result
- 🛑 2nd block is indefinitely in loading state (side-note: the form UI
should preferably explicitly indicate the error).
- 🛑 If saving the document, it is corrupted and will only show a blank
page (also not indicating any error).
Client side:
<img width="1268" alt="Untitled"
src="https://github.com/user-attachments/assets/4b32fdeb-af76-41e2-9181-d2dbd686618a"
/>
API error:
<img width="1272" alt="image"
src="https://github.com/user-attachments/assets/35dc65f7-88ac-4397-b8d4-353bcf6a4bfd"
/>
Client side, when saving and re-opening document (API error of `GET
/admin/collections/${myCollection}/${documentId}` is the same (arguably
the HTTP response status code shouldn't be `200`)):
<img width="1281" alt="image"
src="https://github.com/user-attachments/assets/2e916eb5-6f10-4e82-9b84-1dc41db21d47"
/>
### Result after fix
- `blockType` is sent from the client to the server.
- ✅ 2nd block with rich text is added.
- ✅ Document does not break when saving & re-opening.
<img width="1277" alt="Untitled"
src="https://github.com/user-attachments/assets/84d0c88b-64b2-48c4-864d-610d524ac8fc"
/>
---------
Co-authored-by: Jacob Fletcher <jacobsfletch@gmail.com>
When server rendering custom components within form state, those
components receive a path that is correct at render time, but
potentially stale after manipulating array and blocks rows. This causes
the field to briefly render incorrect values while the form state
request is in flight.
The reason for this is that paths are passed as a prop statically into
those components. Then when we manipulate rows, form state is modified,
potentially changing field paths. The component's `path` prop, however,
hasn't changed. This means it temporarily points to the wrong field in
form state, rendering the data of another row until the server responds
with a freshly rendered component.
This is not an issue with default Payload fields as they are rendered on
the client and can be passed dynamic props.
This is only an issue within custom server components, including rich
text fields which are treated as custom components. Since they are
rendered on the server and passed to the client, props are inaccessible
after render.
The fix for this is to provide paths dynamically through context. This
way as we make changes to form state, there is a mechanism in which
server components can receive the updated path without waiting on its
props to update.
Fixes form state race conditions. Modifying state while a request is in
flight or while the response is being processed could result in those
changes being overridden.
This was happening for a few reasons:
1. Our merge logic was incorrect. We were disregarding local changes to
state that may have occurred while form state requests are pending. This
was because we were iterating over local state, then while building up
new state, we were ignoring any fields that did not exist in the server
response, like this:
```ts
for (const [path, newFieldState] of Object.entries(existingState)) {
if (!incomingState[path]) {
continue
}
// ...
}
```
To fix this, we need to use local state as the source of truth. Then
when the server state arrives, we need to iterate over _that_. If a
field matches in local state, merge in any new properties. This will
ensure all changes to the underlying state are preserved, including any
potential addition or deletions.
However, this logic breaks down if the server might have created _new_
fields, like when populating array rows. This means they, too, would be
ignored. To get around this, there is a new `addedByServer` property
that flags new fields to ensure they are kept.
This new merge strategy also saves an additional loop over form state.
1. We were merging form state based on a mutable ref. This meant that
changes made within one action cause concurrent actions to have dirty
reads. The fix for this is to merge in an isolated manner by copying
state. This will remove any object references. It is generally not good
practice to mutate state without setting it, anyways, as this causes
mismatches between what is rendered and what is in memory.
1. We were merging server form state directly within an effect, then
replacing state entirely. This meant that if another action took place
at the exact moment in time _after_ merge but _before_ dispatch, the
results of that other action would be completely overridden. The fix for
this is to perform the merge within the reducer itself. This will ensure
that we are working with a trustworthy snapshot of state at the exact
moment in time that the action was invoked, and that React can properly
queue the event within its lifecycle.
Significantly optimizes the component rendering strategy within the form
state endpoint by precisely rendering only the fields that require it.
This cuts down on server processing and network response sizes when
invoking form state requests **that manipulate array and block rows
which contain server components**, such as rich text fields, custom row
labels, etc. (results listed below).
Here's a breakdown of the issue:
Previously, when manipulating array and block fields, _all_ rows would
render any server components that might exist within them, including
rich text fields. This means that subsequent changes to these fields
would potentially _re-render_ those same components even if they don't
require it.
For example, if you have an array field with a rich text field within
it, adding the first row would cause the rich text field to render,
which is expected. However, when you add a second row, the rich text
field within the first row would render again unnecessarily along with
the new row.
This is especially noticeable for fields with many rows, where every
single row processes its server components and returns RSC data. And
this does not only affect nested rich text fields, but any custom
component defined on the field level, as these are handled in the same
way.
The reason this was necessary in the first place was to ensure that the
server components receive the proper data when they are rendered, such
as the row index and the row's data. Changing one of these rows could
cause the server component to receive the wrong data if it was not
freshly rendered.
While this is still a requirement that rows receive up-to-date props, it
is no longer necessary to render everything.
Here's a breakdown of the actual fix:
This change ensures that only the fields that are actually being
manipulated will be rendered, rather than all rows. The existing rows
will remain in memory on the client, while the newly rendered components
will return from the server. For example, if you add a new row to an
array field, only the new row will render its server components.
To do this, we send the path of the field that is being manipulated to
the server. The server can then use this path to determine for itself
which fields have already been rendered and which ones need required
rendering.
## Results
The following results were gathered by booting up the `form-state` test
suite and seeding 100 array rows, each containing a rich text field. To
invoke a form state request, we navigate to a document within the
"posts" collection, then add a new array row to the list. The result is
then saved to the file system for comparison.
| Test Suite | Collection | Number of Rows | Before | After | Percentage
Change |
|------|------|---------|--------|--------|--------|
| `form-state` | `posts` | 101 | 1.9MB / 266ms | 80KB / 70ms | ~96%
smaller / ~75% faster |
---------
Co-authored-by: James <james@trbl.design>
Co-authored-by: Alessio Gravili <alessio@gravili.de>
Continuation of #11867. When rendering custom fields nested within
arrays or blocks, such as the Lexical rich text editor which is treated
as a custom field, these fields will sometimes disappear when form state
requests are invoked sequentially. This is especially reproducible on
slow networks.
This is different from the previous PR in that this issue is caused by
adding _rows_ back-to-back, whereas the previous issue was caused when
adding a single row followed by a change to another field.
Here's a screen recording demonstrating the issue:
https://github.com/user-attachments/assets/5ecfa9ec-b747-49ed-8618-df282e64519d
The problem is that `requiresRender` is never sent in the form state
request for row 2. This is because the [task
queue](https://github.com/payloadcms/payload/pull/11579) processes tasks
within a single `useEffect`. This forces React to batch the results of
these tasks into a single rendering cycle. So if request 1 sets state
that request 2 relies on, request 2 will never use that state since
they'll execute within the same lifecycle.
Here's a play-by-play of the current behavior:
1. The "add row" event is dispatched
a. This sets `requiresRender: true` in form state
1. A form state request is sent with `requiresRender: true`
1. While that request is processing, another "add row" event is
dispatched
a. This sets `requiresRender: true` in form state
b. This adds a form state request into the queue
1. The initial form state request finishes
a. This sets `requiresRender: false` in form state
1. The next form state request that was queued up in 3b is sent with
`requiresRender: false`
a. THIS IS EXPECTED, BUT SHOULD ACTUALLY BE `true`!!
To fix this this, we need to ensure that the `requiresRender` property
is persisted into the second request instead of overridden. To do this,
we can add a new `serverPropsToIgnore` to form state which is read when
the processing results from the server. So if `requiresRender` exists in
`serverPropsToIgnore`, we do not merge it. This works because we
actually mutate form state in between requests. So request 2 can read
the results from request 1 without going through an additional rendering
cycle.
Here's a play-by-play of the fix:
1. The "add row" event is dispatched
a. This sets `requiresRender: true` in form state
b. This adds a task in the queue to mutate form state with
`requiresRender: true`
1. A form state request is sent with `requiresRender: true`
1. While that request is processing, another "add row" event is
dispatched
a. This sets `requiresRender: true` in form state AND
`serverPropsToIgnore: [ "requiresRender" ]`
c. This adds a form state request into the queue
1. The initial form state request finishes
a. This returns `requiresRender: false` from the form state endpoint BUT
IS IGNORED
1. The next form state request that was queued up in 3c is sent with
`requiresRender: true`
When rendering custom fields nested within arrays or blocks, such as the
Lexical rich text editor which is treated as a custom field, these
fields will sometimes disappear when form state requests are invoked
sequentially. This is especially reproducible on slow networks.
This is because form state invocations are placed into a [task
queue](https://github.com/payloadcms/payload/pull/11579) which aborts
the currently running tasks when a new one arrives. By doing this, local
form state is never dispatched, and the second task in the queue becomes
stale.
The fix is to _not_ abort the currently running task. This will trigger
a complete rendering cycle, and when the second task is invoked, local
state will be up to date.
Fixes#11340, #11425, and #11824.
The form component's `initializing` and `processing` states do not
disable fields that are rendered outside of `DocumentFields`. Fields
currently rely on the `readOnly` prop provided by `DocumentFields` and
do not subscribe to these states for themselves. This means that fields
that are rendered outright, such as within the bulk edit drawer, they do
not receive a `readOnly` prop and are therefore never disabled.
The fix is add a `disabled` property to the `useField` hook. This
subscribes to the `initializing` and `processing` states in the same way
as `DocumentFields`, however, now each field can determine its own
disabled state instead of relying solely on the `readOnly` prop. Adding
this new prop has no overhead as `processing` and `initializing` is
already being subscribed to within `useField`.
Implements a select-like API into the form state endpoint. This follows
the same spec as the Select API on existing Payload operations, but
works on form state rather than at the db level. This means you can send
the `select` argument through the form state handler, and it will only
process and return the fields you've explicitly identified.
This is especially useful when you only need to generate a partial form
state, for example within the bulk edit form where you select only a
subset of fields to edit. There is no need to iterate all fields of the
schema, generate default values for each, and return them all through
the network. This will also simplify and reduce the amount of
client-side processing required, where we longer need to strip
unselected fields before submission.
The task queue triggers an infinite render of form state. This is
because we return an object from the `useQueues` hook that is recreated
on every render. We then use the `queueTask` function as an unstable
dependency of the `useEffect` responsible for requesting new form state,
ultimately triggering an infinite rendering loop.
The fix is to stabilize the `queueTask` function within a `useCallback`.
Adds a test to prevent future regression.
Implements a form state task queue. This will prevent onChange handlers
within the form component from processing unnecessarily often, sometimes
long after the user has stopped making changes. This leads to a
potentially huge number of network requests if those changes were made
slower than the debounce rate. This is especially noticeable on slow
networks.
Does so through a new `useQueue` hook. This hook maintains a stack of
events that need processing but only processes the final event to
arrive. Every time a new event is pushed to the stack, the currently
running process is aborted (if any), and that event becomes the next in
the queue. This results in a shocking reduction in the time it takes
between final change to form state and the final network response, from
~1.5 minutes to ~3 seconds (depending on the scenario, see below).
This likely fixes a number of existing open issues. I will link those
issues here once they are identified and verifiably fixed.
Before:
I'm typing slowly here to ensure my changes aren't debounce by the form.
There are a total of 60 characters typed, triggering 58 network requests
and taking around 1.5 minutes to complete after the final change was
made.
https://github.com/user-attachments/assets/49ba0790-a8f8-4390-8421-87453ff8b650
After:
Here there are a total of 69 characters typed, triggering 11 network
requests and taking only about 3 seconds to complete after the final
change was made.
https://github.com/user-attachments/assets/447f8303-0957-41bd-bb2d-9e1151ed9ec3
