### What?
Remove `select` parameter from database operations in update operation
during version creation to fix malformed version data.
### Why?
The `select` parameter was being passed to `saveVersion()` in update
operations, causing versions to only contain selected fields
### How?
- Removed `select` parameter from `payload.db.updateOne()` calls in
update operations
- Removed `select` parameter from `saveVersion()` call in update
operation
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1211334352350013
https://github.com/payloadcms/payload/pull/13186 actually made the
select API _more powerful_, as it can reduce the amount of db calls even
for complex collections with blocks down to 1.
This PR adds a test that verifies this.
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1210871676349303
Currently, an optimized DB update (simple data => no
delete-and-create-row) does the following:
1. sql UPDATE
2. sql SELECT
This PR reduces this further to one single DB call for simple
collections:
1. sql UPDATE with RETURNING()
This only works for simple collections that do not have any fields that
need to be fetched from other tables. If a collection has fields like
relationship or blocks, we'll need that separate SELECT call to join in
the other tables.
In 4.0, we can remove all "complex" fields from the jobs collection and
replace them with a JSON field to make use of this optimization
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1210803039809814
While we can use `joins`, `select`, `populate`, `depth` or `draft` on
auth collections when finding or finding by ID, these arguments weren't
supported for `/me` which meant that in some situations like in our
ecommerce template we couldn't optimise these calls.
A workaround would be to make a call to `/me` and then get the user ID
to then use for a `findByID` operation.
This clarifies that jobs.autoRun only *runs* already-queued jobs. It does not queue the jobs for you.
Also adds an e2e test as this functionality had no e2e coverage
I think it's easier to review this PR commit by commit, so I'll explain
it this way:
## Commits
1. [parallelize eslint script (still showing logs results in
serial)](c9ac49c12d):
Previously, `--concurrency 1` was added to the script to make the logs
more readable. However, turborepo has an option specifically for these
use cases: `--log-order=grouped` runs the tasks in parallel but outputs
them serially. As a result, the lint script is now significantly faster.
2. [run pnpm
lint:fix](9c128c276a)
The auto-fix was run, which resolved some eslint errors that were
slipped in due to the use of `no-verify`. Most of these were
`perfectionist` fixes (property ordering) and the removal of unnecessary
assertions. Starting with this PR, this won't happen again in the
future, as we'll be verifying the linter in every PR across the entire
codebase (see commit 7).
3. [fix eslint non-autofixable
errors](700f412a33)
All manual errors have been resolved except for the configuration errors
addressed in commit 5. Most were React compiler violations, which have
been disabled and commented out "TODO" for now. There's also an unused
`use no memo` and a couple of `require` errors.
4. [move react-compiler linter to eslint-config
package](4f7cb4d63a)
To simplify the eslint configuration. My concern was that there would be
a performance regression when used in non-react related packages, but
none was experienced. This is probably because it only runs on .tsx
files.
5. [remove redundant eslint config files and fix
allowDefaultProject](a94347995a)
The main feature introduced by `typescript-eslint` v8 was
`projectService`, which automatically searches each file for the closest
`tsconfig`, greatly simplifying configuration in monorepos
([source](https://typescript-eslint.io/blog/announcing-typescript-eslint-v8#project-service)).
Once I moved `projectService` to `packages/eslint-config`, all the other
configuration files could be easily removed.
I confirmed that pnpm lint still works on individual packages.
The other important change was that the pending eslint errors from
commits 2 and 3 were resolved. That is, some files were giving the
error: "[File] was not found by the project service. Consider either
including it in the tsconfig.json or including it in
allowDefaultProject." Below I copy the explanatory comment I left in the
code:
```ts
// This is necessary because `tsconfig.base.json` defines `"rootDir": "${configDir}/src"`,
// And the following files aren't in src because they aren't transpiled.
// This is typescript-eslint's way of adding files that aren't included in tsconfig.
// See: https://typescript-eslint.io/troubleshooting/typed-linting/#i-get-errors-telling-me--was-not-found-by-the-project-service-consider-either-including-it-in-the-tsconfigjson-or-including-it-in-allowdefaultproject
// The best practice is to have a tsconfig.json that covers ALL files and is used for
// typechecking (with noEmit), and a `tsconfig.build.json` that is used for the build
// (or alternatively, swc, tsup or tsdown). That's what we should ideally do, in which case
// this hardcoded list wouldn't be necessary. Note that these files don't currently go
// through ts, only through eslint.
```
6. [Differentiate errors from warnings in VScode ESLint
Rules](5914d2f48d)
There's no reason to do that. If an eslint rule isn't an error, it
should be disabled or converted to a warning.
7. [Disable skip lint, and lint over the entire repo now that it's
faster](e4b28f1360)
The GitHub action linted only the files that had changed in the PR.
While this seems like a good idea, once exceptions were introduced with
[skip lint], they opened the door to propagating more and more errors.
Often, the linter was skipped, not because someone introduced new
errors, but because they were trying to avoid those that had already
crept in, sometimes accidentally introducing new ones.
On the other hand, `pnpm lint` now runs in parallel (commit 1), so it's
not that slow. Additionally, it runs in parallel with other GitHub
actions like e2e tests, which take much longer, so it can't represent a
bottleneck in CI.
8. [fix lint in next
package](4506595f91)
Small fix missing from commit 5
9. [Merge remote-tracking branch 'origin/main' into
fix-eslint](563d4909c1)
10. [add again eslint.config.js in payload
package](78f6ffcae7)
The comment in the code explains it. Basically, after the merge from
main, the payload package runs out of memory when linting, probably
because it grew in recent PRs. That package will sooner or later
collapse for our tooling, so we may have to split it. It's already too
big.
## Future Actions
- Resolve React compiler violations, as mentioned in commit 3.
- Decouple the `tsconfig` used for typechecking and build across the
entire monorepo (as explained in point 5) to ensure ts coverage even for
files that aren't transpiled (such as scripts).
- Remove the few remaining `eslint.config.js`. I had to leave the
`richtext-lexical` and `next` ones for now. They could be moved to the
root config and scoped to their packages, as we do for example with
`templates/vercel-postgres/**`. However, I couldn't get it to work, I
don't know why.
- Make eslint in the test folder usable. Not only are we not linting
`test` in CI, but now the `pnpm eslint .` command is so large that my
computer freezes. If each suite were its own package, this would be
solved, and dynamic codegen + git hooks to modify tsconfig.base.json
wouldn't be necessary
([related](https://github.com/payloadcms/payload/pull/11984)).
### What?
Adds a new property to collection / global config `forceSelect` which
can be used to ensure that some fields are always selected, regardless
of the `select` query.
### Why?
This can be beneficial for hooks and access control, for example imagine
you need the value of `data.slug` in your hook.
With the following query it would be `undefined`:
`?select[title]=true`
Now, to solve this you can specify
```
forceSelect: {
slug: true
}
```
### How?
Every operation now merges the incoming `select` with
`collectionConfig.forceSelect`.
Previously, behavior with custom IDs and `select` query was incorrect.
By default, the `id` field is guaranteed to be selected, even if it
doesn't exist in the `select` query, this wasn't true for custom IDs.
Improves performance and optimizes memory usage for mongodb adapter by
cutting down copying of results via `JSON.parse(JSON.stringify())`.
Instead, `transform` does necessary transformations (`ObjectID` ->
`string,` `Date` -> `string`) without any copying
Previously, if you selected only upload `hasMany: true` field, you would
receive an empty arrays always, because the `_rels` table wasn't joined
in this case. Fixes the condition to count `field.type === 'upload'` .
Fixes https://github.com/payloadcms/payload/issues/9606
With Postgres / SQLite, select fields (non `hasMany: true`) weren't
properly handled in the `traverseFields.ts` function for `select` query.
### What?
Adds full support for the point field to Postgres and Vercel Postgres
adapters through the Postgis extension. Fully the same API as with
MongoDB, including support for `near`, `within` and `intersects`
operators.
Additionally, exposes to adapter args:
*
`tablesFilter`https://orm.drizzle.team/docs/drizzle-kit-push#including-tables-schemas-and-extensions.
* `extensions` list of extensions to create, for example `['vector',
'pg_search']`, `postgis` is created automatically if there's any point
field
### Why?
It's essential to support that field type, especially if the postgres
adapter should be out of beta on 3.0 stable.
### How?
* Bumps `drizzle-orm` to `0.36.1` and `drizzle-kit` to `0.28.0` as we
need this change https://github.com/drizzle-team/drizzle-orm/pull/3141
* Uses its functions to achieve querying functionality, for example the
`near` operator works through `ST_DWithin` or `intersects` through
`ST_Intersects`.
* Removes MongoDB condition from all point field tests, but keeps for
SQLite
Resolves these discussions:
https://github.com/payloadcms/payload/discussions/8996https://github.com/payloadcms/payload/discussions/8644
### What?
Because of my error, we didn't pass `populate` to `findOperation` from
the Local API.
### Why?
`populate` must work for every operation that has `depth`.
### How?
Passes `populate` in `operations/local/find.ts`, ensures it works with
the test, checked that other operations pass it.
### What?
Generates types for `joins` property.
Example from our `joins` test, keys are type-safe:
<img width="708" alt="image"
src="https://github.com/user-attachments/assets/f1fbbb9d-7c39-49a2-8aa2-a4793ae4ad7e">
Output in `payload-types.ts`:
```ts
collectionsJoins: {
categories: {
relatedPosts: 'posts';
hasManyPosts: 'posts';
hasManyPostsLocalized: 'posts';
'group.relatedPosts': 'posts';
'group.camelCasePosts': 'posts';
filtered: 'posts';
singulars: 'singular';
};
};
```
Additionally, we include type information about on which collection the
join is, it will help when we have types generation for `where` and
`sort`.
### Why?
It provides a better DX as you don't need to memoize your keys.
### How?
Modifies `configToJSONSchema` to generate the json schema for
`collectionsJoins`, uses that type within `JoinQuery`
### What?
Adds `populate` property to Local API and REST API operations that can
be used to specify `select` for a specific collection when it's
populated
```ts
const result = await payload.findByID({
populate: {
// type safe if you have generated types
posts: {
text: true,
},
},
collection: 'pages',
depth: 1,
id: aboutPage.id,
})
result.relatedPost // only has text and id properties
```
```ts
fetch('https://localhost:3000/api/pages?populate[posts][text]=true') // highlight-line
.then((res) => res.json())
.then((data) => console.log(data))
```
It also overrides
[`defaultPopulate`](https://github.com/payloadcms/payload/pull/8934)
Ensures `defaultPopulate` doesn't affect GraphQL.
### How?
Implements the property for all operations that have the `depth`
argument.
### What?
Fixes type for the `select` property when having `strictNullChecks:
true` or `strict: true` in tsconfig.
### Why?
`select` should provide autocompletion for users, at this point it
doesn't work with this condtiion
### How?
Makes `collectionsSelect` and `globalsSelect` properties required in
`configToJSONSchema.ts`.
Fixes
https://github.com/payloadcms/payload/pull/8550#issuecomment-2452669237
### What?
Fixes `select` handling for properties inside of unnamed tabs using the
mongodb adapter.
Additionally, refactors `traverseFields` in drizzle to reuse logic from
groups / collapsible or rows if unnamed.
### Why?
`select` must work for any fields.
### How?
Fixes the `'tab'` case in `buildProjectionFromSelect` to handle when the
field is an unnamed tab.
Adds extra tests for named tabs / unnamed.
### What?
Adds `defaultPopulate` property to collection config that allows to
specify which fields to select when the collection is populated from
another document.
```ts
import type { CollectionConfig } from 'payload'
// The TSlug generic can be passed to have type safety for `defaultPopulate`.
// If avoided, the `defaultPopulate` type resolves to `SelectType`.
export const Pages: CollectionConfig<'pages'> = {
slug: 'pages',
// I need only slug, NOT the WHOLE CONTENT!
defaultPopulate: {
slug: true,
},
fields: [
{
name: 'slug',
type: 'text',
required: true,
},
],
}
```
### Why?
This is essential for example in case of links. You don't need the whole
document, which can contain large data but only the `slug`.
### How?
Implements `defaultPopulate` when populating relationships, including
inside of lexical / slate rich text fields.
Adds `select` which is used to specify the field projection for local
and rest API calls. This is available as an optimization to reduce the
payload's of requests and make the database queries more efficient.
Includes:
- [x] generate types for the `select` property
- [x] infer the return type by `select` with 2 modes - include (`field:
true`) and exclude (`field: false`)
- [x] lots of integration tests, including deep fields / localization
etc
- [x] implement the property in db adapters
- [x] implement the property in the local api for most operations
- [x] implement the property in the rest api
- [x] docs
---------
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
