This PR optimizes the new virtual fields with relationships feature
https://github.com/payloadcms/payload/pull/11805 when the path
references the ID field, for example:
```
{
name: 'postCategoryID',
type: 'number',
virtual: 'post.category.id',
},
```
Previously, we did additional population of `category`, which is
unnecessary as we can always grab the ID from the `category` value
itself. One less querying step.
This PR adds an ability to specify a virtual field in this way
```js
{
slug: 'posts',
fields: [
{
name: 'title',
type: 'text',
required: true,
},
],
},
{
slug: 'virtual-relations',
fields: [
{
name: 'postTitle',
type: 'text',
virtual: 'post.title',
},
{
name: 'post',
type: 'relationship',
relationTo: 'posts',
},
],
},
```
Then, every time you query `virtual-relations`, `postTitle` will be
automatically populated (even if using `depth: 0`) on the db level. This
field also, unlike `virtual: true` is available for querying / sorting /
`useAsTitle`.
Also, the field can be deeply nested to 2 or more relationships, for
example:
```
{
name: 'postCategoryTitle',
type: 'text',
virtual: 'post.category.title',
},
```
Where the current collection has `post` - a relationship to `posts`, the
collection `posts` has `category` that's a relationship to `categories`
and finally `categories` has `title`.
Fixes https://github.com/payloadcms/payload/issues/11975
Previously, this configuration was causing errors in postgres due to
long names, even though `dbName` is used:
```
{
slug: 'aliases',
fields: [
{
name: 'thisIsALongFieldNameThatWillCauseAPostgresErrorEvenThoughWeSetAShorterDBName',
dbName: 'shortname',
type: 'array',
fields: [
{
name: 'nested_field_1',
type: 'array',
dbName: 'short_nested_1',
fields: [],
},
{
name: 'nested_field_2',
type: 'text',
},
],
},
],
},
```
This is because we were generating Drizzle relation name (for arrays)
always based on the field path and internally, drizzle uses this name
for aliasing. Now, if `dbName` is present, we use `_{dbName}` instead
for the relation name.
### What?
This PR adds ability to define indexes on several fields for collections
(compound indexes).
Example:
```ts
{
indexes: [{ unique: true, fields: ['title', 'group.name'] }]
}
```
### Why?
This can be used to either speed up querying/sorting by 2 or more fields
at the same time or to ensure uniqueness between several fields.
### How?
Implements this logic in database adapters. Additionally, adds a utility
`getFieldByPath`.
Previously, data for globals was inconsistent across database adapters.
In Postgres, globals didn't store correct `createdAt`, `updatedAt`
fields and the `updateGlobal` lacked the `globalType` field. This PR
solves that without introducing schema changes.
Field paths within hooks are not correct.
For example, an unnamed tab containing a group field and nested text
field should have the path:
- `myGroupField.myTextField`
However, within hooks that path is formatted as:
- `_index-1.myGroupField.myTextField`
The leading index shown above should not exist, as this field is
considered top-level since it is located within an unnamed tab.
This discrepancy is only evident through the APIs themselves, such as
when creating a request with invalid data and reading the validation
errors in the response. Form state contains proper field paths, which is
ultimately why this issue was never caught. This is because within the
admin panel we merge the API response with the current form state,
obscuring the underlying issue. This becomes especially obvious in
#10580, where we no longer initialize validation errors within form
state until the form has been submitted, and instead rely solely on the
API response for the initial error state.
Here's comprehensive example of how field paths _should_ be formatted:
```
{
// ...
fields: [
{
// path: 'topLevelNamedField'
// schemaPath: 'topLevelNamedField'
// indexPath: ''
name: 'topLevelNamedField',
type: 'text',
},
{
// path: 'array'
// schemaPath: 'array'
// indexPath: ''
name: 'array',
type: 'array',
fields: [
{
// path: 'array.[n].fieldWithinArray'
// schemaPath: 'array.fieldWithinArray'
// indexPath: ''
name: 'fieldWithinArray',
type: 'text',
},
{
// path: 'array.[n].nestedArray'
// schemaPath: 'array.nestedArray'
// indexPath: ''
name: 'nestedArray',
type: 'array',
fields: [
{
// path: 'array.[n].nestedArray.[n].fieldWithinNestedArray'
// schemaPath: 'array.nestedArray.fieldWithinNestedArray'
// indexPath: ''
name: 'fieldWithinNestedArray',
type: 'text',
},
],
},
{
// path: 'array.[n]._index-2'
// schemaPath: 'array._index-2'
// indexPath: '2'
type: 'row',
fields: [
{
// path: 'array.[n].fieldWithinRowWithinArray'
// schemaPath: 'array._index-2.fieldWithinRowWithinArray'
// indexPath: ''
name: 'fieldWithinRowWithinArray',
type: 'text',
},
],
},
],
},
{
// path: '_index-2'
// schemaPath: '_index-2'
// indexPath: '2'
type: 'row',
fields: [
{
// path: 'fieldWithinRow'
// schemaPath: '_index-2.fieldWithinRow'
// indexPath: ''
name: 'fieldWithinRow',
type: 'text',
},
],
},
{
// path: '_index-3'
// schemaPath: '_index-3'
// indexPath: '3'
type: 'tabs',
tabs: [
{
// path: '_index-3-0'
// schemaPath: '_index-3-0'
// indexPath: '3-0'
label: 'Unnamed Tab',
fields: [
{
// path: 'fieldWithinUnnamedTab'
// schemaPath: '_index-3-0.fieldWithinUnnamedTab'
// indexPath: ''
name: 'fieldWithinUnnamedTab',
type: 'text',
},
{
// path: '_index-3-0-1'
// schemaPath: '_index-3-0-1'
// indexPath: '3-0-1'
type: 'tabs',
tabs: [
{
// path: '_index-3-0-1-0'
// schemaPath: '_index-3-0-1-0'
// indexPath: '3-0-1-0'
label: 'Nested Unnamed Tab',
fields: [
{
// path: 'fieldWithinNestedUnnamedTab'
// schemaPath: '_index-3-0-1-0.fieldWithinNestedUnnamedTab'
// indexPath: ''
name: 'fieldWithinNestedUnnamedTab',
type: 'text',
},
],
},
],
},
],
},
{
// path: 'namedTab'
// schemaPath: '_index-3.namedTab'
// indexPath: ''
label: 'Named Tab',
name: 'namedTab',
fields: [
{
// path: 'namedTab.fieldWithinNamedTab'
// schemaPath: '_index-3.namedTab.fieldWithinNamedTab'
// indexPath: ''
name: 'fieldWithinNamedTab',
type: 'text',
},
],
},
],
},
]
}
```
IDs that are supplied directly through the API, such as client-side
generated IDs when adding new blocks and array rows, are overwritten on
create. This is because when adding blocks or array rows on the client,
their IDs are generated first before being sent to the server for
processing. Then when the server receives this data, it incorrectly
overrides them to ensure they are unique when using relational DBs. But
this only needs to happen when no ID was supplied on create, or
specifically when duplicating documents via the `beforeDuplicate` hook.
### What?
Adds full support for the point field to Postgres and Vercel Postgres
adapters through the Postgis extension. Fully the same API as with
MongoDB, including support for `near`, `within` and `intersects`
operators.
Additionally, exposes to adapter args:
*
`tablesFilter`https://orm.drizzle.team/docs/drizzle-kit-push#including-tables-schemas-and-extensions.
* `extensions` list of extensions to create, for example `['vector',
'pg_search']`, `postgis` is created automatically if there's any point
field
### Why?
It's essential to support that field type, especially if the postgres
adapter should be out of beta on 3.0 stable.
### How?
* Bumps `drizzle-orm` to `0.36.1` and `drizzle-kit` to `0.28.0` as we
need this change https://github.com/drizzle-team/drizzle-orm/pull/3141
* Uses its functions to achieve querying functionality, for example the
`near` operator works through `ST_DWithin` or `intersects` through
`ST_Intersects`.
* Removes MongoDB condition from all point field tests, but keeps for
SQLite
Resolves these discussions:
https://github.com/payloadcms/payload/discussions/8996https://github.com/payloadcms/payload/discussions/8644
### What?
Moved the logic for copying the data.id to data._id to the mongoose
adapter.
### Why?
If you have any hooks that need to set the `id`, the value does not get
sent to mongodb as you would expect since it was copied before the
beforeValidate hooks.
### How?
Now data._id is assigned only in the mongodb adapter's `create`
function.
BREAKING CHANGES:
When using custom ID fields, if you have any collection hooks for
beforeValidate, beforeChange then `data._id` will no longer be assigned
as this happens now in the database adapter. Use `data.id` instead.
Adds abillity to customize the generated Drizzle schema with
`beforeSchemaInit` and `afterSchemaInit`. Could be useful if you want to
preserve the existing database schema / override the generated one with
features that aren't supported from the Payload config.
## Docs:
### beforeSchemaInit
Runs before the schema is built. You can use this hook to extend your
database structure with tables that won't be managed by Payload.
```ts
import { postgresAdapter } from '@payloadcms/db-postgres'
import { integer, pgTable, serial } from 'drizzle-orm/pg-core'
postgresAdapter({
beforeSchemaInit: [
({ schema, adapter }) => {
return {
...schema,
tables: {
...schema.tables,
addedTable: pgTable('added_table', {
id: serial('id').notNull(),
}),
},
}
},
],
})
```
One use case is preserving your existing database structure when
migrating to Payload. By default, Payload drops the current database
schema, which may not be desirable in this scenario.
To quickly generate the Drizzle schema from your database you can use
[Drizzle
Introspection](https://orm.drizzle.team/kit-docs/commands#introspect--pull)
You should get the `schema.ts` file which may look like this:
```ts
import { pgTable, uniqueIndex, serial, varchar, text } from 'drizzle-orm/pg-core'
export const users = pgTable('users', {
id: serial('id').primaryKey(),
fullName: text('full_name'),
phone: varchar('phone', { length: 256 }),
})
export const countries = pgTable(
'countries',
{
id: serial('id').primaryKey(),
name: varchar('name', { length: 256 }),
},
(countries) => {
return {
nameIndex: uniqueIndex('name_idx').on(countries.name),
}
},
)
```
You can import them into your config and append to the schema with the
`beforeSchemaInit` hook like this:
```ts
import { postgresAdapter } from '@payloadcms/db-postgres'
import { users, countries } from '../drizzle/schema'
postgresAdapter({
beforeSchemaInit: [
({ schema, adapter }) => {
return {
...schema,
tables: {
...schema.tables,
users,
countries
},
}
},
],
})
```
Make sure Payload doesn't overlap table names with its collections. For
example, if you already have a collection with slug "users", you should
either change the slug or `dbName` to change the table name for this
collection.
### afterSchemaInit
Runs after the Drizzle schema is built. You can use this hook to modify
the schema with features that aren't supported by Payload, or if you
want to add a column that you don't want to be in the Payload config.
To extend a table, Payload exposes `extendTable` utillity to the args.
You can refer to the [Drizzle
documentation](https://orm.drizzle.team/docs/sql-schema-declaration).
The following example adds the `extra_integer_column` column and a
composite index on `country` and `city` columns.
```ts
import { postgresAdapter } from '@payloadcms/db-postgres'
import { index, integer } from 'drizzle-orm/pg-core'
import { buildConfig } from 'payload'
export default buildConfig({
collections: [
{
slug: 'places',
fields: [
{
name: 'country',
type: 'text',
},
{
name: 'city',
type: 'text',
},
],
},
],
db: postgresAdapter({
afterSchemaInit: [
({ schema, extendTable, adapter }) => {
extendTable({
table: schema.tables.places,
columns: {
extraIntegerColumn: integer('extra_integer_column'),
},
extraConfig: (table) => ({
country_city_composite_index: index('country_city_composite_index').on(
table.country,
table.city,
),
}),
})
return schema
},
],
}),
})
```
<!--
For external contributors, please include:
- A summary of the pull request and any related issues it fixes.
- Reasoning for the changes made or any additional context that may be
useful.
Ensure you have read and understand the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository.
-->
## Description
Adds `virtual` property to the fields config. Providing `true`
completely disables the field in the DB, which is useful for [Virtual
Fields](https://payloadcms.com/blog/learn-how-virtual-fields-can-help-solve-common-cms-challenges)
Disables abillity to query by a field with `virtual: true`.
Currently, they bloat the DB with unused tables / columns, which may as
well introduce additional joins.
Discussion https://github.com/payloadcms/payload/discussions/6270
Prev PR (this one contains only this feature):
https://github.com/payloadcms/payload/pull/6983
- [x] I have read and understand the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository.
## Type of change
<!-- Please delete options that are not relevant. -->
- [x] New feature (non-breaking change which adds functionality)
- [x] This change requires a documentation update
## Checklist:
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] Existing test suite passes locally with my changes
- [x] I have made corresponding changes to the documentation
# Breaking Changes
### New file import locations
Exports from the `payload` package have been _significantly_ cleaned up.
Now, just about everything is able to be imported from `payload`
directly, rather than an assortment of subpath exports. This means that
things like `import { buildConfig } from 'payload/config'` are now just
imported via `import { buildConfig } from 'payload'`. The mental model
is significantly simpler for developers, but you might need to update
some of your imports.
Payload now exposes only three exports:
1. `payload` - all types and server-only Payload code
2. `payload/shared` - utilities that can be used in either the browser
or in Node environments
3. `payload/node` - heavy utilities that should only be imported in Node
scripts and never be imported into bundled code like Next.js
### UI library pre-bundling
With this release, we've dramatically sped up the compile time for
Payload by pre-bundling our entire UI package for use inside of the
Payload admin itself. There are new exports that should be used within
Payload custom components:
1. `@payloadcms/ui/client` - all client components
2. `@payloadcms/ui/server` - all server components
For all of your custom Payload admin UI components, you should be
importing from one of these two pre-compiled barrel files rather than
importing from the more deeply nested exports directly. That will keep
compile times nice and speedy, and will also make sure that the bundled
JS for your admin UI is kept small.
For example, whereas before, if you imported the Payload `Button`, you
would have imported it like this:
```ts
import { Button } from '@payloadcms/ui/elements/Button'
```
Now, you would import it like this:
```ts
import { Button } from '@payloadcms/ui/client'
```
This is a significant DX / performance optimization that we're pretty
pumped about.
However, if you are importing or re-using Payload UI components
_outside_ of the Payload admin UI, for example in your own frontend
apps, you can import from the individual component exports which will
make sure that the bundled JS is kept to a minimum in your frontend
apps. So in your own frontend, you can continue to import directly to
the components that you want to consume rather than importing from the
pre-compiled barrel files.
Individual component exports will now come with their corresponding CSS
and everything will work perfectly as-expected.
### Specific exports have changed
- `'@payloadcms/ui/templates/Default'` and
`'@payloadcms/ui/templates/Minimal`' are now exported from
`'@payloadcms/next/templates'`
- Old: `import { LogOut } from '@payloadcms/ui/icons/LogOut'` new:
`import { LogOutIcon } from '@payloadcms/ui/icons/LogOut'`
## Background info
In effort to make local dev as fast as possible, we need to import as
few files as possible so that the compiler has less to process. One way
we've achieved this in the Admin Panel was to _remove_ all .scss imports
from all components in the `@payloadcms/ui` module using a build
process. This stripped all `import './index.scss'` statements out of
each component before injecting them into `dist`. Instead, it bundles
all of the CSS into a single `main.css` file, and we import _that_ at
the root of the app.
While this concept is _still_ the right solution to the problem, this
particular approach is not viable when using these components outside
the Admin Panel, where not only does this root stylesheet not exist, but
where it would also bloat your app with unused styles. Instead, we need
to _keep_ these .scss imports in place so they are imported directly
alongside your components, as expected. Then, we need create a _new_
build step that _separately_ compiles the components _without_ their
stylesheets—this way your app can consume either as needed from the new
`client` and `server` barrel files within `@payloadcms/ui`, i.e. from
within `@payloadcms/next` and all other admin-specific packages and
plugins.
This way, all other applications will simply import using the direct
file paths, just as they did before. Except now they come with
stylesheets.
And we've gotten a pretty awesome initial compilation performance boost.
---------
Co-authored-by: James <james@trbl.design>
Co-authored-by: Alessio Gravili <alessio@gravili.de>
Types are now auto-generated by default.
You can opt-out of this behavior by setting:
```ts
buildConfig({
// Rest of config
typescript: {
autoGenerate: false
},
})
```
