- reduces unnecessary shallow copying within operations by removing
unnecessary spreads or .map()'s
- removes unnecessary `deleteMany` call in `deleteUserPreferences` for
auth-enabled collections
- replaces all instances of `validOperators.includes` with
`validOperatorMap[]`. O(n) => O(1)
- optimizes the `sanitizeInternalFields` function. Previously, it was
doing a **lot** of shallow copying
Based on https://github.com/payloadcms/payload/pull/10154
If the actual database schema is not changed (no new columns, enums,
indexes, tables) - skip calling Drizzle push. This, potentially can
significantly reduce overhead on reloads in development mode especially
when using remote databases.
If for whatever reason you need to preserve the current behavior you can
use `PAYLOAD_FORCE_DRIZZLE_PUSH=true` env flag.
### What?
With Postgres, before join to self like:
```ts
import type { CollectionConfig } from 'payload'
export const SelfJoins: CollectionConfig = {
slug: 'self-joins',
fields: [
{
name: 'rel',
type: 'relationship',
relationTo: 'self-joins',
},
{
name: 'joins',
type: 'join',
on: 'rel',
collection: 'self-joins',
},
],
}
```
wasn't possible, even though it's a valid usage.
### How?
Now, to differentiate parent `self_joins` and children `self_joins` we
do additional alias for the nested select -
`"4d3cf2b6_1adf_46a8_b6d2_3e1c3809d737"`:
```sql
select
"id",
"rel_id",
"updated_at",
"created_at",
(
select
coalesce(
json_agg(
json_build_object('id', "joins_alias".id)
),
'[]' :: json
)
from
(
select
"created_at",
"rel_id",
"id"
from
"self_joins" "4d3cf2b6_1adf_46a8_b6d2_3e1c3809d737"
where
"4d3cf2b6_1adf_46a8_b6d2_3e1c3809d737"."rel_id" = "self_joins"."id"
order by
"4d3cf2b6_1adf_46a8_b6d2_3e1c3809d737"."created_at" desc
limit
$1
) "joins_alias"
) as "joins_alias"
from
"self_joins"
where
"self_joins"."id" = $2
order by
"self_joins"."created_at" desc
limit
$3
```
Fixes https://github.com/payloadcms/payload/issues/10144
-->
### What?
Exposes ability to enable
[AUTOINCREMENT](https://www.sqlite.org/autoinc.html) for Primary Keys
which ensures that the same ID cannot be reused from previously deleted
rows.
```ts
sqliteAdapter({
autoIncrement: true
})
```
### Why?
This may be essential for some systems. Enabled `autoIncrement: true`
also for the SQLite Adapter in our tests, which can be useful when
testing whether the doc was deleted or not when you also have other
create operations.
### How?
Uses Drizzle's `autoIncrement` option.
WARNING:
This cannot be enabled in an existing project without a custom
migration, as it completely changes how primary keys are stored in the
database.
* Avoids additional file system writes (1 for `await writeFile` and then
`npx prettier --write`) instead prettier now formats the javascript
string directly. Went from 650 MS to 250 MS for the prettify block.
* Disables database connection, since the `db.generateSchema` doesn't
need connection, this also disables Drizzle schema push.
* Properly exits the bin script process.
### What?
The join field had a limitation imposed that prevents it from targeting
polymorphic relationship fields. With this change we can support any
relationship fields.
### Why?
Improves the functionality of join field.
### How?
Extended the database adapters and removed the config sanitization that
would throw an error when polymorphic relationships were used.
Fixes #
This PR allows to have full type safety on `payload.drizzle` with a
single command
```sh
pnpm payload generate:db-schema
```
Which generates TypeScript code with Drizzle declarations based on the
current database schema.
Example of generated file with the website template:
https://gist.github.com/r1tsuu/b8687f211b51d9a3a7e78ba41e8fbf03
Video that shows the power:
https://github.com/user-attachments/assets/3ced958b-ec1d-49f5-9f51-d859d5fae236
We also now proxy drizzle package the same way we do for Lexical so you
don't have to install it (and you shouldn't because you may have version
mismatch).
Instead, you can import from Drizzle like this:
```ts
import {
pgTable,
index,
foreignKey,
integer,
text,
varchar,
jsonb,
boolean,
numeric,
serial,
timestamp,
uniqueIndex,
pgEnum,
} from '@payloadcms/db-postgres/drizzle/pg-core'
import { sql } from '@payloadcms/db-postgres/drizzle'
import { relations } from '@payloadcms/db-postgres/drizzle/relations'
```
Fixes https://github.com/payloadcms/payload/discussions/4318
In the future we can also support types generation for mongoose / raw
mongodb results.
### What?
Previously, the `req` argument:
In database operations (e.g `payload.db`) was required and you needed to
pass the whole `req` with all the properties. This is confusing because
in database operations we never use its properties outside of
`req.transactionID` and `req.t`, both of which should be optional as
well.
Now, you don't have to do that cast:
```ts
payload.db.findOne({
collection: 'posts',
req: {} as PayloadRequest,
where: {
id: {
equals: 1,
},
},
})
```
Becomes:
```ts
payload.db.findOne({
collection: 'posts',
where: {
id: {
equals: 1,
},
},
})
```
If you need to use transactions, you're not required to do the `as` cast
as well now, as the `req` not only optional but also partial -
`Partial<PayloadRequest>`.
`initTransaction`, `commitTransaction`, `killTransaction` utilities are
typed better now as well. They do not require to you pass all the
properties of `req`, but only `payload` -
`MarkRequired<Partial<PayloadRequest>, 'payload'>`
```ts
const req = { payload }
await initTransaction(req)
await payload.db.create({
collection: "posts",
data: {},
req
})
await commitTransaction(req)
```
The same for the Local API. Internal operations (for example
`packages/payload/src/collections/operations/find.ts`) still accept the
whole `req`, but local ones
(`packages/payload/src/collections/operations/local/find.ts`) which are
used through `payload.` now accept `Partial<PayloadRequest>`, as they
pass it through to internal operations with `createLocalReq`.
So now, this is also valid, while previously you had to do `as` cast for
`req`.
```ts
const req = { payload }
await initTransaction(req)
await payload.create({
collection: "posts",
data: {},
req
})
await commitTransaction(req)
```
Marked as deprecated `PayloadRequest['transactionIDPromise']` to remove
in the next major version. It was never used anywhere.
Refactored `withSession` that returns an object to `getSession` that
returns just `ClientSession`. Better type safety for arguments
Deduplicated in all drizzle operations to `getTransaction(this, req)`
utility:
```ts
const db = this.sessions[await req?.transactionID]?.db || this.drizzle
```
Added fallback for throwing unique validation errors in database
operations when `req.t` is not available.
In migration `up` and `down` functions our `req` is not partial, while
we used to passed `req` with only 2 properties - `payload` and
`transactionID`. This is misleading and you can't access for example
`req.t`.
Now, to achieve "real" full `req` - we generate it with `createLocalReq`
in all migration functions.
This all is backwards compatible. In all public API places where you
expect the full `req` (like hooks) you still have it.
### Why?
Better DX, more expected types, less errors because of types casting.
### What?
Querying by nested to rows fields in has many relationships like this:
```ts
const result = await payload.find({
collection: 'relationship-fields',
where: {
'relationToRowMany.title': { equals: 'some-title' },
},
})
```
Where the related collection:
```ts
const RowFields: CollectionConfig = {
slug: rowFieldsSlug,
fields: [
{
type: 'row',
fields: [
{
name: 'title',
label: 'Title within a row',
type: 'text',
required: true,
},
],
},
],
}
```
was broken
### Why?
We migrated to use `flattenedFields`, but not in this specific case.
This error would be caught earlier we used `noImplictAny` typescript
rule. https://www.typescriptlang.org/tsconfig/#noImplicitAny which
wouldn't allow us to create variable like this:
```ts
let relationshipFields // relationshipFields is any here
```
Instead, we should write:
```ts
let relationshipFields: FlattenedField[]
```
We should migrate to it and `strictNullChecks` as well.
Fixes https://github.com/payloadcms/payload/issues/9534
Previously, if you selected only upload `hasMany: true` field, you would
receive an empty arrays always, because the `_rels` table wasn't joined
in this case. Fixes the condition to count `field.type === 'upload'` .
### What?
Allow the join field to have a configuration `on` relationships inside
of an array, ie `on: 'myArray.myRelationship'`.
### Why?
This is a more powerful and expressive way to use the join field and not
be limited by usage of array data. For example, if you have a roles
array for multinant sites, you could add a join field on the sites to
show who the admins are.
### How?
This fixes the traverseFields function to allow the configuration to
pass sanitization. In addition, the function for querying the drizzle
tables needed to be ehanced.
Additional changes from https://github.com/payloadcms/payload/pull/9995:
- Significantly improves traverseFields and the 'join' case with a raw
query injection pattern, right now it's internal but we could expose it
at some point, for example for querying vectors.
- Fixes potential issues with not passed locale to traverseFields (it
was undefined always)
- Adds an empty array fallback for joins with localized relationships
Fixes #
https://github.com/payloadcms/payload/discussions/9643
---------
Co-authored-by: Because789 <thomas@because789.ch>
Co-authored-by: Sasha <64744993+r1tsuu@users.noreply.github.com>
### What?
`payload.db.updateOne` (and so `payload.db.upsert`) with drizzle
adapters used incoming `where` incorrectly and worked properly only
either if you passed `id` or some where query path required table joins
(like `where: { 'array.title'`) which is also the reason why `upsert`
_worked_ with user preferences specifically, because we need to join the
`preferences_rels` table to query by `user.relationTo` and `user.value`
Fixes https://github.com/payloadcms/payload/issues/9915
This was found here - https://github.com/payloadcms/payload/pull/9913,
the database KV adapter uses `upsert` with `where` by unique fields.
This PR adds a feature which fixes another issue with migrations in
Postgres and does few refactors that significantly reduce code
duplication.
Previously, if you needed to use the underlying database directly in
migrations with the active transaction (for example to execute raw SQL),
created from `payload create:migration`, as `req` doesn't work there you
had to do something like this:
```ts
// Postgres
export async function up({ payload, req }: MigrateUpArgs): Promise<void> {
const db = payload.db.sessions?.[await req.transactionID!].db ?? payload.db
const { rows: posts } = await db.execute(sql`SELECT * from posts`)
}
// MongoDB
export async function up({ payload, req }: MigrateUpArgs): Promise<void> {
const session = payload.db.sessions?.[await req.transactionID!]
const posts = await payload.db.collections.posts.collection.find({ session }).toArray()
}
```
Which was:
1. Awkward to write
2. Not documented anywhere
Now, we expose `session` and `db` to `up` and `down` functions for you:
#### MongoDB:
```ts
import { type MigrateUpArgs } from '@payloadcms/db-mongodb'
export async function up({ session, payload, req }: MigrateUpArgs): Promise<void> {
const posts = await payload.db.collections.posts.collection.find({ session }).toArray()
}
```
#### Postgres:
```ts
import { type MigrateUpArgs, sql } from '@payloadcms/db-postgres'
export async function up({ db, payload, req }: MigrateUpArgs): Promise<void> {
const { rows: posts } = await db.execute(sql`SELECT * from posts`)
}
```
#### SQLite:
```ts
import { type MigrateUpArgs, sql } from '@payloadcms/db-sqlite'
export async function up({ db, payload, req }: MigrateUpArgs): Promise<void> {
const { rows: posts } = await db.run(sql`SELECT * from posts`)
}
```
This actually was a thing with Postgres migrations, we already were
passing `db`, but:
1. Only for `up` and when running `payload migrate`, not for example
with `payload migrate:fresh`
2. Not documented neither in TypeScript or docs.
By ensuring we use `db`, this also fixes an issue that affects all
Postgres/SQLite migrations:
Currently, if we run `payload migration:create` with the postgres
adapter we get a file like this:
```ts
import { MigrateUpArgs, MigrateDownArgs, sql } from '@payloadcms/db-postgres'
export async function up({ payload, req }: MigrateUpArgs): Promise<void> {
await payload.db.drizzle.execute(sql`
CREATE TABLE IF NOT EXISTS "users" (
"id" serial PRIMARY KEY NOT NULL,
);
```
Looks good?
Not exactly!
`payload.db.drizzle.execute()` doesn't really use the current
transaction which can lead to some problems.
Instead, it should use the `db` from `payload.db.sessions?.[await
req.transactionID!].db` because that's where we store our Drizzle
instance with the transaction.
But now, if we generate `payload migrate:create` we get:
```ts
import { MigrateUpArgs, MigrateDownArgs, sql } from '@payloadcms/db-postgres'
export async function up({ db, payload, req }: MigrateUpArgs): Promise<void> {
await db.execute(sql`
CREATE TABLE IF NOT EXISTS "users" (
"id" serial PRIMARY KEY NOT NULL,
);
```
Which is what we want, as the `db` is passed correctly here:
76428373e4/packages/drizzle/src/migrate.ts (L88-L90)
```ts
export async function up({ db, payload, req }: MigrateUpArgs): Promise<void> {
const dbWithTransaction = payload.db.sessions?.[await req.transactionID!].db
payload.logger.info({ one: db === dbWithTransaction })
payload.logger.info({ two: db === payload.db.drizzle })
```
<img width="336" alt="image"
src="https://github.com/user-attachments/assets/f9fab5a9-44c2-44a9-95dd-8e5cf267f027">
Additionally, this PR refactors:
* `createMigration` with Drizzle - now we have sharable
`buildCreateMigration` in `@payloadcms/drizzle` to reduce copy-pasting
of the same logic.
* the `v2-v3` relationships migration for Postgres is now shared between
`db-postgres` and `db-vercel-postgres`, again to reduce copy-paste.
Fixes https://github.com/payloadcms/payload/issues/9606
With Postgres / SQLite, select fields (non `hasMany: true`) weren't
properly handled in the `traverseFields.ts` function for `select` query.
What?
Fixes issue when on parallel writes in result you can have 0 latest:
true versions.
Why?
There must be always a version with latest: true
How?
Ensures that we always have a version with latest: true by adding a
filter on createdAt < createdVersion.createdAt.
Instead, this ponentially can lead to a situation where we have 2
versions with latest: true, if they were created at the exact same time,
but this shouldn't happen in a real world scenario and it's much less
problematic than not having a version with latest: true.
Fixes https://github.com/payloadcms/payload/issues/5895
Changes from #8986
---------
Co-authored-by: Sasha <64744993+r1tsuu@users.noreply.github.com>
### What?
Previously, using Postgres, select fields with `hasMany: true` weren't
clearable.
Meaning, you couldn't pass an empty array:
```ts
const updatedDoc = await payload.update({
id,
collection: 'select-fields',
data: {
selectHasMany: [],
},
})
```
### Why?
To achieve the same behavior with MongoDB.
### How?
Modifies logic in `packages/drizzle/src/upsertRow/index.ts` to include
empty arrays.
1. Adds flag `--skip-empty` to `migrate:create` to bypass the empty
migration file prompt.
- Blank migration file will not be created if this flag is passed.
3. Adds flag `--force-accept-warning` to `migrate:fresh` to bypass the
drop database prompt
Fixes errors when having joins with versions +drafts on `hasMany: true`
relationships.
Removes `joinQuery` overhead if we don't need it for the current
operation. Right now, in all adapters we support joins only for `find`,
`findOne`, and `queryDrafts`.
Fixes https://github.com/payloadcms/payload/issues/9369
### What?
This command from here:
https://github.com/payloadcms/payload/pull/6339
```sh
payload migrate:create --file @payloadcms/db-postgres/relationships-v2-v3
```
stopped working after db-postgers and drizzle packages were separated
### How?
Passes correct `dirname` to `getPredefinedMigration`
Additionally, adds support for `.js` files in `getPredefinedMigration`
This fixes a peer dependency error in our monorepo, as
eslint-plugin-jsx-a11y finally supports eslint v9.
Additionally, this officially adds TypeScript 5.6 support for
typescript-eslint.
Currently, Payload renders all custom components on initial compile of
the admin panel. This is problematic for two key reasons:
1. Custom components do not receive contextual data, i.e. fields do not
receive their field data, edit views do not receive their document data,
etc.
2. Components are unnecessarily rendered before they are used
This was initially required to support React Server Components within
the Payload Admin Panel for two key reasons:
1. Fields can be dynamically rendered within arrays, blocks, etc.
2. Documents can be recursively rendered within a "drawer" UI, i.e.
relationship fields
3. Payload supports server/client component composition
In order to achieve this, components need to be rendered on the server
and passed as "slots" to the client. Currently, the pattern for this is
to render custom server components in the "client config". Then when a
view or field is needed to be rendered, we first check the client config
for a "pre-rendered" component, otherwise render our client-side
fallback component.
But for the reasons listed above, this pattern doesn't exactly make
custom server components very useful within the Payload Admin Panel,
which is where this PR comes in. Now, instead of pre-rendering all
components on initial compile, we're able to render custom components
_on demand_, only as they are needed.
To achieve this, we've established [this
pattern](https://github.com/payloadcms/payload/pull/8481) of React
Server Functions in the Payload Admin Panel. With Server Functions, we
can iterate the Payload Config and return JSX through React's
`text/x-component` content-type. This means we're able to pass
contextual props to custom components, such as data for fields and
views.
## Breaking Changes
1. Add the following to your root layout file, typically located at
`(app)/(payload)/layout.tsx`:
```diff
/* THIS FILE WAS GENERATED AUTOMATICALLY BY PAYLOAD. */
/* DO NOT MODIFY IT BECAUSE IT COULD BE REWRITTEN AT ANY TIME. */
+ import type { ServerFunctionClient } from 'payload'
import config from '@payload-config'
import { RootLayout } from '@payloadcms/next/layouts'
import { handleServerFunctions } from '@payloadcms/next/utilities'
import React from 'react'
import { importMap } from './admin/importMap.js'
import './custom.scss'
type Args = {
children: React.ReactNode
}
+ const serverFunctions: ServerFunctionClient = async function (args) {
+ 'use server'
+ return handleServerFunctions({
+ ...args,
+ config,
+ importMap,
+ })
+ }
const Layout = ({ children }: Args) => (
<RootLayout
config={config}
importMap={importMap}
+ serverFunctions={serverFunctions}
>
{children}
</RootLayout>
)
export default Layout
```
2. If you were previously posting to the `/api/form-state` endpoint, it
no longer exists. Instead, you'll need to invoke the `form-state` Server
Function, which can be done through the _new_ `getFormState` utility:
```diff
- import { getFormState } from '@payloadcms/ui'
- const { state } = await getFormState({
- apiRoute: '',
- body: {
- // ...
- },
- serverURL: ''
- })
+ const { getFormState } = useServerFunctions()
+
+ const { state } = await getFormState({
+ // ...
+ })
```
## Breaking Changes
```diff
- useFieldProps()
- useCellProps()
```
More details coming soon.
---------
Co-authored-by: Alessio Gravili <alessio@gravili.de>
Co-authored-by: Jarrod Flesch <jarrodmflesch@gmail.com>
Co-authored-by: James <james@trbl.design>
### What?
Adds full support for the point field to Postgres and Vercel Postgres
adapters through the Postgis extension. Fully the same API as with
MongoDB, including support for `near`, `within` and `intersects`
operators.
Additionally, exposes to adapter args:
*
`tablesFilter`https://orm.drizzle.team/docs/drizzle-kit-push#including-tables-schemas-and-extensions.
* `extensions` list of extensions to create, for example `['vector',
'pg_search']`, `postgis` is created automatically if there's any point
field
### Why?
It's essential to support that field type, especially if the postgres
adapter should be out of beta on 3.0 stable.
### How?
* Bumps `drizzle-orm` to `0.36.1` and `drizzle-kit` to `0.28.0` as we
need this change https://github.com/drizzle-team/drizzle-orm/pull/3141
* Uses its functions to achieve querying functionality, for example the
`near` operator works through `ST_DWithin` or `intersects` through
`ST_Intersects`.
* Removes MongoDB condition from all point field tests, but keeps for
SQLite
Resolves these discussions:
https://github.com/payloadcms/payload/discussions/8996https://github.com/payloadcms/payload/discussions/8644
### What?
Handles database name with special characters. For example: `-` -
`my-awesome-app`.
### Why?
Previously, `my-awesome-app` led to this error:
```
Error: failed to create database my-awesome-app.
Details: syntax error at or near "-"
```
This can reproduced for example with `create-payload-app`, as the
generated db name is based on project's name.
### How?
Wraps the query variable to quotes, `create database "my-awesome-app"`
instead of `create database my-awesome-app`.
### What?
Makes it possible to filter join documents using a `where` added
directly in the config.
### Why?
It makes the join field more powerful for adding contextual meaning to
the documents being returned. For example, maybe you have a
`requiresAction` field that you set and you can have a join that
automatically filters the documents to those that need attention.
### How?
In the database adapter, we merge the requested `where` to the `where`
defined on the field.
On the frontend the results are filtered using the `filterOptions`
property in the component.
Fixes
https://github.com/payloadcms/payload/discussions/8936https://github.com/payloadcms/payload/discussions/8937
---------
Co-authored-by: Sasha <64744993+r1tsuu@users.noreply.github.com>
### What?
Fixes `select` handling for properties inside of unnamed tabs using the
mongodb adapter.
Additionally, refactors `traverseFields` in drizzle to reuse logic from
groups / collapsible or rows if unnamed.
### Why?
`select` must work for any fields.
### How?
Fixes the `'tab'` case in `buildProjectionFromSelect` to handle when the
field is an unnamed tab.
Adds extra tests for named tabs / unnamed.
Adds `select` which is used to specify the field projection for local
and rest API calls. This is available as an optimization to reduce the
payload's of requests and make the database queries more efficient.
Includes:
- [x] generate types for the `select` property
- [x] infer the return type by `select` with 2 modes - include (`field:
true`) and exclude (`field: false`)
- [x] lots of integration tests, including deep fields / localization
etc
- [x] implement the property in db adapters
- [x] implement the property in the local api for most operations
- [x] implement the property in the rest api
- [x] docs
---------
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
### What?
Allow specifying the defaultSort and defaultLimit to use for populating
a join field
### Why?
It is much easier to set defaults rather than be forced to always call
the join query using the query pattern ("?joins[categories][limit]=0").
### How?
See docs and type changes
This change adds support for sort with multiple fields in local API and
REST API. Related discussion #2089
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
### What?
Fixes https://github.com/payloadcms/payload/issues/5152 issue related to
sorting by a localized field with SQLite / Postgres database adapters.
### Why?
It was an incorrect behaviour.
### How?
Modifies the `getTableColumnFromPath` file to have correct join
conditions. Previously if you had this structure in the _locales table
_locale title parent
en A 1
es B 1
we sorted by everything that's here, but we need to sort only by the
passed locale.
Additionally fixes a typescript error in `dev.ts` that I added here
https://github.com/payloadcms/payload/pull/8834
Also, removes the condition with `joins.length` in `countDistinct`. It
was there as for this issue
https://github.com/payloadcms/payload/issues/4889 because sorting by a
localized property caused duplication. This can simnifically improve
performance for `.find` with nested querying/sorting on large data sets,
because `count(*)` is faster than `count(DISTINCT id)`
Fixes https://github.com/payloadcms/payload/issues/8752
Previously, trying to define a config like this:
```ts
{
type: 'text',
name: 'someText',
index: true,
},
{
type: 'array',
name: 'some',
index: true,
fields: [
{
type: 'text',
name: 'text',
index: true,
},
],
}
```
Lead to the error:
```
Warning We've found duplicated index name across public schema. Please rename your index in either the demonstration table or the table with the duplicated index name
```
Now, if we encounter duplicates, we increment the name like this:
`collection_some_text_idx`
`collection_some_text_1_idx`
---------
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
Inheriting all the fixes from drizzle moving to latest versions
## BREAKING CHANGES
If you have a prior version of @libsql/client installed in your project,
you must upgrade to 0.14.0
Filtering by `null` `number` field values or normal values with the
`exists` operator was not working in `postgres` & `sqlite`.
Was previously fixed for `mongodb`
[here](https://github.com/payloadcms/payload/pull/8416)
Now fixed for `postgres` & `sqlite` adapters as well.
