Continuation of https://github.com/payloadcms/payload/pull/6245.
This PR allows you to pass `blocksAsJSON: true` to SQL adapters and the
adapter instead of aligning with the SQL preferred relation approach for
blocks will just use a simple JSON column, which can improve performance
with a large amount of blocks.
To try these changes you can install `3.43.0-internal.c5bbc84`.
Adds support for read replicas
https://orm.drizzle.team/docs/read-replicas that can be used to offload
read-heavy traffic.
To use (both `db-postgres` and `db-vercel-postgres` are supported):
```ts
import { postgresAdapter } from '@payloadcms/db-postgres'
database: postgresAdapter({
pool: {
connectionString: process.env.POSTGRES_URL,
},
readReplicas: [process.env.POSTGRES_REPLICA_URL],
})
```
---------
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
You can access the database name from `sanitizedConfig.db.name`. But
currently, it' not possible to access the db name from the unsanitized
config.
Plugins only have access to the unsanitized config. This change allows
db adapters to return the db name early, which will allow plugins to
conditionally initialize db-specific functionality
In 3.0, we made the decision to export all types from the main package
export (e.g. `payload/types` => `payload`). This improves type
discoverability by IDEs and simplifies importing types.
This PR does the same for our db adapters, which still have a separate
`/types` subpath export. While those are kept for
backwards-compatibility, we can remove them in 4.0.
Continuation of #11489. This adds a new, optional `updateJobs` db
adapter method that reduces the amount of database calls for the jobs
queue.
## MongoDB
### Previous: running a set of 50 queued jobs
- 1x db.find (= 1x `Model.paginate`)
- 50x db.updateOne (= 50x `Model.findOneAndUpdate`)
### Now: running a set of 50 queued jobs
- 1x db.updateJobs (= 1x `Model.find` and 1x `Model.updateMany`)
**=> 51 db round trips before, 2 db round trips after**
### Previous: upon task completion
- 1x db.find (= 1x `Model.paginate`)
- 1x db.updateOne (= 1x `Model.findOneAndUpdate`)
### Now: upon task completion
- 1x db.updateJobs (= 1x `Model.findOneAndUpdate`)
**=> 2 db round trips before, 1 db round trip after**
## Drizzle (e.g. Postgres)
### running a set of 50 queued jobs
- 1x db.query[tablename].findMany
- 50x db.select
- 50x upsertRow
This is unaffected by this PR and will be addressed in a future PR
Fixes https://github.com/payloadcms/payload/issues/6884
Adds a new flag `acceptIDOnCreate` that allows you to thread your own
`id` to `payload.create` `data`, for example:
```ts
// doc created with id 1
const doc = await payload.create({ collection: 'posts', data: {id: 1, title: "my title"}})
```
```ts
import { Types } from 'mongoose'
const id = new Types.ObjectId().toHexString()
const doc = await payload.create({ collection: 'posts', data: {id, title: "my title"}})
```
This PR allows to have full type safety on `payload.drizzle` with a
single command
```sh
pnpm payload generate:db-schema
```
Which generates TypeScript code with Drizzle declarations based on the
current database schema.
Example of generated file with the website template:
https://gist.github.com/r1tsuu/b8687f211b51d9a3a7e78ba41e8fbf03
Video that shows the power:
https://github.com/user-attachments/assets/3ced958b-ec1d-49f5-9f51-d859d5fae236
We also now proxy drizzle package the same way we do for Lexical so you
don't have to install it (and you shouldn't because you may have version
mismatch).
Instead, you can import from Drizzle like this:
```ts
import {
pgTable,
index,
foreignKey,
integer,
text,
varchar,
jsonb,
boolean,
numeric,
serial,
timestamp,
uniqueIndex,
pgEnum,
} from '@payloadcms/db-postgres/drizzle/pg-core'
import { sql } from '@payloadcms/db-postgres/drizzle'
import { relations } from '@payloadcms/db-postgres/drizzle/relations'
```
Fixes https://github.com/payloadcms/payload/discussions/4318
In the future we can also support types generation for mongoose / raw
mongodb results.
This PR adds a feature which fixes another issue with migrations in
Postgres and does few refactors that significantly reduce code
duplication.
Previously, if you needed to use the underlying database directly in
migrations with the active transaction (for example to execute raw SQL),
created from `payload create:migration`, as `req` doesn't work there you
had to do something like this:
```ts
// Postgres
export async function up({ payload, req }: MigrateUpArgs): Promise<void> {
const db = payload.db.sessions?.[await req.transactionID!].db ?? payload.db
const { rows: posts } = await db.execute(sql`SELECT * from posts`)
}
// MongoDB
export async function up({ payload, req }: MigrateUpArgs): Promise<void> {
const session = payload.db.sessions?.[await req.transactionID!]
const posts = await payload.db.collections.posts.collection.find({ session }).toArray()
}
```
Which was:
1. Awkward to write
2. Not documented anywhere
Now, we expose `session` and `db` to `up` and `down` functions for you:
#### MongoDB:
```ts
import { type MigrateUpArgs } from '@payloadcms/db-mongodb'
export async function up({ session, payload, req }: MigrateUpArgs): Promise<void> {
const posts = await payload.db.collections.posts.collection.find({ session }).toArray()
}
```
#### Postgres:
```ts
import { type MigrateUpArgs, sql } from '@payloadcms/db-postgres'
export async function up({ db, payload, req }: MigrateUpArgs): Promise<void> {
const { rows: posts } = await db.execute(sql`SELECT * from posts`)
}
```
#### SQLite:
```ts
import { type MigrateUpArgs, sql } from '@payloadcms/db-sqlite'
export async function up({ db, payload, req }: MigrateUpArgs): Promise<void> {
const { rows: posts } = await db.run(sql`SELECT * from posts`)
}
```
This actually was a thing with Postgres migrations, we already were
passing `db`, but:
1. Only for `up` and when running `payload migrate`, not for example
with `payload migrate:fresh`
2. Not documented neither in TypeScript or docs.
By ensuring we use `db`, this also fixes an issue that affects all
Postgres/SQLite migrations:
Currently, if we run `payload migration:create` with the postgres
adapter we get a file like this:
```ts
import { MigrateUpArgs, MigrateDownArgs, sql } from '@payloadcms/db-postgres'
export async function up({ payload, req }: MigrateUpArgs): Promise<void> {
await payload.db.drizzle.execute(sql`
CREATE TABLE IF NOT EXISTS "users" (
"id" serial PRIMARY KEY NOT NULL,
);
```
Looks good?
Not exactly!
`payload.db.drizzle.execute()` doesn't really use the current
transaction which can lead to some problems.
Instead, it should use the `db` from `payload.db.sessions?.[await
req.transactionID!].db` because that's where we store our Drizzle
instance with the transaction.
But now, if we generate `payload migrate:create` we get:
```ts
import { MigrateUpArgs, MigrateDownArgs, sql } from '@payloadcms/db-postgres'
export async function up({ db, payload, req }: MigrateUpArgs): Promise<void> {
await db.execute(sql`
CREATE TABLE IF NOT EXISTS "users" (
"id" serial PRIMARY KEY NOT NULL,
);
```
Which is what we want, as the `db` is passed correctly here:
76428373e4/packages/drizzle/src/migrate.ts (L88-L90)
```ts
export async function up({ db, payload, req }: MigrateUpArgs): Promise<void> {
const dbWithTransaction = payload.db.sessions?.[await req.transactionID!].db
payload.logger.info({ one: db === dbWithTransaction })
payload.logger.info({ two: db === payload.db.drizzle })
```
<img width="336" alt="image"
src="https://github.com/user-attachments/assets/f9fab5a9-44c2-44a9-95dd-8e5cf267f027">
Additionally, this PR refactors:
* `createMigration` with Drizzle - now we have sharable
`buildCreateMigration` in `@payloadcms/drizzle` to reduce copy-pasting
of the same logic.
* the `v2-v3` relationships migration for Postgres is now shared between
`db-postgres` and `db-vercel-postgres`, again to reduce copy-paste.
### What?
This command from here:
https://github.com/payloadcms/payload/pull/6339
```sh
payload migrate:create --file @payloadcms/db-postgres/relationships-v2-v3
```
stopped working after db-postgers and drizzle packages were separated
### How?
Passes correct `dirname` to `getPredefinedMigration`
Additionally, adds support for `.js` files in `getPredefinedMigration`
This fixes a peer dependency error in our monorepo, as
eslint-plugin-jsx-a11y finally supports eslint v9.
Additionally, this officially adds TypeScript 5.6 support for
typescript-eslint.
Currently, Payload renders all custom components on initial compile of
the admin panel. This is problematic for two key reasons:
1. Custom components do not receive contextual data, i.e. fields do not
receive their field data, edit views do not receive their document data,
etc.
2. Components are unnecessarily rendered before they are used
This was initially required to support React Server Components within
the Payload Admin Panel for two key reasons:
1. Fields can be dynamically rendered within arrays, blocks, etc.
2. Documents can be recursively rendered within a "drawer" UI, i.e.
relationship fields
3. Payload supports server/client component composition
In order to achieve this, components need to be rendered on the server
and passed as "slots" to the client. Currently, the pattern for this is
to render custom server components in the "client config". Then when a
view or field is needed to be rendered, we first check the client config
for a "pre-rendered" component, otherwise render our client-side
fallback component.
But for the reasons listed above, this pattern doesn't exactly make
custom server components very useful within the Payload Admin Panel,
which is where this PR comes in. Now, instead of pre-rendering all
components on initial compile, we're able to render custom components
_on demand_, only as they are needed.
To achieve this, we've established [this
pattern](https://github.com/payloadcms/payload/pull/8481) of React
Server Functions in the Payload Admin Panel. With Server Functions, we
can iterate the Payload Config and return JSX through React's
`text/x-component` content-type. This means we're able to pass
contextual props to custom components, such as data for fields and
views.
## Breaking Changes
1. Add the following to your root layout file, typically located at
`(app)/(payload)/layout.tsx`:
```diff
/* THIS FILE WAS GENERATED AUTOMATICALLY BY PAYLOAD. */
/* DO NOT MODIFY IT BECAUSE IT COULD BE REWRITTEN AT ANY TIME. */
+ import type { ServerFunctionClient } from 'payload'
import config from '@payload-config'
import { RootLayout } from '@payloadcms/next/layouts'
import { handleServerFunctions } from '@payloadcms/next/utilities'
import React from 'react'
import { importMap } from './admin/importMap.js'
import './custom.scss'
type Args = {
children: React.ReactNode
}
+ const serverFunctions: ServerFunctionClient = async function (args) {
+ 'use server'
+ return handleServerFunctions({
+ ...args,
+ config,
+ importMap,
+ })
+ }
const Layout = ({ children }: Args) => (
<RootLayout
config={config}
importMap={importMap}
+ serverFunctions={serverFunctions}
>
{children}
</RootLayout>
)
export default Layout
```
2. If you were previously posting to the `/api/form-state` endpoint, it
no longer exists. Instead, you'll need to invoke the `form-state` Server
Function, which can be done through the _new_ `getFormState` utility:
```diff
- import { getFormState } from '@payloadcms/ui'
- const { state } = await getFormState({
- apiRoute: '',
- body: {
- // ...
- },
- serverURL: ''
- })
+ const { getFormState } = useServerFunctions()
+
+ const { state } = await getFormState({
+ // ...
+ })
```
## Breaking Changes
```diff
- useFieldProps()
- useCellProps()
```
More details coming soon.
---------
Co-authored-by: Alessio Gravili <alessio@gravili.de>
Co-authored-by: Jarrod Flesch <jarrodmflesch@gmail.com>
Co-authored-by: James <james@trbl.design>
### What?
Adds full support for the point field to Postgres and Vercel Postgres
adapters through the Postgis extension. Fully the same API as with
MongoDB, including support for `near`, `within` and `intersects`
operators.
Additionally, exposes to adapter args:
*
`tablesFilter`https://orm.drizzle.team/docs/drizzle-kit-push#including-tables-schemas-and-extensions.
* `extensions` list of extensions to create, for example `['vector',
'pg_search']`, `postgis` is created automatically if there's any point
field
### Why?
It's essential to support that field type, especially if the postgres
adapter should be out of beta on 3.0 stable.
### How?
* Bumps `drizzle-orm` to `0.36.1` and `drizzle-kit` to `0.28.0` as we
need this change https://github.com/drizzle-team/drizzle-orm/pull/3141
* Uses its functions to achieve querying functionality, for example the
`near` operator works through `ST_DWithin` or `intersects` through
`ST_Intersects`.
* Removes MongoDB condition from all point field tests, but keeps for
SQLite
Resolves these discussions:
https://github.com/payloadcms/payload/discussions/8996https://github.com/payloadcms/payload/discussions/8644
Adjust drizzle init for changes in drizzle 0.35.0
https://github.com/drizzle-team/drizzle-orm/releases/tag/0.35.0
The pool/connection should now be passed as the `client` arg when
initializing drizzle.
```ts
this.drizzle = drizzle({
client: this.poolOptions ? new VercelPool(this.poolOptions) : sql,
logger,
schema: this.schema,
})
```
This was causing an issue where running `payload migrate` on Vercel was
causing drizzle to attempt to `127.0.0.1:5432` instead of the specified
environment variable in the adapter 🤔
Fixes https://github.com/payloadcms/payload/issues/8752
Previously, trying to define a config like this:
```ts
{
type: 'text',
name: 'someText',
index: true,
},
{
type: 'array',
name: 'some',
index: true,
fields: [
{
type: 'text',
name: 'text',
index: true,
},
],
}
```
Lead to the error:
```
Warning We've found duplicated index name across public schema. Please rename your index in either the demonstration table or the table with the duplicated index name
```
Now, if we encounter duplicates, we increment the name like this:
`collection_some_text_idx`
`collection_some_text_1_idx`
---------
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
Adds `createDatabase` method to Postgres adapters which can be used
either independently like this:
```ts
payload.db.createDatabase({
name: "some-database",
schemaName: "custom-schema"
})
```
Or
```ts
payload.db.createDatabase()
```
Which creates a database from the current configuration, this is used in
`connect` if `autoDatabaseCreate` is set to `true` (default).
You can disable this behaviour with:
```ts
postgresAdapter({ autoDatabaseCreate: false })
```
Example:
<img width="470" alt="image"
src="https://github.com/user-attachments/assets/8d08c79d-9672-454c-af0f-eb802f9dcd99">
Adds abillity to customize the generated Drizzle schema with
`beforeSchemaInit` and `afterSchemaInit`. Could be useful if you want to
preserve the existing database schema / override the generated one with
features that aren't supported from the Payload config.
## Docs:
### beforeSchemaInit
Runs before the schema is built. You can use this hook to extend your
database structure with tables that won't be managed by Payload.
```ts
import { postgresAdapter } from '@payloadcms/db-postgres'
import { integer, pgTable, serial } from 'drizzle-orm/pg-core'
postgresAdapter({
beforeSchemaInit: [
({ schema, adapter }) => {
return {
...schema,
tables: {
...schema.tables,
addedTable: pgTable('added_table', {
id: serial('id').notNull(),
}),
},
}
},
],
})
```
One use case is preserving your existing database structure when
migrating to Payload. By default, Payload drops the current database
schema, which may not be desirable in this scenario.
To quickly generate the Drizzle schema from your database you can use
[Drizzle
Introspection](https://orm.drizzle.team/kit-docs/commands#introspect--pull)
You should get the `schema.ts` file which may look like this:
```ts
import { pgTable, uniqueIndex, serial, varchar, text } from 'drizzle-orm/pg-core'
export const users = pgTable('users', {
id: serial('id').primaryKey(),
fullName: text('full_name'),
phone: varchar('phone', { length: 256 }),
})
export const countries = pgTable(
'countries',
{
id: serial('id').primaryKey(),
name: varchar('name', { length: 256 }),
},
(countries) => {
return {
nameIndex: uniqueIndex('name_idx').on(countries.name),
}
},
)
```
You can import them into your config and append to the schema with the
`beforeSchemaInit` hook like this:
```ts
import { postgresAdapter } from '@payloadcms/db-postgres'
import { users, countries } from '../drizzle/schema'
postgresAdapter({
beforeSchemaInit: [
({ schema, adapter }) => {
return {
...schema,
tables: {
...schema.tables,
users,
countries
},
}
},
],
})
```
Make sure Payload doesn't overlap table names with its collections. For
example, if you already have a collection with slug "users", you should
either change the slug or `dbName` to change the table name for this
collection.
### afterSchemaInit
Runs after the Drizzle schema is built. You can use this hook to modify
the schema with features that aren't supported by Payload, or if you
want to add a column that you don't want to be in the Payload config.
To extend a table, Payload exposes `extendTable` utillity to the args.
You can refer to the [Drizzle
documentation](https://orm.drizzle.team/docs/sql-schema-declaration).
The following example adds the `extra_integer_column` column and a
composite index on `country` and `city` columns.
```ts
import { postgresAdapter } from '@payloadcms/db-postgres'
import { index, integer } from 'drizzle-orm/pg-core'
import { buildConfig } from 'payload'
export default buildConfig({
collections: [
{
slug: 'places',
fields: [
{
name: 'country',
type: 'text',
},
{
name: 'city',
type: 'text',
},
],
},
],
db: postgresAdapter({
afterSchemaInit: [
({ schema, extendTable, adapter }) => {
extendTable({
table: schema.tables.places,
columns: {
extraIntegerColumn: integer('extra_integer_column'),
},
extraConfig: (table) => ({
country_city_composite_index: index('country_city_composite_index').on(
table.country,
table.city,
),
}),
})
return schema
},
],
}),
})
```
<!--
For external contributors, please include:
- A summary of the pull request and any related issues it fixes.
- Reasoning for the changes made or any additional context that may be
useful.
Ensure you have read and understand the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository.
-->
- Adds the upsert method to the database interface
- Adds a mongodb specific option to extend the updateOne to accept
mongoDB Query Options (to pass `upsert: true`)
- Added upsert method to all database adapters
- Uses db.upsert in the payload preferences update operation
Includes a test using payload-preferences
## Description
1. Adds ability to publish a specific individual locale (collections and
globals)
2. Shows published locale in versions list and version comparison
3. Adds new int tests to `versions` test suite
- [X] I have read and understand the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository.
## Type of change
- [X] New feature (non-breaking change which adds functionality)
- [ ] This change requires a documentation update
## Checklist:
- [X] I have added tests that prove my fix is effective or that my
feature works
- [X] Existing test suite passes locally with my changes
- [ ] I have made corresponding changes to the documentation
---------
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
Dedicated adapter for Vercel Postgres
- Uses the `@vercel/postgres` package under the hood.
- No `pg` dependency, speeds up invocation
- Includes refactoring all base postgres functionality into a
`BasePostgresAdapter` type, which will ease implementation of [other
adapters supported by
drizzle-orm](https://orm.drizzle.team/docs/get-started-postgresql)
## Usage
```ts
import { buildConfig } from 'payload'
import { vercelPostgresAdapter } from '@payloadcms/db-vercel-postgres'
export default buildConfig({
db: vercelPostgresAdapter({
pool: {
connectionString: process.env.DATABASE_URI,
},
}),
// ...rest of config
})
```
### Automatic Connection String Detection
Have Vercel automatically detect from environment variable (typically
`process.env.POSTGRES_URL`)
```ts
export default buildConfig({
db: postgresAdapter(),
// ...rest of config
})
```
Allow a compound index to be used for upload collections via a
`filenameCompoundIndex` field. Previously, `filename` was always treated
as unique.
Usage:
```ts
{
slug: 'upload-field',
upload: {
// Slugs to include in compound index
filenameCompoundIndex: ['filename', 'alt'],
},
}
```
## Description
Fixes#7492
In order to run createMigration, we need to read in the previous
snapshot file if one exists. When that snapshot was generated from an
older version of drizzle-kit, we have to first migrate it up match the
latest version for drizzle to generate the new migration. This change
adds in the call to check the version and migrate the snapshot if
needed.
## Description
Prior to this change, the `defaultValue` for fields have only been used
in the application layer of Payload. With this change, you get the added
benefit of having the database columns get the default also. This is
especially helpful when adding new columns to postgres with existing
data to avoid needing to write complex migrations. In MongoDB this
change applies the default to the Mongoose model which is useful when
calling payload.db.create() directly.
This only works for statically defined values.
🙏 A big thanks to @r1tsuu for the feature and implementation idea as I
lifted some code from PR https://github.com/payloadcms/payload/pull/6983
- [x] I have read and understand the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository.
## Type of change
- [x] New feature (non-breaking change which adds functionality)
- [x] This change requires a documentation update
## Checklist:
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] Existing test suite passes locally with my changes
- [x] I have made corresponding changes to the documentation
## Description
### payload
- Removes calls to beginTransaction and commitTransaction from read
operations
### db-sqlite, db-postgres
- beginTransaction() options are passed through and used to create a
transaction
- declare module type adds beginTransaction with proper transaction
config args for postgres and sqlite
Fixes#7402
This fixes a regression from changes to the postgres migration template
that were incorrect. It also fixes other type errors for
`payload.db.drizzle` which needed to be declared for postgres to avoid
confusing it with Libsql for SQLite.
- Abstract shared sql code to a new drizzle package
- Adds sqlite package, not ready to publish until drizzle patches some
issues
- Add `transactionOptions` to allow customizing or disabling db
transactions
- Adds "experimental" label to the `schemaName` property until drizzle
patches an issue
## Description
This is the beta (v3) PR for the v2 PR
[here](https://github.com/payloadcms/payload/pull/6857)
Addresses #6800, #5108
- [x] I have read and understand the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository.
## Type of change
- [x] Bug fix (non-breaking change which fixes an issue)
## Checklist:
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] Existing test suite passes locally with my changes
Removes PayloadRequestWithData in favour of just PayloadRequest with
optional types for `data` and `locale`
`addDataAndFileToRequest` and `addLocalesToRequestFromData` now takes in
a single argument instead of an object
```ts
// before
await addDataAndFileToRequest({ request: req })
addLocalesToRequestFromData({ request: req })
// current
await addDataAndFileToRequest(req)
addLocalesToRequestFromData(req)
```
---------
Co-authored-by: Paul Popus <paul@nouance.io>
BREAKING: `ValidationError` now requires the `global` or `collection`
slug, as well as an `errors` property. The actual errors are no longer
at the top-level.
**BREAKING:**
- Type narrowing for `relationTo` props on filterOptions, relationship
fields and upload fields
- Type narrowing for arguments of lexical relationship, link and upload
features
