### What?
Previously, the `req` argument:
In database operations (e.g `payload.db`) was required and you needed to
pass the whole `req` with all the properties. This is confusing because
in database operations we never use its properties outside of
`req.transactionID` and `req.t`, both of which should be optional as
well.
Now, you don't have to do that cast:
```ts
payload.db.findOne({
collection: 'posts',
req: {} as PayloadRequest,
where: {
id: {
equals: 1,
},
},
})
```
Becomes:
```ts
payload.db.findOne({
collection: 'posts',
where: {
id: {
equals: 1,
},
},
})
```
If you need to use transactions, you're not required to do the `as` cast
as well now, as the `req` not only optional but also partial -
`Partial<PayloadRequest>`.
`initTransaction`, `commitTransaction`, `killTransaction` utilities are
typed better now as well. They do not require to you pass all the
properties of `req`, but only `payload` -
`MarkRequired<Partial<PayloadRequest>, 'payload'>`
```ts
const req = { payload }
await initTransaction(req)
await payload.db.create({
collection: "posts",
data: {},
req
})
await commitTransaction(req)
```
The same for the Local API. Internal operations (for example
`packages/payload/src/collections/operations/find.ts`) still accept the
whole `req`, but local ones
(`packages/payload/src/collections/operations/local/find.ts`) which are
used through `payload.` now accept `Partial<PayloadRequest>`, as they
pass it through to internal operations with `createLocalReq`.
So now, this is also valid, while previously you had to do `as` cast for
`req`.
```ts
const req = { payload }
await initTransaction(req)
await payload.create({
collection: "posts",
data: {},
req
})
await commitTransaction(req)
```
Marked as deprecated `PayloadRequest['transactionIDPromise']` to remove
in the next major version. It was never used anywhere.
Refactored `withSession` that returns an object to `getSession` that
returns just `ClientSession`. Better type safety for arguments
Deduplicated in all drizzle operations to `getTransaction(this, req)`
utility:
```ts
const db = this.sessions[await req?.transactionID]?.db || this.drizzle
```
Added fallback for throwing unique validation errors in database
operations when `req.t` is not available.
In migration `up` and `down` functions our `req` is not partial, while
we used to passed `req` with only 2 properties - `payload` and
`transactionID`. This is misleading and you can't access for example
`req.t`.
Now, to achieve "real" full `req` - we generate it with `createLocalReq`
in all migration functions.
This all is backwards compatible. In all public API places where you
expect the full `req` (like hooks) you still have it.
### Why?
Better DX, more expected types, less errors because of types casting.
This PR adds a feature which fixes another issue with migrations in
Postgres and does few refactors that significantly reduce code
duplication.
Previously, if you needed to use the underlying database directly in
migrations with the active transaction (for example to execute raw SQL),
created from `payload create:migration`, as `req` doesn't work there you
had to do something like this:
```ts
// Postgres
export async function up({ payload, req }: MigrateUpArgs): Promise<void> {
const db = payload.db.sessions?.[await req.transactionID!].db ?? payload.db
const { rows: posts } = await db.execute(sql`SELECT * from posts`)
}
// MongoDB
export async function up({ payload, req }: MigrateUpArgs): Promise<void> {
const session = payload.db.sessions?.[await req.transactionID!]
const posts = await payload.db.collections.posts.collection.find({ session }).toArray()
}
```
Which was:
1. Awkward to write
2. Not documented anywhere
Now, we expose `session` and `db` to `up` and `down` functions for you:
#### MongoDB:
```ts
import { type MigrateUpArgs } from '@payloadcms/db-mongodb'
export async function up({ session, payload, req }: MigrateUpArgs): Promise<void> {
const posts = await payload.db.collections.posts.collection.find({ session }).toArray()
}
```
#### Postgres:
```ts
import { type MigrateUpArgs, sql } from '@payloadcms/db-postgres'
export async function up({ db, payload, req }: MigrateUpArgs): Promise<void> {
const { rows: posts } = await db.execute(sql`SELECT * from posts`)
}
```
#### SQLite:
```ts
import { type MigrateUpArgs, sql } from '@payloadcms/db-sqlite'
export async function up({ db, payload, req }: MigrateUpArgs): Promise<void> {
const { rows: posts } = await db.run(sql`SELECT * from posts`)
}
```
This actually was a thing with Postgres migrations, we already were
passing `db`, but:
1. Only for `up` and when running `payload migrate`, not for example
with `payload migrate:fresh`
2. Not documented neither in TypeScript or docs.
By ensuring we use `db`, this also fixes an issue that affects all
Postgres/SQLite migrations:
Currently, if we run `payload migration:create` with the postgres
adapter we get a file like this:
```ts
import { MigrateUpArgs, MigrateDownArgs, sql } from '@payloadcms/db-postgres'
export async function up({ payload, req }: MigrateUpArgs): Promise<void> {
await payload.db.drizzle.execute(sql`
CREATE TABLE IF NOT EXISTS "users" (
"id" serial PRIMARY KEY NOT NULL,
);
```
Looks good?
Not exactly!
`payload.db.drizzle.execute()` doesn't really use the current
transaction which can lead to some problems.
Instead, it should use the `db` from `payload.db.sessions?.[await
req.transactionID!].db` because that's where we store our Drizzle
instance with the transaction.
But now, if we generate `payload migrate:create` we get:
```ts
import { MigrateUpArgs, MigrateDownArgs, sql } from '@payloadcms/db-postgres'
export async function up({ db, payload, req }: MigrateUpArgs): Promise<void> {
await db.execute(sql`
CREATE TABLE IF NOT EXISTS "users" (
"id" serial PRIMARY KEY NOT NULL,
);
```
Which is what we want, as the `db` is passed correctly here:
76428373e4/packages/drizzle/src/migrate.ts (L88-L90)
```ts
export async function up({ db, payload, req }: MigrateUpArgs): Promise<void> {
const dbWithTransaction = payload.db.sessions?.[await req.transactionID!].db
payload.logger.info({ one: db === dbWithTransaction })
payload.logger.info({ two: db === payload.db.drizzle })
```
<img width="336" alt="image"
src="https://github.com/user-attachments/assets/f9fab5a9-44c2-44a9-95dd-8e5cf267f027">
Additionally, this PR refactors:
* `createMigration` with Drizzle - now we have sharable
`buildCreateMigration` in `@payloadcms/drizzle` to reduce copy-pasting
of the same logic.
* the `v2-v3` relationships migration for Postgres is now shared between
`db-postgres` and `db-vercel-postgres`, again to reduce copy-paste.
1. Adds flag `--skip-empty` to `migrate:create` to bypass the empty
migration file prompt.
- Blank migration file will not be created if this flag is passed.
3. Adds flag `--force-accept-warning` to `migrate:fresh` to bypass the
drop database prompt
### What?
This command from here:
https://github.com/payloadcms/payload/pull/6339
```sh
payload migrate:create --file @payloadcms/db-postgres/relationships-v2-v3
```
stopped working after db-postgers and drizzle packages were separated
### How?
Passes correct `dirname` to `getPredefinedMigration`
Additionally, adds support for `.js` files in `getPredefinedMigration`
This fixes a peer dependency error in our monorepo, as
eslint-plugin-jsx-a11y finally supports eslint v9.
Additionally, this officially adds TypeScript 5.6 support for
typescript-eslint.
### What?
Adds full support for the point field to Postgres and Vercel Postgres
adapters through the Postgis extension. Fully the same API as with
MongoDB, including support for `near`, `within` and `intersects`
operators.
Additionally, exposes to adapter args:
*
`tablesFilter`https://orm.drizzle.team/docs/drizzle-kit-push#including-tables-schemas-and-extensions.
* `extensions` list of extensions to create, for example `['vector',
'pg_search']`, `postgis` is created automatically if there's any point
field
### Why?
It's essential to support that field type, especially if the postgres
adapter should be out of beta on 3.0 stable.
### How?
* Bumps `drizzle-orm` to `0.36.1` and `drizzle-kit` to `0.28.0` as we
need this change https://github.com/drizzle-team/drizzle-orm/pull/3141
* Uses its functions to achieve querying functionality, for example the
`near` operator works through `ST_DWithin` or `intersects` through
`ST_Intersects`.
* Removes MongoDB condition from all point field tests, but keeps for
SQLite
Resolves these discussions:
https://github.com/payloadcms/payload/discussions/8996https://github.com/payloadcms/payload/discussions/8644
### What?
Handles database name with special characters. For example: `-` -
`my-awesome-app`.
### Why?
Previously, `my-awesome-app` led to this error:
```
Error: failed to create database my-awesome-app.
Details: syntax error at or near "-"
```
This can reproduced for example with `create-payload-app`, as the
generated db name is based on project's name.
### How?
Wraps the query variable to quotes, `create database "my-awesome-app"`
instead of `create database my-awesome-app`.
### What?
Fixes https://github.com/payloadcms/payload/issues/5152 issue related to
sorting by a localized field with SQLite / Postgres database adapters.
### Why?
It was an incorrect behaviour.
### How?
Modifies the `getTableColumnFromPath` file to have correct join
conditions. Previously if you had this structure in the _locales table
_locale title parent
en A 1
es B 1
we sorted by everything that's here, but we need to sort only by the
passed locale.
Additionally fixes a typescript error in `dev.ts` that I added here
https://github.com/payloadcms/payload/pull/8834
Also, removes the condition with `joins.length` in `countDistinct`. It
was there as for this issue
https://github.com/payloadcms/payload/issues/4889 because sorting by a
localized property caused duplication. This can simnifically improve
performance for `.find` with nested querying/sorting on large data sets,
because `count(*)` is faster than `count(DISTINCT id)`
Fixes https://github.com/payloadcms/payload/issues/8752
Previously, trying to define a config like this:
```ts
{
type: 'text',
name: 'someText',
index: true,
},
{
type: 'array',
name: 'some',
index: true,
fields: [
{
type: 'text',
name: 'text',
index: true,
},
],
}
```
Lead to the error:
```
Warning We've found duplicated index name across public schema. Please rename your index in either the demonstration table or the table with the duplicated index name
```
Now, if we encounter duplicates, we increment the name like this:
`collection_some_text_idx`
`collection_some_text_1_idx`
---------
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
Inheriting all the fixes from drizzle moving to latest versions
## BREAKING CHANGES
If you have a prior version of @libsql/client installed in your project,
you must upgrade to 0.14.0
Fixes https://github.com/payloadcms/payload/issues/8413 and
https://github.com/payloadcms/payload/issues/6460
- Builds indexes for relationships by default in the SQL schema
- Fixes `unique: true` handling with Postgres / SQLite for every type of
relationships (non-polymorphic. hasMany, polymorphic, polymorphic
hasMany) _note_: disables unique for nested to arrays / blocks
relationships in the `_rels` table.
- adds tests
2.0 PR tha ports only indexes creation
https://github.com/payloadcms/payload/pull/8446, because `unique: true`
could be a breaking change if someone has incosistent unique data in the
database.
Adds `createDatabase` method to Postgres adapters which can be used
either independently like this:
```ts
payload.db.createDatabase({
name: "some-database",
schemaName: "custom-schema"
})
```
Or
```ts
payload.db.createDatabase()
```
Which creates a database from the current configuration, this is used in
`connect` if `autoDatabaseCreate` is set to `true` (default).
You can disable this behaviour with:
```ts
postgresAdapter({ autoDatabaseCreate: false })
```
Example:
<img width="470" alt="image"
src="https://github.com/user-attachments/assets/8d08c79d-9672-454c-af0f-eb802f9dcd99">
fixes https://github.com/payloadcms/payload/issues/8455 and
https://github.com/payloadcms/payload/issues/8462
- Builds the `_locale` column for the `_rels` when it's inside of a
localized group / tab
- Properly builds `sanitizedPath` for blocks in the transform-read
function when it's inside of a localized field. This fixes with fields
inside that have its own table (like `_rels`, select `hasMany: true`
etc)
Adds _more_ tests!
Adds abillity to customize the generated Drizzle schema with
`beforeSchemaInit` and `afterSchemaInit`. Could be useful if you want to
preserve the existing database schema / override the generated one with
features that aren't supported from the Payload config.
## Docs:
### beforeSchemaInit
Runs before the schema is built. You can use this hook to extend your
database structure with tables that won't be managed by Payload.
```ts
import { postgresAdapter } from '@payloadcms/db-postgres'
import { integer, pgTable, serial } from 'drizzle-orm/pg-core'
postgresAdapter({
beforeSchemaInit: [
({ schema, adapter }) => {
return {
...schema,
tables: {
...schema.tables,
addedTable: pgTable('added_table', {
id: serial('id').notNull(),
}),
},
}
},
],
})
```
One use case is preserving your existing database structure when
migrating to Payload. By default, Payload drops the current database
schema, which may not be desirable in this scenario.
To quickly generate the Drizzle schema from your database you can use
[Drizzle
Introspection](https://orm.drizzle.team/kit-docs/commands#introspect--pull)
You should get the `schema.ts` file which may look like this:
```ts
import { pgTable, uniqueIndex, serial, varchar, text } from 'drizzle-orm/pg-core'
export const users = pgTable('users', {
id: serial('id').primaryKey(),
fullName: text('full_name'),
phone: varchar('phone', { length: 256 }),
})
export const countries = pgTable(
'countries',
{
id: serial('id').primaryKey(),
name: varchar('name', { length: 256 }),
},
(countries) => {
return {
nameIndex: uniqueIndex('name_idx').on(countries.name),
}
},
)
```
You can import them into your config and append to the schema with the
`beforeSchemaInit` hook like this:
```ts
import { postgresAdapter } from '@payloadcms/db-postgres'
import { users, countries } from '../drizzle/schema'
postgresAdapter({
beforeSchemaInit: [
({ schema, adapter }) => {
return {
...schema,
tables: {
...schema.tables,
users,
countries
},
}
},
],
})
```
Make sure Payload doesn't overlap table names with its collections. For
example, if you already have a collection with slug "users", you should
either change the slug or `dbName` to change the table name for this
collection.
### afterSchemaInit
Runs after the Drizzle schema is built. You can use this hook to modify
the schema with features that aren't supported by Payload, or if you
want to add a column that you don't want to be in the Payload config.
To extend a table, Payload exposes `extendTable` utillity to the args.
You can refer to the [Drizzle
documentation](https://orm.drizzle.team/docs/sql-schema-declaration).
The following example adds the `extra_integer_column` column and a
composite index on `country` and `city` columns.
```ts
import { postgresAdapter } from '@payloadcms/db-postgres'
import { index, integer } from 'drizzle-orm/pg-core'
import { buildConfig } from 'payload'
export default buildConfig({
collections: [
{
slug: 'places',
fields: [
{
name: 'country',
type: 'text',
},
{
name: 'city',
type: 'text',
},
],
},
],
db: postgresAdapter({
afterSchemaInit: [
({ schema, extendTable, adapter }) => {
extendTable({
table: schema.tables.places,
columns: {
extraIntegerColumn: integer('extra_integer_column'),
},
extraConfig: (table) => ({
country_city_composite_index: index('country_city_composite_index').on(
table.country,
table.city,
),
}),
})
return schema
},
],
}),
})
```
<!--
For external contributors, please include:
- A summary of the pull request and any related issues it fixes.
- Reasoning for the changes made or any additional context that may be
useful.
Ensure you have read and understand the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository.
-->
This PR addresses these issues with localized groups / tabs with
Postgres / SQLite:
- Array fields inside of localized groups. Fixes
https://github.com/payloadcms/payload/issues/8322
- Select fields with `hasMany: true` inside of localized groups. Related
to 1, but still needed its own additional logic.
- Relationship (non-polymorphic / non has-many) inside of localized
groups. Previously, even just trying to define them in the config led to
a crash. Fixes https://github.com/payloadcms/payload/issues/8308
Ensures test coverage for localized groups.
## Description
- Adds a new "join" field type to Payload and is supported by all database adapters
- The UI uses a table view for the new field
- `db-mongodb` changes relationships to be stored as ObjectIDs instead of strings (for now querying works using both types internally to the DB so no data migration should be necessary unless you're querying directly, see breaking changes for details
- Adds a reusable traverseFields utility to Payload to make it easier to work with nested fields, used internally and for plugin maintainers
```ts
export const Categories: CollectionConfig = {
slug: 'categories',
fields: [
{
name: 'relatedPosts',
type: 'join',
collection: 'posts',
on: 'category',
}
]
}
```
BREAKING CHANGES:
All mongodb relationship and upload values will be stored as MongoDB ObjectIDs instead of strings going forward. If you have existing data and you are querying data directly, outside of Payload's APIs, you get different results. For example, a `contains` query will no longer works given a partial ID of a relationship since the ObjectID requires the whole identifier to work.
---------
Co-authored-by: Jacob Fletcher <jacobsfletch@gmail.com>
Co-authored-by: James <james@trbl.design>
## Description
Adds `virtual` property to the fields config. Providing `true`
completely disables the field in the DB, which is useful for [Virtual
Fields](https://payloadcms.com/blog/learn-how-virtual-fields-can-help-solve-common-cms-challenges)
Disables abillity to query by a field with `virtual: true`.
Currently, they bloat the DB with unused tables / columns, which may as
well introduce additional joins.
Discussion https://github.com/payloadcms/payload/discussions/6270
Prev PR (this one contains only this feature):
https://github.com/payloadcms/payload/pull/6983
- [x] I have read and understand the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository.
## Type of change
<!-- Please delete options that are not relevant. -->
- [x] New feature (non-breaking change which adds functionality)
- [x] This change requires a documentation update
## Checklist:
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] Existing test suite passes locally with my changes
- [x] I have made corresponding changes to the documentation
## Description
1. Adds ability to publish a specific individual locale (collections and
globals)
2. Shows published locale in versions list and version comparison
3. Adds new int tests to `versions` test suite
- [X] I have read and understand the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository.
## Type of change
- [X] New feature (non-breaking change which adds functionality)
- [ ] This change requires a documentation update
## Checklist:
- [X] I have added tests that prove my fix is effective or that my
feature works
- [X] Existing test suite passes locally with my changes
- [ ] I have made corresponding changes to the documentation
---------
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
Previously, this worked with MongoDB but failed with Postgres / SQLite
when the `slug` field has both `localized: true` and `unique: true`.
```ts
await payload.create({
collection: "posts",
locale: "en",
data: {
slug: "my-post"
}
})
await payload.create({
collection: "posts",
locale: "de",
data: {
slug: "my-post"
}
})
```
Now, we build unique constraints and indexes in combination with the
_locale column. This should also improve query performance for fields
with both index: true and localized: true.
### Migration steps (Postgres/SQLite only)
This change updates the database schema and requires a migration (if you
have any localized fields). To apply it, run the following commands:
```sh
pnpm payload migration:create locale_unique_indexes
pnpm payload migrate
```
Note that if you use `db.push: true` which is a default, you don't have
to run `pnpm payload migrate` in the development mode, only in the
production, as Payload automatically pushes the schema to your DB with
it.
Fixes an issue where duplicating documents in Postgres / SQLite would
crash because of a foreign key constraint / unique ID issue when you
have nested arrays / blocks within localized arrays / blocks.
We now run `beforeDuplicate` against all locales prior to
`beforeValidate` and `beforeChange` hooks.
This PR also fixes a series of issues in Postgres / SQLite where you
have localized groups / named tabs, and then arrays / blocks within the
localized groups / named tabs.
## Description
In Postgres, localized blocks or arrays that contain other array / block
/ relationship fields were not properly storing locales in the database.
Now they are! Need to check a few things yet:
- Ensure test coverage is sufficient
- Test localized array, with non-localized array inside of it
- Test localized block with relationship field within it
- Ensure `_rels` table gets the `locale` column added if a single
non-localized relationship exists within a localized array / block
Fixes step 6 as identified in #7805
Dedicated adapter for Vercel Postgres
- Uses the `@vercel/postgres` package under the hood.
- No `pg` dependency, speeds up invocation
- Includes refactoring all base postgres functionality into a
`BasePostgresAdapter` type, which will ease implementation of [other
adapters supported by
drizzle-orm](https://orm.drizzle.team/docs/get-started-postgresql)
## Usage
```ts
import { buildConfig } from 'payload'
import { vercelPostgresAdapter } from '@payloadcms/db-vercel-postgres'
export default buildConfig({
db: vercelPostgresAdapter({
pool: {
connectionString: process.env.DATABASE_URI,
},
}),
// ...rest of config
})
```
### Automatic Connection String Detection
Have Vercel automatically detect from environment variable (typically
`process.env.POSTGRES_URL`)
```ts
export default buildConfig({
db: postgresAdapter(),
// ...rest of config
})
```
Supports `hasMany` upload fields, similar to how `hasMany` works in
other fields, i.e.:
```ts
{
type: 'upload',
relationTo: 'media',
hasMany: true
}
```
---------
Co-authored-by: Jacob Fletcher <jacobsfletch@gmail.com>
Co-authored-by: James <james@trbl.design>
