Commit Graph

74 Commits

Author SHA1 Message Date
Sasha
1b1e36e2df fix(db-*): migrate:reset executes in a wrong order (#12445)
fixes https://github.com/payloadcms/payload/issues/12442
2025-05-22 13:30:29 -04:00
Sasha
81d333f4b0 test: add test for sorting by a virtual field with a reference (#12351) 2025-05-20 13:07:48 +00:00
Sasha
219fd01717 fix(db-postgres): allow the same block slug in different places with a different localized value (#12414)
Fixes https://github.com/payloadcms/payload/issues/12409
Now Payload automatically resolves table names conflicts in those cases,
as well as Drizzle relation names.

---------

Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
2025-05-15 16:48:41 -04:00
Sasha
1f6efe9a46 fix: respect hidden: true for virtual fields that have reference to a relationship field (#12219)
Previously, `hidden: true` on a virtual field that references a
relationship field didn't work. Now, this field doesn't get calculated
if there's `hidden: true` and no `showHiddenFields` was passed.

Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
2025-05-15 16:48:08 -04:00
Sasha
5fce501589 fix(db-postgres): dbName in arrays regression with long generated drizzle relation names (#12237)
Fixes https://github.com/payloadcms/payload/issues/12136 which caused by
regression from https://github.com/payloadcms/payload/pull/11995

The previous PR solved an issue where the generated drizzle relation
name was too long because of Payload field names, for example
```
{
  name: 'thisIsALongFieldNameThatWillCauseAPostgresErrorEvenThoughWeSetAShorterDBName',
  dbName: 'shortname',
  type: 'array',
  fields: [
    {
      name: 'nested_field_1',
      type: 'array',
      dbName: 'short_nested_1',
      fields: [],
    },
    {
      name: 'nested_field_2',
      type: 'text',
    },
  ],
},
```
But it caused regression, when custom `dbName` vice versa caused long
relation names:
```
export const Header: GlobalConfig = {
  slug: 'header',
  fields: [
    {
      name: 'itemsLvl1',
      type: 'array',
      dbName: 'header_items_lvl1',
      fields: [
        {
          name: 'label',
          type: 'text',
        },
        {
          name: 'itemsLvl2',
          type: 'array',
          dbName: 'header_items_lvl2',
          fields: [
            {
              name: 'label',
              type: 'text',
            },
            {
              name: 'itemsLvl3',
              type: 'array',
              dbName: 'header_items_lvl3',
              fields: [
                {
                  name: 'label',
                  type: 'text',
                },
                {
                  name: 'itemsLvl4',
                  type: 'array',
                  dbName: 'header_items_lvl4',
                  fields: [
                    {
                      name: 'label',
                      type: 'text',
                    },
                  ],
                },
              ],
            },
          ],
        },
      ],
    },
  ],
}
```

Notice if you calculate the generated relation name for `itemsLvl4` you
get:

`header__header_items_lvl1__header_items_lvl2__header_items_lvl3_header_items_lvl4`
- 81 characters, Drizzle, for joining shrink the alias to 63 characters
-`header__header_items_lvl1__header_items_lvl2__header_items_lvl3` and
Postgres throws:
```
error: table name "header__header_items_lvl1__header_items_lvl2__header_items_lvl3" specified more than once
```
2025-05-15 13:40:24 -04:00
Sasha
4a56597b92 fix(db-postgres): count crashes when query contains subqueries and doesn't return any rows (#12273)
Fixes https://github.com/payloadcms/payload/issues/12264

Uses safe object access in `countDistinct`, fallbacks to `0`
2025-04-30 16:53:36 +00:00
Sasha
564fdb0e17 fix: virtual relationship fields with select (#12266)
Continuation of https://github.com/payloadcms/payload/pull/12265.

Currently, using `select` on new relationship virtual fields:
```
const doc = await payload.findByID({
  collection: 'virtual-relations',
  depth: 0,
  id,
  select: { postTitle: true },
})
```
doesn't work, because in order to calculate `post.title`, the `post`
field must be selected as well. This PR adds logic that sanitizes the
incoming `select` to include those relationships into `select` (that are
related to selected virtual fields)

---------

Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
2025-04-30 12:27:04 -04:00
Sasha
6dc61ae642 fix(db-mongodb): fallback version when not selected (#12158)
When doing `payload.db.queryDrafts` with `select` without `version`, or
simply your select looks like:
`select: { version: { nonExistingField: true } }` - the `queryDrafts`
function will crash because it tries to access the `version` field.
This PR adds a fallback.
2025-04-18 21:43:53 +03:00
Sasha
fdff5871f6 perf: optimize virtual fields that reference ID (#12159)
This PR optimizes the new virtual fields with relationships feature
https://github.com/payloadcms/payload/pull/11805 when the path
references the ID field, for example:
```
{
  name: 'postCategoryID',
  type: 'number',
  virtual: 'post.category.id',
},
```

Previously, we did additional population of `category`, which is
unnecessary as we can always grab the ID from the `category` value
itself. One less querying step.
2025-04-18 21:39:55 +03:00
Sasha
1c99f46e4f feat: queriable / sortable / useAsTitle virtual fields linked with a relationship field (#11805)
This PR adds an ability to specify a virtual field in this way
```js
{
  slug: 'posts',
  fields: [
    {
      name: 'title',
      type: 'text',
      required: true,
    },
  ],
},
{
  slug: 'virtual-relations',
  fields: [
    {
      name: 'postTitle',
      type: 'text',
      virtual: 'post.title',
    },
    {
      name: 'post',
      type: 'relationship',
      relationTo: 'posts',
    },
  ],
},
```

Then, every time you query `virtual-relations`, `postTitle` will be
automatically populated (even if using `depth: 0`) on the db level. This
field also, unlike `virtual: true` is available for querying / sorting /
`useAsTitle`.

Also, the field can be deeply nested to 2 or more relationships, for
example:
```
{
  name: 'postCategoryTitle',
  type: 'text',
  virtual: 'post.category.title',
},
```

Where the current collection has `post` - a relationship to `posts`, the
collection `posts` has `category` that's a relationship to `categories`
and finally `categories` has `title`.
2025-04-16 15:46:18 -04:00
Dan Ribbens
6572bf4ae1 fix(db-sqlite): text field converts to floating point number (#12107)
### What?

Converts numbers passed to a text field to avoid the database/drizzle
from converting it incorrectly.

### Why?

If you have a hook that passes a value to another field you can
experience this problem where drizzle converts a number value for a text
field to a floating point number in sqlite for example.

### How?

Adds logic to `transform/write/traverseFields.ts` to cast text field
values to string.
2025-04-14 17:05:08 -04:00
Sasha
09782be0e0 fix(db-postgres): long array field table aliases cause error even when dbName is used (#11995)
Fixes https://github.com/payloadcms/payload/issues/11975

Previously, this configuration was causing errors in postgres due to
long names, even though `dbName` is used:
```
{
  slug: 'aliases',
  fields: [
    {
      name: 'thisIsALongFieldNameThatWillCauseAPostgresErrorEvenThoughWeSetAShorterDBName',
      dbName: 'shortname',
      type: 'array',
      fields: [
        {
          name: 'nested_field_1',
          type: 'array',
          dbName: 'short_nested_1',
          fields: [],
        },
        {
          name: 'nested_field_2',
          type: 'text',
        },
      ],
    },
  ],
},
```

This is because we were generating Drizzle relation name (for arrays)
always based on the field path and internally, drizzle uses this name
for aliasing. Now, if `dbName` is present, we use `_{dbName}` instead
for the relation name.
2025-04-07 20:12:43 +00:00
Sasha
4ebd3ce668 fix(db-postgres): deleteOne fails when the where query does not resolve to any document (#11632)
Previously, if you called `payload.db.deleteOne` with a `where` query
that does not resolve to anything, an error would be occurred.
2025-04-04 00:46:31 +03:00
Alessio Gravili
20e975b7c6 feat: sort support for payload.update operation (#11769)
Continuation of https://github.com/payloadcms/payload/pull/11768. This
adds support for `sort` in `payload.update`.

## Example

```ts
const { docs } = await payload.update({
  collection: 'posts',
  data: {
    title: 'updated',
  },
  limit: 5,
  sort: '-numberField', // <= new
  where: {
    id: {
      exists: true,
    },
  },
})
```
2025-03-19 17:22:13 +00:00
Alessio Gravili
e96d3c87e2 feat(db-*): support sort in db.updateMany (#11768)
This adds support for `sort` in `payload.db.updateMany`.

## Example

```ts
const updatedDocs = await payload.db.updateMany({
  collection: 'posts',
  data: {
    title: 'updated',
  },
  limit: 5,
  sort: '-numberField', // <= new
  where: {
    id: {
      exists: true,
    },
  },
})
```
2025-03-19 10:47:58 -06:00
Sasha
f442d22237 feat(db-*): allow to thread id to create operation data without custom IDs (#11709)
Fixes https://github.com/payloadcms/payload/issues/6884

Adds a new flag `acceptIDOnCreate` that allows you to thread your own
`id` to `payload.create` `data`, for example:

```ts
// doc created with id 1
const doc = await payload.create({ collection: 'posts', data: {id: 1, title: "my title"}})
```

```ts
import { Types } from 'mongoose'
const id = new Types.ObjectId().toHexString()
const doc = await payload.create({ collection: 'posts', data: {id, title: "my title"}})
```
2025-03-17 23:48:35 -04:00
Germán Jabloñski
885f580b58 chore: fix typo (rename mognoose to mongoose) (#11653)
Fixes #11652
2025-03-12 23:01:41 +02:00
Sasha
2ad035fb7b feat(db-mongodb): strip keys from the data that don't exist in the schema from read results (#11558)
This change makes so that data that exists in MongoDB but isn't defined
in the Payload config won't be included to `payload.find` /
`payload.db.find` calls. Now we strip all the additional keys.

Consider you have a field named `secretField` that's also `hidden: true`
(or `read: () => false`) that contains some sensitive data. Then you
removed this field from the database and as for now with the MongoDB
adapter this field will be included to the Local API / REST API results
without any consideration, as Payload doesn't know about it anymore.

This also fixes https://github.com/payloadcms/payload/issues/11542 if
you removed / renamed a relationship field from the schema, Payload
won't sanitize ObjectIDs back to strings anymore.

Ideally you should create a migration script that completely removes the
deleted field from the database with `$unset`, but people rarely do
this.

If you still need to keep those fields to the result, this PR allows you
to do this with the new `allowAdditionalKeys: true` flag.
2025-03-06 14:31:38 +00:00
Sasha
bacc0f002a feat: compound indexes (#11512)
### What?
This PR adds ability to define indexes on several fields for collections
(compound indexes).

Example:
```ts
{
  indexes: [{ unique: true, fields: ['title', 'group.name'] }]
}
```

### Why?
This can be used to either speed up querying/sorting by 2 or more fields
at the same time or to ensure uniqueness between several fields.

### How?
Implements this logic in database adapters. Additionally, adds a utility
`getFieldByPath`.
2025-03-05 03:09:24 +02:00
Alessio Gravili
6a3d58bb32 feat(db-*): support limit in db.updateMany (#11488)
This PR adds a new `limit` property to `payload.db.updateMany`. This functionality is required for [migrating our job system to use faster, direct db adapter calls](https://github.com/payloadcms/payload/pull/11489)
2025-03-03 05:32:57 +00:00
Alessio Gravili
41c7413f59 feat(db-*): add updateMany method to database adapter (#11441)
This PR adds a new `payload.db.updateMany` method, which is a more performant way to update multiple documents compared to using `payload.update`.
2025-02-27 20:30:17 -07:00
Jarrod Flesch
d2fe9b0807 fix(db-mongodb): ensures same level operators are respected (#11087)
### What?
If you had multiple operator constraints on a single field, the last one
defined would be the only one used.

Example:
```ts
where: {
  id: {
    in: [doc2.id],
    not_in: [], // <-- only respected this operator constraint
  },
}
```

and
```ts
where: {
  id: {
    not_in: [],
    in: [doc2.id], // <-- only respected this operator constraint
  },
}
```

They would yield different results.

### Why?
The results were not merged into an `$and` query inside parseParams.

### How?
Merges the results within an `$and` constraint.

Fixes https://github.com/payloadcms/payload/issues/10944

Supersedes https://github.com/payloadcms/payload/pull/11011
2025-02-10 16:29:08 -05:00
Sasha
57143b37d0 fix(db-postgres): ensure globals have createdAt, updatedAt and globalType fields (#10938)
Previously, data for globals was inconsistent across database adapters.
In Postgres, globals didn't store correct `createdAt`, `updatedAt`
fields and the `updateGlobal` lacked the `globalType` field. This PR
solves that without introducing schema changes.
2025-02-06 23:48:59 +02:00
Jacob Fletcher
0acaf8a7f7 fix: field paths within hooks (#10638)
Field paths within hooks are not correct.

For example, an unnamed tab containing a group field and nested text
field should have the path:
- `myGroupField.myTextField`

However, within hooks that path is formatted as:
- `_index-1.myGroupField.myTextField`

The leading index shown above should not exist, as this field is
considered top-level since it is located within an unnamed tab.

This discrepancy is only evident through the APIs themselves, such as
when creating a request with invalid data and reading the validation
errors in the response. Form state contains proper field paths, which is
ultimately why this issue was never caught. This is because within the
admin panel we merge the API response with the current form state,
obscuring the underlying issue. This becomes especially obvious in
#10580, where we no longer initialize validation errors within form
state until the form has been submitted, and instead rely solely on the
API response for the initial error state.

Here's comprehensive example of how field paths _should_ be formatted:

```
{
  // ...
  fields: [
    {
      // path: 'topLevelNamedField'
      // schemaPath: 'topLevelNamedField'
      // indexPath: ''
      name: 'topLevelNamedField',
      type: 'text',
    },
    {
      // path: 'array'
      // schemaPath: 'array'
      // indexPath: ''
      name: 'array',
      type: 'array',
      fields: [
        {
          // path: 'array.[n].fieldWithinArray'
          // schemaPath: 'array.fieldWithinArray'
          // indexPath: ''
          name: 'fieldWithinArray',
          type: 'text',
        },
        {
          // path: 'array.[n].nestedArray'
          // schemaPath: 'array.nestedArray'
          // indexPath: ''
          name: 'nestedArray',
          type: 'array',
          fields: [
            {
              // path: 'array.[n].nestedArray.[n].fieldWithinNestedArray'
              // schemaPath: 'array.nestedArray.fieldWithinNestedArray'
              // indexPath: ''
              name: 'fieldWithinNestedArray',
              type: 'text',
            },
          ],
        },
        {
          // path: 'array.[n]._index-2'
          // schemaPath: 'array._index-2'
          // indexPath: '2'
          type: 'row',
          fields: [
            {
              // path: 'array.[n].fieldWithinRowWithinArray'
              // schemaPath: 'array._index-2.fieldWithinRowWithinArray'
              // indexPath: ''
              name: 'fieldWithinRowWithinArray',
              type: 'text',
            },
          ],
        },
      ],
    },
    {
      // path: '_index-2'
      // schemaPath: '_index-2'
      // indexPath: '2'
      type: 'row',
      fields: [
        {
          // path: 'fieldWithinRow'
          // schemaPath: '_index-2.fieldWithinRow'
          // indexPath: ''
          name: 'fieldWithinRow',
          type: 'text',
        },
      ],
    },
    {
      // path: '_index-3'
      // schemaPath: '_index-3'
      // indexPath: '3'
      type: 'tabs',
      tabs: [
        {
          // path: '_index-3-0'
          // schemaPath: '_index-3-0'
          // indexPath: '3-0'
          label: 'Unnamed Tab',
          fields: [
            {
              // path: 'fieldWithinUnnamedTab'
              // schemaPath: '_index-3-0.fieldWithinUnnamedTab'
              // indexPath: ''
              name: 'fieldWithinUnnamedTab',
              type: 'text',
            },
            {
              // path: '_index-3-0-1'
              // schemaPath: '_index-3-0-1'
              // indexPath: '3-0-1'
              type: 'tabs',
              tabs: [
                {
                  // path: '_index-3-0-1-0'
                  // schemaPath: '_index-3-0-1-0'
                  // indexPath: '3-0-1-0'
                  label: 'Nested Unnamed Tab',
                  fields: [
                    {
                      // path: 'fieldWithinNestedUnnamedTab'
                      // schemaPath: '_index-3-0-1-0.fieldWithinNestedUnnamedTab'
                      // indexPath: ''
                      name: 'fieldWithinNestedUnnamedTab',
                      type: 'text',
                    },
                  ],
                },
              ],
            },
          ],
        },
        {
          // path: 'namedTab'
          // schemaPath: '_index-3.namedTab'
          // indexPath: ''
          label: 'Named Tab',
          name: 'namedTab',
          fields: [
            {
              // path: 'namedTab.fieldWithinNamedTab'
              // schemaPath: '_index-3.namedTab.fieldWithinNamedTab'
              // indexPath: ''
              name: 'fieldWithinNamedTab',
              type: 'text',
            },
          ],
        },
      ],
    },
  ]
}
```
2025-01-27 14:41:35 -05:00
Patrik
ad553e967b fix: updates field validation error messages to use labels if applicable (#10601)
### What?

Previously, field error messages displayed in toast notifications used
the field path to reference fields that failed validation. This
path-based approach was necessary to distinguish between fields that
might share the same name when nested inside arrays, groups, rows, or
collapsible fields.

However, the human readability of these paths was lacking, especially
for unnamed fields like rows and collapsible fields. For example:

- A text field inside a row could display as: `_index-0.text`
- A text field nested within multiple arrays could display as:
`items.0.subArray.0.text`

These outputs are technically correct but not user-friendly.

### Why?

While the previous format was helpful for pinpointing the specific field
that caused the validation error, it could be more user-friendly and
clearer to read. The goal is to maintain the same level of accuracy
while improving the readability for both developers and content editors.

### How?

To improve readability, the following changes were made:

1. Use Field Labels Instead of Field Paths:
- The ValidationError component now uses the label prop from the field
config (if available) instead of the field’s name.
       - If a label is provided, it will be used in the error message.
       - If no label exists, it will fall back to the field’s name.

2. Remove _index from Paths for Unnamed Fields (In the validationError
component only):
- For unnamed fields like rows and collapsibles, the _index prefix is
now stripped from the output to make it cleaner.
       - Instead of `_index-0.text`, it now outputs just `Text`.

3. Reformat the Error Path for Readability:
- The error message format has been improved to be more human-readable,
showing the field hierarchy in a structured way with array indices
converted to 1-based numbers.

#### Example transformation:

##### Before:
The following fields are invalid: `items.0.subArray.0.text`

##### After:
The following fields are invalid: `Items 1 > SubArray 1 > Text`
2025-01-17 09:42:46 -05:00
Jacob Fletcher
86ff0a434c test: field level validation errors (#10614)
Continuation of #10575. Field level validations error were incorrectly
throwing uniqueness errors. This was fixed but lacking tests.
2025-01-16 15:55:34 -05:00
Sasha
98666eb016 perf(db-postgres): do not push database schema if not changed (#10155)
Based on https://github.com/payloadcms/payload/pull/10154

If the actual database schema is not changed (no new columns, enums,
indexes, tables) - skip calling Drizzle push. This, potentially can
significantly reduce overhead on reloads in development mode especially
when using remote databases.

If for whatever reason you need to preserve the current behavior you can
use `PAYLOAD_FORCE_DRIZZLE_PUSH=true` env flag.
2024-12-27 10:12:01 -05:00
Sasha
ed0d3395c7 test: ensure data strictness (#10123)
Ensures we don't save and read additional properties to the database
with both, Local API and `payload.db`.
2024-12-22 09:37:32 +02:00
Jacob Fletcher
957867f6e2 fix: ensures generated IDs persist on create (#10089)
IDs that are supplied directly through the API, such as client-side
generated IDs when adding new blocks and array rows, are overwritten on
create. This is because when adding blocks or array rows on the client,
their IDs are generated first before being sent to the server for
processing. Then when the server receives this data, it incorrectly
overrides them to ensure they are unique when using relational DBs. But
this only needs to happen when no ID was supplied on create, or
specifically when duplicating documents via the `beforeDuplicate` hook.
2024-12-20 15:14:23 -05:00
Sasha
4e953530df feat(db-sqlite): add autoIncrement option (#9427)
### What?
Exposes ability to enable
[AUTOINCREMENT](https://www.sqlite.org/autoinc.html) for Primary Keys
which ensures that the same ID cannot be reused from previously deleted
rows.

```ts
sqliteAdapter({
  autoIncrement: true
})
```

### Why?
This may be essential for some systems. Enabled `autoIncrement: true`
also for the SQLite Adapter in our tests, which can be useful when
testing whether the doc was deleted or not when you also have other
create operations.

### How?
Uses Drizzle's `autoIncrement` option.

WARNING:
This cannot be enabled in an existing project without a custom
migration, as it completely changes how primary keys are stored in the
database.
2024-12-20 20:13:28 +00:00
Sasha
23f1ed4a48 feat(db-postgres, db-sqlite): drizzle schema generation (#9953)
This PR allows to have full type safety on `payload.drizzle` with a
single command
```sh
pnpm payload generate:db-schema
```
Which generates TypeScript code with Drizzle declarations based on the
current database schema.

Example of generated file with the website template: 
https://gist.github.com/r1tsuu/b8687f211b51d9a3a7e78ba41e8fbf03

Video that shows the power:


https://github.com/user-attachments/assets/3ced958b-ec1d-49f5-9f51-d859d5fae236


We also now proxy drizzle package the same way we do for Lexical so you
don't have to install it (and you shouldn't because you may have version
mismatch).
Instead, you can import from Drizzle like this:
```ts
import {
  pgTable,
  index,
  foreignKey,
  integer,
  text,
  varchar,
  jsonb,
  boolean,
  numeric,
  serial,
  timestamp,
  uniqueIndex,
  pgEnum,
} from '@payloadcms/db-postgres/drizzle/pg-core'
import { sql } from '@payloadcms/db-postgres/drizzle'
import { relations } from '@payloadcms/db-postgres/drizzle/relations'
```


Fixes https://github.com/payloadcms/payload/discussions/4318

In the future we can also support types generation for mongoose / raw
mongodb results.
2024-12-19 11:08:17 -05:00
Sasha
03ff77544e feat(db-sqlite): add idType: 'uuid' support (#10016)
Adds `idType: 'uuid'` to the SQLite adapter support:
```ts
sqliteAdapter({
  idType: 'uuid',
})
```

Achieved through Drizzle's `$defaultFn()`
https://orm.drizzle.team/docs/latest-releases/drizzle-orm-v0283#-added-defaultfn--default-methods-to-column-builders
as SQLite doesn't have native UUID support. Added `sqlite-uuid` to CI.
2024-12-18 22:44:04 -05:00
Sasha
00909ec5c4 fix(db-sqlite): working point field CRUD and default value (#9989)
Previously, the point field with SQLite was incorrectly built to the
schema and not parsed to the result.
Now, it works with SQLite, just doesn't support queries (`near` etc.).
Fixes
https://github.com/payloadcms/payload/pull/9987#discussion_r1885674299
2024-12-16 07:37:47 +02:00
Sasha
5e3963482e fix(db-postgres): payload.db.upsert inserts new rows instead of updating existing ones (#9916)
### What?
`payload.db.updateOne` (and so `payload.db.upsert`) with drizzle
adapters used incoming `where` incorrectly and worked properly only
either if you passed `id` or some where query path required table joins
(like `where: { 'array.title'`) which is also the reason why `upsert`
_worked_ with user preferences specifically, because we need to join the
`preferences_rels` table to query by `user.relationTo` and `user.value`

Fixes https://github.com/payloadcms/payload/issues/9915

This was found here - https://github.com/payloadcms/payload/pull/9913,
the database KV adapter uses `upsert` with `where` by unique fields.
2024-12-12 03:53:57 +02:00
Sasha
e176b8b764 fix: correct migrations sorting in getMigrations (#9330)
### What?
We sorted migrations by `-name` in `getMigrations` as by assumption from
generated file names, however, it may be not true as the improved (+
unflaked, previously it failed sometimes) test for `migrate:down` can
reproduce. As in result, `migrateDown` / `migrateRefresh` may execute in
order different from `migrate`.

Unflakes the 'should commit multiple operations async'  test.
We shouldn't pass the same `req` that doesn't contain a transaction to
different operations that execute in parallel (via `Promise.all`)
without either creating a transaction before or using
`isolateObjectProperty(req, 'transactionID')`. It leads to a race
condition because operation can commit a wrong transaction, different
from inited
2024-11-25 13:53:44 -05:00
James Mikrut
9b00b59df0 fix: corrects cases of false positive identification of custom id fields (#9245)
This PR fixes cases where you may have a field called `id` within a
group or a named tab, which would have incorrectly been treated as a
custom ID field for the collection.

However, custom IDs need to be defined at the root level - and now
Payload only respects custom IDs defined at the root level.
2024-11-16 16:29:48 +00:00
Dan Ribbens
7c6f41936b feat(db-mongodb)!: update mongoose to 8.8.1 (#9115)
### What?
Upgrades mongoose from 6 to latest `v8.8.1`

Fixes https://github.com/payloadcms/payload/issues/9171

### Why?
Compatibilty with Mongodb Atlas

### How?
- Updates deps
- Changed ObjectId from bson-objectid to use `new Type.ObjectId` from
mongoose for compatibility (only inside of db-mongodb)
- Internal type adjustments

https://github.com/payloadcms/payload/discussions/9088

BREAKING CHANGES:
All projects with existing data having versions enabled, or relationship or upload fields will want to create the predefined migration that converts all strings to ObjectIDs where needed. This can be created using `payload migrate:create --file @payloadcms/mongodb/relationships-v2-v3`.
For projects making use of the exposed Models from mongoose, review the
upgrade guides from [v6 to
v7](https://mongoosejs.com/docs/7.x/docs/migrating_to_7.html) and [v7 to
v8](https://mongoosejs.com/docs/migrating_to_8.html) and make
adjustments as needed.

---------

Co-authored-by: Sasha <64744993+r1tsuu@users.noreply.github.com>
2024-11-15 12:03:56 -05:00
Sasha
0a15388edb feat(db-postgres): add point field support (#9078)
### What?
Adds full support for the point field to Postgres and Vercel Postgres
adapters through the Postgis extension. Fully the same API as with
MongoDB, including support for `near`, `within` and `intersects`
operators.

Additionally, exposes to adapter args:
*
`tablesFilter`https://orm.drizzle.team/docs/drizzle-kit-push#including-tables-schemas-and-extensions.
* `extensions` list of extensions to create, for example `['vector',
'pg_search']`, `postgis` is created automatically if there's any point
field

### Why?
It's essential to support that field type, especially if the postgres
adapter should be out of beta on 3.0 stable.

### How?
* Bumps `drizzle-orm` to `0.36.1` and `drizzle-kit` to `0.28.0` as we
need this change https://github.com/drizzle-team/drizzle-orm/pull/3141
* Uses its functions to achieve querying functionality, for example the
`near` operator works through `ST_DWithin` or `intersects` through
`ST_Intersects`.
* Removes MongoDB condition from all point field tests, but keeps for
SQLite

Resolves these discussions:
https://github.com/payloadcms/payload/discussions/8996
https://github.com/payloadcms/payload/discussions/8644
2024-11-11 09:31:47 -05:00
Dan Ribbens
e3172f1e39 fix: migrateRefresh migrates without previously ran migrations (#9073)
fix: migrateRefresh migrates without previously ran migrations
chore: adds tests for database migrate:fresh and migrate:refresh

---------

Co-authored-by: Sasha <64744993+r1tsuu@users.noreply.github.com>
2024-11-08 15:35:46 -05:00
Dan Ribbens
ee117bb616 fix!: handle custom id logic in mongodb adapter (#9069)
### What?
Moved the logic for copying the data.id to data._id to the mongoose
adapter.

### Why?
If you have any hooks that need to set the `id`, the value does not get
sent to mongodb as you would expect since it was copied before the
beforeValidate hooks.

### How?
Now data._id is assigned only in the mongodb adapter's `create`
function.

BREAKING CHANGES:
When using custom ID fields, if you have any collection hooks for
beforeValidate, beforeChange then `data._id` will no longer be assigned
as this happens now in the database adapter. Use `data.id` instead.
2024-11-08 15:34:19 -05:00
Dan Ribbens
d20445b6f3 fix(db-mongodb): write migrations index file (#9071)
fix: remove 'undefined' written into mongodb migrations
fix: write migrations index file
2024-11-08 13:26:45 -05:00
Sasha
9056b9fe9b fix(db-mongodb): virtual fields within row / collapsible / tabs (#8733)
Fixes https://github.com/payloadcms/payload/issues/8674
2024-10-17 16:23:45 -04:00
Dan Ribbens
3110c1b01b fix: local update limit (#8704) 2024-10-14 19:05:13 -04:00
Dan Ribbens
d781624a86 feat: add disableTransaction to local api (#8697) 2024-10-14 14:39:20 -04:00
Patrik
21606ded08 fix(db-mongodb): add validation to relationship ids (#8395)
fixes https://github.com/payloadcms/payload/issues/8652
2024-10-11 13:49:07 -04:00
Sasha
8acbda078e feat(drizzle): customize schema with before / after init hooks (#8196)
Adds abillity to customize the generated Drizzle schema with
`beforeSchemaInit` and `afterSchemaInit`. Could be useful if you want to
preserve the existing database schema / override the generated one with
features that aren't supported from the Payload config.

## Docs:

### beforeSchemaInit

Runs before the schema is built. You can use this hook to extend your
database structure with tables that won't be managed by Payload.

```ts
import { postgresAdapter } from '@payloadcms/db-postgres'
import { integer, pgTable, serial } from 'drizzle-orm/pg-core'

postgresAdapter({
  beforeSchemaInit: [
    ({ schema, adapter }) => {
      return {
        ...schema,
        tables: {
          ...schema.tables,
          addedTable: pgTable('added_table', {
            id: serial('id').notNull(),
          }),
        },
      }
    },
  ],
})
```

One use case is preserving your existing database structure when
migrating to Payload. By default, Payload drops the current database
schema, which may not be desirable in this scenario.
To quickly generate the Drizzle schema from your database you can use
[Drizzle
Introspection](https://orm.drizzle.team/kit-docs/commands#introspect--pull)
You should get the `schema.ts` file which may look like this:

```ts
import { pgTable, uniqueIndex, serial, varchar, text } from 'drizzle-orm/pg-core'

export const users = pgTable('users', {
  id: serial('id').primaryKey(),
  fullName: text('full_name'),
  phone: varchar('phone', { length: 256 }),
})

export const countries = pgTable(
  'countries',
  {
    id: serial('id').primaryKey(),
    name: varchar('name', { length: 256 }),
  },
  (countries) => {
    return {
      nameIndex: uniqueIndex('name_idx').on(countries.name),
    }
  },
)

```

You can import them into your config and append to the schema with the
`beforeSchemaInit` hook like this:

```ts
import { postgresAdapter } from '@payloadcms/db-postgres'
import { users, countries } from '../drizzle/schema'

postgresAdapter({
  beforeSchemaInit: [
    ({ schema, adapter }) => {
      return {
        ...schema,
        tables: {
          ...schema.tables,
          users,
          countries
        },
      }
    },
  ],
})
```

Make sure Payload doesn't overlap table names with its collections. For
example, if you already have a collection with slug "users", you should
either change the slug or `dbName` to change the table name for this
collection.


### afterSchemaInit

Runs after the Drizzle schema is built. You can use this hook to modify
the schema with features that aren't supported by Payload, or if you
want to add a column that you don't want to be in the Payload config.
To extend a table, Payload exposes `extendTable` utillity to the args.
You can refer to the [Drizzle
documentation](https://orm.drizzle.team/docs/sql-schema-declaration).
The following example adds the `extra_integer_column` column and a
composite index on `country` and `city` columns.

```ts
import { postgresAdapter } from '@payloadcms/db-postgres'
import { index, integer } from 'drizzle-orm/pg-core'
import { buildConfig } from 'payload'

export default buildConfig({
  collections: [
    {
      slug: 'places',
      fields: [
        {
          name: 'country',
          type: 'text',
        },
        {
          name: 'city',
          type: 'text',
        },
      ],
    },
  ],
  db: postgresAdapter({
    afterSchemaInit: [
      ({ schema, extendTable, adapter }) => {
        extendTable({
          table: schema.tables.places,
          columns: {
            extraIntegerColumn: integer('extra_integer_column'),
          },
          extraConfig: (table) => ({
            country_city_composite_index: index('country_city_composite_index').on(
              table.country,
              table.city,
            ),
          }),
        })

        return schema
      },
    ],
  }),
})

```



<!--

For external contributors, please include:

- A summary of the pull request and any related issues it fixes.
- Reasoning for the changes made or any additional context that may be
useful.

Ensure you have read and understand the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository.

 -->
2024-09-25 15:14:03 -04:00
Sasha
bf48af411d feat: add virtual property to the fields config (#7621)
## Description

Adds `virtual` property to the fields config. Providing `true`
completely disables the field in the DB, which is useful for [Virtual
Fields](https://payloadcms.com/blog/learn-how-virtual-fields-can-help-solve-common-cms-challenges)
Disables abillity to query by a field with `virtual: true`.
Currently, they bloat the DB with unused tables / columns, which may as
well introduce additional joins.
Discussion https://github.com/payloadcms/payload/discussions/6270
Prev PR (this one contains only this feature):
https://github.com/payloadcms/payload/pull/6983

- [x] I have read and understand the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository.

## Type of change

<!-- Please delete options that are not relevant. -->

- [x] New feature (non-breaking change which adds functionality)
- [x] This change requires a documentation update

## Checklist:

- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] Existing test suite passes locally with my changes
- [x] I have made corresponding changes to the documentation
2024-09-17 10:40:54 -04:00
Dan Ribbens
bb09da08c2 fix: migrate error on windows (#7759)
handle windows compatible file names when reading migrations

---------

Co-authored-by: Alessio Gravili <alessio@gravili.de>
2024-08-19 15:46:40 -04:00
Alessio Gravili
90b7b20699 feat!: beta-next (#7620)
This PR makes three major changes to the codebase:

1. [Component Paths](#component-paths)
Instead of importing custom components into your config directly, they
are now defined as file paths and rendered only when needed. That way
the Payload config will be significantly more lightweight, and ensures
that the Payload config is 100% server-only and Node-safe. Related
discussion: https://github.com/payloadcms/payload/discussions/6938

2. [Client Config](#client-config)
Deprecates the component map by merging its logic into the client
config. The main goal of this change is for performance and
simplification. There was no need to deeply iterate over the Payload
config twice, once for the component map, and another for the client
config. Instead, we can do everything in the client config one time.
This has also dramatically simplified the client side prop drilling
through the UI library. Now, all components can share the same client
config which matches the exact shape of their Payload config (with the
exception of non-serializable props and mapped custom components).

3. [Custom client component are no longer
server-rendered](#custom-client-components-are-no-longer-server-rendered)
Previously, custom components would be server-rendered, no matter if
they are server or client components. Now, only server components are
rendered on the server. Client components are automatically detected,
and simply get passed through as `MappedComponent` to be rendered fully
client-side.

## Component Paths

Instead of importing custom components into your config directly, they
are now defined as file paths and rendered only when needed. That way
the Payload config will be significantly more lightweight, and ensures
that the Payload config is 100% server-only and Node-safe. Related
discussion: https://github.com/payloadcms/payload/discussions/6938

In order to reference any custom components in the Payload config, you
now have to specify a string path to the component instead of importing
it.

Old:

```ts
import { MyComponent2} from './MyComponent2.js'

admin: {
  components: {
    Label: MyComponent2
  },
},
```

New:

```ts
admin: {
  components: {
    Label: '/collections/Posts/MyComponent2.js#MyComponent2', // <= has to be a relative path based on a baseDir configured in the Payload config - NOT relative based on the importing file
  },
},
```

### Local API within Next.js routes

Previously, if you used the Payload Local API within Next.js pages, all
the client-side modules are being added to the bundle for that specific
page, even if you only need server-side functionality.

This `/test` route, which uses the Payload local API, was previously 460
kb. It is now down to 91 kb and does not bundle the Payload client-side
admin panel anymore.

All tests done
[here](https://github.com/payloadcms/payload-3.0-demo/tree/feat/path-test)
with beta.67/PR, db-mongodb and default richtext-lexical:

**dev /admin before:**
![CleanShot 2024-07-29 at 22 49
12@2x](https://github.com/user-attachments/assets/4428e766-b368-4bcf-8c18-d0187ab64f3e)

**dev /admin after:**
![CleanShot 2024-07-29 at 22 50
49@2x](https://github.com/user-attachments/assets/f494c848-7247-4b02-a650-a3fab4000de6)

---

**dev /test before:**
![CleanShot 2024-07-29 at 22 56
18@2x](https://github.com/user-attachments/assets/1a7e9500-b859-4761-bf63-abbcdac6f8d6)

**dev /test after:**
![CleanShot 2024-07-29 at 22 47
45@2x](https://github.com/user-attachments/assets/f89aa76d-f2d5-4572-9753-2267f034a45a)

---

**build before:**
![CleanShot 2024-07-29 at 22 57
14@2x](https://github.com/user-attachments/assets/5f8f7281-2a4a-40a5-a788-c30ddcdd51b5)

**build after::**
![CleanShot 2024-07-29 at 22 56
39@2x](https://github.com/user-attachments/assets/ea8772fd-512f-4db0-9a81-4b014715a1b7)

### Usage of the Payload Local API / config outside of Next.js

This will make it a lot easier to use the Payload config / local API in
other, server-side contexts. Previously, you might encounter errors due
to client files (like .scss files) not being allowed to be imported.

## Client Config

Deprecates the component map by merging its logic into the client
config. The main goal of this change is for performance and
simplification. There was no need to deeply iterate over the Payload
config twice, once for the component map, and another for the client
config. Instead, we can do everything in the client config one time.
This has also dramatically simplified the client side prop drilling
through the UI library. Now, all components can share the same client
config which matches the exact shape of their Payload config (with the
exception of non-serializable props and mapped custom components).

This is breaking change. The `useComponentMap` hook no longer exists,
and most component props have changed (for the better):

```ts
const { componentMap } = useComponentMap() // old
const { config } = useConfig() // new
```

The `useConfig` hook has also changed in shape, `config` is now a
property _within_ the context obj:

```ts
const config = useConfig() // old
const { config } = useConfig() // new
```

## Custom Client Components are no longer server rendered

Previously, custom components would be server-rendered, no matter if
they are server or client components. Now, only server components are
rendered on the server. Client components are automatically detected,
and simply get passed through as `MappedComponent` to be rendered fully
client-side.

The benefit of this change:

Custom client components can now receive props. Previously, the only way
for them to receive dynamic props from a parent client component was to
use hooks, e.g. `useFieldProps()`. Now, we do have the option of passing
in props to the custom components directly, if they are client
components. This will be simpler than having to look for the correct
hook.

This makes rendering them on the client a little bit more complex, as
you now have to check if that component is a server component (=>
already has been rendered) or a client component (=> not rendered yet,
has to be rendered here). However, this added complexity has been
alleviated through the easy-to-use `<RenderMappedComponent />` helper.

This helper now also handles rendering arrays of custom components (e.g.
beforeList, beforeLogin ...), which actually makes rendering custom
components easier in some cases.

## Misc improvements

This PR includes misc, breaking changes. For example, we previously
allowed unions between components and config object for the same
property. E.g. for the custom view property, you were allowed to pass in
a custom component or an object with other properties, alongside a
custom component.

Those union types are now gone. You can now either pass an object, or a
component. The previous `{ View: MyViewComponent}` is now `{ View: {
Component: MyViewComponent} }` or `{ View: { Default: { Component:
MyViewComponent} } }`.

This dramatically simplifies the way we read & process those properties,
especially in buildComponentMap. We can now simply check for the
existence of one specific property, which always has to be a component,
instead of running cursed runtime checks on a shared union property
which could contain a component, but could also contain functions or
objects.

![CleanShot 2024-07-29 at 23 07
07@2x](https://github.com/user-attachments/assets/1e75aa4c-7a4c-419f-9070-216bb7b9a5e5)

![CleanShot 2024-07-29 at 23 09
40@2x](https://github.com/user-attachments/assets/b4c96450-6b7e-496c-a4f7-59126bfd0991)

- [x] I have read and understand the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository.

---------

Co-authored-by: PatrikKozak <patrik@payloadcms.com>
Co-authored-by: Paul <paul@payloadcms.com>
Co-authored-by: Paul Popus <paul@nouance.io>
Co-authored-by: Jacob Fletcher <jacobsfletch@gmail.com>
Co-authored-by: James <james@trbl.design>
2024-08-13 12:54:33 -04:00
Dan Ribbens
075819964d fix(db-postgres, db-sqlite): enum schema (#7453)
- updates drizzle-kit and drizzle-orm
- fix enum creation to fully support custom schemas
- sqlite by default will not use transactions
2024-07-31 16:42:00 -04:00