Commit Graph

221 Commits

Author SHA1 Message Date
Sasha
bacc0f002a feat: compound indexes (#11512)
### What?
This PR adds ability to define indexes on several fields for collections
(compound indexes).

Example:
```ts
{
  indexes: [{ unique: true, fields: ['title', 'group.name'] }]
}
```

### Why?
This can be used to either speed up querying/sorting by 2 or more fields
at the same time or to ensure uniqueness between several fields.

### How?
Implements this logic in database adapters. Additionally, adds a utility
`getFieldByPath`.
2025-03-05 03:09:24 +02:00
Alessio Gravili
6a3d58bb32 feat(db-*): support limit in db.updateMany (#11488)
This PR adds a new `limit` property to `payload.db.updateMany`. This functionality is required for [migrating our job system to use faster, direct db adapter calls](https://github.com/payloadcms/payload/pull/11489)
2025-03-03 05:32:57 +00:00
Sasha
79a7b4ad02 chore(db-mongodb): tsconfig uses strict: true and noUncheckedIndexedAccess: true (#11444)
Migrates the `db-mongodb` package to use `strict: true` and
`noUncheckedIndexedAccess: true` TSConfig properties.
This greatly improves code quality and prevents some runtime errors or
gives better error messages.
2025-03-01 00:17:24 +02:00
Sasha
d4d2bf4617 perf(db-mongodb): faster join field aggregation by replacing mongoose-aggregate-paginate-v2 with a custom implementation (#10936)
Fixes
https://github.com/payloadcms/payload/discussions/10165#discussioncomment-12034047

As described in the discussion, we have an incorrect order of
aggregation pipeline when using aggregations with the join field. We
must use `$sort`, `$skip`, `$limit` before the `$lookup` or otherwise
mongodb scans all the docs, applies `$lookup` for them and only after
applies `$limit`, `$skip`.
Replaces `mongoose-aggregate-paginate-v2` with a custom
`aggregatePaginate` because we need a custom solution here. This was
considered in https://github.com/payloadcms/payload/pull/9594 but it was
reverted as for now.

Fixes https://github.com/payloadcms/payload/issues/11187
2025-02-28 21:30:00 +02:00
Alessio Gravili
41c7413f59 feat(db-*): add updateMany method to database adapter (#11441)
This PR adds a new `payload.db.updateMany` method, which is a more performant way to update multiple documents compared to using `payload.update`.
2025-02-27 20:30:17 -07:00
Alessio Gravili
c21dac1b53 perf(db-*): add option to disable returning modified documents in db methods (#11393)
This PR adds a new `returning` option to various db adapter methods. Setting it to `false` where the return value is not used will lead to performance gains, as we don't have to do additional db calls to fetch the updated document and then sanitize it.
2025-02-27 17:40:22 -05:00
Sasha
3436fb16ea feat: allow to count related docs for join fields (#11395)
### What?
For the join field query adds ability to specify `count: true`, example:
```ts
const result = await payload.find({
  joins: {
    'group.relatedPosts': {
      sort: '-title',
      count: true,
    },
  },
  collection: "categories",
})

result.group?.relatedPosts?.totalDocs // available
```

### Why?
Can be useful to implement full pagination / show total related
documents count in the UI.

### How?
Implements the logic in database adapters. In MongoDB it's additional
`$lookup` that has `$count` in the pipeline. In SQL, it's additional
subquery with `COUNT(*)`. Preserves the current behavior by default,
since counting introduces overhead.


Additionally, fixes a typescript generation error for join fields.
Before, `docs` and `hasNextPage` were marked as nullable, which is not
true, these fields cannot be `null`.
Additionally, fixes threading of `joinQuery` in
`transform/read/traverseFields` for group / tab fields recursive calls.
2025-02-27 16:05:48 +00:00
Sasha
6b6c289d79 fix(db-mongodb): hasNextPage with polymorphic joins (#11394)
Previously, `hasNextPage` was working incorrectly with polymorphic joins
(that have an array of `collection`) in MongoDB.

This PR fixes it and adds extra assertions to the polymorphic joins
test.

---------

Co-authored-by: Jarrod Flesch <jarrodmflesch@gmail.com>
2025-02-25 21:22:47 +00:00
Paul
b014416584 feat: add support for not_like operation (#11326)
This PR adds support for `not_like` as a query operation. Functions just
as a negative to `like`, uses `ilike` in postgres and `like` in sqlite
2025-02-22 00:32:37 +00:00
Sasha
1dc748d341 perf(db-mongodb): remove JSON.parse(JSON.stringify) copying of results (#11293)
Improves performance and optimizes memory usage for mongodb adapter by
cutting down copying of results via `JSON.parse(JSON.stringify())`.
Instead, `transform` does necessary transformations (`ObjectID` ->
`string,` `Date` -> `string`) without any copying
2025-02-21 17:31:24 +02:00
Alessio Gravili
7922d66181 fix(db-mongodb): properly handle document notfound cases for update and delete operations (#11267)
In the `findOne` db operation, we return `null` if the document was not
found.

For single-document delete and update operations, if the document you
wanted to update is not found, the following runtime error is thrown
instead: `Cannot read properties of null (reading '_id')`.

This PR correctly handles these cases and returns `null` from the db
method, just like the `findOne` operation.
2025-02-19 01:19:59 +00:00
Sasha
6d36a28cdc feat: join field across many collections (#10919)
This feature allows you to specify `collection` for the join field as
array.
This can be useful for example to describe relationship linking like
this:
```ts
{
  slug: 'folders',
  fields: [
    {
      type: 'join',
      on: 'folder',
      collection: ['files', 'documents', 'folders'],
      name: 'children',
    },
    {
      type: 'relationship',
      relationTo: 'folders',
      name: 'folder',
    },
  ],
},
{
  slug: 'files',
  upload: true,
  fields: [
    {
      type: 'relationship',
      relationTo: 'folders',
      name: 'folder',
    },
  ],
},
{
  slug: 'documents',
  fields: [
    {
      type: 'relationship',
      relationTo: 'folders',
      name: 'folder',
    },
  ],
},
```

Documents and files can be placed to folders and folders themselves can
be nested to other folders (root folders just have `folder` as `null`).

Output type of `Folder`:
```ts
export interface Folder {
  id: string;
  children?: {
    docs?:
      | (
          | {
              relationTo?: 'files';
              value: string | File;
            }
          | {
              relationTo?: 'documents';
              value: string | Document;
            }
          | {
              relationTo?: 'folders';
              value: string | Folder;
            }
        )[]
      | null;
    hasNextPage?: boolean | null;
  } | null;
  folder?: (string | null) | Folder;
  updatedAt: string;
  createdAt: string;
}
```

While you could instead have many join fields (for example
`childrenFolders`, `childrenFiles`) etc - this doesn't allow you to
sort/filter and paginate things across many collections, which isn't
trivial. With SQL we use `UNION ALL` query to achieve that.

---------

Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
2025-02-18 21:53:45 +02:00
Sasha
847d8d824f feat: add page query parameter for joins (#10998)
Currently, the join field outputs to its result `hasNextPage: boolean`
and have the `limit` query parameter but lacks `page` which can be
useful. This PR adds it.
2025-02-18 16:47:09 +02:00
Alessio Gravili
e6fea1d132 fix: localized fields within block references were not handled properly if any parent is localized (#11207)
The `localized` properly was not stripped out of referenced block fields, if any parent was localized. For normal fields, this is done in sanitizeConfig. As the same referenced block config can be used in both a localized and non-localized config, we are not able to strip it out inside sanitizeConfig by modifying the block config.

Instead, this PR had to bring back tedious logic to handle it everywhere the `field.localized` property is accessed. For backwards-compatibility, we need to keep the existing sanitizeConfig logic. In 4.0, we should remove it to benefit from better test coverage of runtime field.localized handling - for now, this is done for our test suite using the `PAYLOAD_DO_NOT_SANITIZE_LOCALIZED_PROPERTY` flag.
2025-02-17 19:50:32 +00:00
Sasha
2ae670e0e4 test(db-mongodb): unit test assertion for relationship sanitization inside blockReferences (#11195)
This PR adds a new unit test assertion to existing
https://github.com/payloadcms/payload/blob/main/packages/db-mongodb/src/utilities/sanitizeRelationshipIDs.spec.ts
that ensures relationships are sanitized to `ObjectID`s correctly when
saved to the database for relationships inside the new `blockReferences`
https://github.com/payloadcms/payload/pull/10905
2025-02-16 14:07:58 +02:00
Said Akhrarov
12f51bad5f fix(db-mongodb): remove duplicative indexing of timestamps (#11028)
### What?
This PR removes a pair unnecessary calls to `schema.index` against the
timestamp fields. The issue is when a user sets `indexSortableFields` as
this is what will ultimately pass the predicate which then creates
duplicate indexes.

### Why?
These calls are redundant as `index` is [already
passed](https://github.com/payloadcms/payload/blob/main/packages/db-mongodb/src/models/buildSchema.ts#L69)
to the underlying fields base schema options in the process of
formatting and will already be indexed.

These warnings were surfaced after the bump to mongoose to version 8.9.5
as [in 8.9.3 mongoose began throwing these warnings to indicate
duplicative
indexes](https://github.com/Automattic/mongoose/releases/tag/8.9.3).

### How?
By removing these calls and, as a result, silencing the warnings thrown
by mongoose.
2025-02-13 23:07:24 -05:00
Alessio Gravili
4c8cafd6a6 perf: deduplicate blocks used in multiple places using new config.blocks property (#10905)
If you have multiple blocks that are used in multiple places, this can quickly blow up the size of your Payload Config. This will incur a performance hit, as more data is
1.  sent to the client (=> bloated `ClientConfig` and large initial html) and
2. processed on the server (permissions are calculated every single time you navigate to a page - this iterates through all blocks you have defined, even if they're duplicative)

This can be optimized by defining your block **once** in your Payload Config, and just referencing the block slug whenever it's used, instead of passing the entire block config. To do this, the block can be defined in the `blocks` array of the Payload Config. The slug can then be passed to the `blockReferences` array in the Blocks Field - the `blocks` array has to be empty for compatibility reasons.

```ts
import { buildConfig } from 'payload'
import { lexicalEditor, BlocksFeature } from '@payloadcms/richtext-lexical'

// Payload Config
const config = buildConfig({
  // Define the block once
  blocks: [
    {
      slug: 'TextBlock',
      fields: [
        {
          name: 'text',
          type: 'text',
        },
      ],
    },
  ],
  collections: [
    {
      slug: 'collection1',
      fields: [
        {
          name: 'content',
          type: 'blocks',
          // Reference the block by slug
          blockReferences: ['TextBlock'],
          blocks: [], // Required to be empty, for compatibility reasons
        },
      ],
    },
     {
      slug: 'collection2',
      fields: [
        {
          name: 'editor',
          type: 'richText',
          editor: lexicalEditor({
            BlocksFeature({
              // Same reference can be reused anywhere, even in the lexical editor, without incurred performance hit
              blocks: ['TextBlock'],
            })
          })
        },
      ],
    },
  ],
})
```

## v4.0 Plans

In 4.0, we will remove the `blockReferences` property, and allow string block references to be passed directly to the blocks `property`. Essentially, we'd remove the `blocks` property and rename `blockReferences` to `blocks`.

The reason we opted to a new property in this PR is to avoid breaking changes. Allowing strings to be passed to the `blocks` property will prevent plugins that iterate through fields / blocks from compiling.

## PR Changes

- Testing: This PR introduces a plugin that automatically converts blocks to block references. This is done in the fields__blocks test suite, to run our existing test suite using block references.

- Block References support: Most changes are similar. Everywhere we iterate through blocks, we have to now do the following:
1. Check if `field.blockReferences` is provided. If so, only iterate through that.
2. Check if the block is an object (= actual block), or string
3. If it's a string, pull the actual block from the Payload Config or from `payload.blocks`.

The exception is config sanitization and block type generations. This PR optimizes them so that each block is only handled once, instead of every time the block is referenced.

## Benchmarks

60 Block fields, each block field having the same 600 Blocks.

### Before:
**Initial HTML:** 195 kB
**Generated types:** takes 11 minutes, 461,209 lines

https://github.com/user-attachments/assets/11d49a4e-5414-4579-8050-e6346e552f56

### After:
**Initial HTML:** 73.6 kB
**Generated types:** takes 2 seconds, 35,810 lines

https://github.com/user-attachments/assets/3eab1a99-6c29-489d-add5-698df67780a3

### After Permissions Optimization (follow-up PR)
Initial HTML: 73.6 kB

https://github.com/user-attachments/assets/a909202e-45a8-4bf6-9a38-8c85813f1312


## Future Plans

1. This PR does not yet deduplicate block references during permissions calculation. We'll optimize that in a separate PR, as this one is already large enough
2. The same optimization can be done to deduplicate fields. One common use-case would be link field groups that may be referenced in multiple entities, outside of blocks. We might explore adding a new `fieldReferences` property, that allows you to reference those same `config.blocks`.
2025-02-14 00:08:20 +00:00
Jarrod Flesch
d2fe9b0807 fix(db-mongodb): ensures same level operators are respected (#11087)
### What?
If you had multiple operator constraints on a single field, the last one
defined would be the only one used.

Example:
```ts
where: {
  id: {
    in: [doc2.id],
    not_in: [], // <-- only respected this operator constraint
  },
}
```

and
```ts
where: {
  id: {
    not_in: [],
    in: [doc2.id], // <-- only respected this operator constraint
  },
}
```

They would yield different results.

### Why?
The results were not merged into an `$and` query inside parseParams.

### How?
Merges the results within an `$and` constraint.

Fixes https://github.com/payloadcms/payload/issues/10944

Supersedes https://github.com/payloadcms/payload/pull/11011
2025-02-10 16:29:08 -05:00
Sasha
2d8ff7218a fix(db-mongodb): v2-v3 migration versions docs of collections and globals without relationship fields (#10755)
Fixes https://github.com/payloadcms/payload/issues/10753
2025-01-23 12:33:43 -05:00
Sasha
d601300034 fix(db-mongodb): querying polymorphic relationships with the all operator (#10704)
Fixes https://github.com/payloadcms/payload/issues/10678
2025-01-23 18:23:50 +02:00
Dan Ribbens
90f88f8fc0 fix(db-mongodb): beginTransaction invalid type without replicaset (#10690)
Fixes #10603
2025-01-21 03:09:52 +00:00
Sasha
25a70ab455 fix: join field with the target relationship inside localized array (#10621)
Fixes https://github.com/payloadcms/payload/issues/10356
2025-01-21 02:18:26 +02:00
Sasha
46c1b375b8 fix: properly handle nullable minDistance and maxDistance in near query (#10622)
Fixes https://github.com/payloadcms/payload/issues/10521
2025-01-21 02:18:13 +02:00
Jacob Fletcher
86ff0a434c test: field level validation errors (#10614)
Continuation of #10575. Field level validations error were incorrectly
throwing uniqueness errors. This was fixed but lacking tests.
2025-01-16 15:55:34 -05:00
Alessio Gravili
42382b6f6f perf: operations performance optimizations (#10609)
- reduces unnecessary shallow copying within operations by removing
unnecessary spreads or .map()'s
- removes unnecessary `deleteMany` call in `deleteUserPreferences` for
auth-enabled collections
- replaces all instances of `validOperators.includes` with
`validOperatorMap[]`. O(n) => O(1)
- optimizes the `sanitizeInternalFields` function. Previously, it was
doing a **lot** of shallow copying
2025-01-16 17:07:35 +00:00
Sasha
9043b10792 fix(db-mongodb): incorrect errors logging due to invalid logic in handleError (#10575)
Previously, every error from MongoDB was logged as "Value must be
unique", as well the response code should not be `BAD_REQUEST` but
`INTERNAL_SERVER_ERROR`. `throw error` preserves the original error so
it can be traced.
2025-01-15 11:02:09 +02:00
Sasha
b08ff88fbd fix(db-mongodb): mongodb optimizations (#10120)
There are few issues introduced in #9594 that we need to look into with
these changes.
2024-12-21 07:42:44 -05:00
Sasha
a7109ed048 refactor: consistent caps for acronyms in variable names (#10101)
Improves consistency for variable names like `docID`, `insertedID`,
`parentID`
For example: `docId` -> `docID`
2024-12-20 18:10:34 +00:00
Dan Ribbens
d03658de01 feat: join field with polymorphic relationships (#9990)
### What?
The join field had a limitation imposed that prevents it from targeting
polymorphic relationship fields. With this change we can support any
relationship fields.

### Why?
Improves the functionality of join field.

### How?
Extended the database adapters and removed the config sanitization that
would throw an error when polymorphic relationships were used.

Fixes #
2024-12-19 22:34:52 +00:00
Sasha
e468292039 perf(db-mongodb): improve performance of all operations, up to 50% faster (#9594)
This PR improves speed and memory efficiency across all operations with
the Mongoose adapter.

### How?

- Removes Mongoose layer from all database calls, instead uses MongoDB
directly. (this doesn't remove building mongoose schema since it's still
needed for indexes + users in theory can use it)
- Replaces deep copying of read results using
`JSON.parse(JSON.stringify(data))` with the `transform` `operation:
'read'` function which converts Date's, ObjectID's in relationships /
joins to strings. As before, it also handles transformations for write
operations.
- Faster `hasNearConstraint` for potentially large `where`'s
- `traverseFields` now can accept `flattenedFields` which we use in
`transform`. Less recursive calls with tabs/rows/collapsible

Additional fixes
- Uses current transaction for querying nested relationships properties
in `buildQuery`, previously it wasn't used which could've led to wrong
results
- Allows to clear not required point fields with passing `null` from the
Local API. Previously it didn't work in both, MongoDB and Postgres

Benchmarks using this file
https://github.com/payloadcms/payload/blob/chore/db-benchmark/test/_community/int.spec.ts

### Small Dataset Performance

| Metric | Before Optimization | After Optimization | Improvement (%) |

|---------------------------|---------------------|--------------------|-----------------|
| Average FULL (ms) | 1170 | 844 | 27.86% |
| `payload.db.create` (ms) | 1413 | 691 | 51.12% |
| `payload.db.find` (ms) | 2856 | 2204 | 22.83% |
| `payload.db.deleteMany` (ms) | 15206 | 8439 | 44.53% |
| `payload.db.updateOne` (ms) | 21444 | 12162 | 43.30% |
| `payload.db.findOne` (ms) | 159 | 112 | 29.56% |
| `payload.db.deleteOne` (ms) | 3729 | 2578 | 30.89% |
| DB small FULL (ms) | 64473 | 46451 | 27.93% |

---

### Medium Dataset Performance

| Metric | Before Optimization | After Optimization | Improvement (%) |

|---------------------------|---------------------|--------------------|-----------------|
| Average FULL (ms) | 9407 | 6210 | 33.99% |
| `payload.db.create` (ms) | 10270 | 4321 | 57.93% |
| `payload.db.find` (ms) | 20814 | 16036 | 22.93% |
| `payload.db.deleteMany` (ms) | 126351 | 61789 | 51.11% |
| `payload.db.updateOne` (ms) | 201782 | 99943 | 50.49% |
| `payload.db.findOne` (ms) | 1081 | 817 | 24.43% |
| `payload.db.deleteOne` (ms) | 28534 | 23363 | 18.12% |
| DB medium FULL (ms) | 519518 | 342194 | 34.13% |

---

### Large Dataset Performance

| Metric | Before Optimization | After Optimization | Improvement (%) |

|---------------------------|---------------------|--------------------|-----------------|
| Average FULL (ms) | 26575 | 17509 | 34.14% |
| `payload.db.create` (ms) | 29085 | 12196 | 58.08% |
| `payload.db.find` (ms) | 58497 | 43838 | 25.04% |
| `payload.db.deleteMany` (ms) | 372195 | 173218 | 53.47% |
| `payload.db.updateOne` (ms) | 544089 | 288350 | 47.00% |
| `payload.db.findOne` (ms) | 3058 | 2197 | 28.14% |
| `payload.db.deleteOne` (ms) | 82444 | 64730 | 21.49% |
| DB large FULL (ms) | 1461097 | 969714 | 33.62% |
2024-12-19 13:20:39 -05:00
Sasha
5753efb0a4 fix(db-mongodb): querying by localized polymorphic relationships using objects (#10037)
Previously, queries like this didn't work:
```ts
const res = await payload.find({
  collection: 'polymorphic-relationships',
  where: {
    polymorphicLocalized: {
      equals: {
        relationTo: 'movies',
        value: movie.id,
      },
    },
  },
})
```

This was due to the incorrectly passed path to MongoDB without
`.{locale}` suffix.
Additionally, to MongoDB now we send:
```

{
  $or: [
    {
      polymorphic: {
        $eq: {
          relationTo: formattedValue.relationTo,
          value: formattedValue.value,
        },
      },
    },
    {
      polymorphic: {
        $eq: {
          relationTo: 'movies',
          value: 'some-id',
        },
      },
    },
  ],
},
```

Instead of:
```
{
  $and: [
    {
      'polymorphic.relationTo': {
        $eq: 'movies ',
      },
    },
    {
      'polymorphic.value': {
        $eq: 'some-id ',
      },
    },
  ],
}
```

To match the _exact_ value. This is essential when we do querying by
relationships with `hasMany: true` and custom IDs that can be repeated.
`$or` is needed if for some reason keys are stored in the DB in a
different order
2024-12-19 10:42:15 -05:00
Sasha
0c57eef621 fix: unique error message regression (#10064)
Regression from https://github.com/payloadcms/payload/pull/9935, we need
to call `req.t` with `''error:valueMustBeUnique''` instead of just
`''error:valueMustBeUnique''`
2024-12-19 08:13:44 +00:00
Sasha
0e5bda9a74 feat: make req partial and optional in DB / Local API operations (#9935)
### What?
Previously, the `req` argument:
In database operations (e.g `payload.db`) was required and you needed to
pass the whole `req` with all the properties. This is confusing because
in database operations we never use its properties outside of
`req.transactionID` and `req.t`, both of which should be optional as
well.

Now, you don't have to do that cast:
```ts
payload.db.findOne({
  collection: 'posts',
  req: {} as PayloadRequest,
  where: {
    id: {
      equals: 1,
    },
  },
})
```
Becomes:
```ts
payload.db.findOne({
  collection: 'posts',
  where: {
    id: {
      equals: 1,
    },
  },
})
```

If you need to use transactions, you're not required to do the `as` cast
as well now, as the `req` not only optional but also partial -
`Partial<PayloadRequest>`.
`initTransaction`, `commitTransaction`, `killTransaction` utilities are
typed better now as well. They do not require to you pass all the
properties of `req`, but only `payload` -
`MarkRequired<Partial<PayloadRequest>, 'payload'>`
```ts
const req = { payload }
await initTransaction(req)
await payload.db.create({
  collection: "posts",
  data: {},
  req
})
await commitTransaction(req)
```

The same for the Local API. Internal operations (for example
`packages/payload/src/collections/operations/find.ts`) still accept the
whole `req`, but local ones
(`packages/payload/src/collections/operations/local/find.ts`) which are
used through `payload.` now accept `Partial<PayloadRequest>`, as they
pass it through to internal operations with `createLocalReq`.

So now, this is also valid, while previously you had to do `as` cast for
`req`.
```ts
const req = { payload }
await initTransaction(req)
await payload.create({
  collection: "posts",
  data: {},
  req
})
await commitTransaction(req)
```


Marked as deprecated `PayloadRequest['transactionIDPromise']` to remove
in the next major version. It was never used anywhere.
Refactored `withSession` that returns an object to `getSession` that
returns just `ClientSession`. Better type safety for arguments
Deduplicated in all drizzle operations to `getTransaction(this, req)`
utility:
```ts
const db = this.sessions[await req?.transactionID]?.db || this.drizzle
```
Added fallback for throwing unique validation errors in database
operations when `req.t` is not available.

In migration `up` and `down` functions our `req` is not partial, while
we used to passed `req` with only 2 properties - `payload` and
`transactionID`. This is misleading and you can't access for example
`req.t`.
Now, to achieve "real" full `req` - we generate it with `createLocalReq`
in all migration functions.

This all is backwards compatible. In all public API places where you
expect the full `req` (like hooks) you still have it.

### Why?
Better DX, more expected types, less errors because of types casting.
2024-12-18 22:43:37 -05:00
Javier
198763a24e feat(db-mongodb): allow to customize mongoose schema options with collectionsSchemaOptions (#9885)
Adds the ability to pass additional schema options for collections with:
```ts
mongooseAdapter({
  collectionsSchemaOptions: {
    posts: {
      strict: false,
    },
  },
})
``` 

This changes relates to these:

-   https://github.com/payloadcms/payload/issues/4533
-   https://github.com/payloadcms/payload/discussions/4534

It is a proposal to set custom schema options for mongoose driver. 

I understand this got introduced into `main` v2 after `beta` branch was
created so this feature got lost.

- [x] I have read and understand the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository.

<!--

Thank you for the PR! Please go through the checklist below and make
sure you've completed all the steps.

Please review the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository if you haven't already.

The following items will ensure that your PR is handled as smoothly as
possible:

- PR Title must follow conventional commits format. For example, `feat:
my new feature`, `fix(plugin-seo): my fix`.
- Minimal description explained as if explained to someone not
immediately familiar with the code.
- Provide before/after screenshots or code diffs if applicable.
- Link any related issues/discussions from GitHub or Discord.
- Add review comments if necessary to explain to the reviewer the logic
behind a change

### What?

### Why?

### How?

Fixes #

-->

---------

Co-authored-by: Sasha <64744993+r1tsuu@users.noreply.github.com>
2024-12-18 15:46:39 +00:00
Sasha
b73fc586b8 feat: expose session, db in migrations to use the active transaction with the database directly (#9849)
This PR adds a feature which fixes another issue with migrations in
Postgres and does few refactors that significantly reduce code
duplication.

Previously, if you needed to use the underlying database directly in
migrations with the active transaction (for example to execute raw SQL),
created from `payload create:migration`, as `req` doesn't work there you
had to do something like this:
```ts
// Postgres
export async function up({ payload, req }: MigrateUpArgs): Promise<void> {
  const db = payload.db.sessions?.[await req.transactionID!].db ?? payload.db
  const { rows: posts } = await db.execute(sql`SELECT * from posts`)
}

// MongoDB
export async function up({ payload, req }: MigrateUpArgs): Promise<void> {
  const session = payload.db.sessions?.[await req.transactionID!]
  const posts = await payload.db.collections.posts.collection.find({ session }).toArray()
}
```

Which was:
1. Awkward to write
2. Not documented anywhere

Now, we expose `session` and `db` to `up` and `down` functions for you:

#### MongoDB:
```ts
import { type MigrateUpArgs } from '@payloadcms/db-mongodb'

export async function up({ session, payload, req }: MigrateUpArgs): Promise<void> {
  const posts = await payload.db.collections.posts.collection.find({ session }).toArray()
}
```
#### Postgres:
```ts
import { type MigrateUpArgs, sql } from '@payloadcms/db-postgres'

export async function up({ db, payload, req }: MigrateUpArgs): Promise<void> {
  const { rows: posts } = await db.execute(sql`SELECT * from posts`)
}
```

#### SQLite:
```ts
import { type MigrateUpArgs, sql } from '@payloadcms/db-sqlite'

export async function up({ db, payload, req }: MigrateUpArgs): Promise<void> {
  const { rows: posts } = await db.run(sql`SELECT * from posts`)
}
```
This actually was a thing with Postgres migrations, we already were
passing `db`, but:
1. Only for `up` and when running `payload migrate`, not for example
with `payload migrate:fresh`
2. Not documented neither in TypeScript or docs.

By ensuring we use `db`, this also fixes an issue that affects all
Postgres/SQLite migrations:

Currently, if we run `payload migration:create` with the postgres
adapter we get a file like this:

```ts
import { MigrateUpArgs, MigrateDownArgs, sql } from '@payloadcms/db-postgres'

export async function up({ payload, req }: MigrateUpArgs): Promise<void> {
  await payload.db.drizzle.execute(sql`
   CREATE TABLE IF NOT EXISTS "users" (
  	"id" serial PRIMARY KEY NOT NULL,
  );
```
Looks good?
Not exactly!
`payload.db.drizzle.execute()` doesn't really use the current
transaction which can lead to some problems.
Instead, it should use the `db` from `payload.db.sessions?.[await
req.transactionID!].db` because that's where we store our Drizzle
instance with the transaction.

But now, if we generate `payload migrate:create` we get:
```ts
import { MigrateUpArgs, MigrateDownArgs, sql } from '@payloadcms/db-postgres'

export async function up({ db, payload, req }: MigrateUpArgs): Promise<void> {
  await db.execute(sql`
   CREATE TABLE IF NOT EXISTS "users" (
  	"id" serial PRIMARY KEY NOT NULL,
  );
```

Which is what we want, as the `db` is passed correctly here:

76428373e4/packages/drizzle/src/migrate.ts (L88-L90)

```ts
export async function up({ db, payload, req }: MigrateUpArgs): Promise<void> {
  const dbWithTransaction = payload.db.sessions?.[await req.transactionID!].db
  payload.logger.info({ one: db === dbWithTransaction })
  payload.logger.info({ two: db === payload.db.drizzle })
```
<img width="336" alt="image"
src="https://github.com/user-attachments/assets/f9fab5a9-44c2-44a9-95dd-8e5cf267f027">

Additionally, this PR refactors:
* `createMigration` with Drizzle - now we have sharable
`buildCreateMigration` in `@payloadcms/drizzle` to reduce copy-pasting
of the same logic.
* the `v2-v3` relationships migration for Postgres is now shared between
`db-postgres` and `db-vercel-postgres`, again to reduce copy-paste.
2024-12-11 12:23:12 -05:00
Dan Ribbens
631edd4c17 fix: latest: true version disappear on parallel writes (#9032)
What?
Fixes issue when on parallel writes in result you can have 0 latest:
true versions.

Why?
There must be always a version with latest: true

How?
Ensures that we always have a version with latest: true by adding a
filter on createdAt < createdVersion.createdAt.
Instead, this ponentially can lead to a situation where we have 2
versions with latest: true, if they were created at the exact same time,
but this shouldn't happen in a real world scenario and it's much less
problematic than not having a version with latest: true.

Fixes https://github.com/payloadcms/payload/issues/5895

Changes from #8986

---------

Co-authored-by: Sasha <64744993+r1tsuu@users.noreply.github.com>
2024-12-02 14:34:45 -05:00
Sasha
b5f89d5199 fix(db-mongodb): sanitizeRelationshipIDs named tabs within tabs (#9400)
Ensures `sanitizeRelationshipIDs` works properly in any case
Updates predefinedMigration to work with new globals
Skips ObjectID creation errors to not fail with outdated data to the
schema.
2024-11-25 16:27:47 -05:00
Sasha
cae300e8e3 perf: flattenedFields collection/global property, remove deep copying in validateQueryPaths (#9299)
### What?
Improves querying performance of the Local API, reduces copying overhead

### How?

Adds `flattenedFields` property for sanitized collection/global config
which contains fields in database schema structure.
For example, It removes rows / collapsible / unnamed tabs and merges
them to the parent `fields`, ignores UI fields, named tabs are added as
`type: 'tab'`.
This simplifies code in places like Drizzle `transform/traverseFields`.
Also, now we can avoid calling `flattenTopLevelFields` which adds some
overhead and use `collection.flattenedFields` / `field.flattenedFields`.

By refactoring `configToJSONSchema.ts` with `flattenedFields` it also
1. Fixes https://github.com/payloadcms/payload/issues/9467
2. Fixes types for UI fields were generated for `select`



Removes this deep copying for each `where` query constraint
58ac784425/packages/payload/src/database/queryValidation/validateQueryPaths.ts (L69-L73)
which potentially can add overhead if you have a large collection/global
config

UPD:
The overhead is even much more than in the benchmark below if you have
Lexical with blocks.

Benchmark in `relationships/int.spec.ts`:
```ts
const now = Date.now()
for (let i = 0; i < 10; i++) {
  const now = Date.now()
  for (let i = 0; i < 300; i++) {
    const query = await payload.find({
      collection: 'chained',
      where: {
        'relation.relation.name': {
          equals: 'third',
        },
        and: [
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
        ],
      },
    })
  }

  payload.logger.info(`#${i + 1} ${Date.now() - now}`)
}
payload.logger.info(`Total ${Date.now() - now}`)
```

Before:
```
[02:11:48] INFO: #1 3682
[02:11:50] INFO: #2 2199
[02:11:54] INFO: #3 3483
[02:11:56] INFO: #4 2516
[02:11:59] INFO: #5 2467
[02:12:01] INFO: #6 1987
[02:12:03] INFO: #7 1986
[02:12:05] INFO: #8 2375
[02:12:07] INFO: #9 2040
[02:12:09] INFO: #10 1920
    [PASS] Relationships > Querying > Nested Querying > should allow querying two levels deep (24667ms)
[02:12:09] INFO: Total 24657
```

After:
```
[02:12:36] INFO: #1 2113
[02:12:38] INFO: #2 1854
[02:12:40] INFO: #3 1700
[02:12:42] INFO: #4 1797
[02:12:44] INFO: #5 2121
[02:12:46] INFO: #6 1774
[02:12:47] INFO: #7 1670
[02:12:49] INFO: #8 1610
[02:12:50] INFO: #9 1596
[02:12:52] INFO: #10 1576
    [PASS] Relationships > Querying > Nested Querying > should allow querying two levels deep (17828ms)
[02:12:52] INFO: Total 17818
```
2024-11-25 10:28:07 -05:00
Jessica Chowdhury
fb3242df0a feat: add skip and force accept flags to migration commands (#8843)
1. Adds flag `--skip-empty` to `migrate:create` to bypass the empty
migration file prompt.
    - Blank migration file will not be created if this flag is passed.
3. Adds flag `--force-accept-warning` to `migrate:fresh` to bypass the
drop database prompt
2024-11-22 10:37:54 -05:00
Sasha
665b3536d3 fix(db-mongodb): potential errors in sanitizeRelationshipIDs with ref being a non object (#9292)
### What?
Fixes potential errors when passed to `sanitizeRelationships` `ref`
could potentially be a non object (for example `string`) because of
having in the database data in old structure.
```
"Cannot create property 'a' on string 'B'",
``` 

### Why?
Necessary particularly for the migration script, as it migrates
everything including versions that can have outdated data.

### How?
Ensures passed `ref` is an `object`.
2024-11-18 16:10:13 +02:00
Dan Ribbens
7c6f41936b feat(db-mongodb)!: update mongoose to 8.8.1 (#9115)
### What?
Upgrades mongoose from 6 to latest `v8.8.1`

Fixes https://github.com/payloadcms/payload/issues/9171

### Why?
Compatibilty with Mongodb Atlas

### How?
- Updates deps
- Changed ObjectId from bson-objectid to use `new Type.ObjectId` from
mongoose for compatibility (only inside of db-mongodb)
- Internal type adjustments

https://github.com/payloadcms/payload/discussions/9088

BREAKING CHANGES:
All projects with existing data having versions enabled, or relationship or upload fields will want to create the predefined migration that converts all strings to ObjectIDs where needed. This can be created using `payload migrate:create --file @payloadcms/mongodb/relationships-v2-v3`.
For projects making use of the exposed Models from mongoose, review the
upgrade guides from [v6 to
v7](https://mongoosejs.com/docs/7.x/docs/migrating_to_7.html) and [v7 to
v8](https://mongoosejs.com/docs/migrating_to_8.html) and make
adjustments as needed.

---------

Co-authored-by: Sasha <64744993+r1tsuu@users.noreply.github.com>
2024-11-15 12:03:56 -05:00
Francisco Lourenço
2d2d020c29 feat(db-mongodb): support query options in db update operations (#9191)
The mongodb adapter `updateOne` method accepts an `options` argument
that allows query options to be passed to mongoose. This parameter was
added in https://github.com/payloadcms/payload/pull/8397 to support the
`upsert` operation.

This `options` parameter can also be useful when using the database
adaptor directly without going through the local api. It is true that
the Mongoose models could be used directly in such situations, but the
adapter methods include a lot of useful functionality, like for instance
the sanitization of document and relationship ids, so it is desirable to
be able to use the adapter functions while still being able to provide
mongoose query options (e.g. `{timestamps: false}`).

This PR adds the same options parameter to the other update methods of
the mongodb adapter.
2024-11-14 15:15:03 -05:00
Elliot DeNolf
5ff1bb366c chore: misc cleanup (#9206)
- Proper error logger usage
- Some no-fallthrough warning cleanup
2024-11-14 11:14:08 -05:00
James Mikrut
6bb4067bb3 feat: adds option to mongoose to ensure indexes (#9155)
Adds option `ensureIndexes` to Mongoose adapter, which will ensure
indexes are ready prior to completing connection.
2024-11-12 14:42:25 -05:00
Tobias Odendahl
09c41d5c86 fix(db-mongodb)!: use dbName for mongodb model (#9107)
### What?
Uses the `collection.dbName` property for the Mongoose model, if
defined.

### Why?
Currently, `collection.dbName` is used for the version name but not for
the actual collection name. Additionally, `autoPluralization` modifies
the `dbName` regardless. This behavior is inconsistent and contradicts
the documentation.

### How?
- Utilize `collection.dbName` instead of `collection.slug`.
- Disable `autoPluralization` for collections with a defined `dbName`.

Related: https://github.com/payloadcms/payload/discussions/9058

**BREAKING CHANGES**
If a `dbName` was previously provided, it will now be used as the
MongoDB collection name instead of the collection `slug`.
`autoPluralization` will not be applied to `dbName`.

---------

Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
2024-11-12 13:31:23 -05:00
Alessio Gravili
03291472d6 chore: bump all eslint dependencies, run lint and prettier (#9128)
This fixes a peer dependency error in our monorepo, as
eslint-plugin-jsx-a11y finally supports eslint v9.

Additionally, this officially adds TypeScript 5.6 support for
typescript-eslint.
2024-11-12 10:18:22 -05:00
Alessio Gravili
6899a3cc27 fix(db-mongodb): destructuring error when trying to filter date fields by string query (#9116)
Previously, when filtering the internal link relationship in lexical by
typing in the relationship field, it would throw an error, as that
relationship field has a relation to "date-fields".
2024-11-11 20:41:02 -07:00
Jacob Fletcher
c96fa613bc feat!: on demand rsc (#8364)
Currently, Payload renders all custom components on initial compile of
the admin panel. This is problematic for two key reasons:
1. Custom components do not receive contextual data, i.e. fields do not
receive their field data, edit views do not receive their document data,
etc.
2. Components are unnecessarily rendered before they are used

This was initially required to support React Server Components within
the Payload Admin Panel for two key reasons:
1. Fields can be dynamically rendered within arrays, blocks, etc.
2. Documents can be recursively rendered within a "drawer" UI, i.e.
relationship fields
3. Payload supports server/client component composition 

In order to achieve this, components need to be rendered on the server
and passed as "slots" to the client. Currently, the pattern for this is
to render custom server components in the "client config". Then when a
view or field is needed to be rendered, we first check the client config
for a "pre-rendered" component, otherwise render our client-side
fallback component.

But for the reasons listed above, this pattern doesn't exactly make
custom server components very useful within the Payload Admin Panel,
which is where this PR comes in. Now, instead of pre-rendering all
components on initial compile, we're able to render custom components
_on demand_, only as they are needed.

To achieve this, we've established [this
pattern](https://github.com/payloadcms/payload/pull/8481) of React
Server Functions in the Payload Admin Panel. With Server Functions, we
can iterate the Payload Config and return JSX through React's
`text/x-component` content-type. This means we're able to pass
contextual props to custom components, such as data for fields and
views.

## Breaking Changes

1. Add the following to your root layout file, typically located at
`(app)/(payload)/layout.tsx`:

    ```diff
    /* THIS FILE WAS GENERATED AUTOMATICALLY BY PAYLOAD. */
    /* DO NOT MODIFY IT BECAUSE IT COULD BE REWRITTEN AT ANY TIME. */
    + import type { ServerFunctionClient } from 'payload'

    import config from '@payload-config'
    import { RootLayout } from '@payloadcms/next/layouts'
    import { handleServerFunctions } from '@payloadcms/next/utilities'
    import React from 'react'

    import { importMap } from './admin/importMap.js'
    import './custom.scss'

    type Args = {
      children: React.ReactNode
    }

+ const serverFunctions: ServerFunctionClient = async function (args) {
    +  'use server'
    +  return handleServerFunctions({
    +    ...args,
    +    config,
    +    importMap,
    +  })
    + }

    const Layout = ({ children }: Args) => (
      <RootLayout
        config={config}
        importMap={importMap}
    +  serverFunctions={serverFunctions}
      >
        {children}
      </RootLayout>
    )

    export default Layout
    ```

2. If you were previously posting to the `/api/form-state` endpoint, it
no longer exists. Instead, you'll need to invoke the `form-state` Server
Function, which can be done through the _new_ `getFormState` utility:

    ```diff
    - import { getFormState } from '@payloadcms/ui'
    - const { state } = await getFormState({
    -   apiRoute: '',
    -   body: {
    -     // ...
    -   },
    -   serverURL: ''
    - })

    + const { getFormState } = useServerFunctions()
    +
    + const { state } = await getFormState({
    +   // ...
    + })
    ```

## Breaking Changes

```diff
- useFieldProps()
- useCellProps()
```

More details coming soon.

---------

Co-authored-by: Alessio Gravili <alessio@gravili.de>
Co-authored-by: Jarrod Flesch <jarrodmflesch@gmail.com>
Co-authored-by: James <james@trbl.design>
2024-11-11 13:59:05 -05:00
Sasha
0a15388edb feat(db-postgres): add point field support (#9078)
### What?
Adds full support for the point field to Postgres and Vercel Postgres
adapters through the Postgis extension. Fully the same API as with
MongoDB, including support for `near`, `within` and `intersects`
operators.

Additionally, exposes to adapter args:
*
`tablesFilter`https://orm.drizzle.team/docs/drizzle-kit-push#including-tables-schemas-and-extensions.
* `extensions` list of extensions to create, for example `['vector',
'pg_search']`, `postgis` is created automatically if there's any point
field

### Why?
It's essential to support that field type, especially if the postgres
adapter should be out of beta on 3.0 stable.

### How?
* Bumps `drizzle-orm` to `0.36.1` and `drizzle-kit` to `0.28.0` as we
need this change https://github.com/drizzle-team/drizzle-orm/pull/3141
* Uses its functions to achieve querying functionality, for example the
`near` operator works through `ST_DWithin` or `intersects` through
`ST_Intersects`.
* Removes MongoDB condition from all point field tests, but keeps for
SQLite

Resolves these discussions:
https://github.com/payloadcms/payload/discussions/8996
https://github.com/payloadcms/payload/discussions/8644
2024-11-11 09:31:47 -05:00
Dan Ribbens
ee117bb616 fix!: handle custom id logic in mongodb adapter (#9069)
### What?
Moved the logic for copying the data.id to data._id to the mongoose
adapter.

### Why?
If you have any hooks that need to set the `id`, the value does not get
sent to mongodb as you would expect since it was copied before the
beforeValidate hooks.

### How?
Now data._id is assigned only in the mongodb adapter's `create`
function.

BREAKING CHANGES:
When using custom ID fields, if you have any collection hooks for
beforeValidate, beforeChange then `data._id` will no longer be assigned
as this happens now in the database adapter. Use `data.id` instead.
2024-11-08 15:34:19 -05:00