Commit Graph

47 Commits

Author SHA1 Message Date
Sasha
79a7b4ad02 chore(db-mongodb): tsconfig uses strict: true and noUncheckedIndexedAccess: true (#11444)
Migrates the `db-mongodb` package to use `strict: true` and
`noUncheckedIndexedAccess: true` TSConfig properties.
This greatly improves code quality and prevents some runtime errors or
gives better error messages.
2025-03-01 00:17:24 +02:00
Sasha
d4d2bf4617 perf(db-mongodb): faster join field aggregation by replacing mongoose-aggregate-paginate-v2 with a custom implementation (#10936)
Fixes
https://github.com/payloadcms/payload/discussions/10165#discussioncomment-12034047

As described in the discussion, we have an incorrect order of
aggregation pipeline when using aggregations with the join field. We
must use `$sort`, `$skip`, `$limit` before the `$lookup` or otherwise
mongodb scans all the docs, applies `$lookup` for them and only after
applies `$limit`, `$skip`.
Replaces `mongoose-aggregate-paginate-v2` with a custom
`aggregatePaginate` because we need a custom solution here. This was
considered in https://github.com/payloadcms/payload/pull/9594 but it was
reverted as for now.

Fixes https://github.com/payloadcms/payload/issues/11187
2025-02-28 21:30:00 +02:00
Alessio Gravili
41c7413f59 feat(db-*): add updateMany method to database adapter (#11441)
This PR adds a new `payload.db.updateMany` method, which is a more performant way to update multiple documents compared to using `payload.update`.
2025-02-27 20:30:17 -07:00
Sasha
b08ff88fbd fix(db-mongodb): mongodb optimizations (#10120)
There are few issues introduced in #9594 that we need to look into with
these changes.
2024-12-21 07:42:44 -05:00
Sasha
e468292039 perf(db-mongodb): improve performance of all operations, up to 50% faster (#9594)
This PR improves speed and memory efficiency across all operations with
the Mongoose adapter.

### How?

- Removes Mongoose layer from all database calls, instead uses MongoDB
directly. (this doesn't remove building mongoose schema since it's still
needed for indexes + users in theory can use it)
- Replaces deep copying of read results using
`JSON.parse(JSON.stringify(data))` with the `transform` `operation:
'read'` function which converts Date's, ObjectID's in relationships /
joins to strings. As before, it also handles transformations for write
operations.
- Faster `hasNearConstraint` for potentially large `where`'s
- `traverseFields` now can accept `flattenedFields` which we use in
`transform`. Less recursive calls with tabs/rows/collapsible

Additional fixes
- Uses current transaction for querying nested relationships properties
in `buildQuery`, previously it wasn't used which could've led to wrong
results
- Allows to clear not required point fields with passing `null` from the
Local API. Previously it didn't work in both, MongoDB and Postgres

Benchmarks using this file
https://github.com/payloadcms/payload/blob/chore/db-benchmark/test/_community/int.spec.ts

### Small Dataset Performance

| Metric | Before Optimization | After Optimization | Improvement (%) |

|---------------------------|---------------------|--------------------|-----------------|
| Average FULL (ms) | 1170 | 844 | 27.86% |
| `payload.db.create` (ms) | 1413 | 691 | 51.12% |
| `payload.db.find` (ms) | 2856 | 2204 | 22.83% |
| `payload.db.deleteMany` (ms) | 15206 | 8439 | 44.53% |
| `payload.db.updateOne` (ms) | 21444 | 12162 | 43.30% |
| `payload.db.findOne` (ms) | 159 | 112 | 29.56% |
| `payload.db.deleteOne` (ms) | 3729 | 2578 | 30.89% |
| DB small FULL (ms) | 64473 | 46451 | 27.93% |

---

### Medium Dataset Performance

| Metric | Before Optimization | After Optimization | Improvement (%) |

|---------------------------|---------------------|--------------------|-----------------|
| Average FULL (ms) | 9407 | 6210 | 33.99% |
| `payload.db.create` (ms) | 10270 | 4321 | 57.93% |
| `payload.db.find` (ms) | 20814 | 16036 | 22.93% |
| `payload.db.deleteMany` (ms) | 126351 | 61789 | 51.11% |
| `payload.db.updateOne` (ms) | 201782 | 99943 | 50.49% |
| `payload.db.findOne` (ms) | 1081 | 817 | 24.43% |
| `payload.db.deleteOne` (ms) | 28534 | 23363 | 18.12% |
| DB medium FULL (ms) | 519518 | 342194 | 34.13% |

---

### Large Dataset Performance

| Metric | Before Optimization | After Optimization | Improvement (%) |

|---------------------------|---------------------|--------------------|-----------------|
| Average FULL (ms) | 26575 | 17509 | 34.14% |
| `payload.db.create` (ms) | 29085 | 12196 | 58.08% |
| `payload.db.find` (ms) | 58497 | 43838 | 25.04% |
| `payload.db.deleteMany` (ms) | 372195 | 173218 | 53.47% |
| `payload.db.updateOne` (ms) | 544089 | 288350 | 47.00% |
| `payload.db.findOne` (ms) | 3058 | 2197 | 28.14% |
| `payload.db.deleteOne` (ms) | 82444 | 64730 | 21.49% |
| DB large FULL (ms) | 1461097 | 969714 | 33.62% |
2024-12-19 13:20:39 -05:00
Javier
198763a24e feat(db-mongodb): allow to customize mongoose schema options with collectionsSchemaOptions (#9885)
Adds the ability to pass additional schema options for collections with:
```ts
mongooseAdapter({
  collectionsSchemaOptions: {
    posts: {
      strict: false,
    },
  },
})
``` 

This changes relates to these:

-   https://github.com/payloadcms/payload/issues/4533
-   https://github.com/payloadcms/payload/discussions/4534

It is a proposal to set custom schema options for mongoose driver. 

I understand this got introduced into `main` v2 after `beta` branch was
created so this feature got lost.

- [x] I have read and understand the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository.

<!--

Thank you for the PR! Please go through the checklist below and make
sure you've completed all the steps.

Please review the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository if you haven't already.

The following items will ensure that your PR is handled as smoothly as
possible:

- PR Title must follow conventional commits format. For example, `feat:
my new feature`, `fix(plugin-seo): my fix`.
- Minimal description explained as if explained to someone not
immediately familiar with the code.
- Provide before/after screenshots or code diffs if applicable.
- Link any related issues/discussions from GitHub or Discord.
- Add review comments if necessary to explain to the reviewer the logic
behind a change

### What?

### Why?

### How?

Fixes #

-->

---------

Co-authored-by: Sasha <64744993+r1tsuu@users.noreply.github.com>
2024-12-18 15:46:39 +00:00
Francisco Lourenço
2d2d020c29 feat(db-mongodb): support query options in db update operations (#9191)
The mongodb adapter `updateOne` method accepts an `options` argument
that allows query options to be passed to mongoose. This parameter was
added in https://github.com/payloadcms/payload/pull/8397 to support the
`upsert` operation.

This `options` parameter can also be useful when using the database
adaptor directly without going through the local api. It is true that
the Mongoose models could be used directly in such situations, but the
adapter methods include a lot of useful functionality, like for instance
the sanitization of document and relationship ids, so it is desirable to
be able to use the adapter functions while still being able to provide
mongoose query options (e.g. `{timestamps: false}`).

This PR adds the same options parameter to the other update methods of
the mongodb adapter.
2024-11-14 15:15:03 -05:00
James Mikrut
6bb4067bb3 feat: adds option to mongoose to ensure indexes (#9155)
Adds option `ensureIndexes` to Mongoose adapter, which will ensure
indexes are ready prior to completing connection.
2024-11-12 14:42:25 -05:00
Jacob Fletcher
c96fa613bc feat!: on demand rsc (#8364)
Currently, Payload renders all custom components on initial compile of
the admin panel. This is problematic for two key reasons:
1. Custom components do not receive contextual data, i.e. fields do not
receive their field data, edit views do not receive their document data,
etc.
2. Components are unnecessarily rendered before they are used

This was initially required to support React Server Components within
the Payload Admin Panel for two key reasons:
1. Fields can be dynamically rendered within arrays, blocks, etc.
2. Documents can be recursively rendered within a "drawer" UI, i.e.
relationship fields
3. Payload supports server/client component composition 

In order to achieve this, components need to be rendered on the server
and passed as "slots" to the client. Currently, the pattern for this is
to render custom server components in the "client config". Then when a
view or field is needed to be rendered, we first check the client config
for a "pre-rendered" component, otherwise render our client-side
fallback component.

But for the reasons listed above, this pattern doesn't exactly make
custom server components very useful within the Payload Admin Panel,
which is where this PR comes in. Now, instead of pre-rendering all
components on initial compile, we're able to render custom components
_on demand_, only as they are needed.

To achieve this, we've established [this
pattern](https://github.com/payloadcms/payload/pull/8481) of React
Server Functions in the Payload Admin Panel. With Server Functions, we
can iterate the Payload Config and return JSX through React's
`text/x-component` content-type. This means we're able to pass
contextual props to custom components, such as data for fields and
views.

## Breaking Changes

1. Add the following to your root layout file, typically located at
`(app)/(payload)/layout.tsx`:

    ```diff
    /* THIS FILE WAS GENERATED AUTOMATICALLY BY PAYLOAD. */
    /* DO NOT MODIFY IT BECAUSE IT COULD BE REWRITTEN AT ANY TIME. */
    + import type { ServerFunctionClient } from 'payload'

    import config from '@payload-config'
    import { RootLayout } from '@payloadcms/next/layouts'
    import { handleServerFunctions } from '@payloadcms/next/utilities'
    import React from 'react'

    import { importMap } from './admin/importMap.js'
    import './custom.scss'

    type Args = {
      children: React.ReactNode
    }

+ const serverFunctions: ServerFunctionClient = async function (args) {
    +  'use server'
    +  return handleServerFunctions({
    +    ...args,
    +    config,
    +    importMap,
    +  })
    + }

    const Layout = ({ children }: Args) => (
      <RootLayout
        config={config}
        importMap={importMap}
    +  serverFunctions={serverFunctions}
      >
        {children}
      </RootLayout>
    )

    export default Layout
    ```

2. If you were previously posting to the `/api/form-state` endpoint, it
no longer exists. Instead, you'll need to invoke the `form-state` Server
Function, which can be done through the _new_ `getFormState` utility:

    ```diff
    - import { getFormState } from '@payloadcms/ui'
    - const { state } = await getFormState({
    -   apiRoute: '',
    -   body: {
    -     // ...
    -   },
    -   serverURL: ''
    - })

    + const { getFormState } = useServerFunctions()
    +
    + const { state } = await getFormState({
    +   // ...
    + })
    ```

## Breaking Changes

```diff
- useFieldProps()
- useCellProps()
```

More details coming soon.

---------

Co-authored-by: Alessio Gravili <alessio@gravili.de>
Co-authored-by: Jarrod Flesch <jarrodmflesch@gmail.com>
Co-authored-by: James <james@trbl.design>
2024-11-11 13:59:05 -05:00
Dan Ribbens
3847428f0a fix(db-*): make db.begintransactions required (#8419)
Changing the transcations functions on the db so that projects using
typescript `strict: true` do not need to type narrow before using it.
2024-09-30 13:12:07 -04:00
Dan Ribbens
82ba1930e5 feat: add upsert to database interface and adapters (#8397)
- Adds the upsert method to the database interface
- Adds a mongodb specific option to extend the updateOne to accept
mongoDB Query Options (to pass `upsert: true`)
- Added upsert method to all database adapters
- Uses db.upsert in the payload preferences update operation

Includes a test using payload-preferences
2024-09-25 09:23:54 -04:00
Alessio Gravili
7fd736ea5b chore: lint entire codebase including db packages 2024-09-03 11:19:19 -04:00
Elliot DeNolf
631431e006 feat: @payloadcms/db-vercel-postgres adapter (#7806)
Dedicated adapter for Vercel Postgres

- Uses the `@vercel/postgres` package under the hood.
- No `pg` dependency, speeds up invocation
- Includes refactoring all base postgres functionality into a
`BasePostgresAdapter` type, which will ease implementation of [other
adapters supported by
drizzle-orm](https://orm.drizzle.team/docs/get-started-postgresql)

## Usage

```ts
import { buildConfig } from 'payload'
import { vercelPostgresAdapter } from '@payloadcms/db-vercel-postgres'

export default buildConfig({
  db: vercelPostgresAdapter({
    pool: {
      connectionString: process.env.DATABASE_URI,
    },
  }),
  // ...rest of config
})
```

### Automatic Connection String Detection

Have Vercel automatically detect from environment variable (typically
`process.env.POSTGRES_URL`)

```ts
export default buildConfig({
  db: postgresAdapter(),
  // ...rest of config
})
```
2024-08-21 22:54:47 -04:00
James Mikrut
1cb1e5e8b3 feat(db-*): allows for running migrations in production automatically (#7563)
## Description

Introduces a pattern for running migrations upon Payload init in
production.
2024-08-07 13:57:12 -04:00
Patrik
8ba39aa5ca fix(db-mongodb): adds new optional collation feature flag behind mongodb collation option (#7361)
## Description

V2 PR [here](https://github.com/payloadcms/payload/pull/7359)

- [x] I have read and understand the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository.

## Type of change

- [x] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [x] This change requires a documentation update

## Checklist:

- [x] Existing test suite passes locally with my changes
- [x] I have made corresponding changes to the documentation
2024-07-25 12:38:18 -04:00
Alessio Gravili
83fd4c6622 chore: run lint and prettier on entire codebase 2024-07-11 15:27:01 -04:00
Jacob Fletcher
9e76c8f4e3 feat!: prebundle payload, ui, richtext-lexical (#6579)
# Breaking Changes

### New file import locations

Exports from the `payload` package have been _significantly_ cleaned up.
Now, just about everything is able to be imported from `payload`
directly, rather than an assortment of subpath exports. This means that
things like `import { buildConfig } from 'payload/config'` are now just
imported via `import { buildConfig } from 'payload'`. The mental model
is significantly simpler for developers, but you might need to update
some of your imports.

Payload now exposes only three exports:

1. `payload` - all types and server-only Payload code
2. `payload/shared` - utilities that can be used in either the browser
or in Node environments
3. `payload/node` - heavy utilities that should only be imported in Node
scripts and never be imported into bundled code like Next.js

### UI library pre-bundling

With this release, we've dramatically sped up the compile time for
Payload by pre-bundling our entire UI package for use inside of the
Payload admin itself. There are new exports that should be used within
Payload custom components:

1. `@payloadcms/ui/client` - all client components 
2. `@payloadcms/ui/server` - all server components

For all of your custom Payload admin UI components, you should be
importing from one of these two pre-compiled barrel files rather than
importing from the more deeply nested exports directly. That will keep
compile times nice and speedy, and will also make sure that the bundled
JS for your admin UI is kept small.

For example, whereas before, if you imported the Payload `Button`, you
would have imported it like this:

```ts
import { Button } from '@payloadcms/ui/elements/Button'
```

Now, you would import it like this:

```ts
import { Button } from '@payloadcms/ui/client'
```

This is a significant DX / performance optimization that we're pretty
pumped about.

However, if you are importing or re-using Payload UI components
_outside_ of the Payload admin UI, for example in your own frontend
apps, you can import from the individual component exports which will
make sure that the bundled JS is kept to a minimum in your frontend
apps. So in your own frontend, you can continue to import directly to
the components that you want to consume rather than importing from the
pre-compiled barrel files.

Individual component exports will now come with their corresponding CSS
and everything will work perfectly as-expected.

### Specific exports have changed

- `'@payloadcms/ui/templates/Default'` and
`'@payloadcms/ui/templates/Minimal`' are now exported from
`'@payloadcms/next/templates'`
- Old: `import { LogOut } from '@payloadcms/ui/icons/LogOut'` new:
`import { LogOutIcon } from '@payloadcms/ui/icons/LogOut'`

## Background info

In effort to make local dev as fast as possible, we need to import as
few files as possible so that the compiler has less to process. One way
we've achieved this in the Admin Panel was to _remove_ all .scss imports
from all components in the `@payloadcms/ui` module using a build
process. This stripped all `import './index.scss'` statements out of
each component before injecting them into `dist`. Instead, it bundles
all of the CSS into a single `main.css` file, and we import _that_ at
the root of the app.

While this concept is _still_ the right solution to the problem, this
particular approach is not viable when using these components outside
the Admin Panel, where not only does this root stylesheet not exist, but
where it would also bloat your app with unused styles. Instead, we need
to _keep_ these .scss imports in place so they are imported directly
alongside your components, as expected. Then, we need create a _new_
build step that _separately_ compiles the components _without_ their
stylesheets—this way your app can consume either as needed from the new
`client` and `server` barrel files within `@payloadcms/ui`, i.e. from
within `@payloadcms/next` and all other admin-specific packages and
plugins.

This way, all other applications will simply import using the direct
file paths, just as they did before. Except now they come with
stylesheets.

And we've gotten a pretty awesome initial compilation performance boost.

---------

Co-authored-by: James <james@trbl.design>
Co-authored-by: Alessio Gravili <alessio@gravili.de>
2024-06-17 14:25:36 -04:00
Ritsu
d5cbbc472d feat: add count operation to collections (#5930) 2024-04-20 14:45:44 -04:00
Elliot DeNolf
5ef2e35685 test: fix mongodb destroy 2024-03-15 15:11:04 -04:00
Alessio Gravili
6789e61488 chore: run lint & prettier on everything 2024-03-14 23:53:47 -04:00
Dan Ribbens
d193c677c7 chore: attach mongoMemoryServer to db and destroy in tests (#5326)
* chore: attach mongoMemoryServer to db and destroy in tests

* bump mongodb-memory-server to 9.x

---------

Co-authored-by: Paul Popus <paul@nouance.io>
2024-03-14 15:41:20 -04:00
Alessio Gravili
50c7269315 chore: replace .d.ts type imports with .js imports, as .d.ts imports break usage checks in the IDE 2024-03-07 16:08:49 -05:00
Kendell Joseph
3090e7c163 chore: updates db-mongodb imports for ESM 2024-03-06 14:19:13 -05:00
James
12c5100bc8 chore: pulls mongodb from main 2024-02-16 11:27:23 -05:00
James
abf0f7111d chore: type fixes 2024-02-15 14:12:59 -05:00
Jarrod Flesch
d204418530 chore: partial gql changes 2024-02-07 14:01:46 -05:00
Jacob Fletcher
2ee425bdf1 chore(db-mongodb): removes vite and webpack config extensions 2024-01-12 15:31:57 -05:00
Elliot DeNolf
530c825f80 feat(db-mongodb): search for migrations dir intelligently 2023-11-28 11:09:55 -05:00
Alessio Gravili
9f873f8630 chore(db-mongodb): add option to disable index hint optimization, which breaks on AWS DocumentDB (#3997)
* chore: add option to disable pagination count index hinting optimization

* chore: rename hintPaginationCountIndex to disablePaginationCountIndexHint

* chore: fix logic

* chore: disablePaginationCountIndexHint => disableIndexHints
2023-11-10 16:08:14 +01:00
James
eb97acd408 fix: #3521 2023-10-10 14:46:52 -04:00
Elliot DeNolf
74972cf4c9 chore: add predefined migration for version-v1-v2 2023-10-08 12:10:03 -04:00
Elliot DeNolf
e55ec6329c chore: fix up fresh, refresh, and reset migration operations 2023-10-06 17:15:43 -04:00
James
812d1806f5 chore: revisions to local dev suite 2023-10-04 14:21:51 -04:00
James
98b6108eab chore: better alias pattern for db adapters 2023-10-04 14:13:40 -04:00
James
1286ed4706 chore: types mongodb adapter 2023-10-02 16:19:18 -04:00
Elliot DeNolf
71b372211f chore: export migration args from top-level adapter export 2023-10-01 22:07:53 -04:00
James
a458375934 chore: passing versions tests 2023-09-23 10:07:29 -07:00
James
e6e8ab6630 chore: adds updateGlobalVersion to mongo adapter 2023-09-23 09:02:59 -07:00
James
029e319c7b chore: wires up contains in pg, skips point tests 2023-09-18 13:00:00 -04:00
James
2442144c98 chore: fixes id types for graphql in database adapters 2023-09-18 11:42:59 -04:00
Elliot DeNolf
79f9ef2481 chore: migration improvements (#3335)
* chore: better migrationDir handling

* chore: add .migrations to gitignore

* chore: migration cli debugging

* docs: stub out each migration command
2023-09-14 14:23:17 -04:00
Alessio Gravili
ae7d6f97d2 chore: formatting and linting (#3261)
* chore: lint packages/payload

* chore: lint packages/db-postgres

* chore: lint packages/db-mongodb

* chore: update eslintrc exclusion rules

* chore: update eslintrc exclusion rules

* chore: lint misc files

* chore: run prettier through packages

* chore: run eslint on payload again

* chore: prettier misc files

* chore: prettier docs
2023-09-01 17:39:44 +02:00
Alessio Gravili
759d36049d chore: more work on exports 2023-08-25 15:04:05 +00:00
Alessio Gravili
7c4dd5ec5e feat: option to disable auto pluralization for the mongoose adapter (#3153)
* chore: disable slug pluralization for versions model

* chore: disable slug pluralizations for globals version model

* chore: disable auto slug pluralization for globals collection

* feat: autoPluralization option for mongoose adapter
2023-08-21 18:05:39 +02:00
Elliot DeNolf
f8e6914f44 feat: proper type for createMigration func 2023-08-09 17:10:00 -04:00
James
f057e25ec7 chore: more path corrections 2023-07-31 15:27:55 -04:00
James
832c1069f1 chore: moves db adapters to top-level packages 2023-07-31 14:26:14 -04:00