Commit Graph

18 Commits

Author SHA1 Message Date
Sasha
e468292039 perf(db-mongodb): improve performance of all operations, up to 50% faster (#9594)
This PR improves speed and memory efficiency across all operations with
the Mongoose adapter.

### How?

- Removes Mongoose layer from all database calls, instead uses MongoDB
directly. (this doesn't remove building mongoose schema since it's still
needed for indexes + users in theory can use it)
- Replaces deep copying of read results using
`JSON.parse(JSON.stringify(data))` with the `transform` `operation:
'read'` function which converts Date's, ObjectID's in relationships /
joins to strings. As before, it also handles transformations for write
operations.
- Faster `hasNearConstraint` for potentially large `where`'s
- `traverseFields` now can accept `flattenedFields` which we use in
`transform`. Less recursive calls with tabs/rows/collapsible

Additional fixes
- Uses current transaction for querying nested relationships properties
in `buildQuery`, previously it wasn't used which could've led to wrong
results
- Allows to clear not required point fields with passing `null` from the
Local API. Previously it didn't work in both, MongoDB and Postgres

Benchmarks using this file
https://github.com/payloadcms/payload/blob/chore/db-benchmark/test/_community/int.spec.ts

### Small Dataset Performance

| Metric | Before Optimization | After Optimization | Improvement (%) |

|---------------------------|---------------------|--------------------|-----------------|
| Average FULL (ms) | 1170 | 844 | 27.86% |
| `payload.db.create` (ms) | 1413 | 691 | 51.12% |
| `payload.db.find` (ms) | 2856 | 2204 | 22.83% |
| `payload.db.deleteMany` (ms) | 15206 | 8439 | 44.53% |
| `payload.db.updateOne` (ms) | 21444 | 12162 | 43.30% |
| `payload.db.findOne` (ms) | 159 | 112 | 29.56% |
| `payload.db.deleteOne` (ms) | 3729 | 2578 | 30.89% |
| DB small FULL (ms) | 64473 | 46451 | 27.93% |

---

### Medium Dataset Performance

| Metric | Before Optimization | After Optimization | Improvement (%) |

|---------------------------|---------------------|--------------------|-----------------|
| Average FULL (ms) | 9407 | 6210 | 33.99% |
| `payload.db.create` (ms) | 10270 | 4321 | 57.93% |
| `payload.db.find` (ms) | 20814 | 16036 | 22.93% |
| `payload.db.deleteMany` (ms) | 126351 | 61789 | 51.11% |
| `payload.db.updateOne` (ms) | 201782 | 99943 | 50.49% |
| `payload.db.findOne` (ms) | 1081 | 817 | 24.43% |
| `payload.db.deleteOne` (ms) | 28534 | 23363 | 18.12% |
| DB medium FULL (ms) | 519518 | 342194 | 34.13% |

---

### Large Dataset Performance

| Metric | Before Optimization | After Optimization | Improvement (%) |

|---------------------------|---------------------|--------------------|-----------------|
| Average FULL (ms) | 26575 | 17509 | 34.14% |
| `payload.db.create` (ms) | 29085 | 12196 | 58.08% |
| `payload.db.find` (ms) | 58497 | 43838 | 25.04% |
| `payload.db.deleteMany` (ms) | 372195 | 173218 | 53.47% |
| `payload.db.updateOne` (ms) | 544089 | 288350 | 47.00% |
| `payload.db.findOne` (ms) | 3058 | 2197 | 28.14% |
| `payload.db.deleteOne` (ms) | 82444 | 64730 | 21.49% |
| DB large FULL (ms) | 1461097 | 969714 | 33.62% |
2024-12-19 13:20:39 -05:00
Sasha
0e5bda9a74 feat: make req partial and optional in DB / Local API operations (#9935)
### What?
Previously, the `req` argument:
In database operations (e.g `payload.db`) was required and you needed to
pass the whole `req` with all the properties. This is confusing because
in database operations we never use its properties outside of
`req.transactionID` and `req.t`, both of which should be optional as
well.

Now, you don't have to do that cast:
```ts
payload.db.findOne({
  collection: 'posts',
  req: {} as PayloadRequest,
  where: {
    id: {
      equals: 1,
    },
  },
})
```
Becomes:
```ts
payload.db.findOne({
  collection: 'posts',
  where: {
    id: {
      equals: 1,
    },
  },
})
```

If you need to use transactions, you're not required to do the `as` cast
as well now, as the `req` not only optional but also partial -
`Partial<PayloadRequest>`.
`initTransaction`, `commitTransaction`, `killTransaction` utilities are
typed better now as well. They do not require to you pass all the
properties of `req`, but only `payload` -
`MarkRequired<Partial<PayloadRequest>, 'payload'>`
```ts
const req = { payload }
await initTransaction(req)
await payload.db.create({
  collection: "posts",
  data: {},
  req
})
await commitTransaction(req)
```

The same for the Local API. Internal operations (for example
`packages/payload/src/collections/operations/find.ts`) still accept the
whole `req`, but local ones
(`packages/payload/src/collections/operations/local/find.ts`) which are
used through `payload.` now accept `Partial<PayloadRequest>`, as they
pass it through to internal operations with `createLocalReq`.

So now, this is also valid, while previously you had to do `as` cast for
`req`.
```ts
const req = { payload }
await initTransaction(req)
await payload.create({
  collection: "posts",
  data: {},
  req
})
await commitTransaction(req)
```


Marked as deprecated `PayloadRequest['transactionIDPromise']` to remove
in the next major version. It was never used anywhere.
Refactored `withSession` that returns an object to `getSession` that
returns just `ClientSession`. Better type safety for arguments
Deduplicated in all drizzle operations to `getTransaction(this, req)`
utility:
```ts
const db = this.sessions[await req?.transactionID]?.db || this.drizzle
```
Added fallback for throwing unique validation errors in database
operations when `req.t` is not available.

In migration `up` and `down` functions our `req` is not partial, while
we used to passed `req` with only 2 properties - `payload` and
`transactionID`. This is misleading and you can't access for example
`req.t`.
Now, to achieve "real" full `req` - we generate it with `createLocalReq`
in all migration functions.

This all is backwards compatible. In all public API places where you
expect the full `req` (like hooks) you still have it.

### Why?
Better DX, more expected types, less errors because of types casting.
2024-12-18 22:43:37 -05:00
Sasha
cae300e8e3 perf: flattenedFields collection/global property, remove deep copying in validateQueryPaths (#9299)
### What?
Improves querying performance of the Local API, reduces copying overhead

### How?

Adds `flattenedFields` property for sanitized collection/global config
which contains fields in database schema structure.
For example, It removes rows / collapsible / unnamed tabs and merges
them to the parent `fields`, ignores UI fields, named tabs are added as
`type: 'tab'`.
This simplifies code in places like Drizzle `transform/traverseFields`.
Also, now we can avoid calling `flattenTopLevelFields` which adds some
overhead and use `collection.flattenedFields` / `field.flattenedFields`.

By refactoring `configToJSONSchema.ts` with `flattenedFields` it also
1. Fixes https://github.com/payloadcms/payload/issues/9467
2. Fixes types for UI fields were generated for `select`



Removes this deep copying for each `where` query constraint
58ac784425/packages/payload/src/database/queryValidation/validateQueryPaths.ts (L69-L73)
which potentially can add overhead if you have a large collection/global
config

UPD:
The overhead is even much more than in the benchmark below if you have
Lexical with blocks.

Benchmark in `relationships/int.spec.ts`:
```ts
const now = Date.now()
for (let i = 0; i < 10; i++) {
  const now = Date.now()
  for (let i = 0; i < 300; i++) {
    const query = await payload.find({
      collection: 'chained',
      where: {
        'relation.relation.name': {
          equals: 'third',
        },
        and: [
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
        ],
      },
    })
  }

  payload.logger.info(`#${i + 1} ${Date.now() - now}`)
}
payload.logger.info(`Total ${Date.now() - now}`)
```

Before:
```
[02:11:48] INFO: #1 3682
[02:11:50] INFO: #2 2199
[02:11:54] INFO: #3 3483
[02:11:56] INFO: #4 2516
[02:11:59] INFO: #5 2467
[02:12:01] INFO: #6 1987
[02:12:03] INFO: #7 1986
[02:12:05] INFO: #8 2375
[02:12:07] INFO: #9 2040
[02:12:09] INFO: #10 1920
    [PASS] Relationships > Querying > Nested Querying > should allow querying two levels deep (24667ms)
[02:12:09] INFO: Total 24657
```

After:
```
[02:12:36] INFO: #1 2113
[02:12:38] INFO: #2 1854
[02:12:40] INFO: #3 1700
[02:12:42] INFO: #4 1797
[02:12:44] INFO: #5 2121
[02:12:46] INFO: #6 1774
[02:12:47] INFO: #7 1670
[02:12:49] INFO: #8 1610
[02:12:50] INFO: #9 1596
[02:12:52] INFO: #10 1576
    [PASS] Relationships > Querying > Nested Querying > should allow querying two levels deep (17828ms)
[02:12:52] INFO: Total 17818
```
2024-11-25 10:28:07 -05:00
Sasha
dae832c288 feat: select fields (#8550)
Adds `select` which is used to specify the field projection for local
and rest API calls. This is available as an optimization to reduce the
payload's of requests and make the database queries more efficient.

Includes:
- [x] generate types for the `select` property
- [x] infer the return type by `select` with 2 modes - include (`field:
true`) and exclude (`field: false`)
- [x] lots of integration tests, including deep fields / localization
etc
- [x] implement the property in db adapters
- [x] implement the property in the local api for most operations
- [x] implement the property in the rest api 
- [x] docs

---------

Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
2024-10-29 21:47:18 +00:00
Elliot DeNolf
570422ff9a chore(db-mongodb): adjust default exports (#7945) 2024-08-29 04:09:44 +00:00
Patrik
fb72d19d6c fix: graphql query concurrency issues (#6925)
## Description

This is the beta (v3) PR for the v2 PR
[here](https://github.com/payloadcms/payload/pull/6857)

Addresses #6800, #5108

- [x] I have read and understand the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository.

## Type of change

- [x] Bug fix (non-breaking change which fixes an issue)

## Checklist:

- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] Existing test suite passes locally with my changes
2024-07-08 15:55:04 +00:00
Jarrod Flesch
0711f880ff chore!: simplify api handler (#6910)
Removes PayloadRequestWithData in favour of just PayloadRequest with
optional types for `data` and `locale`

`addDataAndFileToRequest` and `addLocalesToRequestFromData` now takes in
a single argument instead of an object

```ts
// before
await addDataAndFileToRequest({ request: req })
addLocalesToRequestFromData({ request: req })

// current
await addDataAndFileToRequest(req)
addLocalesToRequestFromData(req)
```

---------

Co-authored-by: Paul Popus <paul@nouance.io>
2024-07-02 09:47:03 -04:00
Jacob Fletcher
9e76c8f4e3 feat!: prebundle payload, ui, richtext-lexical (#6579)
# Breaking Changes

### New file import locations

Exports from the `payload` package have been _significantly_ cleaned up.
Now, just about everything is able to be imported from `payload`
directly, rather than an assortment of subpath exports. This means that
things like `import { buildConfig } from 'payload/config'` are now just
imported via `import { buildConfig } from 'payload'`. The mental model
is significantly simpler for developers, but you might need to update
some of your imports.

Payload now exposes only three exports:

1. `payload` - all types and server-only Payload code
2. `payload/shared` - utilities that can be used in either the browser
or in Node environments
3. `payload/node` - heavy utilities that should only be imported in Node
scripts and never be imported into bundled code like Next.js

### UI library pre-bundling

With this release, we've dramatically sped up the compile time for
Payload by pre-bundling our entire UI package for use inside of the
Payload admin itself. There are new exports that should be used within
Payload custom components:

1. `@payloadcms/ui/client` - all client components 
2. `@payloadcms/ui/server` - all server components

For all of your custom Payload admin UI components, you should be
importing from one of these two pre-compiled barrel files rather than
importing from the more deeply nested exports directly. That will keep
compile times nice and speedy, and will also make sure that the bundled
JS for your admin UI is kept small.

For example, whereas before, if you imported the Payload `Button`, you
would have imported it like this:

```ts
import { Button } from '@payloadcms/ui/elements/Button'
```

Now, you would import it like this:

```ts
import { Button } from '@payloadcms/ui/client'
```

This is a significant DX / performance optimization that we're pretty
pumped about.

However, if you are importing or re-using Payload UI components
_outside_ of the Payload admin UI, for example in your own frontend
apps, you can import from the individual component exports which will
make sure that the bundled JS is kept to a minimum in your frontend
apps. So in your own frontend, you can continue to import directly to
the components that you want to consume rather than importing from the
pre-compiled barrel files.

Individual component exports will now come with their corresponding CSS
and everything will work perfectly as-expected.

### Specific exports have changed

- `'@payloadcms/ui/templates/Default'` and
`'@payloadcms/ui/templates/Minimal`' are now exported from
`'@payloadcms/next/templates'`
- Old: `import { LogOut } from '@payloadcms/ui/icons/LogOut'` new:
`import { LogOutIcon } from '@payloadcms/ui/icons/LogOut'`

## Background info

In effort to make local dev as fast as possible, we need to import as
few files as possible so that the compiler has less to process. One way
we've achieved this in the Admin Panel was to _remove_ all .scss imports
from all components in the `@payloadcms/ui` module using a build
process. This stripped all `import './index.scss'` statements out of
each component before injecting them into `dist`. Instead, it bundles
all of the CSS into a single `main.css` file, and we import _that_ at
the root of the app.

While this concept is _still_ the right solution to the problem, this
particular approach is not viable when using these components outside
the Admin Panel, where not only does this root stylesheet not exist, but
where it would also bloat your app with unused styles. Instead, we need
to _keep_ these .scss imports in place so they are imported directly
alongside your components, as expected. Then, we need create a _new_
build step that _separately_ compiles the components _without_ their
stylesheets—this way your app can consume either as needed from the new
`client` and `server` barrel files within `@payloadcms/ui`, i.e. from
within `@payloadcms/next` and all other admin-specific packages and
plugins.

This way, all other applications will simply import using the direct
file paths, just as they did before. Except now they come with
stylesheets.

And we've gotten a pretty awesome initial compilation performance boost.

---------

Co-authored-by: James <james@trbl.design>
Co-authored-by: Alessio Gravili <alessio@gravili.de>
2024-06-17 14:25:36 -04:00
Jacob Fletcher
019677b7e6 chore(eslint): consolidates and prevents duplicate imports (#6756)
## Description

Adds ESLint rule to consolidate duplicate imports using the
`import/no-duplicates` rule of the `eslint-plugin-import` plugin. More
here:
https://github.com/import-js/eslint-plugin-import/blob/main/docs/rules/no-duplicates.md.
This was needed as opposed to `no-duplicate-imports` because of the
auto-fix feature.
2024-06-12 16:45:43 -04:00
Jarrod Flesch
22c53392a3 chore: improves types for payloadRequest (#6012) 2024-04-25 10:23:03 -04:00
Alessio Gravili
50c7269315 chore: replace .d.ts type imports with .js imports, as .d.ts imports break usage checks in the IDE 2024-03-07 16:08:49 -05:00
Kendell Joseph
3090e7c163 chore: updates db-mongodb imports for ESM 2024-03-06 14:19:13 -05:00
Alessio Gravili
ae7d6f97d2 chore: formatting and linting (#3261)
* chore: lint packages/payload

* chore: lint packages/db-postgres

* chore: lint packages/db-mongodb

* chore: update eslintrc exclusion rules

* chore: update eslintrc exclusion rules

* chore: lint misc files

* chore: run prettier through packages

* chore: run eslint on payload again

* chore: prettier misc files

* chore: prettier docs
2023-09-01 17:39:44 +02:00
Alessio Gravili
370e38ae30 chore: work on imports 2023-08-25 14:56:42 +00:00
James
8d265ee368 chore: maintains internal sanitize fields for mongodb 2023-07-31 16:52:56 -04:00
James
872b38d378 chore: corrects remaining mongodb adapter paths 2023-07-31 15:57:59 -04:00
Dan Ribbens
aa7bad41e4 chore: import path cleanup continued 2023-07-31 15:27:37 -04:00
James
832c1069f1 chore: moves db adapters to top-level packages 2023-07-31 14:26:14 -04:00