payload

Files

Sasha e468292039 perf(db-mongodb): improve performance of all operations, up to 50% faster (#9594 )

This PR improves speed and memory efficiency across all operations with
the Mongoose adapter.

### How?

- Removes Mongoose layer from all database calls, instead uses MongoDB
directly. (this doesn't remove building mongoose schema since it's still
needed for indexes + users in theory can use it)
- Replaces deep copying of read results using
`JSON.parse(JSON.stringify(data))` with the `transform` `operation:
'read'` function which converts Date's, ObjectID's in relationships /
joins to strings. As before, it also handles transformations for write
operations.
- Faster `hasNearConstraint` for potentially large `where`'s
- `traverseFields` now can accept `flattenedFields` which we use in
`transform`. Less recursive calls with tabs/rows/collapsible

Additional fixes
- Uses current transaction for querying nested relationships properties
in `buildQuery`, previously it wasn't used which could've led to wrong
results
- Allows to clear not required point fields with passing `null` from the
Local API. Previously it didn't work in both, MongoDB and Postgres

Benchmarks using this file
https://github.com/payloadcms/payload/blob/chore/db-benchmark/test/_community/int.spec.ts

### Small Dataset Performance

| Metric | Before Optimization | After Optimization | Improvement (%) |

|---------------------------|---------------------|--------------------|-----------------|
| Average FULL (ms) | 1170 | 844 | 27.86% |
| `payload.db.create` (ms) | 1413 | 691 | 51.12% |
| `payload.db.find` (ms) | 2856 | 2204 | 22.83% |
| `payload.db.deleteMany` (ms) | 15206 | 8439 | 44.53% |
| `payload.db.updateOne` (ms) | 21444 | 12162 | 43.30% |
| `payload.db.findOne` (ms) | 159 | 112 | 29.56% |
| `payload.db.deleteOne` (ms) | 3729 | 2578 | 30.89% |
| DB small FULL (ms) | 64473 | 46451 | 27.93% |

---

### Medium Dataset Performance

| Metric | Before Optimization | After Optimization | Improvement (%) |

|---------------------------|---------------------|--------------------|-----------------|
| Average FULL (ms) | 9407 | 6210 | 33.99% |
| `payload.db.create` (ms) | 10270 | 4321 | 57.93% |
| `payload.db.find` (ms) | 20814 | 16036 | 22.93% |
| `payload.db.deleteMany` (ms) | 126351 | 61789 | 51.11% |
| `payload.db.updateOne` (ms) | 201782 | 99943 | 50.49% |
| `payload.db.findOne` (ms) | 1081 | 817 | 24.43% |
| `payload.db.deleteOne` (ms) | 28534 | 23363 | 18.12% |
| DB medium FULL (ms) | 519518 | 342194 | 34.13% |

---

### Large Dataset Performance

| Metric | Before Optimization | After Optimization | Improvement (%) |

|---------------------------|---------------------|--------------------|-----------------|
| Average FULL (ms) | 26575 | 17509 | 34.14% |
| `payload.db.create` (ms) | 29085 | 12196 | 58.08% |
| `payload.db.find` (ms) | 58497 | 43838 | 25.04% |
| `payload.db.deleteMany` (ms) | 372195 | 173218 | 53.47% |
| `payload.db.updateOne` (ms) | 544089 | 288350 | 47.00% |
| `payload.db.findOne` (ms) | 3058 | 2197 | 28.14% |
| `payload.db.deleteOne` (ms) | 82444 | 64730 | 21.49% |
| DB large FULL (ms) | 1461097 | 969714 | 33.62% |

2024-12-19 13:20:39 -05:00

_community

feat: export sanitizeSelectParam, sanitizePopulateParam, senitizeJoinParams utils (#9777 )

2024-12-06 13:30:04 -05:00

access-control

fix: forgotPassword set expiration time (#9871 )

2024-12-11 08:43:22 -05:00

admin

test: consolidates custom id e2e tests (#10061 )

2024-12-18 22:44:46 +00:00

admin-root

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

app

feat: deprecates getPayloadHMR in favor of simpler getPayload (#9249 )

2024-11-16 15:30:05 -05:00

array-update

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

auth

perf: reduce generated types for select by respecting interfaceName (#9870 )

2024-12-16 17:22:17 +02:00

auth-basic

perf: faster page navigation by speeding up createClientConfig, speed up version fetching, speed up lexical init. Up to 100x faster (#9457 )

2024-11-26 14:31:14 -07:00

collections-graphql

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

collections-rest

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

config

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

create-payload-app

chore(eslint): FlatConfig type deprecated, set to Config

2024-09-20 22:46:40 -04:00

custom-graphql

feat(db-sqlite): add idType: 'uuid' support (#10016 )

2024-12-18 22:44:04 -05:00

database

feat(db-postgres, db-sqlite): drizzle schema generation (#9953 )

2024-12-19 11:08:17 -05:00

dataloader

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

email-nodemailer

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

email-resend

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

endpoints

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

field-error-states

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

field-perf

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

fields

perf(db-mongodb): improve performance of all operations, up to 50% faster (#9594 )

2024-12-19 13:20:39 -05:00

fields-relationship

perf(db-mongodb): improve performance of all operations, up to 50% faster (#9594 )

2024-12-19 13:20:39 -05:00

globals

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

graphql

feat(graphql): graphQL custom field complexity and validationRules (#9955 )

2024-12-13 15:03:57 -05:00

graphql-schema-gen

perf: reduce generated types for select by respecting interfaceName (#9870 )

2024-12-16 17:22:17 +02:00

helpers

feat(db-postgres, db-sqlite): drizzle schema generation (#9953 )

2024-12-19 11:08:17 -05:00

hooks

fix: beforeValidate previousValue argument (#10022 )

2024-12-17 12:08:40 -05:00

i18n

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

import-test

feat: adds jobs queue (#8228 )

2024-10-30 17:56:50 +00:00

joins

perf(db-mongodb): improve performance of all operations, up to 50% faster (#9594 )

2024-12-19 13:20:39 -05:00

lexical-mdx

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

live-preview

fix(next): does not format top-level domains within admin.preview or livePreview.url functions (#9831 )

2024-12-09 11:54:20 -05:00

loader

feat!: beta-next (#7620 )

2024-08-13 12:54:33 -04:00

localization

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

localization-rtl

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

locked-documents

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

chore: better default for useAsTitle with custom auth collections (#9841 )

2024-12-10 08:42:23 -05:00

migrations-cli

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

nested-fields

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

payload-cloud

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

plugin-cloud-storage

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

plugin-form-builder

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

plugin-nested-docs

fix(plugin-nested-docs): publishing parent doc should not publish child doc (#9958 )

2024-12-13 12:04:07 -05:00

plugin-redirects

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

plugin-search

fix(plugin-search): delete proper search document when doc has the same value but different relationTo (#9616 )

2024-11-29 21:25:26 +02:00

plugin-sentry

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

plugin-seo

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

plugin-stripe

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

plugins

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

queues

feat: allow where in payload.jobs.run (#9877 )

2024-12-11 00:33:53 +00:00

relationships

feat(db-postgres, db-sqlite): drizzle schema generation (#9953 )

2024-12-19 11:08:17 -05:00

scripts

feat(next): root admin (#7276 )

2024-07-23 13:44:44 -04:00

select

perf(db-mongodb): improve performance of all operations, up to 50% faster (#9594 )

2024-12-19 13:20:39 -05:00

sort

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

storage-azure

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

storage-gcs

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

storage-s3

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

storage-uploadthing

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

storage-vercel-blob

feat(storage-vercel-blob): allow fallback to disk if token not set (#10005 )

2024-12-16 21:40:22 +00:00

types

ci: add types testing with tstyche (#9803 )

2024-12-07 16:38:25 +02:00

uploads

chore: update generated types for all test suites (#9577 )

2024-11-27 20:36:37 +00:00

versions

feat(next): supports relative preview URLs (#9755 )

2024-12-04 17:01:09 -05:00

.prettierignore

chore: regenerate all types in test dir, and add to eslint & prettier ignores

2024-07-11 15:59:38 -04:00

.swcrc

chore: swcrc syntax fix (#6505 )

2024-05-25 15:45:05 +00:00

buildConfigWithDefaults.ts

fix(richtext-lexical): richtext fields in drawers aren't editable, inline toolbar artifacts are shown for readOnly editors (#8774 )

2024-10-18 05:38:48 +00:00

credentials.ts

chore: hoists tests out of payload package

2023-09-01 14:45:41 -04:00

CustomDashboard.tsx

chore: safely uses deepMerge

2024-03-07 11:33:46 -05:00

dev.ts

feat!: on demand rsc (#8364 )

2024-11-11 13:59:05 -05:00

eslint.config.js

chore(eslint): FlatConfig type deprecated, set to Config

2024-09-20 22:46:40 -04:00

generateDatabaseAdapter.ts

feat(db-sqlite): add idType: 'uuid' support (#10016 )

2024-12-18 22:44:04 -05:00

generateDatabaseSchema.ts

feat(db-postgres, db-sqlite): drizzle schema generation (#9953 )

2024-12-19 11:08:17 -05:00

generateGraphQLSchema.ts

feat!: beta-next (#7620 )

2024-08-13 12:54:33 -04:00

generateImportMap.ts

chore: fix live-preview tests against prod (#9122 )

2024-11-11 19:28:55 -07:00

generateTypes.ts

chore: gitignore test/databaseAdapter (#8235 )

2024-09-16 17:02:08 +00:00

helpers.ts

perf: faster page navigation by speeding up createClientConfig, speed up version fetching, speed up lexical init. Up to 100x faster (#9457 )

2024-11-26 14:31:14 -07:00

initDevAndTest.ts

chore: gitignore test/databaseAdapter (#8235 )

2024-09-16 17:02:08 +00:00

jest-spec-reporter.cjs

feat!: on demand rsc (#8364 )

2024-11-11 13:59:05 -05:00

jest.config.js

chore: use custom jest reporter to achieve sane jest logs (#8607 )

2024-10-11 18:54:39 +00:00

jest.setup.js

test: revert default db adapter in integration tests to mongodb (#10079 )

2024-12-19 18:20:28 +00:00

jestreporter.cjs

chore: use custom jest reporter to achieve sane jest logs (#8607 )

2024-10-11 18:54:39 +00:00

next-env.d.ts

build: fix tsconfig monorepo setup (#10028 )

2024-12-17 14:49:29 -05:00

next.config.mjs

feat!: on demand rsc (#8364 )

2024-11-11 13:59:05 -05:00

package.json

test: mock nodemailer network calls (#9957 )

2024-12-13 11:31:24 -05:00

playwright.bail.config.ts

chore: hide test flakes, improve playwright CI logs, significantly reduce playwright timeouts, add back test retries, cache playwright browsers in CI, disable CI telemetry, improve test throttle utility (#6155 )

2024-05-01 17:35:41 -04:00

playwright.config.ts

feat!: on demand rsc (#8364 )

2024-11-11 13:59:05 -05:00

runE2E.ts

feat!: on demand rsc (#8364 )

2024-11-11 13:59:05 -05:00

runInit.ts

chore: upgrade to pnpm v9, regenerate lockfile (#7369 )

2024-08-14 08:57:04 -04:00

runInitSeparateProcess.ts

chore: upgrade to pnpm v9, regenerate lockfile (#7369 )

2024-08-14 08:57:04 -04:00

safelyRunScript.ts

chore: when dev server restarts, ensure exiting the parent process kills child process (#8703 )

2024-10-14 20:02:26 +00:00

setupProd.ts

feat!: rename @payloadcms/plugin-cloud (#8828 )

2024-10-24 21:19:15 -04:00

test.env

chore: ensure node deprecation warnings stay hidden during e2e test runs

2024-04-06 15:06:04 -04:00

testEmailAdapter.ts

feat!: prebundle payload, ui, richtext-lexical (#6579 )

2024-06-17 14:25:36 -04:00

testHooks.ts

build: fix tsconfig monorepo setup (#10028 )

2024-12-17 14:49:29 -05:00

tsconfig.eslint.json

chore: run lint & prettier on everything

2024-03-14 23:53:47 -04:00

tsconfig.json

build: fix tsconfig monorepo setup (#10028 )

2024-12-17 14:49:29 -05:00

tsconfig.typecheck.json

feat!: rename @payloadcms/plugin-cloud (#8828 )

2024-10-24 21:19:15 -04:00