### What?
The PR #12763 added significant improvements for third-party databases
that are compatible with the MongoDB API. While the original PR was
focused on Firestore, other databases like DocumentDB also benefit from
these compatibility features.
In particular, the aggregate JOIN strategy does not work on AWS
DocumentDB and thus needs to be disabled. The current PR aims to provide
this as a sensible default in the `compatibilityOptions` that are
provided by Payload out-of-the-box.
As a bonus, it also fixes a small typo from `compat(a)bility` to
`compat(i)bility`.
### Why?
Because our Payload instance, which is backed by AWS DocumentDB, crashes
upon trying to access any `join` field.
### How?
By adding the existing `useJoinAggregations` with value `false` to the
compatiblity layer. Individual developers can still choose to override
it in their own local config as needed.
### What?
Adds four more arguments to the `mongooseAdapter`:
```typescript
useJoinAggregations?: boolean /* The big one */
useAlternativeDropDatabase?: boolean
useBigIntForNumberIDs?: boolean
usePipelineInSortLookup?: boolean
```
Also export a new `compatabilityOptions` object from
`@payloadcms/db-mongodb` where each key is a mongo-compatible database
and the value is the recommended `mongooseAdapter` settings for
compatability.
### Why?
When using firestore and visiting
`/admin/collections/media/payload-folders`, we get:
```
MongoServerError: invalid field(s) in lookup: [let, pipeline], only lookup(from, localField, foreignField, as) is supported
```
Firestore doesn't support the full MongoDB aggregation API used by
Payload which gets used when building aggregations for populating join
fields.
There are several other compatability issues with Firestore:
- The invalid `pipeline` property is used in the `$lookup` aggregation
in `buildSortParams`
- Firestore only supports number IDs of type `Long`, but Mongoose
converts custom ID fields of type number to `Double`
- Firestore does not support the `dropDatabase` command
- Firestore does not support the `createIndex` command (not addressed in
this PR)
### How?
```typescript
useJoinAggregations?: boolean /* The big one */
```
When this is `false` we skip the `buildJoinAggregation()` pipeline and resolve the join fields through multiple queries. This can potentially be used with AWS DocumentDB and Azure Cosmos DB to support join fields, but I have not tested with either of these databases.
```typescript
useAlternativeDropDatabase?: boolean
```
When `true`, monkey-patch (replace) the `dropDatabase` function so that
it calls `collection.deleteMany({})` on every collection instead of
sending a single `dropDatabase` command to the database
```typescript
useBigIntForNumberIDs?: boolean
```
When `true`, use `mongoose.Schema.Types.BigInt` for custom ID fields of type `number` which converts to a firestore `Long` behind the scenes
```typescript
usePipelineInSortLookup?: boolean
```
When `false`, modify the sortAggregation pipeline in `buildSortParams()` so that we don't use the `pipeline` property in the `$lookup` aggregation. Results in slightly worse performance when sorting by relationship properties.
### Limitations
This PR does not add support for transactions or creating indexes in firestore.
### Fixes
Fixed a bug (and added a test) where you weren't able to sort by multiple properties on a relationship field.
### Future work
1. Firestore supports simple `$lookup` aggregations but other databases might not. Could add a `useSortAggregations` property which can be used to disable aggregations in sorting.
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Sasha <64744993+r1tsuu@users.noreply.github.com>
Payload is designed with performance in mind, but its customizability
means that there are many ways to configure your app that can impact
performance.
While Payload provides several features and best practices to help you
optimize your app's specific performance needs, these are not currently
well surfaced and can be obscure.
Now:
- A high-level performance doc now exists at `/docs/performance`
- There's a new section on performance within the `/docs/queries` doc
- There's a new section on performance within the `/docs/hooks` doc
- There's a new section on performance within the
`/docs/custom-components` doc
This PR also:
- Restructures and elaborates on the `/docs/queries/pagination` docs
- Adds a new `/docs/database/indexing` doc
- More
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1210743577153856
Fixes https://github.com/payloadcms/payload/issues/6884
Adds a new flag `acceptIDOnCreate` that allows you to thread your own
`id` to `payload.create` `data`, for example:
```ts
// doc created with id 1
const doc = await payload.create({ collection: 'posts', data: {id: 1, title: "my title"}})
```
```ts
import { Types } from 'mongoose'
const id = new Types.ObjectId().toHexString()
const doc = await payload.create({ collection: 'posts', data: {id, title: "my title"}})
```
This change makes so that data that exists in MongoDB but isn't defined
in the Payload config won't be included to `payload.find` /
`payload.db.find` calls. Now we strip all the additional keys.
Consider you have a field named `secretField` that's also `hidden: true`
(or `read: () => false`) that contains some sensitive data. Then you
removed this field from the database and as for now with the MongoDB
adapter this field will be included to the Local API / REST API results
without any consideration, as Payload doesn't know about it anymore.
This also fixes https://github.com/payloadcms/payload/issues/11542 if
you removed / renamed a relationship field from the schema, Payload
won't sanitize ObjectIDs back to strings anymore.
Ideally you should create a migration script that completely removes the
deleted field from the database with `$unset`, but people rarely do
this.
If you still need to keep those fields to the result, this PR allows you
to do this with the new `allowAdditionalKeys: true` flag.
Adds the ability to pass additional schema options for collections with:
```ts
mongooseAdapter({
collectionsSchemaOptions: {
posts: {
strict: false,
},
},
})
```
This changes relates to these:
- https://github.com/payloadcms/payload/issues/4533
- https://github.com/payloadcms/payload/discussions/4534
It is a proposal to set custom schema options for mongoose driver.
I understand this got introduced into `main` v2 after `beta` branch was
created so this feature got lost.
- [x] I have read and understand the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository.
<!--
Thank you for the PR! Please go through the checklist below and make
sure you've completed all the steps.
Please review the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository if you haven't already.
The following items will ensure that your PR is handled as smoothly as
possible:
- PR Title must follow conventional commits format. For example, `feat:
my new feature`, `fix(plugin-seo): my fix`.
- Minimal description explained as if explained to someone not
immediately familiar with the code.
- Provide before/after screenshots or code diffs if applicable.
- Link any related issues/discussions from GitHub or Discord.
- Add review comments if necessary to explain to the reviewer the logic
behind a change
### What?
### Why?
### How?
Fixes #
-->
---------
Co-authored-by: Sasha <64744993+r1tsuu@users.noreply.github.com>
## Description
V2 PR [here](https://github.com/payloadcms/payload/pull/7359)
- [x] I have read and understand the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository.
## Type of change
- [x] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [x] This change requires a documentation update
## Checklist:
- [x] Existing test suite passes locally with my changes
- [x] I have made corresponding changes to the documentation
* wip moves payload, user and data into partial req
* chore: adjust req type
* chore(next): installs sass and resolves type errors
* feat: working login route/view
* fix: me route
* chore(next): scaffolds access routes (#4562)
* chore(next): scaffolds admin layout and dashboard view (#4566)
* chore(next): builds initPage utility (#4589)
* feat(3.0): next route handlers (#4590)
* chore: removes old files
* chore(next): ssr list view (#4594)
* chore: removes old files
* chore: adjusts graphql file imports to align with new operation exports
* chore: allows for custom endpoints
* chore: cleanup
* chore(next): ssr edit view (#4614)
* chore(ui): ssr main nav (#4619)
* chore(next): ssr account view (#4620)
* chore(next): ssr auth views and document create (#4631)
* chore(next): ssr globals view (#4640)
* chore(next): scaffolds document layout (#4644)
* chore(next): ssr versions view (#4645)
* chore(next): ssr field conditions (#4675)
* chore(next): ssr field validations (#4700)
* chore(next): moves dashboard view into next dir
* chore(next): moves account view into next dir
* chore(next): moves global edit view into next dir
* chore(next): returns isolated configs and locale from initPage
* chore(next): ssr api view (#4721)
* feat: adds i18n functionality within Rest API, Local and Client contexts (#4749)
* chore: separate client translation groups with empty line
* chore: add missing translation used in db adapters
* chore: simplify next/routes export and import paths
* chore: renames PayloadT to Payload
* chore(next): custom views (#4748)
* chore: fix translation tsconfig
* chore: adjust other package ts-configs that rely on translations
* chore(next): installs @payloadcms/ui as direct dependency
* chore(next): progress to build
* chore(next): migrates types (#4792)
* fixes acccept-language detection
* chore(next): moves remaining components out from payload core (#4794)
* chore(deps): removes all unused dependencies from payload core (#4797)
* chore(next): achieves buildable state (#4803)
* adds Translation component and removes more react-i18next
* fixes up remaining translation strings
* fixes a few i18n TODO's
* chore: remaining translation strings without colons
* chore: adds missing ja translations
* chore(next): ssr group field (#4830)
* chore: removes placeholder t function
* chore: removes old file
* chore(bundler-webpack): removes webpack bundler
* chore(bundler-vite): removes vite bundler
* chore(next): ssr tabs field (#4863)
* chore(next): ssr row field
* chore(next): ssr textarea field
* chore(next): wires server action into document edit view (#4873)
* chore(next): conditional logic (#4880)
* chore(next): ssr radio, point, code, json, ui, and hidden fields (#4891)
* chore(next): ssr collapsible field (#4894)
* chore: remove findByID from req
* chore: adjusts file property on request type
* comment clarification
* chore: wires up busboy with Requst readstream
* chore: ports over express-fileupload into a NextJS compatible format
* chore: adjust upload file structure
* chore: adds try/catch around routes, corrects a few route responses
* chore: renames file/function
* chore: improve req type safety in local operations, misc req.files replacements
* chore: misc type and fn export changes
* chore: ensures root routes take pass unmodified request to root routes
* chore: improve types
* chore: consolidates locale api req initialization (#4922)
* chore(next): overhauls field rendering strategy (#4924)
* chore(next): ssr array field (#4937)
* chore(next): ssr blocks field (#4942)
* chore(next): ssr upload field and document drawer (#4957)
* chore(next): wires form submissions (#4982)
* chore: api handler adjustments
* feat: adds graphql playground handler
* adds credentials include setting to playground
* remove old playground init, stub graphql handler location
* fix: allow for null fallbackLocale
* fix: correctly prioritize locales passed as null
* chore: move all graphql code into next package
* graphql changes
* chore: semi working version of graphql http layer
* gql fix attempts
* rm console log
* chore: partial gql changes
* chore: adds gql and gql-http back into payload
* chore: removes collection from req
* chore: separates graphql package out for schema generation
* chore: dep cleanup
* chore: move graphql handlers
* chore: removes unused deps
* chore(next): ssr list view (#5032)
* chore: refactor response handler order for custom endpoints
* chore: add back in condition for collection GET path with 2 slugs
* chore: rm optional chain
* chore: import sort route file
* chore: allows custom endpoints to attempt before erroring
* feat: adds memoization to translation functions (#5036)
* chore: fix APIError import
* chore: return attemptCustomEndpointBeforeError responses
* chore(next): properly instantiates table columns
* fix(next): attaches params to req and properly assigns prefs key (#5042)
* chore: reorganize next route order
* chore(next): adds RouteError handler to next routes
* chore: builds payload successfully
* chore: misc file omissions
* fix(ui): maintains proper column order
* fix(ui): ensures first cell is a link
* fix(next): properly copies url object in createPayloadRequest (#5064)
* fix(ui): bumps react-toastify to v10.0.4 to fix hydration warnings
* feat: add route for static file GET requests (#5065)
* chore(next): allows resolved config promise to be thread through initPage (#5071)
* chore(ui): conditionally renders field label from props
* feat(next): next install script
* chore: pass config to route handlers
* feat: initial test suite framework (#4929)
* chore(next): renderable account, api, and create first user views (#5084)
* fix(next): properly parses search params in find, update, and delete handlers (#5088)
* chore(next): ssr versions view (#5085)
* chore: adds homepage for scss testing
* chore: moves dev folder to top, establishes new test pattern
* chore: working turbopack
* chore: sets up working dynamic payload-config imports
* remove unused code
* chore: rm console log
* misc
* feat: correctly subs out ability to boot REST API within same process
* chore: WIP dev suites
* chore: removes need for REST_API folder in test dir
* removes duplicate bootAdminPanel fn
* misc
* specify default export
* chore: sets up jest to work with next/jest
* chore: progress to mongodb and sharp builds
* chore: passing community tests
* chore: sorta workin
* chore: adjust payload-config import
* chore: adds rest client for Next handlers
* chore: removes test garb
* chore: restores payload-config tsconfig path temporarily
* chore: establishes pattern for memory db during tests
* chore: bumps mongoose to 7
* chore(next): 404s on nested create urls
* chore: functional _community e2e
* chore: increases e2e expect timeout
* fix(next): sanitizes locale toString from client config
* chore: type fixes
* chore: pulls mongodb from main
* chore: uses graphql to log user in
* feat: passing auth test suite
* chore(ui): threads params through context and conditionally renders document tabs (#5094)
* feat(ui): adds params context (#5095)
* chore: removes unecessary memory allocation for urlPropertiesObject object
* chore: passing graphql test suite
* chore: removes references to bson
* chore: re-enables mongodb memory server for auth test suite
* chore: replace bson with bson-objectid
* feat: passing collections-rest int suite
* chore: fixes bad imports
* chore: more passing int suites
* feat: passing globals int tests
* feat: passing hooks int test suite
* chore: remove last express file
* chore: start live-preview int test migration
* chore: passing localization int tests
* passing relationships int tests
* chore: partial passing upload int tests
* chore: fixes scss imports
* chore(ui): renders document info provider at root (#5106)
* chore: adds schema path to useFieldPath provider, more passing tests
* chore: begins work to optimize translation imports
* chore: add translations to ui ts-config references
* chore: add exports folder to package json exports
* chore: adds readme how-to-use instructions
* chore: attempts refactor of translation imports
* chore: adds authentication:account translation key to server keys
* chore: finishes translation optimization
* chore: ignores warnings from mongodb
* chore(ui): renders live document title (#5115)
* chore(ui): ssr document tabs (#5116)
* chore: handles redirecting from login
* chore: handle redirect with no searchParams
* chore: handle missing segments
* chore(next): migrates server action into standalone api endpoint (#5122)
* chore: adjust dashboard colection segments
* test: update e2e suites
* fix(ui): prevents unnecessary calls to form state
* chore: fix finding global config fields from schema path
* fix(next): executes root POST endpoints
* chore(ui): ignores values returned by form state polling
* chore: scaffolds ssr rte
* chore: renders client leaves
* chore: server-side rendered rich text elements
* chore: defines ClientFunction pattern
* chore(ui): migrates relationship field
* chore: adds translations, cleans up slate
* chore: functional slate link
* chore: slate upload ssr
* chore: relationship slate ssr
* chore: remaining slate ssr
* chore: fixes circular workspace dep
* chore: correct broken int test import paths
* chore: remove media files from root
* chore: server renders custom edit view
* fix(ui): resolves infinite loading in versions view
* fix(next): resolves global edit view lookup
* chore: payload builds
* chore: delete unused files
* chore: removes local property from payload
* chore: adds mongodb as dev dep in db-mongodb package
* chore: hide deprecation warnings for tempfile and jest-environment-jsdom
* chore: remove all translations from translations dist
* chore: clean ts-config files
* chore: simple type fixes
* chore(ui): server renders custom list view
* chore: fix next config payload-config alias
* chore: adds turbo alias paths
* chore: adjusts translation generation
* chore: improve auth function
* chore: eslint config for packages/ui
* chore(ui): exports FormState
* chore(next): migrates account view to latest patterns
* chore: disable barbie mode
* chore(ui): lints
* chore(next): lints
* chore: for alexical
* chore: custom handler type signature adjustment
* fix: non-boolean condition result causes infinite looping (#4579)
* chore(richtext-lexical): upgrade lexical from v0.12.5 to v0.12.6 (#4732)
* chore(richtext-lexical): upgrade all lexical packages from 0.12.5 to 0.12.6
* fix(richtext-lexical): fix TypeScript errors
* fix indenting
* feat(richtext-lexical): Blocks: generate type definitions for blocks fields (#4529)
* feat(richtext-lexical)!: Update lexical from 0.12.6 to 0.13.1, port over all useful changes from playground (#5066)
* feat(richtext-lexical): Update lexical from 0.12.6 to 0.13.1, port over all useful changes from playground
* chore: upgrade lexical version used in monorepo
* chore: remove the 3
* chore: upgrade nodemon versions (#5059)
* feat: add more options to addFieldStatePromise so that it can be used for field flattening (#4799)
* feat(plugin-seo)!: remove support for payload <2.7.0 (#4765)
* chore(plugin-seo): remove test script from package.json (#4762)
* chore: upgrade @types/nodemailer from v6.4.8 to v6.4.14 (#4733)
* chore: revert auth and initPage changes
* chore(next): moves edit and list views (#5170)
* fix: "The punycode module is deprecated" warning by updating nodemailer
* chore: adjust translations tsconfig paths in root
* chore: fix merge build
---------
Co-authored-by: Jarrod Flesch <jarrodmflesch@gmail.com>
Co-authored-by: Jacob Fletcher <jacobsfletch@gmail.com>
Co-authored-by: Jarrod Flesch <30633324+JarrodMFlesch@users.noreply.github.com>
Co-authored-by: Elliot DeNolf <denolfe@gmail.com>
Co-authored-by: James <james@trbl.design>
Co-authored-by: Alessio Gravili <alessio@gravili.de>
Co-authored-by: Alessio Gravili <70709113+AlessioGr@users.noreply.github.com>
