Commit Graph

6 Commits

Author SHA1 Message Date
Elliott W
41cff6d436 fix(db-mongodb): improve compatibility with Firestore database (#12763)
### What?

Adds four more arguments to the `mongooseAdapter`:

```typescript
  useJoinAggregations?: boolean  /* The big one */
  useAlternativeDropDatabase?: boolean
  useBigIntForNumberIDs?: boolean
  usePipelineInSortLookup?: boolean
```

Also export a new `compatabilityOptions` object from
`@payloadcms/db-mongodb` where each key is a mongo-compatible database
and the value is the recommended `mongooseAdapter` settings for
compatability.

### Why?

When using firestore and visiting
`/admin/collections/media/payload-folders`, we get:

```
MongoServerError: invalid field(s) in lookup: [let, pipeline], only lookup(from, localField, foreignField, as) is supported
```

Firestore doesn't support the full MongoDB aggregation API used by
Payload which gets used when building aggregations for populating join
fields.

There are several other compatability issues with Firestore:
- The invalid `pipeline` property is used in the `$lookup` aggregation
in `buildSortParams`
- Firestore only supports number IDs of type `Long`, but Mongoose
converts custom ID fields of type number to `Double`
- Firestore does not support the `dropDatabase` command
- Firestore does not support the `createIndex` command (not addressed in
this PR)

### How?

 ```typescript
useJoinAggregations?: boolean  /* The big one */
```
When this is `false` we skip the `buildJoinAggregation()` pipeline and resolve the join fields through multiple queries. This can potentially be used with AWS DocumentDB and Azure Cosmos DB to support join fields, but I have not tested with either of these databases.

 ```typescript
useAlternativeDropDatabase?: boolean
```
When `true`, monkey-patch (replace) the `dropDatabase` function so that
it calls `collection.deleteMany({})` on every collection instead of
sending a single `dropDatabase` command to the database

 ```typescript
useBigIntForNumberIDs?: boolean
```
When `true`, use `mongoose.Schema.Types.BigInt` for custom ID fields of type `number` which converts to a firestore `Long` behind the scenes

```typescript
  usePipelineInSortLookup?: boolean
```
When `false`, modify the sortAggregation pipeline in `buildSortParams()` so that we don't use the `pipeline` property in the `$lookup` aggregation. Results in slightly worse performance when sorting by relationship properties.

### Limitations

This PR does not add support for transactions or creating indexes in firestore.

### Fixes

Fixed a bug (and added a test) where you weren't able to sort by multiple properties on a relationship field.

### Future work

1. Firestore supports simple `$lookup` aggregations but other databases might not. Could add a `useSortAggregations` property which can be used to disable aggregations in sorting.

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Sasha <64744993+r1tsuu@users.noreply.github.com>
2025-07-16 15:17:43 -04:00
James
d993f9ac64 chore: bug in isMongoose 2024-04-01 17:46:42 -04:00
James
c461a7fa15 chore: renames mongoose db adapter refs to mongodb 2024-04-01 17:24:43 -04:00
Elliot DeNolf
c5ecf48d94 chore: add test/ to workspace, update most references 2024-03-19 00:59:56 -04:00
Alessio Gravili
8895f6420f chore: fix all esm test suite imports 2024-03-08 14:42:24 -05:00
Alessio Gravili
17f7b94555 chore: improve test suites, upgrade jest and playwright, add debug utilities for lexical (#4011)
* feat(richtext-lexical): 'bottom' position value for plugins

* feat: TestRecorderFeature

* chore: restructuring to seed and clear db before each test

* chore: make sure all tests pass

* chore: make sure indexes are created in seed.ts - this fixes one erroring test

* chore: speed up test runs through db snapshots

* chore: support drizzle when resetting db

* chore: simplify seeding process, by moving boilerplate db reset / snapshot logic into a wrapper function

* chore: add new seeding process to admin test suite

* chore(deps): upgrade jest and playwright

* chore: make sure mongoose-specific tests are not skipped

* chore: fix point test, which was depending on another test (that's bad!)

* chore: fix incorrect import

* chore: remove unnecessary comments

* chore: clearly label lexicalE2E test file as todo

* chore: simplify seed logic

* chore: move versions test suite to new seed system
2023-11-06 16:38:40 +01:00