- Adds documentdb and cosmosdb to CI test matrix
- Adds missing generateDatabaseAdapter configs
- Adjust generateDatabaseAdapter firestore config to make it pure to the
compatibilityOptions we provide as an export
Creating as draft because I expect a few tests to fail based on previous
comments.
---------
Co-authored-by: Sasha <64744993+r1tsuu@users.noreply.github.com>
### What?
This PR adds atomic array operations ($append and $remove) for
relationship fields with `hasMany: true` across all database adapters.
These operations allow developers to add or remove specific items from
relationship arrays without replacing the entire array.
New API:
```
// Append relationships (prevents duplicates)
await payload.db.updateOne({
collection: 'posts',
id: 'post123',
data: {
categories: { $append: ['featured', 'trending'] }
}
})
// Remove specific relationships
await payload.db.updateOne({
collection: 'posts',
id: 'post123',
data: {
tags: { $remove: ['draft', 'private'] }
}
})
// Works with polymorphic relationships
await payload.db.updateOne({
collection: 'posts',
id: 'post123',
data: {
relatedItems: {
$append: [
{ relationTo: 'categories', value: 'category-id' },
{ relationTo: 'tags', value: 'tag-id' }
]
}
}
})
```
### Why?
Currently, updating relationship arrays requires replacing the entire
array which requires fetching existing data before updates. Requiring
more implementation effort and potential for errors when using the API,
in particular for bulk updates.
### How?
#### Cross-Adapter Features:
- Polymorphic relationships: Full support for relationTo:
['collection1', 'collection2']
- Localized relationships: Proper locale handling when fields are
localized
- Duplicate prevention: Ensures `$append` doesn't create duplicates
- Order preservation: Appends to end of array maintaining order
- Bulk operations: Works with `updateMany` for bulk updates
#### MongoDB Implementation:
- Converts `$append` to native `$addToSet` (prevents duplicates in
contrast to `$push`)
- Converts `$remove` to native `$pull` (targeted removal)
#### Drizzle Implementation (Postgres/SQLite):
- Uses optimized batch `INSERT` with duplicate checking for `$append`
- Uses targeted `DELETE` queries for `$remove`
- Implements timestamp-based ordering for performance
- Handles locale columns conditionally based on schema
### Limitations
The current implementation is only on database-adapter level and not
(yet) for the local API. Implementation in the localAPI will be done
separately.
This PR adds support for the following configuration:
```ts
const config = {
collections: [
{
slug: 'categories',
fields: [
{
name: 'title',
type: 'text',
},
],
},
{
slug: 'posts',
fields: [
{
name: 'title',
type: 'text',
},
{
name: 'categories',
type: 'relationship',
relationTo: 'categories',
hasMany: true,
},
],
},
{
slug: 'examples',
fields: [
{
name: 'postCategoriesTitles',
type: 'text',
virtual: 'post.categories.title',
// hasMany: true - added automatically during the sanitization
},
{
type: 'relationship',
relationTo: 'posts',
name: 'post',
},
{
name: 'postsTitles',
type: 'text',
virtual: 'posts.title',
// hasMany: true - added automatically during the sanitization
},
{
type: 'relationship',
relationTo: 'posts',
name: 'posts',
hasMany: true,
},
],
},
],
}
```
In the result:
`postsTitles` - will be always populated with an array of posts titles.
`postCategoriesTitles` - will be always populated with an array of the
categories titles that are related to this post
The virtual `text` field is sanitizated to `hasMany: true`
automatically, but you can specify that manually as well.
This PR adds **atomic** `$push` **support for array fields**. It makes
it possible to safely append new items to arrays, which is especially
useful when running tasks in parallel (like job queues) where multiple
processes might update the same record at the same time. By handling
pushes atomically, we avoid race conditions and keep data consistent -
especially on postgres, where the current implementation would nuke the
entire array table before re-inserting every single array item.
The feature works for both localized and unlocalized arrays, and
supports pushing either single or multiple items at once.
This PR is a requirement for reliably running parallel tasks in the job
queue - see https://github.com/payloadcms/payload/pull/13452.
Alongside documenting `$push`, this PR also adds documentation for
`$inc`.
## Changes to updatedAt behavior
https://github.com/payloadcms/payload/pull/13335 allows us to override
the updatedAt property instead of the db always setting it to the
current date.
However, we are not able to skip updating the updatedAt property
completely. This means, usage of $push results in 2 postgres db calls:
1. set updatedAt in main row
2. append array row in arrays table
This PR changes the behavior to only automatically set updatedAt if it's
undefined. If you explicitly set it to `null`, this now allows you to
skip the db adapter automatically setting updatedAt.
=> This allows us to use $push in just one single db call
## Usage Examples
### Pushing a single item to an array
```ts
const post = (await payload.db.updateOne({
data: {
array: {
$push: {
text: 'some text 2',
id: new mongoose.Types.ObjectId().toHexString(),
},
},
},
collection: 'posts',
id: post.id,
}))
```
### Pushing a single item to a localized array
```ts
const post = (await payload.db.updateOne({
data: {
arrayLocalized: {
$push: {
en: {
text: 'some text 2',
id: new mongoose.Types.ObjectId().toHexString(),
},
es: {
text: 'some text 2 es',
id: new mongoose.Types.ObjectId().toHexString(),
},
},
},
},
collection: 'posts',
id: post.id,
}))
```
### Pushing multiple items to an array
```ts
const post = (await payload.db.updateOne({
data: {
array: {
$push: [
{
text: 'some text 2',
id: new mongoose.Types.ObjectId().toHexString(),
},
{
text: 'some text 3',
id: new mongoose.Types.ObjectId().toHexString(),
},
],
},
},
collection: 'posts',
id: post.id,
}))
```
### Pushing multiple items to a localized array
```ts
const post = (await payload.db.updateOne({
data: {
arrayLocalized: {
$push: {
en: {
text: 'some text 2',
id: new mongoose.Types.ObjectId().toHexString(),
},
es: [
{
text: 'some text 2 es',
id: new mongoose.Types.ObjectId().toHexString(),
},
{
text: 'some text 3 es',
id: new mongoose.Types.ObjectId().toHexString(),
},
],
},
},
},
collection: 'posts',
id: post.id,
}))
```
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1211110462564647
Previously, when manually setting `createdAt` or `updatedAt` in a
`payload.db.*` or `payload.*` operation, the value may have been
ignored. In some cases it was _impossible_ to change the `updatedAt`
value, even when using direct db adapter calls. On top of that, this
behavior sometimes differed between db adapters. For example, mongodb
did accept `updatedAt` when calling `payload.db.updateVersion` -
postgres ignored it.
This PR changes this behavior to consistently respect `createdAt` and
`updatedAt` values for `payload.db.*` operations.
For `payload.*` operations, this also works with the following
exception:
- update operations do no respect `updatedAt`, as updates are commonly
performed by spreading the old data, e.g. `payload.update({ data:
{...oldData} })` - in these cases, we usually still want the `updatedAt`
to be updated. If you need to get around this, you can use the
`payload.db.updateOne` operation instead.
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1210919646303994
Adds a new operation findDistinct that can give you distinct values of a
field for a given collection
Example:
Assume you have a collection posts with multiple documents, and some of
them share the same title:
```js
// Example dataset (some titles appear multiple times)
[
{ title: 'title-1' },
{ title: 'title-2' },
{ title: 'title-1' },
{ title: 'title-3' },
{ title: 'title-2' },
{ title: 'title-4' },
{ title: 'title-5' },
{ title: 'title-6' },
{ title: 'title-7' },
{ title: 'title-8' },
{ title: 'title-9' },
]
```
You can now retrieve all unique title values using findDistinct:
```js
const result = await payload.findDistinct({
collection: 'posts',
field: 'title',
})
console.log(result.values)
// Output:
// [
// 'title-1',
// 'title-2',
// 'title-3',
// 'title-4',
// 'title-5',
// 'title-6',
// 'title-7',
// 'title-8',
// 'title-9'
// ]
```
You can also limit the number of distinct results:
```js
const limitedResult = await payload.findDistinct({
collection: 'posts',
field: 'title',
sortOrder: 'desc',
limit: 3,
})
console.log(limitedResult.values)
// Output:
// [
// 'title-1',
// 'title-2',
// 'title-3'
// ]
```
You can also pass a `where` query to filter the documents.
This is a follow-up to https://github.com/payloadcms/payload/pull/13060.
There are a bunch of other db adapter methods that use `upsertRow` for
updates: `updateGlobal`, `updateGlobalVersion`, `updateJobs`,
`updateMany`, `updateVersion`.
The previous PR had the logic for using the optimized row updating logic
inside the `updateOne` adapter. This PR moves that logic to the original
`upsertRow` function. Benefits:
- all the other db methods will benefit from this massive optimization
as well. This will be especially relevant for optimizing postgres job
queue initial updates - we should be able to close
https://github.com/payloadcms/payload/pull/11865 after another follow-up
PR
- easier to read db adapter methods due to less code.
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1210803039809810
Based on https://github.com/payloadcms/payload/pull/13060 which should
be merged first
This PR adds ability to update number fields atomically, which could be
important with parallel writes. For now we support this only via
`payload.db.updateOne`.
For example:
```js
// increment by 10
const res = await payload.db.updateOne({
data: {
number: {
$inc: 10,
},
},
collection: 'posts',
where: { id: { equals: post.id } },
})
// decrement by 3
const res2 = await payload.db.updateOne({
data: {
number: {
$inc: -3,
},
},
collection: 'posts',
where: { id: { equals: post.id } },
})
```
In case, if `payload.db.updateOne` received simple data, meaning no:
* Arrays / Blocks
* Localized Fields
* `hasMany: true` text / select / number / relationship fields
* relationship fields with `relationTo` as an array
This PR simplifies the logic to a single SQL `set` call. No any extra
(useless) steps with rewriting all the arrays / blocks / localized
tables even if there were no any changes to them. However, it's good to
note that `payload.update` (not `payload.db.updateOne`) as for now
passes all the previous data as well, so this change won't have any
effect unless you're using `payload.db.updateOne` directly (or for our
internal logic that uses it), in the future a separate PR with
optimization for `payload.update` as well may be implemented.
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1210710489889576
Previously, `db.updateOne` calls with `where` queries that lead to no
results would create new rows on drizzle. Essentially, `db.updateOne`
behaved like `db.upsertOne` on drizzle
If you (using the MongoDB adapter) delete a block from the payload
config, but still have some data with that block in the DB, you'd
receive in the admin panel an error like:
```
Block with type "cta" was found in block data, but no block with that type is defined in the config for field with schema path pages.blocks
```
Now, we remove those "unknown" blocks at the DB adapter level.
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
We were running scripts as they were without encompassing our logic in a
function for jest's teardown and we were subsequently running
`process.exit(0)` which meant that tests didn't correctly return an
error status code when they failed in CI.
The following tests have been skipped as well:
```
● postgres vector custom column › should add a vector column and query it
● Sort › Local API › Orderable › should not break with existing base 62 digits
● Sort › Local API › Orderable join › should set order by default
● Sort › Local API › Orderable join › should allow setting the order with the local API
● Sort › Local API › Orderable join › should sort join docs in the correct
```
---------
Co-authored-by: Elliot DeNolf <denolfe@gmail.com>
Co-authored-by: Alessio Gravili <alessio@gravili.de>
Fixes an issue when querying deeply new relationship virtual fields with
`draft: true`. Changes the method for `where` sanitization, before it
was done in `validateSearchParam` which didn't work with versions
properly, now there's a separate `sanitizeWhereQuery` function that does
this.
Continuation of https://github.com/payloadcms/payload/pull/6245.
This PR allows you to pass `blocksAsJSON: true` to SQL adapters and the
adapter instead of aligning with the SQL preferred relation approach for
blocks will just use a simple JSON column, which can improve performance
with a large amount of blocks.
To try these changes you can install `3.43.0-internal.c5bbc84`.
Fixes https://github.com/payloadcms/payload/issues/12628
When using sqlite, the error from the db is a bit different than
Postgres.
This PR allows us to extract the fieldName when using sqlite for the
unique constraint error.
This clarifies that jobs.autoRun only *runs* already-queued jobs. It does not queue the jobs for you.
Also adds an e2e test as this functionality had no e2e coverage
Previously, this was possible in MongoDB but not in Postgres/SQLite
(having `null` in an `in` query)
```
const { docs } = await payload.find({
collection: 'posts',
where: { text: { in: ['text-1', 'text-3', null] } },
})
```
This PR fixes that behavior
Previously, `hidden: true` on a virtual field that references a
relationship field didn't work. Now, this field doesn't get calculated
if there's `hidden: true` and no `showHiddenFields` was passed.
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
Continuation of https://github.com/payloadcms/payload/pull/12265.
Currently, using `select` on new relationship virtual fields:
```
const doc = await payload.findByID({
collection: 'virtual-relations',
depth: 0,
id,
select: { postTitle: true },
})
```
doesn't work, because in order to calculate `post.title`, the `post`
field must be selected as well. This PR adds logic that sanitizes the
incoming `select` to include those relationships into `select` (that are
related to selected virtual fields)
---------
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
When doing `payload.db.queryDrafts` with `select` without `version`, or
simply your select looks like:
`select: { version: { nonExistingField: true } }` - the `queryDrafts`
function will crash because it tries to access the `version` field.
This PR adds a fallback.
This PR optimizes the new virtual fields with relationships feature
https://github.com/payloadcms/payload/pull/11805 when the path
references the ID field, for example:
```
{
name: 'postCategoryID',
type: 'number',
virtual: 'post.category.id',
},
```
Previously, we did additional population of `category`, which is
unnecessary as we can always grab the ID from the `category` value
itself. One less querying step.
This PR adds an ability to specify a virtual field in this way
```js
{
slug: 'posts',
fields: [
{
name: 'title',
type: 'text',
required: true,
},
],
},
{
slug: 'virtual-relations',
fields: [
{
name: 'postTitle',
type: 'text',
virtual: 'post.title',
},
{
name: 'post',
type: 'relationship',
relationTo: 'posts',
},
],
},
```
Then, every time you query `virtual-relations`, `postTitle` will be
automatically populated (even if using `depth: 0`) on the db level. This
field also, unlike `virtual: true` is available for querying / sorting /
`useAsTitle`.
Also, the field can be deeply nested to 2 or more relationships, for
example:
```
{
name: 'postCategoryTitle',
type: 'text',
virtual: 'post.category.title',
},
```
Where the current collection has `post` - a relationship to `posts`, the
collection `posts` has `category` that's a relationship to `categories`
and finally `categories` has `title`.
### What?
Converts numbers passed to a text field to avoid the database/drizzle
from converting it incorrectly.
### Why?
If you have a hook that passes a value to another field you can
experience this problem where drizzle converts a number value for a text
field to a floating point number in sqlite for example.
### How?
Adds logic to `transform/write/traverseFields.ts` to cast text field
values to string.
Fixes https://github.com/payloadcms/payload/issues/11975
Previously, this configuration was causing errors in postgres due to
long names, even though `dbName` is used:
```
{
slug: 'aliases',
fields: [
{
name: 'thisIsALongFieldNameThatWillCauseAPostgresErrorEvenThoughWeSetAShorterDBName',
dbName: 'shortname',
type: 'array',
fields: [
{
name: 'nested_field_1',
type: 'array',
dbName: 'short_nested_1',
fields: [],
},
{
name: 'nested_field_2',
type: 'text',
},
],
},
],
},
```
This is because we were generating Drizzle relation name (for arrays)
always based on the field path and internally, drizzle uses this name
for aliasing. Now, if `dbName` is present, we use `_{dbName}` instead
for the relation name.
Fixes https://github.com/payloadcms/payload/issues/6884
Adds a new flag `acceptIDOnCreate` that allows you to thread your own
`id` to `payload.create` `data`, for example:
```ts
// doc created with id 1
const doc = await payload.create({ collection: 'posts', data: {id: 1, title: "my title"}})
```
```ts
import { Types } from 'mongoose'
const id = new Types.ObjectId().toHexString()
const doc = await payload.create({ collection: 'posts', data: {id, title: "my title"}})
```
This change makes so that data that exists in MongoDB but isn't defined
in the Payload config won't be included to `payload.find` /
`payload.db.find` calls. Now we strip all the additional keys.
Consider you have a field named `secretField` that's also `hidden: true`
(or `read: () => false`) that contains some sensitive data. Then you
removed this field from the database and as for now with the MongoDB
adapter this field will be included to the Local API / REST API results
without any consideration, as Payload doesn't know about it anymore.
This also fixes https://github.com/payloadcms/payload/issues/11542 if
you removed / renamed a relationship field from the schema, Payload
won't sanitize ObjectIDs back to strings anymore.
Ideally you should create a migration script that completely removes the
deleted field from the database with `$unset`, but people rarely do
this.
If you still need to keep those fields to the result, this PR allows you
to do this with the new `allowAdditionalKeys: true` flag.
### What?
This PR adds ability to define indexes on several fields for collections
(compound indexes).
Example:
```ts
{
indexes: [{ unique: true, fields: ['title', 'group.name'] }]
}
```
### Why?
This can be used to either speed up querying/sorting by 2 or more fields
at the same time or to ensure uniqueness between several fields.
### How?
Implements this logic in database adapters. Additionally, adds a utility
`getFieldByPath`.
This PR adds a new `limit` property to `payload.db.updateMany`. This functionality is required for [migrating our job system to use faster, direct db adapter calls](https://github.com/payloadcms/payload/pull/11489)
### What?
If you had multiple operator constraints on a single field, the last one
defined would be the only one used.
Example:
```ts
where: {
id: {
in: [doc2.id],
not_in: [], // <-- only respected this operator constraint
},
}
```
and
```ts
where: {
id: {
not_in: [],
in: [doc2.id], // <-- only respected this operator constraint
},
}
```
They would yield different results.
### Why?
The results were not merged into an `$and` query inside parseParams.
### How?
Merges the results within an `$and` constraint.
Fixes https://github.com/payloadcms/payload/issues/10944
Supersedes https://github.com/payloadcms/payload/pull/11011
