Commit Graph

134 Commits

Author SHA1 Message Date
Sasha
09782be0e0 fix(db-postgres): long array field table aliases cause error even when dbName is used (#11995)
Fixes https://github.com/payloadcms/payload/issues/11975

Previously, this configuration was causing errors in postgres due to
long names, even though `dbName` is used:
```
{
  slug: 'aliases',
  fields: [
    {
      name: 'thisIsALongFieldNameThatWillCauseAPostgresErrorEvenThoughWeSetAShorterDBName',
      dbName: 'shortname',
      type: 'array',
      fields: [
        {
          name: 'nested_field_1',
          type: 'array',
          dbName: 'short_nested_1',
          fields: [],
        },
        {
          name: 'nested_field_2',
          type: 'text',
        },
      ],
    },
  ],
},
```

This is because we were generating Drizzle relation name (for arrays)
always based on the field path and internally, drizzle uses this name
for aliasing. Now, if `dbName` is present, we use `_{dbName}` instead
for the relation name.
2025-04-07 20:12:43 +00:00
Sasha
4ebd3ce668 fix(db-postgres): deleteOne fails when the where query does not resolve to any document (#11632)
Previously, if you called `payload.db.deleteOne` with a `where` query
that does not resolve to anything, an error would be occurred.
2025-04-04 00:46:31 +03:00
Sasha
f310c90211 fix(db-postgres): down migration fails because migrationTableExists doesn't check in the current transaction (#11910)
Fixes https://github.com/payloadcms/payload/issues/11882

Previously, down migration that dropped the `payload_migrations` table
was failing because `migrationTableExists` doesn't check the current
transaction, only in which you can get a `false` value result.
2025-04-03 02:33:34 +03:00
Sasha
dc793d1d14 fix: ValidationError error message when label is a function (#11904)
Fixes https://github.com/payloadcms/payload/issues/11901

Previously, when `ValidationError` `errors.path` was referring to a
field with `label` defined as a function, the error message was
generated with `[object Object]`. Now, we call that function instead.
Since the `i18n` argument is required for `StaticLabel`, this PR
introduces so you can pass a partial `req` to `ValidationError` from
which we thread `req.i18n` to the label args.
2025-04-03 00:38:54 +03:00
Alessio Gravili
f34eb228c4 feat(drizzle): export buildQuery and parseParams (#11935)
This exports `buildQuery` and `parseParams` from @payloadcms/drizzle
2025-04-02 18:17:39 +00:00
Alessio Gravili
9c88af4b20 refactor(drizzle): replace query chaining with dynamic query building (#11923)
This replaces usage of our `chainMethods` helper to dynamically chain
queries with [drizzle dynamic query
building](https://orm.drizzle.team/docs/dynamic-query-building).

This is more type-safe, more readable and requires less code
2025-03-31 20:37:45 +00:00
Alessio Gravili
9a1c3cf4cc fix: support parallel job queue tasks (#11917)
This adds support for running multiple job queue tasks in parallel
within the same workflow while preventing conflicts. Previously, this
would have caused the following issues:
- Job log entries get lost - the final job log is incomplete, despite
all tasks having been executed
- Write conflicts in postgres, leading to unique constraint violation
errors

The solution involves handling job log data updates in a way that avoids
overwriting, and ensuring the final update reflects the latest job log
data. Each job log entry now initializes its own ID, so a given job log
entry’s ID remains the same across multiple, parallel task executions.

## Postgres

In Postgres, we need to enable transactions for the
`payload.db.updateJobs` operation; otherwise, two tasks updating the
same job in parallel can conflict. This happens because Postgres handles
array rows by deleting them all, then re-inserting (rather than
upserting). The rows are stored in a separate table, and the following
scenario can occur:

Op 1: deletes all job log rows
Op 2: deletes all job log rows
Op 1: inserts 200 job log rows
Op 2: insert the same 200 job log rows again => `error: “duplicate key
value violates unique constraint "payload_jobs_log_pkey”`

Because transactions were not used, the rows inserted by Op 1
immediately became visible to Op 2, causing the conflict. Enabling
transactions fixes this. In theory, it can still happen if Op 1 commits
before Op 2 starts inserting (due to the read committed isolation
level), but it should occur far less frequently.

Alongside this change, we should consider inserting the rows using an
upsert (update on conflict), which will get rid of this error
completely. That way, if the insertion of Op 1 is visible to Op 2, Op 2
will simply overwrite it, rather than erroring. Individual job entries
are immutable and job entries cannot be deleted, thus this shouldn't
corrupt any data.

## Mongo

In Mongo, the issue is addressed by ensuring that log row deletions
caused due to different log states in concurrent operations are not
merged back to the client job log, and by making sure the final update
includes all job logs.

There is no duplicate key error in Mongo because the array log resides
in the same document and duplicates are simply upserted. We cannot use
transactions in Mongo, as it appears to lock the document in a way that
prevents reliable parallel updates, leading to:

`MongoServerError: WriteConflict error: this operation conflicted with
another operation. Please retry your operation or multi-document
transaction`
2025-03-31 13:06:05 -06:00
Alessio Gravili
a6f7ef837a feat(db-*): export types from main export (#11914)
In 3.0, we made the decision to export all types from the main package
export (e.g. `payload/types` => `payload`). This improves type
discoverability by IDEs and simplifies importing types.

This PR does the same for our db adapters, which still have a separate
`/types` subpath export. While those are kept for
backwards-compatibility, we can remove them in 4.0.
2025-03-31 15:45:02 +00:00
Alessio Gravili
a5c3aa0e4f perf: reduce job queue db calls (#11846)
Continuation of #11489. This adds a new, optional `updateJobs` db
adapter method that reduces the amount of database calls for the jobs
queue.

## MongoDB

### Previous: running a set of 50 queued jobs
- 1x db.find (= 1x `Model.paginate`)
- 50x db.updateOne (= 50x `Model.findOneAndUpdate`)

### Now: running a set of 50 queued jobs
- 1x db.updateJobs (= 1x `Model.find` and 1x `Model.updateMany`)

**=> 51 db round trips before, 2 db round trips after**


### Previous: upon task completion
- 1x db.find (= 1x `Model.paginate`)
- 1x db.updateOne (= 1x `Model.findOneAndUpdate`)

### Now: upon task completion
- 1x db.updateJobs (= 1x `Model.findOneAndUpdate`)


**=> 2 db round trips before, 1 db round trip after**


## Drizzle (e.g. Postgres)

### running a set of 50 queued jobs
 - 1x db.query[tablename].findMany
 - 50x db.select 
 - 50x upsertRow
 
This is unaffected by this PR and will be addressed in a future PR
2025-03-25 18:09:52 +00:00
Alessio Gravili
73fc3c607a perf(drizzle): remove unnecessary db.select call in updateOne operation (#11847)
This will improve performance when updating a single document in
postgres/drizzle, if the ID is known.

Previously, this resulted in 2 sequential operations:
- `db.select `to fetch the document by the ID
- `upsertRow` to update the document (multiple db operations)

This PR removes the unnecessary `db.select` call, as the document ID is
already known
2025-03-25 10:11:20 -06:00
Sasha
1b2b6a1b15 fix: respect draft: true when querying docs for the join field (#11763)
Previously, if you were querying a collection that has a join field with
`draft: true`, and the join field's collection also has
`versions.drafts: true` our db adapter would still query the original
SQL table / mongodb collection instead of the versions one which isn't
quite right since we respect `draft: true` when populating relationships
2025-03-24 09:49:30 -04:00
Alessio Gravili
e96d3c87e2 feat(db-*): support sort in db.updateMany (#11768)
This adds support for `sort` in `payload.db.updateMany`.

## Example

```ts
const updatedDocs = await payload.db.updateMany({
  collection: 'posts',
  data: {
    title: 'updated',
  },
  limit: 5,
  sort: '-numberField', // <= new
  where: {
    id: {
      exists: true,
    },
  },
})
```
2025-03-19 10:47:58 -06:00
Sasha
f442d22237 feat(db-*): allow to thread id to create operation data without custom IDs (#11709)
Fixes https://github.com/payloadcms/payload/issues/6884

Adds a new flag `acceptIDOnCreate` that allows you to thread your own
`id` to `payload.create` `data`, for example:

```ts
// doc created with id 1
const doc = await payload.create({ collection: 'posts', data: {id: 1, title: "my title"}})
```

```ts
import { Types } from 'mongoose'
const id = new Types.ObjectId().toHexString()
const doc = await payload.create({ collection: 'posts', data: {id, title: "my title"}})
```
2025-03-17 23:48:35 -04:00
Sasha
11d74871ef feat(db-postgres): add vector raw column type (#10422)
Example how you can add a vector column, enable the `vector` extension
and query your embeddings in the included test -

https://github.com/payloadcms/payload/compare/feat/more-types?expand=1#diff-7d876370487cb625eb42ff1ad7cffa78e8327367af3de2930837ed123f5e3ae6R1-R117
2025-03-17 20:50:00 +00:00
Sasha
bacc0f002a feat: compound indexes (#11512)
### What?
This PR adds ability to define indexes on several fields for collections
(compound indexes).

Example:
```ts
{
  indexes: [{ unique: true, fields: ['title', 'group.name'] }]
}
```

### Why?
This can be used to either speed up querying/sorting by 2 or more fields
at the same time or to ensure uniqueness between several fields.

### How?
Implements this logic in database adapters. Additionally, adds a utility
`getFieldByPath`.
2025-03-05 03:09:24 +02:00
Alessio Gravili
6a3d58bb32 feat(db-*): support limit in db.updateMany (#11488)
This PR adds a new `limit` property to `payload.db.updateMany`. This functionality is required for [migrating our job system to use faster, direct db adapter calls](https://github.com/payloadcms/payload/pull/11489)
2025-03-03 05:32:57 +00:00
Alessio Gravili
41c7413f59 feat(db-*): add updateMany method to database adapter (#11441)
This PR adds a new `payload.db.updateMany` method, which is a more performant way to update multiple documents compared to using `payload.update`.
2025-02-27 20:30:17 -07:00
Alessio Gravili
c21dac1b53 perf(db-*): add option to disable returning modified documents in db methods (#11393)
This PR adds a new `returning` option to various db adapter methods. Setting it to `false` where the return value is not used will lead to performance gains, as we don't have to do additional db calls to fetch the updated document and then sanitize it.
2025-02-27 17:40:22 -05:00
Sasha
3436fb16ea feat: allow to count related docs for join fields (#11395)
### What?
For the join field query adds ability to specify `count: true`, example:
```ts
const result = await payload.find({
  joins: {
    'group.relatedPosts': {
      sort: '-title',
      count: true,
    },
  },
  collection: "categories",
})

result.group?.relatedPosts?.totalDocs // available
```

### Why?
Can be useful to implement full pagination / show total related
documents count in the UI.

### How?
Implements the logic in database adapters. In MongoDB it's additional
`$lookup` that has `$count` in the pipeline. In SQL, it's additional
subquery with `COUNT(*)`. Preserves the current behavior by default,
since counting introduces overhead.


Additionally, fixes a typescript generation error for join fields.
Before, `docs` and `hasNextPage` were marked as nullable, which is not
true, these fields cannot be `null`.
Additionally, fixes threading of `joinQuery` in
`transform/read/traverseFields` for group / tab fields recursive calls.
2025-02-27 16:05:48 +00:00
Alessio Gravili
e4712a822b perf(drizzle): use faster, direct db query for getting id to update in updateOne (#11391)
Previously, `updateOne` was using `buildFindManyArgs` and `findFirst` just to retrieve the ID of the document to update, which is a huge function that's not necessary to run just to get the document ID.

This PR refactors it to use a simple `db.select` query to retrieve the ID
2025-02-25 21:54:38 -07:00
Paul
b014416584 feat: add support for not_like operation (#11326)
This PR adds support for `not_like` as a query operation. Functions just
as a negative to `like`, uses `ilike` in postgres and `like` in sqlite
2025-02-22 00:32:37 +00:00
Sasha
6d36a28cdc feat: join field across many collections (#10919)
This feature allows you to specify `collection` for the join field as
array.
This can be useful for example to describe relationship linking like
this:
```ts
{
  slug: 'folders',
  fields: [
    {
      type: 'join',
      on: 'folder',
      collection: ['files', 'documents', 'folders'],
      name: 'children',
    },
    {
      type: 'relationship',
      relationTo: 'folders',
      name: 'folder',
    },
  ],
},
{
  slug: 'files',
  upload: true,
  fields: [
    {
      type: 'relationship',
      relationTo: 'folders',
      name: 'folder',
    },
  ],
},
{
  slug: 'documents',
  fields: [
    {
      type: 'relationship',
      relationTo: 'folders',
      name: 'folder',
    },
  ],
},
```

Documents and files can be placed to folders and folders themselves can
be nested to other folders (root folders just have `folder` as `null`).

Output type of `Folder`:
```ts
export interface Folder {
  id: string;
  children?: {
    docs?:
      | (
          | {
              relationTo?: 'files';
              value: string | File;
            }
          | {
              relationTo?: 'documents';
              value: string | Document;
            }
          | {
              relationTo?: 'folders';
              value: string | Folder;
            }
        )[]
      | null;
    hasNextPage?: boolean | null;
  } | null;
  folder?: (string | null) | Folder;
  updatedAt: string;
  createdAt: string;
}
```

While you could instead have many join fields (for example
`childrenFolders`, `childrenFiles`) etc - this doesn't allow you to
sort/filter and paginate things across many collections, which isn't
trivial. With SQL we use `UNION ALL` query to achieve that.

---------

Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
2025-02-18 21:53:45 +02:00
Sasha
88548fcbe6 fix(db-postgres): querying other collections via relationships inside blocks (#11255)
### What?
Previously, in postgres query like:
```ts
const result = await payload.find({
  collection: 'blocks',
  where: { 'blocks.director.name': { equals: 'Test Director' } },
})
```
where `blocks` is a blocks field, `director` is a relationship field and
`name` is a text field inside `directors`, failed with:

![image](https://github.com/user-attachments/assets/f4b62b69-bd17-4ef0-9f0c-08057e9f2d57)

### Why?
The generated query before was a bit wrong.
Before:
```sql
select distinct
  "blocks"."id",
  "blocks"."created_at",
  "blocks"."created_at"
from
  "blocks"
  left join "directors" "a5ad426a_eda4_4067_af7e_5b294d7f0968" on "a5ad426a_eda4_4067_af7e_5b294d7f0968"."id" = "blocks_blocks_some"."director_id"
   left join "blocks_blocks_some" on "blocks"."id" = "blocks_blocks_some"."_parent_id"
where
  "a5ad426a_eda4_4067_af7e_5b294d7f0968"."name" = 'Test Director'
order by
  "blocks"."created_at" desc
limit
	10
```
Notice `left join directors` _before_ join of `blocks_blocks_some`.
`blocks_blocks_some` doesn't exist yet, this PR changes so now we
generate
```sql
select distinct
  "blocks"."id",
  "blocks"."created_at",
  "blocks"."created_at"
from
  "blocks"
  left join "blocks_blocks_some" on "blocks"."id" = "blocks_blocks_some"."_parent_id"
  left join "directors" "a5ad426a_eda4_4067_af7e_5b294d7f0968" on "a5ad426a_eda4_4067_af7e_5b294d7f0968"."id" = "blocks_blocks_some"."director_id"
where
  "a5ad426a_eda4_4067_af7e_5b294d7f0968"."name" = 'Test Director'
order by
  "blocks"."created_at" desc
limit
	10
```
2025-02-18 14:51:17 -05:00
Sasha
847d8d824f feat: add page query parameter for joins (#10998)
Currently, the join field outputs to its result `hasNextPage: boolean`
and have the `limit` query parameter but lacks `page` which can be
useful. This PR adds it.
2025-02-18 16:47:09 +02:00
Alessio Gravili
e6fea1d132 fix: localized fields within block references were not handled properly if any parent is localized (#11207)
The `localized` properly was not stripped out of referenced block fields, if any parent was localized. For normal fields, this is done in sanitizeConfig. As the same referenced block config can be used in both a localized and non-localized config, we are not able to strip it out inside sanitizeConfig by modifying the block config.

Instead, this PR had to bring back tedious logic to handle it everywhere the `field.localized` property is accessed. For backwards-compatibility, we need to keep the existing sanitizeConfig logic. In 4.0, we should remove it to benefit from better test coverage of runtime field.localized handling - for now, this is done for our test suite using the `PAYLOAD_DO_NOT_SANITIZE_LOCALIZED_PROPERTY` flag.
2025-02-17 19:50:32 +00:00
Sasha
513ba636af fix(db-postgres): ensure countDistinct works correctly and achieve better performance when the query has table joins (#11208)
The fix, added in https://github.com/payloadcms/payload/pull/11096
wasn't sufficient enough. It did handle the case when the same query
path / table was joined twice and caused incorrect `totalDocs`, but it
didn't handle the case when `JOIN` returns more than 1 rows, which 2
added new assertions here check.

Now, we use `COUNT(*)` only if we don't have any joined tables. If we
do, instead of using `SELECT (COUNT DISTINCT id)` which as described in
the previous PR is _very slow_ for large tables, we use the following
query:

```sql
SELECT COUNT(1) OVER() as count -- window function, executes for each row only once
FROM users
LEFT JOIN -- ... here additional rows are added
WHERE -- ...
GROUP BY users.id -- this ensures we're counting only users without additional rows from joins. 
LIMIT 1 -- Since COUNT(1) OVER() executes and resolves before doing LIMIT, we can safely apply LIMIT 1.
```
2025-02-16 14:08:08 +02:00
Alessio Gravili
4c8cafd6a6 perf: deduplicate blocks used in multiple places using new config.blocks property (#10905)
If you have multiple blocks that are used in multiple places, this can quickly blow up the size of your Payload Config. This will incur a performance hit, as more data is
1.  sent to the client (=> bloated `ClientConfig` and large initial html) and
2. processed on the server (permissions are calculated every single time you navigate to a page - this iterates through all blocks you have defined, even if they're duplicative)

This can be optimized by defining your block **once** in your Payload Config, and just referencing the block slug whenever it's used, instead of passing the entire block config. To do this, the block can be defined in the `blocks` array of the Payload Config. The slug can then be passed to the `blockReferences` array in the Blocks Field - the `blocks` array has to be empty for compatibility reasons.

```ts
import { buildConfig } from 'payload'
import { lexicalEditor, BlocksFeature } from '@payloadcms/richtext-lexical'

// Payload Config
const config = buildConfig({
  // Define the block once
  blocks: [
    {
      slug: 'TextBlock',
      fields: [
        {
          name: 'text',
          type: 'text',
        },
      ],
    },
  ],
  collections: [
    {
      slug: 'collection1',
      fields: [
        {
          name: 'content',
          type: 'blocks',
          // Reference the block by slug
          blockReferences: ['TextBlock'],
          blocks: [], // Required to be empty, for compatibility reasons
        },
      ],
    },
     {
      slug: 'collection2',
      fields: [
        {
          name: 'editor',
          type: 'richText',
          editor: lexicalEditor({
            BlocksFeature({
              // Same reference can be reused anywhere, even in the lexical editor, without incurred performance hit
              blocks: ['TextBlock'],
            })
          })
        },
      ],
    },
  ],
})
```

## v4.0 Plans

In 4.0, we will remove the `blockReferences` property, and allow string block references to be passed directly to the blocks `property`. Essentially, we'd remove the `blocks` property and rename `blockReferences` to `blocks`.

The reason we opted to a new property in this PR is to avoid breaking changes. Allowing strings to be passed to the `blocks` property will prevent plugins that iterate through fields / blocks from compiling.

## PR Changes

- Testing: This PR introduces a plugin that automatically converts blocks to block references. This is done in the fields__blocks test suite, to run our existing test suite using block references.

- Block References support: Most changes are similar. Everywhere we iterate through blocks, we have to now do the following:
1. Check if `field.blockReferences` is provided. If so, only iterate through that.
2. Check if the block is an object (= actual block), or string
3. If it's a string, pull the actual block from the Payload Config or from `payload.blocks`.

The exception is config sanitization and block type generations. This PR optimizes them so that each block is only handled once, instead of every time the block is referenced.

## Benchmarks

60 Block fields, each block field having the same 600 Blocks.

### Before:
**Initial HTML:** 195 kB
**Generated types:** takes 11 minutes, 461,209 lines

https://github.com/user-attachments/assets/11d49a4e-5414-4579-8050-e6346e552f56

### After:
**Initial HTML:** 73.6 kB
**Generated types:** takes 2 seconds, 35,810 lines

https://github.com/user-attachments/assets/3eab1a99-6c29-489d-add5-698df67780a3

### After Permissions Optimization (follow-up PR)
Initial HTML: 73.6 kB

https://github.com/user-attachments/assets/a909202e-45a8-4bf6-9a38-8c85813f1312


## Future Plans

1. This PR does not yet deduplicate block references during permissions calculation. We'll optimize that in a separate PR, as this one is already large enough
2. The same optimization can be done to deduplicate fields. One common use-case would be link field groups that may be referenced in multiple entities, outside of blocks. We might explore adding a new `fieldReferences` property, that allows you to reference those same `config.blocks`.
2025-02-14 00:08:20 +00:00
Sasha
98fec35368 fix(db-postgres): incorrect pagination results when querying hasMany relationships multiple times (#11096)
Fixes https://github.com/payloadcms/payload/issues/10810

This was caused by using `COUNT(*)` aggregation instead of
`COUNT(DISTINCT table.id)`. However, we want to use `COUNT(*)` because
`COUNT(DISTINCT table.id)` is slow on large tables. Now we fallback to
`COUNT(DISTINCT table.id)` only when `COUNT(*)` cannot work properly.

Example of a query that leads to incorrect `totalDocs`:
```ts
const res = await payload.find({
  collection: 'directors',
  limit: 10,
  where: {
    or: [
      {
        movies: {
          equals: movie2.id,
        },
      },
      {
        movies: {
          equals: movie1.id,
        },
      },
      {
        movies: {
          equals: movie1.id,
        },
      },
    ],
  },
})
```
2025-02-11 01:16:18 +02:00
Jarrod Flesch
d2fe9b0807 fix(db-mongodb): ensures same level operators are respected (#11087)
### What?
If you had multiple operator constraints on a single field, the last one
defined would be the only one used.

Example:
```ts
where: {
  id: {
    in: [doc2.id],
    not_in: [], // <-- only respected this operator constraint
  },
}
```

and
```ts
where: {
  id: {
    not_in: [],
    in: [doc2.id], // <-- only respected this operator constraint
  },
}
```

They would yield different results.

### Why?
The results were not merged into an `$and` query inside parseParams.

### How?
Merges the results within an `$and` constraint.

Fixes https://github.com/payloadcms/payload/issues/10944

Supersedes https://github.com/payloadcms/payload/pull/11011
2025-02-10 16:29:08 -05:00
Sasha
57143b37d0 fix(db-postgres): ensure globals have createdAt, updatedAt and globalType fields (#10938)
Previously, data for globals was inconsistent across database adapters.
In Postgres, globals didn't store correct `createdAt`, `updatedAt`
fields and the `updateGlobal` lacked the `globalType` field. This PR
solves that without introducing schema changes.
2025-02-06 23:48:59 +02:00
Sasha
3ad56cd86f fix(db-postgres): select hasMany: true with autosave doesn't work properly (#11012)
Previously, select fields with `hasMany: true` didn't save properly in
Postgres on autosave.
2025-02-06 23:47:53 +02:00
Sasha
68a7de2610 fix(db-postgres): select hasMany inside arrays and blocks with versions (#10829)
Fixes https://github.com/payloadcms/payload/issues/10780

Previously, with enabled versions, nested select `hasMany: true` fields
weren't working with SQL database adapters. This was due to wrongly
passed `parent` to select rows data because we store arrays and blocks
in versions a bit differently, using both, `id` and `_uuid` (which
contains the normal Object ID) columns. And unlike with non versions
`_uuid` column isn't actually applicable here as it's not unique, thus
we need to save blocks/arrays first and then map their ObjectIDs to
generated by the database IDs and use them for select fields `parent`
data
2025-01-31 18:26:04 +02:00
Sasha
e1dcb9594c fix(db-postgres): write operations on polymorphic joined collections throw error (#10854)
Fixes https://github.com/payloadcms/payload/issues/10845
See
https://github.com/payloadcms/payload/issues/10845#issuecomment-2620201486
2025-01-31 18:25:47 +02:00
Sasha
d601300034 fix(db-mongodb): querying polymorphic relationships with the all operator (#10704)
Fixes https://github.com/payloadcms/payload/issues/10678
2025-01-23 18:23:50 +02:00
Sasha
46c1b375b8 fix: properly handle nullable minDistance and maxDistance in near query (#10622)
Fixes https://github.com/payloadcms/payload/issues/10521
2025-01-21 02:18:13 +02:00
Alessio Gravili
42382b6f6f perf: operations performance optimizations (#10609)
- reduces unnecessary shallow copying within operations by removing
unnecessary spreads or .map()'s
- removes unnecessary `deleteMany` call in `deleteUserPreferences` for
auth-enabled collections
- replaces all instances of `validOperators.includes` with
`validOperatorMap[]`. O(n) => O(1)
- optimizes the `sanitizeInternalFields` function. Previously, it was
doing a **lot** of shallow copying
2025-01-16 17:07:35 +00:00
Sasha
1525cc6e3a fix(db-postgres): handle undefined fallback for adapter.schemaName in relationships migration (#10384)
Fixes from https://github.com/payloadcms/payload/pull/10322
2025-01-06 15:19:21 +00:00
Nacho Martin
556b8ed393 fix(db-postgres): added missing quotes and schema name to sql statement in v2-v3 migration (#10322)
### What?

* Added missing schema name to the SQL statement
* Added missing quotes to the `parent_id`'s in the SQL statement

### Why?

* Migrations script isn't working for databases that don't use the
`public` schema
* They were missing, causing the database to throw an error.

### Fixes:

- [Discord
Discussion](https://discord.com/channels/967097582721572934/1319313926885736448)
- [Discord
Discussion](https://discord.com/channels/967097582721572934/1324352204915609661)
- [Discord
Discussion](https://discord.com/channels/967097582721572934/1323445259606429716)

---------

Co-authored-by: Sasha <64744993+r1tsuu@users.noreply.github.com>
2025-01-06 16:40:32 +02:00
Sasha
98666eb016 perf(db-postgres): do not push database schema if not changed (#10155)
Based on https://github.com/payloadcms/payload/pull/10154

If the actual database schema is not changed (no new columns, enums,
indexes, tables) - skip calling Drizzle push. This, potentially can
significantly reduce overhead on reloads in development mode especially
when using remote databases.

If for whatever reason you need to preserve the current behavior you can
use `PAYLOAD_FORCE_DRIZZLE_PUSH=true` env flag.
2024-12-27 10:12:01 -05:00
Sasha
374b79d218 fix(db-postgres): prevent indexes from changing name on HMR (#10154)
As we didn't reset our `adapter.indexes` state, on every HMR reload we
incremented every single index name with the `buildIndexName`:

466f109152/packages/drizzle/src/utilities/buildIndexName.ts (L3-L24)

I found this while debugging our internal SQL schema:

Before reload:

```ts
 "payload_preferences": {
    "name": "payload_preferences",
    "columns": {
      "id": {
        "name": "id",
        "type": "serial",
        "primaryKey": true
      },
      "key": {
        "name": "key",
        "type": "varchar"
      },
      "value": {
        "name": "value",
        "type": "jsonb"
      },
      "updatedAt": {
        "name": "updated_at",
        "type": "timestamp",
        "defaultNow": true,
        "mode": "string",
        "notNull": true,
        "precision": 3,
        "withTimezone": true
      },
      "createdAt": {
        "name": "created_at",
        "type": "timestamp",
        "defaultNow": true,
        "mode": "string",
        "notNull": true,
        "precision": 3,
        "withTimezone": true
      }
    },
    "foreignKeys": {},
    "indexes": {
      "payload_preferences_key_idx": {
        "name": "payload_preferences_key_idx",
        "on": "key"
      },
      "payload_preferences_updated_at_idx": {
        "name": "payload_preferences_updated_at_idx",
        "on": "updatedAt"
      },
      "payload_preferences_created_at_idx": {
        "name": "payload_preferences_created_at_idx",
        "on": "createdAt"
      }
    }
  },
```

After:
```ts

"payload_preferences": {
    "name": "payload_preferences",
    "columns": {
      "id": {
        "name": "id",
        "type": "serial",
        "primaryKey": true
      },
      "key": {
        "name": "key",
        "type": "varchar"
      },
      "value": {
        "name": "value",
        "type": "jsonb"
      },
      "updatedAt": {
        "name": "updated_at",
        "type": "timestamp",
        "defaultNow": true,
        "mode": "string",
        "notNull": true,
        "precision": 3,
        "withTimezone": true
      },
      "createdAt": {
        "name": "created_at",
        "type": "timestamp",
        "defaultNow": true,
        "mode": "string",
        "notNull": true,
        "precision": 3,
        "withTimezone": true
      }
    },
    "foreignKeys": {},
    "indexes": {
      "payload_preferences_key_1_idx": {
        "name": "payload_preferences_key_1_idx",
        "on": "key"
      },
      "payload_preferences_updated_at_1_idx": {
        "name": "payload_preferences_updated_at_1_idx",
        "on": "updatedAt"
      },
      "payload_preferences_created_at_1_idx": {
        "name": "payload_preferences_created_at_1_idx",
        "on": "createdAt"
      }
    }
  },
```
Which isn't really great for dev performance and can potentially cause
errors
2024-12-27 10:04:48 -05:00
Sasha
a0d8131649 fix(db-postgres): joins to self collection (#10182)
### What?
With Postgres, before join to self like:
```ts
import type { CollectionConfig } from 'payload'

export const SelfJoins: CollectionConfig = {
  slug: 'self-joins',
  fields: [
    {
      name: 'rel',
      type: 'relationship',
      relationTo: 'self-joins',
    },
    {
      name: 'joins',
      type: 'join',
      on: 'rel',
      collection: 'self-joins',
    },
  ],
}
```
wasn't possible, even though it's a valid usage.

### How?
Now, to differentiate parent `self_joins` and children `self_joins` we
do additional alias for the nested select -
`"4d3cf2b6_1adf_46a8_b6d2_3e1c3809d737"`:
```sql
select 
  "id", 
  "rel_id", 
  "updated_at", 
  "created_at", 
  (
    select 
      coalesce(
        json_agg(
          json_build_object('id', "joins_alias".id)
        ), 
        '[]' :: json
      ) 
    from 
      (
        select 
          "created_at", 
          "rel_id", 
          "id" 
        from 
          "self_joins" "4d3cf2b6_1adf_46a8_b6d2_3e1c3809d737" 
        where 
          "4d3cf2b6_1adf_46a8_b6d2_3e1c3809d737"."rel_id" = "self_joins"."id" 
        order by 
          "4d3cf2b6_1adf_46a8_b6d2_3e1c3809d737"."created_at" desc 
        limit 
          $1
      ) "joins_alias"
  ) as "joins_alias" 
from 
  "self_joins" 
where 
  "self_joins"."id" = $2 
order by 
  "self_joins"."created_at" desc 
limit 
  $3

```

Fixes https://github.com/payloadcms/payload/issues/10144

-->
2024-12-26 20:47:49 +02:00
Sasha
4e953530df feat(db-sqlite): add autoIncrement option (#9427)
### What?
Exposes ability to enable
[AUTOINCREMENT](https://www.sqlite.org/autoinc.html) for Primary Keys
which ensures that the same ID cannot be reused from previously deleted
rows.

```ts
sqliteAdapter({
  autoIncrement: true
})
```

### Why?
This may be essential for some systems. Enabled `autoIncrement: true`
also for the SQLite Adapter in our tests, which can be useful when
testing whether the doc was deleted or not when you also have other
create operations.

### How?
Uses Drizzle's `autoIncrement` option.

WARNING:
This cannot be enabled in an existing project without a custom
migration, as it completely changes how primary keys are stored in the
database.
2024-12-20 20:13:28 +00:00
Sasha
7c4ea5b86e refactor: optimize database schema generation bin script (#10086)
* Avoids additional file system writes (1 for `await writeFile` and then
`npx prettier --write`) instead prettier now formats the javascript
string directly. Went from 650 MS to 250 MS for the prettify block.
* Disables database connection, since the `db.generateSchema` doesn't
need connection, this also disables Drizzle schema push.
* Properly exits the bin script process.
2024-12-20 14:43:11 +02:00
Dan Ribbens
d03658de01 feat: join field with polymorphic relationships (#9990)
### What?
The join field had a limitation imposed that prevents it from targeting
polymorphic relationship fields. With this change we can support any
relationship fields.

### Why?
Improves the functionality of join field.

### How?
Extended the database adapters and removed the config sanitization that
would throw an error when polymorphic relationships were used.

Fixes #
2024-12-19 22:34:52 +00:00
Sasha
07be617963 fix(db-postgres): relationships v2-v3 migration errors when migrating from v2 to stable v3 (#10080)
When migrating from stable v2 to v3, the provided relationships
migration fails because of not having `payload_locked_documents` and
`payload_locked_documents_rels` tables in the migration generated in v2.


![image](https://github.com/user-attachments/assets/0dc57e6a-5e6e-4b74-bcab-70e660f4e939)
2024-12-19 19:09:07 +00:00
Sasha
23f1ed4a48 feat(db-postgres, db-sqlite): drizzle schema generation (#9953)
This PR allows to have full type safety on `payload.drizzle` with a
single command
```sh
pnpm payload generate:db-schema
```
Which generates TypeScript code with Drizzle declarations based on the
current database schema.

Example of generated file with the website template: 
https://gist.github.com/r1tsuu/b8687f211b51d9a3a7e78ba41e8fbf03

Video that shows the power:


https://github.com/user-attachments/assets/3ced958b-ec1d-49f5-9f51-d859d5fae236


We also now proxy drizzle package the same way we do for Lexical so you
don't have to install it (and you shouldn't because you may have version
mismatch).
Instead, you can import from Drizzle like this:
```ts
import {
  pgTable,
  index,
  foreignKey,
  integer,
  text,
  varchar,
  jsonb,
  boolean,
  numeric,
  serial,
  timestamp,
  uniqueIndex,
  pgEnum,
} from '@payloadcms/db-postgres/drizzle/pg-core'
import { sql } from '@payloadcms/db-postgres/drizzle'
import { relations } from '@payloadcms/db-postgres/drizzle/relations'
```


Fixes https://github.com/payloadcms/payload/discussions/4318

In the future we can also support types generation for mongoose / raw
mongodb results.
2024-12-19 11:08:17 -05:00
Sasha
0c57eef621 fix: unique error message regression (#10064)
Regression from https://github.com/payloadcms/payload/pull/9935, we need
to call `req.t` with `''error:valueMustBeUnique''` instead of just
`''error:valueMustBeUnique''`
2024-12-19 08:13:44 +00:00
Sasha
03ff77544e feat(db-sqlite): add idType: 'uuid' support (#10016)
Adds `idType: 'uuid'` to the SQLite adapter support:
```ts
sqliteAdapter({
  idType: 'uuid',
})
```

Achieved through Drizzle's `$defaultFn()`
https://orm.drizzle.team/docs/latest-releases/drizzle-orm-v0283#-added-defaultfn--default-methods-to-column-builders
as SQLite doesn't have native UUID support. Added `sqlite-uuid` to CI.
2024-12-18 22:44:04 -05:00
Sasha
0e5bda9a74 feat: make req partial and optional in DB / Local API operations (#9935)
### What?
Previously, the `req` argument:
In database operations (e.g `payload.db`) was required and you needed to
pass the whole `req` with all the properties. This is confusing because
in database operations we never use its properties outside of
`req.transactionID` and `req.t`, both of which should be optional as
well.

Now, you don't have to do that cast:
```ts
payload.db.findOne({
  collection: 'posts',
  req: {} as PayloadRequest,
  where: {
    id: {
      equals: 1,
    },
  },
})
```
Becomes:
```ts
payload.db.findOne({
  collection: 'posts',
  where: {
    id: {
      equals: 1,
    },
  },
})
```

If you need to use transactions, you're not required to do the `as` cast
as well now, as the `req` not only optional but also partial -
`Partial<PayloadRequest>`.
`initTransaction`, `commitTransaction`, `killTransaction` utilities are
typed better now as well. They do not require to you pass all the
properties of `req`, but only `payload` -
`MarkRequired<Partial<PayloadRequest>, 'payload'>`
```ts
const req = { payload }
await initTransaction(req)
await payload.db.create({
  collection: "posts",
  data: {},
  req
})
await commitTransaction(req)
```

The same for the Local API. Internal operations (for example
`packages/payload/src/collections/operations/find.ts`) still accept the
whole `req`, but local ones
(`packages/payload/src/collections/operations/local/find.ts`) which are
used through `payload.` now accept `Partial<PayloadRequest>`, as they
pass it through to internal operations with `createLocalReq`.

So now, this is also valid, while previously you had to do `as` cast for
`req`.
```ts
const req = { payload }
await initTransaction(req)
await payload.create({
  collection: "posts",
  data: {},
  req
})
await commitTransaction(req)
```


Marked as deprecated `PayloadRequest['transactionIDPromise']` to remove
in the next major version. It was never used anywhere.
Refactored `withSession` that returns an object to `getSession` that
returns just `ClientSession`. Better type safety for arguments
Deduplicated in all drizzle operations to `getTransaction(this, req)`
utility:
```ts
const db = this.sessions[await req?.transactionID]?.db || this.drizzle
```
Added fallback for throwing unique validation errors in database
operations when `req.t` is not available.

In migration `up` and `down` functions our `req` is not partial, while
we used to passed `req` with only 2 properties - `payload` and
`transactionID`. This is misleading and you can't access for example
`req.t`.
Now, to achieve "real" full `req` - we generate it with `createLocalReq`
in all migration functions.

This all is backwards compatible. In all public API places where you
expect the full `req` (like hooks) you still have it.

### Why?
Better DX, more expected types, less errors because of types casting.
2024-12-18 22:43:37 -05:00
Sasha
eee6432715 fix(db-postgres): query has many relationships nested in row fields (#9944) (#9944)
### What?
Querying by nested to rows fields in has many relationships like this:
```ts
const result = await payload.find({
  collection: 'relationship-fields',
  where: {
    'relationToRowMany.title': { equals: 'some-title' },
  },
})
```
Where the related collection:
```ts
const RowFields: CollectionConfig = {
  slug: rowFieldsSlug,
  fields: [
    {
      type: 'row',
      fields: [
        {
          name: 'title',
          label: 'Title within a row',
          type: 'text',
          required: true,
        },
      ],
    },
  ],
}
```

was broken

### Why?
We migrated to use `flattenedFields`, but not in this specific case.
This error would be caught earlier we used `noImplictAny` typescript
rule. https://www.typescriptlang.org/tsconfig/#noImplicitAny which
wouldn't allow us to create variable like this:
```ts
let relationshipFields // relationshipFields is any here
```
Instead, we should write:
```ts
let relationshipFields: FlattenedField[]
```
We should migrate to it and `strictNullChecks` as well.

Fixes https://github.com/payloadcms/payload/issues/9534
2024-12-18 22:26:08 -05:00