Fixes https://github.com/payloadcms/payload/issues/10810
This was caused by using `COUNT(*)` aggregation instead of
`COUNT(DISTINCT table.id)`. However, we want to use `COUNT(*)` because
`COUNT(DISTINCT table.id)` is slow on large tables. Now we fallback to
`COUNT(DISTINCT table.id)` only when `COUNT(*)` cannot work properly.
Example of a query that leads to incorrect `totalDocs`:
```ts
const res = await payload.find({
collection: 'directors',
limit: 10,
where: {
or: [
{
movies: {
equals: movie2.id,
},
},
{
movies: {
equals: movie1.id,
},
},
{
movies: {
equals: movie1.id,
},
},
],
},
})
```
### What?
If you had multiple operator constraints on a single field, the last one
defined would be the only one used.
Example:
```ts
where: {
id: {
in: [doc2.id],
not_in: [], // <-- only respected this operator constraint
},
}
```
and
```ts
where: {
id: {
not_in: [],
in: [doc2.id], // <-- only respected this operator constraint
},
}
```
They would yield different results.
### Why?
The results were not merged into an `$and` query inside parseParams.
### How?
Merges the results within an `$and` constraint.
Fixes https://github.com/payloadcms/payload/issues/10944
Supersedes https://github.com/payloadcms/payload/pull/11011
Previously, data for globals was inconsistent across database adapters.
In Postgres, globals didn't store correct `createdAt`, `updatedAt`
fields and the `updateGlobal` lacked the `globalType` field. This PR
solves that without introducing schema changes.
Fixes https://github.com/payloadcms/payload/issues/10780
Previously, with enabled versions, nested select `hasMany: true` fields
weren't working with SQL database adapters. This was due to wrongly
passed `parent` to select rows data because we store arrays and blocks
in versions a bit differently, using both, `id` and `_uuid` (which
contains the normal Object ID) columns. And unlike with non versions
`_uuid` column isn't actually applicable here as it's not unique, thus
we need to save blocks/arrays first and then map their ObjectIDs to
generated by the database IDs and use them for select fields `parent`
data
- reduces unnecessary shallow copying within operations by removing
unnecessary spreads or .map()'s
- removes unnecessary `deleteMany` call in `deleteUserPreferences` for
auth-enabled collections
- replaces all instances of `validOperators.includes` with
`validOperatorMap[]`. O(n) => O(1)
- optimizes the `sanitizeInternalFields` function. Previously, it was
doing a **lot** of shallow copying
This PR modifies `tsconfig.base.json` by setting the following
strictness properties to true: `strict`, `noUncheckedIndexedAccess` and
`noImplicitOverride`.
In packages where compilation errors were observed, these settings were
opted out, and TODO comments were added to make it easier to track the
roadmap for converting everything to strict mode.
The following packages now have increased strictness, which prevents new
errors from being accidentally introduced:
- storage-vercel-blob
- storage-s3*
- storage-gcs
- plugin-sentry
- payload-cloud*
- email-resend*
- email-nodemailer*
*These packages already had `strict: true`, but now have
`noUncheckedIndexedAccess` and `noImplicitOverride`.
Note that this only affects the `/packages` folder, but not
`/templates`, `/test` or `/examples` which have a different `tsconfig`.
Based on https://github.com/payloadcms/payload/pull/10154
If the actual database schema is not changed (no new columns, enums,
indexes, tables) - skip calling Drizzle push. This, potentially can
significantly reduce overhead on reloads in development mode especially
when using remote databases.
If for whatever reason you need to preserve the current behavior you can
use `PAYLOAD_FORCE_DRIZZLE_PUSH=true` env flag.
Previously we had been downgrading rimraf to v3 simply to handle clean
with glob patterns across platforms. In v4 and newer of rimraf you can
add `-g` to use glob patterns.
This change updates rimraf and adds the flag to handle globs in our
package scripts to be windows compatible.
### What?
With Postgres, before join to self like:
```ts
import type { CollectionConfig } from 'payload'
export const SelfJoins: CollectionConfig = {
slug: 'self-joins',
fields: [
{
name: 'rel',
type: 'relationship',
relationTo: 'self-joins',
},
{
name: 'joins',
type: 'join',
on: 'rel',
collection: 'self-joins',
},
],
}
```
wasn't possible, even though it's a valid usage.
### How?
Now, to differentiate parent `self_joins` and children `self_joins` we
do additional alias for the nested select -
`"4d3cf2b6_1adf_46a8_b6d2_3e1c3809d737"`:
```sql
select
"id",
"rel_id",
"updated_at",
"created_at",
(
select
coalesce(
json_agg(
json_build_object('id', "joins_alias".id)
),
'[]' :: json
)
from
(
select
"created_at",
"rel_id",
"id"
from
"self_joins" "4d3cf2b6_1adf_46a8_b6d2_3e1c3809d737"
where
"4d3cf2b6_1adf_46a8_b6d2_3e1c3809d737"."rel_id" = "self_joins"."id"
order by
"4d3cf2b6_1adf_46a8_b6d2_3e1c3809d737"."created_at" desc
limit
$1
) "joins_alias"
) as "joins_alias"
from
"self_joins"
where
"self_joins"."id" = $2
order by
"self_joins"."created_at" desc
limit
$3
```
Fixes https://github.com/payloadcms/payload/issues/10144
-->
### What?
Exposes ability to enable
[AUTOINCREMENT](https://www.sqlite.org/autoinc.html) for Primary Keys
which ensures that the same ID cannot be reused from previously deleted
rows.
```ts
sqliteAdapter({
autoIncrement: true
})
```
### Why?
This may be essential for some systems. Enabled `autoIncrement: true`
also for the SQLite Adapter in our tests, which can be useful when
testing whether the doc was deleted or not when you also have other
create operations.
### How?
Uses Drizzle's `autoIncrement` option.
WARNING:
This cannot be enabled in an existing project without a custom
migration, as it completely changes how primary keys are stored in the
database.
* Avoids additional file system writes (1 for `await writeFile` and then
`npx prettier --write`) instead prettier now formats the javascript
string directly. Went from 650 MS to 250 MS for the prettify block.
* Disables database connection, since the `db.generateSchema` doesn't
need connection, this also disables Drizzle schema push.
* Properly exits the bin script process.
### What?
The join field had a limitation imposed that prevents it from targeting
polymorphic relationship fields. With this change we can support any
relationship fields.
### Why?
Improves the functionality of join field.
### How?
Extended the database adapters and removed the config sanitization that
would throw an error when polymorphic relationships were used.
Fixes #
This PR allows to have full type safety on `payload.drizzle` with a
single command
```sh
pnpm payload generate:db-schema
```
Which generates TypeScript code with Drizzle declarations based on the
current database schema.
Example of generated file with the website template:
https://gist.github.com/r1tsuu/b8687f211b51d9a3a7e78ba41e8fbf03
Video that shows the power:
https://github.com/user-attachments/assets/3ced958b-ec1d-49f5-9f51-d859d5fae236
We also now proxy drizzle package the same way we do for Lexical so you
don't have to install it (and you shouldn't because you may have version
mismatch).
Instead, you can import from Drizzle like this:
```ts
import {
pgTable,
index,
foreignKey,
integer,
text,
varchar,
jsonb,
boolean,
numeric,
serial,
timestamp,
uniqueIndex,
pgEnum,
} from '@payloadcms/db-postgres/drizzle/pg-core'
import { sql } from '@payloadcms/db-postgres/drizzle'
import { relations } from '@payloadcms/db-postgres/drizzle/relations'
```
Fixes https://github.com/payloadcms/payload/discussions/4318
In the future we can also support types generation for mongoose / raw
mongodb results.
### What?
Previously, the `req` argument:
In database operations (e.g `payload.db`) was required and you needed to
pass the whole `req` with all the properties. This is confusing because
in database operations we never use its properties outside of
`req.transactionID` and `req.t`, both of which should be optional as
well.
Now, you don't have to do that cast:
```ts
payload.db.findOne({
collection: 'posts',
req: {} as PayloadRequest,
where: {
id: {
equals: 1,
},
},
})
```
Becomes:
```ts
payload.db.findOne({
collection: 'posts',
where: {
id: {
equals: 1,
},
},
})
```
If you need to use transactions, you're not required to do the `as` cast
as well now, as the `req` not only optional but also partial -
`Partial<PayloadRequest>`.
`initTransaction`, `commitTransaction`, `killTransaction` utilities are
typed better now as well. They do not require to you pass all the
properties of `req`, but only `payload` -
`MarkRequired<Partial<PayloadRequest>, 'payload'>`
```ts
const req = { payload }
await initTransaction(req)
await payload.db.create({
collection: "posts",
data: {},
req
})
await commitTransaction(req)
```
The same for the Local API. Internal operations (for example
`packages/payload/src/collections/operations/find.ts`) still accept the
whole `req`, but local ones
(`packages/payload/src/collections/operations/local/find.ts`) which are
used through `payload.` now accept `Partial<PayloadRequest>`, as they
pass it through to internal operations with `createLocalReq`.
So now, this is also valid, while previously you had to do `as` cast for
`req`.
```ts
const req = { payload }
await initTransaction(req)
await payload.create({
collection: "posts",
data: {},
req
})
await commitTransaction(req)
```
Marked as deprecated `PayloadRequest['transactionIDPromise']` to remove
in the next major version. It was never used anywhere.
Refactored `withSession` that returns an object to `getSession` that
returns just `ClientSession`. Better type safety for arguments
Deduplicated in all drizzle operations to `getTransaction(this, req)`
utility:
```ts
const db = this.sessions[await req?.transactionID]?.db || this.drizzle
```
Added fallback for throwing unique validation errors in database
operations when `req.t` is not available.
In migration `up` and `down` functions our `req` is not partial, while
we used to passed `req` with only 2 properties - `payload` and
`transactionID`. This is misleading and you can't access for example
`req.t`.
Now, to achieve "real" full `req` - we generate it with `createLocalReq`
in all migration functions.
This all is backwards compatible. In all public API places where you
expect the full `req` (like hooks) you still have it.
### Why?
Better DX, more expected types, less errors because of types casting.
### What?
Querying by nested to rows fields in has many relationships like this:
```ts
const result = await payload.find({
collection: 'relationship-fields',
where: {
'relationToRowMany.title': { equals: 'some-title' },
},
})
```
Where the related collection:
```ts
const RowFields: CollectionConfig = {
slug: rowFieldsSlug,
fields: [
{
type: 'row',
fields: [
{
name: 'title',
label: 'Title within a row',
type: 'text',
required: true,
},
],
},
],
}
```
was broken
### Why?
We migrated to use `flattenedFields`, but not in this specific case.
This error would be caught earlier we used `noImplictAny` typescript
rule. https://www.typescriptlang.org/tsconfig/#noImplicitAny which
wouldn't allow us to create variable like this:
```ts
let relationshipFields // relationshipFields is any here
```
Instead, we should write:
```ts
let relationshipFields: FlattenedField[]
```
We should migrate to it and `strictNullChecks` as well.
Fixes https://github.com/payloadcms/payload/issues/9534
Previously, if you selected only upload `hasMany: true` field, you would
receive an empty arrays always, because the `_rels` table wasn't joined
in this case. Fixes the condition to count `field.type === 'upload'` .
Should fix messed up import suggestions and simplifies all tsconfigs
through inheritance.
One main issue was that packages were inheriting `baseURL: "."` from the
root tsconfig. This caused incorrect import suggestions that start with
"packages/...".
This PR ensures that packages do not inherit this baseURL: "." property,
while ensuring the root, non-inherited tsconfig still keeps it to get
tests to work (the importMap needs it)
### What?
Allow the join field to have a configuration `on` relationships inside
of an array, ie `on: 'myArray.myRelationship'`.
### Why?
This is a more powerful and expressive way to use the join field and not
be limited by usage of array data. For example, if you have a roles
array for multinant sites, you could add a join field on the sites to
show who the admins are.
### How?
This fixes the traverseFields function to allow the configuration to
pass sanitization. In addition, the function for querying the drizzle
tables needed to be ehanced.
Additional changes from https://github.com/payloadcms/payload/pull/9995:
- Significantly improves traverseFields and the 'join' case with a raw
query injection pattern, right now it's internal but we could expose it
at some point, for example for querying vectors.
- Fixes potential issues with not passed locale to traverseFields (it
was undefined always)
- Adds an empty array fallback for joins with localized relationships
Fixes #
https://github.com/payloadcms/payload/discussions/9643
---------
Co-authored-by: Because789 <thomas@because789.ch>
Co-authored-by: Sasha <64744993+r1tsuu@users.noreply.github.com>
