The fix, added in https://github.com/payloadcms/payload/pull/11096
wasn't sufficient enough. It did handle the case when the same query
path / table was joined twice and caused incorrect `totalDocs`, but it
didn't handle the case when `JOIN` returns more than 1 rows, which 2
added new assertions here check.
Now, we use `COUNT(*)` only if we don't have any joined tables. If we
do, instead of using `SELECT (COUNT DISTINCT id)` which as described in
the previous PR is _very slow_ for large tables, we use the following
query:
```sql
SELECT COUNT(1) OVER() as count -- window function, executes for each row only once
FROM users
LEFT JOIN -- ... here additional rows are added
WHERE -- ...
GROUP BY users.id -- this ensures we're counting only users without additional rows from joins.
LIMIT 1 -- Since COUNT(1) OVER() executes and resolves before doing LIMIT, we can safely apply LIMIT 1.
```
Fixes https://github.com/payloadcms/payload/issues/10810
This was caused by using `COUNT(*)` aggregation instead of
`COUNT(DISTINCT table.id)`. However, we want to use `COUNT(*)` because
`COUNT(DISTINCT table.id)` is slow on large tables. Now we fallback to
`COUNT(DISTINCT table.id)` only when `COUNT(*)` cannot work properly.
Example of a query that leads to incorrect `totalDocs`:
```ts
const res = await payload.find({
collection: 'directors',
limit: 10,
where: {
or: [
{
movies: {
equals: movie2.id,
},
},
{
movies: {
equals: movie1.id,
},
},
{
movies: {
equals: movie1.id,
},
},
],
},
})
```
This PR modifies `tsconfig.base.json` by setting the following
strictness properties to true: `strict`, `noUncheckedIndexedAccess` and
`noImplicitOverride`.
In packages where compilation errors were observed, these settings were
opted out, and TODO comments were added to make it easier to track the
roadmap for converting everything to strict mode.
The following packages now have increased strictness, which prevents new
errors from being accidentally introduced:
- storage-vercel-blob
- storage-s3*
- storage-gcs
- plugin-sentry
- payload-cloud*
- email-resend*
- email-nodemailer*
*These packages already had `strict: true`, but now have
`noUncheckedIndexedAccess` and `noImplicitOverride`.
Note that this only affects the `/packages` folder, but not
`/templates`, `/test` or `/examples` which have a different `tsconfig`.
Previously we had been downgrading rimraf to v3 simply to handle clean
with glob patterns across platforms. In v4 and newer of rimraf you can
add `-g` to use glob patterns.
This change updates rimraf and adds the flag to handle globs in our
package scripts to be windows compatible.
### What?
Exposes ability to enable
[AUTOINCREMENT](https://www.sqlite.org/autoinc.html) for Primary Keys
which ensures that the same ID cannot be reused from previously deleted
rows.
```ts
sqliteAdapter({
autoIncrement: true
})
```
### Why?
This may be essential for some systems. Enabled `autoIncrement: true`
also for the SQLite Adapter in our tests, which can be useful when
testing whether the doc was deleted or not when you also have other
create operations.
### How?
Uses Drizzle's `autoIncrement` option.
WARNING:
This cannot be enabled in an existing project without a custom
migration, as it completely changes how primary keys are stored in the
database.
This PR allows to have full type safety on `payload.drizzle` with a
single command
```sh
pnpm payload generate:db-schema
```
Which generates TypeScript code with Drizzle declarations based on the
current database schema.
Example of generated file with the website template:
https://gist.github.com/r1tsuu/b8687f211b51d9a3a7e78ba41e8fbf03
Video that shows the power:
https://github.com/user-attachments/assets/3ced958b-ec1d-49f5-9f51-d859d5fae236
We also now proxy drizzle package the same way we do for Lexical so you
don't have to install it (and you shouldn't because you may have version
mismatch).
Instead, you can import from Drizzle like this:
```ts
import {
pgTable,
index,
foreignKey,
integer,
text,
varchar,
jsonb,
boolean,
numeric,
serial,
timestamp,
uniqueIndex,
pgEnum,
} from '@payloadcms/db-postgres/drizzle/pg-core'
import { sql } from '@payloadcms/db-postgres/drizzle'
import { relations } from '@payloadcms/db-postgres/drizzle/relations'
```
Fixes https://github.com/payloadcms/payload/discussions/4318
In the future we can also support types generation for mongoose / raw
mongodb results.
Should fix messed up import suggestions and simplifies all tsconfigs
through inheritance.
One main issue was that packages were inheriting `baseURL: "."` from the
root tsconfig. This caused incorrect import suggestions that start with
"packages/...".
This PR ensures that packages do not inherit this baseURL: "." property,
while ensuring the root, non-inherited tsconfig still keeps it to get
tests to work (the importMap needs it)
This PR adds a feature which fixes another issue with migrations in
Postgres and does few refactors that significantly reduce code
duplication.
Previously, if you needed to use the underlying database directly in
migrations with the active transaction (for example to execute raw SQL),
created from `payload create:migration`, as `req` doesn't work there you
had to do something like this:
```ts
// Postgres
export async function up({ payload, req }: MigrateUpArgs): Promise<void> {
const db = payload.db.sessions?.[await req.transactionID!].db ?? payload.db
const { rows: posts } = await db.execute(sql`SELECT * from posts`)
}
// MongoDB
export async function up({ payload, req }: MigrateUpArgs): Promise<void> {
const session = payload.db.sessions?.[await req.transactionID!]
const posts = await payload.db.collections.posts.collection.find({ session }).toArray()
}
```
Which was:
1. Awkward to write
2. Not documented anywhere
Now, we expose `session` and `db` to `up` and `down` functions for you:
#### MongoDB:
```ts
import { type MigrateUpArgs } from '@payloadcms/db-mongodb'
export async function up({ session, payload, req }: MigrateUpArgs): Promise<void> {
const posts = await payload.db.collections.posts.collection.find({ session }).toArray()
}
```
#### Postgres:
```ts
import { type MigrateUpArgs, sql } from '@payloadcms/db-postgres'
export async function up({ db, payload, req }: MigrateUpArgs): Promise<void> {
const { rows: posts } = await db.execute(sql`SELECT * from posts`)
}
```
#### SQLite:
```ts
import { type MigrateUpArgs, sql } from '@payloadcms/db-sqlite'
export async function up({ db, payload, req }: MigrateUpArgs): Promise<void> {
const { rows: posts } = await db.run(sql`SELECT * from posts`)
}
```
This actually was a thing with Postgres migrations, we already were
passing `db`, but:
1. Only for `up` and when running `payload migrate`, not for example
with `payload migrate:fresh`
2. Not documented neither in TypeScript or docs.
By ensuring we use `db`, this also fixes an issue that affects all
Postgres/SQLite migrations:
Currently, if we run `payload migration:create` with the postgres
adapter we get a file like this:
```ts
import { MigrateUpArgs, MigrateDownArgs, sql } from '@payloadcms/db-postgres'
export async function up({ payload, req }: MigrateUpArgs): Promise<void> {
await payload.db.drizzle.execute(sql`
CREATE TABLE IF NOT EXISTS "users" (
"id" serial PRIMARY KEY NOT NULL,
);
```
Looks good?
Not exactly!
`payload.db.drizzle.execute()` doesn't really use the current
transaction which can lead to some problems.
Instead, it should use the `db` from `payload.db.sessions?.[await
req.transactionID!].db` because that's where we store our Drizzle
instance with the transaction.
But now, if we generate `payload migrate:create` we get:
```ts
import { MigrateUpArgs, MigrateDownArgs, sql } from '@payloadcms/db-postgres'
export async function up({ db, payload, req }: MigrateUpArgs): Promise<void> {
await db.execute(sql`
CREATE TABLE IF NOT EXISTS "users" (
"id" serial PRIMARY KEY NOT NULL,
);
```
Which is what we want, as the `db` is passed correctly here:
76428373e4/packages/drizzle/src/migrate.ts (L88-L90)
```ts
export async function up({ db, payload, req }: MigrateUpArgs): Promise<void> {
const dbWithTransaction = payload.db.sessions?.[await req.transactionID!].db
payload.logger.info({ one: db === dbWithTransaction })
payload.logger.info({ two: db === payload.db.drizzle })
```
<img width="336" alt="image"
src="https://github.com/user-attachments/assets/f9fab5a9-44c2-44a9-95dd-8e5cf267f027">
Additionally, this PR refactors:
* `createMigration` with Drizzle - now we have sharable
`buildCreateMigration` in `@payloadcms/drizzle` to reduce copy-pasting
of the same logic.
* the `v2-v3` relationships migration for Postgres is now shared between
`db-postgres` and `db-vercel-postgres`, again to reduce copy-paste.
1. Adds flag `--skip-empty` to `migrate:create` to bypass the empty
migration file prompt.
- Blank migration file will not be created if this flag is passed.
3. Adds flag `--force-accept-warning` to `migrate:fresh` to bypass the
drop database prompt
