Continuation of https://github.com/payloadcms/payload/pull/6245.
This PR allows you to pass `blocksAsJSON: true` to SQL adapters and the
adapter instead of aligning with the SQL preferred relation approach for
blocks will just use a simple JSON column, which can improve performance
with a large amount of blocks.
To try these changes you can install `3.43.0-internal.c5bbc84`.
Currently, we globally enable both DOM and Node.js types. While this
mostly works, it can cause conflicts - particularly with `fetch`. For
example, TypeScript may incorrectly allow browser-only properties (like
`cache`) and reject valid Node.js ones like `dispatcher`.
This PR disables DOM types for server-only packages like payload,
ensuring Node-specific typings are applied. This caught a few instances
of incorrect fetch usage that were previously masked by overlapping DOM
types.
This is not a perfect solution - packages that contain both server and
client code (like richtext-lexical or next) will still suffer from this
issue. However, it's an improvement in cases where we can cleanly
separate server and client types, like for the `payload` package which
is server-only.
## Use-case
This change enables https://github.com/payloadcms/payload/pull/12622 to
explore using node-native fetch + `dispatcher`, instead of `node-fetch`
+ `agent`.
Currently, it will incorrectly report that `dispatcher` is not a valid
property for node-native fetch
Fixes https://github.com/payloadcms/payload/issues/12628
When using sqlite, the error from the db is a bit different than
Postgres.
This PR allows us to extract the fieldName when using sqlite for the
unique constraint error.
Adds support for read replicas
https://orm.drizzle.team/docs/read-replicas that can be used to offload
read-heavy traffic.
To use (both `db-postgres` and `db-vercel-postgres` are supported):
```ts
import { postgresAdapter } from '@payloadcms/db-postgres'
database: postgresAdapter({
pool: {
connectionString: process.env.POSTGRES_URL,
},
readReplicas: [process.env.POSTGRES_REPLICA_URL],
})
```
---------
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
### What?
This PR fixes an issue while using `text` & `number` fields with
`hasMany: true` where the last entry would be unreachable, and thus
undeletable, because the `transformForWrite` function did not track
these rows for deletion. This causes values that should've been deleted
to remain in the edit view form, as well as the db, after a submission.
This PR also properly threads the placeholder value from
`admin.placeholder` to `text` & `number` `hasMany: true` fields.
### Why?
To remove rows from the db when a submission is made where these fields
are empty arrays, and to properly show an appropriate placeholder when
one is set in config.
### How?
Adjusting `transformForWrite` and the `traverseFields` to keep track of
rows for deletion.
Fixes#11781
Before:
[Editing---Post-dbpg-before--Payload.webm](https://github.com/user-attachments/assets/5ba1708a-2672-4b36-ac68-05212f3aa6cb)
After:
[Editing---Post--dbpg-hasmany-after-Payload.webm](https://github.com/user-attachments/assets/1292e998-83ff-49d0-aa86-6199be319937)
Previously, this was possible in MongoDB but not in Postgres/SQLite
(having `null` in an `in` query)
```
const { docs } = await payload.find({
collection: 'posts',
where: { text: { in: ['text-1', 'text-3', null] } },
})
```
This PR fixes that behavior
This PR introduces a few changes to improve turbopack compatibility and
ensure e2e tests pass with turbopack enabled
## Changes to improve turbopack compatibility
- Use correct sideEffects configuration to fix scss issues
- Import scss directly instead of duplicating our scss rules
- Fix some scss rules that are not supported by turbopack
- Bump Next.js and all other dependencies used to build payload
## Changes to get tests to pass
For an unknown reason, flaky tests flake a lot more often in turbopack.
This PR does the following to get them to pass:
- add more `wait`s
- fix actual flakes by ensuring previous operations are properly awaited
## Blocking turbopack bugs
- [X] https://github.com/vercel/next.js/issues/76464
- Fix PR: https://github.com/vercel/next.js/pull/76545
- Once fixed: change `"sideEffectsDisabled":` back to `"sideEffects":`
## Non-blocking turbopack bugs
- [ ] https://github.com/vercel/next.js/issues/76956
## Related PRs
https://github.com/payloadcms/payload/pull/12653https://github.com/payloadcms/payload/pull/12652
The databases do not keep track of document order internally so when
sorting by non-unique fields such as shared `order` number values, the
returned order will be random and not consistent.
While this issue is far more noticeable on mongo it could also occur in
postgres on certain environments.
This combined with pagination can lead to the perception of duplicated
or inconsistent data.
This PR adds a second sort parameter to queries so that we always have a
fallback, `-createdAt` will be used by default or `id` if timestamps are
disabled.
⚠️ `orderable` fields will no longer be `required` and `unique`, so your
database may prompt you to accept an automatic migration if you're using
[this
feature](https://payloadcms.com/docs/configuration/collections#config-options).
Note that the `orderable` feature is still experimental, so it may still
receive breaking changes without a major upgrade or contain bugs. Use it
with caution.
___
The `orderable` fields will not have `required` and `unique` constraints
at the database schema level, in order to automatically migrate
collections that incorporate this property.
Now, when a user adds the `orderable` property to a collection or join
field, existing documents will have the order field set to undefined.
The first time you try to reorder them, the documents will be
automatically assigned an initial order, and you will be prompted to
refresh the page.
We believe this provides a better development experience than having to
manually migrate data with a script.
Additionally, it fixes a bug that occurred when using `orderable` in
conjunction with groups and tabs fields.
Closes:
- #12129
- #12331
- #12212
---------
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
Continuation of https://github.com/payloadcms/payload/pull/12185 and fix
https://github.com/payloadcms/payload/issues/12221
The mentioned PR introduced auto sorting by the point field when a
`near` query is used, but it didn't build actual needed query to order
results by their distance to a _given_ (from the `near` query) point.
Now, we build:
```sql
order by pont_field <-> ST_SetSRID(ST_MakePoint(lng, lat), 4326)
```
Which does what we want
This fixes issues identified in the predefined migration for
postgres v2-v3 including the following:
### relation already exists
Can error with the following:
```ts
{
err: [DatabaseError],
msg: 'Error running migration 20250502_020052_relationships_v2_v3 column "relation_id" of relation "table_name" already exists.'
}
```
This was happening when you run a migration with both a required
relationship or upload field and no schema specified in the db adapter.
When both of these are true the function that replaces `ADD COLUMN` and
`ALTER COLUMN` in order to add `NOT NULL` constraints for requried
fields, wasn't working. This resulted in the `ADD COLUMN` statement from
being being called multiple times instead of altering it after data had
been copied over.
### camelCase column change
Enum columns from using `select` or `radio` have changed from camelCase
to snake case in v3. This change was not accounted for in the
relationship migration and needed to be accounted for.
### DROP CONSTRAINT
It was pointed out by
[here](https://github.com/payloadcms/payload/issues/10162#issuecomment-2610018940)
that the `DROP CONSTRAINT` needs to include `IF EXISTS` so that it can
continue if the contraint was already removed in a previous statement.
fixes https://github.com/payloadcms/payload/issues/10162
### What?
Turborepo fails to compile due to type error in the generated drizzle
schema.
### Why?
TypeScript may not include the module augmentation for
@payloadcms/db-postgres, especially in monorepo or isolated module
builds. This causes type errors during the compilation process of
turborepo project. Adding the type-only import guarantees that
TypeScript loads the relevant type definitions and augmentations,
resolving these errors.
### How?
This PR adds a type-only import statement to ensure TypeScript
recognizes the module augmentation for @payloadcms/db-postgres in the
generated drizzle schema from payload, and there is no runtime effect.
Fixes#12311
-->
Fixes https://github.com/payloadcms/payload/issues/12263
This was caused by passing not needed columns to the `SELECT DISTINCT`
query, which we execute in case if we have a filter / sort by a nested
field / relationship. Since the only columns that we need to pass to the
`SELECT DISTINCT` query are: ID and field(s) specified in `sort`, we now
filter the `selectFields` variable.
### What?
Swaps out `deepAssertEqual` for `dequal` package. Further details and
motivation in [this
discussion](https://github.com/payloadcms/payload/discussions/12192).
### Why?
Dequal is about 100x faster in limited local testing. Dequal package
shows 3-5x speed over `deepAssertEqual` in benchmarks. Memory usage is
within acceptable levels.
### How?
Move the result of dequal to a `const` for readability. Replace the `try
{ ... } catch { ... }` with `if { ... } else { ... }` for minimum impact
and change.
When running the v2-v3 migration you might receive prompts for renaming
columns. Since we start a transaction before, you might end up with a
fail if you don't answer within your transaction session period timeout.
This moves the `getTransaction` call after prompts were answered, since
we don't have a reason to start it earlier.
When `payload migrate` is run and a record with name "dev" is returned
having `batch: -1`, then the `batch` is not incrementing as expected as
it is stuck at 1. This change makes it so the batch is incremented from
the correct latest batch, ignoring the `name: "dev"` migration.
This improves performance when querying data in Postgers / SQLite with
`limit: 0`. Before, unless you additionally passed `pagination: false`
we executed additional count query to calculate the pagination. Now we
skip this as this is unnecessary since we can retrieve the count just
from `rows.length`.
This logic already existed in `db-mongodb` -
1b17df9e0b/packages/db-mongodb/src/find.ts (L114-L124)
Fixes https://github.com/payloadcms/payload/issues/12090, in MongoDB the
documents are sorted by distance automatically whenever the `near`
operation is used, which we have in the docs:
> When querying using the near operator, the returned documents will be
sorted by nearest first.
This fixes this incosistensty between Postgres and MongoDB.
⚠️ This change potentially can cause to produce different results, if
you used the `near` operator without `sort: 'pointFieldName'`.
### What?
Converts numbers passed to a text field to avoid the database/drizzle
from converting it incorrectly.
### Why?
If you have a hook that passes a value to another field you can
experience this problem where drizzle converts a number value for a text
field to a floating point number in sqlite for example.
### How?
Adds logic to `transform/write/traverseFields.ts` to cast text field
values to string.
following changes made by Commit a6f7ef8
> feat(db-*): export types from main export (#11914)
In 3.0, we made the decision to export all types from the main package
export (e.g. `payload/types` => `payload`). This improves type
discoverability by IDEs and simplifies importing types.
> This PR does the same for our db adapters, which still have a separate
`/types` subpath export. While those are kept for
backwards-compatibility, we can remove them in 4.0.
a6f7ef837a
the script responsible for generating file generated-schema.ts was not
updated to reflect this change in export paths
drizzle/src/utilities/createSchemaGenerator.ts
CURRENT
```typescript
const finalDeclaration = `
declare module '${this.packageName}/types' {
export interface GeneratedDatabaseSchema {
schema: DatabaseSchema
}
}
```
AFTER THIS PULL REQUEST
```typescript
const finalDeclaration = `
declare module '${this.packageName}' {
export interface GeneratedDatabaseSchema {
schema: DatabaseSchema
}
}
```
this pull request fixes the generation of generated-schema.ts avoiding
errors while building for production with command
```bash
npm run build
```
### What?
This PR adds support for `where` querying by the join field (don't
confuse with `where` querying of related docs via `joins.where`)
Previously, this didn't work:
```
const categories = await payload.find({
collection: 'categories',
where: { 'relatedPosts.title': { equals: 'my-title' } },
})
```
### Why?
This is crucial for bi-directional relationships, can be used for access
control.
### How?
Implements `where` handling for join fields the same as we do for
relationships. In MongoDB it's not as efficient as it can be, the old PR
that improves it and can be updated later is here
https://github.com/payloadcms/payload/pull/8858
Fixes https://github.com/payloadcms/payload/discussions/9683
