Commit Graph

56 Commits

Author SHA1 Message Date
Alessio Gravili
9c88af4b20 refactor(drizzle): replace query chaining with dynamic query building (#11923)
This replaces usage of our `chainMethods` helper to dynamically chain
queries with [drizzle dynamic query
building](https://orm.drizzle.team/docs/dynamic-query-building).

This is more type-safe, more readable and requires less code
2025-03-31 20:37:45 +00:00
Alessio Gravili
a083d47368 feat(db-*): return database name to unsanitized config (#11913)
You can access the database name from `sanitizedConfig.db.name`. But
currently, it' not possible to access the db name from the unsanitized
config.

Plugins only have access to the unsanitized config. This change allows
db adapters to return the db name early, which will allow plugins to
conditionally initialize db-specific functionality
2025-03-31 12:57:17 -06:00
Alessio Gravili
a6f7ef837a feat(db-*): export types from main export (#11914)
In 3.0, we made the decision to export all types from the main package
export (e.g. `payload/types` => `payload`). This improves type
discoverability by IDEs and simplifies importing types.

This PR does the same for our db adapters, which still have a separate
`/types` subpath export. While those are kept for
backwards-compatibility, we can remove them in 4.0.
2025-03-31 15:45:02 +00:00
Alessio Gravili
a5c3aa0e4f perf: reduce job queue db calls (#11846)
Continuation of #11489. This adds a new, optional `updateJobs` db
adapter method that reduces the amount of database calls for the jobs
queue.

## MongoDB

### Previous: running a set of 50 queued jobs
- 1x db.find (= 1x `Model.paginate`)
- 50x db.updateOne (= 50x `Model.findOneAndUpdate`)

### Now: running a set of 50 queued jobs
- 1x db.updateJobs (= 1x `Model.find` and 1x `Model.updateMany`)

**=> 51 db round trips before, 2 db round trips after**


### Previous: upon task completion
- 1x db.find (= 1x `Model.paginate`)
- 1x db.updateOne (= 1x `Model.findOneAndUpdate`)

### Now: upon task completion
- 1x db.updateJobs (= 1x `Model.findOneAndUpdate`)


**=> 2 db round trips before, 1 db round trip after**


## Drizzle (e.g. Postgres)

### running a set of 50 queued jobs
 - 1x db.query[tablename].findMany
 - 50x db.select 
 - 50x upsertRow
 
This is unaffected by this PR and will be addressed in a future PR
2025-03-25 18:09:52 +00:00
Germán Jabloñski
fe9317a0dd chore(db-sqlite): enable TypeScript strict (#11831)
- I installed `@types/uuid` because typescript required it in a file
- In `packages/db-sqlite/src/index.ts` I see four more errors in my IDE
that don't appear when I run the typecheck in the CLI with `tsc
--noEmit`. The same thing happened in
https://github.com/payloadcms/payload/pull/11560. Also referencing
https://github.com/payloadcms/payload/pull/11226#issuecomment-2713898801
for traceability.
2025-03-24 23:41:07 -03:00
Sasha
f442d22237 feat(db-*): allow to thread id to create operation data without custom IDs (#11709)
Fixes https://github.com/payloadcms/payload/issues/6884

Adds a new flag `acceptIDOnCreate` that allows you to thread your own
`id` to `payload.create` `data`, for example:

```ts
// doc created with id 1
const doc = await payload.create({ collection: 'posts', data: {id: 1, title: "my title"}})
```

```ts
import { Types } from 'mongoose'
const id = new Types.ObjectId().toHexString()
const doc = await payload.create({ collection: 'posts', data: {id, title: "my title"}})
```
2025-03-17 23:48:35 -04:00
Alessio Gravili
41c7413f59 feat(db-*): add updateMany method to database adapter (#11441)
This PR adds a new `payload.db.updateMany` method, which is a more performant way to update multiple documents compared to using `payload.update`.
2025-02-27 20:30:17 -07:00
Paul
b014416584 feat: add support for not_like operation (#11326)
This PR adds support for `not_like` as a query operation. Functions just
as a negative to `like`, uses `ilike` in postgres and `like` in sqlite
2025-02-22 00:32:37 +00:00
Sasha
513ba636af fix(db-postgres): ensure countDistinct works correctly and achieve better performance when the query has table joins (#11208)
The fix, added in https://github.com/payloadcms/payload/pull/11096
wasn't sufficient enough. It did handle the case when the same query
path / table was joined twice and caused incorrect `totalDocs`, but it
didn't handle the case when `JOIN` returns more than 1 rows, which 2
added new assertions here check.

Now, we use `COUNT(*)` only if we don't have any joined tables. If we
do, instead of using `SELECT (COUNT DISTINCT id)` which as described in
the previous PR is _very slow_ for large tables, we use the following
query:

```sql
SELECT COUNT(1) OVER() as count -- window function, executes for each row only once
FROM users
LEFT JOIN -- ... here additional rows are added
WHERE -- ...
GROUP BY users.id -- this ensures we're counting only users without additional rows from joins. 
LIMIT 1 -- Since COUNT(1) OVER() executes and resolves before doing LIMIT, we can safely apply LIMIT 1.
```
2025-02-16 14:08:08 +02:00
Sasha
98fec35368 fix(db-postgres): incorrect pagination results when querying hasMany relationships multiple times (#11096)
Fixes https://github.com/payloadcms/payload/issues/10810

This was caused by using `COUNT(*)` aggregation instead of
`COUNT(DISTINCT table.id)`. However, we want to use `COUNT(*)` because
`COUNT(DISTINCT table.id)` is slow on large tables. Now we fallback to
`COUNT(DISTINCT table.id)` only when `COUNT(*)` cannot work properly.

Example of a query that leads to incorrect `totalDocs`:
```ts
const res = await payload.find({
  collection: 'directors',
  limit: 10,
  where: {
    or: [
      {
        movies: {
          equals: movie2.id,
        },
      },
      {
        movies: {
          equals: movie1.id,
        },
      },
      {
        movies: {
          equals: movie1.id,
        },
      },
    ],
  },
})
```
2025-02-11 01:16:18 +02:00
Sasha
4e953530df feat(db-sqlite): add autoIncrement option (#9427)
### What?
Exposes ability to enable
[AUTOINCREMENT](https://www.sqlite.org/autoinc.html) for Primary Keys
which ensures that the same ID cannot be reused from previously deleted
rows.

```ts
sqliteAdapter({
  autoIncrement: true
})
```

### Why?
This may be essential for some systems. Enabled `autoIncrement: true`
also for the SQLite Adapter in our tests, which can be useful when
testing whether the doc was deleted or not when you also have other
create operations.

### How?
Uses Drizzle's `autoIncrement` option.

WARNING:
This cannot be enabled in an existing project without a custom
migration, as it completely changes how primary keys are stored in the
database.
2024-12-20 20:13:28 +00:00
Sasha
23f1ed4a48 feat(db-postgres, db-sqlite): drizzle schema generation (#9953)
This PR allows to have full type safety on `payload.drizzle` with a
single command
```sh
pnpm payload generate:db-schema
```
Which generates TypeScript code with Drizzle declarations based on the
current database schema.

Example of generated file with the website template: 
https://gist.github.com/r1tsuu/b8687f211b51d9a3a7e78ba41e8fbf03

Video that shows the power:


https://github.com/user-attachments/assets/3ced958b-ec1d-49f5-9f51-d859d5fae236


We also now proxy drizzle package the same way we do for Lexical so you
don't have to install it (and you shouldn't because you may have version
mismatch).
Instead, you can import from Drizzle like this:
```ts
import {
  pgTable,
  index,
  foreignKey,
  integer,
  text,
  varchar,
  jsonb,
  boolean,
  numeric,
  serial,
  timestamp,
  uniqueIndex,
  pgEnum,
} from '@payloadcms/db-postgres/drizzle/pg-core'
import { sql } from '@payloadcms/db-postgres/drizzle'
import { relations } from '@payloadcms/db-postgres/drizzle/relations'
```


Fixes https://github.com/payloadcms/payload/discussions/4318

In the future we can also support types generation for mongoose / raw
mongodb results.
2024-12-19 11:08:17 -05:00
Sasha
03ff77544e feat(db-sqlite): add idType: 'uuid' support (#10016)
Adds `idType: 'uuid'` to the SQLite adapter support:
```ts
sqliteAdapter({
  idType: 'uuid',
})
```

Achieved through Drizzle's `$defaultFn()`
https://orm.drizzle.team/docs/latest-releases/drizzle-orm-v0283#-added-defaultfn--default-methods-to-column-builders
as SQLite doesn't have native UUID support. Added `sqlite-uuid` to CI.
2024-12-18 22:44:04 -05:00
Sasha
727fba7b1c refactor: deduplicate and abstract SQL schema building (#9987)
### What?
Abstracts SQL schema building, significantly reducing code duplication
for SQLite / Postgres

db-sqlite lines count From:
```sh
 wc -l **/*.ts
      62 src/connect.ts
      32 src/countDistinct.ts
       9 src/createJSONQuery/convertPathToJSONTraversal.ts
      86 src/createJSONQuery/index.ts
      15 src/defaultSnapshot.ts
       6 src/deleteWhere.ts
      21 src/dropDatabase.ts
      15 src/execute.ts
     178 src/index.ts
     139 src/init.ts
      19 src/insert.ts
      19 src/requireDrizzleKit.ts
     544 src/schema/build.ts
      27 src/schema/createIndex.ts
      38 src/schema/getIDColumn.ts
      13 src/schema/idToUUID.ts
      28 src/schema/setColumnID.ts
     787 src/schema/traverseFields.ts
      18 src/schema/withDefault.ts
     248 src/types.ts
    2304 total
```

To:
```sh
wc -l **/*.ts
      62 src/connect.ts
      32 src/countDistinct.ts
       9 src/createJSONQuery/convertPathToJSONTraversal.ts
      86 src/createJSONQuery/index.ts
      15 src/defaultSnapshot.ts
       6 src/deleteWhere.ts
      21 src/dropDatabase.ts
      15 src/execute.ts
     180 src/index.ts
      39 src/init.ts
      19 src/insert.ts
      19 src/requireDrizzleKit.ts
     149 src/schema/buildDrizzleTable.ts
      32 src/schema/setColumnID.ts
     258 src/types.ts
     942 total
```

Builds abstract schema in shared drizzle package that later gets
converted by SQLite/ Postgres specific implementation into drizzle
schema.
This, apparently can also help here
https://github.com/payloadcms/payload/pull/9953. It should be very
trivial to implement a MySQL adapter with this as well.
2024-12-16 17:17:18 +02:00
Sasha
00909ec5c4 fix(db-sqlite): working point field CRUD and default value (#9989)
Previously, the point field with SQLite was incorrectly built to the
schema and not parsed to the result.
Now, it works with SQLite, just doesn't support queries (`near` etc.).
Fixes
https://github.com/payloadcms/payload/pull/9987#discussion_r1885674299
2024-12-16 07:37:47 +02:00
Sasha
b73fc586b8 feat: expose session, db in migrations to use the active transaction with the database directly (#9849)
This PR adds a feature which fixes another issue with migrations in
Postgres and does few refactors that significantly reduce code
duplication.

Previously, if you needed to use the underlying database directly in
migrations with the active transaction (for example to execute raw SQL),
created from `payload create:migration`, as `req` doesn't work there you
had to do something like this:
```ts
// Postgres
export async function up({ payload, req }: MigrateUpArgs): Promise<void> {
  const db = payload.db.sessions?.[await req.transactionID!].db ?? payload.db
  const { rows: posts } = await db.execute(sql`SELECT * from posts`)
}

// MongoDB
export async function up({ payload, req }: MigrateUpArgs): Promise<void> {
  const session = payload.db.sessions?.[await req.transactionID!]
  const posts = await payload.db.collections.posts.collection.find({ session }).toArray()
}
```

Which was:
1. Awkward to write
2. Not documented anywhere

Now, we expose `session` and `db` to `up` and `down` functions for you:

#### MongoDB:
```ts
import { type MigrateUpArgs } from '@payloadcms/db-mongodb'

export async function up({ session, payload, req }: MigrateUpArgs): Promise<void> {
  const posts = await payload.db.collections.posts.collection.find({ session }).toArray()
}
```
#### Postgres:
```ts
import { type MigrateUpArgs, sql } from '@payloadcms/db-postgres'

export async function up({ db, payload, req }: MigrateUpArgs): Promise<void> {
  const { rows: posts } = await db.execute(sql`SELECT * from posts`)
}
```

#### SQLite:
```ts
import { type MigrateUpArgs, sql } from '@payloadcms/db-sqlite'

export async function up({ db, payload, req }: MigrateUpArgs): Promise<void> {
  const { rows: posts } = await db.run(sql`SELECT * from posts`)
}
```
This actually was a thing with Postgres migrations, we already were
passing `db`, but:
1. Only for `up` and when running `payload migrate`, not for example
with `payload migrate:fresh`
2. Not documented neither in TypeScript or docs.

By ensuring we use `db`, this also fixes an issue that affects all
Postgres/SQLite migrations:

Currently, if we run `payload migration:create` with the postgres
adapter we get a file like this:

```ts
import { MigrateUpArgs, MigrateDownArgs, sql } from '@payloadcms/db-postgres'

export async function up({ payload, req }: MigrateUpArgs): Promise<void> {
  await payload.db.drizzle.execute(sql`
   CREATE TABLE IF NOT EXISTS "users" (
  	"id" serial PRIMARY KEY NOT NULL,
  );
```
Looks good?
Not exactly!
`payload.db.drizzle.execute()` doesn't really use the current
transaction which can lead to some problems.
Instead, it should use the `db` from `payload.db.sessions?.[await
req.transactionID!].db` because that's where we store our Drizzle
instance with the transaction.

But now, if we generate `payload migrate:create` we get:
```ts
import { MigrateUpArgs, MigrateDownArgs, sql } from '@payloadcms/db-postgres'

export async function up({ db, payload, req }: MigrateUpArgs): Promise<void> {
  await db.execute(sql`
   CREATE TABLE IF NOT EXISTS "users" (
  	"id" serial PRIMARY KEY NOT NULL,
  );
```

Which is what we want, as the `db` is passed correctly here:

76428373e4/packages/drizzle/src/migrate.ts (L88-L90)

```ts
export async function up({ db, payload, req }: MigrateUpArgs): Promise<void> {
  const dbWithTransaction = payload.db.sessions?.[await req.transactionID!].db
  payload.logger.info({ one: db === dbWithTransaction })
  payload.logger.info({ two: db === payload.db.drizzle })
```
<img width="336" alt="image"
src="https://github.com/user-attachments/assets/f9fab5a9-44c2-44a9-95dd-8e5cf267f027">

Additionally, this PR refactors:
* `createMigration` with Drizzle - now we have sharable
`buildCreateMigration` in `@payloadcms/drizzle` to reduce copy-pasting
of the same logic.
* the `v2-v3` relationships migration for Postgres is now shared between
`db-postgres` and `db-vercel-postgres`, again to reduce copy-paste.
2024-12-11 12:23:12 -05:00
Sasha
cae300e8e3 perf: flattenedFields collection/global property, remove deep copying in validateQueryPaths (#9299)
### What?
Improves querying performance of the Local API, reduces copying overhead

### How?

Adds `flattenedFields` property for sanitized collection/global config
which contains fields in database schema structure.
For example, It removes rows / collapsible / unnamed tabs and merges
them to the parent `fields`, ignores UI fields, named tabs are added as
`type: 'tab'`.
This simplifies code in places like Drizzle `transform/traverseFields`.
Also, now we can avoid calling `flattenTopLevelFields` which adds some
overhead and use `collection.flattenedFields` / `field.flattenedFields`.

By refactoring `configToJSONSchema.ts` with `flattenedFields` it also
1. Fixes https://github.com/payloadcms/payload/issues/9467
2. Fixes types for UI fields were generated for `select`



Removes this deep copying for each `where` query constraint
58ac784425/packages/payload/src/database/queryValidation/validateQueryPaths.ts (L69-L73)
which potentially can add overhead if you have a large collection/global
config

UPD:
The overhead is even much more than in the benchmark below if you have
Lexical with blocks.

Benchmark in `relationships/int.spec.ts`:
```ts
const now = Date.now()
for (let i = 0; i < 10; i++) {
  const now = Date.now()
  for (let i = 0; i < 300; i++) {
    const query = await payload.find({
      collection: 'chained',
      where: {
        'relation.relation.name': {
          equals: 'third',
        },
        and: [
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
        ],
      },
    })
  }

  payload.logger.info(`#${i + 1} ${Date.now() - now}`)
}
payload.logger.info(`Total ${Date.now() - now}`)
```

Before:
```
[02:11:48] INFO: #1 3682
[02:11:50] INFO: #2 2199
[02:11:54] INFO: #3 3483
[02:11:56] INFO: #4 2516
[02:11:59] INFO: #5 2467
[02:12:01] INFO: #6 1987
[02:12:03] INFO: #7 1986
[02:12:05] INFO: #8 2375
[02:12:07] INFO: #9 2040
[02:12:09] INFO: #10 1920
    [PASS] Relationships > Querying > Nested Querying > should allow querying two levels deep (24667ms)
[02:12:09] INFO: Total 24657
```

After:
```
[02:12:36] INFO: #1 2113
[02:12:38] INFO: #2 1854
[02:12:40] INFO: #3 1700
[02:12:42] INFO: #4 1797
[02:12:44] INFO: #5 2121
[02:12:46] INFO: #6 1774
[02:12:47] INFO: #7 1670
[02:12:49] INFO: #8 1610
[02:12:50] INFO: #9 1596
[02:12:52] INFO: #10 1576
    [PASS] Relationships > Querying > Nested Querying > should allow querying two levels deep (17828ms)
[02:12:52] INFO: Total 17818
```
2024-11-25 10:28:07 -05:00
Jessica Chowdhury
fb3242df0a feat: add skip and force accept flags to migration commands (#8843)
1. Adds flag `--skip-empty` to `migrate:create` to bypass the empty
migration file prompt.
    - Blank migration file will not be created if this flag is passed.
3. Adds flag `--force-accept-warning` to `migrate:fresh` to bypass the
drop database prompt
2024-11-22 10:37:54 -05:00
Elliot DeNolf
5ff1bb366c chore: misc cleanup (#9206)
- Proper error logger usage
- Some no-fallthrough warning cleanup
2024-11-14 11:14:08 -05:00
Alessio Gravili
03291472d6 chore: bump all eslint dependencies, run lint and prettier (#9128)
This fixes a peer dependency error in our monorepo, as
eslint-plugin-jsx-a11y finally supports eslint v9.

Additionally, this officially adds TypeScript 5.6 support for
typescript-eslint.
2024-11-12 10:18:22 -05:00
Jacob Fletcher
c96fa613bc feat!: on demand rsc (#8364)
Currently, Payload renders all custom components on initial compile of
the admin panel. This is problematic for two key reasons:
1. Custom components do not receive contextual data, i.e. fields do not
receive their field data, edit views do not receive their document data,
etc.
2. Components are unnecessarily rendered before they are used

This was initially required to support React Server Components within
the Payload Admin Panel for two key reasons:
1. Fields can be dynamically rendered within arrays, blocks, etc.
2. Documents can be recursively rendered within a "drawer" UI, i.e.
relationship fields
3. Payload supports server/client component composition 

In order to achieve this, components need to be rendered on the server
and passed as "slots" to the client. Currently, the pattern for this is
to render custom server components in the "client config". Then when a
view or field is needed to be rendered, we first check the client config
for a "pre-rendered" component, otherwise render our client-side
fallback component.

But for the reasons listed above, this pattern doesn't exactly make
custom server components very useful within the Payload Admin Panel,
which is where this PR comes in. Now, instead of pre-rendering all
components on initial compile, we're able to render custom components
_on demand_, only as they are needed.

To achieve this, we've established [this
pattern](https://github.com/payloadcms/payload/pull/8481) of React
Server Functions in the Payload Admin Panel. With Server Functions, we
can iterate the Payload Config and return JSX through React's
`text/x-component` content-type. This means we're able to pass
contextual props to custom components, such as data for fields and
views.

## Breaking Changes

1. Add the following to your root layout file, typically located at
`(app)/(payload)/layout.tsx`:

    ```diff
    /* THIS FILE WAS GENERATED AUTOMATICALLY BY PAYLOAD. */
    /* DO NOT MODIFY IT BECAUSE IT COULD BE REWRITTEN AT ANY TIME. */
    + import type { ServerFunctionClient } from 'payload'

    import config from '@payload-config'
    import { RootLayout } from '@payloadcms/next/layouts'
    import { handleServerFunctions } from '@payloadcms/next/utilities'
    import React from 'react'

    import { importMap } from './admin/importMap.js'
    import './custom.scss'

    type Args = {
      children: React.ReactNode
    }

+ const serverFunctions: ServerFunctionClient = async function (args) {
    +  'use server'
    +  return handleServerFunctions({
    +    ...args,
    +    config,
    +    importMap,
    +  })
    + }

    const Layout = ({ children }: Args) => (
      <RootLayout
        config={config}
        importMap={importMap}
    +  serverFunctions={serverFunctions}
      >
        {children}
      </RootLayout>
    )

    export default Layout
    ```

2. If you were previously posting to the `/api/form-state` endpoint, it
no longer exists. Instead, you'll need to invoke the `form-state` Server
Function, which can be done through the _new_ `getFormState` utility:

    ```diff
    - import { getFormState } from '@payloadcms/ui'
    - const { state } = await getFormState({
    -   apiRoute: '',
    -   body: {
    -     // ...
    -   },
    -   serverURL: ''
    - })

    + const { getFormState } = useServerFunctions()
    +
    + const { state } = await getFormState({
    +   // ...
    + })
    ```

## Breaking Changes

```diff
- useFieldProps()
- useCellProps()
```

More details coming soon.

---------

Co-authored-by: Alessio Gravili <alessio@gravili.de>
Co-authored-by: Jarrod Flesch <jarrodmflesch@gmail.com>
Co-authored-by: James <james@trbl.design>
2024-11-11 13:59:05 -05:00
Sasha
9069bd3fac fix(db-postgres): sort by localized fields (#8839)
### What?
Fixes https://github.com/payloadcms/payload/issues/5152 issue related to
sorting by a localized field with SQLite / Postgres database adapters.

### Why?
It was an incorrect behaviour.


### How?
Modifies the `getTableColumnFromPath` file to have correct join
conditions. Previously if you had this structure in the _locales table
_locale title parent
en          A    1
es          B     1
we sorted by everything that's here, but we need to sort only by the
passed locale.

Additionally fixes a typescript error in `dev.ts` that I added here
https://github.com/payloadcms/payload/pull/8834

Also, removes the condition with `joins.length` in `countDistinct`. It
was there as for this issue
https://github.com/payloadcms/payload/issues/4889 because sorting by a
localized property caused duplication. This can simnifically improve
performance for `.find` with nested querying/sorting on large data sets,
because `count(*)` is faster than `count(DISTINCT id)`
2024-10-24 14:47:58 -04:00
Sasha
8af00f2deb fix: join field works on collections with versions enabled (#8715)
- Fixes errors with drizzle when building the schema
https://github.com/payloadcms/payload/issues/8680
- Adds `joins` to `db.queryDrafts` to have them when doing `.find` with
`draft: true`
2024-10-22 11:05:55 -04:00
Sasha
90bca15f52 fix(drizzle): enforce uniqueness on index names (#8754)
Fixes https://github.com/payloadcms/payload/issues/8752

Previously, trying to define a config like this:
```ts
{
  type: 'text',
  name: 'someText',
  index: true,
},
{
  type: 'array',
  name: 'some',
  index: true,
  fields: [
    {
      type: 'text',
      name: 'text',
      index: true,
    },
  ],
}
```

Lead to the error:
```
Warning  We've found duplicated index name across public schema. Please rename your index in either the demonstration table or the table with the duplicated index name
```

Now, if we encounter duplicates, we increment the name like this:
`collection_some_text_idx`
`collection_some_text_1_idx`

---------

Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
2024-10-17 02:05:27 +00:00
Dan Ribbens
93545f3103 fix(db-postgres, db-sqlite)!: bump drizzle-kit drizzle-orm @libsql/client (#8617)
Inheriting all the fixes from drizzle moving to latest versions

## BREAKING CHANGES
If you have a prior version of @libsql/client installed in your project,
you must upgrade to 0.14.0
2024-10-15 20:20:50 +00:00
Sasha
1ffb6c3e13 fix(drizzle): indexes / unique with relationships (#8432)
Fixes https://github.com/payloadcms/payload/issues/8413 and
https://github.com/payloadcms/payload/issues/6460

- Builds indexes for relationships by default in the SQL schema
- Fixes `unique: true` handling with Postgres / SQLite for every type of
relationships (non-polymorphic. hasMany, polymorphic, polymorphic
hasMany) _note_: disables unique for nested to arrays / blocks
relationships in the `_rels` table.
- adds tests

2.0 PR tha ports only indexes creation
https://github.com/payloadcms/payload/pull/8446, because `unique: true`
could be a breaking change if someone has incosistent unique data in the
database.
2024-10-11 13:13:54 -04:00
Dan Ribbens
1bf580fac3 feat: join field works with hasMany relationships (#8493)
Join field works on relationships and uploads having `hasMany: true`

---------

Co-authored-by: Sasha <64744993+r1tsuu@users.noreply.github.com>
2024-10-08 11:40:34 -04:00
Sasha
0128eedf70 fix(drizzle)!: make radio and select column names to snake_case (#8439)
Fixes https://github.com/payloadcms/payload/issues/8402 and
https://github.com/payloadcms/payload/issues/8027

Before DB column names were camelCase:

![image](https://github.com/user-attachments/assets/d2965bcf-290a-4f86-9bf4-dfe7e8613934)

After this change, they are snake_case:
![Screenshot 2024-10-04
114226](https://github.com/user-attachments/assets/bbc8c20b-6745-4dd3-b0c8-56263a4e37b1)

#### Breaking SQLite / Postgres ⚠️  
If you had any select (not `hasMany: true`) or radio fields with the
name in camelCase, for example:
```ts
{
  name: 'selectReadOnly',
  type: 'select',
  admin: {
    readOnly: true,
  },
  options: [
    {
      label: 'Value One',
      value: 'one',
    },
    {
      label: 'Value Two',
      value: 'two',
    },
  ],
},
```
This previously was mapped to the db column name `"selectReadOnly"`. Now
it's `select_read_only`.
Generate a new migration to rename your columns.
```sh
pnpm payload migrate:create
```
Then select "rename column" for targeted columns and Drizzle will handle
the migration.

---------

Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
2024-10-04 16:25:05 +00:00
Dan Ribbens
27b1629927 fix(db-postgres, db-sqlite): joins relation name (#8491)
A join field on a relationship with a camelCase name would cause an
error from drizzle due to the relation name not matching.
2024-09-30 16:47:21 -04:00
Dan Ribbens
3847428f0a fix(db-*): make db.begintransactions required (#8419)
Changing the transcations functions on the db so that projects using
typescript `strict: true` do not need to type narrow before using it.
2024-09-30 13:12:07 -04:00
Sasha
613d3b090e fix(drizzle): hasMany / poly relationships nested to localized fields / nested blocks to localized fields (#8456)
fixes https://github.com/payloadcms/payload/issues/8455 and
https://github.com/payloadcms/payload/issues/8462

- Builds the `_locale` column for the `_rels` when it's inside of a
localized group / tab
- Properly builds `sanitizedPath` for blocks in the transform-read
function when it's inside of a localized field. This fixes with fields
inside that have its own table (like `_rels`, select `hasMany: true`
etc)

Adds _more_ tests!
2024-09-28 17:33:50 +03:00
Sasha
8acbda078e feat(drizzle): customize schema with before / after init hooks (#8196)
Adds abillity to customize the generated Drizzle schema with
`beforeSchemaInit` and `afterSchemaInit`. Could be useful if you want to
preserve the existing database schema / override the generated one with
features that aren't supported from the Payload config.

## Docs:

### beforeSchemaInit

Runs before the schema is built. You can use this hook to extend your
database structure with tables that won't be managed by Payload.

```ts
import { postgresAdapter } from '@payloadcms/db-postgres'
import { integer, pgTable, serial } from 'drizzle-orm/pg-core'

postgresAdapter({
  beforeSchemaInit: [
    ({ schema, adapter }) => {
      return {
        ...schema,
        tables: {
          ...schema.tables,
          addedTable: pgTable('added_table', {
            id: serial('id').notNull(),
          }),
        },
      }
    },
  ],
})
```

One use case is preserving your existing database structure when
migrating to Payload. By default, Payload drops the current database
schema, which may not be desirable in this scenario.
To quickly generate the Drizzle schema from your database you can use
[Drizzle
Introspection](https://orm.drizzle.team/kit-docs/commands#introspect--pull)
You should get the `schema.ts` file which may look like this:

```ts
import { pgTable, uniqueIndex, serial, varchar, text } from 'drizzle-orm/pg-core'

export const users = pgTable('users', {
  id: serial('id').primaryKey(),
  fullName: text('full_name'),
  phone: varchar('phone', { length: 256 }),
})

export const countries = pgTable(
  'countries',
  {
    id: serial('id').primaryKey(),
    name: varchar('name', { length: 256 }),
  },
  (countries) => {
    return {
      nameIndex: uniqueIndex('name_idx').on(countries.name),
    }
  },
)

```

You can import them into your config and append to the schema with the
`beforeSchemaInit` hook like this:

```ts
import { postgresAdapter } from '@payloadcms/db-postgres'
import { users, countries } from '../drizzle/schema'

postgresAdapter({
  beforeSchemaInit: [
    ({ schema, adapter }) => {
      return {
        ...schema,
        tables: {
          ...schema.tables,
          users,
          countries
        },
      }
    },
  ],
})
```

Make sure Payload doesn't overlap table names with its collections. For
example, if you already have a collection with slug "users", you should
either change the slug or `dbName` to change the table name for this
collection.


### afterSchemaInit

Runs after the Drizzle schema is built. You can use this hook to modify
the schema with features that aren't supported by Payload, or if you
want to add a column that you don't want to be in the Payload config.
To extend a table, Payload exposes `extendTable` utillity to the args.
You can refer to the [Drizzle
documentation](https://orm.drizzle.team/docs/sql-schema-declaration).
The following example adds the `extra_integer_column` column and a
composite index on `country` and `city` columns.

```ts
import { postgresAdapter } from '@payloadcms/db-postgres'
import { index, integer } from 'drizzle-orm/pg-core'
import { buildConfig } from 'payload'

export default buildConfig({
  collections: [
    {
      slug: 'places',
      fields: [
        {
          name: 'country',
          type: 'text',
        },
        {
          name: 'city',
          type: 'text',
        },
      ],
    },
  ],
  db: postgresAdapter({
    afterSchemaInit: [
      ({ schema, extendTable, adapter }) => {
        extendTable({
          table: schema.tables.places,
          columns: {
            extraIntegerColumn: integer('extra_integer_column'),
          },
          extraConfig: (table) => ({
            country_city_composite_index: index('country_city_composite_index').on(
              table.country,
              table.city,
            ),
          }),
        })

        return schema
      },
    ],
  }),
})

```



<!--

For external contributors, please include:

- A summary of the pull request and any related issues it fixes.
- Reasoning for the changes made or any additional context that may be
useful.

Ensure you have read and understand the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository.

 -->
2024-09-25 15:14:03 -04:00
Paul
c6519aba8a fix(drizzle): migrate args no longer partial payload request (#8375)
The arguments for migration up or down is now the full PayloadRequest instead of partial
2024-09-25 09:29:37 -04:00
Dan Ribbens
82ba1930e5 feat: add upsert to database interface and adapters (#8397)
- Adds the upsert method to the database interface
- Adds a mongodb specific option to extend the updateOne to accept
mongoDB Query Options (to pass `upsert: true`)
- Added upsert method to all database adapters
- Uses db.upsert in the payload preferences update operation

Includes a test using payload-preferences
2024-09-25 09:23:54 -04:00
Sasha
338c93a229 fix(drizzle): array/relationship/select hasMany in localized field (#8355)
This PR addresses these issues with localized groups / tabs with
Postgres / SQLite:

- Array fields inside of localized groups. Fixes
https://github.com/payloadcms/payload/issues/8322
- Select fields with `hasMany: true` inside of localized groups. Related
to 1, but still needed its own additional logic.
- Relationship (non-polymorphic / non has-many) inside of localized
groups. Previously, even just trying to define them in the config led to
a crash. Fixes https://github.com/payloadcms/payload/issues/8308

Ensures test coverage for localized groups.
2024-09-23 11:34:02 -04:00
Dan Ribbens
6ef2bdea15 feat!: join field (#7518)
## Description

- Adds a new "join" field type to Payload and is supported by all database adapters
- The UI uses a table view for the new field
- `db-mongodb` changes relationships to be stored as ObjectIDs instead of strings (for now querying works using both types internally to the DB so no data migration should be necessary unless you're querying directly, see breaking changes for details
- Adds a reusable traverseFields utility to Payload to make it easier to work with nested fields, used internally and for plugin maintainers

```ts
export const Categories: CollectionConfig = {
    slug: 'categories',
    fields: [
        {
            name: 'relatedPosts',
            type: 'join',
            collection: 'posts',
            on: 'category',
        }
    ]
}
```

BREAKING CHANGES:
All mongodb relationship and upload values will be stored as MongoDB ObjectIDs instead of strings going forward. If you have existing data and you are querying data directly, outside of Payload's APIs, you get different results. For example, a `contains` query will no longer works given a partial ID of a relationship since the ObjectID requires the whole identifier to work. 

---------

Co-authored-by: Jacob Fletcher <jacobsfletch@gmail.com>
Co-authored-by: James <james@trbl.design>
2024-09-20 11:10:16 -04:00
Sasha
bf48af411d feat: add virtual property to the fields config (#7621)
## Description

Adds `virtual` property to the fields config. Providing `true`
completely disables the field in the DB, which is useful for [Virtual
Fields](https://payloadcms.com/blog/learn-how-virtual-fields-can-help-solve-common-cms-challenges)
Disables abillity to query by a field with `virtual: true`.
Currently, they bloat the DB with unused tables / columns, which may as
well introduce additional joins.
Discussion https://github.com/payloadcms/payload/discussions/6270
Prev PR (this one contains only this feature):
https://github.com/payloadcms/payload/pull/6983

- [x] I have read and understand the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository.

## Type of change

<!-- Please delete options that are not relevant. -->

- [x] New feature (non-breaking change which adds functionality)
- [x] This change requires a documentation update

## Checklist:

- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] Existing test suite passes locally with my changes
- [x] I have made corresponding changes to the documentation
2024-09-17 10:40:54 -04:00
Sasha
8520fd9570 fix(drizzle): optimize count querying when no joins (#7749)
Closes https://github.com/payloadcms/payload/issues/6321


To run benchmark:
`git checkout b840222` - from r1tsuu/payload
b840222784
`pnpm dev:postgres _community`

Benchmark results: (Before / After)
Postgres 400 000 rows:

![image](https://github.com/user-attachments/assets/cd7c478f-2057-4c7c-adec-5dbf0b05ec7b)
Postgres 2 000 000 rows:

![image](https://github.com/user-attachments/assets/04224f95-77eb-42ab-9591-887b197c597a)

SQLite 400 000 rows:

![image](https://github.com/user-attachments/assets/ba7482c2-30f1-4498-892d-59710639a7b3)
SQLite 2 000 000 rows:

![image](https://github.com/user-attachments/assets/c0a889f8-8e21-4b98-ac92-65ac735b8b32)



## Description

<!-- Please include a summary of the pull request and any related issues
it fixes. Please also include relevant motivation and context. -->

- [x] I have read and understand the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository.

## Type of change

<!-- Please delete options that are not relevant. -->


- [x] Bug fix (non-breaking change which fixes an issue)

## Checklist:

- [x] Existing test suite passes locally with my changes - See
https://github.com/payloadcms/payload/pull/7749#issuecomment-2295763721
2024-09-16 16:21:06 -04:00
Jessica Chowdhury
b7a0b15786 feat: add publish specific locale (#7669)
## Description

1. Adds ability to publish a specific individual locale (collections and
globals)
2. Shows published locale in versions list and version comparison
3. Adds new int tests to `versions` test suite

- [X] I have read and understand the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository.

## Type of change

- [X] New feature (non-breaking change which adds functionality)
- [ ] This change requires a documentation update

## Checklist:

- [X] I have added tests that prove my fix is effective or that my
feature works
- [X] Existing test suite passes locally with my changes
- [ ] I have made corresponding changes to the documentation

---------

Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
2024-09-16 16:15:29 -04:00
Sasha
b7db53cef4 fix(drizzle)!: localized fields uniqueness per locale (#8230)
Previously, this worked with MongoDB but failed with Postgres / SQLite
when the `slug` field has both `localized: true` and `unique: true`.

```ts
await payload.create({
  collection: "posts",
  locale: "en",
  data: {
    slug: "my-post"
  }
})

await payload.create({
  collection: "posts",
  locale: "de",
  data: {
    slug: "my-post"
  }
})
```

Now, we build unique constraints and indexes in combination with the
_locale column. This should also improve query performance for fields
with both index: true and localized: true.

### Migration steps (Postgres/SQLite only)
This change updates the database schema and requires a migration (if you
have any localized fields). To apply it, run the following commands:

```sh
pnpm payload migration:create locale_unique_indexes
pnpm payload migrate
```

Note that if you use `db.push: true` which is a default, you don't have
to run `pnpm payload migrate` in the development mode, only in the
production, as Payload automatically pushes the schema to your DB with
it.
2024-09-16 14:47:13 -04:00
James Mikrut
5873a3db06 fix: duplicating localized nested arrays (#8220)
Fixes an issue where duplicating documents in Postgres / SQLite would
crash because of a foreign key constraint / unique ID issue when you
have nested arrays / blocks within localized arrays / blocks.

We now run `beforeDuplicate` against all locales prior to
`beforeValidate` and `beforeChange` hooks.

This PR also fixes a series of issues in Postgres / SQLite where you
have localized groups / named tabs, and then arrays / blocks within the
localized groups / named tabs.
2024-09-15 02:51:31 +00:00
Elliot DeNolf
08fdbcacc0 chore: proper error log format (#8105)
Fix some error log formatting to use `{ msg, err }` properly
2024-09-07 02:48:59 +00:00
Alessio Gravili
7fd736ea5b chore: lint entire codebase including db packages 2024-09-03 11:19:19 -04:00
James Mikrut
18b0806b5b fix(db-postgres, db-sqlite): hasMany text, number, poly relationship, blocks, arrays within localized fields (#7900)
## Description

In Postgres, localized blocks or arrays that contain other array / block
/ relationship fields were not properly storing locales in the database.

Now they are! Need to check a few things yet:

- Ensure test coverage is sufficient
- Test localized array, with non-localized array inside of it
- Test localized block with relationship field within it
- Ensure `_rels` table gets the `locale` column added if a single
non-localized relationship exists within a localized array / block

Fixes step 6 as identified in #7805
2024-08-28 17:43:12 +00:00
Elliot DeNolf
631431e006 feat: @payloadcms/db-vercel-postgres adapter (#7806)
Dedicated adapter for Vercel Postgres

- Uses the `@vercel/postgres` package under the hood.
- No `pg` dependency, speeds up invocation
- Includes refactoring all base postgres functionality into a
`BasePostgresAdapter` type, which will ease implementation of [other
adapters supported by
drizzle-orm](https://orm.drizzle.team/docs/get-started-postgresql)

## Usage

```ts
import { buildConfig } from 'payload'
import { vercelPostgresAdapter } from '@payloadcms/db-vercel-postgres'

export default buildConfig({
  db: vercelPostgresAdapter({
    pool: {
      connectionString: process.env.DATABASE_URI,
    },
  }),
  // ...rest of config
})
```

### Automatic Connection String Detection

Have Vercel automatically detect from environment variable (typically
`process.env.POSTGRES_URL`)

```ts
export default buildConfig({
  db: postgresAdapter(),
  // ...rest of config
})
```
2024-08-21 22:54:47 -04:00
Paul
d2571e10d6 feat: upload hasmany (#7796)
Supports `hasMany` upload fields, similar to how `hasMany` works in
other fields, i.e.:

```ts
{
  type: 'upload',
  relationTo: 'media',
  hasMany: true
}
```

---------

Co-authored-by: Jacob Fletcher <jacobsfletch@gmail.com>
Co-authored-by: James <james@trbl.design>
2024-08-21 20:44:04 -04:00
Elliot DeNolf
5fc9f76406 feat: filename compound index (#7651)
Allow a compound index to be used for upload collections via a
`filenameCompoundIndex` field. Previously, `filename` was always treated
as unique.

Usage:

```ts
{
  slug: 'upload-field',
   upload: {
     // Slugs to include in compound index
     filenameCompoundIndex: ['filename', 'alt'],
  },
}
```
2024-08-13 13:55:10 -04:00
Elliot DeNolf
fa3d250053 feat: indent migration sql (#7475)
Properly indent migration sql
2024-08-09 22:41:28 -04:00
James Mikrut
1cb1e5e8b3 feat(db-*): allows for running migrations in production automatically (#7563)
## Description

Introduces a pattern for running migrations upon Payload init in
production.
2024-08-07 13:57:12 -04:00
Patrik
3f5403a52a fix(ui): prevents hasMany text going outside of input boundaries (#7455)
## Description

V2 PR [here](https://github.com/payloadcms/payload/pull/7454)

`Before`:
![Screenshot 2024-07-31 at 12 40
50 PM](https://github.com/user-attachments/assets/ce61f4fc-e676-4273-aa4c-72610cb459b3)

`After`:
![Screenshot 2024-07-31 at 12 40
23 PM](https://github.com/user-attachments/assets/d92631eb-28fb-46ca-bc23-46c7916bba34)

- [x] I have read and understand the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository.

## Type of change

- [x] Bug fix (non-breaking change which fixes an issue)

## Checklist:

- [x] Existing test suite passes locally with my changes
2024-08-05 17:10:35 -04:00