Commit Graph

66 Commits

Author SHA1 Message Date
Gregor Billing
c7b9f0f563 fix(db-mongodb): disable join aggregations in DocumentDB compatibility mode (#13270)
### What?

The PR #12763 added significant improvements for third-party databases
that are compatible with the MongoDB API. While the original PR was
focused on Firestore, other databases like DocumentDB also benefit from
these compatibility features.

In particular, the aggregate JOIN strategy does not work on AWS
DocumentDB and thus needs to be disabled. The current PR aims to provide
this as a sensible default in the `compatibilityOptions` that are
provided by Payload out-of-the-box.

As a bonus, it also fixes a small typo from `compat(a)bility` to
`compat(i)bility`.

### Why?

Because our Payload instance, which is backed by AWS DocumentDB, crashes
upon trying to access any `join` field.

### How?

By adding the existing `useJoinAggregations` with value `false` to the
compatiblity layer. Individual developers can still choose to override
it in their own local config as needed.
2025-08-20 16:31:19 +00:00
chenxi-debugger
379ef87d84 docs(db-mongodb): note on indexing localized fields & per-locale growth (#13469)
Closes #13464

Adds a note to the Indexes docs for localized fields:
- Indexing a `localized: true` field creates one index per locale path
(e.g. `slug.en`, `slug.da-dk`), which can grow index count on MongoDB.
- Recommends defining explicit indexes via collection-level `indexes`
for only the locale paths you actually query.
- Includes a concrete example (index `slug.en` only).

Docs-only change.

---------

Co-authored-by: German Jablonski <43938777+GermanJablo@users.noreply.github.com>
2025-08-19 12:45:38 +00:00
Elliott W
41cff6d436 fix(db-mongodb): improve compatibility with Firestore database (#12763)
### What?

Adds four more arguments to the `mongooseAdapter`:

```typescript
  useJoinAggregations?: boolean  /* The big one */
  useAlternativeDropDatabase?: boolean
  useBigIntForNumberIDs?: boolean
  usePipelineInSortLookup?: boolean
```

Also export a new `compatabilityOptions` object from
`@payloadcms/db-mongodb` where each key is a mongo-compatible database
and the value is the recommended `mongooseAdapter` settings for
compatability.

### Why?

When using firestore and visiting
`/admin/collections/media/payload-folders`, we get:

```
MongoServerError: invalid field(s) in lookup: [let, pipeline], only lookup(from, localField, foreignField, as) is supported
```

Firestore doesn't support the full MongoDB aggregation API used by
Payload which gets used when building aggregations for populating join
fields.

There are several other compatability issues with Firestore:
- The invalid `pipeline` property is used in the `$lookup` aggregation
in `buildSortParams`
- Firestore only supports number IDs of type `Long`, but Mongoose
converts custom ID fields of type number to `Double`
- Firestore does not support the `dropDatabase` command
- Firestore does not support the `createIndex` command (not addressed in
this PR)

### How?

 ```typescript
useJoinAggregations?: boolean  /* The big one */
```
When this is `false` we skip the `buildJoinAggregation()` pipeline and resolve the join fields through multiple queries. This can potentially be used with AWS DocumentDB and Azure Cosmos DB to support join fields, but I have not tested with either of these databases.

 ```typescript
useAlternativeDropDatabase?: boolean
```
When `true`, monkey-patch (replace) the `dropDatabase` function so that
it calls `collection.deleteMany({})` on every collection instead of
sending a single `dropDatabase` command to the database

 ```typescript
useBigIntForNumberIDs?: boolean
```
When `true`, use `mongoose.Schema.Types.BigInt` for custom ID fields of type `number` which converts to a firestore `Long` behind the scenes

```typescript
  usePipelineInSortLookup?: boolean
```
When `false`, modify the sortAggregation pipeline in `buildSortParams()` so that we don't use the `pipeline` property in the `$lookup` aggregation. Results in slightly worse performance when sorting by relationship properties.

### Limitations

This PR does not add support for transactions or creating indexes in firestore.

### Fixes

Fixed a bug (and added a test) where you weren't able to sort by multiple properties on a relationship field.

### Future work

1. Firestore supports simple `$lookup` aggregations but other databases might not. Could add a `useSortAggregations` property which can be used to disable aggregations in sorting.

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Sasha <64744993+r1tsuu@users.noreply.github.com>
2025-07-16 15:17:43 -04:00
Jacob Fletcher
277448d9c0 docs: performance (#13068)
Payload is designed with performance in mind, but its customizability
means that there are many ways to configure your app that can impact
performance.

While Payload provides several features and best practices to help you
optimize your app's specific performance needs, these are not currently
well surfaced and can be obscure.

Now:

- A high-level performance doc now exists at `/docs/performance`
- There's a new section on performance within the `/docs/queries` doc
- There's a new section on performance within the `/docs/hooks` doc
- There's a new section on performance within the
`/docs/custom-components` doc

This PR also:

- Restructures and elaborates on the `/docs/queries/pagination` docs
- Adds a new `/docs/database/indexing` doc
- More

---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
  - https://app.asana.com/0/0/1210743577153856
2025-07-14 23:55:16 -04:00
Paul
c902f14cb3 fix(db-mongodb): add ability to disable fallback sort and no longer adds a fallback for unique fields (#12961)
You can now disable fallback sort in the mongodb adapter by passing
`disableFallbackSort: true` in the options.

We also no longer add fallback sort to sorts on unique fields by default
now.

This came out of a discussion in this issue
https://github.com/payloadcms/payload/issues/12690
and the linked PR https://github.com/payloadcms/payload/pull/12888

Closes https://github.com/payloadcms/payload/issues/12690
2025-06-27 13:45:30 +00:00
Sasha
215f49efa5 feat(db-postgres): allow to store blocks in a JSON column (#12750)
Continuation of https://github.com/payloadcms/payload/pull/6245.
This PR allows you to pass `blocksAsJSON: true` to SQL adapters and the
adapter instead of aligning with the SQL preferred relation approach for
blocks will just use a simple JSON column, which can improve performance
with a large amount of blocks.

To try these changes you can install `3.43.0-internal.c5bbc84`.
2025-06-16 16:03:35 -04:00
Sasha
0a357372e9 feat(db-postgres): support read replicas (#12728)
Adds support for read replicas
https://orm.drizzle.team/docs/read-replicas that can be used to offload
read-heavy traffic.

To use (both `db-postgres` and `db-vercel-postgres` are supported):
```ts
import { postgresAdapter } from '@payloadcms/db-postgres'

database: postgresAdapter({
  pool: {
    connectionString: process.env.POSTGRES_URL,
  },
  readReplicas: [process.env.POSTGRES_REPLICA_URL],
})
```

---------

Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
2025-06-09 19:09:52 +00:00
Sean Zubrickas
7fa879c3a0 docs: typos and links (#12484)
- update SantizeCollection to SanitizedCollection in TypeScript section

- fix new issue link in Multi-Tenant plugin section

- correct You can disabled to disable in Core Features

- fix duplicate section links for MongoDB and Postgres in Migrations
section
2025-05-28 06:47:51 -07:00
Anyu Jiang
e2f7889d72 docs: fix typos, duplicated words, wrong property names etc. (#12480)
### What?
fix typos in doc
### Why?
because they are typos
### How?
checked manually with the help of AI
2025-05-25 10:58:26 -07:00
Jessica Rynkar
b7ae4ee60a docs: adds warning about handling different environments with migrations (#12249)
### What?
Migrating configs that include environment specific options can cause
issues and confusion for users.

### How?
Adds new section to `database/migrations` docs to highlight potential
issues with environment-specific settings when generating and running
migrations and includes some recommendations for addressing these
issues.

Closes #12241
2025-04-29 13:23:49 +01:00
Jacob Fletcher
5f7202bbb8 docs: payload proper nouns (#11792)
Uses proper nouns in the docs where necessary for "Payload" and "Local
API".
2025-03-21 09:04:11 -04:00
Sasha
f442d22237 feat(db-*): allow to thread id to create operation data without custom IDs (#11709)
Fixes https://github.com/payloadcms/payload/issues/6884

Adds a new flag `acceptIDOnCreate` that allows you to thread your own
`id` to `payload.create` `data`, for example:

```ts
// doc created with id 1
const doc = await payload.create({ collection: 'posts', data: {id: 1, title: "my title"}})
```

```ts
import { Types } from 'mongoose'
const id = new Types.ObjectId().toHexString()
const doc = await payload.create({ collection: 'posts', data: {id, title: "my title"}})
```
2025-03-17 23:48:35 -04:00
Alessio Gravili
9f9db3ff81 chore: bump prettier, re-enable prettier for docs (#11695)
## Introducing Prettier for docs

Prettier [was originally disabled for our docs as it didn't support MDX
2.0](1fa636417f),
outputting invalid MDX syntax.

This has since been fixed - prettier now supports MDX 2.0.

## Reducing print width

This also reduces the print width for the docs folder from 100 to 70.
Our docs code field are very narrow - this should help make code more
readable.

**Before**
![CleanShot 2025-03-13 at 19 58
11@2x](https://github.com/user-attachments/assets/0ae9e27b-cddf-44e5-a978-c8e24e99a314)

**After**

![CleanShot 2025-03-13 at 19 59
19@2x](https://github.com/user-attachments/assets/0e424f99-002c-4adc-9b37-edaeef239b0d)



**Before**
![CleanShot 2025-03-13 at 20 00
05@2x](https://github.com/user-attachments/assets/614e51b3-aa0d-45e7-98f4-fcdb1a778bcf)

**After**

![CleanShot 2025-03-13 at 20 00
16@2x](https://github.com/user-attachments/assets/be46988a-2cba-43fc-a8cd-fd3c781da930)
2025-03-14 17:13:08 +00:00
Sasha
3de1636e92 docs: document payload migrate:create flags (#11592)
Related discussion
https://github.com/payloadcms/payload/discussions/10978
2025-03-07 19:25:39 +02:00
Sasha
2ad035fb7b feat(db-mongodb): strip keys from the data that don't exist in the schema from read results (#11558)
This change makes so that data that exists in MongoDB but isn't defined
in the Payload config won't be included to `payload.find` /
`payload.db.find` calls. Now we strip all the additional keys.

Consider you have a field named `secretField` that's also `hidden: true`
(or `read: () => false`) that contains some sensitive data. Then you
removed this field from the database and as for now with the MongoDB
adapter this field will be included to the Local API / REST API results
without any consideration, as Payload doesn't know about it anymore.

This also fixes https://github.com/payloadcms/payload/issues/11542 if
you removed / renamed a relationship field from the schema, Payload
won't sanitize ObjectIDs back to strings anymore.

Ideally you should create a migration script that completely removes the
deleted field from the database with `$unset`, but people rarely do
this.

If you still need to keep those fields to the result, this PR allows you
to do this with the new `allowAdditionalKeys: true` flag.
2025-03-06 14:31:38 +00:00
Sasha
25e8799a09 docs: add few notes about DocumentDB and Azure Cosmos DB (#11336)
Adds few notes about limitations when using Azure Cosmos DB and
DocumentDB

Discussion: https://github.com/payloadcms/payload/discussions/11333
2025-02-28 21:14:51 +00:00
Said Akhrarov
8952662db9 docs: fix links and formatting (#10835)
### What?
This PR fixes many links in the docs as well as a few formatting and
grammar issues.

### Why?
To properly link users to the correct destination in the docs and
present well-formatted docs.

### How?
Changes to a few files in `docs/`
2025-01-27 22:50:54 -07:00
Ben Löffel
5043a8a43f docs: improves grammar in vercel postgres usage note (#10365)
Refined the grammar and structure of the usage note for
`vercelPostgresAdapter`. Replaced the ambiguous phrase "If when using"
with "If you are using" for better readability and clarity.
2025-01-13 13:36:39 -05:00
Said Akhrarov
5e4e27460e docs: fix links and formatting (#10346)
### What?
This PR fixes numerous links across the docs, both internal docs links
and external links. This PR also fixes some minor formatting issues in
some places, as well as optically aligns the markdown tables in tables
that had broken links.

### Why?
To properly link readers to the correct location in the docs, and for
better formatting and easier consumption.

### How?
Changes to many `.mdx` files in the `docs` folder.

Notes:
- There are duplicative section id's in `docs/authentication/email.mdx`,
I've fixed one such link, but have left it as is for now.
2025-01-03 21:41:33 -07:00
Alessio Gravili
5b4730d13c docs: fix incorrect escaping of * (#10274) 2024-12-30 17:02:13 -07:00
Sasha
63bca12f87 docs: fix docs for generate db schema (#10251)
<!--

Thank you for the PR! Please go through the checklist below and make
sure you've completed all the steps.

Please review the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository if you haven't already.

The following items will ensure that your PR is handled as smoothly as
possible:

- PR Title must follow conventional commits format. For example, `feat:
my new feature`, `fix(plugin-seo): my fix`.
- Minimal description explained as if explained to someone not
immediately familiar with the code.
- Provide before/after screenshots or code diffs if applicable.
- Link any related issues/discussions from GitHub or Discord.
- Add review comments if necessary to explain to the reviewer the logic
behind a change

### What?

### Why?

### How?

Fixes #

-->
2024-12-30 16:03:46 +00:00
Alessio Gravili
bb0c8a8c62 docs: fix incorrect mdx (#10201) 2024-12-27 20:18:31 +00:00
Sasha
4e953530df feat(db-sqlite): add autoIncrement option (#9427)
### What?
Exposes ability to enable
[AUTOINCREMENT](https://www.sqlite.org/autoinc.html) for Primary Keys
which ensures that the same ID cannot be reused from previously deleted
rows.

```ts
sqliteAdapter({
  autoIncrement: true
})
```

### Why?
This may be essential for some systems. Enabled `autoIncrement: true`
also for the SQLite Adapter in our tests, which can be useful when
testing whether the doc was deleted or not when you also have other
create operations.

### How?
Uses Drizzle's `autoIncrement` option.

WARNING:
This cannot be enabled in an existing project without a custom
migration, as it completely changes how primary keys are stored in the
database.
2024-12-20 20:13:28 +00:00
Sasha
23f1ed4a48 feat(db-postgres, db-sqlite): drizzle schema generation (#9953)
This PR allows to have full type safety on `payload.drizzle` with a
single command
```sh
pnpm payload generate:db-schema
```
Which generates TypeScript code with Drizzle declarations based on the
current database schema.

Example of generated file with the website template: 
https://gist.github.com/r1tsuu/b8687f211b51d9a3a7e78ba41e8fbf03

Video that shows the power:


https://github.com/user-attachments/assets/3ced958b-ec1d-49f5-9f51-d859d5fae236


We also now proxy drizzle package the same way we do for Lexical so you
don't have to install it (and you shouldn't because you may have version
mismatch).
Instead, you can import from Drizzle like this:
```ts
import {
  pgTable,
  index,
  foreignKey,
  integer,
  text,
  varchar,
  jsonb,
  boolean,
  numeric,
  serial,
  timestamp,
  uniqueIndex,
  pgEnum,
} from '@payloadcms/db-postgres/drizzle/pg-core'
import { sql } from '@payloadcms/db-postgres/drizzle'
import { relations } from '@payloadcms/db-postgres/drizzle/relations'
```


Fixes https://github.com/payloadcms/payload/discussions/4318

In the future we can also support types generation for mongoose / raw
mongodb results.
2024-12-19 11:08:17 -05:00
Sasha
03ff77544e feat(db-sqlite): add idType: 'uuid' support (#10016)
Adds `idType: 'uuid'` to the SQLite adapter support:
```ts
sqliteAdapter({
  idType: 'uuid',
})
```

Achieved through Drizzle's `$defaultFn()`
https://orm.drizzle.team/docs/latest-releases/drizzle-orm-v0283#-added-defaultfn--default-methods-to-column-builders
as SQLite doesn't have native UUID support. Added `sqlite-uuid` to CI.
2024-12-18 22:44:04 -05:00
Javier
198763a24e feat(db-mongodb): allow to customize mongoose schema options with collectionsSchemaOptions (#9885)
Adds the ability to pass additional schema options for collections with:
```ts
mongooseAdapter({
  collectionsSchemaOptions: {
    posts: {
      strict: false,
    },
  },
})
``` 

This changes relates to these:

-   https://github.com/payloadcms/payload/issues/4533
-   https://github.com/payloadcms/payload/discussions/4534

It is a proposal to set custom schema options for mongoose driver. 

I understand this got introduced into `main` v2 after `beta` branch was
created so this feature got lost.

- [x] I have read and understand the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository.

<!--

Thank you for the PR! Please go through the checklist below and make
sure you've completed all the steps.

Please review the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository if you haven't already.

The following items will ensure that your PR is handled as smoothly as
possible:

- PR Title must follow conventional commits format. For example, `feat:
my new feature`, `fix(plugin-seo): my fix`.
- Minimal description explained as if explained to someone not
immediately familiar with the code.
- Provide before/after screenshots or code diffs if applicable.
- Link any related issues/discussions from GitHub or Discord.
- Add review comments if necessary to explain to the reviewer the logic
behind a change

### What?

### Why?

### How?

Fixes #

-->

---------

Co-authored-by: Sasha <64744993+r1tsuu@users.noreply.github.com>
2024-12-18 15:46:39 +00:00
Sasha
41167bfbeb feat(db-vercel-postgres): allow to use a local database using pg instead of @vercel/postgres (#9771)
### What?
This PR allows you to use a local database when using
`vercelPostgresAdapter`. This adapter doesn't work with them because it
requires an SSL connection and Neon's WS proxy. Instead we fallback to
using pool from `pg` if `hostname` is either `127.0.0.1` or `localhost`.

If you still want to use `@vercel/postgres` even locally you can pass
`disableUsePgForLocalDatabase: true` here and you'd have to spin up the
DB with a special Neon's Docker Compose setup -
https://vercel.com/docs/storage/vercel-postgres/local-development#option-2:-local-postgres-instance-with-docker

### Why?
Forcing people to use a cloud database locally isn't great. Not only
they are slow but also paid.

---------

Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
2024-12-16 16:55:08 -05:00
Sasha
b73fc586b8 feat: expose session, db in migrations to use the active transaction with the database directly (#9849)
This PR adds a feature which fixes another issue with migrations in
Postgres and does few refactors that significantly reduce code
duplication.

Previously, if you needed to use the underlying database directly in
migrations with the active transaction (for example to execute raw SQL),
created from `payload create:migration`, as `req` doesn't work there you
had to do something like this:
```ts
// Postgres
export async function up({ payload, req }: MigrateUpArgs): Promise<void> {
  const db = payload.db.sessions?.[await req.transactionID!].db ?? payload.db
  const { rows: posts } = await db.execute(sql`SELECT * from posts`)
}

// MongoDB
export async function up({ payload, req }: MigrateUpArgs): Promise<void> {
  const session = payload.db.sessions?.[await req.transactionID!]
  const posts = await payload.db.collections.posts.collection.find({ session }).toArray()
}
```

Which was:
1. Awkward to write
2. Not documented anywhere

Now, we expose `session` and `db` to `up` and `down` functions for you:

#### MongoDB:
```ts
import { type MigrateUpArgs } from '@payloadcms/db-mongodb'

export async function up({ session, payload, req }: MigrateUpArgs): Promise<void> {
  const posts = await payload.db.collections.posts.collection.find({ session }).toArray()
}
```
#### Postgres:
```ts
import { type MigrateUpArgs, sql } from '@payloadcms/db-postgres'

export async function up({ db, payload, req }: MigrateUpArgs): Promise<void> {
  const { rows: posts } = await db.execute(sql`SELECT * from posts`)
}
```

#### SQLite:
```ts
import { type MigrateUpArgs, sql } from '@payloadcms/db-sqlite'

export async function up({ db, payload, req }: MigrateUpArgs): Promise<void> {
  const { rows: posts } = await db.run(sql`SELECT * from posts`)
}
```
This actually was a thing with Postgres migrations, we already were
passing `db`, but:
1. Only for `up` and when running `payload migrate`, not for example
with `payload migrate:fresh`
2. Not documented neither in TypeScript or docs.

By ensuring we use `db`, this also fixes an issue that affects all
Postgres/SQLite migrations:

Currently, if we run `payload migration:create` with the postgres
adapter we get a file like this:

```ts
import { MigrateUpArgs, MigrateDownArgs, sql } from '@payloadcms/db-postgres'

export async function up({ payload, req }: MigrateUpArgs): Promise<void> {
  await payload.db.drizzle.execute(sql`
   CREATE TABLE IF NOT EXISTS "users" (
  	"id" serial PRIMARY KEY NOT NULL,
  );
```
Looks good?
Not exactly!
`payload.db.drizzle.execute()` doesn't really use the current
transaction which can lead to some problems.
Instead, it should use the `db` from `payload.db.sessions?.[await
req.transactionID!].db` because that's where we store our Drizzle
instance with the transaction.

But now, if we generate `payload migrate:create` we get:
```ts
import { MigrateUpArgs, MigrateDownArgs, sql } from '@payloadcms/db-postgres'

export async function up({ db, payload, req }: MigrateUpArgs): Promise<void> {
  await db.execute(sql`
   CREATE TABLE IF NOT EXISTS "users" (
  	"id" serial PRIMARY KEY NOT NULL,
  );
```

Which is what we want, as the `db` is passed correctly here:

76428373e4/packages/drizzle/src/migrate.ts (L88-L90)

```ts
export async function up({ db, payload, req }: MigrateUpArgs): Promise<void> {
  const dbWithTransaction = payload.db.sessions?.[await req.transactionID!].db
  payload.logger.info({ one: db === dbWithTransaction })
  payload.logger.info({ two: db === payload.db.drizzle })
```
<img width="336" alt="image"
src="https://github.com/user-attachments/assets/f9fab5a9-44c2-44a9-95dd-8e5cf267f027">

Additionally, this PR refactors:
* `createMigration` with Drizzle - now we have sharable
`buildCreateMigration` in `@payloadcms/drizzle` to reduce copy-pasting
of the same logic.
* the `v2-v3` relationships migration for Postgres is now shared between
`db-postgres` and `db-vercel-postgres`, again to reduce copy-paste.
2024-12-11 12:23:12 -05:00
Alessio Gravili
36c2714251 docs: fix typo (#9845) 2024-12-10 01:24:28 +00:00
Arne Wiese
b5ddc328ef docs: fix typo in postgres docs (#9585)
This PR fixes a typo in the postgres docs

```
- disale
+ disable
```
2024-11-28 21:13:28 +00:00
Sasha
0a15388edb feat(db-postgres): add point field support (#9078)
### What?
Adds full support for the point field to Postgres and Vercel Postgres
adapters through the Postgis extension. Fully the same API as with
MongoDB, including support for `near`, `within` and `intersects`
operators.

Additionally, exposes to adapter args:
*
`tablesFilter`https://orm.drizzle.team/docs/drizzle-kit-push#including-tables-schemas-and-extensions.
* `extensions` list of extensions to create, for example `['vector',
'pg_search']`, `postgis` is created automatically if there's any point
field

### Why?
It's essential to support that field type, especially if the postgres
adapter should be out of beta on 3.0 stable.

### How?
* Bumps `drizzle-orm` to `0.36.1` and `drizzle-kit` to `0.28.0` as we
need this change https://github.com/drizzle-team/drizzle-orm/pull/3141
* Uses its functions to achieve querying functionality, for example the
`near` operator works through `ST_DWithin` or `intersects` through
`ST_Intersects`.
* Removes MongoDB condition from all point field tests, but keeps for
SQLite

Resolves these discussions:
https://github.com/payloadcms/payload/discussions/8996
https://github.com/payloadcms/payload/discussions/8644
2024-11-11 09:31:47 -05:00
Dan Ribbens
e2607d4faa chore: database transactions docs formatting (#9068) 2024-11-07 12:48:17 -05:00
James Mikrut
8970c6b3a6 feat: adds jobs queue (#8228)
Adds a jobs queue to Payload.

- [x] Docs, w/ examples for Vercel Cron, additional services
- [x] Type the `job` using GeneratedTypes in `JobRunnerArgs`
(@AlessioGr)
- [x] Write the `runJobs` function 
- [x] Allow for some type of `payload.runTask` 
- [x] Open up a new bin script for running jobs
- [x] Determine strategy for runner endpoint to either await jobs
successfully or return early and stay open until job work completes
(serverless ramifications here)
- [x] Allow for job runner to accept how many jobs to run in one
invocation
- [x] Make a Payload local API method for creating a new job easily
(payload.createJob) or similar which is strongly typed (@AlessioGr)
- [x] Make `payload.runJobs` or similar  (@AlessioGr)
- [x] Write tests for retrying up to max retries for a given step
- [x] Write tests for dynamic import of a runner

The shape of the config should permit the definition of steps separate
from the job workflows themselves.

```js
const config = {
  // Not sure if we need this property anymore
  queues: {
  },
  // A job is an instance of a workflow, stored in DB
  // and triggered by something at some point
  jobs: {
    // Be able to override the jobs collection
    collectionOverrides: () => {},

    // Workflows are groups of tasks that handle
    // the flow from task to task.
    // When defined on the config, they are considered as predefined workflows
    // BUT - in the future, we'll allow for UI-based workflow definition as well.
    workflows: [
      {
        slug: 'job-name',
        // Temporary name for this
        // should be able to pass function 
        // or path to it for Node to dynamically import
        controlFlowInJS: '/my-runner.js',

        // Temporary name as well
        // should be able to eventually define workflows
        // in UI (meaning they need to be serialized in JSON)
        // Should not be able to define both control flows
        controlFlowInJSON: [
          {
            task: 'myTask',
            next: {
              // etc
            }
          }
        ],

        // Workflows take input
        // which are a group of fields
        input: [
          {
            name: 'post',
            type: 'relationship',
            relationTo: 'posts',
            maxDepth: 0,
            required: true,
          },
          {
            name: 'message',
            type: 'text',
            required: true,
          },
        ],
      },
    ],

    // Tasks are defined separately as isolated functions
    // that can be retried on fail
    tasks: [
      {
        slug: 'myTask',
        retries: 2,
        // Each task takes input
        // Used to auto-type the task func args
        input: [
          {
            name: 'post',
            type: 'relationship',
            relationTo: 'posts',
            maxDepth: 0,
            required: true,
          },
          {
            name: 'message',
            type: 'text',
            required: true,
          },
        ],
        // Each task takes output
        // Used to auto-type the function signature
        output: [
          {
            name: 'success',
            type: 'checkbox',
          }
        ],
        onSuccess: () => {},
        onFail: () => {},
        run: myRunner,
      },
    ]
  }
}
```

### `payload.createJob`

This function should allow for the creation of jobs based on either a
workflow (group of tasks) or an individual task.

To create a job using a workflow:

```js
const job = await payload.createJob({
  // Accept the `name` of a workflow so we can match to either a 
  // code-based workflow OR a workflow defined in the DB
  // Should auto-type the input
  workflowName: 'myWorkflow',
  input: {
    // typed to the args of the workflow by name
  }
})
```

To create a job using a task:

```js
const job = await payload.createJob({
  // Accept the `name` of a task
  task: 'myTask',
  input: {
    // typed to the args of the task by name
  }
})
```

---------

Co-authored-by: Alessio Gravili <alessio@gravili.de>
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
2024-10-30 17:56:50 +00:00
Said Akhrarov
c91f21bb78 docs: fix incorrect link for outside-nextjs in local-api importing it section (#8764)
Currently in the `beta` docs at the bottom of [Local API Overview Import
It
section](https://payloadcms.com/docs/beta/local-api/overview#importing-it)
there is a link for _Outside Nextjs_ which incorrectly sends you to
`/docs/beta/beta/local-api/outside-nextjs` instead of
`docs/beta/local-api/outside-nextjs`.

Interestingly enough, a `Not Found` component/message is not rendered
and instead you see a blank screen.

---------

Co-authored-by: Sasha <64744993+r1tsuu@users.noreply.github.com>
2024-10-17 13:13:37 -06:00
Dan Ribbens
d781624a86 feat: add disableTransaction to local api (#8697) 2024-10-14 14:39:20 -04:00
Sasha
256949e331 feat(db-postgres, db-vercel-postgres): createDatabase, auto database create if it does not exist (#8655)
Adds `createDatabase` method to Postgres adapters which can be used
either independently like this:
```ts
payload.db.createDatabase({
  name: "some-database",
  schemaName: "custom-schema"
})
```
Or
```ts
payload.db.createDatabase()
```
Which creates a database from the current configuration, this is used in
`connect` if `autoDatabaseCreate` is set to `true` (default).
You can disable this behaviour with:
```ts
postgresAdapter({ autoDatabaseCreate: false })
```

Example:
<img width="470" alt="image"
src="https://github.com/user-attachments/assets/8d08c79d-9672-454c-af0f-eb802f9dcd99">
2024-10-11 14:17:32 +00:00
Sasha
8acbda078e feat(drizzle): customize schema with before / after init hooks (#8196)
Adds abillity to customize the generated Drizzle schema with
`beforeSchemaInit` and `afterSchemaInit`. Could be useful if you want to
preserve the existing database schema / override the generated one with
features that aren't supported from the Payload config.

## Docs:

### beforeSchemaInit

Runs before the schema is built. You can use this hook to extend your
database structure with tables that won't be managed by Payload.

```ts
import { postgresAdapter } from '@payloadcms/db-postgres'
import { integer, pgTable, serial } from 'drizzle-orm/pg-core'

postgresAdapter({
  beforeSchemaInit: [
    ({ schema, adapter }) => {
      return {
        ...schema,
        tables: {
          ...schema.tables,
          addedTable: pgTable('added_table', {
            id: serial('id').notNull(),
          }),
        },
      }
    },
  ],
})
```

One use case is preserving your existing database structure when
migrating to Payload. By default, Payload drops the current database
schema, which may not be desirable in this scenario.
To quickly generate the Drizzle schema from your database you can use
[Drizzle
Introspection](https://orm.drizzle.team/kit-docs/commands#introspect--pull)
You should get the `schema.ts` file which may look like this:

```ts
import { pgTable, uniqueIndex, serial, varchar, text } from 'drizzle-orm/pg-core'

export const users = pgTable('users', {
  id: serial('id').primaryKey(),
  fullName: text('full_name'),
  phone: varchar('phone', { length: 256 }),
})

export const countries = pgTable(
  'countries',
  {
    id: serial('id').primaryKey(),
    name: varchar('name', { length: 256 }),
  },
  (countries) => {
    return {
      nameIndex: uniqueIndex('name_idx').on(countries.name),
    }
  },
)

```

You can import them into your config and append to the schema with the
`beforeSchemaInit` hook like this:

```ts
import { postgresAdapter } from '@payloadcms/db-postgres'
import { users, countries } from '../drizzle/schema'

postgresAdapter({
  beforeSchemaInit: [
    ({ schema, adapter }) => {
      return {
        ...schema,
        tables: {
          ...schema.tables,
          users,
          countries
        },
      }
    },
  ],
})
```

Make sure Payload doesn't overlap table names with its collections. For
example, if you already have a collection with slug "users", you should
either change the slug or `dbName` to change the table name for this
collection.


### afterSchemaInit

Runs after the Drizzle schema is built. You can use this hook to modify
the schema with features that aren't supported by Payload, or if you
want to add a column that you don't want to be in the Payload config.
To extend a table, Payload exposes `extendTable` utillity to the args.
You can refer to the [Drizzle
documentation](https://orm.drizzle.team/docs/sql-schema-declaration).
The following example adds the `extra_integer_column` column and a
composite index on `country` and `city` columns.

```ts
import { postgresAdapter } from '@payloadcms/db-postgres'
import { index, integer } from 'drizzle-orm/pg-core'
import { buildConfig } from 'payload'

export default buildConfig({
  collections: [
    {
      slug: 'places',
      fields: [
        {
          name: 'country',
          type: 'text',
        },
        {
          name: 'city',
          type: 'text',
        },
      ],
    },
  ],
  db: postgresAdapter({
    afterSchemaInit: [
      ({ schema, extendTable, adapter }) => {
        extendTable({
          table: schema.tables.places,
          columns: {
            extraIntegerColumn: integer('extra_integer_column'),
          },
          extraConfig: (table) => ({
            country_city_composite_index: index('country_city_composite_index').on(
              table.country,
              table.city,
            ),
          }),
        })

        return schema
      },
    ],
  }),
})

```



<!--

For external contributors, please include:

- A summary of the pull request and any related issues it fixes.
- Reasoning for the changes made or any additional context that may be
useful.

Ensure you have read and understand the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository.

 -->
2024-09-25 15:14:03 -04:00
Dan Ribbens
b10f61cb25 fix(db-mongodb): add req to migration templates for transactions (#8407)
- Add `req` to MigrateUpArgs and MigrateDownArgs for mongodb
- Improve docs for transactions and migrations
2024-09-25 12:58:48 -04:00
Elliot DeNolf
8202c3dee8 feat: move logger configuration to Payload config (#7973)
Removes `loggerOptions` and `loggerDestination` from `initOptions`
(these were not able to be used anyway).

Creates new `logger` property on the Payload config.

```ts
// Logger options only
logger: {
   level: 'info',
}

// Logger options with destination stream
logger: {
   options: {
   level: 'info',
   },
   destination: process.stdout
},

// Logger instance
logger: pino({ name: 'my-logger' })
```
2024-08-30 09:37:51 -04:00
Elliot DeNolf
825d8b83d1 feat(templates): add vercel postgres template (#7841)
- docs: add db-vercel-postgres info
- feat: update template variations
2024-08-23 15:17:26 -04:00
James Mikrut
18d9314f22 docs: adds prod migrations (#7631)
## Description

Adds docs for executing migrations in production.
2024-08-12 11:45:39 -04:00
Patrik
8ba39aa5ca fix(db-mongodb): adds new optional collation feature flag behind mongodb collation option (#7361)
## Description

V2 PR [here](https://github.com/payloadcms/payload/pull/7359)

- [x] I have read and understand the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository.

## Type of change

- [x] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [x] This change requires a documentation update

## Checklist:

- [x] Existing test suite passes locally with my changes
- [x] I have made corresponding changes to the documentation
2024-07-25 12:38:18 -04:00
Dan Ribbens
09ad6e4280 feat(drizzle): abstract shared sql code to new package (#7320)
- Abstract shared sql code to a new drizzle package
- Adds sqlite package, not ready to publish until drizzle patches some
issues
- Add `transactionOptions` to allow customizing or disabling db
transactions
- Adds "experimental" label to the `schemaName` property until drizzle
patches an issue
2024-07-24 12:43:29 -04:00
Jacob Fletcher
e9adeecc7a docs: more misc improvements (#7115) 2024-07-11 14:55:13 -04:00
Jacob Fletcher
a1bb661a1a docs: misc improvements (#7107) 2024-07-11 09:54:21 -04:00
James Mikrut
f01fc584ed Update mongodb.mdx 2024-06-25 10:38:12 -04:00
James Mikrut
effba3e45b Chore/overview docs (#6906)
## Description

More progress to docs.
2024-06-24 13:17:50 -04:00
Jacob Fletcher
a20cf70105 docs: removes express 2024-05-13 10:29:59 -04:00
Alessio Gravili
19f8cbcf76 docs: new and improve lexical docs, hoist up all headings (#6639) 2024-06-05 17:08:15 -04:00
Tylan Davis
26ee91eb48 docs: adjust line breaks in code blocks (#6001) 2024-05-01 15:57:39 -04:00