### What?
This PR aims to add reindexing capabilities to `plugin-search` to allow
users to reindex entire searchable collections on demand.
### Why?
As it stands, end users must either perform document reindexing manually
one-by-one or via bulk operations. Both of these approaches are
undesirable because they result in new versions being published on
existing documents. Consider the case when `plugin-search` is only added
_after_ the project has started and documents have been added to
existing collections. It would be nice if users could simply click a
button, choose the searchable collections to reindex, and have the
custom endpoint handle the rest.
### How?
This PR adds on to the existing plugin configuration, creating a custom
endpoint and a custom `beforeListTable` component in the form of a popup
button. Upon clicking the button, a dropdown/popup is opened with
options to select which collection to reindex, as well as a useful `All
Collections` option to run reindexing on all configured search
collections. It also adds a `reindexBatchSize` option in the config to
allow users to specify in what quantity to batch documents to sync with
search.
Big shoutout to @paulpopus & @r1tsuu for the triple-A level support on
this one!
Fixes#8902
See it in action:
https://github.com/user-attachments/assets/ee8dd68c-ea89-49cd-adc3-151973eea28b
Notes:
- Traditionally these kinds of long-running tasks would be better suited
for a job. However, given how many users enjoy deploying to serverless
environments, it would be problematic to offer this feature exclusive to
jobs queues. I thought a significant amount about this and decided it
would be best to ship the feature as-is with the intention of creating
an opt-in method to use job queues in the future if/when this gets
merged.
- In my testing, the collection description somehow started to appear in
the document views after the on-demand RSC merge. I haven't reproduced
this, but this PR has an example of that problem. Super strange.
---------
Co-authored-by: Sasha <64744993+r1tsuu@users.noreply.github.com>
Co-authored-by: Paul Popus <paul@nouance.io>
Deprecates `react-animate-height` in favor of native CSS, specifically
the `interpolate-size: allow-keywords;` property which can be used to
animate to `height: auto`—the primary reason this package exists. This
is one less dependency in our `node_modules`. Tried to replicate the
current DOM structure, class names, and API of `react-animate-height`
for best compatibility.
Note that this CSS property is experimental BUT this PR includes a patch
for browsers without native support. Once full support is reached, the
patch can be safely removed.
TS 5.7 added support for ES2024. By keeping target: “esnext”, we would
have accidentally set our minimum supported ES version to ES2024.
This sets it to ES2022, which is the version supported by Node 18
Fixes https://github.com/payloadcms/payload/issues/9378
We’ve found out that @lexical/markdown imports cannot be reliably
dynamically imported by Node.js for an unknown reason. Frequently,
Node.js simply exits before the dynamic import is done.
We’re suspecting the reason for this to be its dependency on
@lexical/code that installs prism.
This will not only (hopefully) fix the import issue, but also reduce the
bundle size & compilation speed of richtext-lexical.
Supports bi-directional import/export between MDX <=> Lexical. JSX will
be mapped to lexical blocks back and forth.
This will allow editing our mdx docs in payload while keeping mdx as the
source of truth
---------
Co-authored-by: Germán Jabloñski <43938777+GermanJablo@users.noreply.github.com>
Deprecates `getPayloadHMR` and simplifies this pattern into a single
`import { getPayload } from 'payload'`.
We will still retain the exported `getPayloadHMR` but it now will throw
a deprecation warning with instructions for how to migrate.
### What?
Upgrades mongoose from 6 to latest `v8.8.1`
Fixes https://github.com/payloadcms/payload/issues/9171
### Why?
Compatibilty with Mongodb Atlas
### How?
- Updates deps
- Changed ObjectId from bson-objectid to use `new Type.ObjectId` from
mongoose for compatibility (only inside of db-mongodb)
- Internal type adjustments
https://github.com/payloadcms/payload/discussions/9088
BREAKING CHANGES:
All projects with existing data having versions enabled, or relationship or upload fields will want to create the predefined migration that converts all strings to ObjectIDs where needed. This can be created using `payload migrate:create --file @payloadcms/mongodb/relationships-v2-v3`.
For projects making use of the exposed Models from mongoose, review the
upgrade guides from [v6 to
v7](https://mongoosejs.com/docs/7.x/docs/migrating_to_7.html) and [v7 to
v8](https://mongoosejs.com/docs/migrating_to_8.html) and make
adjustments as needed.
---------
Co-authored-by: Sasha <64744993+r1tsuu@users.noreply.github.com>
Upgrade uploadthing to v7
The `options` that can be passed to the plugin now mirror the
`UTApiOptions` of v7.
The most notable change is to pass `token` with
`process.env.UPLOADTHING_TOKEN` instead of `apiKey` with
`process.env.UPLOADTHING_SECRET`.
```diff
options: {
- apiKey: process.env.UPLOADTHING_SECRET,
+ token: process.env.UPLOADTHING_TOKEN,
acl: 'public-read',
},
This fixes a peer dependency error in our monorepo, as
eslint-plugin-jsx-a11y finally supports eslint v9.
Additionally, this officially adds TypeScript 5.6 support for
typescript-eslint.
I'm needing https://github.com/facebook/lexical/pull/6693
I'm going to keep the dependency bump and feature updates in separate
PRs unless they're breaking changes.*
**BREAKING:**
This upgrades our lexical dependencies from 0.18.0 to 0.20.0. If you
have lexical dependencies installed in your project, you will have to
upgrade those.
Additionally, the lexical team may introduce breaking changes in this
upgrade. If you use lexical APIs directly, please consult their
changelog for more information:
https://github.com/facebook/lexical/releases
Currently, Payload renders all custom components on initial compile of
the admin panel. This is problematic for two key reasons:
1. Custom components do not receive contextual data, i.e. fields do not
receive their field data, edit views do not receive their document data,
etc.
2. Components are unnecessarily rendered before they are used
This was initially required to support React Server Components within
the Payload Admin Panel for two key reasons:
1. Fields can be dynamically rendered within arrays, blocks, etc.
2. Documents can be recursively rendered within a "drawer" UI, i.e.
relationship fields
3. Payload supports server/client component composition
In order to achieve this, components need to be rendered on the server
and passed as "slots" to the client. Currently, the pattern for this is
to render custom server components in the "client config". Then when a
view or field is needed to be rendered, we first check the client config
for a "pre-rendered" component, otherwise render our client-side
fallback component.
But for the reasons listed above, this pattern doesn't exactly make
custom server components very useful within the Payload Admin Panel,
which is where this PR comes in. Now, instead of pre-rendering all
components on initial compile, we're able to render custom components
_on demand_, only as they are needed.
To achieve this, we've established [this
pattern](https://github.com/payloadcms/payload/pull/8481) of React
Server Functions in the Payload Admin Panel. With Server Functions, we
can iterate the Payload Config and return JSX through React's
`text/x-component` content-type. This means we're able to pass
contextual props to custom components, such as data for fields and
views.
## Breaking Changes
1. Add the following to your root layout file, typically located at
`(app)/(payload)/layout.tsx`:
```diff
/* THIS FILE WAS GENERATED AUTOMATICALLY BY PAYLOAD. */
/* DO NOT MODIFY IT BECAUSE IT COULD BE REWRITTEN AT ANY TIME. */
+ import type { ServerFunctionClient } from 'payload'
import config from '@payload-config'
import { RootLayout } from '@payloadcms/next/layouts'
import { handleServerFunctions } from '@payloadcms/next/utilities'
import React from 'react'
import { importMap } from './admin/importMap.js'
import './custom.scss'
type Args = {
children: React.ReactNode
}
+ const serverFunctions: ServerFunctionClient = async function (args) {
+ 'use server'
+ return handleServerFunctions({
+ ...args,
+ config,
+ importMap,
+ })
+ }
const Layout = ({ children }: Args) => (
<RootLayout
config={config}
importMap={importMap}
+ serverFunctions={serverFunctions}
>
{children}
</RootLayout>
)
export default Layout
```
2. If you were previously posting to the `/api/form-state` endpoint, it
no longer exists. Instead, you'll need to invoke the `form-state` Server
Function, which can be done through the _new_ `getFormState` utility:
```diff
- import { getFormState } from '@payloadcms/ui'
- const { state } = await getFormState({
- apiRoute: '',
- body: {
- // ...
- },
- serverURL: ''
- })
+ const { getFormState } = useServerFunctions()
+
+ const { state } = await getFormState({
+ // ...
+ })
```
## Breaking Changes
```diff
- useFieldProps()
- useCellProps()
```
More details coming soon.
---------
Co-authored-by: Alessio Gravili <alessio@gravili.de>
Co-authored-by: Jarrod Flesch <jarrodmflesch@gmail.com>
Co-authored-by: James <james@trbl.design>
### What?
Adds full support for the point field to Postgres and Vercel Postgres
adapters through the Postgis extension. Fully the same API as with
MongoDB, including support for `near`, `within` and `intersects`
operators.
Additionally, exposes to adapter args:
*
`tablesFilter`https://orm.drizzle.team/docs/drizzle-kit-push#including-tables-schemas-and-extensions.
* `extensions` list of extensions to create, for example `['vector',
'pg_search']`, `postgis` is created automatically if there's any point
field
### Why?
It's essential to support that field type, especially if the postgres
adapter should be out of beta on 3.0 stable.
### How?
* Bumps `drizzle-orm` to `0.36.1` and `drizzle-kit` to `0.28.0` as we
need this change https://github.com/drizzle-team/drizzle-orm/pull/3141
* Uses its functions to achieve querying functionality, for example the
`near` operator works through `ST_DWithin` or `intersects` through
`ST_Intersects`.
* Removes MongoDB condition from all point field tests, but keeps for
SQLite
Resolves these discussions:
https://github.com/payloadcms/payload/discussions/8996https://github.com/payloadcms/payload/discussions/8644
Adds a jobs queue to Payload.
- [x] Docs, w/ examples for Vercel Cron, additional services
- [x] Type the `job` using GeneratedTypes in `JobRunnerArgs`
(@AlessioGr)
- [x] Write the `runJobs` function
- [x] Allow for some type of `payload.runTask`
- [x] Open up a new bin script for running jobs
- [x] Determine strategy for runner endpoint to either await jobs
successfully or return early and stay open until job work completes
(serverless ramifications here)
- [x] Allow for job runner to accept how many jobs to run in one
invocation
- [x] Make a Payload local API method for creating a new job easily
(payload.createJob) or similar which is strongly typed (@AlessioGr)
- [x] Make `payload.runJobs` or similar (@AlessioGr)
- [x] Write tests for retrying up to max retries for a given step
- [x] Write tests for dynamic import of a runner
The shape of the config should permit the definition of steps separate
from the job workflows themselves.
```js
const config = {
// Not sure if we need this property anymore
queues: {
},
// A job is an instance of a workflow, stored in DB
// and triggered by something at some point
jobs: {
// Be able to override the jobs collection
collectionOverrides: () => {},
// Workflows are groups of tasks that handle
// the flow from task to task.
// When defined on the config, they are considered as predefined workflows
// BUT - in the future, we'll allow for UI-based workflow definition as well.
workflows: [
{
slug: 'job-name',
// Temporary name for this
// should be able to pass function
// or path to it for Node to dynamically import
controlFlowInJS: '/my-runner.js',
// Temporary name as well
// should be able to eventually define workflows
// in UI (meaning they need to be serialized in JSON)
// Should not be able to define both control flows
controlFlowInJSON: [
{
task: 'myTask',
next: {
// etc
}
}
],
// Workflows take input
// which are a group of fields
input: [
{
name: 'post',
type: 'relationship',
relationTo: 'posts',
maxDepth: 0,
required: true,
},
{
name: 'message',
type: 'text',
required: true,
},
],
},
],
// Tasks are defined separately as isolated functions
// that can be retried on fail
tasks: [
{
slug: 'myTask',
retries: 2,
// Each task takes input
// Used to auto-type the task func args
input: [
{
name: 'post',
type: 'relationship',
relationTo: 'posts',
maxDepth: 0,
required: true,
},
{
name: 'message',
type: 'text',
required: true,
},
],
// Each task takes output
// Used to auto-type the function signature
output: [
{
name: 'success',
type: 'checkbox',
}
],
onSuccess: () => {},
onFail: () => {},
run: myRunner,
},
]
}
}
```
### `payload.createJob`
This function should allow for the creation of jobs based on either a
workflow (group of tasks) or an individual task.
To create a job using a workflow:
```js
const job = await payload.createJob({
// Accept the `name` of a workflow so we can match to either a
// code-based workflow OR a workflow defined in the DB
// Should auto-type the input
workflowName: 'myWorkflow',
input: {
// typed to the args of the workflow by name
}
})
```
To create a job using a task:
```js
const job = await payload.createJob({
// Accept the `name` of a task
task: 'myTask',
input: {
// typed to the args of the task by name
}
})
```
---------
Co-authored-by: Alessio Gravili <alessio@gravili.de>
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
BREAKING CHANGE: Rename `@payloadcms/plugin-cloud` to
`@payloadcms/payload-cloud`. Anyone using the existing plugin will need
to switch to using the new package.
## Why?
Since v3 will be using _fixed versioning_, all versions of `^3` must be
available. Unfortunately, the `@payloadcms/plugin-cloud` version has
already breached that version number. Renaming will allow it to be on
the same version as other monorepo packages.
Additionally, the name `plugin-cloud` is quite ambiguous and sometimes
is confused with `plugin-cloud-storage`, so using `payload-cloud` feels
like a good move to make this more evident.
Inheriting all the fixes from drizzle moving to latest versions
## BREAKING CHANGES
If you have a prior version of @libsql/client installed in your project,
you must upgrade to 0.14.0
This PR
- Introduces multiline markdown transformers / mdx support
- Introduce `shouldMergeAdjacentLines` option in
`$convertFromMarkdownString`. If true, merges adjacent lines as per
commonmark spec. This would allow to close:
https://github.com/payloadcms/payload/issues/8049
- Many new features and bug fixes!
- Ports over changes from the lexical playground. Most notably:
- add support for enabling table row stripping
- make table resizing & table cell selection more reliable
**BREAKING**: This upgrades lexical from 0.17.0 to 0.18.0. If you have
any lexical packages installed in your project, please update them
accordingly. Additionally, if you depend on the lexical APIs, please
consult their changelog, as lexical may introduce breaking changes:
https://github.com/facebook/lexical/releases/tag/v0.18.0
---------
Co-authored-by: Alessio Gravili <alessio@gravili.de>
Fixes https://github.com/payloadcms/payload/issues/8013
**BREAKING:**
- Upgrades minimum supported @types/react version from
npm:types-react@19.0.0-rc.0 to npm:types-react@19.0.0-rc.1
- Upgrades minimum supported @types/react-dom version from
npm:types-react-dom@19.0.0-rc.0 to npm:types-react-dom@19.0.0-rc.1
- Upgrades minimum supported react and react-dom version from
19.0.0-rc-06d0b89e-20240801 to 19.0.0-rc-5dcb0097-20240918
- Upgrades minimum supported Next.js version from 15.0.0-canary.104 to
15.0.0-canary.160
---------
Co-authored-by: PatrikKozak <patrik@payloadcms.com>
Co-authored-by: Jacob Fletcher <jacobsfletch@gmail.com>
## Description
- Adds a new "join" field type to Payload and is supported by all database adapters
- The UI uses a table view for the new field
- `db-mongodb` changes relationships to be stored as ObjectIDs instead of strings (for now querying works using both types internally to the DB so no data migration should be necessary unless you're querying directly, see breaking changes for details
- Adds a reusable traverseFields utility to Payload to make it easier to work with nested fields, used internally and for plugin maintainers
```ts
export const Categories: CollectionConfig = {
slug: 'categories',
fields: [
{
name: 'relatedPosts',
type: 'join',
collection: 'posts',
on: 'category',
}
]
}
```
BREAKING CHANGES:
All mongodb relationship and upload values will be stored as MongoDB ObjectIDs instead of strings going forward. If you have existing data and you are querying data directly, outside of Payload's APIs, you get different results. For example, a `contains` query will no longer works given a partial ID of a relationship since the ObjectID requires the whole identifier to work.
---------
Co-authored-by: Jacob Fletcher <jacobsfletch@gmail.com>
Co-authored-by: James <james@trbl.design>
Fixes https://github.com/payloadcms/payload/issues/8111 and
https://github.com/payloadcms/payload/issues/8113
Before: 132 dependencies
After: 123 dependencies
This PR also contains a small performance optimization during telemetry
startup: By using the async `fs.promises.readFile` instead of
`readFileSync` we're not blocking the entire thread anymore and are
allowing other stuff to happen while the file is being read.
Also, in our dependency checker, this moves some variables out of loops,
to the module scope, as they only need to be calculated once.
We have to pin file-type to 19.3.0 and cannot upgrade it further (latest
is 19.5.0). See reasoning in
https://github.com/payloadcms/payload/issues/8111#issuecomment-2348119533
Ajv 8.14.0 => 8.17.1
- Bundle size: 119.6kB => 111kB
- Dependencies: 5 => 4
- Gets rid of dependency on `punycode`. Will help with the annoying
deprecated module console warning spam
This also upgrades TypeScript to 5.6.2 in our monorepo. The most
type-relevant packages are updated as well, e.g. ts-essentials and
@types/node
