Files
payload/test/dev.ts
Alessio Gravili 4c8cafd6a6 perf: deduplicate blocks used in multiple places using new config.blocks property (#10905)
If you have multiple blocks that are used in multiple places, this can quickly blow up the size of your Payload Config. This will incur a performance hit, as more data is
1.  sent to the client (=> bloated `ClientConfig` and large initial html) and
2. processed on the server (permissions are calculated every single time you navigate to a page - this iterates through all blocks you have defined, even if they're duplicative)

This can be optimized by defining your block **once** in your Payload Config, and just referencing the block slug whenever it's used, instead of passing the entire block config. To do this, the block can be defined in the `blocks` array of the Payload Config. The slug can then be passed to the `blockReferences` array in the Blocks Field - the `blocks` array has to be empty for compatibility reasons.

```ts
import { buildConfig } from 'payload'
import { lexicalEditor, BlocksFeature } from '@payloadcms/richtext-lexical'

// Payload Config
const config = buildConfig({
  // Define the block once
  blocks: [
    {
      slug: 'TextBlock',
      fields: [
        {
          name: 'text',
          type: 'text',
        },
      ],
    },
  ],
  collections: [
    {
      slug: 'collection1',
      fields: [
        {
          name: 'content',
          type: 'blocks',
          // Reference the block by slug
          blockReferences: ['TextBlock'],
          blocks: [], // Required to be empty, for compatibility reasons
        },
      ],
    },
     {
      slug: 'collection2',
      fields: [
        {
          name: 'editor',
          type: 'richText',
          editor: lexicalEditor({
            BlocksFeature({
              // Same reference can be reused anywhere, even in the lexical editor, without incurred performance hit
              blocks: ['TextBlock'],
            })
          })
        },
      ],
    },
  ],
})
```

## v4.0 Plans

In 4.0, we will remove the `blockReferences` property, and allow string block references to be passed directly to the blocks `property`. Essentially, we'd remove the `blocks` property and rename `blockReferences` to `blocks`.

The reason we opted to a new property in this PR is to avoid breaking changes. Allowing strings to be passed to the `blocks` property will prevent plugins that iterate through fields / blocks from compiling.

## PR Changes

- Testing: This PR introduces a plugin that automatically converts blocks to block references. This is done in the fields__blocks test suite, to run our existing test suite using block references.

- Block References support: Most changes are similar. Everywhere we iterate through blocks, we have to now do the following:
1. Check if `field.blockReferences` is provided. If so, only iterate through that.
2. Check if the block is an object (= actual block), or string
3. If it's a string, pull the actual block from the Payload Config or from `payload.blocks`.

The exception is config sanitization and block type generations. This PR optimizes them so that each block is only handled once, instead of every time the block is referenced.

## Benchmarks

60 Block fields, each block field having the same 600 Blocks.

### Before:
**Initial HTML:** 195 kB
**Generated types:** takes 11 minutes, 461,209 lines

https://github.com/user-attachments/assets/11d49a4e-5414-4579-8050-e6346e552f56

### After:
**Initial HTML:** 73.6 kB
**Generated types:** takes 2 seconds, 35,810 lines

https://github.com/user-attachments/assets/3eab1a99-6c29-489d-add5-698df67780a3

### After Permissions Optimization (follow-up PR)
Initial HTML: 73.6 kB

https://github.com/user-attachments/assets/a909202e-45a8-4bf6-9a38-8c85813f1312


## Future Plans

1. This PR does not yet deduplicate block references during permissions calculation. We'll optimize that in a separate PR, as this one is already large enough
2. The same optimization can be done to deduplicate fields. One common use-case would be link field groups that may be referenced in multiple entities, outside of blocks. We might explore adding a new `fieldReferences` property, that allows you to reference those same `config.blocks`.
2025-02-14 00:08:20 +00:00

143 lines
4.0 KiB
TypeScript

import nextEnvImport from '@next/env'
import chalk from 'chalk'
import { createServer } from 'http'
import minimist from 'minimist'
import nextImport from 'next'
import fs from 'node:fs'
import path from 'node:path'
import { fileURLToPath } from 'node:url'
import open from 'open'
import { loadEnv } from 'payload/node'
import { parse } from 'url'
import { getNextRootDir } from './helpers/getNextRootDir.js'
import startMemoryDB from './helpers/startMemoryDB.js'
import { runInit } from './runInit.js'
import { child } from './safelyRunScript.js'
import { createTestHooks } from './testHooks.js'
const prod = process.argv.includes('--prod')
if (prod) {
process.argv = process.argv.filter((arg) => arg !== '--prod')
process.env.PAYLOAD_TEST_PROD = 'true'
}
const shouldStartMemoryDB =
process.argv.includes('--start-memory-db') || process.env.START_MEMORY_DB === 'true'
if (shouldStartMemoryDB) {
process.argv = process.argv.filter((arg) => arg !== '--start-memory-db')
process.env.START_MEMORY_DB = 'true'
}
loadEnv()
const filename = fileURLToPath(import.meta.url)
const dirname = path.dirname(filename)
const {
_: [_testSuiteArg = '_community'],
...args
} = minimist(process.argv.slice(2))
let testSuiteArg: string | undefined
let testSuiteConfigOverride: string | undefined
if (_testSuiteArg.includes('#')) {
;[testSuiteArg, testSuiteConfigOverride] = _testSuiteArg.split('#')
} else {
testSuiteArg = _testSuiteArg
}
if (!testSuiteArg || !fs.existsSync(path.resolve(dirname, testSuiteArg))) {
console.log(chalk.red(`ERROR: The test folder "${testSuiteArg}" does not exist`))
process.exit(0)
}
console.log(`Selected test suite: ${testSuiteArg}`)
if (args.turbo === true) {
process.env.TURBOPACK = '1'
}
const { beforeTest } = await createTestHooks(testSuiteArg, testSuiteConfigOverride)
await beforeTest()
const { rootDir, adminRoute } = getNextRootDir(testSuiteArg)
await runInit(testSuiteArg, true)
if (shouldStartMemoryDB) {
await startMemoryDB()
}
// This is needed to forward the environment variables to the next process that were created after loadEnv()
// for example process.env.MONGODB_MEMORY_SERVER_URI otherwise app.prepare() will clear them
nextEnvImport.updateInitialEnv(process.env)
// Open the admin if the -o flag is passed
if (args.o) {
await open(`http://localhost:3000${adminRoute}`)
}
const findOpenPort = (startPort: number): Promise<number> => {
return new Promise((resolve, reject) => {
const server = createServer()
server.listen(startPort, () => {
console.log(`✓ Running on port ${startPort}`)
server.close(() => resolve(startPort))
})
server.on('error', () => {
console.log(`⚠ Port ${startPort} is in use, trying ${startPort + 1} instead.`)
findOpenPort(startPort + 1)
.then(resolve)
.catch(reject)
})
})
}
const port = process.env.PORT ? Number(process.env.PORT) : 3000
const availablePort = await findOpenPort(port)
// @ts-expect-error the same as in test/helpers/initPayloadE2E.ts
const app = nextImport({
dev: true,
hostname: 'localhost',
port: availablePort,
dir: rootDir,
})
const handle = app.getRequestHandler()
let resolveServer: () => void
const serverPromise = new Promise<void>((res) => (resolveServer = res))
void app.prepare().then(() => {
createServer(async (req, res) => {
const parsedUrl = parse(req.url || '', true)
await handle(req, res, parsedUrl)
}).listen(availablePort, () => {
resolveServer()
})
})
await serverPromise
process.env.PAYLOAD_DROP_DATABASE = process.env.PAYLOAD_DROP_DATABASE === 'false' ? 'false' : 'true'
// fetch the admin url to force a render
void fetch(`http://localhost:${availablePort}${adminRoute}`)
void fetch(`http://localhost:${availablePort}/api/access`)
// This ensures that the next-server process is killed when this process is killed and doesn't linger around.
process.on('SIGINT', () => {
if (child) {
child.kill('SIGINT')
}
process.exit(0)
})
process.on('SIGTERM', () => {
if (child) {
child.kill('SIGINT')
}
process.exit(0) // Exit the parent process
})