9a1c3cf4cc
This adds support for running multiple job queue tasks in parallel within the same workflow while preventing conflicts. Previously, this would have caused the following issues: - Job log entries get lost - the final job log is incomplete, despite all tasks having been executed - Write conflicts in postgres, leading to unique constraint violation errors The solution involves handling job log data updates in a way that avoids overwriting, and ensuring the final update reflects the latest job log data. Each job log entry now initializes its own ID, so a given job log entry’s ID remains the same across multiple, parallel task executions. ## Postgres In Postgres, we need to enable transactions for the `payload.db.updateJobs` operation; otherwise, two tasks updating the same job in parallel can conflict. This happens because Postgres handles array rows by deleting them all, then re-inserting (rather than upserting). The rows are stored in a separate table, and the following scenario can occur: Op 1: deletes all job log rows Op 2: deletes all job log rows Op 1: inserts 200 job log rows Op 2: insert the same 200 job log rows again => `error: “duplicate key value violates unique constraint "payload_jobs_log_pkey”` Because transactions were not used, the rows inserted by Op 1 immediately became visible to Op 2, causing the conflict. Enabling transactions fixes this. In theory, it can still happen if Op 1 commits before Op 2 starts inserting (due to the read committed isolation level), but it should occur far less frequently. Alongside this change, we should consider inserting the rows using an upsert (update on conflict), which will get rid of this error completely. That way, if the insertion of Op 1 is visible to Op 2, Op 2 will simply overwrite it, rather than erroring. Individual job entries are immutable and job entries cannot be deleted, thus this shouldn't corrupt any data. ## Mongo In Mongo, the issue is addressed by ensuring that log row deletions caused due to different log states in concurrent operations are not merged back to the client job log, and by making sure the final update includes all job logs. There is no duplicate key error in Mongo because the array log resides in the same document and duplicates are simply upserted. We cannot use transactions in Mongo, as it appears to lock the document in a way that prevents reliable parallel updates, leading to: `MongoServerError: WriteConflict error: this operation conflicted with another operation. Please retry your operation or multi-document transaction`
Explore the Docs · Community Help · Roadmap · View G2 Reviews
Important
🎉 We've released 3.0! Star this repo or keep an eye on it to follow along.
Payload is the first-ever Next.js native CMS that can install directly in your existing /app folder. It's the start of a new era for headless CMS.
Benefits over a regular CMS
- Deploy anywhere, including serverless on Vercel for free
- Combine your front+backend in the same
/appfolder if you want - Don't sign up for yet another SaaS - Payload is open source
- Query your database in React Server Components
- Both admin and backend are 100% extensible
- No vendor lock-in
- Never touch ancient WP code again
- Build faster, never hit a roadblock
Quickstart
Before beginning to work with Payload, make sure you have all of the required software.
pnpx create-payload-app@latest
If you're new to Payload, you should start with the website template (pnpx create-payload-app@latest -t website). It shows how to do everything - including custom Rich Text blocks, on-demand revalidation, live preview, and more. It comes with a frontend built with Tailwind all in one /app folder.
One-click templates
Jumpstart your next project by starting with a pre-made template. These are production-ready, end-to-end solutions designed to get you to market as fast as possible.
🌐 Website
Build any kind of website, blog, or portfolio from small to enterprise. Comes with a fully functional front-end built with RSCs and Tailwind.
We're constantly adding more templates to our Templates Directory. If you maintain your own template, consider adding the payload-template topic to your GitHub repository for others to find.
✨ Features
- Completely free and open-source
- Next.js native, built to run inside your
/appfolder - Use server components to extend Payload UI
- Query your database directly in server components, no need for REST / GraphQL
- Fully TypeScript with automatic types for your data
- Auth out of the box
- Versions and drafts
- Localization
- Block-based layout builder
- Customizable React admin
- Lexical rich text editor
- Conditional field logic
- Extremely granular Access Control
- Document and field-level hooks for every action Payload provides
- Intensely fast API
- Highly secure thanks to HTTP-only cookies, CSRF protection, and more
🗒️ Documentation
Check out the Payload website to find in-depth documentation for everything that Payload offers.
Migrating from v2 to v3? Check out the 3.0 Migration Guide on how to do it.
🙋 Contributing
If you want to add contributions to this repository, please follow the instructions in contributing.md.
📚 Examples
The Examples Directory is a great resource for learning how to setup Payload in a variety of different ways, but you can also find great examples in our blog and throughout our social media.
If you'd like to run the examples, you can either copy them to a folder outside this repo or run them directly by (1) navigating to the example's subfolder (cd examples/your-example-folder) and (2) using the --ignore-workspace flag to bypass workspace restrictions (e.g., pnpm --ignore-workspace install or pnpm --ignore-workspace dev).
You can see more examples at:
🔌 Plugins
Payload is highly extensible and allows you to install or distribute plugins that add or remove functionality. There are both officially-supported and community-supported plugins available. If you maintain your own plugin, consider adding the payload-plugin topic to your GitHub repository for others to find.
🚨 Need help?
There are lots of good conversations and resources in our Github Discussions board and our Discord Server. If you're struggling with something, chances are, someone's already solved what you're up against. 👇
⭐ Like what we're doing? Give us a star
