tabshift/payload
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
1f6efe9a469a03c4d52bcd94d1cfecddef1c75de
payload/test/import-test/import-all-2-exports.ts
James Mikrut 8970c6b3a6 feat: adds jobs queue (#8228) 
Adds a jobs queue to Payload.

- [x] Docs, w/ examples for Vercel Cron, additional services
- [x] Type the `job` using GeneratedTypes in `JobRunnerArgs`
(@AlessioGr)
- [x] Write the `runJobs` function 
- [x] Allow for some type of `payload.runTask` 
- [x] Open up a new bin script for running jobs
- [x] Determine strategy for runner endpoint to either await jobs
successfully or return early and stay open until job work completes
(serverless ramifications here)
- [x] Allow for job runner to accept how many jobs to run in one
invocation
- [x] Make a Payload local API method for creating a new job easily
(payload.createJob) or similar which is strongly typed (@AlessioGr)
- [x] Make `payload.runJobs` or similar  (@AlessioGr)
- [x] Write tests for retrying up to max retries for a given step
- [x] Write tests for dynamic import of a runner

The shape of the config should permit the definition of steps separate
from the job workflows themselves.

```js
const config = {
  // Not sure if we need this property anymore
  queues: {
  },
  // A job is an instance of a workflow, stored in DB
  // and triggered by something at some point
  jobs: {
    // Be able to override the jobs collection
    collectionOverrides: () => {},

    // Workflows are groups of tasks that handle
    // the flow from task to task.
    // When defined on the config, they are considered as predefined workflows
    // BUT - in the future, we'll allow for UI-based workflow definition as well.
    workflows: [
      {
        slug: 'job-name',
        // Temporary name for this
        // should be able to pass function 
        // or path to it for Node to dynamically import
        controlFlowInJS: '/my-runner.js',

        // Temporary name as well
        // should be able to eventually define workflows
        // in UI (meaning they need to be serialized in JSON)
        // Should not be able to define both control flows
        controlFlowInJSON: [
          {
            task: 'myTask',
            next: {
              // etc
            }
          }
        ],

        // Workflows take input
        // which are a group of fields
        input: [
          {
            name: 'post',
            type: 'relationship',
            relationTo: 'posts',
            maxDepth: 0,
            required: true,
          },
          {
            name: 'message',
            type: 'text',
            required: true,
          },
        ],
      },
    ],

    // Tasks are defined separately as isolated functions
    // that can be retried on fail
    tasks: [
      {
        slug: 'myTask',
        retries: 2,
        // Each task takes input
        // Used to auto-type the task func args
        input: [
          {
            name: 'post',
            type: 'relationship',
            relationTo: 'posts',
            maxDepth: 0,
            required: true,
          },
          {
            name: 'message',
            type: 'text',
            required: true,
          },
        ],
        // Each task takes output
        // Used to auto-type the function signature
        output: [
          {
            name: 'success',
            type: 'checkbox',
          }
        ],
        onSuccess: () => {},
        onFail: () => {},
        run: myRunner,
      },
    ]
  }
}
```

### `payload.createJob`

This function should allow for the creation of jobs based on either a
workflow (group of tasks) or an individual task.

To create a job using a workflow:

```js
const job = await payload.createJob({
  // Accept the `name` of a workflow so we can match to either a 
  // code-based workflow OR a workflow defined in the DB
  // Should auto-type the input
  workflowName: 'myWorkflow',
  input: {
    // typed to the args of the workflow by name
  }
})
```

To create a job using a task:

```js
const job = await payload.createJob({
  // Accept the `name` of a task
  task: 'myTask',
  input: {
    // typed to the args of the task by name
  }
})
```

---------

Co-authored-by: Alessio Gravili <alessio@gravili.de>
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
2024-10-30 17:56:50 +00:00

582 lines
12 KiB
TypeScript
Raw Blame History

/* eslint-disable @typescript-eslint/no-unused-vars */
/**
* This is a list of all possible imports from Payload 2.x
*
* All of these should either resolve here
*
* OR
*
* Be documented in the migration guide and breaking changes doc
*/
import payload from 'payload'
import {
CollectionPermission,
FieldPermissions,
GlobalPermission,
IncomingAuthType,
Permission,
Permissions,
User,
VerifyConfig,
} from 'payload/auth'
import {
Banner,
Button,
Check,
Chevron,
ErrorPill,
Menu,
MinimalTemplate,
Pill,
Popup,
Search,
ShimmerEffect,
Tooltip,
X,
} from 'payload/components'
import {
Access,
AccessArgs,
AccessResult,
AdminView,
AdminViewComponent,
AdminViewConfig,
AdminViewProps,
baseBlockFields,
baseIDField,
BaseLocalizationConfig,
buildConfig,
Config,
defaults,
EditView,
EditViewConfig,
EmailOptions,
EmailTransport,
EmailTransportOptions,
Endpoint,
EntityDescription,
FieldTypes,
GeneratePreviewURL,
GraphQLExtension,
hasTransport,
hasTransportOptions,
InitOptions,
LivePreviewConfig,
Locale,
LocalizationConfig,
LocalizationConfigWithLabels,
LocalizationConfigWithNoLabels,
PayloadHandler,
Plugin,
sanitizeConfig,
SanitizedConfig,
SanitizedLocalizationConfig,
sanitizeFields,
} from 'payload/config'
import {
BaseDatabaseAdapter,
BeginTransaction,
combineQueries,
CommitTransaction,
Connect,
Count,
CountArgs,
Create,
CreateArgs,
createDatabaseAdapter,
CreateGlobal,
CreateGlobalArgs,
CreateGlobalVersion,
CreateGlobalVersionArgs,
CreateMigration,
createMigration,
CreateVersion,
CreateVersionArgs,
DBIdentifierName,
DeleteMany,
DeleteManyArgs,
DeleteOne,
DeleteOneArgs,
DeleteVersions,
DeleteVersionsArgs,
Destroy,
EntityPolicies,
Find,
FindArgs,
FindGlobal,
FindGlobalArgs,
FindGlobalVersions,
FindGlobalVersionsArgs,
FindOne,
FindOneArgs,
FindVersions,
FindVersionsArgs,
flattenWhereToOperators,
getLocalizedPaths,
getMigrations,
Init,
migrate,
migrateDown,
migrateRefresh,
migrateReset,
migrateStatus,
Migration,
MigrationData,
migrationsCollection,
migrationTemplate,
PaginatedDocs,
PathToQuery,
QueryDrafts,
QueryDraftsArgs,
readMigrationFiles,
RollbackTransaction,
Transaction,
TypeWithVersion,
UpdateGlobal,
UpdateGlobalArgs,
UpdateGlobalVersion,
UpdateGlobalVersionArgs,
UpdateOne,
UpdateOneArgs,
UpdateVersion,
UpdateVersionArgs,
validateQueryPaths,
validateSearchParam,
} from 'payload/database'
import {
APIError,
AuthenticationError,
DuplicateCollection,
DuplicateGlobal,
ErrorDeletingFile,
FileUploadError,
Forbidden,
InvalidConfiguration,
InvalidFieldName,
InvalidFieldRelationship,
LockedAuth,
MissingCollectionLabel,
MissingFieldInputOptions,
MissingFieldType,
MissingFile,
NotFound,
QueryError,
ValidationError,
} from 'payload/errors'
import { buildPaginatedListType, GraphQL } from 'payload/graphql'
import {
AccessArgs as AccessArgsType,
Access as AccessType,
AllOperations,
ArrayField,
AuthOperations,
BeforeDuplicate,
Block,
BlockField,
CellComponentProps,
CheckboxField,
CodeField,
CollapsibleField,
Collection,
CollectionAfterChangeHook,
CollectionAfterDeleteHook,
CollectionAfterForgotPasswordHook,
CollectionAfterLoginHook,
CollectionAfterOperationHook,
CollectionAfterReadHook,
CollectionBeforeChangeHook,
CollectionBeforeDeleteHook,
CollectionBeforeLoginHook,
CollectionBeforeOperationHook,
CollectionBeforeReadHook,
CollectionBeforeValidateHook,
CollectionConfig,
Condition,
CreateFormData,
CustomPublishButtonProps,
CustomPublishButtonType,
CustomSaveButtonProps,
CustomSaveDraftButtonProps,
Data,
DateField,
docHasTimestamps,
Document,
EmailField,
Field,
FieldAccess,
FieldAffectingData,
fieldAffectsData,
FieldBase,
fieldHasMaxDepth,
fieldHasSubFields,
FieldHook,
FieldHookArgs,
fieldIsArrayType,
fieldIsBlockType,
fieldIsGroupType,
fieldIsLocalized,
fieldIsPresentationalOnly,
FieldPresentationalOnly,
Fields,
fieldSupportsMany,
FieldWithMany,
FieldWithMaxDepth,
FieldWithPath,
FieldWithRichTextRequiredEditor,
FieldWithSubFields,
FileData,
FilterOptions,
FilterOptionsProps,
FormField,
FormFieldsContext,
GlobalAfterChangeHook,
GlobalAfterReadHook,
GlobalBeforeChangeHook,
GlobalBeforeReadHook,
GlobalBeforeValidateHook,
GlobalConfig,
GroupField,
HookName,
ImageSize,
IncomingUploadType,
JSONField,
Labels,
NamedTab,
NonPresentationalField,
NumberField,
Operation,
Operator,
Option,
optionIsObject,
optionIsValue,
OptionObject,
optionsAreObjects,
PayloadRequest,
PointField,
PolymorphicRelationshipField,
RadioField,
RelationshipField,
RelationshipValue,
RichTextAdapter,
RichTextFieldProps,
RichTextFieldRequiredEditor,
RichTextField as RichTextFieldType,
RowAdmin,
RowField,
RowLabel,
SanitizedCollectionConfig,
SanitizedGlobalConfig,
SelectField,
SingleRelationshipField,
Tab,
TabAsField,
tabHasName,
TabsAdmin,
TabsField,
TextareaField,
TextField,
TypeWithID,
UIField,
UnnamedTab,
UploadField,
Validate,
ValidateOptions,
validOperators,
valueIsValueWithRelation,
ValueWithRelation,
VersionOperations,
Where,
WhereField,
} from 'payload/types'
import {
afterReadPromise,
afterReadTraverseFields,
combineMerge,
configToJSONSchema,
createArrayFromCommaDelineated,
deepCopyObject,
deepMerge,
entityToJSONSchema,
extractTranslations,
fieldSchemaToJSON,
fieldsToJSONSchema,
flattenTopLevelFields,
formatLabels,
formatNames,
getCollectionIDFieldTypes,
getIDType,
getTranslation,
i18nInit,
isValidID,
toWords,
withMergedProps,
withNullableJSONSchemaType,
} from 'payload/utilities'
import {
buildVersionCollectionFields,
buildVersionGlobalFields,
deleteCollectionVersions,
enforceMaxVersions,
getLatestCollectionVersion,
getLatestGlobalVersion,
saveVersion,
} from 'payload/versions'
/**
* Plugins
*/
import {
Args,
MigrateDownArgs,
MigrateUpArgs,
MongooseAdapter,
mongooseAdapter,
} from '@payloadcms/db-mongodb'
import {
MigrateDownArgs as MigrateDownArgsPg,
MigrateUpArgs as MigrateUpArgsPg,
postgresAdapter,
} from '@payloadcms/db-postgres'
import { handleMessage, mergeData, ready, subscribe, unsubscribe } from '@payloadcms/live-preview'
import { useLivePreview } from '@payloadcms/live-preview-react'
import { createKey, getStorageClient, payloadCloud } from '@payloadcms/payload-cloud'
import { cloudStorage } from '@payloadcms/plugin-cloud-storage'
import { fields, getPaymentTotal } from '@payloadcms/plugin-form-builder'
import {
BeforeEmail,
BlockConfig,
CountryField,
Email,
FieldConfig,
FieldsConfig,
FieldValues,
Form,
FormattedEmail,
CheckboxField as FormBuilderCheckboxField,
EmailField as FormBuilderEmailField,
SelectField as FormBuilderSelectField,
TextField as FormBuilderTextField,
FormFieldBlock,
FormSubmission,
HandlePayment,
isValidBlockConfig,
MessageField,
PaymentField,
PaymentFieldConfig,
PluginConfig,
PriceCondition,
Redirect,
SelectFieldOption,
StateField,
SubmissionValue,
TextAreaField,
} from '@payloadcms/plugin-form-builder/types'
import nestedDocs from '@payloadcms/plugin-nested-docs'
import { createBreadcrumbsField, createParentField } from '@payloadcms/plugin-nested-docs/fields'
import {
Breadcrumb,
GenerateLabel,
GenerateURL,
PluginConfig as NestedDocsPluginConfig,
} from '@payloadcms/plugin-nested-docs/types'
import redirects from '@payloadcms/plugin-redirects'
import { PluginConfig as RedirectsPluginConfig } from '@payloadcms/plugin-redirects/types'
// Skip plugin-sentry
import search from '@payloadcms/plugin-search'
import {
BeforeSync,
DocToSync,
SearchConfig,
SyncWithSearch,
} from '@payloadcms/plugin-search/types'
import seo from '@payloadcms/plugin-seo'
import {
GenerateDescription,
GenerateImage,
GenerateTitle,
Meta,
GenerateURL as seoGenerateURL,
PluginConfig as SeoPluginConfig,
} from '@payloadcms/plugin-seo/types'
import stripePlugin from '@payloadcms/plugin-stripe'
import {
FieldSyncConfig,
SanitizedStripeConfig,
StripeConfig,
StripeProxy,
StripeWebhookHandler,
StripeWebhookHandlers,
SyncConfig,
} from '@payloadcms/plugin-stripe/types'
import {
$createAutoLinkNode,
$createBlockNode,
$createLinkNode,
$createRelationshipNode,
$createUploadNode,
$isAutoLinkNode,
$isBlockNode,
$isLinkNode,
$isRelationshipNode,
$isUploadNode,
AdapterProps,
addSwipeDownListener,
addSwipeLeftListener,
addSwipeRightListener,
addSwipeUpListener,
AlignFeature,
AutoLinkNode,
BlockFields,
BlockNode,
BlockQuoteFeature,
BlocksFeature,
BlocksFeatureProps,
BoldTextFeature,
CAN_USE_DOM,
CheckListFeature,
cloneDeep,
consolidateHTMLConverters,
convertLexicalNodesToHTML,
convertLexicalToHTML,
convertSlateNodesToLexical,
convertSlateToLexical,
createBlockNode,
defaultEditorConfig,
defaultEditorFeatures,
defaultHTMLConverters,
defaultRichTextValue,
defaultSanitizedEditorConfig,
defaultSlateConverters,
DETAIL_TYPE_TO_DETAIL,
DOUBLE_LINE_BREAK,
EditorConfig,
EditorConfigProvider,
ELEMENT_FORMAT_TO_TYPE,
ELEMENT_TYPE_TO_FORMAT,
ENABLE_SLASH_MENU_COMMAND,
Feature,
FeatureProvider,
FeatureProviderMap,
FloatingToolbarSection,
FloatingToolbarSectionEntry,
FormatSectionWithEntries,
getDOMRangeRect,
getEnabledNodes,
getSelectedNode,
HeadingFeature,
HTMLConverter,
HTMLConverterFeature,
HTMLConverterFeatureProps,
IndentFeature,
InlineCodeTextFeature,
invariant,
IS_ALL_FORMATTING,
isHTMLElement,
isPoint,
ItalicTextFeature,
joinClasses,
LexicalBlock,
lexicalEditor,
LexicalEditorProps,
lexicalHTML,
LexicalPluginToLexicalFeature,
LexicalRichTextAdapter,
LinebreakHTMLConverter,
LinkFeature,
LinkFeatureProps,
LinkFields,
LinkNode,
loadFeatures,
LTR_REGEX,
NodeFormat,
NodeValidation,
NON_BREAKING_SPACE,
OrderedListFeature,
ParagraphFeature,
ParagraphHTMLConverter,
Point,
PopulationPromise,
RawUploadPayload,
Rect,
RelationshipData,
RelationshipFeature,
RelationshipNode,
ResolvedFeature,
ResolvedFeatureMap,
RTL_REGEX,
SanitizedEditorConfig,
SanitizedFeatures,
sanitizeEditorConfig,
sanitizeFeatures,
sanitizeUrl,
SerializedAutoLinkNode,
SerializedBlockNode,
SerializedLinkNode,
SerializedRelationshipNode,
SerializedUploadNode,
setFloatingElemPosition,
setFloatingElemPositionForLinkEditor,
SlashMenuGroup,
SlashMenuOption,
SlateBlockquoteConverter,
SlateHeadingConverter,
SlateIndentConverter,
SlateLinkConverter,
SlateListItemConverter,
SlateNode,
SlateNodeConverter,
SlateOrderedListConverter,
SlateRelationshipConverter,
SlateToLexicalFeature,
SlateUnknownConverter,
SlateUnorderedListConverter,
SlateUploadConverter,
sortFeaturesForOptimalLoading,
StrikethroughTextFeature,
SubscriptTextFeature,
SuperscriptTextFeature,
TestRecorderFeature,
TEXT_MODE_TO_TYPE,
TEXT_TYPE_TO_FORMAT,
TEXT_TYPE_TO_MODE,
TextDropdownSectionWithEntries,
TextHTMLConverter,
TOGGLE_LINK_COMMAND,
TreeViewFeature,
UnderlineTextFeature,
UnorderedListFeature,
UploadData,
UploadFeature,
UploadFeatureProps,
UploadNode,
useEditorConfigContext,
validateUrl,
} from '@payloadcms/richtext-lexical'
import {
defaultEditorLexicalConfig,
RichTextCell,
RichTextField,
ToolbarButton,
ToolbarDropdown,
} from '@payloadcms/richtext-lexical/components'
import {
AdapterArguments,
ElementButton,
ElementNode,
FieldProps,
LeafButton,
nodeIsTextNode,
RichTextCustomElement,
RichTextCustomLeaf,
RichTextElement,
RichTextLeaf,
slateEditor,
TextNode,
toggleElement,
} from '@payloadcms/richtext-slate'
Reference in New Issue View Git Blame Copy Permalink