tabshift/payloadcms
1
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

feat: adds support for both client-side and server-side remote URL uploads fetching (#10004)

Browse Source 
### What?

The `pasteURL` feature for Upload fields has been updated to support
both **client-side** and **server-side** URL fetching. Previously, users
could only paste URLs from the same domain as their Payload instance
(internal) or public domains, which led to **CORS** errors when trying
to fetch files from external URLs.

Now, users can choose between **client-side fetching** (default) and
**server-side fetching** using the new `pasteURL` option in the Upload
collection config.

### How?

- By default, Payload will attempt to fetch the file client-side
directly in the browser.
- To enable server-side fetching, you can configure the new `pasteURL`
option with an `allowList` of trusted domains.
- The new `/api/:collectionSlug/paste-url` endpoint is used to fetch
files server-side and stream them back to the browser.

#### Example

```
import type { CollectionConfig } from 'payload'

export const Media: CollectionConfig = {
  slug: 'media',
  upload: {
    // pasteURL: false, // Can now disable the pasteURL option entirely by passing "false".
    pasteURL: {
      allowList: [
        {
          hostname: 'payloadcms.com', // required
          pathname: '',
          port: '',
          protocol: 'https', // defaults to https - options: "https" | "http"
          search: ''
        },
        {
          hostname: 'example.com',
          pathname: '/images/*',
        },
      ],
    },
  },
}
```

### Why

This update provides more flexibility for users to paste URLs into
Upload fields without running into **CORS errors** and allows Payload to
securely fetch files from trusted domains.
This commit is contained in:
Patrik
2025-01-17 09:16:29 -05:00
committed by GitHub
parent 28b7c04681
commit 38a06e7bd3
13 changed files with 492 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files

99
docs/upload/overview.mdx
View File

@@ -88,26 +88,27 @@ export const Media: CollectionConfig = {
_An asterisk denotes that an option is required._ _An asterisk denotes that an option is required._
| Option | Description | | Option | Description |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **`adminThumbnail`** | Set the way that the [Admin Panel](../admin/overview) will display thumbnails for this Collection. [More](#admin-thumbnails) | | **`adminThumbnail`** | Set the way that the [Admin Panel](../admin/overview) will display thumbnails for this Collection. [More](#admin-thumbnails) |
| **`bulkUpload`** | Allow users to upload in bulk from the list view, default is true | | **`bulkUpload`** | Allow users to upload in bulk from the list view, default is true |
| **`cacheTags`** | Set to `false` to disable the cache tag set in the UI for the admin thumbnail component. Useful for when CDNs don't allow certain cache queries. | | **`cacheTags`** | Set to `false` to disable the cache tag set in the UI for the admin thumbnail component. Useful for when CDNs don't allow certain cache queries. |
| **`crop`** | Set to `false` to disable the cropping tool in the [Admin Panel](../admin/overview). Crop is enabled by default. [More](#crop-and-focal-point-selector) | | **`crop`** | Set to `false` to disable the cropping tool in the [Admin Panel](../admin/overview). Crop is enabled by default. [More](#crop-and-focal-point-selector) |
| **`disableLocalStorage`** | Completely disable uploading files to disk locally. [More](#disabling-local-upload-storage) | | **`disableLocalStorage`** | Completely disable uploading files to disk locally. [More](#disabling-local-upload-storage) |
| **`displayPreview`** | Enable displaying preview of the uploaded file in Upload fields related to this Collection. Can be locally overridden by `displayPreview` option in Upload field. [More](/docs/fields/upload#config-options). | | **`displayPreview`** | Enable displaying preview of the uploaded file in Upload fields related to this Collection. Can be locally overridden by `displayPreview` option in Upload field. [More](/docs/fields/upload#config-options). |
| **`externalFileHeaderFilter`** | Accepts existing headers and returns the headers after filtering or modifying. | | **`externalFileHeaderFilter`** | Accepts existing headers and returns the headers after filtering or modifying. |
| **`filesRequiredOnCreate`** | Mandate file data on creation, default is true. | | **`filesRequiredOnCreate`** | Mandate file data on creation, default is true. |
| **`filenameCompoundIndex`** | Field slugs to use for a compound index instead of the default filename index. | | **`filenameCompoundIndex`** | Field slugs to use for a compound index instead of the default filename index. |
| **`focalPoint`** | Set to `false` to disable the focal point selection tool in the [Admin Panel](../admin/overview). The focal point selector is only available when `imageSizes` or `resizeOptions` are defined. [More](#crop-and-focal-point-selector) | | **`focalPoint`** | Set to `false` to disable the focal point selection tool in the [Admin Panel](../admin/overview). The focal point selector is only available when `imageSizes` or `resizeOptions` are defined. [More](#crop-and-focal-point-selector) |
| **`formatOptions`** | An object with `format` and `options` that are used with the Sharp image library to format the upload file. [More](https://sharp.pixelplumbing.com/api-output#toformat) | | **`formatOptions`** | An object with `format` and `options` that are used with the Sharp image library to format the upload file. [More](https://sharp.pixelplumbing.com/api-output#toformat) |
| **`handlers`** | Array of Request handlers to execute when fetching a file, if a handler returns a Response it will be sent to the client. Otherwise Payload will retrieve and send back the file. | | **`handlers`** | Array of Request handlers to execute when fetching a file, if a handler returns a Response it will be sent to the client. Otherwise Payload will retrieve and send back the file. |
| **`imageSizes`** | If specified, image uploads will be automatically resized in accordance to these image sizes. [More](#image-sizes) | | **`imageSizes`** | If specified, image uploads will be automatically resized in accordance to these image sizes. [More](#image-sizes) |
| **`mimeTypes`** | Restrict mimeTypes in the file picker. Array of valid mimetypes or mimetype wildcards [More](#mimetypes) | | **`mimeTypes`** | Restrict mimeTypes in the file picker. Array of valid mimetypes or mimetype wildcards [More](#mimetypes) |
| **`resizeOptions`** | An object passed to the the Sharp image library to resize the uploaded file. [More](https://sharp.pixelplumbing.com/api-resize) | | **`pasteURL`** | Controls whether files can be uploaded from remote URLs by pasting them into the Upload field. **Enabled by default.** Accepts `false` to disable or an object with an `allowList` of valid remote URLs. [More](#uploading-files-from-remote-urls) |
| **`staticDir`** | The folder directory to use to store media in. Can be either an absolute path or relative to the directory that contains your config. Defaults to your collection slug | | **`resizeOptions`** | An object passed to the the Sharp image library to resize the uploaded file. [More](https://sharp.pixelplumbing.com/api-resize) |
| **`trimOptions`** | An object passed to the the Sharp image library to trim the uploaded file. [More](https://sharp.pixelplumbing.com/api-resize#trim) | | **`staticDir`** | The folder directory to use to store media in. Can be either an absolute path or relative to the directory that contains your config. Defaults to your collection slug |
| **`withMetadata`** | If specified, appends metadata to the output image file. Accepts a boolean or a function that receives `metadata` and `req`, returning a boolean. | | **`trimOptions`** | An object passed to the the Sharp image library to trim the uploaded file. [More](https://sharp.pixelplumbing.com/api-resize#trim) |
| **`withMetadata`** | If specified, appends metadata to the output image file. Accepts a boolean or a function that receives `metadata` and `req`, returning a boolean. |
### Payload-wide Upload Options ### Payload-wide Upload Options
@@ -327,6 +328,64 @@ fetch('api/:upload-slug', {
}) })
``` ```
## Uploading Files from Remote URLs
The `pasteURL` option allows users to fetch files from remote URLs by pasting them into an Upload field. This option is **enabled by default** and can be configured to either **allow unrestricted client-side fetching** or **restrict server-side fetching** to specific trusted domains.
By default, Payload uses **client-side fetching**, where the browser downloads the file directly from the provided URL. However, **client-side fetching will fail if the URL’s server has CORS restrictions**, making it suitable only for internal URLs or public URLs without CORS blocks.
To fetch files from **restricted URLs** that would otherwise be blocked by CORS, use **server-side fetching** by configuring the `pasteURL` option with an `allowList` of trusted domains. This method ensures that Payload downloads the file on the server and streams it to the browser. However, for security reasons, only URLs that match the specified `allowList` will be allowed.
#### Configuration Example
Here’s how to configure the pasteURL option to control remote URL fetching:
```
import type { CollectionConfig } from 'payload'
export const Media: CollectionConfig = {
slug: 'media',
upload: {
pasteURL: {
allowList: [
{
hostname: 'payloadcms.com', // required
pathname: '',
port: '',
protocol: 'https',
search: ''
},
{
hostname: 'example.com',
pathname: '/images/*',
},
],
},
},
}
```
##### Accepted Values for `pasteURL`
| Option | Description |
| --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **`undefined`** | Default behavior. Enables client-side fetching for internal or public URLs. |
| **`false`** | Disables the ability to paste URLs into Upload fields. |
| **`allowList`** | Enables server-side fetching for specific trusted URLs. Requires an array of objects defining trusted domains. See the table below for details on `AllowItem`. |
##### `AllowItem` Properties
_An asterisk denotes that an option is required._
| Option | Description | Example |
| ---------------- | ---------------------------------------------------------------------------------------------------- | ------------- |
| **`hostname`** * | The hostname of the allowed URL. This is required to ensure the URL is coming from a trusted source. | `example.com` |
| **`pathname`** | The path portion of the URL. Supports wildcards to match multiple paths. | `/images/*` |
| **`port`** | The port number of the URL. If not specified, the default port for the protocol will be used. | `3000` |
| **`protocol`** | The protocol to match. Must be either `http` or `https`. Defaults to `https`. | `https` |
| **`search`** | The query string of the URL. If specified, the URL must match this exact query string. | `?version=1` |
## Access Control ## Access Control
All files that are uploaded to each Collection automatically support the `read` [Access Control](/docs/access-control/overview) function from the Collection itself. You can use this to control who should be allowed to see your uploads, and who should not. All files that are uploaded to each Collection automatically support the `read` [Access Control](/docs/access-control/overview) function from the Collection itself. You can use this to control who should be allowed to see your uploads, and who should not.

81
packages/payload/src/collections/endpoints/getFileFromURL.ts Normal file
View File

@@ -0,0 +1,81 @@
import type { PayloadHandler } from '../../config/types.js'
import executeAccess from '../../auth/executeAccess.js'
import { APIError } from '../../errors/APIError.js'
import { Forbidden } from '../../errors/Forbidden.js'
import { getRequestCollectionWithID } from '../../utilities/getRequestEntity.js'
import { isURLAllowed } from '../../utilities/isURLAllowed.js'
// If doc id is provided, it means we are updating the doc
// /:collectionSlug/paste-url/:doc-id?src=:fileUrl
// If doc id is not provided, it means we are creating a new doc
// /:collectionSlug/paste-url?src=:fileUrl
export const getFileFromURLHandler: PayloadHandler = async (req) => {
const { id, collection } = getRequestCollectionWithID(req, { optionalID: true })
if (!req.user) {
throw new Forbidden(req.t)
}
const config = collection?.config
if (id) {
// updating doc
const accessResult = await executeAccess({ req }, config.access.update)
if (!accessResult) {
throw new Forbidden(req.t)
}
} else {
// creating doc
const accessResult = await executeAccess({ req }, config.access?.create)
if (!accessResult) {
throw new Forbidden(req.t)
}
}
try {
if (!req.url) {
throw new APIError('Request URL is missing.', 400)
}
const { searchParams } = new URL(req.url)
const src = searchParams.get('src')
if (!src || typeof src !== 'string') {
throw new APIError('A valid URL string is required.', 400)
}
const validatedUrl = new URL(src)
if (
typeof config.upload?.pasteURL === 'object' &&
!isURLAllowed(validatedUrl.href, config.upload.pasteURL.allowList)
) {
throw new APIError(`The provided URL (${validatedUrl.href}) is not allowed.`, 400)
}
// Fetch the file with no compression
const response = await fetch(validatedUrl.href, {
headers: {
'Accept-Encoding': 'identity',
},
})
if (!response.ok) {
throw new APIError(`Failed to fetch file from ${validatedUrl.href}`, response.status)
}
const decodedFileName = decodeURIComponent(validatedUrl.pathname.split('/').pop() || '')
return new Response(response.body, {
headers: {
'Content-Disposition': `attachment; filename="${decodedFileName}"`,
'Content-Length': response.headers.get('content-length') || '',
'Content-Type': response.headers.get('content-type') || 'application/octet-stream',
},
})
} catch (error) {
throw new APIError(`Error fetching file: ${error.message}`, 500)
}
}

6
packages/payload/src/collections/endpoints/index.ts
View File

@@ -13,6 +13,7 @@ import { findByIDHandler } from './findByID.js'
import { findVersionByIDHandler } from './findVersionByID.js' import { findVersionByIDHandler } from './findVersionByID.js'
import { findVersionsHandler } from './findVersions.js' import { findVersionsHandler } from './findVersions.js'
import { getFileHandler } from './getFile.js' import { getFileHandler } from './getFile.js'
import { getFileFromURLHandler } from './getFileFromURL.js'
import { previewHandler } from './preview.js' import { previewHandler } from './preview.js'
import { restoreVersionHandler } from './restoreVersion.js' import { restoreVersionHandler } from './restoreVersion.js'
import { updateHandler } from './update.js' import { updateHandler } from './update.js'
@@ -46,6 +47,11 @@ export const defaultCollectionEndpoints: Endpoint[] = [
method: 'post', method: 'post',
path: '/access/:id?', path: '/access/:id?',
}, },
{
handler: getFileFromURLHandler,
method: 'get',
path: '/paste-url/:id?',
},
{ {
handler: findVersionsHandler, handler: findVersionsHandler,
method: 'get', method: 'get',

19
packages/payload/src/uploads/types.ts
View File

@@ -83,6 +83,14 @@ export type ImageSize = {
export type GetAdminThumbnail = (args: { doc: Record<string, unknown> }) => false | null | string export type GetAdminThumbnail = (args: { doc: Record<string, unknown> }) => false | null | string
export type AllowList = Array<{
hostname: string
pathname?: string
port?: string
protocol?: 'http' | 'https'
search?: string
}>
export type UploadConfig = { export type UploadConfig = {
/** /**
* The adapter name to use for uploads. Used for storage adapter telemetry. * The adapter name to use for uploads. Used for storage adapter telemetry.
@@ -175,6 +183,17 @@ export type UploadConfig = {
* @default undefined * @default undefined
*/ */
modifyResponseHeaders?: ({ headers }: { headers: Headers }) => Headers modifyResponseHeaders?: ({ headers }: { headers: Headers }) => Headers
/**
* Controls the behavior of pasting/uploading files from URLs.
* If set to `false`, fetching from remote URLs is disabled.
* If an allowList is provided, server-side fetching will be enabled for specified URLs.
* @default true (client-side fetching enabled)
*/
pasteURL?:
| {
allowList: AllowList
}
| false
/** /**
* Sharp resize options for the original image. * Sharp resize options for the original image.
* @link https://sharp.pixelplumbing.com/api-resize#resize * @link https://sharp.pixelplumbing.com/api-resize#resize

35
packages/payload/src/utilities/isURLAllowed.ts Normal file
View File

@@ -0,0 +1,35 @@
import type { AllowList } from '../uploads/types.js'
export const isURLAllowed = (url: string, allowList: AllowList): boolean => {
try {
const parsedUrl = new URL(url)
return allowList.some((allowItem) => {
return Object.entries(allowItem).every(([key, value]) => {
// Skip undefined or null values
if (!value) {
return true
}
// Compare protocol with colon
if (key === 'protocol') {
return typeof value === 'string' && parsedUrl.protocol === `${value}:`
}
if (key === 'pathname') {
// Convert wildcards to a regex
const regexPattern = value
.replace(/\*\*/g, '.*') // Match any path
.replace(/\*/g, '[^/]*') // Match any part of a path segment
.replace(/\/$/, '(/)?') // Allow optional trailing slash
const regex = new RegExp(`^${regexPattern}$`)
return regex.test(parsedUrl.pathname)
}
// Default comparison for all other properties (hostname, port, search)
return parsedUrl[key as keyof URL] === value
})
})
} catch {
return false // If the URL is invalid, deny by default
}
}

99
packages/ui/src/elements/Upload/index.tsx
View File

@@ -2,13 +2,14 @@
import type { FormState, SanitizedCollectionConfig, UploadEdits } from 'payload' import type { FormState, SanitizedCollectionConfig, UploadEdits } from 'payload'
import { isImage } from 'payload/shared' import { isImage } from 'payload/shared'
import React, { useCallback, useEffect, useRef, useState } from 'react' import React, { Fragment, useCallback, useEffect, useRef, useState } from 'react'
import { toast } from 'sonner' import { toast } from 'sonner'
import { FieldError } from '../../fields/FieldError/index.js' import { FieldError } from '../../fields/FieldError/index.js'
import { fieldBaseClass } from '../../fields/shared/index.js' import { fieldBaseClass } from '../../fields/shared/index.js'
import { useForm, useFormProcessing } from '../../forms/Form/index.js' import { useForm, useFormProcessing } from '../../forms/Form/index.js'
import { useField } from '../../forms/useField/index.js' import { useField } from '../../forms/useField/index.js'
import { useConfig } from '../../providers/Config/index.js'
import { useDocumentInfo } from '../../providers/DocumentInfo/index.js' import { useDocumentInfo } from '../../providers/DocumentInfo/index.js'
import { EditDepthProvider } from '../../providers/EditDepth/index.js' import { EditDepthProvider } from '../../providers/EditDepth/index.js'
import { useTranslation } from '../../providers/Translation/index.js' import { useTranslation } from '../../providers/Translation/index.js'
@@ -18,8 +19,8 @@ import { Drawer, DrawerToggler } from '../Drawer/index.js'
import { Dropzone } from '../Dropzone/index.js' import { Dropzone } from '../Dropzone/index.js'
import { EditUpload } from '../EditUpload/index.js' import { EditUpload } from '../EditUpload/index.js'
import { FileDetails } from '../FileDetails/index.js' import { FileDetails } from '../FileDetails/index.js'
import { PreviewSizes } from '../PreviewSizes/index.js'
import './index.scss' import './index.scss'
import { PreviewSizes } from '../PreviewSizes/index.js'
import { Thumbnail } from '../Thumbnail/index.js' import { Thumbnail } from '../Thumbnail/index.js'
const baseClass = 'file-field' const baseClass = 'file-field'
@@ -92,10 +93,17 @@ export type UploadProps = {
export const Upload: React.FC<UploadProps> = (props) => { export const Upload: React.FC<UploadProps> = (props) => {
const { collectionSlug, customActions, initialState, onChange, uploadConfig } = props const { collectionSlug, customActions, initialState, onChange, uploadConfig } = props
const {
config: {
routes: { api },
serverURL,
},
} = useConfig()
const { t } = useTranslation() const { t } = useTranslation()
const { setModified } = useForm() const { setModified } = useForm()
const { resetUploadEdits, updateUploadEdits, uploadEdits } = useUploadEdits() const { resetUploadEdits, updateUploadEdits, uploadEdits } = useUploadEdits()
const { docPermissions, savedDocumentData, setUploadStatus } = useDocumentInfo() const { id, docPermissions, savedDocumentData, setUploadStatus } = useDocumentInfo()
const isFormSubmitting = useFormProcessing() const isFormSubmitting = useFormProcessing()
const { errorMessage, setValue, showError, value } = useField<File>({ const { errorMessage, setValue, showError, value } = useField<File>({
path: 'file', path: 'file',
@@ -111,6 +119,9 @@ export const Upload: React.FC<UploadProps> = (props) => {
const urlInputRef = useRef<HTMLInputElement>(null) const urlInputRef = useRef<HTMLInputElement>(null)
const inputRef = useRef<HTMLInputElement>(null) const inputRef = useRef<HTMLInputElement>(null)
const useServerSideFetch =
typeof uploadConfig?.pasteURL === 'object' && uploadConfig.pasteURL.allowList?.length > 0
const handleFileChange = useCallback( const handleFileChange = useCallback(
(newFile: File) => { (newFile: File) => {
if (newFile instanceof File) { if (newFile instanceof File) {
@@ -174,23 +185,53 @@ export const Upload: React.FC<UploadProps> = (props) => {
) )
const handleUrlSubmit = async () => { const handleUrlSubmit = async () => {
if (fileUrl) { if (!fileUrl || uploadConfig?.pasteURL === false) {
setUploadStatus('uploading') return
try { }
const response = await fetch(fileUrl)
const data = await response.blob()
// Extract the file name from the URL setUploadStatus('uploading')
const fileName = fileUrl.split('/').pop() try {
// Attempt client-side fetch
const clientResponse = await fetch(fileUrl)
// Create a new File object from the Blob data if (!clientResponse.ok) {
const file = new File([data], fileName, { type: data.type }) throw new Error(`Fetch failed with status: ${clientResponse.status}`)
handleFileChange(file)
setUploadStatus('idle')
} catch (e) {
toast.error(e.message)
setUploadStatus('failed')
} }
const blob = await clientResponse.blob()
const fileName = decodeURIComponent(fileUrl.split('/').pop() || '')
const file = new File([blob], fileName, { type: blob.type })
handleFileChange(file)
setUploadStatus('idle')
return // Exit if client-side fetch succeeds
} catch (_clientError) {
if (!useServerSideFetch) {
// If server-side fetch is not enabled, show client-side error
toast.error('Failed to fetch the file.')
setUploadStatus('failed')
return
}
}
// Attempt server-side fetch if client-side fetch fails and useServerSideFetch is true
try {
const pasteURL = `/${collectionSlug}/paste-url${id ? `/${id}?` : '?'}src=${encodeURIComponent(fileUrl)}`
const serverResponse = await fetch(`${serverURL}${api}${pasteURL}`)
if (!serverResponse.ok) {
throw new Error(`Fetch failed with status: ${serverResponse.status}`)
}
const blob = await serverResponse.blob()
const fileName = decodeURIComponent(fileUrl.split('/').pop() || '')
const file = new File([blob], fileName, { type: blob.type })
handleFileChange(file)
setUploadStatus('idle')
} catch (_serverError) {
toast.error('The provided URL is not allowed.')
setUploadStatus('failed')
} }
} }
@@ -272,16 +313,20 @@ export const Upload: React.FC<UploadProps> = (props) => {
ref={inputRef} ref={inputRef}
type="file" type="file"
/> />
<span className={`${baseClass}__orText`}>{t('general:or')}</span> {uploadConfig?.pasteURL !== false && (
<Button <Fragment>
buttonStyle="pill" <span className={`${baseClass}__orText`}>{t('general:or')}</span>
onClick={() => { <Button
setShowUrlInput(true) buttonStyle="pill"
}} onClick={() => {
size="small" setShowUrlInput(true)
> }}
{t('upload:pasteURL')} size="small"
</Button> >
{t('upload:pasteURL')}
</Button>
</Fragment>
)}
</div> </div>
<p className={`${baseClass}__dragAndDropText`}> <p className={`${baseClass}__dragAndDropText`}>

34
packages/ui/src/utilities/isURLAllowed.ts Normal file
View File

@@ -0,0 +1,34 @@
import type { AllowList } from 'payload'
export const isURLAllowed = (url: string, allowList: AllowList): boolean => {
try {
const parsedUrl = new URL(url)
return allowList.some((allowItem) => {
return Object.entries(allowItem).every(([key, value]) => {
// Skip undefined or null values
if (!value) {
return true
}
// Compare protocol with colon
if (key === 'protocol') {
return typeof value === 'string' && parsedUrl.protocol === `${value}:`
}
if (key === 'pathname') {
// Convert wildcards to a regex
const regexPattern = value
.replace(/\*\*/g, '.*') // Match any path
.replace(/\*/g, '[^/]*') // Match any part of a path segment
const regex = new RegExp(`^${regexPattern}$`)
return regex.test(parsedUrl.pathname)
}
// Default comparison for all other properties (hostname, port, search)
return parsedUrl[key as keyof URL] === value
})
})
} catch {
return false // If the URL is invalid, deny by default
}
}

24
test/helpers.ts
View File

@@ -392,7 +392,9 @@ export async function switchTab(page: Page, selector: string) {
* Useful to prevent the e2e test from passing when, for example, there are react missing key prop errors * Useful to prevent the e2e test from passing when, for example, there are react missing key prop errors
* @param page * @param page
*/ */
export function initPageConsoleErrorCatch(page: Page) { export function initPageConsoleErrorCatch(page: Page, options?: { ignoreCORS?: boolean }) {
const { ignoreCORS = false } = options || {} // Default to not ignoring CORS errors
page.on('console', (msg) => { page.on('console', (msg) => {
if ( if (
msg.type() === 'error' && msg.type() === 'error' &&
@@ -407,13 +409,31 @@ export function initPageConsoleErrorCatch(page: Page) {
!msg.text().includes('Error getting document data') && !msg.text().includes('Error getting document data') &&
!msg.text().includes('Failed trying to load default language strings') && !msg.text().includes('Failed trying to load default language strings') &&
!msg.text().includes('TypeError: Failed to fetch') && // This happens when server actions are aborted !msg.text().includes('TypeError: Failed to fetch') && // This happens when server actions are aborted
!msg.text().includes('der-radius: 2px Server Error: Error getting do') // This is a weird error that happens in the console !msg.text().includes('der-radius: 2px Server Error: Error getting do') && // This is a weird error that happens in the console
// Conditionally ignore CORS errors based on the `ignoreCORS` option
!(
ignoreCORS &&
msg.text().includes('Access to fetch at') &&
msg.text().includes("No 'Access-Control-Allow-Origin' header is present")
) &&
// Conditionally ignore network-related errors
!msg.text().includes('Failed to load resource: net::ERR_FAILED')
) { ) {
// "Failed to fetch RSC payload for" happens seemingly randomly. There are lots of issues in the next.js repository for this. Causes e2e tests to fail and flake. Will ignore for now // "Failed to fetch RSC payload for" happens seemingly randomly. There are lots of issues in the next.js repository for this. Causes e2e tests to fail and flake. Will ignore for now
// the the server responded with a status of error happens frequently. Will ignore it for now. // the the server responded with a status of error happens frequently. Will ignore it for now.
// Most importantly, this should catch react errors. // Most importantly, this should catch react errors.
throw new Error(`Browser console error: ${msg.text()}`) throw new Error(`Browser console error: ${msg.text()}`)
} }
// Log ignored CORS-related errors for visibility
if (msg.type() === 'error' && msg.text().includes('Access to fetch at') && ignoreCORS) {
console.log(`Ignoring expected CORS-related error: ${msg.text()}`)
}
// Log ignored network-related errors for visibility
if (msg.type() === 'error' && msg.text().includes('Failed to load resource: net::ERR_FAILED')) {
console.log(`Ignoring expected network error: ${msg.text()}`)
}
}) })
} }

9
test/uploads/collections/Upload1/index.ts
View File

@@ -9,6 +9,15 @@ export const Uploads1: CollectionConfig = {
slug: 'uploads-1', slug: 'uploads-1',
upload: { upload: {
staticDir: path.resolve(dirname, 'uploads'), staticDir: path.resolve(dirname, 'uploads'),
pasteURL: {
allowList: [
{
protocol: 'http',
hostname: 'localhost',
port: '4000',
},
],
},
}, },
fields: [ fields: [
{ {

11
test/uploads/collections/Upload2/index.ts
View File

@@ -9,6 +9,17 @@ export const Uploads2: CollectionConfig = {
slug: 'uploads-2', slug: 'uploads-2',
upload: { upload: {
staticDir: path.resolve(dirname, 'uploads'), staticDir: path.resolve(dirname, 'uploads'),
pasteURL: {
allowList: [
{
protocol: 'https',
hostname: 'some-example-website.com',
pathname: '/images/*',
port: '',
search: '',
},
],
},
}, },
admin: { admin: {
enableRichTextRelationship: false, enableRichTextRelationship: false,

1
test/uploads/config.ts
View File

@@ -385,6 +385,7 @@ export default buildConfigWithDefaults({
width: 300, width: 300,
}, },
], ],
pasteURL: false,
}, },
}, },
{ {

103
test/uploads/e2e.spec.ts
View File

@@ -36,10 +36,11 @@ import {
withOnlyJPEGMetadataSlug, withOnlyJPEGMetadataSlug,
withoutMetadataSlug, withoutMetadataSlug,
} from './shared.js' } from './shared.js'
import { startMockCorsServer } from './startMockCorsServer.js'
const filename = fileURLToPath(import.meta.url) const filename = fileURLToPath(import.meta.url)
const dirname = path.dirname(filename) const dirname = path.dirname(filename)
const { beforeAll, beforeEach, describe } = test const { afterAll, beforeAll, beforeEach, describe } = test
let payload: PayloadTestSDK<Config> let payload: PayloadTestSDK<Config>
let client: RESTClient let client: RESTClient
@@ -58,9 +59,12 @@ let withoutMetadataURL: AdminUrlUtil
let withOnlyJPEGMetadataURL: AdminUrlUtil let withOnlyJPEGMetadataURL: AdminUrlUtil
let relationPreviewURL: AdminUrlUtil let relationPreviewURL: AdminUrlUtil
let customFileNameURL: AdminUrlUtil let customFileNameURL: AdminUrlUtil
let uploadsOne: AdminUrlUtil
let uploadsTwo: AdminUrlUtil
describe('Uploads', () => { describe('Uploads', () => {
let page: Page let page: Page
let mockCorsServer: ReturnType<typeof startMockCorsServer> | undefined
beforeAll(async ({ browser }, testInfo) => { beforeAll(async ({ browser }, testInfo) => {
testInfo.setTimeout(TEST_TIMEOUT_LONG) testInfo.setTimeout(TEST_TIMEOUT_LONG)
@@ -83,11 +87,13 @@ describe('Uploads', () => {
withOnlyJPEGMetadataURL = new AdminUrlUtil(serverURL, withOnlyJPEGMetadataSlug) withOnlyJPEGMetadataURL = new AdminUrlUtil(serverURL, withOnlyJPEGMetadataSlug)
relationPreviewURL = new AdminUrlUtil(serverURL, relationPreviewSlug) relationPreviewURL = new AdminUrlUtil(serverURL, relationPreviewSlug)
customFileNameURL = new AdminUrlUtil(serverURL, customFileNameMediaSlug) customFileNameURL = new AdminUrlUtil(serverURL, customFileNameMediaSlug)
uploadsOne = new AdminUrlUtil(serverURL, 'uploads-1')
uploadsTwo = new AdminUrlUtil(serverURL, 'uploads-2')
const context = await browser.newContext() const context = await browser.newContext()
page = await context.newPage() page = await context.newPage()
initPageConsoleErrorCatch(page) initPageConsoleErrorCatch(page, { ignoreCORS: true })
await ensureCompilationIsDone({ page, serverURL }) await ensureCompilationIsDone({ page, serverURL })
}) })
@@ -106,6 +112,12 @@ describe('Uploads', () => {
await ensureCompilationIsDone({ page, serverURL }) await ensureCompilationIsDone({ page, serverURL })
}) })
afterAll(() => {
if (mockCorsServer) {
mockCorsServer.close()
}
})
test('should show upload filename in upload collection list', async () => { test('should show upload filename in upload collection list', async () => {
await page.goto(mediaURL.list) await page.goto(mediaURL.list)
const audioUpload = page.locator('tr.row-1 .cell-filename') const audioUpload = page.locator('tr.row-1 .cell-filename')
@@ -174,6 +186,16 @@ describe('Uploads', () => {
await saveDocAndAssert(page) await saveDocAndAssert(page)
}) })
test('should remove remote URL button if pasteURL is false', async () => {
// pasteURL option is set to false in the media collection
await page.goto(mediaURL.create)
const pasteURLButton = page.locator('.file-field__upload button', {
hasText: 'Paste URL',
})
await expect(pasteURLButton).toBeHidden()
})
test('should properly create IOS file upload', async () => { test('should properly create IOS file upload', async () => {
await page.goto(mediaURL.create) await page.goto(mediaURL.create)
@@ -650,6 +672,83 @@ describe('Uploads', () => {
expect(webpMediaDoc.sizes.sizeThree.filesize).toEqual(211638) expect(webpMediaDoc.sizes.sizeThree.filesize).toEqual(211638)
}) })
describe('remote url fetching', () => {
beforeAll(async () => {
mockCorsServer = startMockCorsServer()
})
afterAll(() => {
if (mockCorsServer) {
mockCorsServer.close()
}
})
test('should fetch remote URL server-side if pasteURL.allowList is defined', async () => {
// Navigate to the upload creation page
await page.goto(uploadsOne.create)
// Click the "Paste URL" button
const pasteURLButton = page.locator('.file-field__upload button', { hasText: 'Paste URL' })
await pasteURLButton.click()
// Input the remote URL
const remoteImage = 'http://localhost:4000/mock-cors-image'
const inputField = page.locator('.file-field__upload .file-field__remote-file')
await inputField.fill(remoteImage)
// Intercept the server-side fetch to the paste-url endpoint
const encodedImageURL = encodeURIComponent(remoteImage)
const pasteUrlEndpoint = `/api/uploads-1/paste-url?src=${encodedImageURL}`
const serverSideFetchPromise = page.waitForResponse(
(response) => response.url().includes(pasteUrlEndpoint) && response.status() === 200,
{ timeout: 1000 },
)
// Click the "Add File" button
const addFileButton = page.locator('.file-field__add-file')
await addFileButton.click()
// Wait for the server-side fetch to complete
const serverSideFetch = await serverSideFetchPromise
// Assert that the server-side fetch completed successfully
await serverSideFetch.text()
// Wait for the filename field to be updated
const filenameInput = page.locator('.file-field .file-field__filename')
await expect(filenameInput).toHaveValue('mock-cors-image', { timeout: 500 })
// Save and assert the document
await saveDocAndAssert(page)
// Validate the uploaded image
const imageDetails = page.locator('.file-field .file-details img')
await expect(imageDetails).toHaveAttribute('src', /mock-cors-image/, { timeout: 500 })
})
test('should fail to fetch remote URL server-side if the pasteURL.allowList domains do not match', async () => {
// Navigate to the upload creation page
await page.goto(uploadsTwo.create)
// Click the "Paste URL" button
const pasteURLButton = page.locator('.file-field__upload button', { hasText: 'Paste URL' })
await pasteURLButton.click()
// Input the remote URL
const remoteImage = 'http://localhost:4000/mock-cors-image'
const inputField = page.locator('.file-field__upload .file-field__remote-file')
await inputField.fill(remoteImage)
// Click the "Add File" button
const addFileButton = page.locator('.file-field__add-file')
await addFileButton.click()
// Verify the toast error appears with the correct message
await expect(page.locator('.payload-toast-container .toast-error')).toContainText(
'The provided URL is not allowed.',
)
})
})
describe('image manipulation', () => { describe('image manipulation', () => {
test('should crop image correctly', async () => { test('should crop image correctly', async () => {
const positions = { const positions = {

22
test/uploads/startMockCorsServer.ts Normal file
View File

@@ -0,0 +1,22 @@
import fs from 'fs'
import http from 'http'
import path from 'path'
import { fileURLToPath } from 'url'
const filename = fileURLToPath(import.meta.url)
const dirname = path.dirname(filename)
export const startMockCorsServer = () => {
const server = http.createServer((req, res) => {
const filePath = path.resolve(dirname, 'test-image.jpg')
res.setHeader('Content-Type', 'image/jpeg')
fs.createReadStream(filePath).pipe(res)
})
server.listen(4000, () => {
console.log('Mock CORS server running on http://localhost:4000')
})
return server
}