Adds pre-signed URLs support file downloads with the S3 adapter. Can be
enabled per-collection:
```ts
s3Storage({
collections: {
media: { signedDownloads: true }, // or { signedDownloads: { expiresIn: 3600 }} for custom expiresIn (default 7200)
},
bucket: process.env.S3_BUCKET,
config: {
credentials: {
accessKeyId: process.env.S3_ACCESS_KEY_ID,
secretAccessKey: process.env.S3_SECRET_ACCESS_KEY,
},
endpoint: process.env.S3_ENDPOINT,
forcePathStyle: process.env.S3_FORCE_PATH_STYLE === 'true',
region: process.env.S3_REGION,
},
}),
```
The main use case is when you care about the Payload access control (so
you don't want to use `disablePayloadAccessControl: true` but you don't
want your files to be served through Payload (which can affect
performance with large videos for example).
This feature instead generates a signed URL (after verifying the access
control) and redirects you directly to the S3 provider.
This is an addition to https://github.com/payloadcms/payload/pull/11382
which added pre-signed URLs for file uploads.
Ensures all s3 sockets are cleaned up. Now passes through default
request handler options that `@smithy/node-http-handler` now handles
properly.
Fixes#6382
```ts
const defaultRequestHandlerOpts: NodeHttpHandlerOptions = {
httpAgent: {
keepAlive: true,
maxSockets: 100,
},
httpsAgent: {
keepAlive: true,
maxSockets: 100,
},
}
```
If you continue to have socket issues, you can customize any of the
options by setting `requestHandler` property on your s3 config. This
will take precedence if set.
```ts
requestHandler: {
httpAgent: {
maxSockets: 300,
keepAlive: true,
},
httpsAgent: {
maxSockets: 300,
keepAlive: true,
},
// Optional, only set these if you continue to see issues. Be wary of timeouts if you're dealing with large files.
// time limit (ms) for receiving response.
requestTimeout: 5_000,
// time limit (ms) for establishing connection.
connectionTimeout: 5_000,
}),
```
Fixes https://github.com/payloadcms/payload/issues/11473
Previously, when `disablePayloadAccessControl: true` was defined, client
uploads were working improperly. The reason is that
`addDataAndFileToRequest` expects `staticHandler` to be defined and we
don't add in case if `disablePayloadAccessControl: true`.
This PR makes it so otherwise and if we have `clientUploads`, it pushes
the "proxied" handler that responses only when the file was requested in
the context of client upload (from `addDataAndFileToRequest`)
### What?
Fixes client uploads when storage collection config has the `prefix`
property configured. Previously, it failed with "Object key was not
found".
### Why?
This is expected to work.
### How?
The client upload handler now receives to its props `prefix`. Then it
threads it to the server-side `staticHandler` through
`clientUploadContext` and then to `getFilePrefix`, which checks for
`clientUploadContext.prefix` and returns if there is.
Previously, `staticHandler` tried to load the file without including
prefix consideration.
This changes only these adapters:
* S3
* Azure
* GCS
With the Vercel Blob adapter, `prefix` works correctly.
Ensures that even if you pass `enabled: false` to the storage adapter
options, e.g:
```ts
s3Storage({
enabled: false,
collections: {
[mediaSlug]: true,
},
bucket: process.env.S3_BUCKET,
config: {
credentials: {
accessKeyId: process.env.S3_ACCESS_KEY_ID,
secretAccessKey: process.env.S3_SECRET_ACCESS_KEY,
},
},
})
```
the client handler component is added to the import map. This prevents
errors when you use the adapter only on production, but you don't
regenerate the import map before running the build
### What?
Within collections using the `storage-s3` plugins, we eventually start
receiving the following warnings:
`@smithy/node-http-handler:WARN socket usage at capacity=50 and 156
additional requests are enqueued. See
https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html
or increase socketAcquisitionWarningTimeout=(millis) in the
NodeHttpHandler config.`
Also referenced in this issue: #6382
The
[solution](https://github.com/payloadcms/payload/issues/6382#issuecomment-2325468104)
provided by @denolfe in that issue only delayed the reappearance of the
problem somewhat, but did not resolve it.
### Why?
As far as I understand, in the `staticHandler` of the plugin, when
getting items from storage, and they are currently cached, the cached
results are immediately returned without handling the stream. As per
[this](https://github.com/aws/aws-sdk-js-v3/blob/main/supplemental-docs/CLIENTS.md#nodejs-requesthandler)
entry in the aws-sdk docs, if the streaming response is not read, or
manually destroyed, a socket might not properly close.
### How?
Before returning the cached items, manually destroy the streaming
response to make certain the socket is being properly closed.
Additionally, add an error check to also consume/destroy the streaming
response in case an error occurs, to not leave orphaned sockets.
Fixes#6382
Previously we had been downgrading rimraf to v3 simply to handle clean
with glob patterns across platforms. In v4 and newer of rimraf you can
add `-g` to use glob patterns.
This change updates rimraf and adds the flag to handle globs in our
package scripts to be windows compatible.
This PR makes changes to every storage adapter in order to add
browser-based caching by returning etags, then checking for them into
incoming requests and responding a status code of `304` so the data
doesn't have to be returned again.
Performance improvements for cached subsequent requests:
This respects `disableCache` in the dev tools.
Also fixes a bug with getting the latest image when using the Vercel
Blob Storage adapter.
Should fix messed up import suggestions and simplifies all tsconfigs
through inheritance.
One main issue was that packages were inheriting `baseURL: "."` from the
root tsconfig. This caused incorrect import suggestions that start with
"packages/...".
This PR ensures that packages do not inherit this baseURL: "." property,
while ensuring the root, non-inherited tsconfig still keeps it to get
tests to work (the importMap needs it)
