Commit Graph

21 Commits

Author SHA1 Message Date
Sasha
11d74871ef feat(db-postgres): add vector raw column type (#10422)
Example how you can add a vector column, enable the `vector` extension
and query your embeddings in the included test -

https://github.com/payloadcms/payload/compare/feat/more-types?expand=1#diff-7d876370487cb625eb42ff1ad7cffa78e8327367af3de2930837ed123f5e3ae6R1-R117
2025-03-17 20:50:00 +00:00
Sasha
727fba7b1c refactor: deduplicate and abstract SQL schema building (#9987)
### What?
Abstracts SQL schema building, significantly reducing code duplication
for SQLite / Postgres

db-sqlite lines count From:
```sh
 wc -l **/*.ts
      62 src/connect.ts
      32 src/countDistinct.ts
       9 src/createJSONQuery/convertPathToJSONTraversal.ts
      86 src/createJSONQuery/index.ts
      15 src/defaultSnapshot.ts
       6 src/deleteWhere.ts
      21 src/dropDatabase.ts
      15 src/execute.ts
     178 src/index.ts
     139 src/init.ts
      19 src/insert.ts
      19 src/requireDrizzleKit.ts
     544 src/schema/build.ts
      27 src/schema/createIndex.ts
      38 src/schema/getIDColumn.ts
      13 src/schema/idToUUID.ts
      28 src/schema/setColumnID.ts
     787 src/schema/traverseFields.ts
      18 src/schema/withDefault.ts
     248 src/types.ts
    2304 total
```

To:
```sh
wc -l **/*.ts
      62 src/connect.ts
      32 src/countDistinct.ts
       9 src/createJSONQuery/convertPathToJSONTraversal.ts
      86 src/createJSONQuery/index.ts
      15 src/defaultSnapshot.ts
       6 src/deleteWhere.ts
      21 src/dropDatabase.ts
      15 src/execute.ts
     180 src/index.ts
      39 src/init.ts
      19 src/insert.ts
      19 src/requireDrizzleKit.ts
     149 src/schema/buildDrizzleTable.ts
      32 src/schema/setColumnID.ts
     258 src/types.ts
     942 total
```

Builds abstract schema in shared drizzle package that later gets
converted by SQLite/ Postgres specific implementation into drizzle
schema.
This, apparently can also help here
https://github.com/payloadcms/payload/pull/9953. It should be very
trivial to implement a MySQL adapter with this as well.
2024-12-16 17:17:18 +02:00
Sasha
cae300e8e3 perf: flattenedFields collection/global property, remove deep copying in validateQueryPaths (#9299)
### What?
Improves querying performance of the Local API, reduces copying overhead

### How?

Adds `flattenedFields` property for sanitized collection/global config
which contains fields in database schema structure.
For example, It removes rows / collapsible / unnamed tabs and merges
them to the parent `fields`, ignores UI fields, named tabs are added as
`type: 'tab'`.
This simplifies code in places like Drizzle `transform/traverseFields`.
Also, now we can avoid calling `flattenTopLevelFields` which adds some
overhead and use `collection.flattenedFields` / `field.flattenedFields`.

By refactoring `configToJSONSchema.ts` with `flattenedFields` it also
1. Fixes https://github.com/payloadcms/payload/issues/9467
2. Fixes types for UI fields were generated for `select`



Removes this deep copying for each `where` query constraint
58ac784425/packages/payload/src/database/queryValidation/validateQueryPaths.ts (L69-L73)
which potentially can add overhead if you have a large collection/global
config

UPD:
The overhead is even much more than in the benchmark below if you have
Lexical with blocks.

Benchmark in `relationships/int.spec.ts`:
```ts
const now = Date.now()
for (let i = 0; i < 10; i++) {
  const now = Date.now()
  for (let i = 0; i < 300; i++) {
    const query = await payload.find({
      collection: 'chained',
      where: {
        'relation.relation.name': {
          equals: 'third',
        },
        and: [
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
        ],
      },
    })
  }

  payload.logger.info(`#${i + 1} ${Date.now() - now}`)
}
payload.logger.info(`Total ${Date.now() - now}`)
```

Before:
```
[02:11:48] INFO: #1 3682
[02:11:50] INFO: #2 2199
[02:11:54] INFO: #3 3483
[02:11:56] INFO: #4 2516
[02:11:59] INFO: #5 2467
[02:12:01] INFO: #6 1987
[02:12:03] INFO: #7 1986
[02:12:05] INFO: #8 2375
[02:12:07] INFO: #9 2040
[02:12:09] INFO: #10 1920
    [PASS] Relationships > Querying > Nested Querying > should allow querying two levels deep (24667ms)
[02:12:09] INFO: Total 24657
```

After:
```
[02:12:36] INFO: #1 2113
[02:12:38] INFO: #2 1854
[02:12:40] INFO: #3 1700
[02:12:42] INFO: #4 1797
[02:12:44] INFO: #5 2121
[02:12:46] INFO: #6 1774
[02:12:47] INFO: #7 1670
[02:12:49] INFO: #8 1610
[02:12:50] INFO: #9 1596
[02:12:52] INFO: #10 1576
    [PASS] Relationships > Querying > Nested Querying > should allow querying two levels deep (17828ms)
[02:12:52] INFO: Total 17818
```
2024-11-25 10:28:07 -05:00
Alessio Gravili
03291472d6 chore: bump all eslint dependencies, run lint and prettier (#9128)
This fixes a peer dependency error in our monorepo, as
eslint-plugin-jsx-a11y finally supports eslint v9.

Additionally, this officially adds TypeScript 5.6 support for
typescript-eslint.
2024-11-12 10:18:22 -05:00
Sasha
0a15388edb feat(db-postgres): add point field support (#9078)
### What?
Adds full support for the point field to Postgres and Vercel Postgres
adapters through the Postgis extension. Fully the same API as with
MongoDB, including support for `near`, `within` and `intersects`
operators.

Additionally, exposes to adapter args:
*
`tablesFilter`https://orm.drizzle.team/docs/drizzle-kit-push#including-tables-schemas-and-extensions.
* `extensions` list of extensions to create, for example `['vector',
'pg_search']`, `postgis` is created automatically if there's any point
field

### Why?
It's essential to support that field type, especially if the postgres
adapter should be out of beta on 3.0 stable.

### How?
* Bumps `drizzle-orm` to `0.36.1` and `drizzle-kit` to `0.28.0` as we
need this change https://github.com/drizzle-team/drizzle-orm/pull/3141
* Uses its functions to achieve querying functionality, for example the
`near` operator works through `ST_DWithin` or `intersects` through
`ST_Intersects`.
* Removes MongoDB condition from all point field tests, but keeps for
SQLite

Resolves these discussions:
https://github.com/payloadcms/payload/discussions/8996
https://github.com/payloadcms/payload/discussions/8644
2024-11-11 09:31:47 -05:00
Sasha
8af00f2deb fix: join field works on collections with versions enabled (#8715)
- Fixes errors with drizzle when building the schema
https://github.com/payloadcms/payload/issues/8680
- Adds `joins` to `db.queryDrafts` to have them when doing `.find` with
`draft: true`
2024-10-22 11:05:55 -04:00
Sasha
90bca15f52 fix(drizzle): enforce uniqueness on index names (#8754)
Fixes https://github.com/payloadcms/payload/issues/8752

Previously, trying to define a config like this:
```ts
{
  type: 'text',
  name: 'someText',
  index: true,
},
{
  type: 'array',
  name: 'some',
  index: true,
  fields: [
    {
      type: 'text',
      name: 'text',
      index: true,
    },
  ],
}
```

Lead to the error:
```
Warning  We've found duplicated index name across public schema. Please rename your index in either the demonstration table or the table with the duplicated index name
```

Now, if we encounter duplicates, we increment the name like this:
`collection_some_text_idx`
`collection_some_text_1_idx`

---------

Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
2024-10-17 02:05:27 +00:00
Sasha
1ffb6c3e13 fix(drizzle): indexes / unique with relationships (#8432)
Fixes https://github.com/payloadcms/payload/issues/8413 and
https://github.com/payloadcms/payload/issues/6460

- Builds indexes for relationships by default in the SQL schema
- Fixes `unique: true` handling with Postgres / SQLite for every type of
relationships (non-polymorphic. hasMany, polymorphic, polymorphic
hasMany) _note_: disables unique for nested to arrays / blocks
relationships in the `_rels` table.
- adds tests

2.0 PR tha ports only indexes creation
https://github.com/payloadcms/payload/pull/8446, because `unique: true`
could be a breaking change if someone has incosistent unique data in the
database.
2024-10-11 13:13:54 -04:00
Dan Ribbens
1bf580fac3 feat: join field works with hasMany relationships (#8493)
Join field works on relationships and uploads having `hasMany: true`

---------

Co-authored-by: Sasha <64744993+r1tsuu@users.noreply.github.com>
2024-10-08 11:40:34 -04:00
Sasha
0128eedf70 fix(drizzle)!: make radio and select column names to snake_case (#8439)
Fixes https://github.com/payloadcms/payload/issues/8402 and
https://github.com/payloadcms/payload/issues/8027

Before DB column names were camelCase:

![image](https://github.com/user-attachments/assets/d2965bcf-290a-4f86-9bf4-dfe7e8613934)

After this change, they are snake_case:
![Screenshot 2024-10-04
114226](https://github.com/user-attachments/assets/bbc8c20b-6745-4dd3-b0c8-56263a4e37b1)

#### Breaking SQLite / Postgres ⚠️  
If you had any select (not `hasMany: true`) or radio fields with the
name in camelCase, for example:
```ts
{
  name: 'selectReadOnly',
  type: 'select',
  admin: {
    readOnly: true,
  },
  options: [
    {
      label: 'Value One',
      value: 'one',
    },
    {
      label: 'Value Two',
      value: 'two',
    },
  ],
},
```
This previously was mapped to the db column name `"selectReadOnly"`. Now
it's `select_read_only`.
Generate a new migration to rename your columns.
```sh
pnpm payload migrate:create
```
Then select "rename column" for targeted columns and Drizzle will handle
the migration.

---------

Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
2024-10-04 16:25:05 +00:00
Dan Ribbens
27b1629927 fix(db-postgres, db-sqlite): joins relation name (#8491)
A join field on a relationship with a camelCase name would cause an
error from drizzle due to the relation name not matching.
2024-09-30 16:47:21 -04:00
Sasha
613d3b090e fix(drizzle): hasMany / poly relationships nested to localized fields / nested blocks to localized fields (#8456)
fixes https://github.com/payloadcms/payload/issues/8455 and
https://github.com/payloadcms/payload/issues/8462

- Builds the `_locale` column for the `_rels` when it's inside of a
localized group / tab
- Properly builds `sanitizedPath` for blocks in the transform-read
function when it's inside of a localized field. This fixes with fields
inside that have its own table (like `_rels`, select `hasMany: true`
etc)

Adds _more_ tests!
2024-09-28 17:33:50 +03:00
Sasha
338c93a229 fix(drizzle): array/relationship/select hasMany in localized field (#8355)
This PR addresses these issues with localized groups / tabs with
Postgres / SQLite:

- Array fields inside of localized groups. Fixes
https://github.com/payloadcms/payload/issues/8322
- Select fields with `hasMany: true` inside of localized groups. Related
to 1, but still needed its own additional logic.
- Relationship (non-polymorphic / non has-many) inside of localized
groups. Previously, even just trying to define them in the config led to
a crash. Fixes https://github.com/payloadcms/payload/issues/8308

Ensures test coverage for localized groups.
2024-09-23 11:34:02 -04:00
Dan Ribbens
6ef2bdea15 feat!: join field (#7518)
## Description

- Adds a new "join" field type to Payload and is supported by all database adapters
- The UI uses a table view for the new field
- `db-mongodb` changes relationships to be stored as ObjectIDs instead of strings (for now querying works using both types internally to the DB so no data migration should be necessary unless you're querying directly, see breaking changes for details
- Adds a reusable traverseFields utility to Payload to make it easier to work with nested fields, used internally and for plugin maintainers

```ts
export const Categories: CollectionConfig = {
    slug: 'categories',
    fields: [
        {
            name: 'relatedPosts',
            type: 'join',
            collection: 'posts',
            on: 'category',
        }
    ]
}
```

BREAKING CHANGES:
All mongodb relationship and upload values will be stored as MongoDB ObjectIDs instead of strings going forward. If you have existing data and you are querying data directly, outside of Payload's APIs, you get different results. For example, a `contains` query will no longer works given a partial ID of a relationship since the ObjectID requires the whole identifier to work. 

---------

Co-authored-by: Jacob Fletcher <jacobsfletch@gmail.com>
Co-authored-by: James <james@trbl.design>
2024-09-20 11:10:16 -04:00
Sasha
bf48af411d feat: add virtual property to the fields config (#7621)
## Description

Adds `virtual` property to the fields config. Providing `true`
completely disables the field in the DB, which is useful for [Virtual
Fields](https://payloadcms.com/blog/learn-how-virtual-fields-can-help-solve-common-cms-challenges)
Disables abillity to query by a field with `virtual: true`.
Currently, they bloat the DB with unused tables / columns, which may as
well introduce additional joins.
Discussion https://github.com/payloadcms/payload/discussions/6270
Prev PR (this one contains only this feature):
https://github.com/payloadcms/payload/pull/6983

- [x] I have read and understand the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository.

## Type of change

<!-- Please delete options that are not relevant. -->

- [x] New feature (non-breaking change which adds functionality)
- [x] This change requires a documentation update

## Checklist:

- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] Existing test suite passes locally with my changes
- [x] I have made corresponding changes to the documentation
2024-09-17 10:40:54 -04:00
Sasha
b7db53cef4 fix(drizzle)!: localized fields uniqueness per locale (#8230)
Previously, this worked with MongoDB but failed with Postgres / SQLite
when the `slug` field has both `localized: true` and `unique: true`.

```ts
await payload.create({
  collection: "posts",
  locale: "en",
  data: {
    slug: "my-post"
  }
})

await payload.create({
  collection: "posts",
  locale: "de",
  data: {
    slug: "my-post"
  }
})
```

Now, we build unique constraints and indexes in combination with the
_locale column. This should also improve query performance for fields
with both index: true and localized: true.

### Migration steps (Postgres/SQLite only)
This change updates the database schema and requires a migration (if you
have any localized fields). To apply it, run the following commands:

```sh
pnpm payload migration:create locale_unique_indexes
pnpm payload migrate
```

Note that if you use `db.push: true` which is a default, you don't have
to run `pnpm payload migrate` in the development mode, only in the
production, as Payload automatically pushes the schema to your DB with
it.
2024-09-16 14:47:13 -04:00
James Mikrut
5873a3db06 fix: duplicating localized nested arrays (#8220)
Fixes an issue where duplicating documents in Postgres / SQLite would
crash because of a foreign key constraint / unique ID issue when you
have nested arrays / blocks within localized arrays / blocks.

We now run `beforeDuplicate` against all locales prior to
`beforeValidate` and `beforeChange` hooks.

This PR also fixes a series of issues in Postgres / SQLite where you
have localized groups / named tabs, and then arrays / blocks within the
localized groups / named tabs.
2024-09-15 02:51:31 +00:00
Alessio Gravili
7fd736ea5b chore: lint entire codebase including db packages 2024-09-03 11:19:19 -04:00
James Mikrut
18b0806b5b fix(db-postgres, db-sqlite): hasMany text, number, poly relationship, blocks, arrays within localized fields (#7900)
## Description

In Postgres, localized blocks or arrays that contain other array / block
/ relationship fields were not properly storing locales in the database.

Now they are! Need to check a few things yet:

- Ensure test coverage is sufficient
- Test localized array, with non-localized array inside of it
- Test localized block with relationship field within it
- Ensure `_rels` table gets the `locale` column added if a single
non-localized relationship exists within a localized array / block

Fixes step 6 as identified in #7805
2024-08-28 17:43:12 +00:00
Paul
d2571e10d6 feat: upload hasmany (#7796)
Supports `hasMany` upload fields, similar to how `hasMany` works in
other fields, i.e.:

```ts
{
  type: 'upload',
  relationTo: 'media',
  hasMany: true
}
```

---------

Co-authored-by: Jacob Fletcher <jacobsfletch@gmail.com>
Co-authored-by: James <james@trbl.design>
2024-08-21 20:44:04 -04:00
Elliot DeNolf
2835e1d709 feat: abstract postgres base adapter (#7732)
Abstracts Postgres base adapter in order to allow future postgres-based
adapters.
2024-08-16 18:51:39 -04:00