The Good, The Bad, The Ugly - State of Dgraph

Talking with @gajanan (or @gajanansc) earlier today, He requested, I post a discuss topic summarizing a bunch of links for feature requests, current problems, ideas, suggestions, etc. The goal here is to be factual and technical. Oh and for those who may have not known before, @verneleem is also me :wink:

If I have to do all of the raw data filtering, manipulation, and analytics in a client outside of the database, then why have the database at all?

^ Sometimes it seems right now that we have 2 API languages and 0 DB languages with Dgraph.

If you haven’t yet, please see The State of Dgraph’s GraphQL API Notion Document. I will reference many of the same topics from this document over again here to try to get a complete overview all in one place linking to many different subjects.

  • Missing @auth rule for post update state
  • Field level authorization, not the same as ACL predicate control in DQL Enterprise
  • External node/type based auth rules
  • Scalar Validation/Constraints (possible solution pre-hooks in GraphQL API)
  • Edge/Relationship Validation/Constraints in DQL
  • Separating/Combining Interface and Implementing Type @auth rules

Auth rules on interfaces cascade to the implementing type auth rules and get combined that implementing types must match ALL rules of the type and the interface, but sometimes that is not wanted, but rather the need is for the rules to be combined with OR logic instead.

  • Separating @auth outside of the GraphQL schema itself
  • Re-usable/Global @auth rules
  • Scoping/Cascading @auth rules
  • Hard limiting results in GraphQL API — Prevent Data Scraping
  • [Completed?] Combining @auth and @custom DQL resolvers
  • Nested Filtering for DQL and GraphQL API (without using @cascade directive)

    AKA: Dealing with more normalized forms of data

  • Paginating child nodes as a whole irrelevant to their multiple parental levels (related to nested filtering)
  • Logically combining filters together from different levels in the graph (related to nested filtering)
  • Ordering/Sorting by nested data
  • Filter by Aggregated results
  • Scalar comparisons
  • String pattern matching (not full regexp)
  • Date/time filtering and manipulation
  • Order by enums in GraphQL API
  • Sorting by Aggregation
  • Calculated Fields/Triggers
  • Full Text Search Best Match Scoring
  • Full text search across multiple fields:
  • String Functions for inter graph comparisons and manipulation
  • Simplifying/Enhancing groupby
  • Verbosity of Multiple Node Updates in GraphQL
  • Deep Mutations

https://dgraph.io/docs/graphql/mutations/deep/

https://discuss.dgraph.io/t/data-doesnt-update/11489

https://github.com/hasura/graphql-engine/issues/1573

  • Comparative Inputs

https://discuss.dgraph.io/t/atomic-field-operations/14128/3?u=amaster507

  • Auto Incrementing Fields

https://discuss.dgraph.io/t/feature-request-directive-that-indicates-that-id-fields-should-be-auto-populated-on-creation/14992

https://discuss.dgraph.io/t/atomic-field-operations/14128?u=amaster507

  • Need to correct generated payload list nullability in GraphQL API

Payloads right now in Dgraph are generated as nullable items in a list. “queryUser: [User]” But this should be corrected to the tightest possible type such as “queryUser: [User!]!” which means that the result will be an array, it could be an empty array, but no items in the array can or ever will be null.

  • No Arrays—only Sets/Lists

https://discuss.dgraph.io/t/documentation-that-arrays-work-as-a-set-not-storing-duplicates/9590?u=amaster507

But what needs thought out even more so is what this change would bring with it in terms of API capability

  • Add/Update values at positions in a list chosing to replace or skoot over existing values that may exist.
  • Replace a list in its entirety. This is rather difficult to do but is such a simple use case
  • Delete values from a specific index in a list. Right now you can only delete all items in a list or items in a list by their value.
  • Allow lists to maintain order (for lists of scalars and lists of type (aka edges))
  • Move item(s) in a list to a specific location without changing any values
  • Unions sometimes produce unexpected results

https://discuss.dgraph.io/t/using-unions-and-interfaces-and-end-up-with-non-nullable-field-not-present/15680?u=amaster507

https://discuss.dgraph.io/t/dgraph-directive-does-not-work-as-expected-with-union-types/13021

  • @auth on union types

Completed? Needs to be documented or implemented if not currently—haven’t tested personally

https://discuss.dgraph.io/t/union-types-in-graphql/9581?u=amaster507

  • Custom Directives

The idea of custom directives is centered around directives on the developer creating directives available to clients. A developer may wish to allow some kind of direct script being processed on command such as logging the result or adding some metadata to the response (not data, but in the extensions response)

https://discuss.dgraph.io/t/custom-field-concatenation-without-needing-external-script/7283?u=amaster507

https://discuss.dgraph.io/t/can-i-get-a-substring-in-a-query/10522/3?u=amaster507

  • Custom DQL Mutations

This really depends on how much is being refactored in the GraphQL codebase and how that refactoring is done. If GraphQL will still be rewritten into DQL then it is equally important to support not only DQL in custom queries, but also DQL in custom mutations.

https://discuss.dgraph.io/t/why-does-custom-dql-not-support-mutation/10366?u=amaster507

  • Custom DQL Fields

Custom fields can currently be resolved with lambda, but it would be beneficial (again depending on how the refactor is done) to allow custom fields to also be resolved with DQL.

https://discuss.dgraph.io/t/can-i-use-dql-to-calculate-the-value-of-a-custom-field/15877?u=amaster507

  • Additional and Custom Scalars

Developers often finding themselves needing to add custom scalars for various reasons. These can often be represented as strings but with additional constraints such as Email, HexColor, Tuple

https://discuss.dgraph.io/t/json-blob-as-a-scalar/11034?u=amaster507

https://discuss.dgraph.io/t/can-we-have-remote-scalars/11726?u=amaster507

  • Authentication Service

Dgraph built an authentication system for Dgraph Cloud and was discussing open sourcing it. I believe something like that should be made and integrated into the GraphQL API so that users can easily authenticate against their own data and maybe use lambdas to return the claims from the database that the developer wants to use when a client authenticates.

https://discuss.dgraph.io/t/open-sourcing-in-house-dgraph-auth/13398?u=amaster507

  • Open Source all Enterprise Features
  • Auditing
  • Namespacing
  • [Completed in 21.12?] Backups

Should these really be enterprise or are these just enterprise level to force users into the Cloud? Are Enterprise licenses even available anymore? (They were not [or ridiculously purposefully priced outside of the budget to whom it was being quoted] under the last administration)

https://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/89?u=amaster507

  • Schema Migration Tools

There is a need for when users migrate their schema, they expect the data to follow. This is not done in Dgraph now.

https://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/34?u=amaster507

https://discuss.dgraph.io/t/migrating-renaming-predicates-etc/5239?u=amaster507

https://www.edgedb.com/docs/guides/migrations/index

  • Facets are not first class citizens

https://discuss.dgraph.io/t/improve-facets-with-upsert-block/5640

If nested filtering and linking-nodes are not able to be constrained, then we can’t get rid of facets. But even then, maybe we can take the concept of _linkingnodes and make them work without needing to declare the type in the middle, like how prisma creates pivot tables without you needing to specifically create them and in the prisma ORM it lets you link directly through the pivot table like as if it was a 1:1 relationship. For Reference: EdgeDB has what it terms “link properties” and abstracts these onto types.

https://www.edgedb.com/docs/guides/link_properties

  • Mapping GraphQL’s @hasInverse vs DQL’s @reverse

Right now Dgraph GraphQL API, uses the @hasInverse directive to “map” inverse relationships and then the API keeps these pairs of edges balanced with mutation. This creates additional work for adding RDF data with live/bulk loader to add two edges for every inverse relationship.

It might be better if Dgraph would just allow the mapping of the ~ reverse edges.

  • Var Blocks in GraphQL

https://discuss.dgraph.io/t/directive-idea-to-support-var-blocks-in-graphql/14419?u=amaster507

  • Fuzzy Full Text Search

The ability to use TRIGRAMS on phrases and not just words. You cannot search for a fuzzy phrase, only a fuzzy word. This limits full-text search.

https://discuss.dgraph.io/t/fuzzy-full-text-search/16224?u=amaster507

  • Count Words

The ability to count the number of times word appears in a text and sort by that value. This would make it possible to write relevant search algorithms.

https://discuss.dgraph.io/t/how-to-implement-keyword-based-relevance-sorting/16174?u=amaster507

  • DQL Loops

very much needed to simplify algorithms without stepping in and out of queries/mutations with a client.

https://discuss.dgraph.io/t/foreach-func-in-dql-loops-in-bulk-upsert/5533/6

  • More educational material
  • Some docs that allow us to get into the DGraph code easier, so we can contribute
  • Transparent roadmap, open issues and bugs, so we don’t get surprised by missing features or minor bugs

https://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/3?u=amaster507

  • Communicate with your users and get us involved! One post from [the Dgraph Labs] a week giving simple product updates and plans can go a very long way.

  • Provide a generous free tier.

  • An out of the box local (offline) development experience (aka without me having to learn / do much)

  • A way to batch mutations so I can roll back a group of changes.

  • Be able to simply replace a list in a mutation.

https://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/9?u=amaster507

  • Removing Edges using null does not work

https://discuss.dgraph.io/t/removing-edges-using-null-does-not-work/15420

  • Directly integrate other tools into Dgraph like https://magic.link , Auth0, and make it configurable with a few clicks
  • Detailed and accurate GraphQL errors
  • Focus really hard on making it as easy as possible for devs to build side-projects and hobby-projects free/cheap Dgraph instances

Dgraph has all the pieces in place to build the ultimate low-code tool. The simpler you can make it for users, the more users you’re gonna get.

https://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/10?u=amaster507

  • No sharding of predicates
  • No Query Planner (CTO’s current vision)
  • Upsert by xid painful for ingest-heavy workloads
  • Missing a native [time]range type

https://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/14?u=amaster507

  • The upgrade process requires downtime.

https://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/17?u=amaster507

https://discuss.dgraph.io/t/improve-dgraph-upgrade-experience-by-supporting-in-place-rolling-upgrades/8821

  • It would be nice if we have one click hosting on platfroms like digital ocean for open source

https://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/19?u=amaster507

  • Lack of native timestamps.

https://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/21?u=amaster507

  • have is a multi-RAFT approach for regional clusters like CockroachDB is doing

https://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/22?u=amaster507

  • it’s own mobile solution with offline sync

https://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/23?u=amaster507

https://discuss.dgraph.io/t/call-for-collaboration-designing-a-dgraph-offline-first-library/9293/11

  • BM25 and or custom search ranking
  • More and custom tokenizers

https://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/25?u=amaster507

  • Lack of support for some algorithms
  • many problems in path lookup

https://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/26?u=amaster507

  • sub-select statements in graphql
  • More in cloud editor help & messages for when certain changes will orphan data or cause negative side effects

https://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/34?u=amaster507

  • I don’t have time to write my own middleware, and I don’t want to host anything myself to deal with servers
  • lambdas are time consuming and should only be reserved for complex tasks

https://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/40?u=amaster507

  • Cannot Paste with Ctrl+V on Cloud UI

https://discuss.dgraph.io/t/cannot-paste-with-ctrl-v-on-cloud-ui/15195?u=amaster507

  • Separarate GraphQL from Dgraph [as a Plugin?]

https://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/70?u=amaster507

https://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/71?u=amaster507

  • Typescript binding for dql client as well so responses are typed?

Reference: EdgeQL TS Client Achieved this! https://www.edgedb.com/docs/clients/js/index

  • Fix Dgraph’s type system so that the type is not a string value/predicate

https://discuss.dgraph.io/t/how-does-type-system-work/17616?u=amaster507

https://discuss.dgraph.io/t/func-type-error-in-20-11-03-or-later/16237/29?u=amaster507

https://discuss.dgraph.io/t/ask-dgraph-founder-anything/16352/24?u=amaster507

https://discuss.dgraph.io/t/ask-dgraph-founder-anything/16352/28?u=amaster507

  • An official dgraph toolset for data and schema versioning that is manageable within our codebases.

https://discuss.dgraph.io/t/ask-dgraph-founder-anything/16352/31?u=amaster507

  • Math Functions

https://neo4j.com/docs/cypher-manual/current/functions/mathematical-numeric/

  • Subscriptions based on CDC

https://dgraph.io/docs/master/enterprise-features/change-data-capture/

  • Subscriptions should use graph-ws

https://github.com/enisdenjo/graphql-ws/blob/master/PROTOCOL.md

  • [Completed with Learner Nodes?] Add Geo Replication

https://discuss.dgraph.io/t/geographically-distributed-datacenter-replication/13311

  • Cloud Requested Fixes/Improvements from @jdgamble555

    • Add the storage [for file/media uploads management possibly connected to S3]
    • Make data studio have CRUD functionality
    • Allow renaming types in the UI
    • Build an Auth System
  • The problem with Lambdas [A MUST READ!]

https://www.notion.so/Dgraph-Lambdas-d5e6e6614d394fb8afc935e2faa0aacf

  • Pagination with Cascade Cache 22

  • Continue bulk/live from where is stopped earlier

https://discuss.dgraph.io/t/santa-wish-list-for-dgraph-in-2021/12070/4?u=amaster507

  • Offset-based pagination is slow

https://discuss.dgraph.io/t/offset-based-pagination-is-slow/8774/5

  • Add distance for geo

https://discuss.dgraph.io/t/add-distance-for-geo/8406

  • Cannot Run Upsert in Dgraph Cloud DQL UI (only supports JSON not RDF)

https://discuss.dgraph.io/t/cannot-run-upsert-in-dgraph-cloud-dql-ui/14957?u=amaster507

  • Move @default into 22.0[?]

https://discuss.dgraph.io/t/automatic-value-for-creation-date-and-modification-date/11922?u=amaster507

  • Custom [Digging] Function(s)

https://discuss.dgraph.io/t/custom-digging-function/9499?u=amaster507

  • Defer field selection to subquery when using @custom DQL resolver

https://discuss.dgraph.io/t/defer-field-selection-to-subquery-when-using-custom-dql/9221/3

  • Multiple Reverse Edges

https://discuss.dgraph.io/t/why-cant-i-create-two-separate-attributes-to-the-same-node/17067?u=amaster507

https://discuss.dgraph.io/t/one-way-vs-two-way-hasinverse/7255?u=amaster507

https://discuss.dgraph.io/t/support-nested-link-and-multiple-link/10031?u=amaster507

  • Counting within pagination

https://discuss.dgraph.io/t/feature-request-count-property-on-relationships/14949?u=amaster507

https://discuss.dgraph.io/t/metadata-to-determine-if-there-are-more-results/15422?u=amaster507

  • Cursor Based Pagination

https://discuss.dgraph.io/t/cursor-based-pagination/14049?u=amaster507

Support for JSON-LD

https://discuss.dgraph.io/t/support-json-ld-on-dgraph/7162?u=amaster507

  • Dreaded Context Exceeded Bug

https://discuss.dgraph.io/t/mutation-failed-because-dgraph-execution-failed-because-context-deadline-exceeded/15221?u=amaster507

  • HIPAA Compliance

https://discuss.dgraph.io/t/dgraph-and-hipaa/15624?u=amaster507

  • Incomplete Items (I believe) From https://discuss.dgraph.io/t/dgraph-product-roadmap-2021/12284

    Lambda Should continue to resolve GraphQL
    Remote authorization hooks
    Auth on Union type
    Pre/post auth hooks for update mutation
    Global auth rules
    Replacing types in GraphQL schema: show left over data
    Support DQL Variables in Mutations
    String transformation functions
    TF-IDF scoring on full-text search [Dgraph 21.07]
    Integration with Kafka
    Integration with KeyLines
    Support for Gremlin
    Integration with BI Tools (e.g. Tableau)
    Import Neo4j json or CSV
    ORM for top-3 languages ( JS/TS, Py, Java )
    Load/stream data directly from SQL to Dgraph Cloud
    Load/stream data directly from MongoDB to Dgraph Cloud
    Load/stream data directly from Elastic to Dgraph Cloud


This is a start at summing up everything yet again. Others like @BenW @jdgamble555 might have more to add here too…

Probably missed someone’s beloved feature request or problem needing a workaround and I’m sorry, that was not on purpose. :heart_eyes:

6 Likes

Hi @amaster507,

Thanks for quickly putting this together.

2 Likes

Since we brought up in our discussion HL7 you might also be interested in

@gajanan just a followup after editing the OP above, I wanted to provide some other personal topics, ideas, and thoughts that I think might help in your decision making processes building a roadmap for the longterm development of Dgraph.

Just like yourself I came from a heavy SQL background and so have many other users here and potential new audiences. My personal opintion is we need more content like the following worked into official documentation:

https://dgraph.io/learn/courses/datamodel/evolution/develop/graphs-data-modeling/relational-data-model/

https://dgraph.io/learn/courses/datamodel/sql-to-dgraph/develop/graphql-syntax/

And while talking about documentation (as I was involved with it directly before the layoffs) I believe it would be beneficial to put in official documentation clear communication about what Dgraph can and cannot do. It is shady business in my opinion to hide your own known flaws in a discuss forum instead of being the one to first bring them up. I proposed writing some “offical” documentation around Dgraph’s limitation and known workarounds, but that was unnaproved at the time. This would help new developers and teams better evaluate the product knowing in advance what the known limitations (problems) are before investing so much and then being negative PR.


If you haven’t already, I’d aldo HIGHLY recommend reading the #1 post in this forum of all time:

https://discuss.dgraph.io/t/new-to-dgraph-dql-vs-graphql-endpoints-headers-and-schema-translation/10443/12

We (@acarey and I) were working on migrating some of this into the official documentation before the layoffs as well. I think it is important to put this kind of overview level in the “official” documentation.

You can also see all top discuss activity with this link to see what are hot topics:

https://discuss.dgraph.io/top?period=all


And another idea I had and voiced to you over the phone was a way to store data singularities.

For reference, Vaticle (TypeDB) does this and calls it Attribute Instances:

In the data, attribute instances in TypeDB are globally unique by type and value, and immutable. Being globally unique and immutable means your data is maximally normalised at all times.

What does this mean in practice? As an example, in a database of people with ages, there will be at most one instance of age 10, which can never be changed in-place.

So how do multiple people instances have the same age with value 10? We create an ownership of the attribute instance age 10 by each person.

I believe Dgraph should consider this as well. Growing a database to terrabytes of data would then be even more impressive when you are not storing the same exact attribute millions of times.

If Dgraph could leverage the Subject-Predicate-Object (aka SPO/triple) model but at the same time also leverage the ability to operate in the reversed OPS model, then the values could be stored with singularity and provide the new query engine with the ability to find nodes by values just as if it was travering the graph by an id. This would be taking the Dgraph indexing algorithms to a whole new level and playing field.


Another think that I have learned from typeDB is their rules

With their rules you can infer deeper relationships at a closer layer and then break into the rules to show the in-between edges and nodes.

This would be useful to helping to make more normalized data appear less normalized in an API.


EdgeQL (which has an awesome User-DX btw) is another data tool to keep a cose eye on in terms of feature comparison. They are SQL based, but they have built their query language with the idea of GraphQL in mind like Dgraph did with DQL but kept the SQL parts as well (since they are still SQL technically, postgresql)

They have some awesome features (besides the amazing typescript strongly typed client) such as their Computed properties and Links:

This is similar idea of being able to make a deeper part of the graph appear closer in the API or doing some custom business logic in replace of Dgraph’s lambda implementation.


You hinted at making your new query engine pluggable as well to make future development easier. I cannot tell you how much I agree with this idea!

Other users support this idea too:

EdgeDB works with extension in this same way too: Extensions — Schema | EdgeDB Docs