The Good, The Bad, The Ugly - State of Dgraph

[amaster507 (Anthony Master)]

Talking with @gajanan (or @gajanansc) earlier today, He requested, I post a discuss topic summarizing a bunch of links for feature requests, current problems, ideas, suggestions, etc. The goal here is to be factual and technical. Oh and for those who may have not known before, @verneleem is also me

If I have to do all of the raw data filtering, manipulation, and analytics in a client outside of the database, then why have the database at all?

@MichelDiz

We cannot require an API language to do the work of a DB language

^ Sometimes it seems right now that we have 2 API languages and 0 DB languages with Dgraph.

If you haven’t yet, please see The State of Dgraph’s GraphQL API Notion Document. I will reference many of the same topics from this document over again here to try to get a complete overview all in one place linking to many different subjects.

Notion – The all-in-one workspace for your notes, tasks, wikis, and databases.

• Missing @auth rule for post update state

Feature Request: Update After @auth Validation

Experience Report for Feature Request What you wanted to do Prevent a user from updating certain fields based on @auth rules for validations, the same way I can prevent a user from adding certain fields based on @auth rules for validations. What you actually did I can currently prevent a user from adding a certain value to the database by using many different types of field validation @auth rules: By using some rules like this: type Language @withSubscription @auth( add: { rule: "query { …

How to achieve field level auth at the moment?

I will explain my issue, but first let me tell you a quick overview. I am creating a POC for a new project which has little bit similar functionality to the DevJokes Example repo here - graphql-sample-apps/dev-jokes at master · dgraph-io/graphql-sample-apps · GitHub My Frontend is going to be connected to the Slash GraphQL directly and there is no server in-between. to summarize, Users would be able to post stuff that needs to go through an Approval Process. this is present in the example lin…

The Need for Update auth after logic

Continuing the discussion from Mutations - Graphql: Is this still in development? I would like this to be in the auth directive itself and not stuck in a JS Hook. Right now I can prevent a user from adding content as another user with the add rules, but the user can add data as their own and then update it to belong to a different user. This is a security whole as we do not want to allow users updating data and removing them as the author and assigning somebody else to make it appear that the …

• Field level authorization, not the same as ACL predicate control in DQL Enterprise

How to achieve field level auth at the moment?

I will explain my issue, but first let me tell you a quick overview. I am creating a POC for a new project which has little bit similar functionality to the DevJokes Example repo here - graphql-sample-apps/dev-jokes at master · dgraph-io/graphql-sample-apps · GitHub My Frontend is going to be connected to the Slash GraphQL directly and there is no server in-between. to summarize, Users would be able to post stuff that needs to go through an Approval Process. this is present in the example lin…

Walkaraund for common database features in the doc

Design Concepts looks like someone ripped off the first 10 pages of it. I am playing with the Dgraph for a while and I like the idea. I am sure that you are aware of the problems and be sure that we, the community, have the patience. But, please, try to solve the basic issues. From the place where I am concerning Dgrap usage, those are the two issues that I see as the most urgent. Timestamp controlled by the database (or default predicate value, e.g. time.Now()). Backend, or in Dgraph case …

Using @auth on individual fields?

Breaking this into it’s own topic, so here is the context: From reading over the docs on @auth directive I understand that these directives can restrict access to the nodes themselves by using JWT properties to check for user roles, or matching graph fields against those properties. Please correct me if I am wrong, but from how I understand it so far: Once a user has access to a node then the user also has access to all of the fields of that node. If I have a node labeled person and I want…

How do I protect certain fields with Slash GraphQL

HI All, I am new here. Last few days I spent a lot of time reading and watching videos about DGraph and Slash. I have a starter account setup on Slash and we are trying to get a POC rolling for a big project that we are developing. So my setup is that the Frontend is going to be connected to the Slash Graphql directly and there are no other middle servers. I am failing to understand a very basic question. Which is a very common usecase IMO, but I did not find anything on this, so I will ask …

• External node/type based auth rules

Feature Request: Non type related query auth rule

If I have a User type in my schema and I want to do a auth check against that user to access a type, but that user is not related to the type, there is no way to do that. The auth query rules are limited to the query of the type to which the rule applies. What would be nice is if I could use the JWT to do a query based upon a predicate attached to the user. This would allow me to revoke JWTs or even an entire user using just the database and rules. For instance I give the user a JWT with thei…

• Scalar Validation/Constraints (possible solution pre-hooks in GraphQL API)

Field validations

[image] The first thing I thought when I saw this illustration was “amazing! how they managed to make data validation so easy”. But then I was reviewing the issue of validations very thoroughly and I see that they don’t exist. In a realworld application. How would it be possible to mutate data directly from your application to the server without validating simple data like an email?

Composite @id fields

Hi, will this RFC also include composite indexes? E.g. in this example I want to allow unique names within a group of bars, i.e. the uniqueness should only be validated for the composite pair (fooID, name). type Foo { id: String! @id bars: [Bar!]! } type Bar @id(fooId, name) { fooId: String! name: String! }

Support for composite primary keys

This is one of the several standard features that SQL supports that DGraph doesn’t. For now, one work around is to do something like this: type Node { id: String! version: String! composite: String! @id } Add the composite string as id__version so that it is forced to be unique. You could use a lambda to enforce this, although if you add it correctly, it will be be unique without a lambda. J

GraphQL error: Non-nullable field was not present in result from Dgraph

I tend to disagree with this. I believe that there should be a distinguishing factor between a type that generates required inputs and a type that generates required fields. If this is only supported by a custom mutation then all generated mutations for types that reference this type may allow invalid input. type Post { id: ID title: String author: Person! } This would correctly prevent from adding a new post if I did not supply the author, but it would also throw errors if I am author…

How to Improve Unique Constraints in Schema?

Looking for a possible way to improve a portion of my schema while it is young. I have notes and allow users to add reactions to the notes. Users are linked to contacts to hide user info while showing contact names. What I am looking for is a better way to only allow any one reaction to be made once for any one contact on any one post. So in essence, a three edge unique constraint. There was this somewhat similar post a few days ago. I helped @chewxy with that one some and wanted to build upon…

• Edge/Relationship Validation/Constraints in DQL

Feature Request: Cascade Delete & Deep Mutations By Reference Directive

Experience Report for Feature Request Update: 7/1/21 - to comply with Feature Request Template: What you wanted to do 1.) Update nested nodes from the parent Update Mutation. 2.) Delete nested nodes from the parent Delete Mutation 3.) Choose which nodes allow this Cascade Delete and Cascade Update, as not all parent / children relationships should have this ability What you actually did 1.) This can only be done by creating a new update mutation for EVERY SINGLE child individually. If I am …

Hacking the @cascade delete and phantom node problem

So, yes, dgraph can fix this problem by implementing my feature suggestion (I also talk about the basic hack for now in this link): However, there is a bigger issue here. I think we don’t always think about things the way they are. Dgraph handles arrays as shared arrays unlike a noSQL database in example. Example Type Item { id: ID! name: String! @id ... } Type Cart { items: [Item] ... } If we have something like: items: [bread, butter, eggs...] those items may be shared by o…

Slash breaking GraphQL constraints & Cascade delete

Hi all, Broken GraphQL constraint - Example Let’s have a schema (unimportant fields omitted for brevity): type Client { id: String! @id projects: [Project!]! @hasInverse(field: client) time_blocks: [TimeBlock!]! @hasInverse(field: client) } type Project { id: String! @id time_entries: [TimeEntry!]! @hasInverse(field: project) client: Client! } type TimeEntry { id: String! @id project: Project! } type TimeBlock { id: String! @id invoice: Invoice @hasI…

Does dgraph support cascading delete？

Does dgraph support cascading delete？

• Separating/Combining Interface and Implementing Type @auth rules

Auth rules on interfaces cascade to the implementing type auth rules and get combined that implementing types must match ALL rules of the type and the interface, but sometimes that is not wanted, but rather the need is for the rules to be combined with OR logic instead.

• Separating @auth outside of the GraphQL schema itself

Moving GraphQL Authorization to admin API

Experience Report for Feature Request Note: Feature requests are judged based on user experience and modeled on Go Experience Reports. These reports should focus on the problems: they should not focus on and need not propose solutions. What you wanted to do Manage Authorization configuration and rules outside of the schema What you actually did Put Authorization configuration and rules inside of the schema Why that wasn’t great, with examples Any external references to support your…

• Re-usable/Global @auth rules

Moving GraphQL Authorization to admin API

Experience Report for Feature Request Note: Feature requests are judged based on user experience and modeled on Go Experience Reports. These reports should focus on the problems: they should not focus on and need not propose solutions. What you wanted to do Manage Authorization configuration and rules outside of the schema What you actually did Put Authorization configuration and rules inside of the schema Why that wasn’t great, with examples Any external references to support your…

• Scoping/Cascading @auth rules

Scope of Auth Directive Query Rules

This is more for documentation for myself and my companions, but I think it can add value to others. TL;DR; Auth Directive queries are processed before applying any other rules. You can not simply check if you already have access to one item and use it to provide access to related items When writing @auth directives I have tended to think on basis of queries to what the user can already see (as far as deep data). This was actually a flawed concept. Consider the following schema: type Conta…

• Hard limiting results in GraphQL API — Prevent Data Scraping

Bug: Cannot limit number of results using auth directive to prevent malicious queries

We are trying to prevent users from requesting too many documents by setting a big number in first argument. What edition and version of Dgraph are you using? v21.03.1 Have you tried reproducing the issue with the latest release? Yes, the bug can be reproduced in Dgraph Cloud too. Steps to reproduce the issue (paste the query/schema if possible) Try this schema: type Todo @auth( query: { rule: """ query { queryTodo(first: 2) { id } …

• [Completed?] Combining @auth and @custom DQL resolvers

WIP: Auth rules custom DQL query

I am reading this for the first time to understand this in detail. I have a use case where I want to allow some content through a custom query with DQL that a user does not normally have access to under normal @auth rules. Will this still be possible? Use Case Defined: Users should not be able to see a list of all other users in the system Admins in the application need to be able to manage users, so disabling queryUser and allowing only getUser is not a viable solution. A User can share data…

• Nested Filtering for DQL and GraphQL API (without using @cascade directive)

  AKA: Dealing with more normalized forms of data

RFC: Nested Filters in GraphQL

Motivation There are many requests from the community to add nested filters in GraphQL and it’s also one of the feature which can add more flexibility and value in our GraphQL implementation. Nested Filters like below are not supported in GraphQL . query { queryAuthor( filter: { name: { eq: "Alice" }, or: { posts: { postID: ["0x1", "0x2"] } } } ) { name posts { title } } } Although in DQL this Query is possible using var blocks like below query { post1 as var(…

• Paginating child nodes as a whole irrelevant to their multiple parental levels (related to nested filtering)

Pagination on child object that belongs to various parent object types

I have a schema that at a high level looks like this <Restaurant.name>: string @index(fulltext, trigram) . <Restaurant.address>: uid @reverse . <Boutique.name>: string @index(fulltext, trigram) . <Boutique.address>: uid @reverse . <Address.address1>: string @index(fulltext, trigram) . <Address.city>: uid @reverse . <City.name>: string @index(fulltext, trigram) . <City.state>: uid @reverse . <State.name>: string @index(fulltext, trigram) . And I want to do a reverse lookup with pagination of a…

• Logically combining filters together from different levels in the graph (related to nested filtering)

Filtering by multiple child types slow

What I want to do Retrieve paginated results from a large dataset (~10M nodes generated). Effectively filtering out the root query using children predicates. What I did Below is a cut down version of the schema for what I’m trying to do

Basic Schema

type Restaurant { id: ID! name:String @search(by: [fulltext, trigram, exact]) description: String @search(by: [fulltext, trigram]) externalID: String @search(by: [fulltext, term, trigram, exact]) safetyRating: String @search(by…

• Ordering/Sorting by nested data

How to Order by Nested Data using GraphQL

We have discussed filtering on edges multiple times mot recently: Filter on non-scalar fields or relations (Types) - #4 by abhijit-kar, which is closely related to the topic of ordering on edges. Given sample schema: type Contact { name: String! tags: [Tag] category: Category } type Tag { tag @id } type Category { cat @id } With the query: { queryContact(first: 5) { name tags { tag } category { cat } } } I could get a response back like: data.queryContact: [ { n…

• Filter by Aggregated results

Nested aggregation filter / Value Propagation

I’m trying to create a chat system. I want to show the number of new messages (unread). Each time a user reads a message, I update a field called readTime with the current time. This way I can count the number of messages that come after that time. There can be multiple channels in a group. I would like to show the number of unread messages for each channel based on the readTime of each channel. Also, I would like to show the total number of unread messages for the entire group. The followi…

Filter by counts in GraphQL

Looking at the Dgraph tour, I can Find all people who are in their 20s, and have at least 2 friends. { lots_of_friends(func: ge(count(friend), 2)) @filter(ge(age, 20) AND lt(age, 30)) { name@. age friend { name@. } } } How would you do this in GraphQL? A custom query backed by a lambda? If so then you lose all of the power of selecting fields and filtering on the subgraph.

• Scalar comparisons

Compare predicates from two different nodes

I have two different nodes, first one is person {name, age} and second one is family{ person_name }. I want to query all families that match person_name with name in person node. How can I do it? ex: person => [ {‘Alex’, 25} , {‘Bob’, 30}, {‘John’, 50} ], family=> [ {‘John’}, {‘Bob’}, {‘Test’} ]. So my query should print family => [ {'John'}, {'Bob'} ] . @pawan

Comparing two predicate counts in a filter

For curiosity and to better understand the fundamentals, why does dgraph not support two functions: What is preventing dgraph from being able to process this? If it can evaluate a function on one side, shouldn’t it also be able to evaluate a function on the other side of an equation? Maybe this is a little deeper subject than the OP, sorry.

• String pattern matching (not full regexp)

Subtring Matching without Regex?

Continuing the discussion from Simple substring matching without trigram index: The above topic was closed with no replies. So opening anew. @BlakeMScurr, are you still around? What did you find out? I want to allow users to search for an exact phrase. Here is what I have looked at: eq - matches string as whole and not substring, so no matches unless === anyofterms/allofterms - matches terms in a string and not exact phrase, gives false positives anyoftext/alloftext - does more thought matc…

• Date/time filtering and manipulation

DateTime Functions are needed

Continuing the discussion from Filter Datetime on month or day / advanced Datetime functions: I guess I am just now realizing that this is not here anywhere (at least that I can find). Every Database that I have ever worked with before has had Date/Time functions built in. I have the prerequisite of not storing different parts of a DateTime into separate searchable fields. I need to be able to do filtering based upon DateTime that matches a particular month, or a particular day, or a particula…

Date Filtering eq is Useless

Hardly ever do I need to find a node by the exact date to the second which is what the eq filter does. Most often I need to find nodes that match dates based upon the hour, day, month, year. Right now to do any of these we have to use the between filter. Use Case Example type Task { id: ID name: String due: DateTime @search } query thisHour { queryTask(filter: { due: { # eq: "2020-03-10T12" between: { min: "2020-03-10T12:00:00", max: "2020-03-10T12:59:59" } } }) { id }…

• Order by enums in GraphQL API

Order by enum type

I am having a problem ordering by an enum type: enum Direction { forward backward } type Garage { id name cars: [Car] ... } type Car { id: ID! direction: Direction! @search ... } And my query: queryGarage(filter: {id: [$id] } ) { id name cars (order: { asc: direction } ) { direction name } } And I get this message: "input:11: Expected type CarOrderable, found direction.\n" However, when I do order: { asc: name } I get no problems… Can I sort by the enum …

• Sorting by Aggregation

Possible to sort by count of Aggregate field?

For instance, queryContact( order: { desc: hasTasksAggregate { count } ) Is there some way of doing something to that effect?

• Calculated Fields/Triggers

Cooking recipes app

Yes this is similar, except there is an additional id field on ingredient (like “carrot”). And yes here the quantity is a weight for simplification. I was not very clear in my use case: One user has, say, 2g of carrots and 3g of salad. He wants a recipe that use the most of his stock. So he sends to the API a list of tuples [("carrot", 2), ("salad", 3)]. The API should send back a list of recipes names sorted by the “score” value computed with the above algo in pseudocode (the score is a coeff…

• Full Text Search Best Match Scoring

Need to Order by best match

Continuing the discussion from Ordering of results in `anyofterms` query: @MichelDiz forgive me, but I am not seeing an existing feature request, so I am creating this one. Experience Report for Feature Request What you wanted to do Do term based matching and order by best matching first. What you actually did [image] Why that wasn’t great, with examples Because now my head hurts. (See example above) Any external references to support your case It can be done with SQL type languages using C…

Feature request: full text search with tf-idf Scoring

MichelDiz commented : Yes, exactly, it is your own index of search words with tf-idf scoring. I think it would be great to implement this as built-in. Since we use KV indexing, it would be easy to implement this. But it does require some work and thought. @campoy , what do you think of this feature? Can you define exp, status, and priority flags?

• Full text search across multiple fields:

Fulltext search across multiple fields

Experience Report for Feature Request What you wanted to do Fulltext search across multiple fileds What you actually did Nothing, yet, as it is not possible. Why that wasn’t great, with examples Alternatively we are contemplating creating duplicate data fields that hold all the strings concatenated together, but then we break the single source of truth and would rely on other mechanisms to keep this field in sync with the others. Any external references to support your case Use to being …

• String Functions for inter graph comparisons and manipulation

String Functions in DQL

Experience Report for Feature Request What you wanted to do Use String Functions within DQL to do upserts and transform data on server side without client script. What you actually did Started this Topic After I double checked and searched the docs again and searched this forum for “String Functions” Why that wasn’t great, with examples Because there are not any string functions. I understand the GraphQL experience with Dgraph, but when I want to do data work behind the generated AP…

• Simplifying/Enhancing groupby

Remove Duplicates from Results

Given a graph that looks like this: A -> M{status: 1, uid: 0xa} B -> M{status: 1, uid: 0xb} are they really distinct statuses? Perhaps the graph-ness should be taken into account and when inserting the UID of distinct status messages should be kept?

SELECT DISTINCT in Dgraph

Deja vu, I find myself back here again. @dmai, has any of these ideas been explored? UPDATE(for myself next time searching for this): too much to remember sometimes… Here is the syntax query for the above data: { distinctCountries(func: has(Country)) @groupby(Country) { # no need to select any fields, Country will be selected in the group by results # if you want to count the duplicates, use: count(uid) } } This syntax would return the following results: { "data": { "di…

Groupby returning multiple entries

@gajanan this syntax should be supported. @MichelDiz or @matthewmcneely any help with group by? Not much experience with this part of DQL

Postgres: Distinct Query Results | Hasura GraphQL Docs

What is Dgraph lacking?

I think there are some things missing from DQL that could reduce friction when developers consider adopting Dgraph. For example, the ability to include non-aggregate predicates in @groupby query blocks. I’m willing to bet that it’s a common query folks try, especially those that come from SQL backgrounds and are used to queries like SELECT COUNT(1), id, name FROM table GROUP BY id. Improvements to pagination allowing you to query pages using limit/after when applying some non-uid sort on an inde…

What is Dgraph lacking?

Related: can we get @groupby(MONTH(creation_date)) see here: How to groupby date - #19 by geoyws or here: Suggestions to Improve GroupBy - #6 by MichelDiz

Suggestions to Improve GroupBy

Another Groupby usage. One thing I find strange about GB’s behavior is that it returns the grouping only that it is not possible to open the values bound to that grouping. Unless of course, they are children of the group or have a reverse edge. e.g (Dataset in the end): This query { variableblock(func: type(Person)) @groupby(studied_at) { a as count(uid) } data(func: uid(a)){ uid name total: val(a) } } Returns { "data": { "variableblock": [ { "@…

• Verbosity of Multiple Node Updates in GraphQL

Cannot update multiple nodes at once

context: Updating multiple nodes at once with their own separate patches. I can generate a mutation with multiple mutation statements one for each node, but my IDE frowns upon dynamic queries/mutations in this regards. I can’t do this with the generated updateType mutation as it stands because it only takes one filter and patches all nodes that match the filter with the same patch. I thought I could do this using a deep mutation, but deep mutations do not allow you to patch child nodes. type…

What is Dgraph lacking?

Can you point me to where the bug fix for this one is: I tried this on Tuesday and it still wasn’t working. I have a table A, with an array relationship to B. I need to get every row of A with their B and perform an update for each one. So if A1 has B1, B2, B3, I need to do an update with [A1,B1], [A1, B2], [A1,B3], and I need to also include A2…A500 and it’s relationships. AFAIK This is still not possible. Update Many: Fail in GQL, pass in DQL I gave this a fail because of the node l…

• Deep Mutations

http://discuss.dgraph.io/t/deep-mutations-graphql/9789/4?u=amaster507

https://dgraph.io/docs/graphql/mutations/deep/

Feature Request: Cascade Delete & Deep Mutations By Reference Directive

Experience Report for Feature Request Update: 7/1/21 - to comply with Feature Request Template: What you wanted to do 1.) Update nested nodes from the parent Update Mutation. 2.) Delete nested nodes from the parent Delete Mutation 3.) Choose which nodes allow this Cascade Delete and Cascade Update, as not all parent / children relationships should have this ability What you actually did 1.) This can only be done by creating a new update mutation for EVERY SINGLE child individually. If I am …

http://discuss.dgraph.io/t/data-doesnt-update/11489

https://github.com/hasura/graphql-engine/issues/1573

• Comparative Inputs

http://discuss.dgraph.io/t/atomic-field-operations/14128/3?u=amaster507

• Auto Incrementing Fields

http://discuss.dgraph.io/t/feature-request-directive-that-indicates-that-id-fields-should-be-auto-populated-on-creation/14992

http://discuss.dgraph.io/t/atomic-field-operations/14128?u=amaster507

• Need to correct generated payload list nullability in GraphQL API

Payloads right now in Dgraph are generated as nullable items in a list. “queryUser: [User]” But this should be corrected to the tightest possible type such as “queryUser: [User!]!” which means that the result will be an array, it could be an empty array, but no items in the array can or ever will be null.

• No Arrays—only Sets/Lists

http://discuss.dgraph.io/t/documentation-that-arrays-work-as-a-set-not-storing-duplicates/9590?u=amaster507

But what needs thought out even more so is what this change would bring with it in terms of API capability

• Add/Update values at positions in a list chosing to replace or skoot over existing values that may exist.
• Replace a list in its entirety. This is rather difficult to do but is such a simple use case
• Delete values from a specific index in a list. Right now you can only delete all items in a list or items in a list by their value.
• Allow lists to maintain order (for lists of scalars and lists of type (aka edges))
• Move item(s) in a list to a specific location without changing any values

• Unions sometimes produce unexpected results

http://discuss.dgraph.io/t/using-unions-and-interfaces-and-end-up-with-non-nullable-field-not-present/15680?u=amaster507

http://discuss.dgraph.io/t/dgraph-directive-does-not-work-as-expected-with-union-types/13021

• @auth on union types

Completed? Needs to be documented or implemented if not currently—haven’t tested personally

http://discuss.dgraph.io/t/union-types-in-graphql/9581?u=amaster507

• Custom Directives

The idea of custom directives is centered around directives on the developer creating directives available to clients. A developer may wish to allow some kind of direct script being processed on command such as logging the result or adding some metadata to the response (not data, but in the extensions response)

http://discuss.dgraph.io/t/custom-field-concatenation-without-needing-external-script/7283?u=amaster507

http://discuss.dgraph.io/t/can-i-get-a-substring-in-a-query/10522/3?u=amaster507

• Custom DQL Mutations

This really depends on how much is being refactored in the GraphQL codebase and how that refactoring is done. If GraphQL will still be rewritten into DQL then it is equally important to support not only DQL in custom queries, but also DQL in custom mutations.

http://discuss.dgraph.io/t/why-does-custom-dql-not-support-mutation/10366?u=amaster507

• Custom DQL Fields

Custom fields can currently be resolved with lambda, but it would be beneficial (again depending on how the refactor is done) to allow custom fields to also be resolved with DQL.

http://discuss.dgraph.io/t/can-i-use-dql-to-calculate-the-value-of-a-custom-field/15877?u=amaster507

• Additional and Custom Scalars

Developers often finding themselves needing to add custom scalars for various reasons. These can often be represented as strings but with additional constraints such as Email, HexColor, Tuple

http://discuss.dgraph.io/t/json-blob-as-a-scalar/11034?u=amaster507

http://discuss.dgraph.io/t/can-we-have-remote-scalars/11726?u=amaster507

• Authentication Service

Dgraph built an authentication system for Dgraph Cloud and was discussing open sourcing it. I believe something like that should be made and integrated into the GraphQL API so that users can easily authenticate against their own data and maybe use lambdas to return the claims from the database that the developer wants to use when a client authenticates.

http://discuss.dgraph.io/t/open-sourcing-in-house-dgraph-auth/13398?u=amaster507

• Open Source all Enterprise Features

• Auditing
• Namespacing
• [Completed in 21.12?] Backups

Should these really be enterprise or are these just enterprise level to force users into the Cloud? Are Enterprise licenses even available anymore? (They were not [or ridiculously purposefully priced outside of the budget to whom it was being quoted] under the last administration)

http://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/89?u=amaster507

• Schema Migration Tools

There is a need for when users migrate their schema, they expect the data to follow. This is not done in Dgraph now.

http://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/34?u=amaster507

http://discuss.dgraph.io/t/migrating-renaming-predicates-etc/5239?u=amaster507

https://www.edgedb.com/docs/guides/migrations/index

• Facets are not first class citizens

http://discuss.dgraph.io/t/improve-facets-with-upsert-block/5640

If nested filtering and linking-nodes are not able to be constrained, then we can’t get rid of facets. But even then, maybe we can take the concept of _linkingnodes and make them work without needing to declare the type in the middle, like how prisma creates pivot tables without you needing to specifically create them and in the prisma ORM it lets you link directly through the pivot table like as if it was a 1:1 relationship. For Reference: EdgeDB has what it terms “link properties” and abstracts these onto types.

https://www.edgedb.com/docs/guides/link_properties

• Mapping GraphQL’s @hasInverse vs DQL’s @reverse

Right now Dgraph GraphQL API, uses the @hasInverse directive to “map” inverse relationships and then the API keeps these pairs of edges balanced with mutation. This creates additional work for adding RDF data with live/bulk loader to add two edges for every inverse relationship.

It might be better if Dgraph would just allow the mapping of the ~ reverse edges.

• Var Blocks in GraphQL

http://discuss.dgraph.io/t/directive-idea-to-support-var-blocks-in-graphql/14419?u=amaster507

• Fuzzy Full Text Search

The ability to use TRIGRAMS on phrases and not just words. You cannot search for a fuzzy phrase, only a fuzzy word. This limits full-text search.

http://discuss.dgraph.io/t/fuzzy-full-text-search/16224?u=amaster507

• Count Words

The ability to count the number of times word appears in a text and sort by that value. This would make it possible to write relevant search algorithms.

http://discuss.dgraph.io/t/how-to-implement-keyword-based-relevance-sorting/16174?u=amaster507

• DQL Loops

very much needed to simplify algorithms without stepping in and out of queries/mutations with a client.

http://discuss.dgraph.io/t/foreach-func-in-dql-loops-in-bulk-upsert/5533/6

• More educational material
• Some docs that allow us to get into the DGraph code easier, so we can contribute
• Transparent roadmap, open issues and bugs, so we don’t get surprised by missing features or minor bugs

http://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/3?u=amaster507

• Communicate with your users and get us involved! One post from [the Dgraph Labs] a week giving simple product updates and plans can go a very long way.

• Provide a generous free tier.

• An out of the box local (offline) development experience (aka without me having to learn / do much)

• A way to batch mutations so I can roll back a group of changes.

• Be able to simply replace a list in a mutation.

http://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/9?u=amaster507

• Removing Edges using null does not work

http://discuss.dgraph.io/t/removing-edges-using-null-does-not-work/15420

• Directly integrate other tools into Dgraph like https://magic.link , Auth0, and make it configurable with a few clicks
• Detailed and accurate GraphQL errors
• Focus really hard on making it as easy as possible for devs to build side-projects and hobby-projects free/cheap Dgraph instances

Dgraph has all the pieces in place to build the ultimate low-code tool. The simpler you can make it for users, the more users you’re gonna get.

http://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/10?u=amaster507

• No sharding of predicates
• No Query Planner (CTO’s current vision)
• Upsert by xid painful for ingest-heavy workloads
• Missing a native [time]range type

http://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/14?u=amaster507

• The upgrade process requires downtime.

http://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/17?u=amaster507

http://discuss.dgraph.io/t/improve-dgraph-upgrade-experience-by-supporting-in-place-rolling-upgrades/8821

• It would be nice if we have one click hosting on platfroms like digital ocean for open source

http://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/19?u=amaster507

• Lack of native timestamps.

http://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/21?u=amaster507

• have is a multi-RAFT approach for regional clusters like CockroachDB is doing

http://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/22?u=amaster507

• it’s own mobile solution with offline sync

http://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/23?u=amaster507

http://discuss.dgraph.io/t/call-for-collaboration-designing-a-dgraph-offline-first-library/9293/11

• BM25 and or custom search ranking
• More and custom tokenizers

http://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/25?u=amaster507

• Lack of support for some algorithms
• many problems in path lookup

http://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/26?u=amaster507

• sub-select statements in graphql
• More in cloud editor help & messages for when certain changes will orphan data or cause negative side effects

http://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/34?u=amaster507

• I don’t have time to write my own middleware, and I don’t want to host anything myself to deal with servers
• lambdas are time consuming and should only be reserved for complex tasks

http://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/40?u=amaster507

• Cannot Paste with Ctrl+V on Cloud UI

http://discuss.dgraph.io/t/cannot-paste-with-ctrl-v-on-cloud-ui/15195?u=amaster507

• Separarate GraphQL from Dgraph [as a Plugin?]

http://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/70?u=amaster507

http://discuss.dgraph.io/t/what-is-dgraph-lacking/16010/71?u=amaster507

• Typescript binding for dql client as well so responses are typed?

Reference: EdgeQL TS Client Achieved this! https://www.edgedb.com/docs/clients/js/index

• Fix Dgraph’s type system so that the type is not a string value/predicate

http://discuss.dgraph.io/t/how-does-type-system-work/17616?u=amaster507

http://discuss.dgraph.io/t/func-type-error-in-20-11-03-or-later/16237/29?u=amaster507

http://discuss.dgraph.io/t/ask-dgraph-founder-anything/16352/24?u=amaster507

http://discuss.dgraph.io/t/ask-dgraph-founder-anything/16352/28?u=amaster507

@mrjn

If one type has billions of nodes, then instead of using type(X), would be better to retrieve a field which only belongs to that type of node. Then check the existence of that to determine if the type is legit or not.

• An official dgraph toolset for data and schema versioning that is manageable within our codebases.

http://discuss.dgraph.io/t/ask-dgraph-founder-anything/16352/31?u=amaster507

• Math Functions

https://neo4j.com/docs/cypher-manual/current/functions/mathematical-numeric/

• Subscriptions based on CDC

https://dgraph.io/docs/master/enterprise-features/change-data-capture/

• Subscriptions should use graph-ws

https://github.com/enisdenjo/graphql-ws/blob/master/PROTOCOL.md

• [Completed with Learner Nodes?] Add Geo Replication

http://discuss.dgraph.io/t/geographically-distributed-datacenter-replication/13311

• Cloud Requested Fixes/Improvements from @jdgamble555

  • Add the storage [for file/media uploads management possibly connected to S3]
  • Make data studio have CRUD functionality
  • Allow renaming types in the UI
  • Build an Auth System

• The problem with Lambdas [A MUST READ!]

https://www.notion.so/Dgraph-Lambdas-d5e6e6614d394fb8afc935e2faa0aacf

• Pagination with Cascade Cache 22

• Continue bulk/live from where is stopped earlier

http://discuss.dgraph.io/t/santa-wish-list-for-dgraph-in-2021/12070/4?u=amaster507

• Offset-based pagination is slow

http://discuss.dgraph.io/t/offset-based-pagination-is-slow/8774/5

• Add distance for geo

http://discuss.dgraph.io/t/add-distance-for-geo/8406

• Cannot Run Upsert in Dgraph Cloud DQL UI (only supports JSON not RDF)

http://discuss.dgraph.io/t/cannot-run-upsert-in-dgraph-cloud-dql-ui/14957?u=amaster507

• Move @default into 22.0[?]

http://discuss.dgraph.io/t/automatic-value-for-creation-date-and-modification-date/11922?u=amaster507

• Custom [Digging] Function(s)

http://discuss.dgraph.io/t/custom-digging-function/9499?u=amaster507

• Defer field selection to subquery when using @custom DQL resolver

http://discuss.dgraph.io/t/defer-field-selection-to-subquery-when-using-custom-dql/9221/3

• Multiple Reverse Edges

http://discuss.dgraph.io/t/why-cant-i-create-two-separate-attributes-to-the-same-node/17067?u=amaster507

http://discuss.dgraph.io/t/one-way-vs-two-way-hasinverse/7255?u=amaster507

http://discuss.dgraph.io/t/support-nested-link-and-multiple-link/10031?u=amaster507

• Counting within pagination

http://discuss.dgraph.io/t/feature-request-count-property-on-relationships/14949?u=amaster507

http://discuss.dgraph.io/t/metadata-to-determine-if-there-are-more-results/15422?u=amaster507

• Cursor Based Pagination

http://discuss.dgraph.io/t/cursor-based-pagination/14049?u=amaster507

Support for JSON-LD

http://discuss.dgraph.io/t/support-json-ld-on-dgraph/7162?u=amaster507

• Dreaded Context Exceeded Bug

http://discuss.dgraph.io/t/mutation-failed-because-dgraph-execution-failed-because-context-deadline-exceeded/15221?u=amaster507

• HIPAA Compliance

http://discuss.dgraph.io/t/dgraph-and-hipaa/15624?u=amaster507

• Incomplete Items (I believe) From http://discuss.dgraph.io/t/dgraph-product-roadmap-2021/12284

  Lambda Should continue to resolve GraphQL
  Remote authorization hooks
  Auth on Union type
  Pre/post auth hooks for update mutation
  Global auth rules
  Replacing types in GraphQL schema: show left over data
  Support DQL Variables in Mutations
  String transformation functions
  TF-IDF scoring on full-text search [Dgraph 21.07]
  Integration with Kafka
  Integration with KeyLines
  Support for Gremlin
  Integration with BI Tools (e.g. Tableau)
  Import Neo4j json or CSV
  ORM for top-3 languages ( JS/TS, Py, Java )
  Load/stream data directly from SQL to Dgraph Cloud
  Load/stream data directly from MongoDB to Dgraph Cloud
  Load/stream data directly from Elastic to Dgraph Cloud

---

@amaster507

I’ve talked to a lot of new-comers. Do you want their voices?

They say almost in this order:

• There is too much confusion between GraphQL and DQL, what is the differences/purpose.
• The documentation is AWFUL!
• If I search docs I end up on a mixture of GraphQL and DQL again too jumbled together.
• I love Dgraphs implementation, but how do I extend it without using lambdas to do more things like my custom directives, scalars, input validation, more schema control such as requiring inputs but not outputs.
• You mean there is no nested filtering??
• Can you help model my data?
• I imported my data with rdf, why doesn’t it appear? (DQL schema mapping, missing inverse edges)
• How do I setup authentication? (Not just auth rules)
• Where do my backend files go? (Have to use a different service with a another separate auth mechanism)

@amaster507

If the same questions keep coming up, just having blog posts or discuss topics do not solve the issues. Something needs to change to prevent these things for smooth user adoption.

@amaster507

Whenever you get some free time, could you do a introduction into the Dgraph codebase for a beginner wanting to contribute to Dgraph and what are the key things he should know or do. If you need a real use case, I can provide something for you.

This is a start at summing up everything yet again. Others like @BenW @jdgamble555 might have more to add here too…

Probably missed someone’s beloved feature request or problem needing a workaround and I’m sorry, that was not on purpose.

[gajanan]

Hi @amaster507,

Thanks for quickly putting this together.

[amaster507 (Anthony Master)]

Since we brought up in our discussion HL7 you might also be interested in

Dgraph in Healthcare Interfaces (HL7) - an Edge/List ordering problem

This whole conversation pends on: What I want to do Use Dgraph as a db used alongside a healthcare interface engine for HL7 message storage, manipulation, and retrieval. What I did Tried to map HL7 2.x to RDF The HL7 Spec comes in a few different versions. The FHIR spec does have some discussion around RDF, but of course it is not the same RDF that Dgraph uses, but similar in nature. More specfically though, the 2.x versions of HL7 are not going away anytime soon, and these are what I de…

[amaster507 (Anthony Master)]

@gajanan just a followup after editing the OP above, I wanted to provide some other personal topics, ideas, and thoughts that I think might help in your decision making processes building a roadmap for the longterm development of Dgraph.

Just like yourself I came from a heavy SQL background and so have many other users here and potential new audiences. My personal opintion is we need more content like the following worked into official documentation:

Transitioning from a Relational DB to a Graph DB

Well, we made probably one of the most difficult choices halfway through Developing our Application, but it ended up being an easy decision. Here is our (somewhat) short story of making the transition from a Relational Databse (MySQL in our Case) to a Graph Database (Dgraph). Most of this post will be pointers on how to make the transition and what we learned that applies specifically to Dgraph but can also apply in the broader spectrum to most Graph Databases. Our setup was complicated to say …

https://dgraph.io/learn/courses/datamodel/evolution/develop/graphs-data-modeling/relational-data-model/

https://dgraph.io/learn/courses/datamodel/sql-to-dgraph/develop/graphql-syntax/

And while talking about documentation (as I was involved with it directly before the layoffs) I believe it would be beneficial to put in official documentation clear communication about what Dgraph can and cannot do. It is shady business in my opinion to hide your own known flaws in a discuss forum instead of being the one to first bring them up. I proposed writing some “offical” documentation around Dgraph’s limitation and known workarounds, but that was unnaproved at the time. This would help new developers and teams better evaluate the product knowing in advance what the known limitations (problems) are before investing so much and then being negative PR.

---

If you haven’t already, I’d aldo HIGHLY recommend reading the #1 post in this forum of all time:

http://discuss.dgraph.io/t/new-to-dgraph-dql-vs-graphql-endpoints-headers-and-schema-translation/10443/12

We (@acarey and I) were working on migrating some of this into the official documentation before the layoffs as well. I think it is important to put this kind of overview level in the “official” documentation.

You can also see all top discuss activity with this link to see what are hot topics:

http://discuss.dgraph.io/top?period=all

---

And another idea I had and voiced to you over the phone was a way to store data singularities.

For reference, Vaticle (TypeDB) does this and calls it Attribute Instances:

https://docs.vaticle.com/docs/general/introduction#attribute-instances

In the data, attribute instances in TypeDB are globally unique by type and value, and immutable. Being globally unique and immutable means your data is maximally normalised at all times.

What does this mean in practice? As an example, in a database of people with ages, there will be at most one instance of age 10, which can never be changed in-place.

So how do multiple people instances have the same age with value 10? We create an ownership of the attribute instance age 10 by each person.

I believe Dgraph should consider this as well. Growing a database to terrabytes of data would then be even more impressive when you are not storing the same exact attribute millions of times.

If Dgraph could leverage the Subject-Predicate-Object (aka SPO/triple) model but at the same time also leverage the ability to operate in the reversed OPS model, then the values could be stored with singularity and provide the new query engine with the ability to find nodes by values just as if it was travering the graph by an id. This would be taking the Dgraph indexing algorithms to a whole new level and playing field.

---

Another think that I have learned from typeDB is their rules

https://docs.vaticle.com/docs/schema/rules

With their rules you can infer deeper relationships at a closer layer and then break into the rules to show the in-between edges and nodes.

This would be useful to helping to make more normalized data appear less normalized in an API.

---

EdgeQL (which has an awesome User-DX btw) is another data tool to keep a cose eye on in terms of feature comparison. They are SQL based, but they have built their query language with the idea of GraphQL in mind like Dgraph did with DQL but kept the SQL parts as well (since they are still SQL technically, postgresql)

They have some awesome features (besides the amazing typescript strongly typed client) such as their Computed properties and Links:

Computed properties and links — Schema | EdgeDB Docs

This is similar idea of being able to make a deeper part of the graph appear closer in the API or doing some custom business logic in replace of Dgraph’s lambda implementation.

---

You hinted at making your new query engine pluggable as well to make future development easier. I cannot tell you how much I agree with this idea!

Other users support this idea too:

Develop Lambdas with any programming language

I am thinking if loading in WASM and executing it like-native works well… maybe that could be the future of custom tokenizers, custom dql functions, custom data types… What a cool draw that would be as an extensibility feature on this system. Custom tokenizers right now are doable but not fun since you have to build a go plugin shared object, which has a big restriction of having to have 100% the same imports/versions to be able to be loaded into dgraph :/. Something sandboxed may be better.

EdgeDB works with extension in this same way too: Extensions — Schema | EdgeDB Docs