IMPORTANT: new behavior for expand, deprecation of _forward_ and _reverse_

Announce
1

Hi everyone!

Now that types have made it into Dgraph with v1.1.0, it is time to reconsider how expand works.

In v1.2.0 we will update the behavior of expand to the following specs:

  • expand(_all_) expands all of the predicates found in all of the types associated with any of the nodes at the level where expand appears. This includes any predicates appearing in those types, which could be reversed edges.

  • expand(_predicate_) expands all of the predicates for all of the nodes at the level where expand appears. This includes any predicate coming out of the nodes, no matter whether any types are associated with those nodes.

  • expand(_forward_) and expand(_reverse_) will be REMOVED FROM THE LANGUAGE, as their interaction with types makes them redundant.

  • expand(typeNameA, typeNameB) will expand all of the predicates in the types given as parameters.

Some examples

Let’s define this schema:

<name>: string @index(exact) .
<parent>: [uid] @reverse .
<lives_in>: string .

type Person {
    name: string
    parent: [uid]
    <~parent>: [uid]
}

The type Person has three fields, corresponding to two predicates defined in the schema plus a reversed predicate. Let’s assume we have performed this mutation:

{
  set {
    _:f <name> "Francesc" .
    _:f <dgraph.type> "Person" .
    _:f <lives_in> "San Francisco" .
    _:f <parent> _:p .
    _:f <parent> _:l .

    _:p <name> "Paco" .
    _:p <dgraph.type> "Person" .

    _:l <name> "Lucia" .
    _:l <dgraph.type> "Person" .
  }
}

Using expand(_all_) on this query will return only the predicates in Person type:

{
  q(func: eq(name, "Francesc")) @recurse(depth:3) {
    expand(_all_)
  }
}

Since we know that all of the nodes in this dataset are of type Person, the query above is equivalent to:

{
  q(func: eq(name, "Francesc")) @recurse(depth:3) {
    expand(Person)
  }
}

Both of the queries above return the same response:

{
  "data": {
    "q": [
      {
        "name": "Francesc",
        "parent": [
          {
            "name": "Paco",
            "~parent": [
              {
                "name": "Francesc"
              }
            ]
          },
          {
            "name": "Lucia",
            "~parent": [
              {
                "name": "Francesc"
              }
            ]
          }
        ]
      }
    ]
  }
}

Notice how even though "Francesc" also has a predicate lives_in it will not appear in the response, as it’s not part of the type Person. Also, notice how ~parent is included in the response, as it’s also part of the type.

On the other hand, expanding on _predicate_ will return all of the predicates associated with the nodes, no matter whether they’re part of the type or not.

{
  q(func: eq(name, "Francesc")) @recurse(depth:3) {
    expand(_predicate_)
  }
}

{
  "data": {
    "q": [
      {
        "name": "Francesc",
        "parent": [
          {
            "name": "Paco",
            "~parent": [
              {
                "name": "Francesc",
                "lives_in": "San Francisco"
              }
            ]
          },
          {
            "name": "Lucia",
            "~parent": [
              {
                "name": "Francesc",
                "lives_in": "San Francisco"
              }
            ]
          }
        ],
        "lives_in": "San Francisco"
      }
    ]
  }
}

Notice how lives_in appears, although it’s not part of the type, and how ~parent appears as reverse predicates will still be fetched.

We need your opinion!

It is your time to tell us whether this will impact your usage of Dgraph so we can consider your needs in our design. Please let us know in this thread (or directly to francesc@dgraph.io) if you have any concerns regarding these changes.

We have other options, such as keeping _forward_ and _reverse_ but basing them on the behavior of _predicate_ rather than _all_. This would lead to not very performant queries, so that’s why we’re considering removing them completely.

Thanks!

:badger:

1 Like
2

The expand by type is a great idea !
I already need it desperately :scream:

1 Like
3

This looks interesting and what cames first to my mind, it’s the right evolution of expand for the types.
But one line in the type definition of this example leaves me sceptical:

<~parent>: [uid]

Isn’t this declaring an index as a type? That’s some kind out of the scope of a semantic definition, dont you think?
The following type declaration should be enough:

type Person {
    name: string
    parent: [uid]
}
1 Like
4

Hey @AugustHell,

Thanks for your feedback!

I’d say <~parent>: [uid] is a totally valid field, as <~parent> is a predicate, not an index.

That’s why you can even use it on a query block:

{
  q(func: has(name)) {
    name
    ~parent {
      name
    }
  }
}

We’re also working on a different proposal that would allow you to give a better name to a reverse predicate by passing it as a parameter to the @recurse directive.

You could then write:

<name>: string @index(exact) .
<parent>: [uid] @reverse(child) .

Then you could write the query above as follows:

{
  q(func: has(name)) {
    name
    child {
      name
    }
  }
}

I think it’s time to be able to call your children something better than your ~parents :smile:

3 Likes
5

Hmmm, the reverse being a predicate not really convincing me.
If so you should be able to store something different, then with parent.
If not, it’s redundance, which is to avoid in data stores.
~parent is storing the same information as parent, just in another order, so you get easier access to the information you want. That’s what an index does.
In the schema it is declared not on first position, as all other types names.
schema {
name
parent
~parent (wont fit here)
}
Instead it is declared at the end and till now not even with its name
parent: [uid] @reverse
the same position like the other indexes
name: string @index(exact)

if you place now in parantheses a name like
: [uid] @reverse(child)
it’s even get more confusing, as there is usually till yet the type of the index.

Defining that schema in go as struct

type Person struct {
    Name string `dgraph:"person @index(exact)"`
    Parent []Person `dgraph:"parent @reverse"`
}

… the reversed part is not even defined as a variable.
How on earth you should know now, that you need it to define separately for the type if you want to have the full usage of it? That’s more something like hidden secrets of an untyped language.

I think declaring the reverse in the type Person { <~parent> } is wrong and not logical. That should be done by the database or the data user needs to handle types and schemas separately while being always aware of, that if he changes the first he could break the other.

6

Would you consider adding the ‘dgraph.type’ of node in the response of expand() ? If you do, i think it’s will be more convenient to recognize the type of node and to deal with it.

10

Shouldn’t expand(__all__) include predicates that are not associated with any types too? Semantically, all implies everything. Maybe it would be a better for expand(__type__) to return predicates that belong to all types

11

You can do it yourself by also adding <dgraph.type> in the request, or even in the type definition if you prefer.

12

~parent is technically not an existing predicate, it’s an index.

We are not proposing to add a new feature here, we’re just proposing a better name for reverse edges.
There’s no change in what information will be stored, just a syntactic help to avoid having field names in a type with awkward names like ~parent when child is more concise.

I sincerely not follow what your concern is, I do not really appreciate your dismissive tone to be honest.
If you’d like to explain why this is “wrong and not logical” please expand on your reasoning.

1 Like
13

We’re considering that option too, thanks for your feedback @san8pai!

14

Sorry, I apologize, that was not intended. I’m not a native english speaker, nor am I that educated and skilled that you are. I have a huge respect for what you’re doing and am curious on all new I can see or read. I don’t wanted to sound destructive, instead I’d like to bring a constructive point to develop this feature further.

My concern here is the declaration the reverse edge (<~parent>) in the type:

If this reversed edge <~parent> need to be declared this way in the future in types to be expandable, it is questionable, because

  1. it’s not named that way in the schema. There it is noted as @reverse, like an index,
  2. it is by its own not an independent value as the others in the type. It depends on parent, so declaring it in the type it is redundant and that might cause other problems, (like, what happens if someone removes @reverse later in the scheme)
  3. as its not in the schema by name, it’s not in a struct as a variable/member if working with go. So finding a way to programmatically create the type this way is cumbersome.
  4. it is confusing being new with graphql, as it is not logical (in my eyes) to need to define something that looks like an index as a type.

I might be totally wrong and misunderstand it. Maybe the declaration of the @reverse as a type member is not needed and wont effect the results if not done. I just can’t test it yet and dont have the fully insight and understanding to be sure. But if results are affected, I need to go back to see how I can handle it in my go code :roll_eyes:

Again sorry, hope it is more clear now, what my thoughts are.

15

@francesc I think I have similar concerns to @AugustHell

This may be an abuse of the system, as Im new to dgraph, but my assumption was that if I add @reverse to the index, expand() would return me all the reverse relationships.

As it stand, this is not quite true, we’re required to add <~myrelation> on the type of the thing we are expand()ing, which reduces the utility of expand, since it requires defining all of the reverse relations explicitly.

A use case: I have entities that represent physical servers, with their components (disk, cpu memory, etc).
Someone else who is interested in application health builds an Application type and then builds a Consumes relationship to my PhysicalHost type, specifying the relationship should be @reverse. This lets someone expand() from the application down to the hardware it’s running on, but does not allow us to expand() the physical host entity to see all the upstream entities it reverse relates to unless I also add a ~Consumes entry to the PhysicalHost type. It feels like @reverse on the index should do this behind the scenes for me.

Again, my mental model of how this is intended to work might be very off, but it seemed like a nice idea to restrict modification of the PhysicalHost type to one set of users in my org, but allow users in other domains create relationships to it (without modifying the underlying PhysicalHost type). This would allow us to know all the upstream users to notify if we wanted to shut down a server, without having the infrastructure team keep making changes to the PhysicalHost type for all the possible upstream reverse relations.

16

Hi @francesc. This comment by @san8pai is important for our use case.

I want to use dgraph to combine both structured/typed data and unstructured/generic data like MongoDB.

I understand if I cant query or filter based on predicates that were not registered or typed, but I still want to associate arbitrary predicates to a node that were not registered and are not part of the type system and be able to retrieve those when getting the node without having to explicitely specify the predicates I want.

Is this possible? I understand this was possible before types were introduced using expand(_all_), isn’t it?

17

@MichelDiz, do you know if this is possible? It seems @francesc is not online these days :wave:

18

You can do it, but all Dgraph’s functions that needs the Type won’t work.

You can use “has()” func tho. Has does not take Type System into account.

19

Thanks, @MichelDiz. I guess what I would need is a function that gives all the predicate names for which the has() function will return true. And then returns the value of those predicates.

20

Is it possible to chain these? In my experience, this doesn’t work (at least I couldn’t get it to work). Expand may be useful for “simple” queries, but for complex structures the expand functionality is pretty limited.

I would like to be able to use expand(all) together with expand(Type) and also filters. This way I can get all the values and define what I want to be returned from related types.

For example (may contain syntax errors, just to demonstrate idea):

q(func: uid($id)){
  expand(_all_)
  expand(TypeA) {
    uid
    name
    desc
    expand(Type) {
      ...
    }
  }
  expand(TypeB) {
    uid
    title
    color
    expand(Type) {
      ...
    }
  }
  expand(_predicate_) @filter(...) {
    uid
    format
    deadline
    expand(Type) {
      ...
    }
  }
}

Or is this a stupid suggestion? :stuck_out_tongue: