Query sever timestamps in GraphQL?

Thanks! Looking forward to seeing server generated timestamps implemented.

1 Like

Yeah! you could surely do it that way for now.

2 Likes

Any update on this? I think that this functionality is super-important.

@abhimanyusinghgaur How do you guys keep track of feature requests? Is there some overview about their status?

@amaster507 Could you give an example how to do this with custom mutations? I

@maaft, 20.11 should be out as I have been informed next month. When this arrives, I believe the lambda directive will best handle this use case. This was exemplified on the October community call which you can view: https://youtu.be/XryDxyb0fT4?t=3073 ← that link starts at the lambda part.

1 Like

In that video they show an example on how to implement the @lambda directive with Slash-Graphql.

Can you ELI5 if (and potenitally how) I can use this with the “normal” dgraph-executables? (alpha and zero)

Edit: I’m seeing a --graphql_lambda_url flag for dgraph alpha :sunglasses: I guess I will start from there.

Yeah! You can have a look at this docker-compose file. You will find how to set-up a lambda service running along with Alpha and Zero. Notice how the --graphql_lambda_url flag is passed to the Alpha.

These files will also be useful:

The documentation for lambda is not yet there, but it would surely be out by 20.11 release.

Thank you very much @abhimanyusinghgaur!

Feature requests from Discuss are discussed on a weekly basis, and added to our internal backlog. A public status overview may not be available. But, till now we have posted our roadmap on an annual basis. And, going forward we have been considering all the feedback from the community to make our roadmap more visible.

About this particular one, It didn’t make it to 20.11 release, instead we added @lambda as that covers a broad range of cases. I will confirm and let you know whether we still plan to work on this particular request, as @lambda is there.

We will be working on it for the 21.03 release.

1 Like

Okay, thank you.

Is there a rough ETA for 21.03 ?

Just trying to figure out if it makes sense to start building our stack with @lambda now on the latest-release or if we should roll without @lambda for the moment.

EDIT:
I guess the only documentation for @lambda is the code itself for now?

Edit2:
Am I correct that for implementing createdAt and updatedAt timestamps I would have to mirror every add- and update- mutation to Javascript? That’s honestly very super complicated and you would have to track and maintain two files instead of just the gql-schema. I hope there will be @createdAt and @updatedAt directives instead! If I’m wrong though, I’m happy to hear otherwise.

2 Likes

21.03 = 2021 March
Major releases are named as yy.mm for the year and month when they will be available.

Yeah! At present, there is no documentation, you can follow the sample setup I had shared yesterday.
Along with that, you can also have a look at the integration tests for @lambda. The documentation for the feature is in progress, and should be out in a couple of weeks.

Yeah! with @lambda you will have to do that. That would be complex and also less performant. We will notify you as soon as we are done implementing the new features.

Alright, thank you very much for your answers! We will not use @lambda then and wait for @createdAt and @updatedAt directives instead. In the meantime, clients will just have to deliver these timestamps on their own (which of course could be a security issue).

2 Likes

Hi! Is there any progress on this?

No progress yet, But it is one of the features we are aiming for 21.03 release.

Thanks

4 Likes

Here is my updated thoughts on the matter. Can we make it mongoDB like. Mongoose v8.0.3: Schemas

With a directive @timestamps( createdAt: Boolean, updatedAt: Boolean )

It could be used the following way:

type Post @timestamps { # defaults both createdAt and updatedAt to true
  id: ID
  title: String!
}

The predicate _createdAt and _updatedAt should be added to the list of reserved names.

If a user needed to manually set or update the _createdAt or _updatedAt I believe it should be allowed through DQL only. Meaning that GraphQL does not allow any input for these values other than searching.

Cons: There is no way to timestamp edges without either using facets or a linking type. Facets are further out than 21.03 I believe and even then are not first rate citizens. Like many other implementations of timestamps, it does not indicate which predicates/edges were updated, just that there was something updated. With other databases using joining tables it is possible to update timestamps in a join table without updating timestamps on the tables that were joined.

Pros: All timestamps are handled 100% by the rewriting methods in Go. These timestamps are directly immutable by a user within GraphQL. This ensures that the _createdAt is true and was not updated by a later GraphQL mutation. Auto updating timestamps on nodes when edges are mutated, gives a sync script the ability to refresh the data for that entire node and their referenced edges.

Implementation:

  • When an add mutation is rewritten and its type has the @timestamp directive, it gets the _createdAt predicate added with the equivalent of new Date()
  • When an update mutation is rewritten and its type has the @timestamp directive, the _updatedAt predicate is set to the equivalent of new Date()
  • When an edge is created/removed that utilizes the @hasInverse directive, then both nodes on either side are updated with the _updatedAt predicate with the equivalent of new Date().

This would then generate this GraphQL schema:

type Post {
  id: ID
  title: String!
  _createdAt: DateTime!
  _updatedAt: DateTime
}
input PostFilter {
  id: [ID!]
  _createdAt: DateTimeFilter
  _updatedAt: DateTimeFilter
  has: PostHasFilter
  and: [PostFilter]
  or: [PostFilter]
  not: PostFilter
}
enum PostHasFilter {
  "title"
  "_createdAt" # seems kinda silly having a required field, but following precedents already set.
  "_updatedAt"
}
input PostOrder {
  asc: PostOrderable
  desc: PostOrderable
  then: PostOrder
}
enum PostOrderable {
  "title"
  "_createdAt"
  "_updatedAt"
}
type PostAggregateResult {
  count: Int
  titleMin: String
  titleMax: String
  _createdAtMin: DateTime
  _createdAtMax: DateTime
  _updatedAtMin: DateTime
  _updatedAtMax: DateTime
}
input AddPostInput {
  title: String!
  # Notice, do not include _createdAt and _updatedAt here because they are handled internally
}
type AddPostPayload {
  post(
    filter: PostFilter
    order: PostOrder
    first: Int
    offset: Int
  ): [Post]
  numUids: Int
}
input UpdatePostInput {
  filter: PostFilter!
  set: PostPatch
  remove: PostPatch
}
input PostPatch {
  title: String
  # Notice, do not include _createdAt and _updatedAt here because they are handled internally
}
type UpdatePostPayload {
  post(
    filter: PostFilter
    order: PostOrder
    first: Int
    offset: Int
  ): [Post]
  numUids: Int
}
type DeletePostPayload {
  post(
    filter: PostFilter
    order: PostOrder
    first: Int
    offset: Int
  ): [Post]
  msg: String
  numUids: Int
}
PostRef {
  id: ID
  title: String
  # Notice, do not include _createdAt and _updatedAt here because they are handled internally
}
type Query {
  getPost(id: ID!): Post
  queryPost(
    filter: PostFilter
    first: Int
    offset: Int
    order: PostOrder
  ): [Post]
  aggregatePost(filter: PostFilter): PostAggregateResult
}
type Mutation {
  addPost(input: [AddPostInput!]!): AddPostPayload
  updatePost(input: UpdatePostInput!): UpdatePostPayload
  deletePost(filter: PostFilter!): DeletePostPayload
}

Which would translate roughly to this DQL schema:

Post.title: string .
Post._createdAt: dateTime @index(hour) .
Post._updatedAt: dateTime @index(hour) .

Maybe the index should be configurable from the @timestamp directive, hmm…

Alternatively, the _createdAt and _updatedAt GraphQL fields could all map to the same two predicates. This would make DQL statements easier to get a list of everything that was updated filtering on a single predicate. However, this may degrade distributed performance for larger databases as the number of predicates that could be sharded is dropped leaving a large amount of work always in a single alpha/group.


After Thoughts through PM:

So would that be sortable?

Timestamps would be sortable but not mutable directly. Meaning in GraphQL you couldn’t do set: {_createdAt: "2020-01-20"} as that is reserved only for the add mutation or when doing any input ref that creates new.

What if you needed to update a time stamp? For instance for importing data with current timestamps

for importing, I think it should go 100% into DQL if you are importing something that has a preset timestamp. Anything imported using GraphQL you could create your own DateTime field and write to that instead. That would allow to do:

  • This was imported at [_createdAt]
  • The imported data was originally created at [custom importedDateTime field]

Sounds good. Just seems like there needs to be some sort of way to modify that. Not super easily of course since it should only be used in special cases

It would be modifiable through DQL. I think that is the preferred method anyways. Thinking of GraphQL as the API and DQL as the database language. It would be similar to a REST API serving MySQL. In MySQL you could update the timestamps as needed, but when using the REST API, the timestamps are all handled internally blindly, outside of your control.

5 Likes

Thank you for this well thought out push for server-side timestamps!

I love this. Would help a lot with getting only newest data and not receiving old data from sub-graphs over and over again.

Does this need to be so strict? Why not allow both? For instance, we would definitely need mutable timestamps for client ↔ server synchronization.

When implemented like this, you could choose to allow it or not:

type TimestampConfig {
   active: Boolean!
   mutable: Boolean!
}
@timestamps( createdAt: TimestampConfig, updatedAt: TimestampConfig )
1 Like

This is just my opinion on how and why I would do it this way. I am not developing this, so the core-devs would have to make the final decision in this regards by my $.02 is:

It should be strict to ensure that the API layer never does actions it should not be doing. With Dgraph GraphQL, this is a little harder to see at first. But as I explained in my edit after thoughts above, the GraphQL implementation inside of Dgraph is just that, an API. It takes the GraphQL and rewrites it using rules into DQL. In an API layer, actions such as adjusting timestamps are not permitted. If it was, then any user would be able to adjust a timetamp willy-nilly. Think of it how it may be with other database and APIs. If a timestamp is automated by the databse, then the API uses that automation of the timestamps and does not allow writing to that through the API. And looking at the implementation I wrote above, if the _updatedAt field gets set in a rewriting process into the DQL mutation, then allowing a user to also set this _updatedAt predicate could result in writing two values and then the rewriting process becomes more complex with needing then to decide when not to add in the automated predicate if it is supplied by the user. But I think the issue goes deeper than this…

I believe a client->server synchronization should be different than a server->client synchronization. Hear me out. Right now there are no good implementation that I have found for offline GraphQL. The only things really out there is client side cache of GraphQL (ie: Apollo Client). To update the client cache with a source of truth, the client should first update the source of truth (server) and then with the response update the cache. Therefore a client->server sync is not really pushing the source of truth, but is setting pieces of truths and returns the source of truth for those pieces (that would contain the timestamps). For any server<->server syncs that needs to set these timestamps, that should be done using DQL live imports and rdf streams IMO.

It would be interesting to see any implementation of client->server syncs where the client is collecting mutations and then running those in batches at a later point to perform sync. This would add more complications because the client would then be responsible for ensuring that there were no conflicts of unique ids and also require some sort of blank node implementation in GraphQL. I don’t think the GraphQL spec is ready yet for client->server sync.

For the time being, if you have a timestamp field you want to let the database control, let it control it, if you have a timestamp field that you want to control, then do so with a regular DateTime field as I stated above. This would get a timestamp feature into production quickly which could then be iterated upon for feature enhancements later.

Sure - the implementation would be a bit more complex, but is this really a deciding criterion?

I’m fine with this approach. But I fear that once it is implemented immutable-only, there won’t be a reiteration for a loooooong time given the vast amount of (important) feature- and bug-requests currently.

We didn’t find any either, that’s why we build it from scratch. We have one dgraph database running on the client (+ electron react app) and one dgraph database on our server. The user can work offline, all data is stored in his dgraph instance and when he goes online, we synchronize both databases using GQL mutations. And I can tell you - it’s working just fine and isn’t much effort either when using code-generation. And that’s why we would need mutable timestamps. When the user creates a post offline on date XXX, the same data should show online on that post.

I’d also happily implement the synchronization with DQL if you think that this is better suited. Can you point me to some ressources where I can learn how to use “DQL live imports and rdf streams” for server<->server synchronization purposes?

@amaster507 Thank you for the detailed post. What you have here is pretty good and can be implemented pretty much as it is.

Yeah, we’d properly like to have the predicates named as Type._createdAt and Type._updatedAt so that they can be sharded across Alpha nodes as the data grows.

I also agree that they should be set automatically and shouldn’t be exposed via the GraphQL API. This is also because the GraphQL API is supposed to be used by browsers that are clients and it’s not a good practice to be setting timestamps via the client.

1 Like

Sorry, why do you want to limit clients to browsers? What about my @custom logic resolvers? They are doing a lot of stuff that dgraph-gql can’t do (and might never be able to do) and use GQL clients to write back to my dgraph instance. They are running on my servers, so I trust them to write correct timestamps.

Can’t we find some kind of agreement here? E.g. GQL-clients can change timestamps when they send some kind of authentication header?

The thing is, that this decision is very important for us. It will decide if we have to learn DQL, throw month of developer work away and rewrite our complete synchronization logic. And if it is like this, we better start yesterday than next week.

Also if you decided to make it read-only, I need to be sure that server<->server sync can be achieved (with partial data) with DQL. Can you give your opinion on this @pawan ?

1 Like