Major changes in v0.9


(Manish R Jain) #1

v0.9 is introducing support for transactions, which is a MAJOR change throughout the codebase, and significantly affects how users interact with Dgraph.

HTTP /query endpoint is going away

All the queries and mutations would be done via a transaction API, which is going to be coordinated via Dgraph clients. As such, we have significantly simplified the clients, so that we can easily replicate the functionality and maintain various clients for different languages.


Update (1/11) - We have heard your feedback and brought back the http /query endpoint. Infact we have full support for transactions using http.

We will have the following endpoints

  1. POST /query to do queries.
  2. POST /mutate to do mutations. Mutation block doesn’t require the mutation keyword anymore since we have a separate endpoint for it.
  3. POST /commit to commit a transaction.
  4. PUT /abort to abort a transaction.
  5. PUT /alter to drop a predicate, drop all data and do schema mutations.

Queries and mutations are separate

Queries would no longer allow a mutation { ... } block. All mutations would be done via a Mutate Grpc endpoint, called via the client or the /mutate http handler.

Mutations would accept set / delete JSON, or set / delete list of NQuads, and various other operations.

/mutate http handler would only accept raw mutations but without the mutation keyword.
E.g.

{
	set {
		<name> <is> <something> .
		<hometown> <is> <san/francisco> .
	}
	delete {
		<name> <is> <something-else> .
	}
}

No more upsert and mutation variables

With the availability of transactions, a user can query for data, and then write their mods back atomically. Thus, upsert and mutation variables can be done via client-side logic, instead of baking this limiting functionality in the server.

JSON over protobuf messages

JSON is taking a prominent role. All responses would now be in JSON format, wrapped in a protobuf along with other information, useful to the client. We would no longer encode the subgraph in proto messages.

We’re getting rid of the dgraph tag in Go structs, which used to help unmarshal these proto messages. Now Dgraph’s reponse can just be unmarshaled using standard JSON.

Schema mutations

Schema mutations would be fed via a new Alter GRPC endpoint. It would take a schema block string, and execute it. All the parsing would be done at the server side. This endpoint would also be used to drop a certain predicate or to drop the entire DB.

Breaking: Given this, sending a <*> <pred> <*> . delete mutation to delete a predicate wouldn’t be allowed via Mutate endpoint.

Live Loader [under consideration]

Dgraph live loader should no longer store the external ids in Dgraph. That seems like a very specific solution for a broader problem. So, a user could use the client directly to load data and achieve custom functionality.

Dgraph Flag Changes

A change went in today to automatically allocate Raft ids to Dgraph servers. A user can optionally still specify an ID, via --idx flag. But, if this flag is not specified, auto-allocation would happen.

--peer flag which was used to specify another Dgraph instance’s IP address is being replaced by --zero flag. This flag would need the IP address : Port of a Dgraph zero instance.

--groups, --gentlecommit and a couple of other flags are now removed.

Embedded Dgraph goes away

We haven’t seen much usage of this feature. And it adds unnecessary maintenance overhead to the code. If there are clear reasons for its benefits, let us know (https://github.com/dgraph-io/dgraph/pull/1734).

Go client

Facets

  • Facets response structure has been modified and is a lot flatter. Facet key is now predicate:facet_name.

Retrieving uid

Instead of fetching uid using _uid_, you can just use uid.

Diff of changes https://github.com/dgraph-io/dgraph/compare/release/v0.8.3...release/v0.9.0?expand=1


Give us your feedback

If there are specific things you need, let us know; so we can discuss them and potentially incorporate them before the release.


Transactions in Dgraph
Why can't Dgraph work well when I kill one node in the cluster
(Manish R Jain) #2

(Manish R Jain) #4

One of the side effects of transactional support is that indices can frequently result in collisions among concurrently running transactions updating the same keys, and hence aborted transactions. So, the fewer indices, the better write throughput can be achieved.


(Jeff Hull) #5

Thanks @mrjn for this writeup. These sound like some really positive changes, and transactions will definitely help my team’s product by allowing us to abort midway through a group of statements if we need to.

My main concern right now is migrating away from the /query endpoint to whatever client we should be using in v0.9. Our server runs on node currently. I see from the documentation that it looks like official clients support Go, Python, and Java. Will the dgraph team be supporting an official nodejs client for v0.9, or maybe an HTTP client could be provided for languages that do not yet have an official supported client?


(Lazhar Ichir) #6

I am confused – is the API /query endpoint disappearing totally? So, my own current PHP APIs and microservices that were querying my dgraph through HTTP endpoint won’t work with 0.9?

Ouch, that hurts and is definitely problematic…


(Peter Stace) #7

Yes, that is the plan, to remove the /query endpoint. Users will interact with dgraph only via clients or directly via gRPC.

We definitely want to work closely with users though to come up with a solution that will work smoothly for everyone.

Is it mutations that you are doing via the HTTP endpoint, or just read-only queries?


(Jeff Hull) #8

Hi @peter, as for us, we use strictly the query endpoint for all queries and mutations. For a server running on node, what do you think the smoothest path to migration would be after you guys release v0.9? Direct interaction with grpc or would the dgraph team provide a node client?

Sounds like I may need to do some reading on grpc. Never used it before.


(Peter Stace) #9

Ideally, there will be a JS client made available, either by the dgraph team or as a community effort (there was some talk of this in Slack). Grpc would always be there as an alternative though.

EDIT: We’re definitely going to have a JS client.


(Abbas) #10

@peter So you mean as mentioned in the Dgraph Documentation, the Go (gRPC) way will still work and Go (HTTP) will not right ? as given below in the examples:

https://docs.dgraph.io/query-language/#uid


(Peter Stace) #11

@Abbas, yes that’s correct. Go (or any other language with grpc support) will be able to interact via the grpc interface. The HTTP interface will be removed.

We also have a Go client which is nicer to work with compared to the grpc interface. Clients will likely become available for other languages as well, although they wouldn’t be strictly required since it’s always possible to use grpc directly.

EDIT: We’re going to have a JS client as well.


(Lazhar Ichir) #12

I do a lot of both:

  • writes/mutations (mainly upserts, which are being removed from what I
    understand) to update my knowledge graph (one microservice)
  • reads to work with it (most other ones just read)

I am not in production yet as I am unsure which graph db to use but this
move extremely tough. I was seeing the HTTP Endpoint as a selling point
actually.

Will wait and see how it is evolving, thanks for keeping everybody updated
though!


(Manish R Jain) #13

If HTTP endpoint is something folks really want, then we could bring a query-only (read-only) version of it back. It won’t be transactional, so mutations would need to be run via Grpc. Not sure how helpful that would be; but if it is, let us know.

We’d definitely going to have Go, JS and Java client in shape before the release. JS is going to be one of our top priorities, right next to Go. So, with these clients support, it shouldn’t be something that Dgraph users need to worry about. It would require breaking up queries from mutations, but in the long run, that’s going to avoid complexity and potential for injection attacks.


(Jeff Hull) #14

Great news for us :slight_smile:

I guess this is moot for my team now since you guys will be supporting a JS client. If I were interacting with gRPC for mutations, personally I would want to stick with gRPC for everything including queries to keep to one protocol instead of two.


(Lazhar Ichir) #15

I assume a PHP client is not a priority? :eyes:


(Wolf Dan) #16

Looks like an amazing update!

But sadly there is no usable Grpc libraries for Elixir e-e so seems like need to do Grpc library and Dgraph /sighs


(Manish R Jain) #17

We have a very small team, and none of us have PHP experience. That’s something we could really use help with.

Unfortunately, Grpc doesn’t support Elixir. We would definitely come up with some solution later on, but it might not be part of the v0.9 release.


(Michel Conrado (Support Engineer)) #18

What about this one? https://github.com/tony612/grpc-elixir

more about the link above https://speakerdeck.com/tony612/the-way-to-grpc-elixir


(Michel Conrado (Support Engineer)) #19

@mrjn When the 0.9 comes?


(Pawan Rawal) #20

Hi @MichelDiz

You can expect the v0.9 release by mid-November.


(Michel Conrado (Support Engineer)) #21

Thank you!
One more thing.
Technically, the changes are drastic?
And If so, will there be a guide in the documentation on migrating from 0.8.X to 0.9?