Major changes in v0.9


(Peter Stace) #42

Hi @calummoore. You’re correct, the transaction (and the mutation made within it) hasn’t been committed. You have to use the /commit endpoint (passing the start_ts as a header) to commit the transaction.

There are some instructions here on how to use the raw HTTP endpoints (I updated them last night, so you were possibly looking at an old version).

EDIT: To make clear: all mutations and queries are now executed inside transactions.


(Calum Moore) #43

Thanks for the quick response!

Ah, I didn’t realize, might be worth pointing that out in the documentation, as I didn’t see anything on it. Does that mean we now run 2 x HTTP calls in order to run any mutation?

Also, I’m not exactly clear on what lin_read is doing, or what I should do with it.

lin_read is initialized by copying the client’s lin_read

I think this means copy the clients lin_read into the transactions lin_read. But what should the initial client lin_read be? Can I randomly set it the first time? Do I ever update it independently (e.g. add a new item to the map), or should I only ever merge from a response?

Each transaction needs its own lin_read

The lin_read in the response is {“1”: 14}. The merged result is {“1”: 14}, since we take the max all of the keys.

It looks like at this point you just want me to merge it with the transaction’s lin_read only but as far as I can see lin_read is only sent in the header for query and not the mutation/commit? So what am I merging it for?

We get another lin_read map, which needs to be merged (the new lin_read map for both the client and transaction

Ok, so when I get a lin_read back from a mutation, then I merge it with the clients lin_read?

Notice that we receive a commit_ts in the response

Ok, do you want me to do anything with that?


Sorry for so many questions! Just trying to get a handle on this. :slight_smile:

Thanks!


(Manish R Jain) #44

If you only want to send a mutation and immediately commit it, you can send a “Commit-Now” header during mutation.
That would incur only a single HTTP call. @peter is adding that part in the documentation.

lin_read enables linearizable reads. This is not something that we expect our users to care about or understand, it’s meant to be hidden away in the client code. The raw HTTP, OTOH, would enable a contributor to build a client in a new language, hence that section explains what to do with it. Basically, the idea is that every time you get back the lin read map, that needs to be merged back into the txn lin reads map, and the client lin reads map, taking the max value for each entry. And that lin reads map must be passed into the queries. This would allow a replica which might be behind the leader, to catch up to the read index, before it replies back. This is important in a replicated setup, if you wish to see your updates made to leader, which might not have been yet applied to the follower.

You can ignore all that if you don’t have a replicated setup. Something that we’ll a note about in the docs.

We’re doing one more change to simplify the HTTP interaction. @pawan is working on that. Expect that to be out by end of day today.


(Peter Stace) #45

@calummoore, thanks for the feedback on the docs, I’ve updated them to be a bit clearer.

To quickly answer your questions though (look at the now updated docs for more details):

Does that mean we now run 2 x HTTP calls in order to run any mutation?

Normally yes, although you can get around this by using the X-Dgraph-CommitNow header.

I think this means copy the clients lin_read into the transactions lin_read.

Correct.

But what should the initial client lin_read be?

It should be initially empty.

Do I ever update it independently (e.g. add a new item to the map), or should I only ever merge from a response?

Only from a response.

Ok, so when I get a lin_read back from a mutation, then I merge it with the clients lin_read?

Correct. lin_read maps from all mutations should be merged with the client lin_read, and also with the lin_read for the transaction the mutation was part of.

Ok, do you want me to do anything with that?

Nothing at all. The statement about commit_ts is probably not useful, so I’ve removed it.


(Calum Moore) #46

Thanks both, very helpful! :slight_smile:

I’m building myself a mini-Javascript client, so this is useful.

It seems like it’s a case of always update the client’s lin_read to the maximum values received from any transaction/request.

Any new requests use the client’s lin_read to start, and any on-going transactions can use the latest lin_read they received in a response related to that transaction (even if it’s not as high as the client’s lin_read).


(Manish R Jain) #47

If you’re building a mini-JS client, I’d recommend doing it in Grpc :-).


(Calum Moore) #48

I’ve created a JS client, all seems to be working fine, but when I test a conflicting transaction, it seems like the grpc is aborting with Transaction Aborted. Is that the expected behaviour, or should I receive a JSON response { aborted: true }?


(Manish R Jain) #49

This is amazing! Thanks a lot for this, Cal! We don’t have any JS engineer in our current team, so this comes at a great time.

We’d like to promote this as the default JS client. Before we do so, can you please write integration tests for bank and acctupsert? (Possibly others, but these two would do).

Yeah, that’s right. We just return an error for aborted, which is provisioned by Grpc. If it’s hard to parse, let us know – and we can try to make it friendlier to JS.


(Calum Moore) #50

No problem :slight_smile: , Dgraph is awesome and you guys have been incredible at adding new features!

Happy to add the integration tests, I assume you just want me to replicate these tests into JS, or do you need me to match it to a Go produced output?

It’s not a problem to error on aborting the transaction, I can handle it on the JS side. At the moment I just pass the errors on.


(Manish R Jain) #51

Thanks for your kind words. It always feels amazing when our users find Dgraph useful.

Yeah, the idea is to port these to JS, so they could act as integration tests for JS client.


(Manish R Jain) #52

Hey @calummoore,

We have a couple of JS engineers joining the team. If you’re okay with it, we could potentially fork your JS client, and base our official JS on.


(Calum Moore) #53

Great news, no problem. If you’d rather I transferred ownership of the repo, also happy to do that.


(Marc Erdmann) #54

Hi, any news regarding the JS client?


(Pawan Rawal) #55

Hi @MarcErdmann

The JS client code is ready at https://github.com/dgraph-io/dgraph-js. It has working integration tests but needs more documentation. Expect that to be done in the next couple of days.


(Marc Erdmann) #56

Wow, that are great news! Thanks for the awesome work


(Michel Conrado (Support Engineer)) #57

@mrjn I have few questions about the transactions.

Is there a way, or will there be one, to query transactions without being committed? I mean, if we somehow “lose” the information because of a technical problem, we lose “start_ts” and so on. How to proceed? how to “list” the uncommitted transactions? - if applies*.

Will there be a “timer” commit limit? Is there a way to define a limit ourselves? Could we define “rules” for commitment to happen automatically? - if applies*.

And if we need to revert the transaction without needing a new mutation. There could be something like “uncommit” / “revert” / “checkout” / “Undo”/“rollback”? and the commitment “rules” could be applied too for this question.

And are there more future plans for something new* with transactions?

Of course all these things from my questions should be a Queries / mutations affected by admin. Maybe only “reverted” commitment transactions should be informed to clients.

ps. Maybe my questions are silly. But they seem to be pertinent to me. Because I see them related to ACID logic idea.

Best Regards.


(Manish R Jain) #58

There are a lot of questions here, I can’t answer all of them. We’re writing a paper which explains how transactions work, so maybe when that comes out, you could read more about that. But, let me handle the top question here. Feel free to ask any specific questions which might affect how you interact with Dgraph; but as per design related questions, I’d wait for the paper.

No, there’s no way to query a list of pending txns across the cluster. If you lose the start ts somehow, then the best way forward is just to create a new txn and make the changes there. The pending txns automatically get cancelled at some point.