What version of Dgraph are you using?
20.07.02
java client: 20.03.2(can’t not find 20.07 version)
Have you tried reproducing the issue with the latest release?
This is the latest release
What is the hardware spec (RAM, OS)?
os: linux
cluster info : 3 machines with 140G memory and 16 cores
Steps to reproduce the issue (command/config used to run Dgraph).
zero
dgraph zero --my xxxx:5080 --replicas 3
alpha
dgraph alpha --whitelist xxxx --lru_mb 42000 --my xxxx:7080 --zero xxxx:5080 --snapshot_after 1000 --graphql_debug
I spent large time to look for the answer about TxnConflictException. All guys say if you concurrent modify the same node, your will get this error. But I’m sure I don’t concurrent modify the same node. My scene is I will migrate large user relations to Dgraph, about six hundred million users, six billion edges. So, I must concurrent insert nodes to Dgraph in order to save times. For simplify the question. I just concurrent batch insert nodes( one batch 1000 nodes), and this nodes just one predicate. the schme and snippet code below:
schema
arith_name:int @index(int) @count @upsert .
type ARITH_U {
arith_name
}
java client version
<dependency>
<groupId>io.dgraph</groupId>
<artifactId>dgraph4j</artifactId>
<version>20.03.2</version>
</dependency>
test code
Divide from 0 to 100000000L, each size 1000 , then use 10 threads concurrent insert.
public void test1() throws InterruptedException {
long start = 0;
long end = 100000000L; //
int worker = 10;
int page = 1000;
ThreadPoolExecutor executorService = (ThreadPoolExecutor) Executors.newFixedThreadPool(worker);
for (; start < end; ) {
long taskStart = start;
start += page;
long taskEnd = start;
executorService.execute(() -> {
List<Long> names = new ArrayList<>();
for (long k = taskStart; k < taskEnd; k++) {
names.add(k);
}
upsertAddUsersWithRDFFormatById(names, taskStart, taskEnd);
});
}
executorService.shutdown();
while (!executorService.awaitTermination(1, TimeUnit.MINUTES)) {
System.out.println(executorService);
}
}
public void upsertAddUsersWithRDFFormatById(Collection<Long> ids, long start, long end) {
try {
if (ids.size() < 1) {
return;
}
StringBuilder sbQuery = new StringBuilder();
StringBuilder sbMutation = new StringBuilder();
sbQuery.append("query ");
sbQuery.append("{");
int index = 1;
for (Long id : ids) {
sbQuery.append("user").append(index).append(" as var(func:eq(arith_name, ")
.append(id).append(")){\n");
sbQuery.append("}\n");
sbMutation.append(String.format("uid(%s) <arith_name> \"%s\" .\n", "user" + index, id));
sbMutation.append(String.format("uid(%s) <dgraph.type> \"ARITH_U\" .\n", "user" + index));
index++;
}
sbQuery.append("}");
DgraphProto.Mutation mutation = DgraphProto.Mutation
.newBuilder()
.setSetNquads(ByteString.copyFromUtf8(sbMutation.toString()))
.build();
DgraphProto.Request request = DgraphProto.Request
.newBuilder()
.setQuery(sbQuery.toString())
.addMutations(mutation)
.setCommitNow(true)
.build();
Transaction transaction = dgraphClient.newTransaction();
try {
DgraphProto.Response response = transaction.doRequest(request);
// System.out.println(response);
} finally {
transaction.discard();
transaction.close();
}
} catch (Exception e) {
if (NestedExceptionUtils.getRootCause(e) instanceof TxnConflictException) {
logger.error(String.format("[CONFLICT] start:%d, end:%d, ids:%s", start, end, ids), e);
}else {
logger.error("unknown error", e);
}
}
a little while，I got below errors. As you see, the node inserted are irrelevantly. so I think this is a serious bug in dgraph or dgraph java client.
java.util.concurrent.CompletionException: io.dgraph.TxnConflictException: Transaction has been aborted. Please retry
at io.dgraph.AsyncTransaction.lambda$doRequest$2(AsyncTransaction.java:187) ~[dgraph4j-20.03.2.jar:?]
at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822) ~[?:1.8.0_191]
at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797) ~[?:1.8.0_191]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_191]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1595) ~[?:1.8.0_191]
at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1582) ~[?:1.8.0_191]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) ~[?:1.8.0_191]
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) ~[?:1.8.0_191]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) ~[?:1.8.0_191]
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) ~[?:1.8.0_191]
Caused by: java.util.concurrent.CompletionException: io.dgraph.TxnConflictException: Transaction has been aborted. Please retry
at io.dgraph.DgraphAsyncClient.lambda$runWithRetries$2(DgraphAsyncClient.java:210) ~[dgraph4j-20.03.2.jar:?]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_191]
... 5 more
Caused by: io.dgraph.TxnConflictException: Transaction has been aborted. Please retry
at io.dgraph.DgraphAsyncClient.lambda$runWithRetries$2(DgraphAsyncClient.java:210) ~[dgraph4j-20.03.2.jar:?]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_191]
... 5 more
Every node is irrelevant, but throw conflict exception, Isn’t that strange？
Please let us know if we can help in any other aspects.