Upsert Failed for unique data

BigBunnie · October 27, 2020, 12:27pm

Hi Team,

we are facing a transaction aborted error while mutating in a concurrent manner, where my data set is completely unique, so no chances of updating the same node.
here are my upsert blocks via goroutine ( tested with 3 routines, out of 2 failed)

go routine - 1 ( aborted)

upsert{
	query{
		var(func: eq(product.swidtag,"ph02")) @filter(eq(type_name,"product")){
                product as uid
      }
		var(func: eq(product.swidtag,"Smart Opt2")) @filter(eq(type_name,"product")){
                        child as uid
      }
	}
	mutation {
	 set{ 
		uid(product) <product.swidtag> "ph02" .
		uid(product) <type_name> "product" .
		uid(product) <dgraph.type> "Product" .
		uid(product) <scopes> "m2" .
		uid(child) <product.swidtag> "Smart Opt2" .
		uid(child) <type_name> "product" .
		uid(child) <dgraph.type> "Product" .
		uid(child) <scopes> "m2" .
		uid(child) <product.child> uid(product) . 
	 }
	}
}

===================
go routine - 2 ( aborted)

upsert{
	query{
		var(func: eq(product.swidtag,"ph03")) @filter(eq(type_name,"product")){
                product as uid
      }
		var(func: eq(product.swidtag,"Smart Opt3")) @filter(eq(type_name,"product")){
                        child as uid
      }
	}
	mutation {
	 set{ 
		uid(product) <product.swidtag> "ph03" .
		uid(product) <type_name> "product" .
		uid(product) <dgraph.type> "Product" .
		uid(product) <scopes> "m3" .
		uid(child) <product.swidtag> "Smart Opt3" .
		uid(child) <type_name> "product" .
		uid(child) <dgraph.type> "Product" .
		uid(child) <scopes> "m3" .
		uid(child) <product.child> uid(product) . 
	 }
	}
}

================
go routine - 3 ( pass)

upsert{
	query{
		var(func: eq(product.swidtag,"ph01")) @filter(eq(type_name,"product")){
                product as uid
      }
		var(func: eq(product.swidtag,"Smart Opt1")) @filter(eq(type_name,"product")){
                        child as uid
      }
	}
	mutation {
	 set{ 
		uid(product) <product.swidtag> "ph01" .
		uid(product) <type_name> "product" .
		uid(product) <dgraph.type> "Product" .
		uid(product) <scopes> "m1" .
		uid(child) <product.swidtag> "Smart Opt1" .
		uid(child) <type_name> "product" .
		uid(child) <dgraph.type> "Product" .
		uid(child) <scopes> "m1" .
		uid(child) <product.child> uid(product) . 
	 }
	}
}

Here is schema

<product.child>: [uid] @count @reverse .
<product.name>: string @index(exact, trigram) .
<product.swidtag>: string @index(exact, trigram) @upsert .
<scopes>: [string] @index(exact) .
<type_name>: string @index(exact) .

I have tried without var block as I have seen in other threads, that didn’t work
tried with immediate commit true so that node will be created asap.

anand · October 28, 2020, 3:45am

Welcome Keshav!
I believe this is a bug. Note the trigram index on the field with an @upsert tag. The @upsert tag tries to bring in the value of data being stored into the conflict detection mechanism. I believe the trigram index is comparing parts of the value (“Smart Opt1, Smart Opt2”) and finds matches.
I was able to get two of the transactions to go through after removing the trigram index. Could you please try your go routines after removing the trigram index and let us know?

Edit
After further review, we found that this is not a bug and is actually working as designed.

anand · October 28, 2020, 6:46am

Hi Keshav, On subsequent review, I can confirm that this is not a bug. The upsert directive with trigram index is expected to behave this way.

In this use case, you may want to choose only the exact index to support upsert in detecting conflicts.

BigBunnie · October 28, 2020, 6:49am

Hi Anand, Thanks for the solution, It worked. I removed the trigram index from those predicates who has upsert tag. But it means upsert tag cann’t be used with trigram based index?

BigBunnie · October 28, 2020, 9:22am

Hi Anand I tried the solution by removing the trigram index but while upserting below mutation still getting aborted error. but this frequency is not constant, like 3 times out of 5 coming aborted.

routine 1 ( failed)

upsert{
	query{
		var(func: eq(product.swidtag,"ph02")) @filter(eq(type_name,"product")){
                product as uid
      }
		var(func: eq(product.swidtag,"Smart Opt2")) @filter(eq(type_name,"product")){
                        child as uid
      }
	}
	mutation {
	 set{ 
		uid(product) <product.swidtag> "ph02" .
		uid(product) <type_name> "product" .
		uid(product) <dgraph.type> "Product" .
		uid(product) <scopes> "m2" .
		uid(child) <product.swidtag> "Smart Opt2" .
		uid(child) <type_name> "product" .
		uid(child) <dgraph.type> "Product" .
		uid(child) <scopes> "m2" .
		uid(child) <product.child> uid(product) . 
	 }
	}
}

==========
routine - 2 ( failed)

upsert{
	query{
		var(func: eq(product.swidtag,"ph03")) @filter(eq(type_name,"product")){
                product as uid
      }
		var(func: eq(product.swidtag,"Smart Opt2")) @filter(eq(type_name,"product")){
                        child as uid
      }
	}
	mutation {
	 set{ 
		uid(product) <product.swidtag> "ph03" .
		uid(product) <type_name> "product" .
		uid(product) <dgraph.type> "Product" .
		uid(product) <scopes> "m2" .
		uid(child) <product.swidtag> "Smart Opt2" .
		uid(child) <type_name> "product" .
		uid(child) <dgraph.type> "Product" .
		uid(child) <scopes> "m2" .
		uid(child) <product.child> uid(product) . 
	 }
	}
}

==============

routine -3 ( failed)

upsert{
	query{
		var(func: eq(product.swidtag,"ph09")) @filter(eq(type_name,"product")){
                product as uid
      }
		var(func: eq(product.swidtag,"Smart Opt2")) @filter(eq(type_name,"product")){
                        child as uid
      }
	}
	mutation {
	 set{ 
		uid(product) <product.swidtag> "ph09" .
		uid(product) <type_name> "product" .
		uid(product) <dgraph.type> "Product" .
		uid(product) <scopes> "m2" .
		uid(child) <product.swidtag> "Smart Opt2" .
		uid(child) <type_name> "product" .
		uid(child) <dgraph.type> "Product" .
		uid(child) <scopes> "m2" .
		uid(child) <product.child> uid(product) . 
	 }
	}
}

I have removed trigram, but still getting transaction aborted for these. In mutations I have added conditional upsert also via checking the length of the node then only mutate,[@if(eq(len(child),0)].

And sometimes it works fine but sometimes not. Here the child of a node is the same for all three products.

anand · October 28, 2020, 9:29am

The @upsert directive is still operating with exact index on the <product.swidtag> attribute. So the concurrent mutations with the pattern * <product.swidtag> "Smart Opt2" . will trigger conflict. If you are confident that input data does not contain duplicates, you might want to drop @upsert directive.

BigBunnie · October 28, 2020, 10:23am

so we shouldn’t use @upsert tag with any index? , If so then the concurrent mutation will not trigger conflict? or vise versa!!

anand · October 28, 2020, 10:28am

An index is mandatory for the @upsert directive to work.

BigBunnie · October 28, 2020, 12:48pm

Okay, but a few of my data set are duplicate but not complete like one predicate is repeated (Smart Opt2) in all three data set, but I am using conditional upsert as if that node is already created or not, if not then mutate only else just link them. then how to handle this situation?
And does there is any locking mechanism while mutation?

anand · October 28, 2020, 1:18pm

If you are having failures due to transaction aborts, simply retry them several times. In the first iteration, one of the transactions (with “Smart Opt2”) will go through and create new nodes, and the others will abort. Upon retry, the linking mechanism you have put in will kick in as the query will find data (query for “Smart Opt2”) and link between the products can be established .

BigBunnie · November 3, 2020, 5:10am

Anand, Is @upsert directive is mandatory for upsert Because I am upserting same data without @upsert tag but trigram index, It’s creating two different nodes for same data ( which is supposed to create a single node and link to another node) and I am also using conditional mutation

anand · November 3, 2020, 5:16am

Hi Keshav, @upsert is not mandatory for the upsert block. In case you are still facing issues, please create a new post with exact details of the schema, the data you are inserting, the expected output and what you actually got. It will clear out some of the confusion and help to answer your question effectively.

BigBunnie · November 3, 2020, 5:17am

Okay , thank you Anand.

Topic		Replies	Views
Unique Identifer based on specified unique predicates Dgraph mutation , example	3	895	June 11, 2019
Writing only unique nodes based on values not working Users	3	403	May 16, 2019
Upserting for a list of predicates Dgraph	10	671	June 24, 2020
Upsert resulting duplicate node Dgraph dgraph	8	570	January 29, 2022
"Transaction has been aborted. Please retry" when creating nodes with different predicate values Dgraph kind:question	3	962	September 11, 2020

Upsert Failed for unique data

Related topics