Crash when online write

Copied over from Crash when online write [2]

@JimWen wrote

Following the issue Crash when online write

@ibrahim How can i use the dgraph debug to find reason, the sst file is too big, it’s hard to print all the keys, could you give me some advice?

@ibrahim It seems that ‘dgraph.type’ is too big, i have dump an alpha node data whose sst file is bigger than 65M here, but i have no idea how to find the duplicate keys.

The file is here
link:百度网盘-链接不存在 password:o3h7
A same file has been sent to your email.

Wait for your help.

@JimWen I looked at the shared p directory and I see you have many dgraph.TypeAction keys. The p directory you shared has around 850,000 dgraph.TypeAction keys. These are duplicate keys and that’s why you have big tables.

@martinmr @ashishgoswami do you know why would we have so many duplicate keys?

@JimWen can you please share your complete schema?

Yes, the complete schema is as folllowings:

curl ‘localhost:8080/alter?runInBackground=true’ -XPOST -d $’
type DeviceId {
gid_device
}

type Idfa {
gid_idfa
}

type DID {
gid_did
}

type Ip{
gid_ip
}

type Gps {
gid_gps
loc
}

type Wifi {
gid_wifi
}

type Mac {
gid_mac
}

type Job {
gid_job
title
}

type Brand {
gid_brand
}

type Company {
gid_company
}

type Interview {
gid_interview
}

type Expect {
gid_expect
}

type User{
gid_user
src
}

type Action{
aid
t
bg
chl
pl
net
status
msg
extmsg
from
to
withDeviceId
withIdfa
withDID
withIp
withGps
withWifi
withMac
withJob
withBrand
withCompany
withInterview
withExpect
}

gid_device: string @index(hash) @noconflict .
gid_idfa: string @index(hash) @noconflict .
gid_did: string @index(hash) @noconflict .
gid_ip: string @index(hash) @noconflict .

gid_gps: string @index(hash) @noconflict .
loc: geo @index(geo) @noconflict .

gid_wifi: string @index(hash) @noconflict .
gid_mac: string @index(hash) @noconflict .

gid_job: int @index(int) @noconflict .
gid_brand: int @index(int) @noconflict .
gid_company: int @index(int) @noconflict .
gid_interview: int @index(int) @noconflict .
gid_expect: int @index(int) @noconflict .

title: string @noconflict .
extmsg: string @noconflict .

gid_user: int @index(int) @noconflict .
src: int @noconflict .

aid:string @noconflict .
bg:int @noconflict .
t:int @noconflict .
chl:int @noconflict .
pl:string @noconflict .
net:string @noconflict .
status:int @noconflict .
msg:string @noconflict .
from:uid @reverse @noconflict .
to:uid @reverse @noconflict .
withDeviceId:uid @reverse @noconflict .
withIdfa:uid @reverse @noconflict .
withDID:uid @reverse @noconflict .
withIp:uid @reverse @noconflict .
withGps:uid @reverse @noconflict .
withWifi:uid @reverse @noconflict .
withMac:uid @reverse @noconflict .
withJob:uid @reverse @noconflict .
withBrand:uid @reverse @noconflict .
withCompany:uid @reverse @noconflict .
withInterview:uid @reverse @noconflict .
withExpect:uid @reverse @noconflict .
’ | python -m json.tool | less

And We use self-def uid to generate rdf data and overwrite data, so there would have duplicate writes. Some writing sample is as followings:

<0x29c56dd695c04710> <dgraph.type> "Action" . <0x29c56dd695c04710> <from> <0x4000005a186d2> . <0x29c56dd695c04710> <aid> "detail-geek-addfriend" . <0x29c56dd695c04710> <t> "1595242741" . <0x29c56dd695c04710> <bg> "1" . <0x29c56dd695c04710> <chl> "13" . <0x29c56dd695c04710> <pl> "Android" . <0x29c56dd695c04710> <net> "4G" . <0x29c56dd695c04710> <withDeviceId> <0x18000001dcbacb> . <0x29c56dd695c04710> <withIp> <0x14000075884f6d> . <0x29c56dd695c04710> <withGps> <0x24000730100a3f> . <0x29c56dd695c04710> <withJob> <0x8000004215430> . <0x29c56dd695c04710> <withExpect> <0x100000018dc5ac> . <0x29c56dd695c04710> <to> <0x40000026d7783> . <0x29c56dd198140511> <dgraph.type> "Action" . <0x29c56dd198140511> <from> <0x400000706b433> . <0x29c56dd198140511> <aid> "quick-reply" . <0x29c56dd198140511> <t> "1595242740" . <0x29c56dd198140511> <bg> "0" . <0x29c56dd198140511> <chl> "15" . <0x29c56dd198140511> <pl> "Android" . <0x29c56dd198140511> <net> "4G" . <0x29c56dd198140511> <withDeviceId> <0x18000002e22b44> . <0x29c56dd198140511> <withIp> <0x14000075883f82> . <0x29c56dd198140511> <withGps> <0x240007263f5d59> . <0x29c56dd198140511> <to> <0x400000214ad39> . <0x29c56dd1308d9f90> <dgraph.type> "Action" . <0x29c56dd1308d9f90> <from> <0x4000000c110a6> . <0x29c56dd1308d9f90> <aid> "detail-geek-addfriend" . <0x29c56dd1308d9f90> <t> "1595242740" . <0x29c56dd1308d9f90> <bg> "1" . <0x29c56dd1308d9f90> <chl> "13" . <0x29c56dd1308d9f90> <pl> "Android" . <0x29c56dd1308d9f90> <net> "4G" . <0x29c56dd1308d9f90> <withDeviceId> <0x180000000653eb> . <0x29c56dd1308d9f90> <withIp> <0x140000df68d41d> . <0x29c56dd1308d9f90> <withGps> <0x24000733c1b1dd> . <0x29c56dd1308d9f90> <withJob> <0x8000004947c7f> . <0x29c56dd1308d9f90> <withExpect> <0x100000079e8621> . <0x29c56dd1308d9f90> <to> <0x4000000b63dd1> . 
<0x4000000c69a46> <gid_user> "13015622" . <0x4000000c69a46> <dgraph.type> "User" . <0x4000000c69a46> <src> "0" . <0x1800000027a53e> <gid_device> "150588292923005512" . <0x1800000027a53e> <dgraph.type> "DeviceId" . <0x140000728a13b5> <gid_ip> "114.138.19.181" . <0x140000728a13b5> <dgraph.type> "Ip" . <0x18000001a9fdf6> <gid_device> "DDB54D8299ACEA9FB14ACBB6BB4EF0BA" . <0x18000001a9fdf6> <dgraph.type> "DeviceId" . <0x4000006f4f034> <gid_user> "116715572" . <0x4000006f4f034> <dgraph.type> "User" . <0x4000006f4f034> <src> "0" . <0x100000032b363d> <gid_expect> "53163581" . <0x100000032b363d> <dgraph.type> "Expect" . <0x400000473e362> <gid_user> "74703714" . <0x400000473e362> <dgraph.type> "User" . <0x400000473e362> <src> "0" . <0x180000000492f3> <gid_device> "158357062644057606" . <0x180000000492f3> <dgraph.type> "DeviceId" . <0x2400073010ed61> <gid_gps> "ws11vc1" . <0x2400073010ed61> <dgraph.type> "Gps" . <0x2400073010ed61> <loc> "{'type':'Point','coordinates':[114.161336,22.813517]}" . <0x2400073010ed61> <gid_gps> "ws11vc1" . <0x2400073010ed61> <dgraph.type> "Gps" . <0x2400073010ed61> <loc> "{'type':'Point','coordinates':[114.161284,22.813414]}" . <0x40000024b6312> <gid_user> "38494994" . <0x40000024b6312> <dgraph.type> "User" . <0x40000024b6312> <src> "0" . <0x1c00000083ed28> <gid_idfa> "644B0535-B0AB-4843-9742-BE647E5201D4" . <0x1c00000083ed28> <dgraph.type> "Idfa" . <0x4000003cdd659> <gid_user> "63821401" . <0x4000003cdd659> <dgraph.type> "User" . <0x4000003cdd659> <src> "0" . <0x80000046aebda> <gid_job> "74116058" . <0x80000046aebda> <dgraph.type> "Job" . <0x80000046aebda> <title> "" . <0x400000263a32f> <gid_user> "40084271" . <0x400000263a32f> <dgraph.type> "User" . <0x400000263a32f> <src> "0" . <0x40000049d0628> <gid_user> "77399592" . <0x40000049d0628> <dgraph.type> "User" . <0x40000049d0628> <src> "0" . <0x800000539ab44> <gid_job> "87665476" . <0x800000539ab44> <dgraph.type> "Job" . <0x800000539ab44> <title> "" . <0x18000001b181c4> <gid_device> "DBBCF24A672546C9652D4F24BF19E919" . <0x18000001b181c4> <dgraph.type> "DeviceId" . <0x1800000278edb2> <gid_device> "158849370727685176" . <0x1800000278edb2> <dgraph.type> "DeviceId" . <0x4000002233c85> <gid_user> "35863685" . <0x4000002233c85> <dgraph.type> "User" . <0x4000002233c85> <src> "0" . <0x800000538900d> <gid_job> "87592973" . <0x800000538900d> <dgraph.type> "Job" . <0x800000538900d> <title> "" . <0x4000001e83533> <gid_user> "31995187" . <0x4000001e83533> <dgraph.type> "User" . <0x4000001e83533> <src> "0" . <0x1400006f16b482> <gid_ip> "111.22.180.130" . <0x1400006f16b482> <dgraph.type> "Ip" . <0x18000000580523> <gid_device> "158734449357601123" . <0x18000000580523> <dgraph.type> "DeviceId" . <0x800000537d14e> <gid_job> "87544142" . <0x800000537d14e> <dgraph.type> "Job" . <0x800000537d14e> <title> "" . <0x400000679da20> <gid_user> "108648992" . <0x400000679da20>

And there is only default dgraph.type on this alpha node as followings:

{"1":{"members":{"1":{"id":"1","groupId":1,"addr":"172.21.46.20:7080","leader":true,"lastUpdate":"1595230814"}},"tablets":{"dgraph.graphql.schema":{"groupId":1,"predicate":"dgraph.graphql.schema"},"dgraph.graphql.xid":{"groupId":1,"predicate":"dgraph.graphql.xid"},"dgraph.type":{"groupId":1,"predicate":"dgraph.type","space":"439011234"}},"checksum":"15480156157935355611"}

We use dgraph to track user actions and releated info to do user-job recommend, so there would have huge actions here, what do you mean by saying " The p directory you shared has around 850,000 dgraph.TypeAction keys", could you please explain it in more detail. @ibrahim

Dgraph uses badger to store keys and values. Dgraph builds keys from predicates and types. In your case, you have many duplicate keys. The key dgraph.TypeAction is duplicated many times. Duplicate keys are okay. Let’s say you do some update on a predicate and it will generate duplicates.

These duplicate keys are stored in a single table in badger (to ensure reads are correct). When you have too many duplicate keys (850,000 is acceptable), the table size will increase and that’s what caused the crash.

The directory you shared, was it from a crash? That directory shouldn’t cause any crashes. Your biggest table is around 100 MB and that cannot cause a crash. A table around 4 GB will cause a crash because uint32 overflows at 4 GB.

Yes, it does not crash yet, but its size grow on time, and it will crash finally.

And would it generate duplicate keys if i write the same content every time or duplicate keys mean same key with different values.

Could you give me some duplicate keys here, and what’s strange is that there is only duplicate keys in ‘dgraph.type’ not in any other predicates where both of them have duplicates writes.

Yes, it would generate duplicate keys if you write the same content multiple times.We’re not concerned with values right now.

Duplicate keys mean exactly duplicate keys. Usually, the duplicate keys are dropped when we do a snapshot (which performs badger compaction). @JimWen please capture all the logs. When the DB crashes, I’d like to see if there were any snapshots.

Could you give me some duplicate keys here, and what’s strange is that there is only duplicate keys in ‘dgraph.type’ not in any other predicates where both of them have duplicates writes.

Yeah, I am not 100% why would there be so many duplicate type keys. @martinmr might have some clue so I’ll wait for his reply.

I also noticed that there are no snapshots in the provided data directory.

Actually, you can have a look at that run script i have paste before

nohup $dgpath alpha --enable_sentry=false --ludicrous_mode --snapshot_after=1000000 --abort_older_than 15m --my=$local_ip:$port -o $offset --zero=$zero_1:5080 --config $cfpath --cwd alpha$index > ./log/alpha$index.log 2>&1 &

Because the badger compaction is time-consuming and could stop writing before, so i increase snapshot_after to 100000, could it because that the snapshot frequency is too low to clear all the duplicate keys. But that can still not explain why there is only ‘dgraph.type’ pred too big.

I looked into this issue and it looks like the problem is that the key is never rolled up. During a rollup, all the deltas for the affected keys would be compacted into one list that contains the new complete list. When this list is written the previous versions are discarded.

The problem is that right now Dgraph only rolls up lists as they are queried. So my guess is that you keep inserting data into this key but the list is not queried so it’s not being put into the list for rollups.

Internally, we will change Dgraph to also queue lists for rollups during mutations.

For now, you can do two things.

  1. Use upsert to insert the dgraph.type triples. In the query part of the upsert, check if the type is already there and don’t insert the triple if that’s the case (I am assuming the type is always the same).

  2. Force a rollup. Try to query the specific list (I show an example below) to get this query into the queue. Keep in mind that this might require sending a lot of queries in quick succession to force the queues to fill quickly. Another tip is to send other queries along with this query to add more keys into the queue.

If you are starting from scratch with a new live load, you don’t need to do step 2. If you are using the live loader you can’t really use upserts. What you could do is removing the type triples from the initial live load, then figure out the types for each node, and start another live load that sets the types of the nodes only once.

What you want to avoid is getting into another situation where you are writing the data to the same node but never querying the node.

query to touch the types:

If you want to query a specific node.

{ q(func: uid(<uid of the node>)) { dgraph.type } }

If you want to query all the nodes of a specific type:

{ q(func: eq(dgraph.type, <name of your type>)) { dgraph.type } }
2 Likes

@martinmr
Thank you, i will try step 2 first. But why did Dgraph only rolls up lists as they are queried, we just use it as data warehouse which join all data here and only query when we need, it’s wierd.

And as you mention to query the specific list to get this query into the queue, many predicates in our situation could be first queryed after a long time after first indgest, so we are thinking to query in background like this to improve query performance , I am wondering could Dgraph add some function like warmup in neo4j which can improve first query performance of specified predicates.

It was a design decision but we’ll change it to trigger rollups using queries and mutations.

I am not familiar with query warmup so I can’t comment. We don’t have a cache (have plans for that but it’s not in the next release). Once that’s done querying in advance could have some benefits.

@JimWen Did you get a chance to try Martin’s suggestion?

yes, i have try that solution, but it seems to help nothing.

Not sending repeated triples will fix your issue but it might require to reload your data.

We’ll look into the proper fix on our end (triggering rollups when mutations happen).

The changes made by @martinmr in fix(Dgraph): Queue keys for rollup during mutation. by martinmr · Pull Request #6145 · dgraph-io/dgraph · GitHub should fix this issue.

Thanks,that is great ! I will verify it later

2 Likes