We have around 150 Predicates. When I apply our schema in Ratel, with 1 dgraph zero and 3 dgraph servers on 1.0.4, it seems that not all of our predicates are actually written into the schema, or they are at first, and then upon a predicate move, they are lost? We are getting odd errors like Predicate ENTITY_hasMaster doesn't have reverse edge when that is most certainly in our schema.
Can you clarify your setup here? Are the three dgraph servers serving the same group i.e. they are replicas of each other or are they serving different groups? If they are serving the same group, then predicate move won’t even happen. Do you see any logs on Zero about any predicate move that happened for the given predicate?
We have 3 servers, 1 replica due to my other issue I took out the replicas. It even seemed from the start that I had to run the alter schema command multiple times in ratel to get it to have all of the predicates I gave it. There are empty lines in our schema file, and I copy and paste that and run it. It acts like everything is alright though.
How did you know that it didn’t have all the predicates even though it got completed? I can check if Ratel behaves properly with empty lines.
I knew because immediately, when I tried to query something after VERY little data was run, it said that an edge didnt exist, when I know it should have. Now if a predicate move could have already happen, then its possible it did.
Predicate moves happen every 8 mins. I’d need logs from all servers and the error that you get while querying to be able to help. Also, try this against the nightly, the logging has been improved a little.
ok, is there a log dump somewhere that I can get those? I run this all in coreos
Logs are sent to stdout, do you run the binaries directly or use something like systemd to run the processes? If you run the binaries directly, you could perhaps pipe stdout and stderr to a file like https://stackoverflow.com/a/7219252.
alright I will look into it
I started the new servers (3) with the master build from docker. I have 1 dgraphzero, and replica of 1. I added our schema, which has 180 total predicates. I also removed any spaces in the query for the predicates. I ran the query in the alter window of Ratel, and it showed 180 predicates in the schema window, so all was good. After about an hour (I took lunch) with nothing running or being added to the schema or graph in general, I now have 179 predicates in the schema window. It looks like, from the logs, that ACCOUNT_id had something happen to it, and sure enough, it is the one missing in the schema window. The logs follow →
– Logs begin at Fri 2018-03-23 14:19:28 UTC. –
Mar 23 14:20:44 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:20:44 raft.go:749: INFO: 1 is starting a new election at term 1
Mar 23 14:20:44 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:20:44 raft.go:580: INFO: 1 became candidate at term 2
Mar 23 14:20:44 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:20:44 raft.go:664: INFO: 1 received MsgVoteResp from 1 at term 2
Mar 23 14:20:44 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:20:44 raft.go:621: INFO: 1 became leader at term 2
Mar 23 14:20:44 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:20:44 node.go:301: INFO: raft.node: 1 elected leader 1 at term 2
Mar 23 14:20:51 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:20:51 zero.go:333: Got connection request: id:1 addr:“dev-dgraph-master-01:7080”
Mar 23 14:20:51 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:20:51 pool.go:118: == CONNECT ==> Setting dev-dgraph-master-01:7080
Mar 23 14:20:51 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:20:51 zero.go:433: Connected
Mar 23 14:23:31 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:23:31 oracle.go:85: purging below ts:2, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 14:23:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:23:41 oracle.go:85: purging below ts:4, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 14:26:51 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:26:51 zero.go:333: Got connection request: id:2 addr:“dev-dgraph-master-02:7080”
Mar 23 14:26:51 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:26:51 pool.go:118: == CONNECT ==> Setting dev-dgraph-master-02:7080
Mar 23 14:26:51 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:26:51 zero.go:433: Connected
Mar 23 14:28:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:28:41 tablet.go:184:
Mar 23 14:28:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: Groups sorted by size: [{gid:2 size:0} {gid:1 size:17}]
Mar 23 14:28:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:28:41 tablet.go:189: size_diff 17
Mar 23 14:31:46 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:31:46 zero.go:333: Got connection request: id:3 addr:“dev-dgraph-master-03:7080”
Mar 23 14:31:46 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:31:46 pool.go:118: == CONNECT ==> Setting dev-dgraph-master-03:7080
Mar 23 14:31:46 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:31:46 zero.go:433: Connected
Mar 23 14:33:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:33:41 oracle.go:85: purging below ts:6, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 14:36:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:36:41 tablet.go:184:
Mar 23 14:36:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: Groups sorted by size: [{gid:2 size:0} {gid:3 size:0} {gid:1 size:17}]
Mar 23 14:36:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:36:41 tablet.go:189: size_diff 17
Mar 23 14:36:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:36:41 tablet.go:189: size_diff 0
Mar 23 14:36:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:36:41 oracle.go:85: purging below ts:9, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 14:37:31 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:37:31 oracle.go:85: purging below ts:11, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 14:38:31 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:38:31 oracle.go:85: purging below ts:13, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 14:38:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:38:41 oracle.go:85: purging below ts:16, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 14:38:51 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:38:51 oracle.go:85: purging below ts:18, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 14:44:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:44:41 tablet.go:184:
Mar 23 14:44:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: Groups sorted by size: [{gid:2 size:0} {gid:3 size:0} {gid:1 size:17}]
Mar 23 14:44:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:44:41 tablet.go:189: size_diff 17
Mar 23 14:44:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:44:41 tablet.go:189: size_diff 0
Mar 23 14:49:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:49:41 oracle.go:85: purging below ts:20, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 14:49:51 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:49:51 oracle.go:85: purging below ts:21, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 14:50:21 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:50:21 oracle.go:85: purging below ts:24, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 14:50:31 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:50:31 oracle.go:85: purging below ts:25, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 14:52:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:52:41 tablet.go:184:
Mar 23 14:52:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: Groups sorted by size: [{gid:2 size:0} {gid:3 size:0} {gid:1 size:17}]
Mar 23 14:52:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:52:41 tablet.go:189: size_diff 17
Mar 23 14:52:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:52:41 tablet.go:189: size_diff 0
Mar 23 14:53:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:53:41 oracle.go:85: purging below ts:29, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 14:53:51 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 14:53:51 oracle.go:85: purging below ts:32, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 15:00:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:00:41 tablet.go:184:
Mar 23 15:00:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: Groups sorted by size: [{gid:2 size:0} {gid:3 size:0} {gid:1 size:17}]
Mar 23 15:00:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:00:41 tablet.go:189: size_diff 17
Mar 23 15:00:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:00:41 tablet.go:189: size_diff 0
Mar 23 15:08:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:08:41 tablet.go:184:
Mar 23 15:08:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: Groups sorted by size: [{gid:2 size:0} {gid:3 size:0} {gid:1 size:17}]
Mar 23 15:08:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:08:41 tablet.go:189: size_diff 17
Mar 23 15:08:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:08:41 tablet.go:189: size_diff 0
Mar 23 15:08:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:08:41 oracle.go:85: purging below ts:36, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 15:08:51 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:08:51 oracle.go:85: purging below ts:39, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 15:16:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:16:41 tablet.go:184:
Mar 23 15:16:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: Groups sorted by size: [{gid:2 size:0} {gid:3 size:0} {gid:1 size:17}]
Mar 23 15:16:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:16:41 tablet.go:189: size_diff 17
Mar 23 15:16:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:16:41 tablet.go:189: size_diff 0
Mar 23 15:23:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:23:41 oracle.go:85: purging below ts:43, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 15:23:51 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:23:51 oracle.go:85: purging below ts:46, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 15:24:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:24:41 tablet.go:184:
Mar 23 15:24:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: Groups sorted by size: [{gid:2 size:0} {gid:3 size:0} {gid:1 size:17}]
Mar 23 15:24:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:24:41 tablet.go:189: size_diff 17
Mar 23 15:24:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:24:41 tablet.go:189: size_diff 0
Mar 23 15:32:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:32:41 tablet.go:184:
Mar 23 15:32:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: Groups sorted by size: [{gid:2 size:0} {gid:3 size:0} {gid:1 size:17}]
Mar 23 15:32:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:32:41 tablet.go:189: size_diff 17
Mar 23 15:32:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:32:41 tablet.go:189: size_diff 0
Mar 23 15:38:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:38:41 oracle.go:85: purging below ts:50, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 15:38:51 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:38:51 oracle.go:85: purging below ts:52, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 15:39:01 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:39:01 oracle.go:85: purging below ts:53, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 15:40:01 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:40:01 oracle.go:85: purging below ts:56, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 15:40:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:40:41 tablet.go:184:
Mar 23 15:40:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: Groups sorted by size: [{gid:2 size:0} {gid:3 size:0} {gid:1 size:17}]
Mar 23 15:40:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:40:41 tablet.go:189: size_diff 17
Mar 23 15:40:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:40:41 tablet.go:189: size_diff 0
Mar 23 15:41:31 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:41:31 oracle.go:85: purging below ts:57, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 15:41:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:41:41 oracle.go:85: purging below ts:58, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 15:41:51 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:41:51 oracle.go:85: purging below ts:59, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 15:41:53 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:41:53 raft.go:351: Tablet for attr: [predicate], gid: [3] is already being served by group: [1]
Mar 23 15:41:53 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:41:53 raft.go:351: Tablet for attr: [ACCOUNT_id], gid: [3] is already being served by group: [1]
Mar 23 15:42:01 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:42:01 oracle.go:85: purging below ts:61, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 15:44:21 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:44:21 oracle.go:85: purging below ts:63, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 15:44:31 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:44:31 oracle.go:85: purging below ts:65, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 15:44:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:44:41 oracle.go:85: purging below ts:66, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 15:48:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:48:41 tablet.go:184:
Mar 23 15:48:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: Groups sorted by size: [{gid:2 size:0} {gid:3 size:0} {gid:1 size:17}]
Mar 23 15:48:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:48:41 tablet.go:189: size_diff 17
Mar 23 15:48:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:48:41 tablet.go:189: size_diff 0
Mar 23 15:53:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:53:41 oracle.go:85: purging below ts:69, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 15:53:51 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:53:51 oracle.go:85: purging below ts:72, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 15:54:01 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:54:01 oracle.go:85: purging below ts:73, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 15:56:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:56:41 tablet.go:184:
Mar 23 15:56:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: Groups sorted by size: [{gid:2 size:0} {gid:3 size:0} {gid:1 size:17}]
Mar 23 15:56:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:56:41 tablet.go:189: size_diff 17
Mar 23 15:56:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 15:56:41 tablet.go:189: size_diff 0
Mar 23 16:04:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:04:41 tablet.go:184:
Mar 23 16:04:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: Groups sorted by size: [{gid:2 size:0} {gid:3 size:0} {gid:1 size:17}]
Mar 23 16:04:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:04:41 tablet.go:189: size_diff 17
Mar 23 16:04:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:04:41 tablet.go:189: size_diff 0
Mar 23 16:08:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:08:41 oracle.go:85: purging below ts:75, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 16:08:51 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:08:51 oracle.go:85: purging below ts:79, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 16:09:01 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:09:01 oracle.go:85: purging below ts:80, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 16:12:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:12:41 tablet.go:184:
Mar 23 16:12:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: Groups sorted by size: [{gid:3 size:0} {gid:2 size:0} {gid:1 size:17}]
Mar 23 16:12:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:12:41 tablet.go:189: size_diff 17
Mar 23 16:12:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:12:41 tablet.go:189: size_diff 0
Mar 23 16:20:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:20:41 tablet.go:184:
Mar 23 16:20:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: Groups sorted by size: [{gid:3 size:0} {gid:2 size:0} {gid:1 size:17}]
Mar 23 16:20:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:20:41 tablet.go:189: size_diff 17
Mar 23 16:20:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:20:41 tablet.go:189: size_diff 0
Mar 23 16:23:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:23:41 oracle.go:85: purging below ts:82, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 16:23:51 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:23:51 oracle.go:85: purging below ts:86, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 16:24:01 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:24:01 oracle.go:85: purging below ts:87, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 16:28:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:28:41 tablet.go:184:
Mar 23 16:28:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: Groups sorted by size: [{gid:2 size:0} {gid:3 size:0} {gid:1 size:17}]
Mar 23 16:28:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:28:41 tablet.go:189: size_diff 17
Mar 23 16:28:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:28:41 tablet.go:189: size_diff 0
Mar 23 16:36:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:36:41 tablet.go:184:
Mar 23 16:36:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: Groups sorted by size: [{gid:2 size:0} {gid:3 size:0} {gid:1 size:17}]
Mar 23 16:36:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:36:41 tablet.go:189: size_diff 17
Mar 23 16:36:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:36:41 tablet.go:189: size_diff 0
Mar 23 16:38:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:38:41 oracle.go:85: purging below ts:89, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 16:38:51 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:38:51 oracle.go:85: purging below ts:93, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 16:39:01 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:39:01 oracle.go:85: purging below ts:94, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 16:44:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:44:41 tablet.go:184:
Mar 23 16:44:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: Groups sorted by size: [{gid:2 size:0} {gid:3 size:0} {gid:1 size:17}]
Mar 23 16:44:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:44:41 tablet.go:189: size_diff 17
Mar 23 16:44:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:44:41 tablet.go:189: size_diff 0
Mar 23 16:52:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:52:41 tablet.go:184:
Mar 23 16:52:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: Groups sorted by size: [{gid:3 size:0} {gid:2 size:0} {gid:1 size:17}]
Mar 23 16:52:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:52:41 tablet.go:189: size_diff 17
Mar 23 16:52:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:52:41 tablet.go:189: size_diff 0
Mar 23 16:53:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:53:41 oracle.go:85: purging below ts:96, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 16:53:51 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:53:51 oracle.go:85: purging below ts:99, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 16:54:01 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 16:54:01 oracle.go:85: purging below ts:101, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:00:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:00:41 tablet.go:184:
Mar 23 17:00:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: Groups sorted by size: [{gid:2 size:0} {gid:3 size:0} {gid:1 size:17}]
Mar 23 17:00:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:00:41 tablet.go:189: size_diff 17
Mar 23 17:00:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:00:41 tablet.go:189: size_diff 0
Mar 23 17:08:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:08:41 raft.go:351: Tablet for attr: [ACCOUNT_id], gid: [1] is already being served by group: [2]
Mar 23 17:08:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:08:41 tablet.go:184:
Mar 23 17:08:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: Groups sorted by size: [{gid:2 size:0} {gid:3 size:0} {gid:1 size:17}]
Mar 23 17:08:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:08:41 tablet.go:189: size_diff 17
Mar 23 17:08:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:08:41 tablet.go:189: size_diff 0
Mar 23 17:08:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:08:41 oracle.go:85: purging below ts:103, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:08:51 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:08:51 oracle.go:85: purging below ts:105, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:09:01 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:09:01 oracle.go:85: purging below ts:108, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:10:01 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:10:01 oracle.go:85: purging below ts:109, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Update to the previous reply: In the time it took me to write that, and post it, if I ran another refresh for the schema in ratel, it now shows ACCOUNT_id back, I tested like 10 times earlier before I wrote the post and it didn’t return ACCOUNT_id, and shoed only 179, not 180. Now, at this moment, it looks back to normal. However, I literally have done nothing.
Another Update, upon starting to run data, we saw this error repeatedly → rpc error: code = Unknown desc = : rpc error: code = Unknown desc = Got error: Schema not defined for predicate: ENTITY_id. while running: name:"eq" args:"0eac446436286dd3" " . I confirmed that it is in my schema file (that I use for hydrating the schema through ratel) and confirmed it was there before we ran data. It shows that it is still there now too. We are also back down to 179 predicates and ENTITY_id and ACCOUNT_id both show up as if they are still there, so some other random predicate is now gone. This seems to not be working at all. Dgraphzero logs are as follows → Mar 23 17:16:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: Groups sorted by size: [{gid:2 size:0} {gid:3 size:0} {gid:1 size:17}]
Mar 23 17:16:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:16:41 tablet.go:189: size_diff 17
Mar 23 17:16:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:16:41 tablet.go:189: size_diff 0
Mar 23 17:19:21 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:19:21 oracle.go:85: purging below ts:110, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:19:31 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:19:31 oracle.go:85: purging below ts:111, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:20:11 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:20:11 oracle.go:85: purging below ts:113, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:22:11 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:22:11 oracle.go:85: purging below ts:116, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:22:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:22:41 oracle.go:85: purging below ts:118, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:23:51 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:23:51 oracle.go:85: purging below ts:122, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:24:01 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:24:01 oracle.go:85: purging below ts:125, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:24:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:24:41 tablet.go:184:
Mar 23 17:24:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: Groups sorted by size: [{gid:2 size:0} {gid:3 size:0} {gid:1 size:17}]
Mar 23 17:24:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:24:41 tablet.go:189: size_diff 17
Mar 23 17:24:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:24:41 tablet.go:189: size_diff 0
Mar 23 17:26:21 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:26:21 oracle.go:85: purging below ts:126, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:32:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:32:41 tablet.go:184:
Mar 23 17:32:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: Groups sorted by size: [{gid:2 size:0} {gid:3 size:0} {gid:1 size:17}]
Mar 23 17:32:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:32:41 tablet.go:189: size_diff 17
Mar 23 17:32:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:32:41 tablet.go:189: size_diff 0
Mar 23 17:34:31 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:34:31 oracle.go:85: purging below ts:127, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:38:51 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:38:51 oracle.go:85: purging below ts:131, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:39:01 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:39:01 oracle.go:85: purging below ts:134, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:40:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:40:41 tablet.go:184:
Mar 23 17:40:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: Groups sorted by size: [{gid:2 size:0} {gid:3 size:0} {gid:1 size:17}]
Mar 23 17:40:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:40:41 tablet.go:189: size_diff 17
Mar 23 17:40:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:40:41 tablet.go:189: size_diff 0
Mar 23 17:48:31 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:48:31 oracle.go:85: purging below ts:135, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:48:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:48:41 tablet.go:184:
Mar 23 17:48:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: Groups sorted by size: [{gid:2 size:0} {gid:3 size:0} {gid:1 size:17}]
Mar 23 17:48:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:48:41 tablet.go:189: size_diff 17
Mar 23 17:48:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:48:41 tablet.go:189: size_diff 0
Mar 23 17:56:01 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:56:01 oracle.go:85: purging below ts:215, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:56:11 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:56:11 oracle.go:85: purging below ts:321, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:56:21 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:56:21 oracle.go:85: purging below ts:429, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:56:31 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:56:31 oracle.go:85: purging below ts:539, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:56:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:56:41 tablet.go:184:
Mar 23 17:56:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: Groups sorted by size: [{gid:2 size:0} {gid:3 size:0} {gid:1 size:17}]
Mar 23 17:56:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:56:41 tablet.go:189: size_diff 17
Mar 23 17:56:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:56:41 tablet.go:189: size_diff 0
Mar 23 17:56:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:56:41 oracle.go:85: purging below ts:599, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:56:51 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:56:51 oracle.go:85: purging below ts:600, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:57:01 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:57:01 oracle.go:85: purging below ts:602, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:57:11 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:57:11 oracle.go:85: purging below ts:603, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:57:31 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:57:31 oracle.go:85: purging below ts:605, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:57:51 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:57:51 oracle.go:85: purging below ts:606, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:58:01 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:58:01 oracle.go:85: purging below ts:608, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:58:11 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:58:11 oracle.go:85: purging below ts:609, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:58:21 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:58:21 oracle.go:85: purging below ts:622, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:58:31 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:58:31 oracle.go:85: purging below ts:624, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:58:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:58:41 oracle.go:85: purging below ts:625, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:59:01 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:59:01 oracle.go:85: purging below ts:629, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:59:31 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:59:31 oracle.go:85: purging below ts:631, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 17:59:51 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 17:59:51 oracle.go:85: purging below ts:632, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 18:00:01 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 18:00:01 oracle.go:85: purging below ts:634, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 18:00:11 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 18:00:11 oracle.go:85: purging below ts:635, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 18:00:31 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 18:00:31 oracle.go:85: purging below ts:637, len(o.commits):0, len(o.aborts):0, len(o.rowCommit)
Also, after refreshing over and over, for minutes, we are back to 180 predicates, with no logging of ANYTHING moving in dgraphzero.
back to 179 again with these logs → Mar 23 18:09:31 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 18:09:31 oracle.go:85: purging below ts:797, len(o.commits):25, len(o.aborts):0, len(o.rowCommit):187
Mar 23 18:09:31 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 18:09:31 raft.go:504: Writing snapshot at index: 4507, applied mark: 4508
Mar 23 18:09:41 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 18:09:41 oracle.go:85: purging below ts:799, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
Mar 23 18:10:01 dev-dgraph-master-01.c.charlotte-161616.internal docker[1442]: 2018/03/23 18:10:01 oracle.go:85: purging below ts:801, len(o.commits):0, len(o.aborts):0, len(o.rowCommit):0
We also just noticed, it seemed to move an @reverse off of one of our predicates just recently.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.