The same query will return different results

query 1:

query 2:

Two query in the same time almost,but return different results. This node has not schema type.

Is it a cluster? Did you perform the first query on one node and then the second query on another node? If so your Alpha nodes have diverged data, cluster is out of sync. Yet I have no idea how to fix such state as we struggle with the same problem from time to time.

Do you have steps to reproduce this behavior, @tss and @lukaszlenart? The same query shouldn’t be returning different data if there are no commits that changed the data in-between.

We are seeing a similar issue elsewhere too. Cc @ibrahim

If this is the same issue, let’s mark this as a bug @ibrahim

I do not have exact steps to reproduce this issue, but it can happen after Alpha leader crash/hard reset/box force restart and under heavy load. At this point other Alphas we elect a new leader but once the old leader will be available it tries to be leader again to commit pending transactions (you can read my other threads about this). We observe this in Grafana by monitoring pending mutations - if they start keeping on the same level we have an issue.

At this point I disable connection to that broken Alpha - our service has urls defined to all three Alphas as pointed out here and leave the Alpha in the cluster to give it time to rebalance - it take a few days (!?!?!) to get the Alpha in-sync with others.

During the sync procedure I can observer a lot of such entries coming in

I0315 15:07:24.474969      18 draft.go:1611] Skipping snapshot at index: 3714359. Insufficient discard entries: 0. MinPendingStartTs: 3384351
I0315 15:07:58.938631      18 draft.go:602] Skipping snapshot at 3714359, because found one at 3714359

Also I can notice some delay in getting updates from leader (like dropping and applying index, the broken Alpha will do the same but after a few hours after the leader).