# Queries returning empty results randomly

**URL:** https://discuss.dgraph.io/t/queries-returning-empty-results-randomly/7335
**Category:** Dgraph
**Tags:** java-client, query-processing, area:data-loss, dql, kind:bug
**Created:** [June 23, 2020, 3:54pm UTC](https://discuss.dgraph.io/t/queries-returning-empty-results-randomly/7335 "2020-06-23T15:54:38Z")
**Posts on this page:** 12
**Page:** 1

<div class="post-metadata">

### Author: ![vtomar](https://avatars.discourse-cdn.com/v4/letter/v/f05b48/32.png) [@vtomar](https://discuss.dgraph.io/u/vtomar)
#### Post date: [June 23, 2020, 3:54pm UTC](https://discuss.dgraph.io/t/queries-returning-empty-results-randomly/7335/1 "2020-06-23T15:54:38Z")

</div>

we have deployed dgraph cluster using official helm charts in production with 3 zero and 3 alpha nodes.

One of our queries look like this

```
  {
    stores(func: eq(site, %s)) @filter(eq(state, "active") and not uid(%s)){
      %s
    }
  }

```

This query gives correct results most of the time but gives an empty response a few times.  
It neither fails nor does it give any other wrong result but only empty results as wrong results.

Also, this is not the only query with this issue, there are other queries also which gives empty results occasionally.

```
  {
    inactiveStores(func: type(store)) @filter(eq(site, %s) and lt(createdOn, %s) and eq(state, inactive)) {
      uid
      productCount: count(hasProduct)
    }
  }

```

For example, I hit ratel 5-6 times with this query, it gives empty results 1 or 2 times. Also, there are no mutations running which can affect the results.  
How to debug such an issue?  
Is this a known issue?

This is affecting our production systems because we can’t build reliable systems on top of it.

Any help would be highly appreciated.

---

<div class="post-metadata">

### Author: ![ahsan](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/ahsan/32/3517_2.png) [@ahsan](https://discuss.dgraph.io/u/ahsan)
#### Post date: [June 23, 2020, 8:54pm UTC](https://discuss.dgraph.io/t/queries-returning-empty-results-randomly/7335/2 "2020-06-23T20:54:05Z")

</div>

Hi @vtomar,

Thanks for reaching out to us. Please let us know the following so that we can better dig into the issue.

- Which dgraph version you are using?
- Could you provide us an example dummy data or just the schema so that we can create dummy data?

---

<div class="post-metadata">

### Author: ![vtomar](https://avatars.discourse-cdn.com/v4/letter/v/f05b48/32.png) [@vtomar](https://discuss.dgraph.io/u/vtomar)
#### Post date: [June 24, 2020, 3:49am UTC](https://discuss.dgraph.io/t/queries-returning-empty-results-randomly/7335/3 "2020-06-24T03:49:33Z")

</div>

This is our helm chart values

```auto
dgraph:
  ## Global Docker image parameters
  ## Please, note that this will override the image parameters, including dependencies, configured to use the global value
  ## Current available global Docker image parameters: imageRegistry and imagePullSecrets
  ##
  # global:
  # imageRegistry: myRegistryName
  # imagePullSecrets:
  # - myRegistryKeySecretName

  image:
    registry: docker.io
    repository: dgraph/dgraph
    tag: v20.03.0
    ## Specify a imagePullPolicy
    ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'
    ## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images
    ##
    pullPolicy: Always
    ## Optionally specify an array of imagePullSecrets.
    ## Secrets must be manually created in the namespace.
    ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
    ##
    # pullSecrets:
    # - myRegistryKeySecretName
    ## Set to true if you would like to see extra information on logs
    ## It turns BASH and NAMI debugging in minideb
    ## ref: https://github.com/bitnami/minideb-extras/#turn-on-bash-debugging
    ##
    debug: false

  zero:
    name: zero
    monitorLabel: zero-dgraph-io
    ## StatefulSet controller supports automated updates. There are two valid update strategies: RollingUpdate and OnDelete
    ## ref: https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#updating-statefulsets
    ##
    updateStrategy: RollingUpdate

    ## Partition update strategy
    ## https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#partitions
    ##
    # rollingUpdatePartition:

    ## StatefulSet controller supports relax its ordering guarantees while preserving its uniqueness and identity guarantees. There are two valid pod management policies: OrderedReady and Parallel
    ## ref: https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#pod-management-policy
    ##
    podManagementPolicy: OrderedReady

    ## Number of dgraph zero pods
    ##
    replicaCount: 3

    ## Max number of replicas per data shard.
    ## i.e., the max number of Dgraph Alpha instances per group (shard).
    ##
    shardReplicaCount: 3

    ## zero server pod termination grace period
    ##
    terminationGracePeriodSeconds: 60

    ## Hard means that by default pods will only be scheduled if there are enough nodes for them
    ## and that they will never end up on the same node. Setting this to soft will do this "best effort"
    antiAffinity: soft

    # By default this will make sure two pods don't end up on the same node
    # Changing this to a region would allow you to spread pods across regions
    podAntiAffinitytopologyKey: "kubernetes.io/hostname"

    ## This is the node affinity settings as defined in
    ## https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity-beta-feature
    nodeAffinity: {}

    ## Kubernetes configuration
    ## For minikube, set this to NodePort, elsewhere use LoadBalancer
    ##
    service:
      type: ClusterIP

    ## dgraph Pod Security Context
    securityContext:
      enabled: false
      fsGroup: 1001
      runAsUser: 1001

    ## dgraph data Persistent Volume Storage Class
    ## If defined, storageClassName: <storageClass>
    ## If set to "-", storageClassName: "", which disables dynamic provisioning
    ## If undefined (the default) or set to null, no storageClassName spec is
    ## set, choosing the default provisioner. (gp2 on AWS, standard on
    ## GKE, AWS & OpenStack)
    ##
    persistence:
      enabled: true
      storageClass: iopsssd
      persistentVolumeReclaimPolicy: Retain
      accessModes:
        - ReadWriteOnce
      size: 10Gi

    ## Node labels and tolerations for pod assignment
    ## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector
    ## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#taints-and-tolerations-beta-feature
    ##
    nodeSelector:
      spotinst.io/node-lifecycle: od
    tolerations: []

    ## Configure resource requests
    ## ref: http://kubernetes.io/docs/user-guide/compute-resources/
    ##
    resources:
      requests:
        memory: 3096Mi
        cpu: 2

    ## Configure extra options for liveness and readiness probes
    ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes)
    ##
    livenessProbe:
      enabled: false
      port: 6080
      path: /health
      initialDelaySeconds: 15
      periodSeconds: 10
      timeoutSeconds: 5
      failureThreshold: 6
      successThreshold: 1

    readinessProbe:
      enabled: false
      port: 6080
      path: /state
      initialDelaySeconds: 15
      periodSeconds: 10
      timeoutSeconds: 5
      failureThreshold: 6
      successThreshold: 1

  alpha:
    name: alpha
    monitorLabel: alpha-dgraph-io
    ## StatefulSet controller supports automated updates. There are two valid update strategies: RollingUpdate and OnDelete
    ## ref: https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#updating-statefulsets
    ##
    updateStrategy: RollingUpdate

    ## Partition update strategy
    ## https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#partitions
    ##
    # rollingUpdatePartition:

    ## StatefulSet controller supports relax its ordering guarantees while preserving its uniqueness and identity guarantees. There are two valid pod management policies: OrderedReady and Parallel
    ## ref: https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#pod-management-policy
    ##
    podManagementPolicy: OrderedReady

    ## Number of dgraph nodes
    ##
    replicaCount: 3

    ## zero server pod termination grace period
    ##
    terminationGracePeriodSeconds: 600

    ## Hard means that by default pods will only be scheduled if there are enough nodes for them
    ## and that they will never end up on the same node. Setting this to soft will do this "best effort"
    antiAffinity: soft

    # By default this will make sure two pods don't end up on the same node
    # Changing this to a region would allow you to spread pods across regions
    podAntiAffinitytopologyKey: "kubernetes.io/hostname"

    ## This is the node affinity settings as defined in
    ## https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity-beta-feature
    nodeAffinity: {}

    ## Kubernetes configuration
    ## For minikube, set this to NodePort, elsewhere use LoadBalancer
    ##
    service:
      type: ClusterIP

    ## dgraph Pod Security Context
    securityContext:
      enabled: false
      fsGroup: 1001
      runAsUser: 1001

    ## dgraph data Persistent Volume Storage Class
    ## If defined, storageClassName: <storageClass>
    ## If set to "-", storageClassName: "", which disables dynamic provisioning
    ## If undefined (the default) or set to null, no storageClassName spec is
    ## set, choosing the default provisioner. (gp2 on AWS, standard on
    ## GKE, AWS & OpenStack)
    ##
    persistence:
      enabled: true
      storageClass: iopsssd
      persistentVolumeReclaimPolicy: Retain
      accessModes:
        - ReadWriteOnce
      size: 50Gi
      annotations: {}

    ## Node labels and tolerations for pod assignment
    ## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector
    ## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#taints-and-tolerations-beta-feature
    ##
    nodeSelector:
      spotinst.io/node-lifecycle: od
    tolerations: []

    ## Configure resource requests
    ## ref: http://kubernetes.io/docs/user-guide/compute-resources/
    ##
    resources:
      requests:
        memory: 12Gi
        cpu: 8
    ## Configure value for lru_mb flag
    ## Typically a third of available memory is recommended, keeping the default value to 2048mb
    # lru_mb: 3096

    ## Configure extra options for liveness and readiness probes
    ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes)
    ##
    livenessProbe:
      enabled: false
      port: 8080
      path: /health?live=1
      initialDelaySeconds: 15
      periodSeconds: 10
      timeoutSeconds: 5
      failureThreshold: 6
      successThreshold: 1

    readinessProbe:
      enabled: false
      port: 8080
      path: /health
      initialDelaySeconds: 15
      periodSeconds: 10
      timeoutSeconds: 5
      failureThreshold: 6
      successThreshold: 1

  ratel:
    name: ratel
    ## Number of dgraph nodes
    ##
    replicaCount: 1

    ## Kubernetes configuration
    ## For minikube, set this to NodePort, elsewhere use ClusterIP or LoadBalancer
    ##
    service:
      type: ClusterIP

    ## dgraph Pod Security Context
    securityContext:
      enabled: false
      fsGroup: 1001
      runAsUser: 1001

    ## Configure resource requests
    ## ref: http://kubernetes.io/docs/user-guide/compute-resources/
    ##
    ## resources:
    ## requests:
    ## memory: 256Mi
    ## cpu: 250m

    ## Configure extra options for liveness and readiness probes
    ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes)
    ##
    livenessProbe:
      enabled: false
      port: 8000
      path: /
      initialDelaySeconds: 30
      periodSeconds: 10
      timeoutSeconds: 5
      failureThreshold: 6
      successThreshold: 1

    readinessProbe:
      enabled: false
      port: 8000
      path: /
      initialDelaySeconds: 5
      periodSeconds: 10
      timeoutSeconds: 5
      failureThreshold: 6
      successThreshold: 1

```

This is our schema.

```auto
  schema: |
    <hasChild>: [uid] @reverse .
    <hasProduct>: [uid] @reverse .
    <hasDeletedProduct>: [uid] @reverse .
    <state>: string @index(exact) .
    <site>: string @index(hash) .
    <store.version>: int @index(int) @upsert .
    <preceding>: [int] .
    <mergedTo>: int .
    <createdOn>: int @index(int) .
    <updatedOn>: int .

  siteKeySchemaTemplate: |
    <_siteKey_.feedAttributeHash>: string @index(hash) .
    <_siteKey_.productHash>: string @index(hash) .
    <_siteKey_.product.id>: string @index(hash) .
    <_siteKey_.deletedProduct.id>: string @index(hash) .
    <_siteKey_.store.id>: string @index(hash) @upsert .
    <_siteKey_.variant.id>: string @index(hash) .

```

siteKeySchemaTemplate is a variable schema in which _siteKey_ is replaced by an actual value like this.

```auto
<1977dd954a21fd562718f15df628a6647f395abde2ac85719a83fdd6060a2c93.deletedProduct.id>: string @index(hash) .
<1977dd954a21fd562718f15df628a6647f395abde2ac85719a83fdd6060a2c93.feedAttributeHash>: string @index(hash) .
<1977dd954a21fd562718f15df628a6647f395abde2ac85719a83fdd6060a2c93.product.id>: string @index(hash) .
<1977dd954a21fd562718f15df628a6647f395abde2ac85719a83fdd6060a2c93.productHash>: string @index(hash) .
<1977dd954a21fd562718f15df628a6647f395abde2ac85719a83fdd6060a2c93.store.id>: string @index(hash) @upsert .
<1977dd954a21fd562718f15df628a6647f395abde2ac85719a83fdd6060a2c93.variant.id>: string @index(hash) .

```

@ahsan

---

<div class="post-metadata">

### Author: ![vtomar](https://avatars.discourse-cdn.com/v4/letter/v/f05b48/32.png) [@vtomar](https://discuss.dgraph.io/u/vtomar)
#### Post date: [June 25, 2020, 5:55pm UTC](https://discuss.dgraph.io/t/queries-returning-empty-results-randomly/7335/4 "2020-06-25T17:55:05Z")

</div>

@ahsan any updates on this?

---

<div class="post-metadata">

### Author: ![ahsan](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/ahsan/32/3517_2.png) [@ahsan](https://discuss.dgraph.io/u/ahsan)
#### Post date: [June 29, 2020, 6:19am UTC](https://discuss.dgraph.io/t/queries-returning-empty-results-randomly/7335/7 "2020-06-29T06:19:14Z")

</div>

@vtomar  
Can you verify that the cluster is not failing when you are getting the random results. One possible reason for this can be that some pods might have died and aren’t able to return all the data due to consensus.

---

<div class="post-metadata">

### Author: ![vtomar](https://avatars.discourse-cdn.com/v4/letter/v/f05b48/32.png) [@vtomar](https://discuss.dgraph.io/u/vtomar)
#### Post date: [June 29, 2020, 4:14pm UTC](https://discuss.dgraph.io/t/queries-returning-empty-results-randomly/7335/8 "2020-06-29T16:14:07Z")

</div>

@ahsan  
So I don’t see any error logs in pods, also all the pods are up. Also the results are either empty or complete, never a non empty subset.

Also even if there is any issue with the cluster, given that dgraph is CP, should not the queries fail instead of giving wrong result ?

---

<div class="post-metadata">

### Author: ![MichelDiz](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/micheldiz/32/11873_2.png) [@MichelDiz](https://discuss.dgraph.io/u/MichelDiz)
#### Post date: [June 29, 2020, 4:42pm UTC](https://discuss.dgraph.io/t/queries-returning-empty-results-randomly/7335/9 "2020-06-29T16:42:48Z")

</div>

Can you start from scratch? clean your k8s (like “docker prune”) and start over. BTW, what was the process you used to add the dataset?

---

<div class="post-metadata">

### Author: ![vtomar](https://avatars.discourse-cdn.com/v4/letter/v/f05b48/32.png) [@vtomar](https://discuss.dgraph.io/u/vtomar)
#### Post date: [June 29, 2020, 6:12pm UTC](https://discuss.dgraph.io/t/queries-returning-empty-results-randomly/7335/10 "2020-06-29T18:12:47Z")

</div>

It’s our production system, can’t start from scratch.  
Data is ingested using dgraph mutations using JAVA client, it comes from various external sources S3 files, REST APIs etc, transformed into our data model and pushed to dgraph using mutations.  
Besides, how would clean up help and recreate same problem?

---

<div class="post-metadata">

### Author: ![MichelDiz](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/micheldiz/32/11873_2.png) [@MichelDiz](https://discuss.dgraph.io/u/MichelDiz)
#### Post date: [June 29, 2020, 6:32pm UTC](https://discuss.dgraph.io/t/queries-returning-empty-results-randomly/7335/11 "2020-06-29T18:32:43Z")

</div>

> [@vtomar](#):
>
> ` shardReplicaCount: 3`

I see that you have changed this from 5 to 3. There is a reason why?

> [@vtomar](#):
>
> Besides, how would clean up help and recreate same problem?

That depends. First of all, it’s just a guess - starting from scratch avoids having to do a detailed scan to find a problem, and take more time than you intended. To solve the problem.

Often users do several tests using the same volume and sometimes mix previous configurations and even run posting and wall files on top of each other. Overwriting files. That can work, but generate problems.

A while ago I saw this happen with other users. We don’t know what was done to achieve this. But starting from scratch usually helps.

It would be great to understand the steps to replicate this and then see if it is a problem in Dgraph or in the use of DB. But, the last time I saw someone report this was over 9+ months ago. And was a heavy user.

> [@vtomar](#):
>
> It’s our production system, can’t start from scratch.

Either way, you will have to stop your production to solve the problem. It is not possible to understand exactly what is happening only through logs, yaml, or testimony.

---

<div class="post-metadata">

### Author: ![joaquin](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/joaquin/32/2619_2.png) [@joaquin](https://discuss.dgraph.io/u/joaquin)
#### Post date: [August 19, 2020, 3:49am UTC](https://discuss.dgraph.io/t/queries-returning-empty-results-randomly/7335/12 "2020-08-19T03:49:57Z")

</div>

On reproducing the use case, you could do an export and re-import into a test cluster. How is ratel and alpha accessed? If using port-forward to a particular pod, are all the pods getting same frequency of empty results? Sometimes, when port-forward errors may occur, and you have to terminate tunnel and reconnect.

The Java client is experiencing similar behavior in similar frequency? Is the Java client running as a pod within the cluster, or outside the cluster?

What Kubernetes implementation, version are you using? Are you using the default scheduler or another scheduler (noticed the spotinst in the nodeSelector)? Were any underlying nodes affected swapped during testing?

---

<div class="post-metadata">

### Author: ![BlankRain](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/blankrain/32/1592_2.png) [@BlankRain](https://discuss.dgraph.io/u/BlankRain)
#### Post date: [August 19, 2020, 4:02am UTC](https://discuss.dgraph.io/t/queries-returning-empty-results-randomly/7335/13 "2020-08-19T04:02:06Z")

</div>

> [@Query result not as expected](http://discuss.hypermode.com/t/query-result-not-as-expected/9544):
>
> OS info: [image] dgraph.exe version [Decoder]: Using assembly version of decoder Dgraph version : v20.03.0 Dgraph SHA-256 : 2dd8a84711404f19a13375c377084795321f5e022eefafb869dbba6822b4d4fd Commit SHA-1 : 147c8df9 Commit timestamp : 2020-03-30 17:28:31 -0700 Branch : HEAD Go version : go1.14.1 For Dgraph official documentation, visit https://docs.dgraph.io. For discussions about Dgraph , visit http://discuss.hypermode.com. To say hi to the community , visit htt…

This two topics is similar.  
I also got an empty result.

---

<div class="post-metadata">

### Author: ![chewxy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/chewxy/32/4339_2.png) [@chewxy](https://discuss.dgraph.io/u/chewxy)
#### Post date: [August 19, 2020, 12:27pm UTC](https://discuss.dgraph.io/t/queries-returning-empty-results-randomly/7335/14 "2020-08-19T12:27:09Z")

</div>

Hi @vtomar, in [another thread](http://discuss.hypermode.com/t/query-result-not-as-expected/9544), the simple fix is to upgrade to dgraph 20.07. I was wondering if you could do the same to see if this issue has been fixed?