Update throughput of dGraph using Python Client

Users Dgraph
, ,
1

Hello everyone !

I would like to know what is the best way to achieve the “highest throughput” regarding the update of dGraph nodes using Python. In my current setup I have to update nodes 50-100 times in a second & whenever a new “value” occurs it should get updated individually. I would like to avoid “batch” update, because it may result in loss of data.

To do this, I’ve tried two different strategies, one strategy is using the GraphqlClient, and the other one is using the official pydgraph client. Basically, what am I doing is updating the same node 100 times and measuring the time.

GraphqlClient:

from python_graphql_client import GraphqlClient
import time

client = GraphqlClient(endpoint="http://localhost:8080/graphql")

msg_count = 1
sensorOne = "temperatureSensor"

t0 = time.time()
while True:
    if (msg_count <=100):
        updateSensor = 'mutation MyMutation{ updateSensor(input: {filter: {name: {eq: "' + sensorOne + '"}' + '}, set: {value: ' + str(msg_count) +'}' + '}){ sensor { name value}' + ' }' + '}'
        data = client.execute(query=updateSensor)    
    else:
        t1 = time.time()
        break
    
    msg_count +=1
print("Total time: "+ str(t1-t0))

In average, I takes around 13-15 seconds to update 100 values, and that would be around 6 updates/second.

pydgraph client:

import pydgraph, time

client_stub = pydgraph.DgraphClientStub('localhost:9080')
client = pydgraph.DgraphClient(client_stub)

nodeType = "temperatureSensor"
msg_count = 1

t0 = time.time()
while True:
    if (msg_count <=100):
        txn = client.txn()
        try:
            queryV2 = '{var(func: eq(Sensor.name, "' + nodeType + '")) {Type as uid}}'
            nquad = f"""uid(Type) <Sensor.value> "{msg_count}" ."""
            mutation = txn.create_mutation(set_nquads=nquad)
            request = txn.create_request(query=queryV2, mutations=[mutation], commit_now=True)
            txn.do_request(request)  
        except pydgraph.AbortedError:
            pass
        finally:
            txn.discard()
    else:
        t1 = time.time()
        break
    msg_count+=1
print("Total time: "+ str(t1-t0))

This strategy is in average twice as fast as the one using GraphqlClient. It takes around 6 seconds to update 100 values, and that is 15 updates/second.

This is the schema for Sensor:

type Sensor @withSubscription {
    id: ID!
    name: String! @id @search(by: [exact, regexp])
    value: Float @search
    timestamp: DateTime
    unit: String
}

So I would like to know is there a way to achieve around 50-100 updates/second. Important to say is that I’m using a Standalone image deployed to Docker container. Should I get better results if I increase the resources of container or if I use the cluster schema?

Thanks in advance!

2

Can you try this?

I get consistently ~115 updates/second. I’m running a simple local cluster on my Mac M1 Mini. Perhaps a lot of the lag in your version is all the python string parsing per update?

import datetime
from python_graphql_client import GraphqlClient

# Instantiate the client with an endpoint.
client = GraphqlClient(endpoint="http://localhost:8080/graphql")

query = """
mutation($id: String!, $value: Float!) {
  updateSensor(input: {filter: {name: {eq: $id}}, set: {value: $value}}) {
    sensor {
      id
    }
  }
}
"""

name = "temperatureSensor"

start = datetime.datetime.now()
for i in range(1, 101):
    data = client.execute(query=query, variables={"id": name, "value": i*i})
end = datetime.datetime.now()

elapsed = end - start

print(i, "updates in", elapsed.microseconds / 1000, "milliseconds")
print("updates per second", i / (elapsed.microseconds / 1000000))
3

Hello @matthewmcneely,
thank You for your suggestions, they helped me a bit, but still I’ve some doubts.

At the time when I was conducting “test” from the question, dGraph database was deployed to the Azure Container Instance and I think that was the main issue. When I’ve deployed “standalone” dGraph base on my laptop, I got similar results as You did.

So, I’ve thought that the source of the problem was the resources given to that specific Azure Container Instance, but it seems that was not the case. To prove this, I’ve created a simple python script and deployed it also to Azure as Container Instance, and I got much better results compared to the one mentioned in the question. This is what I’ve done (through whole process, the database was deployed to the Azure):

  1. This python script had the same code as the one You’ve proposed, and I’ve set it up to do exactly 500 updates.

  2. When script was executed as an Azure Container instance, the result was around 70 updates/seconds. That results were 10 times better regarding the one I’ve achieved when the script was executed on my laptop (6-7 updates/second).

  3. The results from the Step 2. were achieved using the exact same amount of resources as the first time. Then I’ve decided to increase resources by 2 & 4 times, and the results weren’t better. The resources are related to the database, not the script.

  4. After exactly 1000 requests (2 executions with 500 requests) dGraph base deployed in Azure got “stuck”, when I say stuck, I mean that it wasn’t able to process any more requests, and I had to wait several minutes (Time-out error). This only occurred when the base was updated using a script that was also deployed to Azure. Local script never caused this issue, probably because it was to slow regarding those 6-7 updates/second…

Based on all of these results, I’ve concluded that the network definitely affected the results. It probably took some time for each request to “travel” from my laptop to some Azure server that hosted the database. But I’m wondering why I can’t achieve results similar to the one achieved locally on the laptop, and why the database wasn’t responding after 1000 requests.

I would be very happy If You could share Your thoughts with me regarding this issue.
Thanks in advance :smiley: