# Protocol buffer vs JSON time graphs

**URL:** <https://discuss.dgraph.io/t/protocol-buffer-vs-json-time-graphs/143>\
**Category:** Users\
**Created:** [May 4, 2016, 2:29pm UTC](https://discuss.dgraph.io/t/protocol-buffer-vs-json-time-graphs/143 "2016-05-04T14:29:31Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![pawan](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/pawan/32/1946_2.png) [@pawan](https://discuss.dgraph.io/u/pawan)\
**Post date:** [May 4, 2016, 2:29pm UTC](https://discuss.dgraph.io/t/protocol-buffer-vs-json-time-graphs/143/1 "2016-05-04T14:29:31Z")

</div>

I did some analysis for this.

Results for the actors query are good. The query was performed for 478936 actors.

```auto
{
  me(_xid_: m.0f4vbz) {
    type.object.name.en
    film.actor.film {
      film.performance.film {
        type.object.name.en
      }
    }
  }
}

```

[https://plot.ly/~pawanrawal/24/actors-query/](https://plot.ly/~pawanrawal/24/actors-query/)

The results are not as great as expected for the directors query. Protocol buffer performs well when number of entities are upto say 200 but for greater values it is debatable.  
Here are results for the director query.(This query was performed with 98000 different directors)

```auto
{
  me(_xid_: m.06pj8) {
    type.object.name.en
    film.director.film {
      film.film.genre {
        type.object.name.en
      }
    }
  }
}

```

[https://plot.ly/~pawanrawal/27/directors-query/](https://plot.ly/~pawanrawal/27/directors-query/)

How do the results look @mrjn ?

---

<div class="post-metadata">

**Author:** ![mrjn](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/mrjn/32/775_2.png) [@mrjn](https://discuss.dgraph.io/u/mrjn)\
**Post date:** [May 4, 2016, 8:22pm UTC](https://discuss.dgraph.io/t/protocol-buffer-vs-json-time-graphs/143/2 "2016-05-04T20:22:45Z")

</div>

What were the queries? Can you post them. Also, can you add the JSON latency data in addition to the ratio?

I hunch is that memory allocation is the issue here. We can also look through your code and see if there are any ways to optimize it.

If we want to spend more time on this, then we could dump the Flatbuffer responses in a file. Have a standalone script pick the response, run the PB and JSON converter, and calculate time. Go profiler could also help us dig into the real issue better, with this component isolation.

The other option is that we send back Flatbuffer response directly to Go client, and do the parsing in the client. That would surely be faster and more network efficient – but that’s a someday/maybe task if we really want to squeeze the extra juice for better performance.

---

<div class="post-metadata">

**Author:** ![pawan](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/pawan/32/1946_2.png) [@pawan](https://discuss.dgraph.io/u/pawan)\
**Post date:** [May 5, 2016, 2:18am UTC](https://discuss.dgraph.io/t/protocol-buffer-vs-json-time-graphs/143/3 "2016-05-05T02:18:16Z")

</div>

> [@mrjn](#):
>
> Also, can you add the JSON latency data in addition to the ratio?

I will run the script again and store that in the CSV file too.

> [@mrjn](#):
>
> If we want to spend more time on this, then we could dump the Flat buffer responses in a file. Have a standalone script pick the response, run the PB and JSON converter, and calculate time. Go profiler could also help us dig into the real issue better, with this component isolation.

Yes, we could do this for the cases in which the ratio is \> 1. Should we spend more time on this?

> [@mrjn](#):
>
> The other option is that we send back Flat buffers response directly to Go client, and do the parsing in the client.

Hmm, can we do that? I thought that wasn’t possible hence we went for proto buffers. If we could do that, still the total time would be the same right?

---

<div class="post-metadata">

**Author:** ![mrjn](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/mrjn/32/775_2.png) [@mrjn](https://discuss.dgraph.io/u/mrjn)\
**Post date:** [May 5, 2016, 2:34am UTC](https://discuss.dgraph.io/t/protocol-buffer-vs-json-time-graphs/143/4 "2016-05-05T02:34:02Z")

</div>

> [@pawan](#):
>
> Should we spend more time on this?

If you can find a way to dump the Flatbuffer responses to a file, and bring them back in memory in the standalone script – the rest of the thing is relatively straightforward. So, might be worth.

---

<div class="post-metadata">

**Author:** ![pawan](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/pawan/32/1946_2.png) [@pawan](https://discuss.dgraph.io/u/pawan)\
**Post date:** [May 5, 2016, 4:39am UTC](https://discuss.dgraph.io/t/protocol-buffer-vs-json-time-graphs/143/5 "2016-05-05T04:39:12Z")

</div>

Modified the graphs to also show JSON latency data.

Checking if there is a way to dump flatbuffer data to a file. Though that would need to happen on the server.

---

<div class="post-metadata">

**Author:** ![pawan](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/pawan/32/1946_2.png) [@pawan](https://discuss.dgraph.io/u/pawan)\
**Post date:** [May 11, 2016, 9:06am UTC](https://discuss.dgraph.io/t/protocol-buffer-vs-json-time-graphs/143/6 "2016-05-11T09:06:40Z")

</div>

Regarding profiling

I was successfully able to gobencode the SubGraph and write it to a file but on retrieving it back I realised that since in SubGraph

```auto
type SubGraph struct {
	Attr string
	Count int
	Offset int
	Children []*SubGraph

	query []byte
	result []byte
}

```

query and result are not exported, values for those were not encoded and I could not bring them back to memory. I will make some code changes to export them temporarily, but if we want to have the benchmarking code as part of the repo they would have to exported permanently. What do you think @mrjn?

---

<div class="post-metadata">

**Author:** ![mrjn](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/mrjn/32/775_2.png) [@mrjn](https://discuss.dgraph.io/u/mrjn)\
**Post date:** [May 11, 2016, 9:08am UTC](https://discuss.dgraph.io/t/protocol-buffer-vs-json-time-graphs/143/7 "2016-05-11T09:08:07Z")

</div>

Should be alright. Just check how they’re being used – directly like `sg.query`, or always from a function.

Off the top of my head, I can’t think of any issues making them publicly available.

---

<div class="post-metadata">

**Author:** ![pawan](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/pawan/32/1946_2.png) [@pawan](https://discuss.dgraph.io/u/pawan)\
**Post date:** [May 11, 2016, 9:18am UTC](https://discuss.dgraph.io/t/protocol-buffer-vs-json-time-graphs/143/8 "2016-05-11T09:18:49Z")

</div>

> [@mrjn](#):
>
> Just check how they’re being used – directly like sg.query, or always from a function.

It’s from within a function always. Cool, I will change this.

---

<div class="post-metadata">

**Author:** ![pawan](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/pawan/32/1946_2.png) [@pawan](https://discuss.dgraph.io/u/pawan)\
**Post date:** [May 11, 2016, 5:08pm UTC](https://discuss.dgraph.io/t/protocol-buffer-vs-json-time-graphs/143/9 "2016-05-11T17:08:25Z")

</div>

Saw this amazing talk by Brad Fitzpatrick [Profiling & Optimizing in Go / Brad Fitzpatrick - YouTube](https://www.youtube.com/watch?v=xxDZuPEgbBU) about profiling Go Code. @ashwin95r would definitely recommend you to see this.

Did some analysis on the directors query which had 992 elements in its result. Got [svg images](https://www.dropbox.com/sh/jevj2ameodnyiie/AADmY2p5Skg43MGK5du-HyAna?dl=0) for the CPU and Memory profiles. Will study these tomorrow and see how we can optimize this.

Very good idea to do this @mrjn.

---

<div class="post-metadata">

**Author:** ![pawan](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/pawan/32/1946_2.png) [@pawan](https://discuss.dgraph.io/u/pawan)\
**Post date:** [May 20, 2016, 9:14am UTC](https://discuss.dgraph.io/t/protocol-buffer-vs-json-time-graphs/143/10 "2016-05-20T09:14:35Z")

</div>

Comparsion of ToJSON vs JSON encoding of protocol buffer for the query in `query_test.go`

```auto
{
	me(_uid_:0x01) {
		name
		gender
		status
		friend {
			name
		}
	}
}

```

ToJSON -

```auto
{  
   "_root_":[  
      {  
         "_uid_":"0x1",
         "friend":[  
            {  
               "_uid_":"0x17",
               "name":"Rick Grimes"
            },
            {  
               "_uid_":"0x18",
               "name":"Glenn Rhee"
            },
            {  
               "_uid_":"0x19",
               "name":"Daryl Dixon"
            },
            {  
               "_uid_":"0x1f",
               "name":"Andrea"
            },
            {  
               "_uid_":"0x65"
            }
         ],
         "gender":"female",
         "name":"Michonne",
         "status":"alive"
      }
   ]
}

```

JSON encoding of Protocol buffer

```auto
{  
   "uid":"1",
   "attribute":"_root_",
   "properties":[  
      {  
         "prop":"name",
         "val":"TWljaG9ubmU="
      },
      {  
         "prop":"gender",
         "val":"ZmVtYWxl"
      },
      {  
         "prop":"status",
         "val":"YWxpdmU="
      }
   ],
   "children":[  
      {  
         "uid":"23",
         "attribute":"friend",
         "properties":[  
            {  
               "prop":"name",
               "val":"UmljayBHcmltZXM="
            }
         ]
      },
      {  
         "uid":"24",
         "attribute":"friend",
         "properties":[  
            {  
               "prop":"name",
               "val":"R2xlbm4gUmhlZQ=="
            }
         ]
      },
      {  
         "uid":"25",
         "attribute":"friend",
         "properties":[  
            {  
               "prop":"name",
               "val":"RGFyeWwgRGl4b24="
            }
         ]
      },
      {  
         "uid":"31",
         "attribute":"friend",
         "properties":[  
            {  
               "prop":"name",
               "val":"QW5kcmVh"
            }
         ]
      },
      {  
         "uid":"101",
         "attribute":"friend",
         "properties":[  
            {  
               "prop":"name",
               "val":""
            }
         ]
      }
   ]
}

```

Ignore that the `val` are base64 encoded. The main point is that the schema of GraphQL isn’t retained.

---

<div class="post-metadata">

**Author:** ![mrjn](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/mrjn/32/775_2.png) [@mrjn](https://discuss.dgraph.io/u/mrjn)\
**Post date:** [May 20, 2016, 9:19am UTC](https://discuss.dgraph.io/t/protocol-buffer-vs-json-time-graphs/143/11 "2016-05-20T09:19:45Z")

</div>

I see what you mean. I think the best thing to do here would be to have another look at the ToJson code. And ensure the logic is correct.

---

<div class="post-metadata">

**Author:** ![pawan](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/pawan/32/1946_2.png) [@pawan](https://discuss.dgraph.io/u/pawan)\
**Post date:** [May 20, 2016, 9:22am UTC](https://discuss.dgraph.io/t/protocol-buffer-vs-json-time-graphs/143/12 "2016-05-20T09:22:20Z")

</div>

Right, I will have a look at it and see why we get the error.

---

<div class="post-metadata">

**Author:** ![compton](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/compton/32/127_2.png) [@compton](https://discuss.dgraph.io/u/compton)\
**Post date:** [May 22, 2016, 6:17pm UTC](https://discuss.dgraph.io/t/protocol-buffer-vs-json-time-graphs/143/13 "2016-05-22T18:17:10Z")

</div>

Oh, that’s fun stuff. You could write a graphql-generating [protobuf plugin](https://github.com/gogo/protobuf/tree/master/plugin) and manually create [scalar types](https://github.com/graphql-go/graphql/blob/master/scalars.go) for your [leaf values](https://github.com/dgraph-io/dgraph/blob/query/query/graph/graphresponse.proto#L21)

ccing @snorecone who is the master of this stuff

---

<div class="post-metadata">

**Author:** ![pawan](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/pawan/32/1946_2.png) [@pawan](https://discuss.dgraph.io/u/pawan)\
**Post date:** [November 28, 2017, 12:59am UTC](https://discuss.dgraph.io/t/protocol-buffer-vs-json-time-graphs/143/14 "2017-11-28T00:59:39Z")

</div>

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.