# Fix queryWithVars to handle UTF-8 values correctly

**URL:** <https://discuss.dgraph.io/t/fix-querywithvars-to-handle-utf-8-values-correctly/7685>\
**Category:** Issues\
**Tags:** dgraph-js-http, kind:enhancement\
**Created:** [December 11, 2018, 2:21am UTC](https://discuss.dgraph.io/t/fix-querywithvars-to-handle-utf-8-values-correctly/7685 "2018-12-11T02:21:17Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![diggy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/diggy/32/3666_2.png) [@diggy](https://discuss.dgraph.io/u/diggy)\
**Post date:** [December 11, 2018, 2:21am UTC](https://discuss.dgraph.io/t/fix-querywithvars-to-handle-utf-8-values-correctly/7685/1 "2018-12-11T02:21:17Z")

</div>

_Posted by_ [srfrog](https://github.com/srfrog):

Apparently `queryWithVars` is altering the content of vars with UTF-8 strings when passing them to Dgraph. The main culprit seems to be the `toString()` method.

Reference [Query variables parsed differently to raw values · Issue #2803 · dgraph-io/dgraph · GitHub](https://github.com/dgraph-io/dgraph/issues/2803)

---

<div class="post-metadata">

**Author:** ![diggy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/diggy/32/3666_2.png) [@diggy](https://discuss.dgraph.io/u/diggy)\
**Post date:** [March 1, 2019, 5:52pm UTC](https://discuss.dgraph.io/t/fix-querywithvars-to-handle-utf-8-values-correctly/7685/2 "2019-03-01T17:52:21Z")

</div>

[paulftw](https://github.com/paulftw) _commented_ :

@srfrog I took a look at this and the referenced dgaph bug.

`toString()` in `txn.ts` only forces arbitrary js values to be strings, JS is pretty ok for unicode I wouldn’t suspect foul play here.

`toString` is really funky in there and can probably be removed altogether - the line above it makes sure type is string-like.  
Seems to be an overkill to [ignore non-strings](https://github.com/dgraph-io/dgraph-js-http/blob/master/tests/txn.spec.ts#L39) - something I wouldn’t enforce if I were designing an API.  
If I had to come up with a reason to enforce this requirement and even write a test for it - my guess it’s related to how Dgraph used to parse JSON in headers.

Original bug mentions the HTTP header encoding/parsing and I think that makes a lot of sense, because `vars` object does get sent as a header [here](https://github.com/dgraph-io/dgraph-js-http/blob/master/src/clientStub.ts#L50)

**tl;dr** Dgaph parses unicode differently in headers, than it does in protobufs.  
We need to either fix the parser or require client libraries to use some encoding + make sure it works for both nodejs and browsers

---

<div class="post-metadata">

**Author:** ![diggy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/diggy/32/3666_2.png) [@diggy](https://discuss.dgraph.io/u/diggy)\
**Post date:** [March 1, 2019, 7:32pm UTC](https://discuss.dgraph.io/t/fix-querywithvars-to-handle-utf-8-values-correctly/7685/3 "2019-03-01T19:32:28Z")

</div>

[srfrog](https://github.com/srfrog) _commented_ :

We can send unicode in HTTP headers but seems more trouble than it’s worth - [RFC 5978](https://tools.ietf.org/html/rfc5987#section-3.2.2)

I think the safest path would be to detect unicode and switch to regular POST form, then all the vars can be encoded and handled correctly. We would need to send the body in there too. Then we can add a new header that instructs Dgraph to get values from **form** , e.g., `X-Dgraph-FormVars: true`

For the API string restriction, I agree also. But I think it can be simple to correct. Dgraph expects var values to be string, so you could try to `toString()` all and send the ones that convert. It’s just a matter refactoring [this test](https://github.com/dgraph-io/dgraph-js-http/blob/master/src/txn.ts#L69) a bit.

---

<div class="post-metadata">

**Author:** ![diggy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/diggy/32/3666_2.png) [@diggy](https://discuss.dgraph.io/u/diggy)\
**Post date:** [March 1, 2019, 7:54pm UTC](https://discuss.dgraph.io/t/fix-querywithvars-to-handle-utf-8-values-correctly/7685/4 "2019-03-01T19:54:16Z")

</div>

[srfrog](https://github.com/srfrog) _commented_ :

Looking at [c.query()](https://github.com/dgraph-io/dgraph-js-http/blob/master/src/clientStub.ts#L47) I take back my form idea. We could just send the whole query, vars and all, as a JSON object.

```json
{
  "vars": {
    "key1": "value1",
    "key2": "value2",
    "key3": "value3",
  },
  "body": "... the query ..."
}

```

We can check for `Content-Type: application/json` at the server and parse accordingly. I think that should have the least amount of changes. Easier to expand too. What do you think?

---

<div class="post-metadata">

**Author:** ![diggy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/diggy/32/3666_2.png) [@diggy](https://discuss.dgraph.io/u/diggy)\
**Post date:** [March 4, 2019, 2:53pm UTC](https://discuss.dgraph.io/t/fix-querywithvars-to-handle-utf-8-values-correctly/7685/5 "2019-03-04T14:53:46Z")

</div>

[paulftw](https://github.com/paulftw) _commented_ :

Yes! haven’t seen your comment until now but been thinking the same idea on weekend - we should stop relying on HTTP headers, esp on custom ones.  
commitNow, vars (and everything else I’m not aware of) should be parts of the body.  
I was going to write a proposal for that.  
Not sure how easy it is to control Content-Type in browser and nodejs, if it is it’s a good idea, otherwise a query string param would do `/mutate?format=json`.

---

<div class="post-metadata">

**Author:** ![diggy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/diggy/32/3666_2.png) [@diggy](https://discuss.dgraph.io/u/diggy)\
**Post date:** [March 4, 2019, 3:01pm UTC](https://discuss.dgraph.io/t/fix-querywithvars-to-handle-utf-8-values-correctly/7685/6 "2019-03-04T15:01:16Z")

</div>

[paulftw](https://github.com/paulftw) _commented_ :

there’s also X-Dgraph-MutationType

---

<div class="post-metadata">

**Author:** ![diggy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/diggy/32/3666_2.png) [@diggy](https://discuss.dgraph.io/u/diggy)\
**Post date:** [October 3, 2019, 1:27pm UTC](https://discuss.dgraph.io/t/fix-querywithvars-to-handle-utf-8-values-correctly/7685/7 "2019-10-03T13:27:27Z")

</div>

[paulftw](https://github.com/paulftw) _commented_ :

This was an issue in the old API, in v1.1 it shouldn’t be a problem.

---

<div class="post-metadata">

**Author:** ![diggy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/diggy/32/3666_2.png) [@diggy](https://discuss.dgraph.io/u/diggy)\
**Post date:** [July 11, 2020, 1:12am UTC](https://discuss.dgraph.io/t/fix-querywithvars-to-handle-utf-8-values-correctly/7685/8 "2020-07-11T01:12:23Z")

</div>


