How can I set text with JSON encoded emojis using DQL?

Users Dgraph
#1

I try to set an emoji like this to a node predicate.
:grinning:
This character is converted to encoded string in JSON “\uD83D\uDE00”

{
	set {
		<0x1f> <post-text> "\uD83D\uDE00" .
  }
}

After executing above mutation I get the following:

image

I also checked the returned text in browser and clearly Dgraph corrupts encoded string values.

"post-text": "\uFFFD\uFFFD",

This is the value I get. That is why smiley is shown as question marks in the browser.

Is this a bug or am I doing something wrong?

1 Like
#2

Try \\uD83D\\uDE00

#3

That is not a valid unicode encoding. \u is a special indicator that following characters are the hex representation of underlying unicode character.
Check in the browser console as below.

image

Apparently, Dgraph serializer does not respect Unicode encoding or there is a bug.

#4

Dgraph doesn’t support special characters. You have to follow the JSON escaping rules in order to store those values. You have to(on your end) scape and unscape those before sending/reading them to Dgraph.

@docs maybe we should document this.

#5

The JSON specification says: ’ To escape an extended character that is not in the Basic Multilingual Plane, the character is represented as a twelve-character sequence, encoding the UTF-16 surrogate pair. So, for example, a string containing only the G clef character (U+1D11E) may be represented as “\uD834\uDD1E”. ’

This is not special character. This is the definition of JSON serialization and deserialization.
I send values to Dgraph using .NET JsonSerializer and it converts all emojis encoded.
If DQL language designed to work on JSON data, then it would be wise to support this.

https://tools.ietf.org/html/rfc4627#section-2.5

Related discussion: JsonSerializer.Serialize support for emojis defined with surrogate pairs · Issue #42847 · dotnet/runtime · GitHub

#6

Dgraphs’ string doesn’t support JSON. But it follows the JSON rules of scaping special characters. A Slash is a special character.

#7

We might create a bug / feature request to not to convert given input
“\uD83D\uDE00”
into this:
“\uFFFD\uFFFD”

If this syntax is not supported, Dgraph should throw an exception instead of converting the bytes into some other number.

#8

If we throw an exception every time we see a special character, the users will be really mad cuz it will happen all the time. We could have a scaping directive, but for now, is perfectly fine to have it on your end.

#9

Yes, I handled the situation on my end. I insist on this to improve Dgraph.
These kind of issues can be send to backlog to be implemented in the future.

#11

If I am not wrong, DQL parser unquotes a string using primitive strconv Unquote function.

image
image1398×687 87.3 KB

I am newbie on Go lang however I found following:

This might end up with one line code change sth like below.

json.Unmarshal([]byte(str), &str)
#12

I’ll look into it - on a related note there is a PR I have for GraphQL (not DQL) that is for a similar issue (it’s yet to be merged):

1 Like