Support JSON-LD in Dgraph

MichelDiz · June 12, 2020, 4:26pm

Internet media type: application/ld+json

Motivation

There was the intention to provide support for JSON-LD in the past. But while we have support for standard JSON, JSON-LD has been pushed aside.

JSON-LD is used as an RDF representation in JSON. It provides context for the parser to understand this dataset as if it were RDF. Dgraph’s JSON does not support contextualization except through the blank node (which is a basic type of contextualization).

If we were to try to insert JSON-LD into Dgraph today, it would be a simple dataset, with no context and repeated data. It would be useless. However, it would not be possible, as some characters used in JSON-LD are blocked in Dgraph (see issue #4897 - Support reserved character "@" to be used in predicate naming). That is, it is not possible to import JSON-LD at all.

My proposal is that we support JSON-LD so that people using Triple Stores and other DBs that export in JSON-LD could bring their datasets to Dgraph. JSON-LD is also used in Google’s Knowledge Graph, Biomedical applications, provenance information, ActivityStreams, ActivityPub, Internet of things(IoT) and etc according to Wikipedia*.

Someone coming from a Triple Store or Ontology would feel comfortable if we had JSON-LD support.

The tool JSON-LD Playground is my big reference here. It can converts JSON-LD to N-Quads easily using JS. But it can’t handle megabytes or GB, or TB…

First things first

Support reserved character “@” to be used in predicate naming

This one the mandatory to fix, cuz some characters won’t work on Dgraph.

github.com/dgraph-io/dgraph

Support reserved character "@" to be used in predicate naming

opened 09:46PM - 07 Mar 20 UTC

MichelDiz

kind/bug priority/P3 status/accepted area/parsing area/mutations dgraph

## Experience Report reference: https://discuss.dgraph.io/t/support-json-ld-o…n-dgraph/7162 ### What you wanted to do Have the possibility to insert JSON-LD directly into Dgraph instead of having to convert the entire dataset manually. ### What you actually did I tried to run a JSON-LD with JSON mutation. ```JSON { "set": [ { "@context": "http://schema.org/", "@type": "Movie", "name": "Avatar", "Director": { "@type": "Person", "name": "James Cameron", "birthDate": "1954-08-16" }, "genre": "Science fiction", "trailer": "../movies/avatar-theatrical-trailer.html" } ] } ``` ### Why that wasn't great, with examples The "@" character is a character reserved for language support. However, it is a character that is used between two words and not at the beginning of a word. Therefore, it makes no sense to reserve it that way. Releasing this character would have a "kind of JSON-LD support". This would be a good step to allow different RDF based datasets to be loaded into Dgraph. ### Any external references to support your case Code reference https://github.com/dgraph-io/dgraph/blob/2a091db1608e4d3035f5f51546e67f5ff6c0b13b/worker/mutation.go#L356 User comment about JSON-LD https://github.com/dgraph-io/dgraph/issues/1#issuecomment-281198671 Also check https://github.com/dgraph-io/dgraph/issues/616 Tests I made with other characters #### Can not be mutate _: b0 <@context> "test". _: b0 <\\ context> "test". _: b0 <\ context> "test". _: b0 <~ context> "test". _: b0 <^ context> "test". _: b0 <`context>" test ". _: b0 <{context> "test". _: b0 <} context> "test". #### Can _: b0 <´context> "test". _: b0 <[context> "test". _: b0 <] context> "test". _: b0 <'context> "test". ![Captura de Tela 2020-03-07 às 18 33 12](https://user-images.githubusercontent.com/6570955/76152814-ec152a80-60a2-11ea-82e6-f8d72e74fc7a.png) Other JSON-LD examples: ```JSON { "@context": "http://schema.org/", "@type": "Person", "name": "Jane Doe", "jobTitle": "Professor", "telephone": "(425) 123-4567", "url": "http://www.janedoe.com" } ``` ```JSON { "@context": { "ical": "http://www.w3.org/2002/12/cal/ical#", "xsd": "http://www.w3.org/2001/XMLSchema#", "ical:dtstart": { "@type": "xsd:dateTime" } }, "ical:summary": "Lady Gaga Concert", "ical:location": "New Orleans Arena, New Orleans, Louisiana, USA", "ical:dtstart": "2011-04-09T20:00:00Z" } ``` ```JSON { "@context": { "name": "http://schema.org/name", "description": "http://schema.org/description", "image": { "@id": "http://schema.org/image", "@type": "@id" }, "geo": "http://schema.org/geo", "latitude": { "@id": "http://schema.org/latitude", "@type": "xsd:float" }, "longitude": { "@id": "http://schema.org/longitude", "@type": "xsd:float" }, "xsd": "http://www.w3.org/2001/XMLSchema#" }, "name": "The Empire State Building", "description": "The Empire State Building is a 102-story landmark in New York City.", "image": "http://www.civil.usherbrooke.ca/cours/gci215a/empire-state-building.jpg", "geo": { "latitude": "40.75", "longitude": "73.98" } } ``` ```JSON { "@context": "https://www.w3.org/ns/activitystreams", "@type": "Create", "actor": { "@type": "Person", "@id": "acct:sally@example.org", "name": "Sally" }, "object": { "@type": "Note", "content": "This is a simple note" }, "published": "2015-01-25T12:34:56Z" } ```

Add aliases at the schema level (In type)

This one would help a LOT. The users coming from Triple Store or Ontology.

github.com/dgraph-io/dgraph

Add aliases at the schema level (In type)

opened 03:37AM - 08 Mar 20 UTC

MichelDiz

kind/feature area/schema dgraph

## Experience Report reference: https://discuss.dgraph.io/t/support-json-ld-o…n-dgraph/7162 ### What you wanted to do Add aliases at the schema level. To preserve my dataset. ### Why that wasn't great, with examples It is not possible. e.g Take this RDF for example. ``` _:b0 <http://www.w3.org/2002/12/cal/ical#location> "New Orleans Arena, New Orleans, Louisiana, USA" . ``` It is a very common type of RDF dataset source. But it is hard to daily work or to use in APIs (like GraphQL). And every time you need to make a query, you need to manually write this whole line with an alias. So, my suggestion would have an alias at the schema level like this: ``` type Example { location : <http://www.w3.org/2002/12/cal/ical#location> } ``` So, instead of querying for `<http://www.w3.org/2002/12/cal/ical#location>` they would use `location` instead. And the predicate would be preserved in storage level. It is good to preserve this information, in case the user needs to use the RDF in an application that uses web semantics from W3C standards and other information that this type of file (RDF) usually has. It also prevents the user from having to sanitize the whole RDF, thereby completely changing its semantic information structure. >Even though we don't natively support the W3C's RDF standards. Still, if the user exports as JSON (Simulating JSON-LD). He can convert to W3C RDF easily using third-party applications.

Can also be related:

Deal better with Unrecognized RDF types.

Dgraph doesn’t support some types of RDF, so that would be a way to not lose information.

github.com/dgraph-io/dgraph

Deal better with Unrecognized RDF types.

opened 05:12PM - 11 Mar 20 UTC

MichelDiz

kind/feature status/accepted area/bulk-loader area/live-loader area/mutations dgraph

## Experience Report reference: https://discuss.dgraph.io/t/support-json-ld-o…n-dgraph/7162 ### What you wanted to do Import an RDF from a public dump. ### What you actually did ``` dgraph live -f ./agrovoc_2019-11-04_lod.nt --format=rdf Processing data file "./agrovoc_2019-11-04_lod.nt" [14:00:03-0300] Elapsed: 05s Txns: 470 N-Quads: 470000 N-Quads/s [last 5s]: 94000 Aborts: 0 Error while mutating: Attr: [http://www.w3.org/2004/02/skos/core#scopeNote] should have @lang directive in schema to mutate edge: [entity:6090028 attr:"http://www.w3.org/2004/02/skos/core#scopeNote" value:"Trozas que en el aserradero, se usan para la elaboraci\303\263n de la madera aserrada" lang:"es" ] s.Code Unknown Error while processing data file "./agrovoc_2019-11-04_lod.nt": During parsing chunk in processLoadFile: while parsing line "<http://aims.fao.org/aos/agrovoc/xl_tr_433_1321789680651> <http://www.w3.org/2004/02/skos/core#notation> \"1339129579719\"^^<http://aims.fao.org/aos/agrovoc/AgrovocCode> .\n": Unrecognized rdf type http://aims.fao.org/aos/agrovoc/AgrovocCode ``` e.g of the dataset ``` <http://aims.fao.org/aos/agrovoc/xl_cs_1299485709333> <http://www.w3.org/2004/02/skos/core#notation> "34938"^^<http://aims.fao.org/aos/agrovoc/AgrovocCode> . ``` ### Why that wasn't great, with examples We support the main RDF types from w3.org https://docs.dgraph.io/mutations/#language-and-rdf-types But that doesn't cover all types existent out there in Knowledge databases using RDF. My proposal would be to accept these types as default and to throw the type value to a facet. e.g: ``` <http://aims.fao.org/aos/agrovoc/xl_cs_1299485709333> <http://www.w3.org/2004/02/skos/core#notation> "34938" (unknown_type="<http://aims.fao.org/aos/agrovoc/AgrovocCode>") . ``` This is just an abstract example, Dgraph is not meant to transform the RDF during a load. I'm just exemplifying what I mean above. Supporting this, the user can later do something with it, such as using https://docs.dgraph.io/mutations/#external-ids-and-upsert-block

Example

It would not be complicated to support JSON-LD.

This JSON-LD:

{
  "@context": {
    "name": "http://xmlns.com/foaf/0.1/name",
    "homepage": {
      "@id": "http://xmlns.com/foaf/0.1/workplaceHomepage",
      "@type": "@id"
    },
    "Person": "http://xmlns.com/foaf/0.1/Person"
  },
  "@id": "https://me.example.com",
  "@type": "Person",
  "name": "John Smith",
  "homepage": "https://www.example.com/"
}

Would look like this in Dgraph’s RDF:

Use JSON-LD Playground to convert it to N-Quads. Only to analyze yourself.

_:Blank_node0 <xid> "http://xmlns.com/foaf/0.1/Person" .

_:Blank_node1 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> _:Blank_node0 .
_:Blank_node1 <http://xmlns.com/foaf/0.1/name> "John Smith" .
_:Blank_node1 <http://xmlns.com/foaf/0.1/workplaceHomepage> "https://www.example.com/" .
_:Blank_node1 <dgraph.type> "Person" .

Would be a simplified JSON:

It just ignores the “@context”.

{
  "id": "https://me.example.com",
  "dgraph.type": "Person",
  "name": "John Smith",
  "homepage": "https://www.example.com/"
}

Or a Contextualized one

Using Upsert Block cuz its easy to exemplify with it. Internally it should not use Upsert Block.

e.g In JSON-LD, the “@context” part is used to guide the parser and transform the JSON-LD into RDF like in a Triple Store.

   upsert {
      query {
        var(func: eq(xid, "http://xmlns.com/foaf/0.1/Person")) {
          Type as uid
        }
        var(func: eq(<http://xmlns.com/foaf/0.1/name>, "John Smith")) {
          Person as uid
        }
      }
      mutation {
          set {
		   uid(Type) <xid> "http://xmlns.com/foaf/0.1/Person" .
           uid(Person) <id> "https://me.example.com" .
           uid(Person) <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> uid(Type) .
		   uid(Person) <http://xmlns.com/foaf/0.1/workplaceHomepage> "https://www.example.com/" .
           uid(Person) <http://xmlns.com/foaf/0.1/name> "John Smith" .
           uid(Person) <dgraph.type> "Person" .
          }
      }
    }

“@id” is converted to “id”.
“@type” is converted to <dgraph.type>.
“name” is converted based on the context key “name”
“homepage” is converted based on the context key “homepage”

Simplified JSON-LD example

{
  "@context": "http://schema.org/",
  "@type": "Person",
  "name": "Jane Doe",
  "jobTitle": "Professor",
  "telephone": "(425) 123-4567",
  "url": "http://www.janedoe.com"
}

   upsert {
      query {
        var(func: eq(xid, "http://xmlns.com/foaf/0.1/Person")) {
          Type as uid
        }
        var(func: eq(<http://xmlns.com/foaf/0.1/name>, "Jane Doe")) {
          Person as uid
        }
      }
      mutation {
          set {
		   uid(Type) <xid> "http://xmlns.com/foaf/0.1/Person" .
		   uid(Person) <http://schema.org/jobTitle> "Professor" .
		   uid(Person) <http://schema.org/name> "Jane Doe" .
		   uid(Person) <http://schema.org/telephone> "(425) 123-4567" .
		   uid(Person) <http://schema.org/url> "http://www.janedoe.com" .
		   uid(Person) <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> uid(Type) .
		   uid(Person) <dgraph.type> "Person" .
          }
      }
    }

More examples

{
  "@context": {
    "ical": "http://www.w3.org/2002/12/cal/ical#",
    "xsd": "http://www.w3.org/2001/XMLSchema#",
    "ical:dtstart": {
      "@type": "xsd:dateTime"
    }
  },
  "ical:summary": "Lady Gaga Concert",
  "ical:location": "New Orleans Arena, New Orleans, Louisiana, USA",
  "ical:dtstart": "2011-04-09T20:00:00Z"
}

 _:b0 <http://www.w3.org/2002/12/cal/ical#dtstart> "2011-04-09T20:00:00Z"^^<http://www.w3.org/2001/XMLSchema#dateTime> .
 _:b0 <http://www.w3.org/2002/12/cal/ical#location> "New Orleans Arena, New Orleans, Louisiana, USA" .
 _:b0 <http://www.w3.org/2002/12/cal/ical#summary> "Lady Gaga Concert" .

The Simpsons example:

[
  {
    "@id": "#homer",
    "http://example.com/vocab#name": "Homer"
  },
  {
    "@id": "#bart",
    "http://example.com/vocab#name": "Bart",
    "http://example.com/vocab#parent": { "@id": "#homer" }
  },
  {
    "@id": "#lisa",
    "http://example.com/vocab#name": "Lisa",
    "http://example.com/vocab#parent": { "@id": "#homer" }
  }
]

_:bart <http://example.com/vocab#name> "Bart" .
_:bart <http://example.com/vocab#parent> _:homer .
_:homer <http://example.com/vocab#name> "Homer" .
_:lisa <http://example.com/vocab#name> "Lisa" .
_:lisa <http://example.com/vocab#parent> _:homer .

Syntax Tokens and Keywords

JSON-LD specifies a number of syntax tokens and keywords that are a core part of the language:

@context

Used to define the short-hand names that are used throughout a JSON-LD document. These short-hand names are called terms and help developers to express specific identifiers in a compact manner. The @context keyword is described in detail in section 5.1 The Context.

@id

Used to uniquely identify things that are being described in the document with IRIs or blank node identifiers. This keyword is described in section 5.3 Node Identifiers.

@value

Used to specify the data that is associated with a particular property in the graph. This keyword is described in section 6.9 String Internationalization and section 6.4 Typed Values.

@language

Used to specify the language for a particular string value or the default language of a JSON-LD document. This keyword is described in section 6.9 String Internationalization.

@type

Used to set the data type of a node or typed value. This keyword is described in section 6.4 Typed Values.

@container

Used to set the default container type for a term. This keyword is described in section 6.11 Sets and Lists.

@list

Used to express an ordered set of data. This keyword is described in section 6.11 Sets and Lists.

@set

Used to express an unordered set of data and to ensure that values are always represented as arrays. This keyword is described in section 6.11 Sets and Lists.

@reverse

Used to express reverse properties. This keyword is described in section 6.12 Reverse Properties.

@index

Used to specify that a container is used to index information and that processing should continue deeper into a JSON data structure. This keyword is described in section 6.16 Data Indexing.

@base

Used to set the base IRI against which relative IRIs are resolved. This keyword is described in section 6.1 Base IRI.

@vocab

Used to expand properties and values in @type with a common prefix IRI. This keyword is described in section 6.2 Default Vocabulary.

@graph

Used to express a graph. This keyword is described in section 6.13 Named Graphs.

:

The separator for JSON keys and values that use compact IRIs.

All keys, keywords, and values in JSON-LD are case-sensitive.

SOURCE: JSON-LD 1.0

JSON-LD 1.1 context for the JSON schema vocabulary

{
    "@context": {
        "@version": 1.1,
        "xsd": "http://www.w3.org/2001/XMLSchema#",
        "@vocab": "https://www.w3.org/2019/wot/json-schema#",
        "id": { "@id": "@id" },
        "type": { "@id": "@type" },
        "object": "ObjectSchema",
        "array": "ArraySchema",
        "boolean": "BooleanSchema",
        "string": "StringSchema",
        "number": "NumberSchema",
        "integer": "IntegerSchema",
        "null": "NullSchema",
        "properties": {
            "@type": "@vocab",
            "@container": "@index",
            "@index": "propertyName"
        },
        "items": { "@type": "@vocab" },
        "oneOf": { "@type": "@vocab", "@container": "@set" },
        "allOf": { "@type": "@vocab", "@container": "@set" },
        "anyOf": { "@type": "@vocab", "@container": "@set" },
        "minItems": { "@type": "xsd:decimal" },
        "maxItems": { "@type": "xsd:decimal" },
        "minimum": { "@type": "xsd:decimal" },
        "maximum": { "@type": "xsd:decimal" },
        "enum": { "@container": "@set", "@type": "@json" },
        "enum": { "@container": "@set", "@type": "@json" },
        "writeOnly": { "@type": "xsd:boolean" },
        "readOnly": { "@type": "xsd:boolean" },
        "format": { "@type": "xsd:string" },
        "required": { "@type": "xsd:string", "@container": "@set" },
        "title": { "@type": "xsd:string" },
        "description": { "@type": "xsd:string" }
    }
}

User Impact

The impact on the user would be none. As it would be optional.

Research

Most important one => JSON-LD Playground

https://w3c.github.io/wot-thing-description/ontology/jsonschema.html
https://www.w3.org/TR/2014/REC-json-ld-20140116/

keyvan-m-sadeghi · August 13, 2020, 5:35am

Perfect!

This is the only missing peace until I make the leap to Dgraph.

Anton · May 16, 2021, 5:07am

Likewise

Topic		Replies	Views
Support reserved character "@" to be used in predicate naming Dgraph dgraph , status:accepted , kind:bug , area:parsing , area:mutations	1	734	March 11, 2020
JSON-DGRAPH: RFC about Dgraph's JSON format Dev rfc , rdf , json	0	751	June 10, 2020
Increase RDF compliance support (N-Triples) Dev rdf	1	1262	July 29, 2020
Do you have plans for gpu-accelerated dgraph-ratel UI? Ratel	23	2577	September 4, 2018
Is Dgraph an RDF store? Dgraph faq	3	2498	July 15, 2020

Support JSON-LD in Dgraph

Motivation

First things first

Example

Simplified JSON-LD example

Syntax Tokens and Keywords

JSON-LD 1.1 context for the JSON schema vocabulary

User Impact

Research

Related topics