How do I know that my data is in correct JSON format?
Because I am trying to run my JSON formatted data which I have converted from csv,but it is showing an error that invalid mutation.
We have docs related to JSON usage https://docs.dgraph.io/mutations/#json-mutation-format
I have converted my csv formatted data into JSON which contains 98522 triples. So when I add 20-25 triples it works fine but as I increase the data size it is showing rpc error: Transaction Aborted.
Is there any problem with data size?
I don’t thing so. I have a small project with a big JSON(see image below) and I run it via dgraph-js - It is fast and has no issues. This tree is a set of cities and gps localizations. Has more than 5k cities and double of gps track.
I performed this in a single transaction.
I want to perform data scaling operation. Like, 1Gb dataset[25 lac triples] then 5Gb dataset. But in Mozilla this 200mb dataset[98522 triples] is not running so any alternative way if you can suggest.
You can use LiveLoad https://docs.dgraph.io/deploy/#live-loader
It accepts JSON load.
Aborts can happen if you’re writing to the same data from different transactions. When you run live loader, try to set the concurrency so there are very few aborts. Aborts would cause your load speed to go down significantly, so avoiding them is the best. If you set the concurrency to 1, there’d be no aborts. So, start with the default, then go down until you reach a sweet spot (hopefully greater than 1).
Live loader doesn’t accept JSON Data. It only handles RDF N-Quad.
Yeah, but it is on master. You can use the Docker tag “master” to use the binarie just for this liveloader instance.
#pick the flag -f, --files string Location of *.rdf(.gz) or *.json(.gz) file(s) to load
[node1] (local) email@example.com ~ $ docker run -it -p 5080:5080 -p 6080:6080 dgraph/dgraph:master dgraph live -h Unable to find image 'dgraph/dgraph:master' locally master: Pulling from dgraph/dgraph 124c757242f8: Already exists 9d866f8bde2a: Already exists fa3f2f277e67: Already exists 398d32b153e8: Already exists afde35469481: Already exists 275fe35919d5: Already exists 43b451c9b8c2: Pull complete cf89c21a4062: Pull complete Digest: sha256:7aad04f3895a3fdca1eac9b5a06c6d63d40312aa9e95b3af7af0a1e488050f79 Status: Downloaded newer image for dgraph/dgraph:master Run Dgraph live loader Usage: dgraph live [flags] Flags: -a, --auth_token string The auth token passed to the server for Alter operation of the schema file -b, --batch int Number of N-Quads to send as part of a mutation. (default 1000) -c, --conc int Number of concurrent requests to make to Dgraph (default 10) -d, --dgraph string Dgraph alpha gRPC server address (default "127.0.0.1:9080") -f, --files string Location of *.rdf(.gz) or *.json(.gz) file(s) to load --format string Specify file format (rdf or json) instead of getting it from filename -h, --help help for live -i, --ignore_index_conflict Ignores conflicts on index keys during transaction (default true) --new_uids Ignore UIDs in load files and assign new ones. -s, --schema string Location of schema file --tls_cacert string The CA Cert file used to verify server certificates. --tls_cert string (optional) The Cert file provided by the client to the server. --tls_key string (optional) The private key file provided by the client to the server. --tls_server_name string Used to verify the server hostname. --tls_use_system_ca Include System CA into CA Certs. (default true) -C, --use_compression Enable compression on connection to alpha server -x, --xidmap string Directory to store xid to uid mapping -z, --zero string Dgraph zero gRPC server address (default "127.0.0.1:5080") Global Flags: --alsologtostderr log to standard error as well as files --bindall Use 0.0.0.0 instead of localhost to bind to all addresses on local machine. (default true) --block_rate int Block profiling rate. Must be used along with block profile_mode --config string Configuration file. Takes precedence over default values, but is overridden to values set with environment variables and flags. --cwd string Change working directory to the path specified. The parent must exist. --expose_trace Allow trace endpoint to be accessible from remote --log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0) --log_dir string If non-empty, write log files in this directory --logtostderr log to standard error instead of files --profile_mode string Enable profiling mode, one of [cpu, mem, mutex, block] -v, --v Level log level for V logs --vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
Note: Don’t use master for other purposes. Just to use liveloader. You can also use Bulkloader that already has JSON support.
Okay…but without docker I can’t use …right?
Yes you can! (LOL) - Just build Dgraph from master branch. On Dgraph’s Readme there’s the steps to do it.
ohh, okay okay…!
Thanks a ton to answer all my questions(some silly). I will surely mention your team in my thesis acknowledgement.
To build Dgraph from master’s branch I have followed README.md and it shows that you have already installed it. So does it mean that version v1.0.13 is from master’s branch?
Because Dgraph live does not have flags --files in my installed version.
When you build from master you gonna have a bin at ~/go/src/github.com/dgraph-io/dgraph/dgraph path. You need to move this bin to a desired place and run like so:
No, master is master. Master is like v1.1 not v1.0.xx