Hello,
How do I know that my data is in correct JSON format?
Because I am trying to run my JSON formatted data which I have converted from csv,but it is showing an error that invalid mutation.
We have docs related to JSON usage https://docs.dgraph.io/mutations/#json-mutation-format
I have converted my csv formatted data into JSON which contains 98522 triples. So when I add 20-25 triples it works fine but as I increase the data size it is showing rpc error: Transaction Aborted.
Is there any problem with data size?
I donât thing so. I have a small project with a big JSON(see image below) and I run it via dgraph-js - It is fast and has no issues. This tree is a set of cities and gps localizations. Has more than 5k cities and double of gps track.
I performed this in a single transaction.
I want to perform data scaling operation. Like, 1Gb dataset[25 lac triples] then 5Gb dataset. But in Mozilla this 200mb dataset[98522 triples] is not running so any alternative way if you can suggest.
You can use LiveLoad https://docs.dgraph.io/deploy/#live-loader
It accepts JSON load.
1 Like
Aborts can happen if youâre writing to the same data from different transactions. When you run live loader, try to set the concurrency so there are very few aborts. Aborts would cause your load speed to go down significantly, so avoiding them is the best. If you set the concurrency to 1, thereâd be no aborts. So, start with the default, then go down until you reach a sweet spot (hopefully greater than 1).
1 Like
Live loader doesnât accept JSON Data. It only handles RDF N-Quad.
Yeah, but it is on master. You can use the Docker tag âmasterâ to use the binarie just for this liveloader instance.
#pick the flag
-f, --files string Location of *.rdf(.gz) or *.json(.gz) file(s) to load
[node1] (local) root@192.168.0.23 ~
$ docker run -it -p 5080:5080 -p 6080:6080 dgraph/dgraph:master dgraph live -h
Unable to find image 'dgraph/dgraph:master' locally
master: Pulling from dgraph/dgraph
124c757242f8: Already exists
9d866f8bde2a: Already exists
fa3f2f277e67: Already exists
398d32b153e8: Already exists
afde35469481: Already exists
275fe35919d5: Already exists
43b451c9b8c2: Pull complete
cf89c21a4062: Pull complete
Digest: sha256:7aad04f3895a3fdca1eac9b5a06c6d63d40312aa9e95b3af7af0a1e488050f79
Status: Downloaded newer image for dgraph/dgraph:master
Run Dgraph live loader
Usage:
dgraph live [flags]
Flags:
-a, --auth_token string The auth token passed to the server for Alter operation of the schema file
-b, --batch int Number of N-Quads to send as part of a mutation. (default 1000)
-c, --conc int Number of concurrent requests to make to Dgraph (default 10)
-d, --dgraph string Dgraph alpha gRPC server address (default "127.0.0.1:9080")
-f, --files string Location of *.rdf(.gz) or *.json(.gz) file(s) to load
--format string Specify file format (rdf or json) instead of getting it from filename
-h, --help help for live
-i, --ignore_index_conflict Ignores conflicts on index keys during transaction (default true)
--new_uids Ignore UIDs in load files and assign new ones.
-s, --schema string Location of schema file
--tls_cacert string The CA Cert file used to verify server certificates.
--tls_cert string (optional) The Cert file provided by the client to the server.
--tls_key string (optional) The private key file provided by the client to the server.
--tls_server_name string Used to verify the server hostname.
--tls_use_system_ca Include System CA into CA Certs. (default true)
-C, --use_compression Enable compression on connection to alpha server
-x, --xidmap string Directory to store xid to uid mapping
-z, --zero string Dgraph zero gRPC server address (default "127.0.0.1:5080")
Global Flags:
--alsologtostderr log to standard error as well as files
--bindall Use 0.0.0.0 instead of localhost to bind to all addresses on local machine. (default true)
--block_rate int Block profiling rate. Must be used along with block profile_mode
--config string Configuration file. Takes precedence over default values, but is overridden to values set with environment variables and flags.
--cwd string Change working directory to the path specified. The parent must exist.
--expose_trace Allow trace endpoint to be accessible from remote
--log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)
--log_dir string If non-empty, write log files in this directory
--logtostderr log to standard error instead of files
--profile_mode string Enable profiling mode, one of [cpu, mem, mutex, block]
-v, --v Level log level for V logs
--vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
Note: Donât use master for other purposes. Just to use liveloader. You can also use Bulkloader that already has JSON support.
OkayâŚbut without docker I canât use âŚright?
Yes you can! (LOL) - Just build Dgraph from master branch. On Dgraphâs Readme thereâs the steps to do it.
ohh, okay okayâŚ!
Thanks a ton to answer all my questions(some silly). I will surely mention your team in my thesis acknowledgement.
1 Like
Hey,
To build Dgraph from masterâs branch I have followed README.md and it shows that you have already installed it. So does it mean that version v1.0.13 is from masterâs branch?
Because Dgraph live does not have flags --files in my installed version.
When you build from master you gonna have a bin at ~/go/src/github.com/dgraph-io/dgraph/dgraph path. You need to move this bin to a desired place and run like so:
./dgraph something...
No, master is master. Master is like v1.1 not v1.0.xx
Cheers.
1 Like
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.