Insert become slow until can't insert

Issues Dgraph Clients
,
21

lxwithgod commented :

i don’t konw why，so i wait until 1.0

22

manishrjain commented :

There’s not enough information here for us to reproduce this. Maybe @janardhan1993 can have a look, and see if we should try and figure what’s going on here.

23

deepakjois commented :

I have now run the java client code against a dgraph server in EC2 for 24 hours without any interruption. I have not encountered any errors. I think we can safely rule out that this is a problem related to the Java client.

24

janardhan1993 commented :

@lxwithgod: Can you please try again with the master build.

25

lxwithgod commented :

@janardhan1993 :i don’t know build the project,i never use golang.so,i use curl https://get.dgraph.io -sSf | bash.so,can you help me?

26

lxwithgod commented :

please

27

janardhan1993 commented :

I will update this thread once we publish the binary for master. You can then use the binary directly.

28

lxwithgod commented :

thanks @janardhan1993

29

janardhan1993 commented :

@lxwithgod : I have triggered the nightly build. Please use curl https://nightly.dgraph.io -sSf | bash after 1 hour to get the latest binary.

30

deepakjois commented :

@lxwithgod : you might need to use the latest java client code from master if you are going to use the nightly build of dgraph. We havent made a new release, but there are quite a few changes in the master branch.

31

lxwithgod commented :

server have not broken,but,when i write data until 189*30000,data can’t be writen.

32

lxwithgod commented :

query slow,about 16s.

33

janardhan1993 commented :

Sorry, we need to fix one more issue. I will test the performance and trigger another binary build.

34

janardhan1993 commented :

@lxwithgod : Can you please try now with latest nightly build. Btw is it possible to share your data, may be you can mask confidential data and share a subset.

35

lxwithgod commented :

ok，i try it.
i use the cell phone number as node.

36

lxwithgod commented :

2017/11/30 14:19:37 groups.go:300: Asking if I can serve tablet for: phone
2017/11/30 14:24:01 groups.go:300: Asking if I can serve tablet for: dummy
已杀死
the number of data reach 10 million.

37

janardhan1993 commented :

@lxwithgod : What happened after it reached 10 million ?

38

lxwithgod commented :

server crash.but i don’t use index.

39

janardhan1993 commented :

can you share your code ?

40

lxwithgod commented :

package driver;

import com.dataPlatform.protobuf.ByteString;
import com.dataPlatform.protobuf.InvalidProtocolBufferException;
import io.dgraph.DgraphClient;
import io.dgraph.DgraphProto;
import tool.Json;
import tool.Log;

import java.io.;
import java.util.;

/**

  • Created by lx on 17-11-20.
    */
    public class DgraphDiver extends DgraphBaseDriver{
    final static Log log=Log.newInstance(DgraphDiver.class);

    private Object mutate(Object… args){
    DgraphClient.Transaction transaction = driver.newTransaction();
    try {

         DgraphProto.Assigned rs = transaction.mutate((DgraphProto.Mutation) args[0]);
     transaction.commit();
     return rs;
 }catch (Exception e) {
     log.warn(e);
     return null;
 }finally {
     transaction.discard();
 }

    }

    @Override
    public Object client(Object args){
    return driver;

    }

    @Override
    public Object update(Object… args) {
    DgraphProto.Mutation mu =
    DgraphProto.Mutation.newBuilder()
    .setSetNquads(ByteString.copyFromUtf8(args[0].toString()))
    .build();
    return mutate(mu);
    }

    @Override
    public Object create(Object… args) {
    DgraphProto.Mutation mu =
    DgraphProto.Mutation.newBuilder()
    .setSetJson(ByteString.copyFromUtf8(
    Json.objectAsString(args[0])))
    .build();
    return mutate(mu);
    }

    @Override
    public Object retrieve(Object… args) {
    try {
    if(args.length==1) {

             return driver.newTransaction().query((String) args[0])
                 .getJson().toStringUtf8();
     }else {
         return driver.newTransaction().queryWithVars((String) args[0],(Map)args[1])
                 .getJson().toStringUtf8();
     }

 }catch (Exception e) {
     log.warn(e);
     return null;
 }

    }

    @Override
    public Object delete(Object… args) {
    DgraphProto.Mutation mu =
    DgraphProto.Mutation.newBuilder()
    .setDelNquads(ByteString.copyFromUtf8(args[0].toString()))
    .build();
    return mutate(mu);
    }

    public void setSchema(String schema){
    DgraphProto.Operation op = DgraphProto.Operation
    .newBuilder().setSchema(schema).build();
    driver.alter(op);
    }

    public static void main(String args) throws IOException {
    DgraphDiver dgraphDriver = new DgraphDiver();
    dgraphDriver.client(null,null,null,null);
    // dgraphDriver.driver.alter(DgraphProto.Operation.newBuilder().setDropAll(true).build());
    // dgraphDriver.setSchema(“phone:string@index(exact) .”);
    // dgraphDriver.setSchema(“call:uid@count @reverse .”);
    String phone_path="/media/lx/0c430a79-8c39-4d87-865a-b54bb8d4f738/idealProjects/gxwl/phone_distinct_2";
    BufferedReader bf = new BufferedReader(new FileReader(phone_path));
    String line=null;

     LinkedList<Map> phones=new LinkedList<>();
 int i=0;
 while ((line=bf.readLine())!=null){
     String phone=line;
     if(!phone.contains("*") && phone.length()==11){
         LinkedHashMap<String,String> js=new LinkedHashMap<>();
         js.put("phone",phone);
         phones.add(js);
     }
     if(phones.size()==40000){
         dgraphDriver.create(phones);
         phones.clear();
         System.out.println(i++);
     }
 }
 dgraphDriver.create(phones);

// dgraphDriver.setSchema(“phone:string @index(exact) .”);
// dgraphDriver.setSchema(“call: uid @count @reverse .”);

// DgraphClient.Transaction transaction = dgraphDriver.driver.newTransaction();
//
// String nqs="<0x120><0x119>.";
// DgraphProto.Mutation mutation = DgraphProto.Mutation.newBuilder()
// .setSetNquads(ByteString.copyFromUtf8(
// nqs))
// .build();
// transaction.mutate(mutation);
// transaction.commit();

}

}