Distributed Tracing through the dgo client

Users Dgraph
,
1

Hello Team,

For my services Distributed Tracing needs, I am using opencensus and the good thing is it already has integrations to capture grpc/http/sql logs. I know that Dgraph provides distributed traces but these are from database end. Is it possible to integrate it using Dgo Client, so that we see the complete trace including dgraph spans.

2

You can set up your app so that it sends distributed traces to Jaeger. It’d look like the example in OpenCensus tracing docs: https://opencensus.io/guides/http/go/net_http/client/. You’ll want to call trace.RegisterExporter from your app and then pass in your context.Context from your app through to dgo.

Jaeger would show the distributed traces from your app to Dgraph.

1 Like
3

Hi @dmai , Thanks for your response. We already have these configuration for our application.
Jaeger Configuration

`	// Configure Jaeger
	if cfg.Instrumentation.Jaeger.Enabled {
		logger.Log.Info("jaeger exporter enabled")

		exporter, err := jaeger.NewExporter(cfg.Instrumentation.Jaeger.Config)
		if err != nil {
			logger.Log.Fatal("Jaeger Exporter Error")
		}
		trace.RegisterExporter(exporter)
	}`

dgo client invoke

_, err := lr.dg.NewTxn().Do(ctx, &api.Request{
		CommitNow: true,
		Query:     q,
		Mutations: []*api.Mutation{mu}},
	)

Trace with Dgraph

image
image1361Ă—330 32.8 KB

Trace with SQL DB

image
image1361Ă—410 42.7 KB

Is there some reference code/example we can look at?

4

@dmai, Do you think I am missing something.

5

I think you are not putting your Jaeger trace into ctx

6

Hello @dmai @chewxy, Context is being passed to grpc service method as visible in jaeger screenshot above(equipment-service), which is then propagated to Drgaph dgo client. I am not sure if something is getting missed in between.

7

Hello Team, Any update on this.

8

Can you send me a larger snippet of your code? I’m trying to understand what you want to do here

9

Hello @chewxy,

Thanks for your response. The code is there on github and below is some of the information.

Our goal is to see Dgraph spans as well in Distributed trace of application requests. We are not sure if we need to enable some dgraph feature for this or add some handler/wrapper in dgo client. As per the initial response from @dmai, we are passing context down to dgo client.

Register Exporter - optisam-backend/server.go at 08d020476f70e7c47614a4f9305ce2cdbd878e57 · Orange-OpenSource/optisam-backend · GitHub

GRPC Server Setup - optisam-backend/server.go at 08d020476f70e7c47614a4f9305ce2cdbd878e57 · Orange-OpenSource/optisam-backend · GitHub

This is all the setup we have as of now. And we can see from screenshots in the discussion thread, that we see spans for HTTP receive, HTTP sent, GRPC recieve.

GRPC Service Method - optisam-backend/equip.go at 08d020476f70e7c47614a4f9305ce2cdbd878e57 · Orange-OpenSource/optisam-backend · GitHub

Dgraph DB implementation - optisam-backend/equip.go at 08d020476f70e7c47614a4f9305ce2cdbd878e57 · Orange-OpenSource/optisam-backend · GitHub

Let me know if you need more info.

Thanks in advance

10

OK Will look into it now. give me about 12 hrs (it’s 11pm for me now)

1 Like
11

Hi, dharmjit, I think it’s best we schedule a call. It’d be faster. Sending you a PM

12

Hello @chewxy, As discussed in the call please find the details below

The code is there on github and below is some of the information.

Our goal is to see Dgraph spans as well in Distributed trace of application requests. We are not sure if we need to enable some dgraph feature for this or add some handler/wrapper in dgo client. As per the initial response from @dmai, we are passing context down to dgo client.

Register Exporter - optisam-backend/server.go at 08d020476f70e7c47614a4f9305ce2cdbd878e57 · Orange-OpenSource/optisam-backend · GitHub

GRPC Server Setup - optisam-backend/server.go at 08d020476f70e7c47614a4f9305ce2cdbd878e57 · Orange-OpenSource/optisam-backend · GitHub

This is all the setup we have as of now. And we can see from screenshots in the discussion thread, that we see spans for HTTP receive, HTTP sent, GRPC recieve.

GRPC Service Method - optisam-backend/equip.go at 08d020476f70e7c47614a4f9305ce2cdbd878e57 · Orange-OpenSource/optisam-backend · GitHub

Dgraph DB implementation - optisam-backend/equip.go at 08d020476f70e7c47614a4f9305ce2cdbd878e57 · Orange-OpenSource/optisam-backend · GitHub

Let me know if you need more info.

Thanks in advance

1 Like
13

Hi, a bit of an update. I was not able to get resources for last week to add Jaeger into dgo as most of the team last week were busy with some other problem. Will try this week

14

No worries @chewxy , let me know if I can help in some way on this.

Thanks

15

Hello @chewxy any update for me. Thanks

16

For those following/finding this thread via search, here are the important bits to set up a dgo/grpc app to work with distributed tracing.

  1. It’s important to register your app as an exporter
	exporter, err := jaeger.NewExporter(jaeger.Options{
		ServiceName:       "your.app.name",
		CollectorEndpoint: "http://localhost:14268/api/traces",
		Process: jaeger.Process{
			ServiceName: "your.app.name",
		},
	})
	if err != nil {
		log.Fatal(err)
	}
	trace.RegisterExporter(exporter)
  1. You may want to increase the sampling ratio to test that your integration is working
	trace.ApplyConfig(trace.Config{
		DefaultSampler:             trace.ProbabilitySampler(1.0),
		MaxAnnotationEventsPerSpan: 256,
	})
  1. You’ll need to add a stats handler to your grpc dial connection request
import "go.opencensus.io/plugin/ocgrpc"
...
conn, err := grpc.Dial(addr, grpc.WithInsecure(), 
    grpc.WithStatsHandler(&ocgrpc.ClientHandler{}))

With this setup, you can create spans that will show up under your top level “service” and the distributed traces that are called from the request (spans in alpha and zero) will be viewable.

ctx, span := trace.StartSpan(context.Background(), "test.mutate-something")
span.AddAttributes(...)
defer span.End()

_, err := txn.Mutate(ctx, &mu)
1 Like