Dgraph Live/Increment StackTrace with System CA (mac client)

joaquin · August 31, 2020, 10:15pm

Report a Dgraph Bug

When using Dgraph Root CA installed in System CA, dgraph increment or dgraph live will stack trace.

What version of Dgraph are you using?

v20.03.4

Have you tried reproducing the issue with the latest release?

yes

What is the hardware spec (RAM, OS)?

$sw_vers
ProductName:	Mac OS X
ProductVersion:	10.14.6
BuildVersion:	18G103

Steps to reproduce the issue (command/config used to run Dgraph).

Steps

# create certificates/keys
dgraph cert -n localhost,dgraph.test -c dgraphuser

# install root ca into system
sudo security add-trusted-cert \
  -d -r trustRoot \
  -k /Library/Keychains/System.keychain \
  ./tls/ca.crt

# run dgraph alpha/zero
docker-compose up -d

# run tests - use clients

###### SUCCESS
curl https://localhost:8080/health | jq

###### FAIL: STACK TRACE
dgraph increment --tls_server_name "localhost" --tls_use_system_ca

###### FAIL: STACK TRACE
dgraph live \
  --tls_server_name localhost \
  --tls_use_system_ca \
  -f ./data/1million.rdf.gz \
  -s ./data/1million.schema \
  -a localhost:9080 \
  -z localhost:5080

docker-compose: .env

DGRAPH_ALPHA_TLS_CLIENT_AUTH=VERIFYIFGIVEN
DGRAPH_VERSION=v20.03.4

docker-compose.yaml

# docker-compose.yaml
version: "3.5"
services:
  tls-zero1:
    image: dgraph/dgraph:$DGRAPH_VERSION
    container_name: tls-zero1
    working_dir: /data/zero1
    ports:
      - 5080:5080
      - 6080:6080
    command: dgraph zero --my=tls-zero1:5080 --replicas 1 --idx 1

  tls-alpha1:
    image: dgraph/dgraph:$DGRAPH_VERSION
    container_name: tls-alpha1
    working_dir: /data/alpha1
    volumes:
      - type: bind
        source: ./tls/
        target: /tls
        read_only: true
    environment:
      DGRAPH_ALPHA_TLS_DIR: /tls
      DGRAPH_ALPHA_TLS_CLIENT_AUTH: ${DGRAPH_ALPHA_TLS_CLIENT_AUTH}
    ports:
      - 8080:8080
      - 9080:9080
    command: dgraph alpha --my=tls-alpha1:7080 --lru_mb=1024 --zero=tls-zero1:5080

Expected behaviour and actual result.

After installing Dgraph Root CA ./ca.crt into the system, it should work without the need to explicitly specify the ca.crt.

This works fine for curl but fails for dgraph increment and dgraph live

Curl Works Fine

Before, this was required --cacert ./tls/ca.crt, but it is no longer needed:

curl https://localhost:8000/health

Crash Using Increment

You need to include --tls_cacert ./tls/ca.crt or dgraph increment stack traces:

$ dgraph increment --tls_server_name "localhost" --tls_use_system_ca
[Decoder]: Using assembly version of decoder

Running transaction with dgraph endpoint: localhost:9080
2020/08/31 14:52:12 --tls_cacert is required for enabling TLS
github.com/dgraph-io/dgraph/x.LoadClientTLSConfig
	/ext-go/1/src/github.com/dgraph-io/dgraph/x/tls_helper.go:113
github.com/dgraph-io/dgraph/x.GetDgraphClient
	/ext-go/1/src/github.com/dgraph-io/dgraph/x/x.go:762
github.com/dgraph-io/dgraph/dgraph/cmd/counter.run
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/counter/increment.go:172
github.com/dgraph-io/dgraph/dgraph/cmd/counter.init.0.func1
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/counter/increment.go:45
github.com/spf13/cobra.(*Command).execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:70
main.main
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/main.go:78
runtime.main
	/usr/local/go/src/runtime/proc.go:203
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1373
While loading TLS configuration
github.com/dgraph-io/dgraph/x.Checkf
	/ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:51
github.com/dgraph-io/dgraph/x.GetDgraphClient
	/ext-go/1/src/github.com/dgraph-io/dgraph/x/x.go:763
github.com/dgraph-io/dgraph/dgraph/cmd/counter.run
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/counter/increment.go:172
github.com/dgraph-io/dgraph/dgraph/cmd/counter.init.0.func1
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/counter/increment.go:45
github.com/spf13/cobra.(*Command).execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:70
main.main
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/main.go:78
runtime.main
	/usr/local/go/src/runtime/proc.go:203
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1373

Crash Using Live Loader

You need to include --tls_cacert ./tls/ca.crt or dgraph increment stack traces:

$ dgraph live --tls_server_name localhost --tls_use_system_ca -f ./data/1million.rdf.gz -s ./data/1million.schema -a localhost:9080 -z localhost:5080
[Decoder]: Using assembly version of decoder
I0831 14:59:10.590105   41449 init.go:99] 

Dgraph version   : v20.03.1
Dgraph SHA-256   : cbd3c7fd96d08f6b78259a10e03d70ab85a4afd3c9a4c68daac6358fe84a4b6e
Commit SHA-1     : c201611d6
Commit timestamp : 2020-04-24 13:53:41 -0700
Branch           : HEAD
Go version       : go1.14.1

For Dgraph official documentation, visit https://docs.dgraph.io.
For discussions about Dgraph     , visit http://discuss.dgraph.io.
To say hi to the community       , visit https://dgraph.slack.com.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2020 Dgraph Labs, Inc.



Running transaction with dgraph endpoint: localhost:9080
2020/08/31 14:59:10 --tls_cacert is required for enabling TLS
github.com/dgraph-io/dgraph/x.LoadClientTLSConfig
	/ext-go/1/src/github.com/dgraph-io/dgraph/x/tls_helper.go:113
github.com/dgraph-io/dgraph/x.GetDgraphClient
	/ext-go/1/src/github.com/dgraph-io/dgraph/x/x.go:762
github.com/dgraph-io/dgraph/dgraph/cmd/live.run
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/run.go:435
github.com/dgraph-io/dgraph/dgraph/cmd/live.init.0.func1
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/run.go:119
github.com/spf13/cobra.(*Command).execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:70
main.main
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/main.go:78
runtime.main
	/usr/local/go/src/runtime/proc.go:203
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1373
While loading TLS configuration
github.com/dgraph-io/dgraph/x.Checkf
	/ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:51
github.com/dgraph-io/dgraph/x.GetDgraphClient
	/ext-go/1/src/github.com/dgraph-io/dgraph/x/x.go:763
github.com/dgraph-io/dgraph/dgraph/cmd/live.run
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/run.go:435
github.com/dgraph-io/dgraph/dgraph/cmd/live.init.0.func1
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/run.go:119
github.com/spf13/cobra.(*Command).execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:70
main.main
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/main.go:78
runtime.main
	/usr/local/go/src/runtime/proc.go:203
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1373

dmai · August 31, 2020, 10:21pm

I don’t think this is a macOS-specific issue. That --tls_cacert option is required for TLS connections.

github.com

dgraph-io/dgraph/blob/bf799992473cf334947f8507b001883333ad2d88/x/tls_helper.go#L131


		}

		return &tlsCfg, nil
	} else
	// Attempt to determine if user specified *any* TLS option. Unfortunately and contrary to
	// Viper's own documentation, there's no way to tell whether an option value came from a
	// command-line option or a built-it default.
	if v.GetString("tls_server_name") != "" ||
		v.GetString("tls_cert") != "" ||
		v.GetString("tls_key") != "" {
		return nil, errors.Errorf("--tls_cacert is required for enabling TLS")
	}
	return nil, nil
}

func generateCertPool(certPath string, useSystemCA bool) (*x509.CertPool, error) {
	var pool *x509.CertPool
	if useSystemCA {
		var err error
		if pool, err = x509.SystemCertPool(); err != nil {
			return nil, err

joaquin · August 31, 2020, 11:08pm

@dmai The CA Certificate is installed into system system, so it works with browsers and curl, why not dgraph increment/live?

This is a confusing from UX standpoint, for these two use cases:

operator installed Dgraph Root CA into system, so the option to for tls_use_system_ca doesn’t make sense, if you have to explicitly and redundantly specify the ca.crt.
this looks like it would prevent using actual trusted Root CA, because the provider would unlikely export their trusted root ca for dgraph.

Topic		Replies	Views
Dgraph live fails on macOS Mojave Dgraph dgraph , kind:bug , area:live-loader	1	536	May 6, 2020
State.commit is out of range Dgraph	3	1201	February 15, 2021
Dgraph + TLS is not working Users	17	1307	June 3, 2020
Tour intro does not implement new superflags Dgraph	4	678	April 14, 2021
[Alpha] Problems with --auth_token GraphQL dgraph , status:accepted , kind:bug , ticket:created	3	678	January 5, 2021