PyDgraph failing to call self-hosted dgraph (in k8s) over grpc

Issues Dgraph Clients
,
1

What I want to do

I am trying to use pydgraph client over a private ingress on kubernetes to talk to a self-hosted dgraph (based on the tutorial on how to host dgraph in kubernetes).

So I have definitely verified that things work over graphql with a private ingress. But when I try to create a client with pydgraph and query my graph, it throws the following error:

---------------------------------------------------------------------------
_InactiveRpcError  
...
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses"
	debug_error_string = "{"created":"@1637784489.545899000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3009,"referenced_errors":[{"created":"@1637784489.545897000","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":398,"grpc_status":14}]}"
>

What I did

  • Since pydgraph seems to talk to dgraph over grpc, I created a separate grpc ingress (https://dgraph-grpc.hostname.com), and tried calling it with that, and I get that error but I can definitely make graphql queries on the http ingress (https://dgraph.hostname.com)

  • I tried creating a client from within k8s with the IP and port and it worked fine, which makes me wonder if there’s something that I am missing with TLS.

  • I also tried to create a dgraph client with my free dgraph cloud and I still get similar RPC errors.

_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses"
	debug_error_string = "{"created":"@1637785094.658682000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3009,"referenced_errors":[{"created":"@1637785094.658679000","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":398,"grpc_status":14}]}"
>
  • But if I try to create a pydgraph client with pydgraph.DgraphClientStub.from_cloud(host, api_key), it works for dgraph-cloud. Does this mean I need to set an api_key somewhere in my own private ingress?

Any help here would be deeply appreciated!

1 Like
2

gRPC in K8s is quite a challenge sometimes. There are some topics here in discuss about it.

3

Do you have any particular post? I looked around the forum a bit and didn’t really find anything relevant but I may be searching for the wrong things.

4

This one How to: Live Load distributed with Kubernetes, Docker or Binary - Users / Dgraph - Discuss Dgraph

this Latest area:kubernetes topics - Discuss Dgraph

This one Using Kubernetes - Deploy - Documentation - Discuss Dgraph

5

TY. The third link is how I set up our local dgraph. I will look into your first link to figure it out - I don’t think it talks about the same things but I may be missing something. I will read it in detail and respond back if I can figure out what’s going on. Thanks MichelDiz.

6

Not sure if I am missing something - but neither of those links really do anything with ingresses. As mentioned in the original post, hitting the internal IP with the port works fine, it’s the private ingress that’s causing me issues.

7

That’s it, I don’t know how to help you there. “gRPC in K8s is quite a challenge sometimes”. I know that someone figure out that envoy works with gRPC. Not sure, I haven’t touched K8s for a long time. I hope someone with k8s experience jump in to help you.

Nope, that’s a Dgraph Cloud thing.

I personally don’t know how the Dgraph Cloud was implemented. And who built it isn’t with us anymore. But I can ask.

1 Like
9

@StrangerDanger When you’re connecting to your TLS endpoint in Pydgraph, are you setting up your gRPC channel with SSL using grpc.ssl_channel_credentials? If you don’t pass in secure channel credentials to your DgraphClientStub, then by default the client tries to connect without TLS.

Here’s an example of setting up a TLS connection in pydgraph:

If your host already has the trusted CA and certificates, then the default grpc.ssl_channel_credentials() with no arguments should work for you:

creds = grpc.ssl_channel_credentials()
client_stub = pydgraph.DgraphClientStub(addr, credentials=creds)
client = pydgraph.DgraphClient(client_stub)

If you’re connecting to your own self-hosted Dgraph then you don’t need an API key.

2 Likes