Ratel not connecting using example k8s deployment


(Aaron Clay) #1

using deployment of https://github.com/dgraph-io/dgraph/blob/master/contrib/config/kubernetes/dgraph-ha.yaml

Entered server URL of http://redacted-lb.us-west-2.elb.amazonaws.com:8080/ in ratel. (unrelated nit - please rename this in ratel to alpha since that’ what documentation uses, instead of “server” which is ambiguous.)

This results in an HTTP request from the browser to http://redacted-lb.us-west-2.elb.amazonaws.com:8080/health?debug=true
which 503s.

Adding a \ prior to the ? in the url makes it 200:

curl -v "http://redacted-lb.us-west-2.elb.amazonaws.com:8080/health\?debug=true
 Trying redacted-ip...
* TCP_NODELAY set
* Connected to redacted-lb.us-west-2.elb.amazonaws.com (34.218.194.34) port 8080 (#0)
> GET /health\?debug=true HTTP/1.1
> Host: redacted-lb.us-west-2.elb.amazonaws.com:8080
> User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)
> Accept: */*
> Referer: 
> 
< HTTP/1.1 200 OK
< Date: Wed, 19 Dec 2018 21:38:58 GMT
< Content-Length: 80
< Content-Type: text/plain; charset=utf-8
< 
* Connection #0 to host redacted-lb.us-west-2.elb.amazonaws.com left intact
Dgraph browser is available for running separately using the dgraph-ratel binary

but using a curl for the one that the browser makes also 503s.

➜  dgraph-playground curl -v "http://redacted-lb.us-west-2.elb.amazonaws.com:8080/health?debug=true" 
*   Trying redacted-ip...
* TCP_NODELAY set
* Connected to redacted-lb.us-west-2.elb.amazonaws.com (redacted-ip port 8080 (#0)
> GET /health?debug=true HTTP/1.1
> Host: redacted-lb.us-west-2.elb.amazonaws.com:8080
> User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)
> Accept: */*
> Referer: 
> 
< HTTP/1.1 503 Service Unavailable
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Headers: Content-Type, Content-Length, Accept-Encoding, X-CSRF-Token, X-Auth-Token, Cache-Control, X-Requested-With, X-Dgraph-CommitNow, X-Dgraph-Vars, X-Dgraph-MutationType, X-Dgraph-IgnoreIndexConflict
< Access-Control-Allow-Methods: POST, OPTIONS
< Access-Control-Allow-Origin: *
< Connection: close
< Date: Wed, 19 Dec 2018 21:40:47 GMT
< Content-Length: 0
< 
* Closing connection 0

Not quite sure what is causing this without digging into the handler code, or why it’d 503 instead of more graceful 40x with an error message.

Built at 2018-12-18T15:46:45.490Z
Commit: fcd7bed
Commit Info: fcd7bed Tue Dec 18 17:23:20 2018 +0200 (HEAD -> d3graphs, origin/d3graphs)```

(Michel Conrado) #2

Jus use /health - There’s no option for health endpoint.

Can you go to http://play.dgraph.io and point to your URL? Just to see.

I’m not sure why your Ratel is in the /d3graphs branch. So, please can you test it and let me know if it works? I’ll investigate this.

Cheers.


(Aaron Clay) #3

I don’t get to choose the url that ratel adds unless I’m missing something. Everything after the :8080/ is added by ratel itself.

The play URL works, which is more confusing. I was testing with ?/dev. Same issue with latest though.

Built at 2018-11-02T14:55:42.722Z
Commit: 196c527
Commit Info: 196c527 Fri Nov 2 16:43:33 2018 +0200 (HEAD -> master, origin/master, origin/HEAD)```

(Michel Conrado) #4

I mean the URL that AWS gives you to the K8s instance. As the Play Dgraph is outside of your context, you have to use an external URL to your machine.

Ratel is just a service. It don’t dealswith url or ports. The Ratel’s port is 8000
So you access it in localhost:8000

humm, yeah there is a dev option right now. @paulftw can you check this?


(Michel Conrado) #5

Are you sure that you’re using the “latest” build?

Ratel’s info from Latest image:

Built at 2018-11-02T14:55:42.722Z
Commit: 196c527
Commit Info: 196c527 Fri Nov 2 16:43:33 2018 +0200 
(HEAD -> master, origin/master, origin/HEAD)

Update: Okay, I see. /Dev version is broken. Don’t work here too, so go to
http://yourURL:8000/ and choose latest or local.


(Aaron Clay) #6

@MichelDiz In my last reply, I mentioned that I had tried latest (and linked the “About” section there as well). It’s broken in the same way in both “latest” and “dev”. I’m honestly not sure if ratel is broken, or if I’m doing something wrong here though. The fact that a modified curl works makes me think that it’s ratel mungling with the URL, or that dgraph’s server code is expecting something wrong.


(Michel Conrado) #7

I thought It had broken. Because Latest was working and Dev did not. But it was something in my Chrome. It’s running in anonymous and in Safari.

Well, your case is certainly URL exposure by DNS or something.

1- If you can run Ratel via yourURL:8000/some = Nice
2 - if you can point your Ratel to https://play.dgraph.io/and you are able to query and so on.
= All fine with your Ratel. So Dgraph’s URL isn’t exposed or reversed for some reason. Check your (reverse)|proxy or DNS. Or even if you did any change in the K8s YAML.


(Michel Conrado) #8

Hey, We just used that yml a day ago but in gcloud.

Can you share the Alphas and Zeros logs?


(Aaron Clay) #9

Zero logs:

Alpha logs:

Ratel logs
2018/12/19 21:06:50 Listening on port 8000...

This is the specific yaml I ran:

There are a bunch of errors in zeros/alphas. I had reported errors here, which resulted in a “fixed” yaml. Perhaps I need to update again? Kubernetes contrib config doesn’t work with minikube?

I can delete the dgraph namespace and recreate it with a new yaml if you think that would help, but this was working locally, so I think it should be ok?


(Daniel Mai) #10

Are you running Dgraph in a specific Kubernetes namespace? i.e., not the default namespace?

In the logs it says that there’s no such host for the Zeros. The DNS name it’s trying to resolve to includes the k8s namespace default. If you’re deploying to a different namespace you’ll also need to update the DNS name used in the commands.


(Aaron Clay) #11

I’m running it in a dgraph namespace. That might be the issue (I’m testing that theory now, and will update). Thanks for the catch.


(Aaron Clay) #12

That worked. Thanks @dmai, sorry for not catching that.


(Daniel Mai) #13

The k8s configs have been updated (commit bebf84) so that the namespace isn’t assumed to be the default one. It should now work out of the box with any k8s namespace.