Too many open files - logs in server

Users
1 
2016/10/05 04:50:09 server.go:2317: http: Accept error: accept tcp [::]:8082: accept4: too many open files; retrying in 1s
2016/10/05 04:50:10 server.go:2317: http: Accept error: accept tcp [::]:8082: accept4: too many open files; retrying in 1s
2016/10/05 04:50:11 server.go:2317: http: Accept error: accept tcp [::]:8082: accept4: too many open files; retrying in 1s
2016/10/05 04:50:12 server.go:2317: http: Accept error: accept tcp [::]:8082: accept4: too many open files; retrying in 1s
2016/10/05 04:50:13 server.go:2317: http: Accept error: accept tcp [::]:8082: accept4: too many open files; retrying in 1s
2016/10/05 04:50:14 server.go:2317: http: Accept error: accept tcp [::]:8082: accept4: too many open files; retrying in 1s

I’ve seen these logs a few times in the server (dgraph demo instance). Just wondering If we aren’t closing some connections properly. When this happens, the response time of server increases.

2

Hey @ashwin95r,

Forgot to mention, please file a bug on Github.

3

The sockets info of the 2 dgraph processes

$ cat /proc/16304/net/sockstat
sockets: used 1861
TCP: inuse 884 orphan 4 tw 2 alloc 1890 mem 160
UDP: inuse 2 mem 0
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0

$ cat /proc/16302/net/sockstat                                                                                                      
sockets: used 1861
TCP: inuse 884 orphan 4 tw 2 alloc 1890 mem 162
UDP: inuse 2 mem 0
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0

TCP inuse is large, hence we get the error. I doubt we don’t close the connections in connection pools properly.

4

Could also be the http requests not closing the tcp sockets. Try to watch it and throw more and more queries and see how this increases. Because all conn.Pool stuff gets created at the beginning and just reused.

5

True. Also since the log says port 8082, which is the port to which the queries are sent and not the internal ones. But I tried doing curl requests in a loop (around 100000) but the fd count doesn’t change. Will try to find some other way to reproduce this.

2016/11/21 05:36:16 server.go:2317: http: Accept error: accept tcp [::]:8082: accept4: too many open files; retrying in 1s
1 Like
6

Probably this has the answer to our issue here. We need to set some timeouts.

2 Likes
7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.