Dgraph October Community Call

Register here

Please submit your questions on this thread

We are delighted to announce our next community call on October 28, 2020 9:30 AM Pacific Time . This month we are featuring an exciting talk by Daniel Posthuma from Martin.ai. We will also have a Dgraph product update from @mrjn and the rest of the team. As always, you will have the opportunity to get your questions answered by the Dgraph team.

Agenda :

9:30 AM - 10:00 AM: Daniel Posthuma from Martin.ai and Q&A
10:00 AM - 10:20 AM: Product update from the Dgraph Team
10:20 AM - 10:30 AM: Q&A with Dgraph team

How did I reduce an almost 1TB Redis cluster down to 10GB using Dgraph’s Ristretto by Daniel Posthuma

When you need low latency access to large datasets, it’s easy to cache yourself into massive cloud hosting bills. Off-the-shelf solutions may solve parts of your problem but don’t address the full picture. This is the situation we found ourselves in at Martin where we are building real-time bidding systems that have strict latency requirements while receiving well over 200k RPS. Solving this problem requires a robust caching and storage strategy, but before devising a strategy we first need to ask the right questions. What kind of usage patterns does the solution need to support? How does data enter into the system? What needs to be fast - reads, writes or both? What kind of scaling requirements must be supported? How important is consistency to your application? What metrics determine success in the solution? Join me as we address these questions by walking through our caching strategy at Martin.
You will learn:

  • How to isolate caching in its own service
  • How to identify the right off-the-shelf software to include in your solution
  • How Dgraph’s Ristretto library can play a part in your caching strategy
  • A data driven approach to measuring the performance of your solution

About Daniel Posthuma

Daniel Posthuma is a software engineer building high performance bidding systems for Martin that support billions of requests per day. Most of the code he writes is in Go, C, or C++ and his interests primarily revolve around operating systems, computer architecture, and infrastructure engineering. In his spare time you could find him hacking on Plan 9 from Bell Labs, picking up algorithmic trading, or enjoying the outdoors with his family.

Recording

2 Likes

Question from @mikehawkes to Daniel Posthuma:

Can I ask how you manage the infrastructure for your caching - machine failure, exception management and drop-outs etc?

Talking of zones … how do you manage EU vs US traffic for compliance?

Hi dgraph team.

We’ve been running with the new memory improvements… way less! Thanks :slight_smile:

3 Likes

Regarding the parameterized cascade, will this address some of the outstanding issues with pagination and cascades over filters?

Question from @mikehawkes to the Dgraph team:

Things like auth on GraphQL - are these enterprise only?

@matthewmcneely can you elaborate/link on the outstanding issues you mentioned?

@matthewmcneely Thanks for clarifying. As @pawan said during the call, parameterized cascade doesn’t directly address the issue, but we are aware of it and the issue should be fixed in the next patch release.

1 Like

@mikehawkes GraphQL Auth is an open source feature of Dgraph including all the new features coming out in v20.11.

1 Like

This was a pretty amazing point! Looking at this screenshot, the commend was made that these were on the same axis, but this is not the case, this is actually better than it looks. The 20.07.2 diagram is 9Gb on the bottom whereas 20.11 is 0Gb on the bottom. The go memory is consistently below 9Gb in 20.11 where it was never under 9Gb in 20.07.2


The lambda functionality coming in 20.11 will support npm. Any stable package on npm can be used in the JS hook. :slight_smile: For example, you imported React and actually rendered an html string from a query with data from the database all in one request to the database!

I loved the examples provided they were great. It did spur a question, which I think is possible.

With the addGraphQLResolver does it have access to the query request info? I am thinking about a way to possibly expand this which may already be built in.

In your example you provided a Item.TITLE predicate and used a function to convert Item.title to uppercase. What if a user wanted to put that into a switch/if-else statement and with a single predicate allow a user to support uppercase, lowercase, substring, etc. Does the lambda directive accept and pass any parameters? I will go find the codebase and try to answer my own question.