A lot of sstables, each is 67Mb - is it ok?

Users Badger
1

seems like they are never compacted. only 16Gb is loaded but query performance is way slow.
should these files be compacted? how to debug why it’s not happening? thanks

root@ip-172-16-25-70:/dgraph/p# ls -lh
total 16G
-rw-r--r-- 1 root root 159M Mar 29 16:14 000477.vlog
-rw-r--r-- 1 root root  70M Mar 29 09:19 014865.sst
-rw-r--r-- 1 root root  70M Mar 29 09:19 014888.sst
-rw-r--r-- 1 root root  70M Mar 29 09:19 014889.sst
-rw-r--r-- 1 root root  70M Mar 29 09:20 014890.sst
-rw-r--r-- 1 root root  70M Mar 29 09:21 014928.sst
-rw-r--r-- 1 root root  70M Mar 29 09:21 014931.sst
-rw-r--r-- 1 root root  70M Mar 29 09:21 014951.sst
-rw-r--r-- 1 root root  70M Mar 29 09:21 014952.sst
-rw-r--r-- 1 root root  70M Mar 29 09:21 014954.sst
-rw-r--r-- 1 root root  70M Mar 29 09:22 014977.sst
-rw-r--r-- 1 root root  70M Mar 29 09:22 014978.sst
-rw-r--r-- 1 root root  70M Mar 29 09:22 014997.sst
-rw-r--r-- 1 root root  67M Mar 29 09:22 014999.sst
-rw-r--r-- 1 root root  70M Mar 29 09:23 015023.sst
-rw-r--r-- 1 root root  70M Mar 29 09:23 015024.sst
-rw-r--r-- 1 root root  70M Mar 29 09:23 015025.sst
-rw-r--r-- 1 root root 8.2K Mar 29 09:23 015026.sst
-rw-r--r-- 1 root root  70M Mar 29 09:23 015027.sst
-rw-r--r-- 1 root root  70M Mar 29 09:23 015029.sst
-rw-r--r-- 1 root root  70M Mar 29 09:23 015053.sst
-rw-r--r-- 1 root root  70M Mar 29 09:23 015054.sst
-rw-r--r-- 1 root root  70M Mar 29 09:23 015059.sst
-rw-r--r-- 1 root root  70M Mar 29 09:23 015062.sst
-rw-r--r-- 1 root root  70M Mar 29 09:23 015063.sst
-rw-r--r-- 1 root root  70M Mar 29 09:23 015087.sst
-rw-r--r-- 1 root root  70M Mar 29 09:23 015088.sst
-rw-r--r-- 1 root root  70M Mar 29 09:23 015089.sst
-rw-r--r-- 1 root root  67M Mar 29 09:23 015091.sst
-rw-r--r-- 1 root root  70M Mar 29 09:23 015092.sst
-rw-r--r-- 1 root root  70M Mar 29 09:23 015093.sst
-rw-r--r-- 1 root root  70M Mar 29 09:23 015094.sst
-rw-r--r-- 1 root root  67M Mar 29 09:24 015124.sst
-rw-r--r-- 1 root root  67M Mar 29 09:24 015125.sst
-rw-r--r-- 1 root root  67M Mar 29 09:24 015126.sst
-rw-r--r-- 1 root root  67M Mar 29 09:24 015127.sst
-rw-r--r-- 1 root root  67M Mar 29 09:24 015150.sst
-rw-r--r-- 1 root root  67M Mar 29 09:24 015151.sst
-rw-r--r-- 1 root root  67M Mar 29 09:24 015152.sst
-rw-r--r-- 1 root root  67M Mar 29 09:24 015153.sst
-rw-r--r-- 1 root root  67M Mar 29 09:24 015175.sst
-rw-r--r-- 1 root root  67M Mar 29 09:24 015176.sst
-rw-r--r-- 1 root root  67M Mar 29 09:24 015177.sst
-rw-r--r-- 1 root root  67M Mar 29 09:24 015178.sst
-rw-r--r-- 1 root root  67M Mar 29 09:25 015201.sst
-rw-r--r-- 1 root root  67M Mar 29 09:25 015202.sst
-rw-r--r-- 1 root root  67M Mar 29 09:25 015203.sst
-rw-r--r-- 1 root root  67M Mar 29 09:25 015232.sst
-rw-r--r-- 1 root root  67M Mar 29 09:25 015233.sst
-rw-r--r-- 1 root root  67M Mar 29 09:25 015234.sst
-rw-r--r-- 1 root root  67M Mar 29 09:25 015235.sst
-rw-r--r-- 1 root root  67M Mar 29 09:25 015268.sst
-rw-r--r-- 1 root root  67M Mar 29 09:25 015269.sst
-rw-r--r-- 1 root root  67M Mar 29 09:25 015270.sst
-rw-r--r-- 1 root root  67M Mar 29 09:25 015271.sst
-rw-r--r-- 1 root root  67M Mar 29 09:26 015289.sst
-rw-r--r-- 1 root root  66M Mar 29 09:26 015290.sst
-rw-r--r-- 1 root root  66M Mar 29 09:26 015291.sst
-rw-r--r-- 1 root root  66M Mar 29 09:26 015328.sst
-rw-r--r-- 1 root root  66M Mar 29 09:27 015339.sst
-rw-r--r-- 1 root root  66M Mar 29 09:27 015340.sst
-rw-r--r-- 1 root root  66M Mar 29 09:27 015343.sst
-rw-r--r-- 1 root root  67M Mar 29 09:28 015359.sst
-rw-r--r-- 1 root root  66M Mar 29 09:28 015383.sst
-rw-r--r-- 1 root root  66M Mar 29 09:30 015426.sst
-rw-r--r-- 1 root root  67M Mar 29 09:31 015473.sst
-rw-r--r-- 1 root root  67M Mar 29 09:31 015474.sst
-rw-r--r-- 1 root root  67M Mar 29 09:31 015475.sst
-rw-r--r-- 1 root root  67M Mar 29 09:31 015476.sst
-rw-r--r-- 1 root root  67M Mar 29 09:31 015491.sst
-rw-r--r-- 1 root root  67M Mar 29 09:31 015495.sst
-rw-r--r-- 1 root root  67M Mar 29 09:31 015496.sst
-rw-r--r-- 1 root root  67M Mar 29 09:31 015497.sst
-rw-r--r-- 1 root root  67M Mar 29 09:31 015498.sst
-rw-r--r-- 1 root root  67M Mar 29 09:31 015499.sst
-rw-r--r-- 1 root root  67M Mar 29 09:31 015500.sst
-rw-r--r-- 1 root root  67M Mar 29 09:32 015533.sst
-rw-r--r-- 1 root root  67M Mar 29 09:32 015534.sst
-rw-r--r-- 1 root root  67M Mar 29 09:32 015535.sst
-rw-r--r-- 1 root root  67M Mar 29 09:32 015536.sst
-rw-r--r-- 1 root root  67M Mar 29 09:32 015537.sst
-rw-r--r-- 1 root root  67M Mar 29 09:32 015555.sst
-rw-r--r-- 1 root root  67M Mar 29 09:32 015557.sst
-rw-r--r-- 1 root root  67M Mar 29 09:32 015560.sst
-rw-r--r-- 1 root root  67M Mar 29 09:32 015562.sst
-rw-r--r-- 1 root root  67M Mar 29 09:32 015563.sst
-rw-r--r-- 1 root root  66M Mar 29 09:32 015564.sst
-rw-r--r-- 1 root root  67M Mar 29 09:33 015583.sst
-rw-r--r-- 1 root root  67M Mar 29 09:33 015586.sst
-rw-r--r-- 1 root root  66M Mar 29 09:33 015588.sst
-rw-r--r-- 1 root root  67M Mar 29 09:33 015611.sst
-rw-r--r-- 1 root root  66M Mar 29 09:33 015612.sst
-rw-r--r-- 1 root root  67M Mar 29 09:33 015613.sst
-rw-r--r-- 1 root root  67M Mar 29 09:33 015614.sst
-rw-r--r-- 1 root root  66M Mar 29 09:33 015615.sst
-rw-r--r-- 1 root root  67M Mar 29 09:33 015616.sst
-rw-r--r-- 1 root root  66M Mar 29 09:33 015636.sst
-rw-r--r-- 1 root root  67M Mar 29 09:33 015637.sst
-rw-r--r-- 1 root root  66M Mar 29 09:33 015639.sst
-rw-r--r-- 1 root root  66M Mar 29 09:34 015664.sst
-rw-r--r-- 1 root root  66M Mar 29 09:34 015665.sst
-rw-r--r-- 1 root root  66M Mar 29 09:34 015668.sst
-rw-r--r-- 1 root root  67M Mar 29 09:34 015689.sst
-rw-r--r-- 1 root root  66M Mar 29 09:35 015720.sst
-rw-r--r-- 1 root root  66M Mar 29 09:36 015762.sst
-rw-r--r-- 1 root root  67M Mar 29 09:36 015781.sst
-rw-r--r-- 1 root root  67M Mar 29 09:36 015783.sst
-rw-r--r-- 1 root root  67M Mar 29 09:37 015807.sst
-rw-r--r-- 1 root root  66M Mar 29 09:37 015809.sst
-rw-r--r-- 1 root root  67M Mar 29 09:38 015832.sst
-rw-r--r-- 1 root root  67M Mar 29 09:38 015833.sst
-rw-r--r-- 1 root root  67M Mar 29 09:38 015835.sst
-rw-r--r-- 1 root root  67M Mar 29 09:38 015837.sst
-rw-r--r-- 1 root root 3.5K Mar 29 09:38 015838.sst
-rw-r--r-- 1 root root  67M Mar 29 09:38 015858.sst
-rw-r--r-- 1 root root  67M Mar 29 09:38 015859.sst
-rw-r--r-- 1 root root  67M Mar 29 09:38 015860.sst
-rw-r--r-- 1 root root  67M Mar 29 09:38 015861.sst
-rw-r--r-- 1 root root 4.3K Mar 29 09:38 015863.sst
-rw-r--r-- 1 root root  67M Mar 29 09:39 015874.sst
-rw-r--r-- 1 root root 1.7K Mar 29 09:39 015878.sst
-rw-r--r-- 1 root root  67M Mar 29 09:39 015906.sst
-rw-r--r-- 1 root root  794 Mar 29 09:39 015907.sst
-rw-r--r-- 1 root root  67M Mar 29 09:40 015933.sst
-rw-r--r-- 1 root root  70M Mar 29 09:40 015952.sst
-rw-r--r-- 1 root root  67M Mar 29 09:41 015962.sst
-rw-r--r-- 1 root root  67M Mar 29 09:41 015963.sst
-rw-r--r-- 1 root root  732 Mar 29 09:41 015964.sst
-rw-r--r-- 1 root root  67M Mar 29 09:41 015980.sst
-rw-r--r-- 1 root root  67M Mar 29 09:41 015982.sst
-rw-r--r-- 1 root root  66M Mar 29 09:41 015983.sst
-rw-r--r-- 1 root root  66M Mar 29 09:41 016013.sst
-rw-r--r-- 1 root root  66M Mar 29 09:41 016014.sst
-rw-r--r-- 1 root root  70M Mar 29 09:41 016015.sst
-rw-r--r-- 1 root root  66M Mar 29 09:41 016016.sst
-rw-r--r-- 1 root root  70M Mar 29 09:41 016019.sst
-rw-r--r-- 1 root root  70M Mar 29 09:41 016021.sst
-rw-r--r-- 1 root root  70M Mar 29 09:41 016023.sst
-rw-r--r-- 1 root root  70M Mar 29 09:41 016024.sst
-rw-r--r-- 1 root root  67M Mar 29 09:42 016048.sst
-rw-r--r-- 1 root root  70M Mar 29 09:42 016051.sst
-rw-r--r-- 1 root root  70M Mar 29 09:42 016054.sst
-rw-r--r-- 1 root root  66M Mar 29 09:42 016089.sst
-rw-r--r-- 1 root root  67M Mar 29 09:42 016092.sst
-rw-r--r-- 1 root root  66M Mar 29 09:42 016093.sst
-rw-r--r-- 1 root root  66M Mar 29 09:42 016113.sst
-rw-r--r-- 1 root root  444 Mar 29 09:42 016117.sst
-rw-r--r-- 1 root root  66M Mar 29 09:42 016118.sst
-rw-r--r-- 1 root root  66M Mar 29 09:42 016119.sst
-rw-r--r-- 1 root root  66M Mar 29 09:43 016142.sst
-rw-r--r-- 1 root root  66M Mar 29 09:43 016145.sst
-rw-r--r-- 1 root root  66M Mar 29 09:43 016172.sst
-rw-r--r-- 1 root root  70M Mar 29 09:43 016173.sst
-rw-r--r-- 1 root root  70M Mar 29 09:43 016174.sst
-rw-r--r-- 1 root root  70M Mar 29 09:43 016175.sst
-rw-r--r-- 1 root root  67M Mar 29 09:43 016176.sst
-rw-r--r-- 1 root root  66M Mar 29 09:43 016178.sst
-rw-r--r-- 1 root root  66M Mar 29 09:43 016179.sst
-rw-r--r-- 1 root root  66M Mar 29 09:44 016205.sst
-rw-r--r-- 1 root root  66M Mar 29 09:45 016251.sst
-rw-r--r-- 1 root root  66M Mar 29 09:45 016257.sst
-rw-r--r-- 1 root root  66M Mar 29 09:45 016282.sst
-rw-r--r-- 1 root root  67M Mar 29 09:47 016345.sst
-rw-r--r-- 1 root root  67M Mar 29 09:47 016346.sst
-rw-r--r-- 1 root root  67M Mar 29 09:47 016349.sst
-rw-r--r-- 1 root root  67M Mar 29 09:47 016371.sst
-rw-r--r-- 1 root root  67M Mar 29 09:47 016373.sst
-rw-r--r-- 1 root root  67M Mar 29 09:47 016379.sst
-rw-r--r-- 1 root root  67M Mar 29 09:47 016380.sst
-rw-r--r-- 1 root root  67M Mar 29 09:48 016418.sst
-rw-r--r-- 1 root root  67M Mar 29 09:48 016419.sst
-rw-r--r-- 1 root root  67M Mar 29 09:48 016423.sst
-rw-r--r-- 1 root root  67M Mar 29 09:48 016424.sst
-rw-r--r-- 1 root root  67M Mar 29 09:48 016425.sst
-rw-r--r-- 1 root root  67M Mar 29 09:48 016426.sst
-rw-r--r-- 1 root root  67M Mar 29 09:48 016427.sst
-rw-r--r-- 1 root root  67M Mar 29 09:48 016428.sst
-rw-r--r-- 1 root root  67M Mar 29 09:48 016430.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016433.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016434.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016435.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016436.sst
-rw-r--r-- 1 root root  67M Mar 29 09:48 016437.sst
-rw-r--r-- 1 root root 3.0K Mar 29 09:48 016439.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016440.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016441.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016442.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016443.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016444.sst
-rw-r--r-- 1 root root  28K Mar 29 09:48 016445.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016446.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016447.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016448.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016449.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016450.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016451.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016452.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016454.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016455.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016457.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016458.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016459.sst
-rw-r--r-- 1 root root 2.8K Mar 29 09:48 016460.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016461.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016462.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016463.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016464.sst
-rw-r--r-- 1 root root 7.0K Mar 29 09:48 016466.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016467.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016468.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016470.sst
-rw-r--r-- 1 root root  70M Mar 29 09:48 016471.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016472.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016473.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016474.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016475.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016476.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016477.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016479.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016480.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016481.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016482.sst
-rw-r--r-- 1 root root 1004 Mar 29 09:48 016483.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016484.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016487.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016488.sst
-rw-r--r-- 1 root root  304 Mar 29 09:48 016489.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016490.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016492.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016493.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016494.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016495.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016496.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016497.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016498.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016499.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016501.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016502.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016503.sst
-rw-r--r-- 1 root root  66M Mar 29 09:48 016504.sst
-rw-r--r-- 1 root root  66M Mar 29 09:49 016505.sst
-rw-r--r-- 1 root root  66M Mar 29 09:49 016506.sst
-rw-r--r-- 1 root root  66M Mar 29 09:49 016507.sst
-rw-r--r-- 1 root root  67M Mar 29 09:49 016508.sst
-rw-r--r-- 1 root root  67M Mar 29 09:49 016509.sst
-rw-r--r-- 1 root root  67M Mar 29 09:49 016510.sst
-rw-r--r-- 1 root root  67M Mar 29 09:49 016511.sst
-rw-r--r-- 1 root root  67M Mar 29 09:49 016512.sst
-rw-r--r-- 1 root root 1.6M Mar 29 09:49 016513.sst
-rw-r--r-- 1 root root  13K Mar 29 09:49 016514.sst
-rw-r--r-- 1 root root  67M Mar 29 09:49 016515.sst
-rw-r--r-- 1 root root  11K Mar 29 09:49 016516.sst
-rw-r--r-- 1 root root  66M Mar 29 09:49 016517.sst
-rw-r--r-- 1 root root 4.7K Mar 29 09:49 016518.sst
-rw-r--r-- 1 root root  66M Mar 29 09:49 016519.sst
-rw-r--r-- 1 root root 2.4K Mar 29 09:49 016520.sst
-rw-r--r-- 1 root root 2.2K Mar 29 09:49 016521.sst
-rw-r--r-- 1 root root  66M Mar 29 09:49 016522.sst
-rw-r--r-- 1 root root 2.3K Mar 29 09:49 016523.sst
-rw-r--r-- 1 root root 1.8K Mar 29 09:49 016524.sst
-rw-r--r-- 1 root root 5.4M Mar 29 16:01 016526.sst
-rw-r--r-- 1 root root    2 Mar 29 16:02 LOCK
-rw-r--r-- 1 root root 110K Mar 29 16:02 MANIFEST
2

on previous load we already had 800 sstables - so node took 6 minutes to start

3

Can you share your badger info output? And yes, it is ok to have a lot of SSTables. Badger does not look through each and every one of them when doing a key lookup.

4

sorry, but i don’t think that is’s ok. now it’s 1466, and it is increasing more and more. insertion dramatically slows down, queries slow down as well
i don’t know any other lsm-based systems with such a lot of sstables being not merged. and if you insert data in a random order (which is true for all realtime data) you have to look into all of them.
for now, we are loading ~1 billion of edges for more than 4 days and insertion speed now is way low, so i am not sure that we will complete this insertion

image
image700×640 12.4 KB

will share badger info a bit later

5

so, yep, now alpha is crashing all the time with OOM error on 64Gb machine. so, no way to continue insertion.
whereas there is only 44Gb data on one node and 115G on other (replicas is set to 3, so full copy on each node). i suppose, that just because of a lot of not merged sstables it eats so much space and memory, and running so slow.

[ec2-user@ip-172-18-xxx-xxx ~]$ for ip in $ips; do echo $ip; ssh $ip 'sudo du /vol/dgraph -sh'; done
172.18.xxx.xxx
115G    /vol/dgraph
172.18.xxx.xxx
107G    /vol/dgraph
172.18.xxx.xxx
44G     /vol/dgraph
6

badger info on one node (which is the biggest one)

full listing is here: badger-info.txt - Google Drive

bash-4.2# /home/ec2-user/go/bin/badger info --dir .
Listening for /debug HTTP requests at port: 8080

[     2019-04-06T10:39:14Z] MANIFEST     694 kB MA
[         12 hours earlier] 121374.sst    43 MB L0
[         12 hours earlier] 121089.sst    68 MB L1
[         12 hours earlier] 121091.sst    70 MB L1
[         12 hours earlier] 121093.sst    69 MB L1
[         12 hours earlier] 121094.sst    29 MB L1
[           2 days earlier] 091226.sst    379 B L2
[         19 hours earlier] 114155.sst   1.3 MB L2
[         17 hours earlier] 116204.sst   2.7 MB L2
[         15 hours earlier] 118977.sst    68 MB L2
[         13 hours earlier] 120159.sst    12 kB L2
[         12 hours earlier] 121095.sst    68 MB L2
[         12 hours earlier] 121097.sst    68 MB L2
[         12 hours earlier] 121098.sst    68 MB L2
[         12 hours earlier] 121100.sst    68 MB L2
[         12 hours earlier] 121101.sst    68 MB L2
[         12 hours earlier] 121102.sst    68 MB L2
[         12 hours earlier] 121103.sst    68 MB L2
[         12 hours earlier] 121104.sst    68 MB L2
[         12 hours earlier] 121105.sst    68 MB L2
[         12 hours earlier] 121106.sst    68 MB L2
[         12 hours earlier] 121107.sst    68 MB L2
[         12 hours earlier] 121108.sst    68 MB L2
[         12 hours earlier] 121109.sst    68 MB L2
[         12 hours earlier] 121110.sst    68 MB L2
[         12 hours earlier] 121111.sst    68 MB L2
[         12 hours earlier] 121112.sst    68 MB L2
[         12 hours earlier] 121113.sst    68 MB L2
[         12 hours earlier] 121114.sst    68 MB L2
[         12 hours earlier] 121115.sst    68 MB L2
[         12 hours earlier] 121116.sst    68 MB L2
[         12 hours earlier] 121117.sst    68 MB L2
[         12 hours earlier] 121119.sst    68 MB L2
[         12 hours earlier] 121120.sst    68 MB L2
[         12 hours earlier] 121122.sst    68 MB L2
[         12 hours earlier] 121123.sst    68 MB L2
[         12 hours earlier] 121125.sst    68 MB L2
[         12 hours earlier] 121127.sst    68 MB L2
[         12 hours earlier] 121129.sst    68 MB L2
[         12 hours earlier] 121130.sst    68 MB L2
[         12 hours earlier] 121132.sst    68 MB L2
[         12 hours earlier] 121133.sst    68 MB L2
[         12 hours earlier] 121135.sst    68 MB L2
[         12 hours earlier] 121137.sst    68 MB L2
[         12 hours earlier] 121138.sst    68 MB L2
[         12 hours earlier] 121140.sst    68 MB L2
[         12 hours earlier] 121141.sst    68 MB L2
[         12 hours earlier] 121143.sst    68 MB L2
[         12 hours earlier] 121146.sst   9.8 MB L2
[           2 days earlier] 091236.sst    70 MB L3
[           2 days earlier] 091238.sst    70 MB L3
[           2 days earlier] 091239.sst    70 MB L3
[           2 days earlier] 091240.sst    70 MB L3
[           2 days earlier] 091241.sst    70 MB L3
[           2 days earlier] 091473.sst    70 MB L3
[           2 days earlier] 091475.sst    70 MB L3
[           2 days earlier] 091477.sst    70 MB L3
[           2 days earlier] 091479.sst    70 MB L3
[           2 days earlier] 091481.sst    70 MB L3
[           2 days earlier] 091482.sst    70 MB L3
[           2 days earlier] 091484.sst    70 MB L3
[           2 days earlier] 092631.sst    70 MB L3
[           2 days earlier] 092962.sst    70 MB L3
[           2 days earlier] 093476.sst    70 MB L3
[           2 days earlier] 094137.sst    70 MB L3
[           2 days earlier] 094570.sst    70 MB L3
[           2 days earlier] 095165.sst    70 MB L3
[           2 days earlier] 095746.sst    70 MB L3
[           2 days earlier] 096285.sst    70 MB L3
[           2 days earlier] 096837.sst    70 MB L3
[           2 days earlier] 097701.sst    70 MB L3
[           2 days earlier] 098531.sst    70 MB L3
[           2 days earlier] 099350.sst    70 MB L3
[            1 day earlier] 100197.sst    70 MB L3
[            1 day earlier] 100805.sst    70 MB L3
[            1 day earlier] 101693.sst    70 MB L3
[            1 day earlier] 102528.sst    70 MB L3
[            1 day earlier] 103605.sst    70 MB L3
[            1 day earlier] 105281.sst    70 MB L3
[            1 day earlier] 105569.sst    70 MB L3
[            1 day earlier] 106595.sst    70 MB L3
[            1 day earlier] 106934.sst    70 MB L3
[            1 day earlier] 107886.sst    70 MB L3
[            1 day earlier] 109047.sst    70 MB L3
[            1 day earlier] 110167.sst    70 MB L3
[            1 day earlier] 110575.sst    70 MB L3
[         22 hours earlier] 111483.sst    70 MB L3
[         17 hours earlier] 115548.sst    70 MB L3
[         17 hours earlier] 115779.sst    70 MB L3
[         17 hours earlier] 116215.sst    70 MB L3
[         17 hours earlier] 116217.sst    70 MB L3
[         17 hours earlier] 116219.sst    22 MB L3
[         17 hours earlier] 116333.sst    29 MB L3
[         17 hours earlier] 116347.sst    68 MB L3
[         17 hours earlier] 116350.sst    68 MB L3
[         17 hours earlier] 116351.sst    68 MB L3
[         17 hours earlier] 116354.sst    68 MB L3
[         17 hours earlier] 116601.sst    70 MB L3
[         16 hours earlier] 117589.sst    70 MB L3
[         16 hours earlier] 118028.sst    68 MB L3
[         16 hours earlier] 118029.sst    68 MB L3
[         16 hours earlier] 118031.sst    68 MB L3
[         16 hours earlier] 118032.sst    68 MB L3
[         16 hours earlier] 118034.sst    68 MB L3
[         16 hours earlier] 118035.sst    17 MB L3
[         15 hours earlier] 118771.sst    12 MB L3
[         14 hours earlier] 119143.sst    68 MB L3
[         14 hours earlier] 119279.sst    70 MB L3
[         13 hours earlier] 120353.sst    68 MB L3
[         13 hours earlier] 120356.sst    70 MB L3
[         13 hours earlier] 120359.sst    70 MB L3
[         13 hours earlier] 120362.sst    70 MB L3
[         13 hours earlier] 120489.sst    68 MB L3
[         13 hours earlier] 120490.sst    68 MB L3
[         13 hours earlier] 120491.sst    53 MB L3
[         13 hours earlier] 120529.sst    70 MB L3
[         13 hours earlier] 120531.sst    70 MB L3
[         13 hours earlier] 120533.sst    70 MB L3
[         13 hours earlier] 120535.sst    70 MB L3
[         13 hours earlier] 120537.sst    70 MB L3
[         13 hours earlier] 120539.sst    70 MB L3
[         13 hours earlier] 120542.sst    70 MB L3
[         13 hours earlier] 120544.sst    70 MB L3
[         13 hours earlier] 120545.sst    70 MB L3
[         13 hours earlier] 120547.sst    70 MB L3
[         13 hours earlier] 120549.sst    70 MB L3
[         13 hours earlier] 120552.sst    70 MB L3
[         13 hours earlier] 120554.sst    70 MB L3
[         13 hours earlier] 120556.sst    70 MB L3
[         12 hours earlier] 120875.sst    70 MB L3
[         12 hours earlier] 120877.sst    70 MB L3
[         12 hours earlier] 120878.sst    70 MB L3
[         12 hours earlier] 120880.sst    70 MB L3
[         12 hours earlier] 120881.sst    70 MB L3
[         12 hours earlier] 120883.sst    70 MB L3
[         12 hours earlier] 120885.sst    70 MB L3
[         12 hours earlier] 120887.sst    70 MB L3
[         12 hours earlier] 120889.sst    70 MB L3
[         12 hours earlier] 120891.sst    70 MB L3
[         12 hours earlier] 120893.sst    70 MB L3
[         12 hours earlier] 120895.sst    70 MB L3
[         12 hours earlier] 120897.sst    70 MB L3
[         12 hours earlier] 120899.sst    70 MB L3
[         12 hours earlier] 120901.sst    73 MB L3
[         12 hours earlier] 120903.sst    73 MB L3
[         12 hours earlier] 120905.sst    73 MB L3
[         12 hours earlier] 120907.sst    73 MB L3
[         12 hours earlier] 120909.sst    73 MB L3
[         12 hours earlier] 120911.sst    73 MB L3
[         12 hours earlier] 120913.sst    73 MB L3
[         12 hours earlier] 120915.sst    73 MB L3
[         12 hours earlier] 120917.sst    73 MB L3
[         12 hours earlier] 120920.sst    70 MB L3
[         12 hours earlier] 120923.sst    70 MB L3
[         12 hours earlier] 120925.sst    73 MB L3
[         12 hours earlier] 120927.sst    73 MB L3
[         12 hours earlier] 120929.sst    73 MB L3
[         12 hours earlier] 120931.sst    73 MB L3
[         12 hours earlier] 120933.sst    73 MB L3
[         12 hours earlier] 120937.sst    73 MB L3
[         12 hours earlier] 120939.sst    73 MB L3
[         12 hours earlier] 120943.sst    73 MB L3
[         12 hours earlier] 120945.sst    73 MB L3
[         12 hours earlier] 120947.sst    73 MB L3
[         12 hours earlier] 120950.sst    70 MB L3
[         12 hours earlier] 120952.sst    68 MB L3
[         12 hours earlier] 120954.sst    68 MB L3
[         12 hours earlier] 120956.sst    68 MB L3
[         12 hours earlier] 120958.sst    68 MB L3
[         12 hours earlier] 120960.sst    68 MB L3
[         12 hours earlier] 120962.sst    68 MB L3
[         12 hours earlier] 120964.sst    68 MB L3
[         12 hours earlier] 120966.sst    68 MB L3
[         12 hours earlier] 120968.sst    68 MB L3
[         12 hours earlier] 120970.sst    68 MB L3
[         12 hours earlier] 120972.sst    68 MB L3
[         12 hours earlier] 120974.sst    68 MB L3
[         12 hours earlier] 120976.sst    68 MB L3
[         12 hours earlier] 120978.sst    68 MB L3
[         12 hours earlier] 120980.sst    68 MB L3
[         12 hours earlier] 120982.sst    68 MB L3
[         12 hours earlier] 120984.sst    68 MB L3
[         12 hours earlier] 120986.sst    68 MB L3
[         12 hours earlier] 120988.sst    68 MB L3
[         12 hours earlier] 120990.sst    68 MB L3
[         12 hours earlier] 120992.sst    68 MB L3
[         12 hours earlier] 120994.sst    68 MB L3
[         12 hours earlier] 120996.sst    68 MB L3
[         12 hours earlier] 120998.sst    68 MB L3
[         12 hours earlier] 121000.sst    68 MB L3
[         12 hours earlier] 121002.sst    68 MB L3
[         12 hours earlier] 121004.sst    68 MB L3
[         12 hours earlier] 121006.sst    68 MB L3
[         12 hours earlier] 121008.sst    68 MB L3
[         12 hours earlier] 121010.sst    68 MB L3
[         12 hours earlier] 121012.sst    68 MB L3
[         12 hours earlier] 121014.sst    68 MB L3
[         12 hours earlier] 121016.sst    68 MB L3
[         12 hours earlier] 121018.sst    68 MB L3
[         12 hours earlier] 121020.sst    68 MB L3
[         12 hours earlier] 121022.sst    68 MB L3
[         12 hours earlier] 121024.sst    68 MB L3
[         12 hours earlier] 121026.sst    68 MB L3
[         12 hours earlier] 121028.sst    68 MB L3
[         12 hours earlier] 121030.sst    67 MB L3
[         12 hours earlier] 121118.sst    70 MB L3
[         12 hours earlier] 121121.sst    70 MB L3
[         12 hours earlier] 121124.sst    70 MB L3
[         12 hours earlier] 121126.sst    70 MB L3
[         12 hours earlier] 121128.sst    70 MB L3
[         12 hours earlier] 121131.sst    70 MB L3
[         12 hours earlier] 121134.sst    70 MB L3
[         12 hours earlier] 121136.sst    70 MB L3
[         12 hours earlier] 121139.sst    70 MB L3
[         12 hours earlier] 121142.sst    70 MB L3
[         12 hours earlier] 121145.sst    70 MB L3
[         12 hours earlier] 121147.sst    68 MB L3
[         12 hours earlier] 121148.sst    70 MB L3
[         12 hours earlier] 121149.sst    68 MB L3
[         12 hours earlier] 121150.sst    68 MB L3
[         12 hours earlier] 121151.sst    70 MB L3
[         12 hours earlier] 121152.sst    68 MB L3
[         12 hours earlier] 121153.sst    70 MB L3
[         12 hours earlier] 121154.sst    68 MB L3
[         12 hours earlier] 121155.sst    68 MB L3
[         12 hours earlier] 121156.sst    70 MB L3
[         12 hours earlier] 121157.sst    68 MB L3
[         12 hours earlier] 121158.sst    70 MB L3
[         12 hours earlier] 121159.sst    68 MB L3
[         12 hours earlier] 121160.sst    70 MB L3
[         12 hours earlier] 121161.sst    70 MB L3
[         12 hours earlier] 121162.sst    70 MB L3
[         12 hours earlier] 121163.sst    70 MB L3
[         12 hours earlier] 121164.sst    70 MB L3
[         12 hours earlier] 121165.sst    70 MB L3
[         12 hours earlier] 121166.sst    70 MB L3
[         12 hours earlier] 121167.sst    70 MB L3
[         12 hours earlier] 121168.sst    70 MB L3
[         12 hours earlier] 121169.sst    70 MB L3
[         12 hours earlier] 121170.sst    70 MB L3
[         12 hours earlier] 121171.sst    70 MB L3
[         12 hours earlier] 121172.sst    70 MB L3
[         12 hours earlier] 121173.sst    70 MB L3
[         12 hours earlier] 121174.sst    70 MB L3
[         12 hours earlier] 121175.sst    70 MB L3
[         12 hours earlier] 121176.sst    70 MB L3
[         12 hours earlier] 121177.sst    70 MB L3
[         12 hours earlier] 121178.sst    70 MB L3
[         12 hours earlier] 121179.sst    70 MB L3
[         12 hours earlier] 121180.sst    70 MB L3
[         12 hours earlier] 121181.sst    70 MB L3
[         12 hours earlier] 121182.sst    70 MB L3
[         12 hours earlier] 121183.sst    70 MB L3
[         12 hours earlier] 121184.sst    70 MB L3
[         12 hours earlier] 121185.sst    70 MB L3
[         12 hours earlier] 121186.sst    70 MB L3
[         12 hours earlier] 121187.sst    70 MB L3
[         12 hours earlier] 121188.sst    70 MB L3
[         12 hours earlier] 121189.sst    70 MB L3
[         12 hours earlier] 121190.sst    70 MB L3
[         12 hours earlier] 121191.sst    70 MB L3
[         12 hours earlier] 121192.sst    70 MB L3
[         12 hours earlier] 121193.sst    70 MB L3
[         12 hours earlier] 121194.sst    70 MB L3
[         12 hours earlier] 121195.sst    70 MB L3
[         12 hours earlier] 121196.sst    68 MB L3
[         12 hours earlier] 121197.sst    70 MB L3
[         12 hours earlier] 121198.sst    70 MB L3
[         12 hours earlier] 121199.sst    38 MB L3
[         12 hours earlier] 121200.sst    70 MB L3
[         12 hours earlier] 121201.sst    70 MB L3
[         12 hours earlier] 121202.sst    70 MB L3
[         12 hours earlier] 121203.sst    70 MB L3
[         12 hours earlier] 121204.sst    70 MB L3
[         12 hours earlier] 121205.sst    70 MB L3
[         12 hours earlier] 121206.sst    70 MB L3
[         12 hours earlier] 121207.sst    70 MB L3
[         12 hours earlier] 121208.sst    70 MB L3
[         12 hours earlier] 121209.sst    70 MB L3
[         12 hours earlier] 121210.sst    70 MB L3
[         12 hours earlier] 121211.sst    70 MB L3
[         12 hours earlier] 121212.sst    70 MB L3
[         12 hours earlier] 121213.sst    70 MB L3
[         12 hours earlier] 121214.sst    68 MB L3
[         12 hours earlier] 121215.sst    68 MB L3
[         12 hours earlier] 121216.sst    70 MB L3
[         12 hours earlier] 121217.sst    68 MB L3
[         12 hours earlier] 121218.sst    68 MB L3
[         12 hours earlier] 121219.sst    70 MB L3
[         12 hours earlier] 121220.sst    71 MB L3
[         12 hours earlier] 121221.sst    73 MB L3
[         12 hours earlier] 121222.sst    68 MB L3
[         12 hours earlier] 121223.sst    70 MB L3
[         12 hours earlier] 121224.sst    73 MB L3
[         12 hours earlier] 121225.sst    70 MB L3
[         12 hours earlier] 121226.sst    73 MB L3
[         12 hours earlier] 121227.sst    70 MB L3
[         12 hours earlier] 121228.sst    70 MB L3
[         12 hours earlier] 121229.sst    73 MB L3
[         12 hours earlier] 121230.sst    70 MB L3
[         12 hours earlier] 121231.sst    73 MB L3
[         12 hours earlier] 121232.sst    64 MB L3
[         12 hours earlier] 121233.sst    73 MB L3
[         12 hours earlier] 121235.sst    73 MB L3
[         12 hours earlier] 121237.sst    73 MB L3
[         12 hours earlier] 121238.sst    73 MB L3
[         12 hours earlier] 121239.sst    73 MB L3
[         12 hours earlier] 121240.sst    70 MB L3
[         12 hours earlier] 121241.sst    70 MB L3
[         12 hours earlier] 121242.sst    73 MB L3
[         12 hours earlier] 121243.sst    73 MB L3
[         12 hours earlier] 121244.sst    73 MB L3
[         12 hours earlier] 121245.sst    73 MB L3
[         12 hours earlier] 121246.sst    73 MB L3
[         12 hours earlier] 121247.sst    73 MB L3
[         12 hours earlier] 121248.sst    73 MB L3
[         12 hours earlier] 121250.sst    73 MB L3
[         12 hours earlier] 121251.sst    73 MB L3
[         12 hours earlier] 121252.sst    73 MB L3
[         12 hours earlier] 121253.sst    70 MB L3
[         12 hours earlier] 121254.sst    70 MB L3
[         12 hours earlier] 121255.sst    70 MB L3
[         12 hours earlier] 121256.sst    70 MB L3
[         12 hours earlier] 121257.sst    70 MB L3
[         12 hours earlier] 121258.sst    70 MB L3
[         12 hours earlier] 121259.sst    70 MB L3
[         12 hours earlier] 121260.sst    70 MB L3
[         12 hours earlier] 121261.sst    73 MB L3
[         12 hours earlier] 121262.sst    73 MB L3
[         12 hours earlier] 121263.sst    73 MB L3
[         12 hours earlier] 121264.sst    73 MB L3
[         12 hours earlier] 121265.sst    73 MB L3
[         12 hours earlier] 121266.sst    73 MB L3
[         12 hours earlier] 121267.sst    73 MB L3
[         12 hours earlier] 121268.sst    73 MB L3
[         12 hours earlier] 121269.sst    73 MB L3
[         12 hours earlier] 121270.sst    73 MB L3
[         12 hours earlier] 121271.sst    70 MB L3
(a lot of lines here)
[         17 hours earlier] 003579.vlog   87 MB VL
[         17 hours earlier] 003586.vlog   71 MB VL
[         17 hours earlier] 003587.vlog   72 MB VL
[         17 hours earlier] 003588.vlog   70 MB VL
[         17 hours earlier] 003589.vlog   67 MB VL
[         17 hours earlier] 003591.vlog   69 MB VL
[         17 hours earlier] 003593.vlog   64 MB VL
[         17 hours earlier] 003594.vlog   67 MB VL
[         17 hours earlier] 003595.vlog   61 MB VL
[         17 hours earlier] 003597.vlog   62 MB VL
[         17 hours earlier] 003598.vlog   57 MB VL
[         17 hours earlier] 003599.vlog   57 MB VL
[         17 hours earlier] 003600.vlog   81 MB VL
[         17 hours earlier] 003601.vlog   81 MB VL
[         17 hours earlier] 003602.vlog   81 MB VL
[         17 hours earlier] 003603.vlog   81 MB VL
[         17 hours earlier] 003604.vlog   77 MB VL
[         17 hours earlier] 003605.vlog   93 MB VL
[         17 hours earlier] 003606.vlog   85 MB VL
[         17 hours earlier] 003607.vlog   88 MB VL
[         17 hours earlier] 003608.vlog   81 MB VL
[         17 hours earlier] 003609.vlog   88 MB VL
[         17 hours earlier] 003610.vlog   78 MB VL
[         17 hours earlier] 003611.vlog   85 MB VL
[         17 hours earlier] 003612.vlog   83 MB VL
[         17 hours earlier] 003624.vlog   90 MB VL
[         17 hours earlier] 003625.vlog  207 MB VL
[         17 hours earlier] 003626.vlog   94 MB VL
[         17 hours earlier] 003627.vlog   76 MB VL
[         17 hours earlier] 003628.vlog   70 MB VL
[         17 hours earlier] 003629.vlog   70 MB VL
[         17 hours earlier] 003630.vlog   68 MB VL
[         17 hours earlier] 003631.vlog   68 MB VL
[         17 hours earlier] 003632.vlog   68 MB VL
[         17 hours earlier] 003633.vlog   97 MB VL
[         17 hours earlier] 003635.vlog   70 MB VL
[         16 hours earlier] 003637.vlog   69 MB VL
[         16 hours earlier] 003638.vlog   67 MB VL
[         16 hours earlier] 003639.vlog   69 MB VL
[         16 hours earlier] 003640.vlog   69 MB VL
[         16 hours earlier] 003641.vlog   68 MB VL
[         16 hours earlier] 003642.vlog   67 MB VL
[         16 hours earlier] 003643.vlog   67 MB VL
[         16 hours earlier] 003644.vlog   69 MB VL
[         16 hours earlier] 003645.vlog   68 MB VL
[         16 hours earlier] 003646.vlog   67 MB VL
[         16 hours earlier] 003647.vlog   68 MB VL
[         16 hours earlier] 003648.vlog   68 MB VL
[         16 hours earlier] 003652.vlog   70 MB VL
[         16 hours earlier] 003653.vlog   68 MB VL
[         16 hours earlier] 003654.vlog   68 MB VL
[         16 hours earlier] 003655.vlog   67 MB VL
[         16 hours earlier] 003656.vlog   68 MB VL
[         16 hours earlier] 003657.vlog   70 MB VL
[         16 hours earlier] 003658.vlog   68 MB VL
[         16 hours earlier] 003659.vlog   68 MB VL
[         16 hours earlier] 003660.vlog   69 MB VL
[         16 hours earlier] 003661.vlog   71 MB VL
[         16 hours earlier] 003662.vlog   71 MB VL
[         16 hours earlier] 003663.vlog   70 MB VL
[         16 hours earlier] 003665.vlog   68 MB VL
[         16 hours earlier] 003667.vlog   67 MB VL
[         16 hours earlier] 003669.vlog   69 MB VL
[         16 hours earlier] 003670.vlog   68 MB VL
[         16 hours earlier] 003671.vlog   68 MB VL
[         16 hours earlier] 003672.vlog   68 MB VL
[         16 hours earlier] 003673.vlog  109 MB VL
[         16 hours earlier] 003674.vlog   68 MB VL
[         16 hours earlier] 003675.vlog   70 MB VL
[         16 hours earlier] 003676.vlog   68 MB VL
[         16 hours earlier] 003677.vlog   68 MB VL
[         16 hours earlier] 003678.vlog   67 MB VL
[         16 hours earlier] 003679.vlog   67 MB VL
[         15 hours earlier] 003680.vlog   69 MB VL
[         15 hours earlier] 003681.vlog   69 MB VL
[         15 hours earlier] 003682.vlog   69 MB VL
[         15 hours earlier] 003683.vlog   70 MB VL
[         15 hours earlier] 003684.vlog   68 MB VL
[         15 hours earlier] 003685.vlog   68 MB VL
[         15 hours earlier] 003686.vlog   69 MB VL
[         15 hours earlier] 003687.vlog   69 MB VL
[         15 hours earlier] 003688.vlog   70 MB VL
[         15 hours earlier] 003689.vlog   68 MB VL
[         15 hours earlier] 003690.vlog   68 MB VL
[         15 hours earlier] 003691.vlog   68 MB VL
[         15 hours earlier] 003692.vlog   69 MB VL
[         15 hours earlier] 003693.vlog   73 MB VL
[         15 hours earlier] 003694.vlog   69 MB VL
[         15 hours earlier] 003695.vlog   68 MB VL
[         15 hours earlier] 003696.vlog   68 MB VL
[         15 hours earlier] 003697.vlog   68 MB VL
[         15 hours earlier] 003698.vlog   68 MB VL
[         15 hours earlier] 003699.vlog   69 MB VL
[         15 hours earlier] 003700.vlog   69 MB VL
[         15 hours earlier] 003701.vlog   69 MB VL
[         15 hours earlier] 003702.vlog   72 MB VL
[         15 hours earlier] 003703.vlog   69 MB VL
[         15 hours earlier] 003704.vlog   67 MB VL
[         15 hours earlier] 003705.vlog   70 MB VL
[         15 hours earlier] 003706.vlog   68 MB VL
[         14 hours earlier] 003707.vlog   69 MB VL
[         14 hours earlier] 003708.vlog   68 MB VL
[         14 hours earlier] 003709.vlog   68 MB VL
[         14 hours earlier] 003710.vlog   69 MB VL
[         14 hours earlier] 003711.vlog   70 MB VL
[         14 hours earlier] 003712.vlog   70 MB VL
[         14 hours earlier] 003713.vlog   68 MB VL
[         14 hours earlier] 003714.vlog   68 MB VL
[         14 hours earlier] 003715.vlog   69 MB VL
[         14 hours earlier] 003716.vlog   67 MB VL
[         14 hours earlier] 003717.vlog   68 MB VL
[         14 hours earlier] 003718.vlog   71 MB VL
[         14 hours earlier] 003719.vlog   68 MB VL
[         14 hours earlier] 003720.vlog   67 MB VL
[         14 hours earlier] 003721.vlog   68 MB VL
[         14 hours earlier] 003722.vlog   68 MB VL
[         14 hours earlier] 003723.vlog   68 MB VL
[         14 hours earlier] 003724.vlog   68 MB VL
[         14 hours earlier] 003725.vlog   70 MB VL
[         14 hours earlier] 003726.vlog   68 MB VL
[         14 hours earlier] 003727.vlog   69 MB VL
[         14 hours earlier] 003728.vlog   67 MB VL
[         14 hours earlier] 003729.vlog   67 MB VL
[         13 hours earlier] 003730.vlog   68 MB VL
[         13 hours earlier] 003731.vlog   69 MB VL
[         13 hours earlier] 003732.vlog   95 MB VL
[         13 hours earlier] 003733.vlog   68 MB VL
[         13 hours earlier] 003734.vlog   68 MB VL
[         13 hours earlier] 003735.vlog   87 MB VL
[         13 hours earlier] 003736.vlog   68 MB VL
[         13 hours earlier] 003737.vlog   70 MB VL
[         13 hours earlier] 003738.vlog   80 MB VL
[         13 hours earlier] 003739.vlog  447 MB VL
[         13 hours earlier] 003740.vlog   70 MB VL
[         13 hours earlier] 003741.vlog   68 MB VL
[         13 hours earlier] 003742.vlog   69 MB VL
[         13 hours earlier] 003743.vlog   69 MB VL
[         13 hours earlier] 003744.vlog   69 MB VL
[         13 hours earlier] 003745.vlog   69 MB VL
[         13 hours earlier] 003746.vlog   68 MB VL
[         13 hours earlier] 003747.vlog  121 MB VL
[         12 hours earlier] 003748.vlog  559 MB VL
[         12 hours earlier] 003749.vlog   69 MB VL
[         12 hours earlier] 003750.vlog   69 MB VL
[         12 hours earlier] 003751.vlog   69 MB VL
[         12 hours earlier] 003752.vlog   69 MB VL
[         12 hours earlier] 003753.vlog  102 MB VL
[         12 hours earlier] 003754.vlog   67 MB VL
[         12 hours earlier] 003755.vlog   69 MB VL
[         12 hours earlier] 003756.vlog   33 MB VL

[EXTRA]
[2019-04-06T10:39:14Z] LOCK            2 B

[Summary]
Level 0 size:        43 MB
Level 1 size:       236 MB
Level 2 size:       2.6 GB
Level 3 size:        27 GB
Level 4 size:        64 GB
Total index size:    94 GB
Value log size:      18 GB

Abnormalities:
1 extra file.
0 missing files.
0 empty files.
0 truncated manifests.
7

ok, sorry guys, seems like i was not familiar with level compaction strategy, only with size tiered one from cassandra. i’ve read some articles how rocksdb and hence badger works. so, number of sstables is not the case. anyway, now 2 nodes are crashing with OOM in a regular basis and loading is not continuing…

fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x148dd8a, 0x16)
	/usr/local/go/src/runtime/panic.go:608 +0x72
runtime.sysMap(0xcdd0000000, 0x4000000, 0x1fd43b8)
	/usr/local/go/src/runtime/mem_linux.go:156 +0xc7
runtime.(*mheap).sysAlloc(0x1fba720, 0x4000000, 0x1fba738, 0x7f9e93260e58)
	/usr/local/go/src/runtime/malloc.go:619 +0x1c7
runtime.(*mheap).grow(0x1fba720, 0x1, 0x0)
	/usr/local/go/src/runtime/mheap.go:920 +0x42
runtime.(*mheap).allocSpanLocked(0x1fba720, 0x1, 0x1fd43c8, 0x400)
	/usr/local/go/src/runtime/mheap.go:848 +0x337
runtime.(*mheap).alloc_m(0x1fba720, 0x1, 0x16, 0x7f9e9359efff)
	/usr/local/go/src/runtime/mheap.go:692 +0x119
runtime.(*mheap).alloc.func1()
	/usr/local/go/src/runtime/mheap.go:759 +0x4c
runtime.(*mheap).alloc(0x1fba720, 0x1, 0x7f9e93010016, 0x7f9e93260e58)
	/usr/local/go/src/runtime/mheap.go:758 +0x8a
runtime.(*mcentral).grow(0x1fbbfd8, 0x0)
	/usr/local/go/src/runtime/mcentral.go:232 +0x94
runtime.(*mcentral).cacheSpan(0x1fbbfd8, 0x7f9e93260e58)
	/usr/local/go/src/runtime/mcentral.go:106 +0x2f8
runtime.(*mcache).refill(0x7fbdafe456c0, 0xc1bbacad16)
	/usr/local/go/src/runtime/mcache.go:122 +0x95
runtime.(*mcache).nextFree.func1()
	/usr/local/go/src/runtime/malloc.go:749 +0x32
runtime.systemstack(0x0)
	/usr/local/go/src/runtime/asm_amd64.s:351 +0x66
runtime.mstart()
	/usr/local/go/src/runtime/proc.go:1229

goroutine 1864 [running]:
runtime.systemstack_switch()
	/usr/local/go/src/runtime/asm_amd64.s:311 fp=0xc1bbacad20 sp=0xc1bbacad18 pc=0x8e1d60
runtime.(*mcache).nextFree(0x7fbdafe456c0, 0xe0e416, 0xc63ec97480, 0x20, 0x40)
	/usr/local/go/src/runtime/malloc.go:748 +0xb6 fp=0xc1bbacad78 sp=0xc1bbacad20 pc=0x8923d6
runtime.mallocgc(0xa0, 0x14401e0, 0xc586e8eb01, 0x203373)
	/usr/local/go/src/runtime/malloc.go:903 +0x793 fp=0xc1bbacae18 sp=0xc1bbacad78 pc=0x892d23
runtime.newobject(0x14401e0, 0x1)
	/usr/local/go/src/runtime/malloc.go:1032 +0x38 fp=0xc1bbacae48 sp=0xc1bbacae18 pc=0x893108
github.com/dgraph-io/dgraph/protos/pb.(*PostingList).Unmarshal(0xcdcfffb4f0, 0xc6dbd349a0, 0x36, 0x6c, 0x8b1e42, 0x8)
	/ext-go/1/src/github.com/dgraph-io/dgraph/protos/pb/pb.pb.go:13680 +0x44e fp=0xc1bbacaf48 sp=0xc1bbacae48 pc=0xe98a0e
github.com/dgraph-io/dgraph/posting.ReadPostingList.func1(0xc6dbd349a0, 0x36, 0x6c, 0x6c, 0x0)
	/ext-go/1/src/github.com/dgraph-io/dgraph/posting/mvcc.go:208 +0x74 fp=0xc1bbacafa0 sp=0xc1bbacaf48 pc=0xf99844
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*Item).Value(0xcdcf673d90, 0xc1bbacb070, 0x0, 0x0)
	/ext-go/1/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/iterator.go:109 +0x17a fp=0xc1bbacb010 sp=0xc1bbacafa0 pc=0xe2ac3a
github.com/dgraph-io/dgraph/posting.ReadPostingList(0xc7212a13c0, 0x18, 0x18, 0xcdcffd8fa0, 0x0, 0x64, 0x100)
	/ext-go/1/src/github.com/dgraph-io/dgraph/posting/mvcc.go:206 +0x305 fp=0xc1bbacb0b8 sp=0xc1bbacb010 pc=0xf95415
github.com/dgraph-io/dgraph/posting.getNew(0xc7212a13c0, 0x18, 0x18, 0xc0001b0300, 0x0, 0x0, 0x0)
	/ext-go/1/src/github.com/dgraph-io/dgraph/posting/mvcc.go:245 +0x1d3 fp=0xc1bbacb170 sp=0xc1bbacb0b8 pc=0xf956f3
github.com/dgraph-io/dgraph/posting.(*LocalCache).Get(0xcd89ea1ae0, 0xc7212a13c0, 0x18, 0x18, 0x18, 0x4, 0xc7212a13c0)
	/ext-go/1/src/github.com/dgraph-io/dgraph/posting/lists.go:212 +0xc2 fp=0xc1bbacb1c8 sp=0xc1bbacb170 pc=0xf941a2
github.com/dgraph-io/dgraph/posting.(*Txn).Get(0xcd89eef360, 0xc7212a13c0, 0x18, 0x18, 0xc7212a13c0, 0x18, 0x18)
	/ext-go/1/src/github.com/dgraph-io/dgraph/posting/oracle.go:76 +0x4d fp=0xc1bbacb210 sp=0xc1bbacb1c8 pc=0xf959bd
github.com/dgraph-io/dgraph/posting.(*Txn).addIndexMutation(0xcd89eef360, 0x15b9f00, 0xc000038018, 0xcdcfd02dc0, 0xc77ee8ce4c, 0x4, 0x0, 0x0)
	/ext-go/1/src/github.com/dgraph-io/dgraph/posting/index.go:120 +0x7c fp=0xc1bbacb260 sp=0xc1bbacb210 pc=0xf8a4ec
github.com/dgraph-io/dgraph/posting.(*Txn).addIndexMutations(0xcd89eef360, 0x15b9f00, 0xc000038018, 0xc1bbacb440, 0x1, 0x1)
	/ext-go/1/src/github.com/dgraph-io/dgraph/posting/index.go:109 +0x166 fp=0xc1bbacb2f0 sp=0xc1bbacb260 pc=0xf8a366
github.com/dgraph-io/dgraph/posting.(*List).AddMutationWithIndex(0xcdcf3a0930, 0x15b9f00, 0xc000038018, 0xc1cc1a7e00, 0xcd89eef360, 0x0, 0x0)
	/ext-go/1/src/github.com/dgraph-io/dgraph/posting/index.go:390 +0x479 fp=0xc1bbacb4d0 sp=0xc1bbacb2f0 pc=0xf8c2e9
github.com/dgraph-io/dgraph/worker.runMutation(0x15b9f00, 0xc000038018, 0xc1cc1a7e00, 0xcd89eef360, 0x0, 0x0)
	/ext-go/1/src/github.com/dgraph-io/dgraph/worker/mutation.go:88 +0x2e6 fp=0xc1bbacb688 sp=0xc1bbacb4d0 pc=0x10d4fc6
github.com/dgraph-io/dgraph/worker.(*node).applyMutations(0xc192b591a0, 0x15b9f00, 0xc000038018, 0xc1c0645580, 0x0, 0x0)
	/ext-go/1/src/github.com/dgraph-io/dgraph/worker/draft.go:246 +0x6e4 fp=0xc1bbacb920 sp=0xc1bbacb688 pc=0x10c7604
github.com/dgraph-io/dgraph/worker.(*node).applyCommitted(0xc192b591a0, 0xc1c0645580, 0xc1cc16d660, 0x15)
	/ext-go/1/src/github.com/dgraph-io/dgraph/worker/draft.go:271 +0xbc2 fp=0xc1bbacbc60 sp=0xc1bbacb920 pc=0x10c8692
github.com/dgraph-io/dgraph/worker.(*node).processApplyCh.func1(0xc1c0645a00, 0xb, 0x10)
	/ext-go/1/src/github.com/dgraph-io/dgraph/worker/draft.go:385 +0x1a5 fp=0xc1bbacbd68 sp=0xc1bbacbc60 pc=0x10f4375
github.com/dgraph-io/dgraph/worker.(*node).processApplyCh(0xc192b591a0)
	/ext-go/1/src/github.com/dgraph-io/dgraph/worker/draft.go:412 +0x241 fp=0xc1bbacbfd8 sp=0xc1bbacbd68 pc=0x10c8d71
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc1bbacbfe0 sp=0xc1bbacbfd8 pc=0x8e3e41
created by github.com/dgraph-io/dgraph/worker.(*node).InitAndStartNode
	/ext-go/1/src/github.com/dgraph-io/dgraph/worker/draft.go:1161 +0x4e0
8

If you can share a heap profile of the Alpha before it runs out of memory we can take a closer look.

https://docs.dgraph.io/howto/#profiling-information