klauspost commented :
IIUC, I can decompress the data that was created by dataDog/zstd using klauspost/compres (which means they’re compatible), is this correct?
Yes they are compatible both ways. The only exception is 0 bytes of input which will give 0 bytes output with the Go zstd. But you already have the zstd.WithZeroFrames(true) which will wrap 0 bytes in a header so it can be fed to DD zstd. This will of course only be relevant when downgrading.
I am fuzz testing the change above. It will have much less compression impact than completely disabling entropy coding, but will handle the random input blocks better. I would probably leave out the WithNoEntropyCompression and upgrade to the next version that will select this automatically when it makes sense.
number of major page fails
The dd library allocates a lot. That could be why. Go zstd does not allocate for 4K blocks after a few runs.
why wouldn’t you use CGO in your code?
Compiling is much slower. Cross compilation is a pain to set up/impossible for some. Deployment requires dependencies whereas plain Go is just the executable. cgo is inherently less secure since none of the c code has the security features the Go runtime provides.