log: pull in logrotator, switch to ZSTD compressor

[jharveyb]

The log rotator we currently use is from a now-unmaintained repo: https://github.com/jrick/logrotate/tree/master

Which is fine, it works. But this issue lightninglabs/taproot-assets#1084 prompted me to look at how we could switch from gzip to zstd for log compression (better compression at the default level: https://pkg.go.dev/github.com/klauspost/compress/zstd#readme-performance).

Since the log rotator is one short file, it seems easiest to just pull it into this repo directly vs. fork the logrotate package and release this updated version with ZSTD support, and then pull that in as a dependency.

If this gets merged to btcd, we could do the same lnd and then drop the logrotate dependency there as well. Once it's in the lnd build package, it could be used from the taproot-assets project.

The diff from upstream logrotate is quite small, and can be checked easily:

wget https://github.com/jrick/logrotate/raw/master/rotator/rotator.go
wget https://github.com/jharveyb/btcd/raw/logrotator_vendor_zstd/logrotator.go
diff -y --suppress-common-lines rotator.go logrotator.go

This is a more minimal diff from difftastic (https://difftastic.wilfred.me.uk/) with upstream on the left, and my version on the right:

logrotator.go --- 1/6 --- Go
14 //                                                      14 //
15 // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS A 15 // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS A
.. ND CONTRIBUTORS "AS IS"                                 .. ND CONTRIBUTORS "AS IS"
16 // AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BU 16 // AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BU
.. T NOT LIMITED TO, THE                                   .. T NOT LIMITED TO, THE
17 // IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FO 17 // IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FO
.. R A PARTICULAR PURPOSE ARE                              .. R A PARTICULAR PURPOSE
18 // DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER O 18 // ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLD
.. R CONTRIBUTORS BE LIABLE                                .. ER OR CONTRIBUTORS BE
19 // FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMP 19 // LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL
.. LARY, OR CONSEQUENTIAL                                  .. , EXEMPLARY, OR
20 // DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT  20 // CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO
.. OF SUBSTITUTE GOODS OR                                  .. , PROCUREMENT OF
21 // SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 21 // SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 
..  INTERRUPTION) HOWEVER                                  .. PROFITS; OR BUSINESS
22 // CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CO 22 // INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LI
.. NTRACT, STRICT LIABILITY,                               .. ABILITY, WHETHER IN
23 // OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING  23 // CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLI
.. IN ANY WAY OUT OF THE USE                               .. GENCE OR OTHERWISE)
24 // OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY 24 // ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, 
..  OF SUCH DAMAGE.                                        .. EVEN IF ADVISED OF THE
..                                                         25 // POSSIBILITY OF SUCH DAMAGE.
25                                                         26 
26 // Package rotator implements a simple logfile rotator. 27 // Package rotator implements a simple logfile rotator.
..  Logs are read from an                                  ..  Logs are read from an
27 // io.Reader and are written to a file until they reach 28 // io.Reader and are written to a file until they reach
..  a specified size. The                                  ..  a specified size. The
28 // log is then gzipped to another file and truncated.   29 // log is then gzipped to another file and truncated.
29 package rotator                                         30 package main
30                                                         31 
31 import (                                                32 import (
32     "bufio"                                             33     "bufio"

logrotator.go --- 2/6 --- Go
39 40     "strings"
40 41     "sync"
.. 42 
.. 43     "github.com/klauspost/compress/zstd"
41 44 )
42 45 
43 46 // nl is a byte slice containing a newline byte.  It is used to avoid creating
44 47 // additional allocations when writing newlines to the log file.
45 48 var nl = []byte{'\n'}
.. 49 
.. 50 // Compressor represents the supported compression algorithms.
.. 51 type Compressor uint8
.. 52 
.. 53 const (
.. 54     // Gzip is the gzip compression algorithm, implemented in the stdlib.
.. 55     Gzip Compressor = 0
.. 56 
.. 57     // Zstd is the zstd compression algorithm.
.. 58     Zstd Compressor = 1
.. 59 )
.. 60 
.. 61 // Validate checks that the Compressor is valid, and returns the appropriate
.. 62 // file suffix for the compressed file.
.. 63 func (c Compressor) Validate() (string, error) {
.. 64     switch c {
.. 65     case Gzip:
.. 66         return ".gz", nil
.. 67     case Zstd:
.. 68         return ".zst", nil
.. 69 
.. 70     default:
.. 71         return "", fmt.Errorf("unknown compression algorithm: %d", c)
.. 72     }
.. 73 }
.. 74 
.. 75 // Init creates a new writer to use for compression.
.. 76 func (c Compressor) Init(w io.Writer) (io.WriteCloser, error) {
.. 77     switch c {
.. 78     case Gzip:
.. 79         dst := gzip.NewWriter(w)
.. 80         return dst, nil
.. 81 
.. 82     case Zstd:
.. 83         return zstd.NewWriter(w)
.. 84 
.. 85     default:
.. 86         return nil, fmt.Errorf("unknown compression algorithm: %d", c)
.. 87     }
.. 88 }
46 89 
47 90 // A Rotator writes input to a file, splitting it up into gzipped chunks once
48 91 // the filesize reaches a certain threshold.

logrotator.go --- 3/6 --- Go
54     out       *os.File                                  97     out       *os.File
55     tee       bool                                      98     tee       bool
56     wg        sync.WaitGroup                            99     wg        sync.WaitGroup
..                                                        100     comp      Compressor
57 }                                                      101 }
58                                                        102 
59 // New returns a new Rotator.  The rotator can be used 103 // NewRotator returns a new Rotator. The rotator can b
..  either by reading input                               ... e used either by reading
60 // from an io.Reader by calling Run, or writing direct 104 // input from an io.Reader by calling Run, or writing 
.. ly to the Rotator with                                 ... directly to the Rotator
61 // Write.                                              105 // with Write. The default compression algorithm is Gz
..                                                        ... ip.
62 func New(filename string, thresholdKB int64, tee bool, 106 func NewRotator(filename string, thresholdKB int64, te
..  maxRolls int) (*Rotator, error) {                     ... e bool,
..                                                        107     maxRolls int) (*Rotator, error) {
..                                                        108 
63     f, err := os.OpenFile(filename, os.O_CREATE|os.O_A 109     f, err := os.OpenFile(filename, os.O_CREATE|os.O_A
   PPEND|os.O_RDWR, 0644)                                     PPEND|os.O_RDWR, 0644)

logrotator.go --- 4/6 --- Go
77 123         filename:  filename,
78 124         out:       f,
79 125         tee:       tee,
.. 126         comp:      Gzip,
80 127     }, nil
81 128 }
.. 129 
.. 130 // NewRotatorWithCompressor returns a new Rotator that will use a specific
.. 131 // compression algorithm.
.. 132 func NewRotatorWithCompressor(filename string, thresholdKB int64, tee bool,
.. 133     maxRolls int, comp Compressor) (*Rotator, error) {
.. 134 
.. 135     _, err := comp.Validate()
.. 136     if err != nil {
.. 137         return nil, err
.. 138     }
.. 139 
.. 140     rotator, err := NewRotator(filename, thresholdKB, tee, maxRolls)
.. 141     if err != nil {
.. 142         return nil, err
.. 143     }
.. 144 
.. 145     rotator.comp = comp
.. 146 
.. 147     return rotator, nil
.. 148 }
82 149 
83 150 // Run begins reading lines from the reader and rotating logs as necessary.  Run
84 151 // should not be called concurrently with Write.

logrotator.go --- 5/6 --- Go
202 269 
203 270     r.wg.Add(1)
204 271     go func() {
205 272         err := compress(rotname, r.comp)
206 273         if err == nil {
207 274             os.Remove(rotname)
208 275         }

logrotator.go --- 6/6 --- Go
212     return nil                                         279     return nil
213 }                                                      280 }
214                                                        281 
215 func compress(name string) (err error) {               282 func compress(name string, comp Compressor) (err error
...                                                        ... ) {
216     f, err := os.Open(name)                            283     f, err := os.Open(name)
217     if err != nil {                                    284     if err != nil {
218         return err                                     285         return err
219     }                                                  286     }
220     defer f.Close()                                    287     defer f.Close()
...                                                        288 
...                                                        289     suffix, err := comp.Validate()
...                                                        290     if err != nil {
...                                                        291         return err
...                                                        292     }
221                                                        293 
222     arc, err := os.OpenFile(name+".gz", os.O_CREATE|os 294     arc, err := os.OpenFile(
... .O_EXCL|os.O_WRONLY, 0644)                             ... 
...                                                        295         name+suffix, os.O_CREATE|os.O_EXCL|os.O_WRONLY
...                                                        ... , 0644,
...                                                        296     )
223     if err != nil {                                    297     if err != nil {
224         return err                                     298         return err
225     }                                                  299     }
226                                                        300 
227     z := gzip.NewWriter(arc)                           301     z, err := comp.Init(arc)
...                                                        302     if err != nil {
...                                                        303         return err
...                                                        304     }
...                                                        305 
228     if _, err = io.Copy(z, f); err != nil {            306     if _, err = io.Copy(z, f); err != nil {
229         return err                                     307         return err

[jrick]

I still maintain it, it just has not needed any changes :)

[jharveyb]

I still maintain it, it just has not needed any changes :)

Fair enough; having a configurable compressor would be a useful option. It seems like having zstd in the stdlib may happen, eventually:

golang/go#62513

But I don't see a reason to wait on that. Opened a PR on upstream: jrick/logrotate#2

[guggero]

I'm in favor of merging jrick/logrotate#2 and then using the latest version.

[jharveyb]

Closed in favor of #2238