v0.8.13

Change spooled threshold from 2MB to 500KB

v0.8.12

fix: remove nil assignments (#18)

v0.8.11

feat: Use a pool for spooledTempFile buffers (#17)

v0.8.10

Add TLS specific things & tempDir (#16)

* Restore old tests. (They still pass, of course.)

* Remove files after test.

* tempDir and TLS

* feat: simply ask NewWARCWritingHTTPClient for the temporary directory. Empty == "/tmp" or the default temporary directory according to Go.

Co-authored-by: Jake L <[email protected]>

v0.8.9

Using spooledTempFile to dynamically move payloads to disk when they …

v0.8.8

Add toggle for verifying HTTP certificates. (#14)

* feat: add toggle for verifying HTTP certificates.

before: we had a function to verify, but it would never get called as InsecureSkipVerify was never changed, as such, things could not get archived due to invalid certificates, when we would like them to.

New: Having InsecureSkipVerify set on the tls.Config allows the normal TLS functions to check the certificate. We've configured this to a toggle on NewWARCWritingHTTPClient to allow us to turn the verification of certificates on or off.

If the setting is set to false, certificates will not be checked. If the setting is set to true, certificates are checked.

* Fix: tests

* feat: add tests to test this new feature.

* fix: remove unused TLSClientConfig

* fix: change verifyCertificates to verifyCerts

v0.8.7

Add error channel (#13)

v0.8.6

Ignore CDX errors (#11)

feat: ignore errors on CDX revisit at this time, as they are not crucial and would cause the response to not be captured.

Co-authored-by: Jake LaFountain <[email protected]>

v0.8.5

Memory management fixes (#8)

* feat: add initial "read" testing on our dynamically written warcs

on every single HTTPClient test, look for all files for that test (since we can't really return the WARC file name) and check that WARC for the correct amount of response records and Payload-Digest. Once we've reached EOF, check that we read the expected number, if we haven't error out. If we have, all good, everything works!

This vastly improves our current tests, as we can ensure we're actually writing WARCs, and that they most likely hold the correct content.

* feat: add test for files over 2MB to ensure we're properly handling them still, with our move to temporary files (for files over 2MB!)

(do note, this SHA hash has been verified with SPN and wget-at.)

* Add: Test1MConcurrentWARCWritingWithHTTPClientLargerThan2MB

* Fix: Test1MConcurrentWARCWritingWithHTTPClientLargerThan2MB

* remove tmpFiles and remove duplicate memory usage

* Comment out Test1MConcurrentWARCWritingWithHTTPClientLargerThan2MB

* Fix: TestReader, more specifically, the "testFIleHash" portion that was failing.

* Some small syntax fixes

* Remove useless log line

* Remove useless code, make HTTP status code check happen ASAP

* Fix comment

Co-authored-by: Jake L <[email protected]>
Co-authored-by: Fionera <[email protected]>

v0.8.4

DRAFT: Write WARC responses to temporary files to (hopefully) avoid O…