[Bug]: session cleanup with file system storage

[aritas1]

zot version

v2.0.0-rc6

Describe the bug

After running rc6 for ~2 months now i found aporx. 4.3 million session stored on disk, blocking around 16GB.

To reproduce

image: ghcr.io/project-zot/zot-linux-amd64:v2.0.0-rc6

config.yaml

distspecversion: 1.1.0-dev
http:
  address: 0.0.0.0
  port: 8080
  # externalUrl: "https://registry.example.com"  # as of v2.0.0-rc7
  #realm: zot
  auth:
    #faildelay: 5
    faildelay: 1
    htpasswd:
      path: /config/htpasswd
    openid:
      providers:
        dex:
          issuer: "https://sso.zitadel.com"
          clientid: zot@zitadel
          clientsecret: secret
          scopes:
          - openid
          - email
  accessControl:
    adminPolicy:
      users:
      - admin
      actions:
      - read
      - create
      - update
      - delete
      - detectManifestCollision
    repositories:
      "**":
        defaultPolicy:
        - read
        - create
log:
  level: info
storage:
  rootdirectory: /data/zot
  dedupe: true
  gc: true
extensions:
  metrics:
    enable: true
    prometheus:
      path: "/metrics_silly"
  search:
    enable: true
    cve:
      updateInterval: 24h
  ui:
    enable: true
  mgmt:
    enable: true

Expected behavior

It would be nice if (expired) sessions can be deleted automatically and session duration can be configured.

Screenshots

No response

Additional context

No response

[rchincha]

@aritas1 Thanks for trying out zot (for two months running!) and reporting this.

Will triage this.

[rchincha]

@aritas1 Assuming you have social login/auth enabled in zot config?

[aritas1]

Yes, see the config above.
both, htpasswd and openid are enabled.

Im testing/debugging my instance for a few minutes now, seems like all basic auth clients generate a session for every http request.
All requests by docker, kubernetes, python or prometheus results in a "user profile successfully set" message.

zot-zot-1  | {"level":"info","identity":"abc","goroutine":259344,"caller":"zotregistry.io/zot/pkg/api/authn.go:177","time":"2023-10-09T17:59:13.766229779Z","message":"user profile successfully set"}
zot-zot-1  | {"level":"info","module":"http","username":"abc","clientIP":"172.16.0.33:35258","method":"HEAD","path":"/v2/xxxxx/xxxx/app/manifests/latest","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["application/vnd.docker.distribution.manifest.v2+json","application/vnd.docker.distribution.manifest.list.v2+json","application/vnd.docker.distribution.manifest.v1+json","application/vnd.oci.image.index.v1+json"],"Authorization":["******"],"Connection":["close"],"User-Agent":["Watchtower/1.6.0"],"X-Forwarded-For":["::ffff:xxx.xxx.xxx.219"],"X-Forwarded-Port":["443"],"X-Forwarded-Proto":["https"],"X-Real-Ip":["::ffff:xxx.xxx.xxx.219"]},"goroutine":259344,"caller":"zotregistry.io/zot/pkg/api/session.go:131","time":"2023-10-09T17:59:13.766697872Z","message":"HTTP API"}

zot-zot-1  | {"level":"info","identity":"monitoring","goroutine":265576,"caller":"zotregistry.io/zot/pkg/api/authn.go:177","time":"2023-10-09T19:02:38.290300165Z","message":"user profile successfully set"}
zot-zot-1  | {"level":"info","module":"http","username":"monitoring","clientIP":"172.16.0.33:35116","method":"GET","path":"/metrics_silly","statusCode":200,"latency":"0s","bodySize":3153,"headers":{"Accept":["application/openmetrics-text; version=0.0.1,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"Accept-Encoding":["gzip"],"Authorization":["******"],"Connection":["close"],"User-Agent":["Prometheus/2.32.0"],"X-Forwarded-For":["::ffff:172.18.0.1"],"X-Forwarded-Port":["443"],"X-Forwarded-Proto":["https"],"X-Prometheus-Scrape-Timeout-Seconds":["10"],"X-Real-Ip":["::ffff:172.18.0.1"]},"goroutine":265576,"caller":"zotregistry.io/zot/pkg/api/session.go:131","time":"2023-10-09T19:02:38.291848025Z","message":"HTTP API"}

I assume none of the basic auth clients is complying with the session cookie part? so storing the session in the first place may be an issue.

If i try the webinterface, the sesison cookie is set and present, so no new session gets created on the disk.

[andaaron]

#1919 takes care of the logic creating the sessions for requests coming from command-line tools.

We still need logic for handling old sessions in a separate PR. Right now we only delete for the UI, on logout

[andaaron]

#1939 Also takes care of cleaning the existing sessions.

[rchincha]

@aritas1 do you want to try deploying top of main to verify. We are waiting for a few more key PRs before we do another release.

[aritas1]

running on a44ca57 now, everything looks great.