Cannot upload session to Azure Blob Storage

[AnNHSVT]

Hello team,

Have you ever encountered this case? Please help me with action recommendations to help troubleshoot.

Thanks and Regards!

Expected behavior:
Can start service and access sucessfully.

Current behavior:
Failed authorized on Auth_services and App_Service

Bug details:

• Teleport version: 16.1
• Recreation steps:

1. Create teleport.yaml on proxy and auth node
   Auth_service config:

teleport:
  nodename: teleport-auth-01
  advertise_ip: 10.20.11.52
  log:
    output: stderr
    severity: DEBUG
  data_dir: /var/lib/teleport

  storage:
    type: etcd
    peers: ["http://127.0.0.1:2379"]
    prefix: /teleport/
    insecure: true
    username: root
    password_file: /etc/teleport/.etcd-passwd
    etcd_max_client_msg_size_bytes: 15728640
    audit_sessions_uri: azblob://mystorageaccount.blob.core.windows.net#session_container=teleport-session&inprogress_container=teleport-ingress

ssh_service:
  enabled: no

proxy_service:
  enabled: no

auth_service:
  enabled: yes
  keep_alive_interval: 1m
  keep_alive_count_max: 3
  listen_addr: 0.0.0.0:3025
  public_addr: 1.1.1.1:3025
  proxy_listener_mode: multiplex
  authentication:
    type: local
    second_factor: otp
  cluster_name: 'teleport'
  session_recording: "node"
  proxy_checks_host_keys: no

3. Create teleport.yaml on Agent node

version: v3
teleport:
  nodename: teleport-agent-01
  data_dir: /etc
  join_params:
    token_name: b4afe55519ed83d87a6a75XXXXXXXX
    method: token
  proxy_server: rcenter.example.com:443
  log:
    output: stderr
    severity: INFO
    format:
      output: text
  ca_pin: ""
  diag_addr: ""
auth_service:
  enabled: "no"
ssh_service:
  enabled: "no"
proxy_service:
  enabled: "no"
  https_keypairs: []
  https_keypairs_reload_interval: 0s
  acme: {}
app_service:
  enabled: "yes"
  debug_app: true
  apps:
  - name: myapp
    uri: https://webserver.domain2.com/
    public_addr: "myapp.example.com"
    insecure_skip_verify: true
    labels:
      env: "prod"

4. Run app with command teleport start or systemd service

• Debug logs:
• ------ Auth_service node:

Aug 05 16:50:26 teleport-auth-01 teleport[31486]: 2024-08-05T16:50:26+07:00 ERRO [AUTH:GRPC] Failed to create audit stream: "AuthorizationFailure\n\taccess denied". auth/grpcserver.go:355
Aug 05 16:50:26 teleport-auth-01 teleport[31486]: 2024-08-05T16:50:26+07:00 ERRO [AUTH:GRPC] Failed to create audit stream: "AuthorizationFailure\n\taccess denied". auth/grpcserver.go:355
Aug 05 16:50:26 teleport-auth-01. teleport[31486]: 2024-08-05T16:50:26+07:00 ERRO [AUTH:GRPC] Failed to create audit stream: "AuthorizationFailure\n\taccess denied". auth/grpcserver.go:355
Aug 05 16:50:26 teleport-auth-01 teleport[31486]: 2024-08-05T16:50:26+07:00 ERRO [AUTH:GRPC] Failed to create audit stream: "AuthorizationFailure\n\taccess denied". auth/grpcserver.go:355
Aug 05 16:50:27 teleport-auth-01. teleport[31486]: 2024-08-05T16:50:27+07:00 ERRO [AUTH:GRPC] Failed to create audit stream: "AuthorizationFailure\n\taccess denied". auth/grpcserver.go:355
Aug 05 16:50:27 teleport-auth-01. teleport[31486]: 2024-08-05T16:50:27+07:00 ERRO [AUTH:GRPC] Failed to create audit stream: "AuthorizationFailure\n\taccess denied". auth/grpcserver.go:355
Aug 05 16:50:27 teleport-auth-01. teleport[31486]: 2024-08-05T16:50:27+07:00 ERRO [AUTH:GRPC] Failed to create audit stream: "AuthorizationFailure\n\taccess denied". auth/grpcserver.go:355
Aug 05 16:50:27 teleport-auth-01 teleport[31486]: 2024-08-05T16:50:27+07:00 ERRO [AUTH:GRPC] Failed to create audit stream: "AuthorizationFailure\n\taccess denied". auth/grpcserver.go:355

--- Agent node:

2024-08-05T16:50:26+07:00 WARN [UPLOAD]    Upload failed. error:[
ERROR REPORT:
Original Error: *trace.ConnectionProblemError AuthorizationFailure
        access denied
Stack Trace:
        github.com/gravitational/teleport/lib/events/filesessions/fileasync.go:537 github.com/gravitational/teleport/lib/events/filesessions.(*Uploader).upload
        github.com/gravitational/teleport/lib/events/filesessions/fileasync.go:464 github.com/gravitational/teleport/lib/events/filesessions.(*Uploader).startUpload.func2
        runtime/asm_amd64.s:1695 runtime.goexit
User Message: AuthorizationFailure
        access denied] session:8051a098-05e8-4c59-b770-ae88dfc51bf8 filesessions/fileasync.go:465
2024-08-05T16:50:26+07:00 WARN [UPLOAD]    Increasing session upload backoff due to error, will retry after 30.059671279s. filesessions/fileasync.go:228
2024-08-05T16:50:26+07:00 WARN [UPLOAD]    Upload failed. error:[
ERROR REPORT:
Original Error: *trace.ConnectionProblemError AuthorizationFailure
        access denied
Stack Trace:
        github.com/gravitational/teleport/lib/events/filesessions/fileasync.go:537 github.com/gravitational/teleport/lib/events/filesessions.(*Uploader).upload
        github.com/gravitational/teleport/lib/events/filesessions/fileasync.go:464 github.com/gravitational/teleport/lib/events/filesessions.(*Uploader).startUpload.func2
        runtime/asm_amd64.s:1695 runtime.goexit
User Message: AuthorizationFailure
        access denied] session:467252d0-5044-4ce5-a276-cde4541b2ca8 filesessions/fileasync.go:465
2024-08-05T16:50:26+07:00 WARN [UPLOAD]    Increasing session upload backoff due to error, will retry after 37.664050075s. filesessions/fileasync.go:228
2024-08-05T16:50:26+07:00 WARN [UPLOAD]    Upload failed. error:[
ERROR REPORT:
Original Error: *trace.ConnectionProblemError AuthorizationFailure
        access denied
Stack Trace:
        github.com/gravitational/teleport/lib/events/filesessions/fileasync.go:537 github.com/gravitational/teleport/lib/events/filesessions.(*Uploader).upload
        github.com/gravitational/teleport/lib/events/filesessions/fileasync.go:464 github.com/gravitational/teleport/lib/events/filesessions.(*Uploader).startUpload.func2
        runtime/asm_amd64.s:1695 runtime.goexit
User Message: AuthorizationFailure
        access denied] session:4d117e6f-0b0c-46fa-b7c9-e7d15ce99a48 filesessions/fileasync.go:465
2024-08-05T16:50:26+07:00 WARN [UPLOAD]    Increasing session upload backoff due to error, will retry after 41.614451838s. filesessions/fileasync.go:228
2024-08-05T16:50:26+07:00 WARN [UPLOAD]    Upload failed. error:[
ERROR REPORT:
Original Error: *trace.ConnectionProblemError AuthorizationFailure
        access denied
Stack Trace:
        github.com/gravitational/teleport/lib/events/filesessions/fileasync.go:537 github.com/gravitational/teleport/lib/events/filesessions.(*Uploader).upload
        github.com/gravitational/teleport/lib/events/filesessions/fileasync.go:464 github.com/gravitational/teleport/lib/events/filesessions.(*Uploader).startUpload.func2
        runtime/asm_amd64.s:1695 runtime.goexit
User Message: AuthorizationFailure
        access denied] session:97a90149-08d5-4d4b-abea-497eb1745eb4 filesessions/fileasync.go:465
2024-08-05T16:50:26+07:00 WARN [UPLOAD]    Increasing session upload backoff due to error, will retry after 48.676121707s. filesessions/fileasync.go:228
2024-08-05T16:50:26+07:00 WARN [UPLOAD]    Upload failed. error:[
ERROR REPORT:
Original Error: *trace.ConnectionProblemError AuthorizationFailure
        access denied
Stack Trace:
        github.com/gravitational/teleport/lib/events/filesessions/fileasync.go:537 github.com/gravitational/teleport/lib/events/filesessions.(*Uploader).upload
        github.com/gravitational/teleport/lib/events/filesessions/fileasync.go:464 github.com/gravitational/teleport/lib/events/filesessions.(*Uploader).startUpload.func2
        runtime/asm_amd64.s:1695 runtime.goexit
User Message: AuthorizationFailure
        access denied] session:181266a0-3e2c-4de7-bd17-706ebaa4006a filesessions/fileasync.go:465

[webvictim]

How did you grant the required authorization for your Teleport auth_service to upload to Azure?

[AnNHSVT]

How did you grant the required authorization for your Teleport auth_service to upload to Azure?

I have create App register on azure and i have try add role "Storage Blob Data Owner" to this user but it doesn't help.