Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When resume the pipeline halted by 404 error, the client doesn't as expected. #630

Closed
xdxxxdx opened this issue May 14, 2024 · 1 comment · Fixed by #723
Closed

When resume the pipeline halted by 404 error, the client doesn't as expected. #630

xdxxxdx opened this issue May 14, 2024 · 1 comment · Fixed by #723
Assignees
Labels
needs triage Issue needs to be evaluated by team

Comments

@xdxxxdx
Copy link

xdxxxdx commented May 14, 2024

Versions
ldes/ldi-orchestrator:2.6.0-SNAPSHOT

Describe the bug
Thanks to #605, when the client now faces 404 error, the pipeline is halted as expected.
But when the source LDES stream is recovered, manually resume the pipeline by API call, the pipeline doesn't continue consuming the LDES stream.

To Reproduce

  1. ldes/ldi-orchestrator:2.6.0-SNAPSHOT is up and running with the following configuration
server:
  port: 8080

orchestrator:
  pipelines:
    - name: bear-b-pipeline-client
      description: "bear-b dataset client for configured VSDS server"
      input:
        name: Ldio:LdesClient
        config:
          urls: http://ldes-server:8080/bearb/timebased-50
          source-format: "text/turtle"
      outputs:
            - name: Ldio:ConsoleOut
              config:
                content-type: text/turtle

spring:
  codec:
    max-in-memory-size: 10MB
    
logging:
  level:
    root: INFO
    Ldio.Application: INFO
    Ldio:HttpIn: DEBUG
    Ldio:HttpOut: DEBUG

http://ldes-server:8080/bearb/timebased-50 is not available for moment => UPDATED status for pipeline 'bear-b-pipeline-client' to HALTED as expected
3.http://ldes-server:8080/bearb/timebased-50 is back online
4. Call API http://localhost:8082/admin/api/v1/pipeline/bear-b-pipeline-client/resume to resume the pipeline

Current behavior
In the trace of LDIO we can see the pipeline is recovered, but it doesn't restart to consume the LDES stream.
image

Expected behavior
The pipeline is recovered and continue consuming the stream http://ldes-server:8080/bearb/timebased-50.
If resuming the pipeline doesn't' resume the function, there is no need to resume it..

@xdxxxdx xdxxxdx added the needs triage Issue needs to be evaluated by team label May 14, 2024
@github-project-automation github-project-automation bot moved this to 📋 Backlog in VSDS Backlog May 14, 2024
@jobulcke
Copy link
Collaborator

A little investigation shows that the task that is responsible for everything in the ExecutorService is considered finished after an exception is thrown, which is not the case when the client is paused manually. So a way to "restart" the process(es) responsible in the client in a clean way needs to be found and implemented

@Yalz Yalz self-assigned this Nov 20, 2024
@Yalz Yalz linked a pull request Nov 26, 2024 that will close this issue
@Yalz Yalz closed this as completed Nov 27, 2024
@github-project-automation github-project-automation bot moved this from 📋 Backlog to 👀 In review in VSDS Backlog Nov 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs triage Issue needs to be evaluated by team
Projects
Status: 👀 In review
Development

Successfully merging a pull request may close this issue.

3 participants