Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Inconsistent data when consulting jobs/instances #5484

Closed
khaeghar opened this issue Sep 27, 2023 · 0 comments
Closed

Inconsistent data when consulting jobs/instances #5484

khaeghar opened this issue Sep 27, 2023 · 0 comments
Assignees
@corneil
Labels
type/bug Is a bug report
Milestone
2.11.2

Comments

@khaeghar
Copy link

Description:
When I use the SCDF API and use the jobs/instances endpoint to consult the status of a jobInstance, it returns empty values and confusing data about the jobExecutions it has, like, for example, a never-ending STARTING status, and null or empty date stamps.

Weird thing is, when I use the jobs/executions endpoint to check the jobExecution data, all data comes correctly, and when I query manually against the database the same query used by SCDF to return the values of the jobs/instances endpoint, it returns the data correctly.

So I think it may be some export data error that's not working properly.

Release versions:

image

Steps to reproduce:

Execute a SCDF Server with a PostgreSQL database, and launch a task with a job. Then, make use of the SCDF API to query the data of the job instances and check the data obtained.

Additional context:

Additional info:

The data queried from the database, exactly the same way SCDF does it (literally copied the query and pasted it into the DBeaver):

SELECT I.JOB_INSTANCE_ID as JOB_INSTANCE_ID, I.JOB_NAME as JOB_NAME, I.SCHEMA_TARGET as SCHEMA_TARGET, E.JOB_EXECUTION_ID as JOB_EXECUTION_ID, E.START_TIME as START_TIME, E.END_TIME as END_TIME, E.STATUS as STATUS, E.EXIT_CODE as EXIT_CODE, E.EXIT_MESSAGE as EXIT_MESSAGE, E.CREATE_TIME as CREATE_TIME, E.LAST_UPDATED as LAST_UPDATED, E.VERSION as VERSION, T.TASK_EXECUTION_ID as TASK_EXECUTION_ID, (SELECT COUNT(*) FROM AGGREGATE_STEP_EXECUTION S WHERE S.JOB_EXECUTION_ID = E.JOB_EXECUTION_ID AND S.SCHEMA_TARGET = E.SCHEMA_TARGET) as STEP_COUNT from AGGREGATE_JOB_INSTANCE I JOIN AGGREGATE_JOB_EXECUTION E ON I.JOB_INSTANCE_ID = E.JOB_INSTANCE_ID AND I.SCHEMA_TARGET = E.SCHEMA_TARGET LEFT OUTER JOIN AGGREGATE_TASK_BATCH TT ON E.JOB_EXECUTION_ID = TT.JOB_EXECUTION_ID AND E.SCHEMA_TARGET = TT.SCHEMA_TARGET LEFT OUTER JOIN AGGREGATE_TASK_EXECUTION T ON TT.TASK_EXECUTION_ID = T.TASK_EXECUTION_ID AND TT.SCHEMA_TARGET = T.SCHEMA_TARGET where I.JOB_INSTANCE_ID = 2 AND I.SCHEMA_TARGET = 'boot3';

The data obtained from the database. The JobExecutionId is 2:

image

The JSON obtained from the cUrl to the SCDF Server with the incorrect data:

The endpoint:

{{scdf_host}}/jobs/instances/2?schemaTarget=boot3

The obtained JSON:

{
    "jobName": "customerJob",
    "jobInstanceId": 2,
    "jobExecutions": [
        {
            "executionId": 2,
            "stepExecutionCount": 3,
            "jobId": 2,
            "taskExecutionId": 5,
            "name": "customerJob",
            "startDate": "",
            "startTime": "",
            "duration": "",
            "jobExecution": {
                "id": 2,
                "version": null,
                "jobParameters": {
                    "parameters": {
                        "a": {
                            "identifying": true,
                            "value": "c",
                            "type": "STRING"
                        },
                        "app.apabatch.2": {
                            "identifying": true,
                            "value": "sleepTime=0",
                            "type": "STRING"
                        },
                        "run.id": {
                            "identifying": true,
                            "value": 2.0,
                            "type": "DOUBLE"
                        },
                        "app.apabatch.1": {
                            "identifying": true,
                            "value": "failChunk=-1",
                            "type": "STRING"
                        },
                        "app.apabatch.3": {
                            "identifying": true,
                            "value": "failTasklet=false",
                            "type": "STRING"
                        },
                        "app.apabatch.0": {
                            "identifying": true,
                            "value": "message=test_lanzamiento_scdf",
                            "type": "STRING"
                        }
                    }
                },
                "jobInstance": {
                    "id": 2,
                    "jobName": "customerJob",
                    "version": null
                },
                "stepExecutions": [],
                "status": "STARTING",
                "startTime": null,
                "createTime": "2023-09-27T14:31:49.878+0000",
                "endTime": null,
                "lastUpdated": null,
                "exitStatus": {
                    "exitCode": "UNKNOWN",
                    "exitDescription": ""
                },
                "executionContext": {
                    "dirty": false,
                    "empty": true,
                    "values": []
                },
                "failureExceptions": [],
                "jobConfigurationName": null,
                "allFailureExceptions": []
            },
            "jobParameters": {
                "a": "c",
                "app.apabatch.2": "sleepTime=0",
                "app.apabatch.1": "failChunk=-1",
                "app.apabatch.3": "failTasklet=false",
                "run.id(double)": "2.0",
                "app.apabatch.0": "message=test_lanzamiento_scdf"
            },
            "jobParametersString": "a=c\napp.apabatch.2=sleepTime\\=0\napp.apabatch.1=failChunk\\=-1\napp.apabatch.3=failTasklet\\=false\nrun.id(double)=2.0\napp.apabatch.0=message\\=test_lanzamiento_scdf\n",
            "restartable": false,
            "abandonable": false,
            "stoppable": true,
            "defined": true,
            "timeZone": "UTC",
            "schemaTarget": "boot3"
        }
    ],

Here in the JSON, you can see the data that it returns, but the same query against the /jobs/executions endpoint returns the correct data for the jobExecution:

{
    "executionId": 2,
    "stepExecutionCount": 3,
    "jobId": 2,
    "taskExecutionId": 5,
    "name": "customerJob",
    "startDate": "2023-09-25",
    "startTime": "08:45:43",
    "duration": "00:00:01",
    "jobExecution": {
        "id": 2,
        "version": 2,
        "jobParameters": {
            "parameters": {
                "a": {
                    "identifying": true,
                    "value": "c",
                    "type": "STRING"
                },
                "app.apabatch.2": {
                    "identifying": true,
                    "value": "sleepTime=0",
                    "type": "STRING"
                },
                "run.id": {
                    "identifying": true,
                    "value": 2.0,
                    "type": "DOUBLE"
                },
                "app.apabatch.1": {
                    "identifying": true,
                    "value": "failChunk=-1",
                    "type": "STRING"
                },
                "app.apabatch.3": {
                    "identifying": true,
                    "value": "failTasklet=false",
                    "type": "STRING"
                },
                "app.apabatch.0": {
                    "identifying": true,
                    "value": "message=test_lanzamiento_scdf",
                    "type": "STRING"
                }
            }
        },
        "jobInstance": {
            "id": 2,
            "jobName": "customerJob",
            "version": null
        },
        "stepExecutions": [
            {
                "stepName": "customerCollectorChunkStep",
                "id": 2,
                "version": 3,
                "status": "COMPLETED",
                "readCount": 22,
                "writeCount": 22,
                "commitCount": 1,
                "rollbackCount": 0,
                "readSkipCount": 0,
                "processSkipCount": 0,
                "writeSkipCount": 0,
                "startTime": "2023-09-25T08:45:43.589+0000",
                "endTime": "2023-09-25T08:45:44.193+0000",
                "lastUpdated": "2023-09-25T08:45:44.196+0000",
                "executionContext": {
                    "dirty": false,
                    "empty": true,
                    "values": []
                },
                "exitStatus": {
                    "exitCode": "COMPLETED",
                    "exitDescription": ""
                },
                "terminateOnly": false,
                "filterCount": 0,
                "failureExceptions": [],
                "jobParameters": {
                    "parameters": {
                        "a": {
                            "identifying": true,
                            "value": "c",
                            "type": "STRING"
                        },
                        "app.apabatch.2": {
                            "identifying": true,
                            "value": "sleepTime=0",
                            "type": "STRING"
                        },
                        "run.id": {
                            "identifying": true,
                            "value": 2.0,
                            "type": "DOUBLE"
                        },
                        "app.apabatch.1": {
                            "identifying": true,
                            "value": "failChunk=-1",
                            "type": "STRING"
                        },
                        "app.apabatch.3": {
                            "identifying": true,
                            "value": "failTasklet=false",
                            "type": "STRING"
                        },
                        "app.apabatch.0": {
                            "identifying": true,
                            "value": "message=test_lanzamiento_scdf",
                            "type": "STRING"
                        }
                    }
                },
                "jobExecutionId": 2,
                "summary": "StepExecution: id=2, version=3, name=customerCollectorChunkStep, status=COMPLETED, exitStatus=COMPLETED, readCount=22, filterCount=0, writeCount=22 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=1, rollbackCount=0",
                "skipCount": 0
            },
            {
                "stepName": "customerTaskletStep",
                "id": 3,
                "version": 3,
                "status": "COMPLETED",
                "readCount": 0,
                "writeCount": 0,
                "commitCount": 1,
                "rollbackCount": 0,
                "readSkipCount": 0,
                "processSkipCount": 0,
                "writeSkipCount": 0,
                "startTime": "2023-09-25T08:45:44.285+0000",
                "endTime": "2023-09-25T08:45:44.376+0000",
                "lastUpdated": "2023-09-25T08:45:44.377+0000",
                "executionContext": {
                    "dirty": false,
                    "empty": true,
                    "values": []
                },
                "exitStatus": {
                    "exitCode": "COMPLETED",
                    "exitDescription": ""
                },
                "terminateOnly": false,
                "filterCount": 0,
                "failureExceptions": [],
                "jobParameters": {
                    "parameters": {
                        "a": {
                            "identifying": true,
                            "value": "c",
                            "type": "STRING"
                        },
                        "app.apabatch.2": {
                            "identifying": true,
                            "value": "sleepTime=0",
                            "type": "STRING"
                        },
                        "run.id": {
                            "identifying": true,
                            "value": 2.0,
                            "type": "DOUBLE"
                        },
                        "app.apabatch.1": {
                            "identifying": true,
                            "value": "failChunk=-1",
                            "type": "STRING"
                        },
                        "app.apabatch.3": {
                            "identifying": true,
                            "value": "failTasklet=false",
                            "type": "STRING"
                        },
                        "app.apabatch.0": {
                            "identifying": true,
                            "value": "message=test_lanzamiento_scdf",
                            "type": "STRING"
                        }
                    }
                },
                "jobExecutionId": 2,
                "summary": "StepExecution: id=3, version=3, name=customerTaskletStep, status=COMPLETED, exitStatus=COMPLETED, readCount=0, filterCount=0, writeCount=0 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=1, rollbackCount=0",
                "skipCount": 0
            },
            {
                "stepName": "customerViabilityChunkStep",
                "id": 4,
                "version": 3,
                "status": "COMPLETED",
                "readCount": 22,
                "writeCount": 22,
                "commitCount": 1,
                "rollbackCount": 0,
                "readSkipCount": 0,
                "processSkipCount": 0,
                "writeSkipCount": 0,
                "startTime": "2023-09-25T08:45:44.405+0000",
                "endTime": "2023-09-25T08:45:44.593+0000",
                "lastUpdated": "2023-09-25T08:45:44.593+0000",
                "executionContext": {
                    "dirty": false,
                    "empty": true,
                    "values": []
                },
                "exitStatus": {
                    "exitCode": "COMPLETED",
                    "exitDescription": ""
                },
                "terminateOnly": false,
                "filterCount": 0,
                "failureExceptions": [],
                "jobParameters": {
                    "parameters": {
                        "a": {
                            "identifying": true,
                            "value": "c",
                            "type": "STRING"
                        },
                        "app.apabatch.2": {
                            "identifying": true,
                            "value": "sleepTime=0",
                            "type": "STRING"
                        },
                        "run.id": {
                            "identifying": true,
                            "value": 2.0,
                            "type": "DOUBLE"
                        },
                        "app.apabatch.1": {
                            "identifying": true,
                            "value": "failChunk=-1",
                            "type": "STRING"
                        },
                        "app.apabatch.3": {
                            "identifying": true,
                            "value": "failTasklet=false",
                            "type": "STRING"
                        },
                        "app.apabatch.0": {
                            "identifying": true,
                            "value": "message=test_lanzamiento_scdf",
                            "type": "STRING"
                        }
                    }
                },
                "jobExecutionId": 2,
                "summary": "StepExecution: id=4, version=3, name=customerViabilityChunkStep, status=COMPLETED, exitStatus=COMPLETED, readCount=22, filterCount=0, writeCount=22 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=1, rollbackCount=0",
                "skipCount": 0
            }
        ],
        "status": "COMPLETED",
        "startTime": "2023-09-25T08:45:43.492+0000",
        "createTime": "2023-09-25T08:45:43.295+0000",
        "endTime": "2023-09-25T08:45:44.684+0000",
        "lastUpdated": "2023-09-25T08:45:44.685+0000",
        "exitStatus": {
            "exitCode": "COMPLETED",
            "exitDescription": ""
        },
        "executionContext": {
            "dirty": false,
            "empty": true,
            "values": []
        },
        "failureExceptions": [],
        "jobConfigurationName": null,
        "allFailureExceptions": []
    },
    "jobParameters": {
        "a": "c",
        "app.apabatch.2": "sleepTime=0",
        "app.apabatch.1": "failChunk=-1",
        "app.apabatch.3": "failTasklet=false",
        "run.id(double)": "2.0",
        "app.apabatch.0": "message=test_lanzamiento_scdf"
    },
    "jobParametersString": "a=c\napp.apabatch.2=sleepTime\\=0\napp.apabatch.1=failChunk\\=-1\napp.apabatch.3=failTasklet\\=false\nrun.id(double)=2.0\napp.apabatch.0=message\\=test_lanzamiento_scdf\n",
    "restartable": false,
    "abandonable": false,
    "stoppable": false,
    "defined": true,
    "timeZone": "UTC",
    "schemaTarget": "boot3",
@github-actions github-actions bot added the status/need-triage Team needs to triage and take a first look label Sep 27, 2023
@cppwfs cppwfs added type/bug Is a bug report and removed status/need-triage Team needs to triage and take a first look labels Sep 28, 2023
@cppwfs cppwfs added this to the 2.11.1 milestone Sep 28, 2023
@corneil corneil self-assigned this Oct 3, 2023
@onobc onobc modified the milestones: 2.11.1, 2.11.2 Oct 3, 2023
corneil added a commit that referenced this issue Nov 22, 2023
@corneil
Ensure sub-query use for Job instances.
1bf1052 
 Fix find by name query ordering.
 Fix find by name not found query.
 Improve test case readability.
 #5484
This was referenced Nov 22, 2023
Merged
Closed
corneil added a commit that referenced this issue Nov 23, 2023
@corneil
Ensure sub-query use for Job instances. (#5559)
0af69dd 
Fix find by name query ordering.
 Fix find by name not found query.
 Improve test case readability.
 #5484
@corneil corneil mentioned this issue Nov 23, 2023
Merged
corneil added a commit that referenced this issue Nov 28, 2023
@corneil
Ensure sub-query use for Job instances. (#5563)
90cd253 
* Ensure sub-query use for Job instances.
 Fix find by name query ordering.
 Fix find by name not found query.
 Improve test case readability.

* Fix imports and spacing.
Removed .andDo(print())

* Fixed wildcard imports.
 #5484
@corneil corneil closed this as completed Dec 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@corneil corneil

Labels
type/bug Is a bug report
Projects
None yet
Milestone
2.11.2
Development

No branches or pull requests
4 participants
@corneil @cppwfs @khaeghar @onobc