DevOps Publish Test Results Task Fails #1

carllp · 2019-12-05T15:28:37Z

Hello,

Receiving the following error in the Publish Test Results task after successfully running both Bash scripts as found on your reporting documentation (https://marketplace.visualstudio.com/items?itemName=CSE-DevOps.zap-scanner). Test results are not shown on the test tab even though the task is marked as having succeeded.:

2019-12-05T14:36:03.5747957Z ##[debug]OS type: Linux
2019-12-05T14:36:04.4503434Z ##[debug]Reading test results from file '/home/vsts/work/r1/a/owaspzap/test-results.xml'
2019-12-05T14:36:04.4569568Z ##[warning]Failed to read /home/vsts/work/r1/a/owaspzap/test-results.xml. Error : Data at the root level is invalid. Line 22, position 4..
2019-12-05T14:36:04.4591686Z ##[warning]Failed to publish test results: Object reference not set to an instance of an object.
2019-12-05T14:36:04.4717111Z ##[debug]Processed: ##vso[results.publish type=NUnit;mergeResults=false;publishRunAttachments=true;resultFiles=/home/vsts/work/r1/a/owaspzap/test-results.xml;failTaskOnFailedTests=false;testRunSystem=VSTS - PTR;]
2019-12-05T14:36:04.4718265Z ##[debug]task result: Succeeded
2019-12-05T14:36:04.4718634Z ##[debug]Processed: ##vso[task.complete result=Succeeded;]

As a possible alternative solution if handlebars is causing issues, perhaps outputting the test results file as an .xml and then using the template found on this azuredevblog might work?

https://devblogs.microsoft.com/premier-developer/azure-devops-pipelines-leveraging-owasp-zap-in-the-release-pipeline/

Thanks for your time! Appreciate your work.

ExpressDead · 2019-12-05T15:49:13Z

This doesn't look like a problem related to the handlebars template. The error is related to the scanner not being able to publish the result files. Normally this happens on a self-hosted agent that has a version of docker installed via the SNAP package manager. Are you by any chance running a similar self-hosted agent @carllp ?

carllp · 2019-12-06T06:59:41Z

@doymturner The agent the task is being run on is the Ubuntu 18.04 Microsoft-hosted agent.

ExpressDead · 2019-12-06T07:04:10Z

Can you share your full pipeline (cleansed)? At least all of the bits related to zap scanner?

carllp · 2019-12-06T07:57:07Z

tasklog_12.log

Sure. I believe it's been sufficiently cleansed of all unique identifiers.

As a side note, if you could point me to some documentation (via DM or here) to speed up the cleansing of pipeline logs it would be much appreciated.

ExpressDead · 2019-12-11T17:53:41Z

Took a second look at the original error you posted. Did you make any changes to the handlebar template when adding it to your pipeline? Could you share the bits of your pipeline YAML related to the extension and report publishing (including the template)?

FYI: We are working on additional reporting options for the extension, but plan to leave the options for custom reporting such as with the handlebars template so I'm keen to figure out the issue.

carllp · 2019-12-16T09:28:25Z

Sorry for the late reply @doymturner.

Fixed quoted text below by including only the bash scripting code and not displayName and condition lines. Please ignore.

We did edit the bash script convert report.json to test-results.xml slightly as it was throwing errors (see log below). I assume that this is the culprit as to why the nunit test report doesn't work. The two removed lines work fine in the handlebars template script so we're not sure why they throw errors in this step.

We removed the following two lines so that the conversion would run:
displayName: 'owasp nunit template'
condition: always()

The error report if the lines are included:

=========================================================
Generating script.
========================== Starting Command Output ====
/home/vsts/work/_temp/: line 2: displayName:: command not found
/home/vsts/work/_temp/: line 3: syntax error near unexpected token (' /home/vsts/work/_temp/: line 3: condition: always()'

##[error]Bash exited with code '2'.
##[section]Finishing: Bash Script - Convert report.json to test-results.xml

Here's how it looks in our pipeline. If you know what's syntactically wrong, please let me know.

Appreciate your help with this.

carllp · 2019-12-16T09:31:32Z

As requested, 1st snippet is the handlebars YAML and the 2nd is the report conversion. 3rd is the Publish Test Results YAML.

steps:

bash: |
sudo npm install -g handlebars-cmd

cat <<EOF > owaspzap/nunit-template.hbs
{{#each site}}

<test-run
    id="2"
    name="Owasp test"
    start-time="{{../[@generated]}}"  >
    <test-suite
        id="{{@index}}"
        type="Assembly"
        name="{{[@name]}}"
        result="Failed"
        failed="{{alerts.length}}">
        <attachments>
            <attachment>
                <filePath>owaspzap/report.html</filePath>
            </attachment>
        </attachments>
    {{#each alerts}}<test-case
        id="{{@index}}"
        name="{{alert}}"
        result="Failed"
        fullname="{{alert}}"
        time="1">
            <failure>
                <message>
                    <![CDATA[{{{desc}}}]]>
                </message>
                <stack-trace>
                    <![CDATA[
Solution:
{{{solution}}}

Reference:
{{{reference}}}

instances:{{#each instances}}
* {{uri}}
    - {{method}}
    {{#if evidence}}- {{{evidence}}}{{/if}}
                    {{/each}}]]>
                </stack-trace>
            </failure>
    </test-case>
    {{/each}}
    </test-suite>
</test-run>
{{/each}}
EOF

displayName: 'Bash Script - Install handlebars and generate report template'

steps:

bash: |
handlebars owaspzap/report.json < owaspzap/nunit-template.hbs > owaspzap/test-results.xml

displayName: 'Bash Script - Convert report.json to test-results.xml'

steps:

task: PublishTestResults@2
displayName: 'Publish Test Results owaspzap/test-results.xml'
inputs:
testResultsFormat: NUnit
testResultsFiles: 'owaspzap/test-results.xml'
condition: succeededOrFailed()

carllp · 2019-12-16T10:05:09Z

Error in the publish test results task is slightly different now.

It now shows:
##[warning]Failed to read /home/vsts/work/r1/a/owaspzap/test-results.xml. Error : Data at the root level is invalid. Line 284, position 4..

Instead of
/home/vsts/work/r1/a/owaspzap/test-results.xml. Error : Data at the root level is invalid. Line 22, position 4..

Full log:

` ##[section]Starting: Publish Test Results test-results.xml

Task : Publish Test Results
Description : Publish test results to Azure Pipelines
Version : 2.161.1
Author : Microsoft Corporation
Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/test/publish-test-results

[command]/usr/bin/dotnet --version
3.0.101
##[warning]Failed to read /home/vsts/work/r1/a/owaspzap/test-results.xml. Error : Data at the root level is invalid. Line 284, position 4..
##[warning]Failed to publish test results: Object reference not set to an instance of an object.
##[section]Async Command Start: Publish test results
##[section]Async Command End: Publish test results
##[section]Finishing: Publish Test Results test-results.xml
`

TomaszPiec · 2020-01-31T07:29:49Z

I have the same issue as carllp and I have solved it by creating whole task in yaml instead of DevOps separate components.

Eldorados · 2020-02-19T13:47:42Z

@carllp Take a look in your test-results.xml file (copy it to artifacts so you can download it), it likely has 2 <test-run> nodes which is invalid xml format, and Nunit only allows one per file

I've opened an issue for it as the script itself is written to do this specifically for each site that is scanned, in my case it was http vs https

adamtuliper · 2020-04-18T00:00:03Z

In my case, this was resolved by moving my final EOF over two spaces. In the spacing adjustments in the YAML file it became offset thus causing a problem as shown here. Once it was aligned, the warning didn't show up anymore. To be clear - the task was succeeding but a warning was being output on the PublishTestResults task.

cjazinski · 2020-05-31T20:53:17Z

@carllp Take a look in your test-results.xml file (copy it to artifacts so you can download it), it likely has 2 <test-run> nodes which is invalid xml format, and Nunit only allows one per file

I've opened an issue for it as the script itself is written to do this specifically for each site that is scanned, in my case it was http vs https

So how did you resolve the issue? NM. Thanks!

leparduk · 2020-08-06T13:07:05Z

The sample shows the following

sudo npm install -g handlebars-cmd

cat < owaspzap/nunit-template.hbs

... snip

EOF

When running in a bash task this causes failure. Fix by moving everything from cat to EoF two spaces to the left.

ramesh1983 · 2020-12-03T18:11:56Z

hi
I am getting same issue now.
I have tried all the options but nothing works (yaml,azure devops tasks) , removing EOF and other stuff..
Result Attachments will be stored in LogStore
Run Attachments will be stored in LogStore
##[warning]Failed to read /home/vsts/work/1/s/owaspzap/test-results.xml. Error : Data at the root level is invalid. Line 59, position 4..
Async Command Start: Publish test results
Async Command End: Publish test results
Finishing: Publish Test Results owaspzap/test-results.xml

I am not able to see test tab eventhough build is successful

ramesh1983 · 2020-12-03T18:13:32Z

trigger:

master

pool:
vmImage: 'ubuntu-latest'

steps:

task: owaspzap@1
inputs:
threshold: '150'
scantype: 'targetedScan'
url: 'example.com'
port: '443'
task: CopyFiles@2
inputs:
SourceFolder: 'owaspzap/'
Contents: '**'
TargetFolder: '$(Build.ArtifactStagingDirectory)'
task: PublishBuildArtifacts@1
condition: always()
inputs:
ArtifactName: 'owasp_zap_reports'
bash: |
sudo npm install -g handlebars-cmd

cat < owaspzap/nunit-template.hbs

{{#each site}}

$BUILD_SOURCESDIRECTORY/owaspzap/report.html

{{#each alerts}}

<![CDATA[
Solution:
{{{solution}}}

Reference:
{{{reference}}}

instances:{{#each instances}}
- {{uri}}
  - {{method}}
    {{#if evidence}}- {{{evidence}}}{{/if}}
    {{/each}}]]>
    
    {{/each}}
  {{/each}}
EOF

displayName: 'owasp nunit template'
condition: always()

bash: ' handlebars owaspzap/report.json < owaspzap/nunit-template.hbs > owaspzap/test-results.xml'
displayName: 'generate nunit type file'
condition: always()
task: PublishTestResults@2
inputs:
testResultsFormat: 'Nunit'
testResultsFiles: 'owaspzap/test-results.xml'

ramesh1983 · 2020-12-03T18:14:02Z

Can anyone help me

kocercan · 2021-02-05T09:59:41Z

I've found solution for this, I'll just paste it from marketplace reviews.

Sithu Kyaw
9/23/2020
I tried it just now and it works fine except not able to attach "owaspzap/report.html" for Publish Test Result task.

After a couple of troubleshooting, I found Handlebars script needs minor adjustment by adding $(System.DefaultWorkingDirectory)/ in front of 'owasp/report.html' inside . Good work!


<attachment>
       <filePath>$(System.DefaultWorkingDirectory)/owaspzap/report.html</filePath>
</attachment>

deepak-kannan · 2021-03-19T08:20:35Z

In Step 1 - Remove the EOF statement at the end. Keep only the below code. It will work fine

sudo npm install -g handlebars-cmd

 cat <<EOF > $(System.DefaultWorkingDirectory)/owaspzap/nunit-template.hbs
 {{#each site}}

 <test-run
     id="2"
     name="Owasp test"
     start-time="{{../[@generated]}}"  >
     <test-suite
         id="{{@index}}"
         type="Assembly"
         name="{{[@name]}}"
         result="Failed"
         failed="{{alerts.length}}">
         <attachments>
             <attachment>

$(System.DefaultWorkingDirectory)/owaspzap/report.html

{{#each alerts}}

<![CDATA[
Solution:
{{{solution}}}

 Reference:
 {{{reference}}}

 instances:{{#each instances}}
 * {{uri}}
     - {{method}}
     {{#if evidence}}- {{{evidence}}}{{/if}}
                     {{/each}}]]>
                 </stack-trace>
             </failure>
     </test-case>
     {{/each}}
     </test-suite>
 </test-run>
 {{/each}}

codewode · 2022-02-03T16:34:45Z

I have had the same problem, tried all different possibilites. But always get error file not found..

##[warning]File at 
                  $(Build.SourcesDirectory)/OWASP-ZAP-Report.xml
                 was not found
Failed to upload run logs to LogStore, Trying to upload to FileService
##[warning]Skipping attachment as it is not available on disk: 
                  $(Build.SourcesDirectory)/OWASP-ZAP-Report.xml

also by injecting env variable to xslt


##[warning]File at /home/vsts/work/1/s/OWASP-ZAP-Report.xml
                 was not found
Failed to upload run logs to LogStore, Trying to upload to FileService
##[warning]Skipping attachment as it is not available on disk: /home/vsts/work/1/s/OWASP-ZAP-Report.xml

also added echo to confirm the location of file

echo `pwd`/`ls OWASP-ZAP-Report.xml`

/home/vsts/work/1/s/OWASP-ZAP-Report.xml

mgrandillo · 2022-02-25T18:13:21Z

In my case the pipeline runs successfully. However the Passed test results are not shown in the test report. I only see the failed or others. This is the template I am using.

stages:

stage: OWASP_ZAP_Stage # !CHANGE! per env
jobs:
- job : OWASP_ZAP
  variables:
  websiteurl: 'https://qatest.gdc.net/status'
  pool:
  vmImage: 'ubuntu-latest'
  steps:
  - task: Bash@3
    displayName: 'Prepare OWASPZAP output directory for Docker'
    inputs:
    targetType: 'inline'
    script: |
    sudo mkdir $(System.DefaultWorkingDirectory)/owaspzap
    sudo chmod -R 777 $(System.DefaultWorkingDirectory)/owaspzap
  - task: Bash@3
    displayName: 'Run OWASPZAP Docker Image and scan $(websiteurl)'
    inputs:
    targetType: 'inline'
    script: |
    docker run -v $(System.DefaultWorkingDirectory)/owaspzap:/zap/wrk:rw -t owasp/zap2docker-stable zap-baseline.py -t $(websiteurl) -J report.json -i --autooff
  - bash: |
    sudo npm install -g handlebars-cmd
    
    cat < owaspzap/nunit-template.hbs
    
    {{#each site}}
    
    $(System.DefaultWorkingDirectory)/owaspzap/report.html
    
    {{#each alerts}}
    
    <![CDATA[
    Solution:
    {{{solution}}}
    
    Reference:
    {{{reference}}}
    
    instances:{{#each instances}}
    - {{uri}}
      - {{method}}
        {{#if evidence}}- {{{evidence}}}{{/if}}
        {{/each}}]]>
        
        {{/each}}
      {{/each}}
    displayName: 'OWASP NUnit template'
  - bash: ' handlebars owaspzap/report.json < owaspzap/nunit-template.hbs > owaspzap/test-results.xml'
    displayName: 'Generate NUnit type file for $(websiteurl)'
  Publish results to AZDO Test page
  - task: PublishTestResults@2
    displayName: 'Publish Test Results to AZDO'
    inputs:
    testResultsFormat: NUnit
    testResultsFiles: 'owaspzap/test-results.xml'
    testRunTitle: '$(websiteurl)'
    failTaskOnFailedTests: false
    condition: always()

roman-avseenko · 2022-05-20T14:49:52Z

Hello everyone! I'm don't use this extension for ZAP testing, but got the same issue - unable to upload test results. So my pipeline yaml looks like this:

trigger: none

pr: none

pool:
  vmImage: ubuntu-latest

steps:
- script: docker run --rm --user root -v $(System.DefaultWorkingDirectory):/zap/wrk/:rw -i owasp/zap2docker-stable zap-full-scan.py -d -j -I -t https://blah-blah -J owasp-zap/zap-report.json
  displayName: 'Run dynamic testing'

- bash: |
    sudo npm install -g handlebars-cmd
    handlebars owasp-zap/zap-report.json < owasp-zap/nunit-template.hbs > owasp-zap/test-results.xml
    python3 owasp-zap/splitter.py test-results.xml
  displayName: 'Generate test result files'

- task: PublishTestResults@2
  displayName: 'Publish rest results'
  inputs:
    testResultsFormat: NUnit
    testResultsFiles: 'owasp-zap/test-results-*.xml'
    mergeTestResults: true

So, at the root of my repo i had directory owasp-zap that contains file nunit-template.hbs:

{{#each site}}

<test-run
    id="2"
    name="Owasp test"
    start-time="{{../[@generated]}}"  >
    <test-suite
        id="{{@index}}"
        type="Assembly"
        name="{{[@name]}}"
        result="Failed"
        failed="{{alerts.length}}">
    {{#each alerts}}<test-case
        id="{{@index}}"
        name="{{alert}}"
        result="Failed"
        fullname="{{alert}}"
        time="1">
            <failure>
                <message>
                    <![CDATA[{{{desc}}}]]>
                </message>
                <stack-trace>
                    <![CDATA[
Solution:
{{{solution}}}

Reference:
{{{reference}}}

instances:{{#each instances}}
* {{uri}}
    - {{method}}
    {{#if evidence}}- {{{evidence}}}{{/if}}
                    {{/each}}]]>
                </stack-trace>
            </failure>
    </test-case>
    {{/each}}
    </test-suite>
</test-run>
{{/each}}

And file splitter.py:

import xml.dom.minidom as minidom
import argparse


def parse_file():
    parser = argparse.ArgumentParser(description='File path parser')
    parser.add_argument('-f', '--file', type=str, help='A required string - path to report file.')
    args = parser.parse_args()
    return args.file


def get_xml(document: str) -> list:
    with open(document, 'r') as file:
        return [line for line in file]


def to_file(line: str, file_name: str):
    with open(file_name, 'a') as f:
        f.write(line)


def extract_separate_documents(lines: list):
    count = 0

    for line in lines:
        line: str
        if '<test-run' in line:
            count += 1
            print('report:', count)
        if count > 0:
            to_file(line, f'test-results-{count}.xml')


document = parse_file()

if not document:
    print("Exception! Report file not specified.")
    exit(1)

lines = get_xml(document=document)
extract_separate_documents(lines)

My ZAP testing run generates json file, that should be procced by handlebars tool and I got test-results.xml in owasp-zap dir. In my case test-results.xml file contains two root block that doesn't correspond to NUnit type. Than I wrote a Python script that splits this xml result file on more files named test-results-{count}.xml based on tag. And in PublishTestResults task i provided those files as an input and set mergeTestResults parameter to true. This is working fine in my case and may be suitable for yours.

DariuszPorowski added the OwaspZap label Oct 19, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DevOps Publish Test Results Task Fails #1

DevOps Publish Test Results Task Fails #1

carllp commented Dec 5, 2019

ExpressDead commented Dec 5, 2019

carllp commented Dec 6, 2019

ExpressDead commented Dec 6, 2019

carllp commented Dec 6, 2019

ExpressDead commented Dec 11, 2019

carllp commented Dec 16, 2019 •

edited

Loading

carllp commented Dec 16, 2019 •

edited

Loading

carllp commented Dec 16, 2019

TomaszPiec commented Jan 31, 2020 •

edited

Loading

Eldorados commented Feb 19, 2020 •

edited

Loading

adamtuliper commented Apr 18, 2020

cjazinski commented May 31, 2020 •

edited

Loading

leparduk commented Aug 6, 2020 •

edited

Loading

ramesh1983 commented Dec 3, 2020

ramesh1983 commented Dec 3, 2020

ramesh1983 commented Dec 3, 2020

kocercan commented Feb 5, 2021 •

edited

Loading

deepak-kannan commented Mar 19, 2021

codewode commented Feb 3, 2022

mgrandillo commented Feb 25, 2022

Publish results to AZDO Test page

roman-avseenko commented May 20, 2022

DevOps Publish Test Results Task Fails #1

DevOps Publish Test Results Task Fails #1

Comments

carllp commented Dec 5, 2019

ExpressDead commented Dec 5, 2019

carllp commented Dec 6, 2019

ExpressDead commented Dec 6, 2019

carllp commented Dec 6, 2019

ExpressDead commented Dec 11, 2019

carllp commented Dec 16, 2019 • edited Loading

carllp commented Dec 16, 2019 • edited Loading

carllp commented Dec 16, 2019

TomaszPiec commented Jan 31, 2020 • edited Loading

Eldorados commented Feb 19, 2020 • edited Loading

adamtuliper commented Apr 18, 2020

cjazinski commented May 31, 2020 • edited Loading

leparduk commented Aug 6, 2020 • edited Loading

ramesh1983 commented Dec 3, 2020

ramesh1983 commented Dec 3, 2020

ramesh1983 commented Dec 3, 2020

kocercan commented Feb 5, 2021 • edited Loading

deepak-kannan commented Mar 19, 2021

codewode commented Feb 3, 2022

mgrandillo commented Feb 25, 2022

Publish results to AZDO Test page

roman-avseenko commented May 20, 2022

carllp commented Dec 16, 2019 •

edited

Loading

carllp commented Dec 16, 2019 •

edited

Loading

TomaszPiec commented Jan 31, 2020 •

edited

Loading

Eldorados commented Feb 19, 2020 •

edited

Loading

cjazinski commented May 31, 2020 •

edited

Loading

leparduk commented Aug 6, 2020 •

edited

Loading

kocercan commented Feb 5, 2021 •

edited

Loading