Report displayed incorrectly when displayed non-locally due to CSR problems

[kriegaex]

Is There An Existing Issue

• I have searched the issue tracker for similar problems.

What Are You Seeing

When attaching reports to my Jenkins build, ...

(cannot insert screenshot)

... and clicking one of the HTML file links on the left, e.g. "test-result-IT.html", the report is displayed like this:

(cannot insert screenshot)

Using developer tools in the browser, I see that it reports "blocked (csp)" for 4 different CSS stylesheets:

(cannot insert screenshot)

If I donwload the same HTML file and open it in the browser from my local drive, it works normally.

Steps To Reproduce The Issue

Run Newman from Jenkins

Full Newman Command Or Node Script

newman run mycollection.json --environment my_environment.json --reporters cli,htmlextra --reporter-htmlextra-export test-result-IT.html

HTMLEXTRA Version

latest

Newman Version

6.0.0

Additional Context

x

[github-actions]

Hey @kriegaex: 👋 - Thanks for helping to make the reporter better by raising this ticket. I will take a look as soon as I can and get back to you. Cheers, Danny.

[kriegaex]

For some reason, I see GitHub uploading my screenshots, but then they vanish and are not attached to the issue. I never experienced such a problem with inline screenshots before.

[DannyDainton]

Hey @kriegaex

This sounds similar to an issue that people were seeing a while ago with Jenkins.

Here a link to the solution that one user found:

#45 (comment)

This is more of a Jenkins issue than something specific to the reporter.

[kriegaex]

Yes, that is the same problem, @DannyDainton. Thanks for the hint. It actually seems to be a Jenkins problem, but I am not sure if I can get it resolved within our organisation for all our hundreds of Jenkins instances on OpenShift. For now, I have no alternative URL for my Jenkins instance, which I could use as a resource URL.

If there was an alternative way to optionally generate the reports in a way that would work on any Jenkins instance, or more generically, any server (also non-Jenkins) instance with a CSP (content security policy) in place, by rendering HTML not relying on external CSS but embedding all necessary resources directly into the generated HTML file, even at the expense of bigger files with redundant information, that would be great. I think, you can easily test that by downloading the current Jenkins WAR, running it localy and creating a simple scripted or pipeline Jenkins build, attaching a sample report to the build manually.

That way, you could dramatically enlarge your user base in enterprise environments with CSPs in place and without readily available resource URLs.

[DannyDainton]

Just putting it out there, I'm not going to be spinning up any Jenkins instances to test that.

You could do all that you want to do, in a template file. The reporter runs from a single generic template that presents the data in the way that you seen it in the final report.

The reporter takes in a custom template file using the --reporter-htmlextra-template file.hbs flag.

https://github.com/DannyDainton/newman-reporter-htmlextra/blob/main/lib%2Fdashboard-template.hbs

You could get the raw responses from all the css and js CDNs and paste those directly into a new template file. Then run your Collection's using that.

[kriegaex]

@DannyDainton, Jenkins is just an example for any website sporting a CSP. I just suggested Jenkins, because it has one in place by default, is setup locally in 5 minutes by just running java -jar jenkins.war and going through the rather trivial config assistant. It also happens to be the system where I noticed the problem, i.e. I can easily contribute by re-testing. So I thought, I would suggest that to you as a local testing sandbox. It requires no programming, just starting a program.

I am sure what you said about templates is correct. I have no idea how those templates work, speak next to no CSS and no JS at all. I am a Java guy. Maybe I can get it done with some pointers, but what I was striving for when opening this issue was an improvement for all users who happen to generate reports on sites with CSPs in place. If they could just set a reporting option in newman for your custom reporter, it would be easier. The leverage would be much bigger for you to reach the users than by forcing every single one of them to do what you just suggested to me.

Creating a custom template, IIUC, would also mean that I would have to re-do that after upgrading your plugin to match the external CSS versions you are referencing from there. The thing is, I do not even notice version changes, because npm simply pulls the latest one into my Docker container. It is meant to run in a "touch once, (next to) never change" mode. If you would deliver two templates - the current one and a big one inlining all the other CSS files - and let the user switch using an option, probably it would solve the problem described here and in the other issue you linked to before.

If it is just about the download and 5 minutes to set up Jenkins for you, I will gladly volunteer to test the new version, once it is out or if you can instruct me how to pull a test version via npm. (You are guessing correctly, I am not just a JS noob but a node.js and npm one to boot.)

[DannyDainton]

It wasn't about me not being able to set it up, I've used Jenkins a lot. I just wasn't going to do it for this as I wouldn't be making a change in the reporter for that issue. There are steps to mitigate that which can be done outside of the reporter.

As I mentioned, this is the template file for the reporter

Template file

You can make a copy of this file on your local machine which can be used for every run, on any version. Nothing needs to be updated, it would be your template and your custom report.

At the top of the template are the the style tags with the CDN referenced here:

You can go to each one of those URLs and grab the response and paste it in the file. It will be huge and ugly and you would need to get the correct syntax in order for it to work correctly.

There are script tags at the bottom of the file which are doing a similar thing and have links to the CDNs, you can follow the same method and add the massive response to the template. It's going to make the starting size of the file very big before you have even started pushing data into it - There are already perf issue when reports reach a certain size so i think that is what will happen here too.

[kriegaex]

Like I said, this is something to be done during a tool's build process automatically, not something I want to customise and cut off myself from future updates.

Anyway, I tried manually, and it still does not work as expected, probably due to JS, see here:
https://www.jenkins.io/doc/book/security/configuring-content-security-policy/:

The Default Rule Set

The default rule set results in the following:

• No JavaScript allowed at all
• No plugins (object/embed) allowed
• No inline CSS, or CSS from other sites allowed
• No images from other sites allowed
• No frames allowed
• No web fonts allowed
• No XHR/AJAX allowed
• etc.

Before after almost 1.5 years I left my customer last week - I have no more access to their environment - I already told them to set up a resource root URL for Jenkins, which will solve the problem. I tried locally. I hope they have done that in my absence.

[DannyDainton]

If you have a custom template created, that's always going to render that if you use that in the execute command. Even if the underlying reporter changed or the version was bumped that would step be the same file that would generate your report.

There isn't anything that I'm going to be doing here to the reporter, I'm closing this issue.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.