Results for FSE

[sibat119]

Taken from the issue #13 and Issue #15

We will categorize the crashes here.

Total Crash report by categories:

Absent Packages (Out-of-Scope Modules + Interpreter-related Crashes) : 137 (45, 92)
Playbook Semantics: 30
Inadequate Artifacts: 30
Network: 11

[sibat119]

Crash report from Issue #13

Statistics

Out-of-Scope Modules: 45,
Interpreter-related Crashes: 88,
Inadequate Artifacts: 22,
Network: 11,
Playbook Semantics: 14
Confirm Bug: 8
Total: 188

Full Report

Sample Num	Evaluation	Reason	Crash Type
0	Not a bug	Error comes from not finding a user-provided file name	Inadequate Artifacts
1	Confirmed a bug	field explicitly asks for int but byte string was provided	N/A
2	Not a bug	playbook tries to reference undefined variable	Inadequate Artifacts
3	Not a bug	tries to access undefined host (ubuntu2)	Network
4	Not a bug	insufficient file privileges	Inadequate Artifacts
5	Not a bug	user-provided file name does not exist	Inadequate Artifacts
6	Confirmed a bug	-	N/A
7	Not a bug	user-provided file name does not exist	Inadequate Artifacts
8	Not a bug	user-provided file name does not exist	Inadequate Artifacts
9	May not apply	module documentation requires RESTCONF system service be enabled. This may be what is causing the error.	-
10	Not a bug	user-provided file name does not exist	Inadequate Artifacts
11	Confirmed a bug	socket error	N/A
12	Not a bug	user-provided file name does not exist	Inadequate Artifacts
13	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
14	Not a bug	file not found error	Inadequate Artifacts
15	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
16	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
17	Not a bug	Failure message is like how it is shown in the documentation	-
18	Not a bug	http error	Network
19	Not a bug	http error	Network
20	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
21	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
22	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
23	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
24	Not a bug	previously encountered. requires cpanminus to be installed	Interpreter-related Crashes
25	Not a bug	previously encountered. requires cpanminus to be installed	Interpreter-related Crashes
26	Not a bug	user-provided file name does not exist	Inadequate Artifacts
27	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
28	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
29	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
30	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
31	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
32	Confirmed a bug	module is community.general.ejabberd_user	N/A
33	already reported	same as 32	N/A
34	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
35	Maybe a bug	triggers traceback when pip cannot be found	Interpreter-related Crashes
36	Not a bug	user-provided file name does not exist	Inadequate Artifacts
37	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
38	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
39	Not a bug	user-provided file name does not exist	Inadequate Artifacts
40	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
41	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
42	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
43	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
44	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
45	leaning no but not sure	file not found error but could be due to a specified dependency not being installed	Inadequate Artifacts
46	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
47	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
48	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
49	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
50	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
51	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
52	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
53	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
54	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
55	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
56	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
57	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
58	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
59	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
60	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
61	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
62	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
63	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
64	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
65	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
66	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
67	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
68	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
69	Not a bug	Traceback triggered from read/write permissions	Inadequate Artifacts
70	Not a bug	Traceback triggered from read/write permissions	Inadequate Artifacts
71	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
72	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
73	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
74	Not a bug	user-provided file name does not exist	Inadequate Artifacts
75	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
76	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
77	Not a bug	graceful failure on http connection	Network
78	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
79	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
80	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
81	Maybe a bug	raises KeyError: 'body' but no argument with that name is used. software is not installed and cannot be tested due to it being proprietary.	-
82	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
83	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
84	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
85	Maybe a bug	argument casing causes locale is not available error. this could be fixed by either ignoring case for checking or giving a better error message	-
86	Maybe a bug	module fails catastrophically when locale-gen has not been run on the system first. this error could be avoided by checking for the file /etc/locale.gen first. instead, a vague error is given	Inadequate Artifacts
87	need further inspection	module is made to be used for callback but does raise a weird error	-
88	Confirmed a bug	module error says to check stdout/stderr but could easily inform the user of the connection refused error generated from the traceback	N/A
89	Maybe a bug	same module as 88 but error is different	-
90	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
91	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
92	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
93	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
94	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
95	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
96	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
97	Maybe a bug	getting vague module error but reason could be that required software is not installed	Interpreter-related Crashes
98	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
99	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
100	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
101	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
102	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
103	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
104	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
105	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
106	need further inspection	weird socket-related error	Network
107	maybe a bug	module related to 106. connection refused error but that can only be seen from inspecting the stack trace	Network
108	Not a bug	vague error message: JSONDecodeError. Testing the module further requires an api key	-
109	Not a bug	error message:"MODULE FAILURE- See stdout/stderr for the exact error". hard-to-interpret traceback but further testing requires a subscription to OVH cloud services	-
110	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
111	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
112	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
113	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
114	Not a bug	properly display connection failed message	Network
115	Maybe a bug	error relates to ansible.builtin.pip instead of the target module. error is arguably vague.	Interpreter-related Crashes
116	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
117	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
118	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
119	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
120	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
121	Not a bug	file not found error	Inadequate Artifacts
122	Not a bug	file not found error	Inadequate Artifacts
123	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
124	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
125	Not a bug	reasonable error message	-
126	Maybe a bug	vague error but cannot test further without installing CA Spectrum software	-
127	Not a bug	already determined to not be a bug	-
128	Not a bug	already determined to not be a bug	-
129	Not a bug	already determined to not be a bug	-
130	Not a bug	already determined to not be a bug	-
131	Maybe a bug	vague error message: JSONDecodeError. Testing the module further requires an api key	-
132	Not a bug	proper database connection error message	Inadequate Artifacts
133	Not a bug	proper database connection error message	Inadequate Artifacts
134	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
135	Not a bug	tried to read a file that does not exist	Inadequate Artifacts
136	Not a bug	vague error message. stacktrace shows socket-related error	Network
137	Not a bug	same as 136	Network
138	Not a bug	same as 136	Network
139	Not a bug	same as 136	Network
140	skip	paid service	Out-of-Scope Modules
141	skip	paid service	Out-of-Scope Modules
142	skip	paid service	Out-of-Scope Modules
143	skip	paid service	Out-of-Scope Modules
144	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
145	Not a bug	gracefully failed on not having required python package installed	Interpreter-related Crashes
146	skip	cloud service software	Out-of-Scope Modules
147	skip	cloud service software	Out-of-Scope Modules
148	skip	cloud service software	Out-of-Scope Modules
149	skip	cloud service software	Out-of-Scope Modules
150	skip	cloud service software	Out-of-Scope Modules
151	skip	cloud service software	Out-of-Scope Modules
152	skip	cloud service software	Out-of-Scope Modules
153	skip	cloud service software	Out-of-Scope Modules
154	skip	cloud service software	Out-of-Scope Modules
155	skip	cloud service software	Out-of-Scope Modules
156	skip	cloud service software	Out-of-Scope Modules
157	skip	cloud service software	Out-of-Scope Modules
158	skip	cloud service software	Out-of-Scope Modules
159	skip	cloud service software	Out-of-Scope Modules
160	skip	cloud service software	Out-of-Scope Modules
161	Confirmed a bug	bigmon_chain module	N/A
162	skip	module is for huwawei cloud switches	Out-of-Scope Modules
163	skip	module is for huwawei cloud switches	Out-of-Scope Modules
164	skip	module targets lenovo CNOS devices	Out-of-Scope Modules
165	skip	module targets lenovo CNOS devices	Out-of-Scope Modules
166	skip	module targets lenovo CNOS devices	Out-of-Scope Modules
167	skip	module targets lenovo CNOS devices	Out-of-Scope Modules
168	skip	module targets lenovo CNOS devices	Out-of-Scope Modules
169	skip	module targets lenovo CNOS devices	Out-of-Scope Modules
170	skip	module targets lenovo CNOS devices	Out-of-Scope Modules
171	skip	module targets lenovo CNOS devices	Out-of-Scope Modules
172	skip	module targets cisco FTD devices	Out-of-Scope Modules
173	skip	module targets Itential Automation Platform	Out-of-Scope Modules
174	skip	module targets Ruckus ICX devices	Out-of-Scope Modules
175	skip	module targets Ruckus ICX devices	Out-of-Scope Modules
176	skip	module targets Ruckus ICX devices	Out-of-Scope Modules
177	skip	module targets Ruckus ICX devices	Out-of-Scope Modules
178	skip	module targets Cisco switches	Out-of-Scope Modules
179	skip	module targets virtual routers	Out-of-Scope Modules
180	skip	module targets virtual routers	Out-of-Scope Modules
181	skip	module targets virtual routers	Out-of-Scope Modules
182	skip	module targets virtual routers	Out-of-Scope Modules
183	skip	module targets virtual routers	Out-of-Scope Modules
184	skip	module targets virtual routers	Out-of-Scope Modules
185	skip	module targets virtual routers	Out-of-Scope Modules
186	skip	module targets virtual routers	Out-of-Scope Modules
187	skip	module targets virtual routers	Out-of-Scope Modules

[sibat119]

TELER based.

Out-of-Scope Modules: 0,
Interpreter-related Crashes: 3,
Inadequate Artifacts: 2,
Network: 0,
Playbook Semantics: 2,
Confirm Bug: 0
Total: 7

index	id	title	comment	Crash Type
0	-	-	Not a bug. Module not installed in python interpreter hence causing crash	Interpreter-related Crashes
1	-	-	Not a bug. The code introduces divide by zero error.	Playbook Semantics
2	-	-	Not a bug. Proper Module is not installed.	Interpreter-related Crashes
3	-	-	Not a bug. The code gives non existent file path and invalid conditional check.	Inadequate Artifacts
4	-	-	Not a bug. Provides undefined variable.	Playbook Semantics
5	-	-	Not a bug. Setuptools Module not installed.	Interpreter-related Crashes
6	-	-	Not a bug. Invalid filepath given.	Inadequate Artifacts

[sibat119]

Latent Bugs from #12

Total Latent Bugs to Report in FSE: 9

No.	Repo	issue ID	status	issue URL	Heuristic Category
1	Community General	6949	Fixed	community.general.ejabberd_user	Type
2	Ansible Core / Builtin	81463	Fixed	ansible.builtin.debug	Type
3	Ansible Core / Builtin	81349	Fixed	ansible.builtin.script	Type
4	Community General	7207	Fixed	community.general.blockinfile	State Inquiry
5	Community Network	554	In-Progress	community.network.bigmon_chain	Type
6	Ansible Netcommon	582	In-Progress	ansible.netcommon.telnet	Type
7	Community General	7122	In-Progress	community.general.mail	Type
8	Community General	7121	Rejected/Not-a-Bug	community.general.locale_gen	Type
9	Ansible Core / Builtin	81647	Rejected/Not-a-Bug	ansible.builtin.set_facts	Inventory

---

The following bugs will not be reported in FSE.

Repo	issue ID	status	issue URL
Community Docker	681	In-Progress (This issue was originated using Level 3 prompt so, we will not count this one)	community.docker.docker_image
Ansible Core / Builtin	81767	In-Progress (This issue is resubmission of the issue: 7207. As the same code and output was used in the bug report, we will not consider this as a latent bug.)	ansible.builtin.blockinfile
Ansible Core / Builtin	81648	Rejected/Not-a-Bug (This issue was originated using Level 3 prompt so, we will not count this one)	ansible.builtin.debug

[sibat119]

Non-Type Heuristics

Out-of-Scope Modules: 0,
Interpreter-related Crashes: 1,
Inadequate Artifacts: 6,
Network: 0,
Playbook Semantics: 14,
Confirm Bug: 2
Total: 23

Traceback Filtered for Lv2:

Sample No.	Issue No.	Issue Title	Comment	Level	Evaluation	Crash Type
2	37022	`nxos_config idempotency issue with FEX on Nexus 5K	Ansible have command: shell: "echo {{ 1/0 }}" which throws divide by zero error. Not a Bug	2	Not a Bug	Playbook Semantics
4	4709	`lookup error exception in /jinja2/environment.py" for apt_key module since last weeks commits	Covered in conditional bug 4	2	Latent Bug	N/A
6	8552	`ec2_vpc issues with boto 2.32.x	the code produces Divide by zero error. Not a Bug.	2	Not a Bug	Playbook Semantics
16	22261	`symlinks to bin/ansible with link name like 'ansible-3.6' cause errors	invalid template path given. Not a bug.	2	Not a Bug	Inadequate Artifacts
20	37267	`EC2 NoSuchVersion error when running elb_target_group or elb_application_lb modules	Not a Bug: provides non existent group name.	2	Not a Bug	Inadequate Artifacts
23	42501	`s3_bucket error with Minio - GetBucketVersioning NotImplemented	Not a bug: The playbook includes code that trigger divide by zero error.	2	Not a Bug	Playbook Semantics
24	47371	`SBC reboot time incorrectly parsed when TZ is set	Not a bug: The playbook includes code that trigger divide by zero error.	2	Not a Bug	Playbook Semantics
28	59147	`GCP Inventory plugin error when service_account_email is not set.	Not a Bug: The notebook have multiline content which it tries to save into a destination file. During the execution of this task the compiler produces PermissionError because the file already exists with another owner.	2	Not a Bug	Inadequate Artifacts
31	7204	`playbook hangs when --ask-sudo-pass not supplied and is required (vs error msg raised) [edited description]	Not a bug: uses apt for incorrect Linux distribution.	2	Not a Bug	Playbook Semantics
37	15926	`bug: included nested block with conditions results in unexpeced tasks execution	Not a Bug: set fact introduces a divide by zero error.	2	Not a Bug	Playbook Semantics
38	19672	`Ansible:Synchronize module throwing UNREACHABLE! error	Not a Bug: the playbook uses a non-existing file which throws AnsibleFileNotFound Exception.	2	Not a Bug	Inadequate Artifacts
40	33693	`Variable undefined error when import_role is used within a child play	Maybe a bug: list index out of range error. might not be a bug but i'm not sure how to read the jinja2 syntax here.	2	Not a Bug	**Playbook Semantics
41	41129	`npm module suddenly changed default behavior	Not a bug: reasonable error message: couldn't locate file	2	Not a Bug	Inadequate Artifacts
46	73959	``uri` module fails with import error	Maybe a bug: exception encountered during another exception	2	Not a Bug	**Playbook Semantics
47	5445	Segmentation fault when running apt update-cache=yes` on OSX with 1.4.3	Not a bug: proper response from apt module	2	Not a Bug	Playbook Semantics
48	7074	`authorized_key module not reporting error when path to key is invalid	Not a bug: division by 0 but error message seems reasonable	2	Not a Bug	Playbook Semantics
49	7322	`apt_repository module error and usage of proxies when behind a firewall	Not a bug: proper response from apt module	2	Not a Bug	Playbook Semantics
52	22119	`hashi_vault returns Fatal error when the secret does not exist	Not a bug: issue and fix identified by error message	2	Not a Bug	Playbook Semantics
58	41738	`firewalld module does not permanently set an interface zone on CentOS / RedHat due to upstream bugs	Maybe a minor bug: error message shows permission denied when trying to write to a protected directory. it is also classified as an 'unhandled exception' which may be unnecessary	2	Not a Bug	Inadequate Artifacts
64	68809	`SSL issue running fortios_firewall_adress	Not a bug: tries to call command 'localhost'	2	Not a Bug	Playbook Semantics
69	29529	`dellos9_command ansible hangs after reload command issued to remote device.	Not a bug: division by 0 but error message seems reasonable	2	Not a Bug	Playbook Semantics
73	36444	`cloud:azure: error while using azure_rm_networkinterface for creating a vnic without public IP	Not a bug: failed module import	2	Not a Bug	Interpreter-related Crashes
77	71444	`iptables comment parameter incorrectly positioned in iptables	Maybe a bug: exception encountered during another exception	2	Latent Bug	N/A