Initial commit for plugin internals refactor #763

Schamper · 2024-07-25T14:46:04Z

Initial commit for the plugin internals refactor. This removes a lot of slow and hard to understand code with (hopefully) faster and easier to understand code. Also fixes some consistency issues I came across while working on this.

There are some things I want to add, but I may do those in a separate PR:

Re-add support for Nested namespaces do not work #758 (it was lost in between version 2 and 3)
Add support for "plugin directories", similar to _os.py, but then _plugin.py or something similar (Add support for plugin directories #788)
Add helper functions for getting runtime information from functions, e.g. the class object, the record descriptor objects, etc. This will finally allow us to query plugins by e.g. ones that generate records that have a path field, for example.

This should also cover #759.

Closes #889.

codecov · 2024-07-25T14:53:45Z

Codecov Report

Attention: Patch coverage is 85.64516% with 89 lines in your changes missing coverage. Please review.

Project coverage is 75.50%. Comparing base (6968253) to head (e830c4e).

Files	Patch %	Lines
dissect/target/plugin.py	88.73%	41 Missing ⚠️
dissect/target/target.py	74.64%	18 Missing ⚠️
dissect/target/tools/query.py	72.72%	12 Missing ⚠️
dissect/target/helpers/docs.py	85.36%	6 Missing ⚠️
dissect/target/plugins/general/plugins.py	86.84%	5 Missing ⚠️
dissect/target/tools/build_pluginlist.py	0.00%	3 Missing ⚠️
dissect/target/tools/utils.py	81.25%	3 Missing ⚠️
dissect/target/plugins/os/windows/registry.py	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #763      +/-   ##
==========================================
- Coverage   75.51%   75.50%   -0.02%     
==========================================
  Files         305      305              
  Lines       26348    26295      -53     
==========================================
- Hits        19897    19853      -44     
+ Misses       6451     6442       -9

Flag	Coverage Δ
unittests	`75.50% <85.64%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

pyrco

I noticed plugins/general/example.py and plugins/os/unix/packagemanager.py still have a __findable__ property, which is now no longer needed.

Another thing I noticed is that the -l no longer lists the plugins from _os.py.

And a third thing I noticed that if I run a namespace plugin, e.g. webserver.access, which is listed when doing target-query MyTarget -l, before it printed an error like:

Error while parsing sysvol/windows/system32/inetsrv/config/applicationHost.config: ...

but now it errors with:

target-query: error: argument -f/--function contains invalid plugin(s): webserver.access

pyrco · 2024-11-11T16:07:20Z

dissect/target/plugin.py

+        arglist = getattr(obj, "__args__", [])
+        arglist.append((args, kwargs))


Suggested change

arglist = getattr(obj, "__args__", [])

arglist.append((args, kwargs))

obj.__args__.append((args, kwargs))

This can be simplified similar as in alias()

pyrco · 2024-11-11T16:09:09Z

dissect/target/plugin.py

@@ -182,7 +293,6 @@ class attribute. Namespacing results in your plugin needing to be prefixed

    With the following three being assigned in :func:`register`:


Suggested change

With the following three being assigned in :func:`register`:

With the following two being assigned in :func:`register`:

pyrco · 2024-11-25T11:42:41Z

dissect/target/plugin.py

-            The hostname as string.
-        """
-        raise NotImplementedError
+                    members: dict[str, list] = function_index.setdefault(name, {})


Suggested change

members: dict[str, list] = function_index.setdefault(name, {})

members: dict[str, FunctionDescriptor] = function_index.setdefault(name, {})

pyrco · 2024-11-25T12:34:27Z

dissect/target/plugin.py

+    # Function descriptor lookup
+    # {"<function_name>": {"<module_path>": FunctionDescriptor}}
+    "__functions__": {},


From the code it looks like this has the same None/__os__/__child__ subdivision as the __plugins__ key.

pyrco · 2024-11-25T16:02:31Z

dissect/target/plugin.py

-    DefaultPlugin, then plugins are returned as if no ``osfilter`` was
-    specified.
+    registry = _get_plugins()
+    __functions__: dict[str, dict[str, FunctionDescriptor]] = registry.get("__functions__", {})


Suggested change

__functions__: dict[str, dict[str, FunctionDescriptor]] = registry.get("__functions__", {})

__functions__: dict[str, dict[str, dict[str, FunctionDescriptor]]] = registry.get("__functions__", {})

There is an other None / __os__ / __child__ level

pyrco · 2024-11-25T16:14:56Z

dissect/target/plugin.py

-    """Retrieve all child plugin descriptors."""
-    yield from plugins(special_keys={"_child"}, only_special_keys=True)
+                # Skip plugins that don't want to be found by wildcards
+                if not descriptor or not descriptor.exported or (not show_hidden and not descriptor.findable):


Suggested change

if not descriptor or not descriptor.exported or (not show_hidden and not descriptor.findable):

if not descriptor.exported or (not show_hidden and not descriptor.findable):

I don't think it is possible for _generate_long_paths() to give back an entry without a descriptor.

pyrco · 2024-11-25T16:16:31Z

dissect/target/plugin.py

+def _generate_long_paths() -> dict[str, FunctionDescriptor]:
+    """Generate a dictionary of all long paths to their function descriptors."""
+    paths = {}
+    for value in _get_plugins().get("__functions__", {}).get(None, {}).values():


Shouldn't the functions for __os__ not also be added? Otherwise they will not be found by a non-exact match in find_plugin_functions().

pyrco · 2024-11-25T16:18:22Z

dissect/target/plugin.py

+    result = []
+    seen = set()
+    for descriptor in descriptors:
+        print(descriptor)


Suggested change

print(descriptor)

debug left over

pyrco · 2024-11-25T16:29:51Z

dissect/target/plugin.py

+    return list(descriptors)
+
+
+# Class for specific types of plugins


Suggested change

# Class for specific types of plugins

# Classes for specific types of plugins

pyrco · 2024-11-26T10:17:44Z

tests/test_plugin.py

@@ -265,7 +326,7 @@ def test_incompatible_plugin(target_bare: Target) -> None:
        target_bare.add_plugin(_TestIncompatiblePlugin)


-MOCK_PLUGINS = {
+MOCK_PLUGINS_OLD = {


As this is no longer used, it can be removed I think?

Schamper force-pushed the refactor-plugin-registration branch from bfa1029 to f4c1af4 Compare July 25, 2024 14:50

Schamper added plugin Related to a plugin core Related to a core component of dissect.target and removed plugin Related to a plugin labels Aug 5, 2024

Schamper force-pushed the refactor-plugin-registration branch 2 times, most recently from 68299b4 to 84e83ea Compare August 5, 2024 15:11

Horofic mentioned this pull request Aug 7, 2024

Add Windows Jumplist plugin #669

Merged

Schamper force-pushed the refactor-plugin-registration branch 3 times, most recently from 2192994 to 85f4ebc Compare August 8, 2024 13:37

This was referenced Aug 12, 2024

Refactor target-shell #812

Merged

Overview of unique plugins instead of available (duplicate) plugins #822

Closed

Schamper mentioned this pull request Aug 27, 2024

Improve DPAPI plugin #711

Merged

Schamper force-pushed the refactor-plugin-registration branch from 85f4ebc to 43b0585 Compare August 27, 2024 12:27

Initial commit for plugin internals refactor

e830c4e

Schamper force-pushed the refactor-plugin-registration branch from 43b0585 to e830c4e Compare August 27, 2024 14:56

JSCU-CNI mentioned this pull request Sep 25, 2024

Add output_def to PluginFunction #759

Closed

pyrco requested changes Nov 26, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial commit for plugin internals refactor #763

Initial commit for plugin internals refactor #763

Schamper commented Jul 25, 2024 •

edited

Loading

codecov bot commented Jul 25, 2024 •

edited

Loading

pyrco left a comment

pyrco Nov 11, 2024

pyrco Nov 11, 2024

pyrco Nov 25, 2024

pyrco Nov 25, 2024

pyrco Nov 25, 2024

pyrco Nov 25, 2024

pyrco Nov 25, 2024

pyrco Nov 25, 2024

pyrco Nov 25, 2024

pyrco Nov 26, 2024

		arglist = getattr(obj, "__args__", [])
		arglist.append((args, kwargs))

		@@ -182,7 +293,6 @@ class attribute. Namespacing results in your plugin needing to be prefixed

		With the following three being assigned in :func:`register`:

	With the following three being assigned in :func:`register`:
	With the following two being assigned in :func:`register`:

	members: dict[str, list] = function_index.setdefault(name, {})
	members: dict[str, FunctionDescriptor] = function_index.setdefault(name, {})

	__functions__: dict[str, dict[str, FunctionDescriptor]] = registry.get("__functions__", {})
	__functions__: dict[str, dict[str, dict[str, FunctionDescriptor]]] = registry.get("__functions__", {})

	if not descriptor or not descriptor.exported or (not show_hidden and not descriptor.findable):
	if not descriptor.exported or (not show_hidden and not descriptor.findable):

		return list(descriptors)


		# Class for specific types of plugins

	# Class for specific types of plugins
	# Classes for specific types of plugins

Initial commit for plugin internals refactor #763

Are you sure you want to change the base?

Initial commit for plugin internals refactor #763

Conversation

Schamper commented Jul 25, 2024 • edited Loading

codecov bot commented Jul 25, 2024 • edited Loading

Codecov Report

pyrco left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Schamper commented Jul 25, 2024 •

edited

Loading

codecov bot commented Jul 25, 2024 •

edited

Loading