Syncing recent changes (version bump to 3.4.7.1). (#1045)

* FileFinderOS bugfix for handling broken symlinks.
* Further decoupling interrogation from artifact parsers.
* Decoupling RRG protos from GRR protos.
* Updated fleetspeak-client-bin and fleetspeak-server-bin dependencies
  versions to 0.1.13.
* Butmping current version to 3.4.7.1.
* Adding changelog for 3.4.7.1 release.

CHANGELOG.md

@@ -5,21 +5,75 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Planned for removal
+
+Note: GRR release 3.4.7.1 is the **last release** containing the following
+features:
+
+* **Artifact parsers**. ArtifactCollector flow supports parsing collected files
+  and output of executed commands. Its parsers are not properly maintained,
+  are often outdated and fragile. We're going to convert selected parsers
+  into standalone flows and remove the artifact parsing subsystem:
+  the ArtifactCollector will always work as if "apply_parsers" arguments
+  attribute is set to False. Afterwards the "apply_parsers" attribute will be
+  deprecated completely. We will provide documentation on integrating
+  GRR and ArtifactCollector with well-maintained parsing frameworks like
+  [Plaso](https://plaso.readthedocs.io/en/latest/index.html).
+
+* **Built-in cron jobs**. Built-in cron jobs are primarily used for periodic
+  hunts. We will provide documentation on how to easily replicate the
+  current functionality using external scheduling systems (like Linux cron,
+  for example).
+
+  If your workflow depends on GRR built in cron jobs and you anticipate problems
+  when migrating it to external schedulers, please reach out to us via email
+  or GitHub.
+
+* **GRR server Debian package**. We will stop providing the GRR server Debian
+  package as the main way of distributing GRR server and client binaries.
+  Instead we will make GRR Docker image a preferred way for running GRR in a
+  demo or production environment.
+
+If your workflow depends on any of the above, please feel free reach out to
+us via [grr-users](https://groups.google.com/forum/#!forum/grr-users) Google
+Group or [GitHub](https://github.com/google/grr/issues).
+
+## [3.4.7.1] - 2023-10-23
+
 ### Added
 
 * Created a flow for collecting an identifier of the CrowdStrike agent.
+* Podman-based zero-setup development environment.
+* Added StatMultipleFiles and HashMultipleFiles flows to be used in
+  UIv2.
 
 ### Changed
 
 * Renamed AdminUI.new_hunt_wizard.default_output_plugin to
   AdminUI.new_hunt_wizard.default_output_plugins (note the "s" in the end).
   The new option accepts a comma-separated list of names.
+* Newly interrogated clients now pick up active hunts automatically.
+* Hunts workflow is now available in the new UI: creating hunts from a flow,
+  duplicating existing hunts, monitoring hunt progress and inspecting results.
 
 ### Removed
 
 * Fully removed deprecated use_tsk flag.
 * Removed deprecated plugin_args field from OutputPluginDescriptor.
-* Removed deprecated flows: FingerprintFile
+* Removed deprecated flows: FingerprintFile, KeepAlive, FingerprintFile, FindFiles, SendFile, Uninstall,
+  UpdateClient, CollectEfiHashes, DumpEfiImage.
+* Deprecated GetFile flow in favor of MultiGetFile.
+* Made FileFinder an alias to ClientFileFinder, using ClientFileFinder
+  by default everywhere. Legacy FileFinder is still available as
+  LegacyFileFinder. Fixed several inconsistencies in ClientFileFinder
+  client action. Same for RegistryFinder.
+* Removed deprecated client actions: EficheckCollectHashes, EficheckDumpImage, Uninstall, SendFile.
+* Removed "Checks" functionality.
+
+### API removed
+
+* Deprecated no-op "keep_client_alive" attribute in ApiCreateClientApprovalArgs.
+* Deprecated ListClientActionRequests API call (was no-op after Fleetspeak migration).
 
 ## [3.4.6.7] - 2023-03-22
 

grr/client/grr_response_client/client_actions/file_finder.py

@@ -2,6 +2,7 @@
 """The file finder client action."""
 
 import io
+import logging
 from typing import Callable, Iterator, List, Text
 
 from grr_response_client import actions
@@ -124,11 +125,14 @@ def _ValidateMetadata(self, stat, filepath):
     # the client file finder. The legacy file finder was automatically
     # following symlinks to regular files.
     if stat.IsSymlink():
-      target_stat = filesystem.Stat.FromPath(
-          stat.GetPath(), follow_symlink=True
-      )
-      if target_stat.IsRegular():
-        stat = target_stat
+      link_path = stat.GetPath()
+      try:
+        target_stat = filesystem.Stat.FromPath(link_path, follow_symlink=True)
+      except FileNotFoundError:
+        logging.info("Broken link: %s", link_path)
+      else:
+        if target_stat.IsRegular():
+          stat = target_stat
 
     for metadata_condition in self._metadata_conditions:
       if not metadata_condition.Check(stat):
@@ -140,11 +144,14 @@ def _ValidateContent(self, stat, filepath, matches):
       # and the client file finder. The legacy file finder was automatically
       # following symlinks to regular files.
       if stat.IsSymlink():
-        target_stat = filesystem.Stat.FromPath(
-            stat.GetPath(), follow_symlink=True
-        )
-        if not target_stat.IsRegular():
-          raise _SkipFileException()
+        link_path = stat.GetPath()
+        try:
+          target_stat = filesystem.Stat.FromPath(link_path, follow_symlink=True)
+        except FileNotFoundError:
+          logging.info("Broken link: %s", link_path)
+        else:
+          if not target_stat.IsRegular():
+            raise _SkipFileException()
       else:
         raise _SkipFileException()
 

grr/client/grr_response_client/client_actions/file_finder_test.py

@@ -252,6 +252,19 @@ def testFollowLinks(self):
       except OSError:
         pass
 
+  def testBrokenLink(self):
+    """Broken links should be collected when follow_links=False."""
+    with temp.AutoTempDirPath(remove_non_empty=True) as test_dir:
+      link_path = os.path.join(test_dir, "link")
+
+      os.symlink("nowhere", link_path)
+
+      paths = [f"{test_dir}/*"]
+      results = self._RunFileFinder(paths, self.stat_action, follow_links=False)
+      relative_results = self._GetRelativeResults(results, base_path=test_dir)
+
+      self.assertIn("link", relative_results)
+
   def _PrepareTimestampedFiles(self):
     searching_path = os.path.join(self.base_path, "searching")
     test_dir = os.path.join(self.temp_dir, "times_test")

grr/client/setup.py

@@ -137,7 +137,7 @@ def run(self):
         "grr-response-core==%s" % VERSION.get("Version", "packagedepends"),
         "pytsk3==20230125",
         "libfsntfs-python==20230606",
-        "fleetspeak-client-bin==0.1.12",
+        "fleetspeak-client-bin==0.1.13",
     ],
     extras_require={
         # The following requirements are needed in Windows.

grr/client_builder/setup.py

@@ -65,7 +65,7 @@ def make_release_tree(self, base_dir, files):
         "distro==1.7.0",
         "grr-response-client==%s" % VERSION.get("Version", "packagedepends"),
         "grr-response-core==%s" % VERSION.get("Version", "packagedepends"),
-        "fleetspeak-client-bin==0.1.12",
+        "fleetspeak-client-bin==0.1.13",
         "olefile==0.46",
         "PyInstaller==5.8.0",
     ],

grr/core/grr_response_core/config/artifacts.py

@@ -12,9 +12,6 @@
 
 config_lib.DEFINE_list(
     "Artifacts.knowledge_base", [
-        "LinuxReleaseInfo",
-        "LinuxUserProfiles",
-        "WindowsTimezone",
     ], "List of artifacts that are collected regularly by"
     " interrogate and used for interpolation of client-side"
     " variables. Includes artifacts for all supported OSes. "

grr/core/grr_response_core/lib/parsers/linux_release_parser.py

@@ -139,7 +139,10 @@ class LinuxReleaseParser(parsers.MultiFileParser[rdf_protodict.Dict]):
   """Parser for Linux distribution information."""
 
   output_types = [rdf_protodict.Dict]
-  supported_artifacts = ['LinuxReleaseInfo']
+
+  # TODO: The parser has to be invoked explicitly, we should not
+  # relly on magic parsing anymore.
+  supported_artifacts = []
 
   # Multiple files exist to define a Linux distribution, some of which are more
   # accurate than others under certain circumstances. We assign a weight and

grr/core/grr_response_core/lib/rdfvalue.py

@@ -610,12 +610,6 @@ def Lerp(cls, t, start_time, end_time):
 
     return cls(round((1 - t) * start_time._value + t * end_time._value))  # pylint: disable=protected-access
 
-  @classmethod
-  def EarliestDatabaseSafeValue(cls):
-    """Returns the earliest datetime supported by all database backends."""
-    # See https://bugs.mysql.com/77232
-    return cls(1000000)
-
   def __add__(self, other):
     # TODO(hanuszczak): Disallow `float` initialization.
     if isinstance(other, (int, float)):

grr/core/grr_response_core/lib/rdfvalues/structs.py

@@ -1986,6 +1986,9 @@ class RDFProtoStruct(RDFStruct):
   # dependencies setting.
   recorded_rdf_deps = None
 
+  def __init__(self, initializer=None, **kwargs):
+    super().__init__(initializer, **kwargs)
+
   def AsPrimitiveProto(self):
     """Return an old style protocol buffer object."""
     if self.protobuf:

grr/core/grr_response_core/lib/utils.py

@@ -672,30 +672,19 @@ class NotAValue(object):
 class HeartbeatQueue(queue.Queue):
   """A queue that periodically calls a provided callback while waiting."""
 
-  def __init__(self, callback=None, fast_poll_time=60, *args, **kw):
+  def __init__(self, *args, callback=None, **kw):
     queue.Queue.__init__(self, *args, **kw)
     self.callback = callback or (lambda: None)
-    self.last_item_time = time.time()
-    self.fast_poll_time = fast_poll_time
 
   def get(self, poll_interval=5):
     while True:
       try:
-        # Using Queue.get() with a timeout is really expensive - Python uses
-        # busy waiting that wakes up the process every 50ms - so we switch
-        # to a more efficient polling method if there is no activity for
-        # <fast_poll_time> seconds.
-        if time.time() - self.last_item_time < self.fast_poll_time:
-          message = queue.Queue.get(self, block=True, timeout=poll_interval)
-        else:
-          time.sleep(poll_interval)
-          message = queue.Queue.get(self, block=False)
+        message = queue.Queue.get(self, block=True, timeout=poll_interval)
         break
 
       except queue.Empty:
         self.callback()
 
-    self.last_item_time = time.time()
     return message
 
 

grr/proto/grr_response_proto/rrg.proto

@@ -34,6 +34,10 @@ enum Action {
   LIST_USERS = 9;
   // Get the snapshot of the entire filesystem.
   GET_FILESYSTEM_TIMELINE = 10;
+  // List network interfaces available on the system.
+  LIST_INTERFACES = 11;
+  // List filesystem mounts available on the system.
+  LIST_MOUNTS = 12;
 
   // TODO: Define more actions that should be supported.
 

grr/proto/grr_response_proto/rrg/action/list_interfaces.proto

@@ -0,0 +1,14 @@
+// Copyright 2023 Google LLC
+//
+// Use of this source code is governed by an MIT-style license that can be found
+// in the LICENSE file or at https://opensource.org/licenses/MIT.
+syntax = "proto3";
+
+package rrg.action.list_interfaces;
+
+import "grr_response_proto/rrg/net.proto";
+
+message Result {
+  // Information about the individual network interface.
+  rrg.net.Interface interface = 1;
+}

grr/proto/grr_response_proto/rrg/action/list_mounts.proto

@@ -0,0 +1,14 @@
+// Copyright 2023 Google LLC
+//
+// Use of this source code is governed by an MIT-style license that can be found
+// in the LICENSE file or at https://opensource.org/licenses/MIT.
+syntax = "proto3";
+
+package rrg.action.list_mounts;
+
+import "grr_response_proto/rrg/fs.proto";
+
+message Result {
+  // Information about the individual filesystem mount.
+  rrg.fs.Mount mount = 1;
+}

grr/proto/grr_response_proto/rrg/fs.proto

@@ -70,3 +70,13 @@ message FileExtAttr {
     // This can be an arbitrary sequence of bytes both on macOS and Linux.
     bytes value = 2;
 }
+
+// Information about a mounted filesystem.
+message Mount {
+    // Name or other identifier of the mounted device.
+    string name = 1;
+    // Path at which the mounted filesystem is available (a mount point).
+    Path path = 2;
+    // Type of the mounted filesystem (e.g. `ext4`, `ramfs`, `NTFS`).
+    string fs_type = 3;
+}

grr/proto/grr_response_proto/rrg/net.proto

@@ -91,3 +91,20 @@ message Connection {
     UdpConnection udp = 2;
   }
 }
+
+// Information about a network interface.
+message Interface {
+  // A name of the interface as reported by the system.
+  //
+  // Note that on some system (e.g. Linux), the interface may consist of pretty
+  // much arbitrary bytes and might not be compatible with Unicode. Because this
+  // is not very probable and ergonomics of using a raw `bytes` field, invalid
+  // bytes are going to be subsituted with the replacement character ("�").
+  string name = 1;
+
+  // MAC address associated with the interface.
+  MacAddress mac_address = 2;
+
+  // IP addresses associated with the interface.
+  repeated IpAddress ip_addresses = 3;
+}