Towards better inference: bits → nibbles

[originalsouth]

Changes

Bits → Nibbles

Issue link

N/A

Demo

T.B.D.

QA notes

T.B.D.

---

Code Checklist

• All the commits in this PR are properly PGP-signed and verified.
• This PR only contains functionality relevant to the issue.
• I have written unit tests for the changes or fixes I made.
• I have checked the documentation and made changes where necessary.
• I have performed a self-review of my code and refactored it to the best of my abilities.

• Tickets have been created for newly discovered issues.
• For any non-trivial functionality, I have added integration and/or end-to-end tests.
• I have informed others of any required .env changes files if required and changed the .env-dist accordingly.
• I have included comments in the code to elaborate on what is not self-evident from the code itself, including references to issues and discussions online, or implicit behavior of an interface.

---

Checklist for code reviewers:

Copy-paste the checklist from the docs/source/templates folder into your comment.

---

Checklist for QA:

Copy-paste the checklist from the docs/source/templates folder into your comment.

[ammar92]

Nice work! I've provided mostly Python related feedback since I missed the discussion about Nibbles. I'd love to know more about it, why is it better than bits, and what are the main differences? Perhaps we can meet offline for coffee soon?

[ammar92]

I was wondering, what is the reason this isn't implemented as an e.g. Pydantic class but instead as a POJO-like class?

[originalsouth]

Somehow the Pydantic class does not work well with the importlib yielding the payload... not sure why but it fixed the issues so I moved on -- perhaps hoping one day you would fix it ;)

[noamblitz]

General flow

The Nibbles "flow" Is as follows. Some ideas are explained below.

graph TD
    A[Start] --> Z[Object is newly created]
    Z --> Y[Add object to stack]
    A --> X[Object is updated]
    Y --> C[For all objects on the stack]
    X --> Y
    X -.-> H[Perform Dollar Sign Magic]
    C --> D[Check which Nibbles have the object's type in their signature]
    D --> E[For all Nibbles with matching type]
    E -->|Query attribute present?| F{Query Attribute?}
    F -->|No| O[Nibble requires only one OOI]
    O --> G[Run Nibble]
    F -->|Yes| H[Perform Dollar Sign Magic]
    H --> I[Run the query]
    I --> J[Output in Octopoes JSON format]
    J --> K[Use Jamespath JSON parser for all types in nibble signature]
    K --> L[Cross-product of all outputs]
    L --> G
    G --> M[Output objects go back to the stack]
    M --> Y

Loading

Jmespath queries

Jmespath allows for easy parsing of the Octopoes JSON output. From the Octopoes output, a Jmespath query will be run for all elements of the signature to get the input for the Nibble. A Jmespath can be [*][?object_type == 'Network'][] to get all Networks from the response. But an element of the signature might also be count: int where the Jmespath will find that count. We can also let Jmespath query wrap all Findings that match a specific condition in an array.

Example

Let's say we want to create a Nibble that checks whether the name of a Network has the same length as the raw of a URL. The signature will have to NibbleParameters:

1. NibbleParameter(object_type=Network, parser="[*][?object_type == 'Network'][]")
2. NibbleParameter(object_type=URL, parser="[*][?object_type == 'URL'][]")

The Octopoes query can be (we still need to add that this query checks specifically for the OOI that is added/changed):

{
        :query {
            :find [(pull ?var [*])]
            :where [
                (or
                    (and [?var :object_type "URL" ] [?var :URL/primary_key $2 ])
                    (and [?var :object_type "Network" ] [?var :Network/primary_key $1])
                )
            ]
        }
    }

After this the product of all results from this query will be created (note that there is either one URL and multiple Networks or one Network and multiple URLs). When we have the product, the nibble can be run:

def find_network_url(network: Network, url: URL) -> Iterator[OOI]:
    if len(network.name) == len(str(url.raw)):
        yield Finding(
            finding_type=KATFindingType(id="Network and URL have same name length").reference,
            ooi=network.reference,
            proof=url.reference,
        )

[originalsouth]

We need a fast, insecure, deterministic hashing algorithm: I am proposing xxhash by zip-hero Yann Collett (so actually python-xxhash); let us discuss!

[sonarcloud]

Quality Gate failed

Failed conditions
28.7% Coverage on New Code (required ≥ 80%)
7.1% Duplication on New Code (required ≤ 3%)

See analysis details on SonarQube Cloud