Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update minimum Python to 3.10 #2467

Merged
merged 4 commits into from
Oct 22, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .devcontainer/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# See here for image contents: https://github.com/microsoft/vscode-dev-containers/tree/v0.233.0/containers/python-3/.devcontainer/base.Dockerfile

# [Choice] Python version (use -bullseye variants on local arm64/Apple Silicon): 3, 3.10, 3.9, 3.8, 3.7, 3.6, 3-bullseye, 3.10-bullseye, 3.9-bullseye, 3.8-bullseye, 3.7-bullseye, 3.6-bullseye, 3-buster, 3.10-buster, 3.9-buster, 3.8-buster, 3.7-buster, 3.6-buster
# [Choice] Python version (use -bullseye variants on local arm64/Apple Silicon): 3, 3.10, 3-bullseye, 3.10-bullseye, 3-buster, 3.10-buster, etc.
ARG VARIANT="3.10-bullseye"
FROM mcr.microsoft.com/vscode/devcontainers/python:0-${VARIANT}

Expand Down
2 changes: 1 addition & 1 deletion .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
"dockerfile": "Dockerfile",
"context": "..",
"args": {
// Update 'VARIANT' to pick a Python version: 3, 3.10, 3.9, 3.8, 3.7, 3.6
// Update 'VARIANT' to pick a Python version: 3, 3.10, etc.
// Append -bullseye or -buster to pin to an OS version.
// Use -bullseye variants on local on arm64/Apple Silicon.
"VARIANT": "3.10",
Expand Down
11 changes: 5 additions & 6 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,26 +21,25 @@ jobs:
# set to false for debugging
fail-fast: true
matrix:
# using Python 3.8 to support running across multiple operating systems including Windows 7
include:
- os: ubuntu-20.04
# use old linux so that the shared library versioning is more portable
artifact_name: capa
asset_name: linux
python_version: 3.8
python_version: '3.10'
- os: ubuntu-20.04
artifact_name: capa
asset_name: linux-py312
python_version: 3.12
python_version: '3.12'
- os: windows-2019
artifact_name: capa.exe
asset_name: windows
python_version: 3.8
python_version: '3.10'
- os: macos-12
# use older macOS for assumed better portability
artifact_name: capa
asset_name: macos
python_version: 3.8
python_version: '3.10'
steps:
- name: Checkout capa
uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
Expand Down Expand Up @@ -107,7 +106,7 @@ jobs:
# upload zipped binaries to Release page
if: github.event_name == 'release'
name: zip and upload ${{ matrix.asset_name }}
runs-on: ubuntu-20.04
runs-on: ubuntu-latest
needs: [build]
strategy:
matrix:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/changelog.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ jobs:
check_changelog:
# no need to check for dependency updates via dependabot
if: github.actor != 'dependabot[bot]' && github.actor != 'dependabot-preview[bot]'
runs-on: ubuntu-20.04
runs-on: ubuntu-latest
env:
NO_CHANGELOG: '[x] No CHANGELOG update needed'
steps:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@0a5c61591373683505ea898e09a3ea4f39ef2b9c # v5.0.0
with:
python-version: '3.8'
python-version: '3.10'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/tag.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ permissions: read-all
jobs:
tag:
name: Tag capa rules
runs-on: ubuntu-20.04
runs-on: ubuntu-latest
steps:
- name: Checkout capa-rules
uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
Expand Down
26 changes: 13 additions & 13 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ env:

jobs:
changelog_format:
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04
steps:
- name: Checkout capa
uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
Expand All @@ -37,15 +37,15 @@ jobs:
if [ $number != 1 ]; then exit 1; fi

code_style:
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04
steps:
- name: Checkout capa
uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
# use latest available python to take advantage of best performance
- name: Set up Python 3.11
- name: Set up Python 3.12
uses: actions/setup-python@0a5c61591373683505ea898e09a3ea4f39ef2b9c # v5.0.0
with:
python-version: "3.11"
python-version: "3.12"
- name: Install dependencies
run: |
pip install -r requirements.txt
Expand All @@ -64,16 +64,16 @@ jobs:
run: pre-commit run deptry --hook-stage manual

rule_linter:
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04
steps:
- name: Checkout capa with submodules
uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
with:
submodules: recursive
- name: Set up Python 3.11
- name: Set up Python 3.12
uses: actions/setup-python@0a5c61591373683505ea898e09a3ea4f39ef2b9c # v5.0.0
with:
python-version: "3.11"
python-version: "3.12"
- name: Install capa
run: |
pip install -r requirements.txt
Expand All @@ -90,15 +90,15 @@ jobs:
matrix:
os: [ubuntu-20.04, windows-2019, macos-12]
# across all operating systems
python-version: ["3.8", "3.11"]
python-version: ["3.10", "3.11"]
include:
# on Ubuntu run these as well
- os: ubuntu-20.04
python-version: "3.8"
python-version: "3.10"
- os: ubuntu-20.04
python-version: "3.9"
python-version: "3.11"
- os: ubuntu-20.04
python-version: "3.10"
python-version: "3.12"
steps:
- name: Checkout capa with submodules
uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
Expand Down Expand Up @@ -131,7 +131,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ["3.9", "3.11"]
python-version: ["3.10", "3.11"]
steps:
- name: Checkout capa with submodules
# do only run if BN_SERIAL is available, have to do this in every step, see https://github.com/orgs/community/discussions/26726#discussioncomment-3253118
Expand Down Expand Up @@ -173,7 +173,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ["3.8", "3.11"]
python-version: ["3.10", "3.11"]
java-version: ["17"]
ghidra-version: ["11.0.1"]
public-version: ["PUBLIC_20240130"] # for ghidra releases
Expand Down
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@

### Breaking Changes

- remove support for Python 3.8 and use Python 3.10 as minimum now #1966 @mr-tz

### New Rules (0)

-
Expand Down
4 changes: 2 additions & 2 deletions capa/capabilities/common.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
import logging
import itertools
import collections
from typing import Any, Tuple
from typing import Any

from capa.rules import Scope, RuleSet
from capa.engine import FeatureSet, MatchResults
Expand Down Expand Up @@ -64,7 +64,7 @@ def has_file_limitation(rules: RuleSet, capabilities: MatchResults, is_standalon

def find_capabilities(
ruleset: RuleSet, extractor: FeatureExtractor, disable_progress=None, **kwargs
) -> Tuple[MatchResults, Any]:
) -> tuple[MatchResults, Any]:
from capa.capabilities.static import find_static_capabilities
from capa.capabilities.dynamic import find_dynamic_capabilities

Expand Down
12 changes: 6 additions & 6 deletions capa/capabilities/dynamic.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
import logging
import itertools
import collections
from typing import Any, List, Tuple
from typing import Any

import capa.perf
import capa.features.freeze as frz
Expand All @@ -24,7 +24,7 @@

def find_call_capabilities(
ruleset: RuleSet, extractor: DynamicFeatureExtractor, ph: ProcessHandle, th: ThreadHandle, ch: CallHandle
) -> Tuple[FeatureSet, MatchResults]:
) -> tuple[FeatureSet, MatchResults]:
"""
find matches for the given rules for the given call.

Expand All @@ -51,7 +51,7 @@ def find_call_capabilities(

def find_thread_capabilities(
ruleset: RuleSet, extractor: DynamicFeatureExtractor, ph: ProcessHandle, th: ThreadHandle
) -> Tuple[FeatureSet, MatchResults, MatchResults]:
) -> tuple[FeatureSet, MatchResults, MatchResults]:
"""
find matches for the given rules within the given thread.

Expand Down Expand Up @@ -89,7 +89,7 @@ def find_thread_capabilities(

def find_process_capabilities(
ruleset: RuleSet, extractor: DynamicFeatureExtractor, ph: ProcessHandle
) -> Tuple[MatchResults, MatchResults, MatchResults, int]:
) -> tuple[MatchResults, MatchResults, MatchResults, int]:
"""
find matches for the given rules within the given process.

Expand Down Expand Up @@ -127,15 +127,15 @@ def find_process_capabilities(

def find_dynamic_capabilities(
ruleset: RuleSet, extractor: DynamicFeatureExtractor, disable_progress=None
) -> Tuple[MatchResults, Any]:
) -> tuple[MatchResults, Any]:
all_process_matches: MatchResults = collections.defaultdict(list)
all_thread_matches: MatchResults = collections.defaultdict(list)
all_call_matches: MatchResults = collections.defaultdict(list)

feature_counts = rdoc.DynamicFeatureCounts(file=0, processes=())

assert isinstance(extractor, DynamicFeatureExtractor)
processes: List[ProcessHandle] = list(extractor.get_processes())
processes: list[ProcessHandle] = list(extractor.get_processes())
n_processes: int = len(processes)

with capa.helpers.CapaProgressBar(
Expand Down
14 changes: 7 additions & 7 deletions capa/capabilities/static.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
import logging
import itertools
import collections
from typing import Any, List, Tuple
from typing import Any

import capa.perf
import capa.helpers
Expand All @@ -26,7 +26,7 @@

def find_instruction_capabilities(
ruleset: RuleSet, extractor: StaticFeatureExtractor, f: FunctionHandle, bb: BBHandle, insn: InsnHandle
) -> Tuple[FeatureSet, MatchResults]:
) -> tuple[FeatureSet, MatchResults]:
"""
find matches for the given rules for the given instruction.

Expand All @@ -53,7 +53,7 @@ def find_instruction_capabilities(

def find_basic_block_capabilities(
ruleset: RuleSet, extractor: StaticFeatureExtractor, f: FunctionHandle, bb: BBHandle
) -> Tuple[FeatureSet, MatchResults, MatchResults]:
) -> tuple[FeatureSet, MatchResults, MatchResults]:
"""
find matches for the given rules within the given basic block.

Expand Down Expand Up @@ -93,7 +93,7 @@ def find_basic_block_capabilities(

def find_code_capabilities(
ruleset: RuleSet, extractor: StaticFeatureExtractor, fh: FunctionHandle
) -> Tuple[MatchResults, MatchResults, MatchResults, int]:
) -> tuple[MatchResults, MatchResults, MatchResults, int]:
"""
find matches for the given rules within the given function.

Expand Down Expand Up @@ -131,16 +131,16 @@ def find_code_capabilities(

def find_static_capabilities(
ruleset: RuleSet, extractor: StaticFeatureExtractor, disable_progress=None
) -> Tuple[MatchResults, Any]:
) -> tuple[MatchResults, Any]:
all_function_matches: MatchResults = collections.defaultdict(list)
all_bb_matches: MatchResults = collections.defaultdict(list)
all_insn_matches: MatchResults = collections.defaultdict(list)

feature_counts = rdoc.StaticFeatureCounts(file=0, functions=())
library_functions: Tuple[rdoc.LibraryFunction, ...] = ()
library_functions: tuple[rdoc.LibraryFunction, ...] = ()

assert isinstance(extractor, StaticFeatureExtractor)
functions: List[FunctionHandle] = list(extractor.get_functions())
functions: list[FunctionHandle] = list(extractor.get_functions())
n_funcs: int = len(functions)
n_libs: int = 0
percentage: float = 0
Expand Down
14 changes: 7 additions & 7 deletions capa/engine.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

import copy
import collections
from typing import TYPE_CHECKING, Set, Dict, List, Tuple, Union, Mapping, Iterable, Iterator
from typing import TYPE_CHECKING, Union, Mapping, Iterable, Iterator

import capa.perf
import capa.features.common
Expand All @@ -27,7 +27,7 @@
# to collect the locations of a feature, do: `features[Number(0x10)]`
#
# aliased here so that the type can be documented and xref'd.
FeatureSet = Dict[Feature, Set[Address]]
FeatureSet = dict[Feature, set[Address]]


class Statement:
Expand Down Expand Up @@ -94,7 +94,7 @@ class And(Statement):
match if all of the children evaluate to True.

the order of evaluation is dictated by the property
`And.children` (type: List[Statement|Feature]).
`And.children` (type: list[Statement|Feature]).
a query optimizer may safely manipulate the order of these children.
"""

Expand Down Expand Up @@ -127,7 +127,7 @@ class Or(Statement):
match if any of the children evaluate to True.

the order of evaluation is dictated by the property
`Or.children` (type: List[Statement|Feature]).
`Or.children` (type: list[Statement|Feature]).
a query optimizer may safely manipulate the order of these children.
"""

Expand Down Expand Up @@ -176,7 +176,7 @@ class Some(Statement):
match if at least N of the children evaluate to True.

the order of evaluation is dictated by the property
`Some.children` (type: List[Statement|Feature]).
`Some.children` (type: list[Statement|Feature]).
a query optimizer may safely manipulate the order of these children.
"""

Expand Down Expand Up @@ -267,7 +267,7 @@ def evaluate(self, features: FeatureSet, short_circuit=True):
# inspect(match_details)
#
# aliased here so that the type can be documented and xref'd.
MatchResults = Mapping[str, List[Tuple[Address, Result]]]
MatchResults = Mapping[str, list[tuple[Address, Result]]]


def get_rule_namespaces(rule: "capa.rules.Rule") -> Iterator[str]:
Expand All @@ -292,7 +292,7 @@ def index_rule_matches(features: FeatureSet, rule: "capa.rules.Rule", locations:
features[capa.features.common.MatchedRule(namespace)].update(locations)


def match(rules: List["capa.rules.Rule"], features: FeatureSet, addr: Address) -> Tuple[FeatureSet, MatchResults]:
def match(rules: list["capa.rules.Rule"], features: FeatureSet, addr: Address) -> tuple[FeatureSet, MatchResults]:
"""
match the given rules against the given features,
returning an updated set of features and the matches.
Expand Down
3 changes: 1 addition & 2 deletions capa/features/com/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and limitations under the License.
from enum import Enum
from typing import Dict, List

from capa.helpers import assert_never

Expand All @@ -22,7 +21,7 @@ class ComType(Enum):
}


def load_com_database(com_type: ComType) -> Dict[str, List[str]]:
def load_com_database(com_type: ComType) -> dict[str, list[str]]:
# lazy load these python files since they are so large.
# that is, don't load them unless a COM feature is being handled.
import capa.features.com.classes
Expand Down
3 changes: 1 addition & 2 deletions capa/features/com/classes.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,8 @@
# Unless required by applicable law or agreed to in writing, software distributed under the License
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and limitations under the License.
from typing import Dict, List

COM_CLASSES: Dict[str, List[str]] = {
COM_CLASSES: dict[str, list[str]] = {
"ClusAppWiz": ["24F97150-6689-11D1-9AA7-00C04FB93A80"],
"ClusCfgAddNodesWizard": ["BB8D141E-C00A-469F-BC5C-ECD814F0BD74"],
"ClusCfgCreateClusterWizard": ["B929818E-F5B0-44DC-8A00-1B5F5F5AA1F0"],
Expand Down
Loading
Loading