Smoke_test.py Compile error: Epoch Queue is unintialized. backtrace

[danindiana]

Not sure what exactly is going on here. Although my first suspicion was it may have to do with duel cards on the same machine but I've gotten the smoke test to run with prior install/config setups before so I don't think that's the issue.

baruch@spinoza
OS: Ubuntu 22.04 jammy
Kernel: x86_64 Linux 6.8.0-51-generic
Uptime: 1h 6m
Packages: 2079
Shell: bash 5.1.16
Disk: 41G / 1.8T (3%)
CPU: AMD Ryzen 7 7700 8-Core @ 16x 5.389GHz
GPU: AMD/ATI
RAM: 3321MiB / 63437MiB

(venv) baruch@spinoza:~/Downloads/tt-smi/tt_smi$ tt-smi -ls
Detected Chips: 2
Detecting ARC: /
Detecting DRAM: /
[] ETH: /
Gathering Information ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
All available boards on host:
┏━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ PCI Dev ID ┃ Board Type ┃ Device Series ┃ Board Number ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ 0 │ Grayskull │ e150 │ 10000361171a0fe │
│ 1 │ Grayskull │ e75 │ 10000741171f143 │
└────────────┴────────────┴───────────────┴─────────────────┘
Boards that can be reset:
┏━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ PCI Dev ID ┃ Board Type ┃ Device Series ┃ Board Number ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ 0 │ Grayskull │ e150 │ 10000361171a0fe │
│ 1 │ Grayskull │ e75 │ 10000741171f143 │
└────────────┴────────────┴───────────────┴─────────────────┘

$which python
/home/baruch/Downloads/tt-buda/venv/bin/python

$ python --version
Python 3.10.12

(venv) baruch@spinoza:~/Downloads$ python smoke_test.py 
2025-01-06 17:13:16.281 | INFO     | Backend         - initialize_child_process called on pid 8648
2025-01-06 17:13:17.156 | DEBUG    | pybuda.tvm_to_python:_determine_node_dtype:1712 - Node 'weights1' does not have a framework dtype specified. Using TVM generated dtype.
2025-01-06 17:13:17.156 | DEBUG    | pybuda.tvm_to_python:_determine_node_dtype:1712 - Node 'weights2' does not have a framework dtype specified. Using TVM generated dtype.
2025-01-06 17:13:17.159 | DEBUG    | pybuda.ttdevice:_create_input_queue_device_connector:1408 - Creating input queue connector on TTDevice 'auto_tt0'
2025-01-06 17:13:17.159 | DEBUG    | pybuda.ttdevice:_create_intermediates_queue_device_connector:1418 - Creating fwd intermediates queue connector on TTDevice 'auto_tt0'
2025-01-06 17:13:17.159 | DEBUG    | pybuda.ttdevice:_create_forward_output_queue_device_connector:1398 - Creating forward output queue connector on TTDevice 'auto_tt0'
2025-01-06 17:13:19.659 | INFO     | pybuda.device_connector:pusher_thread_main:148 - Pusher thread on <pybuda.device_connector.InputQueueDirectPusherDeviceConnector object at 0x70d47611f7f0> starting
2025-01-06 17:13:19.659 | INFO     | Backend         - initialize_child_process called on pid 8698
2025-01-06 17:13:19.659 | DEBUG    | pybuda.device:run_next_command:455 - Received COMPILE command on TTDevice 'auto_tt0' / 8698
2025-01-06 17:13:19.659 | DEBUG    | pybuda.ttdevice:compile_for:785 - Compiling for Inference mode on TTDevice 'auto_tt0'
2025-01-06 17:13:19.674 | INFO     | pybuda.compile:pybuda_compile_from_context:239 - Running compile stage init_compile
2025-01-06 17:13:19.680 | INFO     | pybuda.ci:initialize_output_build_directory:98 - Pybuda output build directory for compiled artifacts: /tmp/baruch/6778b7184359
2025-01-06 17:13:19.680 | INFO     | pybuda.ci:create_symlink:89 - Symlink created from /home/baruch/Downloads/tt_build/test_out to /tmp/baruch/6778b7184359
2025-01-06 17:13:19.688 | INFO     | pybuda.compile:init_compile:511 - Device architecture: grayskull
2025-01-06 17:13:19.688 | INFO     | pybuda.compile:init_compile:512 - Device grid size: r = 10, c = 12
2025-01-06 17:13:19.688 | INFO     | pybuda.compile:init_compile:522 - Using chips: [0]
2025-01-06 17:13:19.688 | INFO     | pybuda.compile:pybuda_compile_from_context:239 - Running compile stage generate_initial_graph
2025-01-06 17:13:19.690 | INFO     | pybuda.compile:pybuda_compile_from_context:239 - Running compile stage post_initial_graph_pass
2025-01-06 17:13:19.691 | INFO     | pybuda.compile:pybuda_compile_from_context:239 - Running compile stage consteval_graph
2025-01-06 17:13:19.692 | INFO     | pybuda.compile:pybuda_compile_from_context:239 - Running compile stage optimized_graph
2025-01-06 17:13:19.694 | INFO     | pybuda.compile:pybuda_compile_from_context:239 - Running compile stage post_autograd_pass
2025-01-06 17:13:19.695 | INFO     | pybuda.compile:pybuda_compile_from_context:239 - Running compile stage pre_lowering_pass
2025-01-06 17:13:19.695 | INFO     | pybuda.compile:pybuda_compile_from_context:239 - Running compile stage buda_graph_pre_placer
2025-01-06 17:13:19.696 | INFO     | GraphCompiler   - Running with Automatic Mixed Precision Level = 0.
2025-01-06 17:13:19.697 | INFO     | pybuda.compile:pybuda_compile_from_context:239 - Running compile stage balancer_pass
2025-01-06 17:13:19.697 | INFO     | Always          - Running Balancer with Policy: PolicyType::NLP
2025-01-06 17:13:19.698 | INFO     | Balancer        - Based on NLP matmul analysis, target cycle count is set to 45000
2025-01-06 17:13:19.698 | INFO     | Balancer        - Balancing 100% completed!
2025-01-06 17:13:19.698 | INFO     | Balancer        - Balancer perf score : 2538071
2025-01-06 17:13:19.698 | WARNING  | Placer          - Compilation Option with input queue placed on host, but Grayskull does not support fast device reads from host. Placer opting to allocate the queue on device instead.
2025-01-06 17:13:19.698 | INFO     | Backend         - Lookup contexts -- arch:system scope:device0 name:harvesting_mask
2025-01-06 17:13:19.718 | INFO     | Runtime         - Cluster descriptor file does not exist at path=
2025-01-06 17:13:19.723 | INFO     | Placer          - Running DRAM allocator for device 0
2025-01-06 17:13:19.723 | INFO     | PerfModel       - Running performance model...
2025-01-06 17:13:19.723 | INFO     | pybuda.compile:pybuda_compile_from_context:239 - Running compile stage pre_netlist_pass
2025-01-06 17:13:19.724 | INFO     | pybuda.compile:pybuda_compile_from_context:239 - Running compile stage generate_netlist
2025-01-06 17:13:19.724 | INFO     | pybuda.compile:generate_netlist:1075 - Generating Netlist
2025-01-06 17:13:19.726 | INFO     | pybuda.ci:create_symlink:89 - Symlink created from /home/baruch/Downloads/direct_pt_netlist.yaml to /tmp/baruch/6778b7184359/direct_pt_netlist.yaml
2025-01-06 17:13:19.741 | INFO     | pybuda.compile:pybuda_compile_from_context:239 - Running compile stage backend_golden_verify
2025-01-06 17:13:19.741 | INFO     | Runtime         - Running tt_runtime on host: 'spinoza'
2025-01-06 17:13:19.741 | INFO     | PerfInfra       - Backend profiler is disabled
2025-01-06 17:13:19.741 | INFO     | PerfInfra       - Memory profiler is enabled
2025-01-06 17:13:19.742 | WARNING  | Runtime         - Config.soc_descriptor_path='/tmp/baruch/6778b7184359/device_descs/grayskull_0_0x0.yaml' doesn't exist, defaulting to '/home/baruch/Downloads/pybudrelease/venv/lib/python3.10/site-packages/budabackend/device/grayskull_10x12.yaml'
2025-01-06 17:13:19.747 | INFO     | SiliconDriver   - Detected 2 PCI devices : [0, 1]
2025-01-06 17:13:19.792 | INFO     | Runtime         - Compiling Firmware for TT device
2025-01-06 17:13:20.112 | FATAL    | Always          - Not every worker core is allocated a hexq
2025-01-06 17:13:20.113 | ERROR    | Always          - Not every worker core is allocated a hexq
backtrace:
 --- tt_epoch_control::init_epoch_ctrl(int, int, int, bool)
 --- tt_epoch_loader::create_and_allocate_epoch_queues(bool)
 --- tt_runtime::create_graphs_and_init_queues()
 --- tt_runtime::initialize(tt::tt_compile_result*)
 --- /home/baruch/Downloads/pybudrelease/venv/lib/python3.10/site-packages/pybuda/_C.cpython-310-x86_64-linux-gnu.so(+0x8ef784) [0x70d432eef784]
 --- /home/baruch/Downloads/pybudrelease/venv/lib/python3.10/site-packages/pybuda/_C.cpython-310-x86_64-linux-gnu.so(+0x2c33f6) [0x70d4328c33f6]
 --- /home/baruch/Downloads/pybudrelease/venv/bin/python(+0x18b282) [0x5f5da0064282]
 --- /home/baruch/Downloads/pybudrelease/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x5f5da005ab4b]
 --- /home/baruch/Downloads/pybudrelease/venv/bin/python(+0x198ebb) [0x5f5da0071ebb]
 --- /home/baruch/Downloads/pybudrelease/venv/bin/python(_PyEval_EvalFrameDefault+0x58aa) [0x5f5da0053b7a]
 --- /home/baruch/Downloads/pybudrelease/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x5f5da0059d14]
 --- /home/baruch/Downloads/pybudrelease/venv/bin/python(+0x195005) [0x5f5da006e005]
 --- /home/baruch/Downloads/pybudrelease/venv/bin/python(_PyObject_MakeTpCall+0x1fc) [0x5f5da005aaec]
 --- /home/baruch/Downloads/pybudrelease/venv/bin/python(_PyEval_EvalFrameDefault+0x5c7d) [0x5f5da0053f4d]
 --- /home/baruch/Downloads/pybudrelease/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5f5da0064aec]
 --- /home/baruch/Downloads/pybudrelease/venv/bin/python(_PyEval_EvalFrameDefault+0x818) [0x5f5da004eae8]
 --- /home/baruch/Downloads/pybudrelease/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5f5da0064aec]
 --- /home/baruch/Downloads/pybudrelease/venv/bin/python(_PyEval_EvalFrameDefault+0x818) [0x5f5da004eae8]
 --- /home/baruch/Downloads/pybudrelease/venv/bin/python(+0x198d2e) [0x5f5da0071d2e]
 --- /home/baruch/Downloads/pybudrelease/venv/bin/python(_PyEval_EvalFrameDefault+0x2c89) [0x5f5da0050f59]
 --- /home/baruch/Downloads/pybudrelease/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5f5da0064aec]
 --- /home/baruch/Downloads/pybudrelease/venv/bin/python(_PyEval_EvalFrameDefault+0x818) [0x5f5da004eae8]
 --- /home/baruch/Downloads/pybudrelease/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5f5da0064aec]
 --- /home/baruch/Downloads/pybudrelease/venv/bin/python(_PyEval_EvalFrameDefault+0x818) [0x5f5da004eae8]
 --- /home/baruch/Downloads/pybudrelease/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5f5da0064aec]
 --- /home/baruch/Downloads/pybudrelease/venv/bin/python(_PyEval_EvalFrameDefault+0x6d2) [0x5f5da004e9a2]
 --- /home/baruch/Downloads/pybudrelease/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5f5da0064aec]
 --- /home/baruch/Downloads/pybudrelease/venv/bin/python(_PyEval_EvalFrameDefault+0x1a22) [0x5f5da004fcf2]
 --- /home/baruch/Downloads/pybudrelease/venv/bin/python(+0x25ae56) [0x5f5da0133e56]
 --- /home/baruch/Downloads/pybudrelease/venv/bin/python(PyEval_EvalCode+0x86) [0x5f5da0133d26]
 --- /home/baruch/Downloads/pybudrelease/venv/bin/python(+0x281ae8) [0x5f5da015aae8]
 --- /home/baruch/Downloads/pybudrelease/venv/bin/python(+0x27c2ef) [0x5f5da01552ef]
 --- /home/baruch/Downloads/pybudrelease/venv/bin/python(PyRun_StringFlags+0x81) [0x5f5da014ef61]
 --- /home/baruch/Downloads/pybudrelease/venv/bin/python(PyRun_SimpleStringFlags+0x41) [0x5f5da014ee11]
 --- /home/baruch/Downloads/pybudrelease/venv/bin/python(Py_RunMain+0x3d0) [0x5f5da014e140]
 --- /home/baruch/Downloads/pybudrelease/venv/bin/python(Py_BytesMain+0x2d) [0x5f5da0127d6d]
 --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x70d4db429d90]
 --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x70d4db429e40]
 --- /home/baruch/Downloads/pybudrelease/venv/bin/python(_start+0x25) [0x5f5da0127c65]

2025-01-06 17:13:20.113 | INFO     | Runtime         - Cleaning up runtime and closing device.
2025-01-06 17:13:20.113 | FATAL    | Always          - Epoch Queue is unintialized.
2025-01-06 17:13:20.124 | ERROR    | pybuda.device:run_next_command:469 - Compile error: Epoch Queue is unintialized.
backtrace:
 --- tt_epoch_queue::push_to_dram(tt_cluster*, std::shared_ptr<tt_hex>, bool, bool)
 --- tt_epoch_queue::push_command(std::shared_ptr<tt_hex>, tt_cluster*, bool)
 --- tt_epoch_control::push_command(tt_epoch_queue*, std::shared_ptr<tt_hex>, bool)
 --- tt_epoch_control::push_command_to_all_workers_and_update_l1_wptr(std::vector<unsigned int, std::allocator<unsigned int> > const&)
 --- tt_epoch_loader::send_end_program_commands()
 --- /home/baruch/Downloads/pybudrelease/venv/lib/python3.10/site-packages/pybuda/../budabackend/build/lib/libtt.so(+0x71ca95) [0x70d43211ca95]
 --- /home/baruch/Downloads/pybudrelease/venv/lib/python3.10/site-packages/pybuda/../budabackend/build/lib/libtt.so(+0x771680) [0x70d432171680]
 --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x70d4c00dc253]
 --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x70d4db494ac3]
 --- /lib/x86_64-linux-gnu/libc.so.6(+0x126850) [0x70d4db526850]

Traceback (most recent call last):
  File "/home/baruch/Downloads/pybudrelease/venv/lib/python3.10/site-packages/pybuda/device.py", line 458, in run_next_command
    ret = self.compile_for(
  File "/home/baruch/Downloads/pybudrelease/venv/lib/python3.10/site-packages/pybuda/ttdevice.py", line 832, in compile_for
    self.backend_api = BackendAPI(
  File "/home/baruch/Downloads/pybudrelease/venv/lib/python3.10/site-packages/pybuda/backend.py", line 84, in __init__
    if self.be_api.initialize(self.compile_result) != BackendStatusCode.Success:
RuntimeError: Epoch Queue is unintialized.
backtrace:
 --- tt_epoch_queue::push_to_dram(tt_cluster*, std::shared_ptr<tt_hex>, bool, bool)
 --- tt_epoch_queue::push_command(std::shared_ptr<tt_hex>, tt_cluster*, bool)
 --- tt_epoch_control::push_command(tt_epoch_queue*, std::shared_ptr<tt_hex>, bool)
 --- tt_epoch_control::push_command_to_all_workers_and_update_l1_wptr(std::vector<unsigned int, std::allocator<unsigned int> > const&)
 --- tt_epoch_loader::send_end_program_commands()
 --- /home/baruch/Downloads/pybudrelease/venv/lib/python3.10/site-packages/pybuda/../budabackend/build/lib/libtt.so(+0x71ca95) [0x70d43211ca95]
 --- /home/baruch/Downloads/pybudrelease/venv/lib/python3.10/site-packages/pybuda/../budabackend/build/lib/libtt.so(+0x771680) [0x70d432171680]
 --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x70d4c00dc253]
 --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x70d4db494ac3]
 --- /lib/x86_64-linux-gnu/libc.so.6(+0x126850) [0x70d4db526850]


2025-01-06 17:13:20.124 | INFO     | Runtime         - Cleaning up runtime and closing device.
2025-01-06 17:13:20.124 | FATAL    | Always          - Epoch Queue is unintialized.
terminate called after throwing an instance of 'std::runtime_error'
  what():  Epoch Queue is unintialized.
backtrace:
 --- tt_epoch_queue::push_to_dram(tt_cluster*, std::shared_ptr<tt_hex>, bool, bool)
 --- tt_epoch_queue::push_command(std::shared_ptr<tt_hex>, tt_cluster*, bool)
 --- tt_epoch_control::push_command(tt_epoch_queue*, std::shared_ptr<tt_hex>, bool)
 --- tt_epoch_control::push_command_to_all_workers_and_update_l1_wptr(std::vector<unsigned int, std::allocator<unsigned int> > const&)
 --- tt_epoch_loader::send_end_program_commands()
 --- /home/baruch/Downloads/pybudrelease/venv/lib/python3.10/site-packages/pybuda/../budabackend/build/lib/libtt.so(+0x71ca95) [0x70d43211ca95]
 --- /home/baruch/Downloads/pybudrelease/venv/lib/python3.10/site-packages/pybuda/../budabackend/build/lib/libtt.so(+0x771680) [0x70d432171680]
 --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x70d4c00dc253]
 --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x70d4db494ac3]
 --- /lib/x86_64-linux-gnu/libc.so.6(+0x126850) [0x70d4db526850]

^CTraceback (most recent call last):
  File "/home/baruch/Downloads/smoke_test.py", line 27, in <module>
    test_module_direct_pytorch()
  File "/home/baruch/Downloads/smoke_test.py", line 21, in test_module_direct_pytorch
    output = pybuda.PyTorchModule("direct_pt", PyTorchTestModule()).run(input1, input2)
  File "/home/baruch/Downloads/pybudrelease/venv/lib/python3.10/site-packages/pybuda/module.py", line 95, in run
    output_q = pybuda.run_inference(self, inputs=[args])
  File "/home/baruch/Downloads/pybudrelease/venv/lib/python3.10/site-packages/pybuda/run/api.py", line 90, in run_inference
    return _run_inference(module, inputs, input_count, output_queue, _sequential, _perf_trace, _verify_cfg)
  File "/home/baruch/Downloads/pybudrelease/venv/lib/python3.10/site-packages/pybuda/run/impl.py", line 277, in _run_inference
    return _run_devices_inference(
  File "/home/baruch/Downloads/pybudrelease/venv/lib/python3.10/site-packages/pybuda/run/impl.py", line 467, in _run_devices_inference
    output_queue = _initialize_pipeline(False, output_queue, sequential=sequential, verify_cfg=verify_cfg)
  File "/home/baruch/Downloads/pybudrelease/venv/lib/python3.10/site-packages/pybuda/run/impl.py", line 414, in _initialize_pipeline
    _compile_devices(sequential, training=training, sample_inputs=sample_inputs, sample_targets=sample_targets, microbatch_count=microbatch_count, verify_cfg=verify_cfg)
  File "/home/baruch/Downloads/pybudrelease/venv/lib/python3.10/site-packages/pybuda/run/impl.py", line 1250, in _compile_devices
    ret = _run_command(d, sequential, Command.compile(inputs, compiler_cfg, dev_targets, microbatch_size, microbatch_count, verify_cfg), response=True)
  File "/home/baruch/Downloads/pybudrelease/venv/lib/python3.10/site-packages/pybuda/run/impl.py", line 294, in _run_command
    return device.get_command_queue_response()
  File "/home/baruch/Downloads/pybudrelease/venv/lib/python3.10/site-packages/pybuda/device.py", line 307, in get_command_queue_response
    resp = self._command_queue_resp.get(timeout=0.1)
  File "/usr/lib/python3.10/multiprocessing/queues.py", line 113, in get
    if not self._poll(timeout):
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 424, in _poll
    r = wait([self], timeout)
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 931, in wait
    ready = selector.select(timeout)
  File "/usr/lib/python3.10/selectors.py", line 416, in select
    fd_event_list = self._selector.poll(timeout)
KeyboardInterrupt
2025-01-06 17:27:59.278 | DEBUG    | pybuda.run.impl:_shutdown:1269 - PyBuda shutdown
^CException ignored in atexit callback: <function shutdown at 0x71d07cf75510>
Traceback (most recent call last):
  File "/home/baruch/Downloads/pybudrelease/venv/lib/python3.10/site-packages/pybuda/run/api.py", line 475, in shutdown
    return _shutdown()
  File "/home/baruch/Downloads/pybudrelease/venv/lib/python3.10/site-packages/pybuda/run/impl.py", line 1280, in _shutdown
    ctx.final_barrier.wait()
  File "/usr/lib/python3.10/threading.py", line 668, in wait
    self._wait(timeout)
  File "/usr/lib/python3.10/threading.py", line 703, in _wait
    if not self._cond.wait_for(lambda : self._state != 0, timeout):
  File "/usr/lib/python3.10/multiprocessing/synchronize.py", line 313, in wait_for
    self.wait(waittime)
  File "/usr/lib/python3.10/multiprocessing/synchronize.py", line 261, in wait
    return self._wait_semaphore.acquire(True, timeout)
KeyboardInterrupt: 

pip list
Package                      Version
---------------------------- --------------------
absl-py                      2.1.0
accelerate                   0.23.0
addict                       2.4.0
aiohappyeyeballs             2.4.4
aiohttp                      3.11.11
aiosignal                    1.3.2
alembic                      1.14.0
antlr4-python3-runtime       4.9.3
anyio                        3.7.1
appdirs                      1.4.4
asttokens                    3.0.0
astunparse                   1.6.3
async-timeout                5.0.1
attrs                        24.3.0
audioread                    3.0.1
backcall                     0.2.0
bitarray                     2.5.1
boto3                        1.35.93
botocore                     1.35.93
cachetools                   5.5.0
certifi                      2024.12.14
cffi                         1.17.1
charset-normalizer           3.4.1
chex                         0.1.6
clang-format                 14.0.3
click                        8.1.8
colorama                     0.4.6
coloredlogs                  15.0.1
colorlog                     6.9.0
cycler                       0.12.1
dataclasses-json             0.5.7
datasets                     2.3.2
decorator                    5.1.1
deepdiff                     6.7.1
diffusers                    0.27.2
dill                         0.3.5.1
dm-tree                      0.1.8
efficientnet-pytorch         0.7.1
exceptiongroup               1.2.2
execnet                      2.1.1
executing                    2.1.0
fastapi                      0.103.2
filelock                     3.16.1
fire                         0.7.0
flatbuffers                  23.5.26
fonttools                    4.55.3
frozenlist                   1.5.0
fsspec                       2024.12.0
gast                         0.4.0
gitdb                        4.0.12
GitPython                    3.1.44
google-auth                  2.37.0
google-auth-oauthlib         1.0.0
google-pasta                 0.2.0
graphviz                     0.8.4
greenlet                     3.1.1
grpcio                       1.69.0
h5py                         3.12.1
huggingface-hub              0.27.1
humanfriendly                10.0
hydra-core                   1.3.2
idna                         3.10
imageio                      2.36.1
importlib_metadata           8.5.0
iniconfig                    2.0.0
ipython                      8.8.0
jax                          0.4.13
jaxlib                       0.4.11
jedi                         0.19.2
Jinja2                       3.1.3
jmespath                     1.0.1
joblib                       1.4.2
keras                        2.13.1
kiwisolver                   1.4.8
lazy_loader                  0.4
libclang                     18.1.1
librosa                      0.10.0.post2
lightning                    2.5.0.post0
lightning-utilities          0.11.9
llvmlite                     0.43.0
loguru                       0.5.3
Mako                         1.3.8
Markdown                     3.7
MarkupSafe                   3.0.2
marshmallow                  3.24.0
marshmallow-enum             1.5.1
matplotlib                   3.5.1
matplotlib-inline            0.1.7
ml-dtypes                    0.2.0
mock                         5.1.0
mpmath                       1.3.0
msgpack                      1.1.0
multidict                    6.1.0
multiprocess                 0.70.13
munch                        4.0.0
mxnet                        1.9.1
mypy-extensions              1.0.0
networkx                     2.8.5
numba                        0.60.0
numpy                        1.23.1
nvidia-ml-py3                7.352.0
oauthlib                     3.2.2
omegaconf                    2.3.0
onnx                         1.16.0
onnxruntime                  1.18.1
opencv-contrib-python        4.9.0.80
opencv-python                4.10.0.84
opencv-python-headless       4.6.0.66
opt_einsum                   3.4.0
optuna                       3.6.1
ordered-set                  4.1.0
packaging                    24.2
pandas                       1.5.3
parso                        0.8.4
patool                       3.1.0
patsy                        1.0.1
pexpect                      4.9.0
pickleshare                  0.7.5
pillow                       11.1.0
pip                          22.0.2
pluggy                       0.13.1
pooch                        1.6.0
portalocker                  3.1.1
pretrainedmodels             0.7.4
prettytable                  3.0.0
prompt_toolkit               3.0.48
propcache                    0.2.1
protobuf                     3.20.3
psutil                       6.1.1
ptyprocess                   0.7.0
pure_eval                    0.2.3
py                           1.11.0
py-cpuinfo                   9.0.0
pyarrow                      18.1.0
pyasn1                       0.6.1
pyasn1_modules               0.4.1
pybboxes                     0.1.6
pybind11                     2.6.2
pycocotools                  2.0.8
pycparser                    2.22
pydantic                     1.10.19
Pygments                     2.19.1
pyinstrument                 4.1.1
pyparsing                    3.2.1
pytest                       6.2.4
pytest-forked                1.6.0
pytest-timeout               2.0.1
pytest-xdist                 2.5.0
python-dateutil              2.9.0.post0
python-gitlab                4.4.0
pytorch-forecasting          1.0.0
pytorch-lightning            2.5.0.post0
pytorch_optimizer            2.12.0
pytorchcv                    0.0.67
pytz                         2024.2
PyWavelets                   1.8.0
PyYAML                       6.0.2
regex                        2024.11.6
requests                     2.28.2
requests-oauthlib            2.0.0
requests-toolbelt            1.0.0
responses                    0.18.0
rsa                          4.9
s3transfer                   0.10.4
sacrebleu                    2.1.0
sacremoses                   0.0.53
safetensors                  0.5.0
sahi                         0.11.20
scikit-image                 0.20.0
scikit-learn                 1.6.0
scipy                        1.8.0
seaborn                      0.13.2
segmentation-models-pytorch  0.3.2
sentencepiece                0.1.96
setuptools                   59.6.0
shapely                      2.0.6
six                          1.17.0
smmap                        5.0.2
sniffio                      1.3.1
soundfile                    0.12.1
soxr                         0.5.0.post1
SQLAlchemy                   2.0.24
stack-data                   0.6.3
starlette                    0.27.0
statsmodels                  0.14.4
subword-nmt                  0.3.8
sympy                        1.13.3
tabulate                     0.9.0
tensorboard                  2.13.0
tensorboard-data-server      0.7.2
tensorflow-cpu               2.13.0
tensorflow-estimator         2.13.0
tensorflow-hub               0.12.0
tensorflow-io-gcs-filesystem 0.37.1
termcolor                    2.5.0
terminaltables               3.1.10
tf2onnx                      1.15.1
tflite                       2.10.0
thop                         0.1.1.post2209072238
threadpoolctl                3.5.0
tifffile                     2024.12.12
timm                         0.6.12
tokenizers                   0.19.1
toml                         0.10.2
toolz                        1.0.0
torch                        2.1.0+cpu.cxx11.abi
torchmetrics                 1.6.1
torchvision                  0.16.0+cpu
torchxrayvision              0.0.39
tqdm                         4.67.1
traitlets                    5.14.3
transformers                 4.41.0
typing_extensions            4.5.0
typing-inspect               0.9.0
ultralytics                  8.0.145
urllib3                      1.26.14
vgg-pytorch                  0.3.0
wcwidth                      0.2.13
Werkzeug                     3.1.3
wheel                        0.45.1
wrapt                        1.17.0
xxhash                       3.5.0
yarl                         1.18.3
yolov5                       7.0.9
yolov6detect                 0.4.1
zipp                         3.21.0