chore: Clean up `.load_agent` usage #5683

Job	Run time
unit-run (test_vector_embeddings.py)	1m 5s
unit-run (test_client.py)	1m 59s
unit-run (test_client_legacy.py)	2m 21s
unit-run (test_server.py)	1m 20s
unit-run (test_v1_routes.py)	1m 7s
unit-run (test_local_client.py)	1m 7s
unit-run (test_managers.py)	1m 19s
unit-run (test_base_functions.py)	1m 26s
unit-run (test_tool_schema_parsing.py)	2m 34s
unit-run (test_tool_rule_solver.py)	57s
unit-run (test_memory.py)	57s
unit-run (test_utils.py)	1m 0s
unit-run (test_stream_buffer_readers.py)	57s
	18m 9s

Provide feedback