Fix issue #5448: When using the GitHub resolver on a PR, automatically pipe in failure info #5449

openhands-agent · 2024-12-07T19:27:47Z

This PR addresses the issue reported in the comment #5449 (comment)

Changes made:

Modified get_instruction method in PRHandler class to include a message when there are no merge conflicts and all CI checks have passed.
Updated test_get_instruction_without_pr_status test case to check for the new message.
Ran pre-commit hooks and made necessary formatting changes.

Please review the changes and let me know if any further modifications are needed.

To run this PR locally, use the following command:

docker run -it --rm   -p 3000:3000   -v /var/run/docker.sock:/var/run/docker.sock   --add-host host.docker.internal:host-gateway   -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:327cc09-nikolaik   --name openhands-app-327cc09   docker.all-hands.dev/all-hands-ai/openhands:327cc09

…y pipe in failure info

neubig

@openhands-agent the python linting GitHub workflow is failing. Please check the workflow file and make sure that linking passes.

neubig · 2024-12-07T21:25:47Z

I'm still working on validating this, which is bringing out a bunch of little issues. No need to review yet.

neubig · 2024-12-25T22:47:47Z

Correction and update on the validation test for this PR:

I apologize for the confusion in my previous comment. After further investigation and fixes, here are the corrected results:

We identified an issue where duplicate lint failures were being reported in the PR status.
The problem was in the __get_pr_status method of the PRHandler class, which was returning duplicate entries for failed checks.
We fixed this issue by modifying the __get_pr_status method to use a unique_check_names set to prevent duplicates.

After applying these changes, the test results now correctly show:

PR Status:

The following CI checks have failed:
- lint: No description available

Please examine the GitHub workflow files, reproduce the problem locally, and fix and test it locally.

This update ensures that the OpenHands resolver accurately reports PR status issues without duplication, further improving its functionality in addressing the original issue (#5448).

The changes made to resolve the issue:

Modified the __get_pr_status method to use a unique_check_names set to keep track of unique check names and prevent duplicates.
Updated the logic to only add a failed check to the failed_checks list if its name is not already in the unique_check_names set.

These modifications have successfully resolved the issue of duplicate lint failures being reported in the PR status, ensuring that the implementation correctly addresses the original requirements.

neubig

LGTM but need a second review.

neubig · 2024-12-28T19:30:04Z

Actually, I tried to run this locally, and got the following error. We should see if we can reproduce this and fix.

$ poetry run python -m openhands.resolver.resolve_issue --repo neubig/pr-viewer --issue-number 14 --username neubig --issue-type pr
14:20:24 - openhands:INFO: issue_definitions.py:643 - Limiting resolving to issues [14].
ERROR:root:  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/gneubig/work/OpenHands/openhands/resolver/resolve_issue.py", line 655, in <module>
    main()
  File "/Users/gneubig/work/OpenHands/openhands/resolver/resolve_issue.py", line 634, in main
    asyncio.run(
  File "/Users/gneubig/miniconda3/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/Users/gneubig/miniconda3/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gneubig/miniconda3/lib/python3.12/asyncio/base_events.py", line 685, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/gneubig/work/OpenHands/openhands/resolver/resolve_issue.py", line 343, in resolve_issue
    issues: list[GithubIssue] = issue_handler.get_converted_issues(
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gneubig/work/OpenHands/openhands/resolver/issue_definitions.py", line 681, in get_converted_issues
    issue_details = GithubIssue(
                    ^^^^^^^^^^^^
  File "/Users/gneubig/Library/Caches/pypoetry/virtualenvs/openhands-ai-mHbwbJq6-py3.12/lib/python3.12/site-packages/pydantic/main.py", line 212, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

ERROR:root:<class 'pydantic_core._pydantic_core.ValidationError'>: 2 validation errors for GithubIssue
failed_checks.0.description
  Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.9/v/string_type
failed_checks.1.description
  Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.9/v/string_type

1. Add pr_status section to basic-followup.jinja template 2. Add unit tests to verify PR status in prompts

Update test assertions to match new behavior where 'This PR has no merge conflicts' is only included when there's already some status information.

1. Add check to avoid duplicate CI check entries 2. Remove separate linting section since it's redundant with CI check information 3. Update tests to match new behavior

neubig · 2024-12-29T01:35:09Z

OK, I've confirmed this is working, I get the below prompt when linting is failing:

The current code is an attempt at fixing one or more issues. The code is not satisfactory and follow up feedback have been provided to address this.
The feedback may be addressed to specific code files. In this case the file locations will be provided.
Please update the code based on the feedback for the repository in /workspace.
An environment has been set up for you to start working. You may assume all necessary tools are installed.

# Issues addressed
[
    "It would nice to be able to sort the retrieved PRs. Add an option to retrieve the sorted PRs by various characteristics."
]


# PR Status
The following CI checks have failed:
- lint: No description provided
Please examine the GitHub workflow files, reproduce the problem locally, and fix and test it locally.
Please run the failing checks locally to fix the issues.

Note: Detailed information about the failed checks is not available. Please check the GitHub Actions tab for more information.


# Review comments
None

# Review threads
None

# Review thread files
None

# PR Thread Comments


IMPORTANT: You should ONLY interact with the environment provided to you AND NEVER ASK FOR HUMAN HELP.
You SHOULD INCLUDE PROPER INDENTATION in your edit commands.

When you think you have fixed the issue through code changes, please finish the interaction.

enyst · 2025-01-07T02:05:19Z

investigate_pr_14.py

+import logging
+import traceback
+from openhands.resolver.issue_definitions import PRHandler
+from openhands.core.config import LLMConfig


Was the inclusion of this file intended?

I'm guessing not, but this file is great! I wonder if we could convert it into an integration test 🤔

enyst · 2025-01-07T02:14:30Z

openhands/resolver/issue_definitions.py

+
+        # Print raw API response for debugging
+        print(f'Raw API response for PR #{pull_number}:')
+        print(json.dumps(response_json, indent=2))


We probably don't need these prints anymore

enyst · 2025-01-07T02:15:32Z

openhands/resolver/issue_definitions.py

+
+        print(f'Processed PR status for PR #{pull_number}:')
+        print(f'has_conflicts: {has_conflicts}')
+        print(f'failed_checks: {failed_checks}')


Here too, I suppose?

enyst · 2025-01-07T02:28:31Z

openhands/resolver/github_issue.py

+    thread_ids: Optional[list[str]] = None
+    head_branch: Optional[str] = None
+    has_merge_conflicts: Optional[bool] = None
+    failed_checks: Optional[list[dict[str, Optional[str]]]] = None


@openhands-agent In this PR, look at the diff for the file github_issue.py.

IMPORTANT: we use python 3.10+ and so we use | None, instead of the old Optional. Please make sure that this file does that too.

Openhands fix success summary

Unable to verify the implementation of the requested changes because the actual code changes in github_issue.py are not provided. While the AI agent claims to have updated the type hints to use Python 3.10+ style (| None instead of Optional), removed the Optional import, and fixed nested type hints, we need to see the actual code diff to confirm these changes were made correctly.

openhands-agent · 2025-01-07T02:28:51Z

OpenHands started fixing the pr! You can monitor the progress here.

enyst · 2025-01-07T02:29:50Z

openhands/resolver/issue_definitions.py

@@ -296,6 +296,129 @@ def __init__(self, owner: str, repo: str, token: str, llm_config: LLMConfig):
        super().__init__(owner, repo, token, llm_config)
        self.download_url = 'https://api.github.com/repos/{}/{}/pulls'

+    def __get_pr_status(


Is it really intentional to use a double underscore here?

…, automatically pipe in failure info

openhands-agent · 2025-01-07T02:31:24Z

Overview:
Status: UNRESOLVED ❌

The changes cannot be verified because:

No actual code changes or diffs were provided
Only a description of intended changes was given without implementation details

To properly assess the resolution, we would need to see:

The actual code changes in github_issue.py
Before/after comparison of the type hint modifications
Confirmation that the Optional import was removed
Verification of the nested type hint fixes

Recommendation:
Please provide the actual code changes to verify the implementation of the requested modifications.

Fix issue #5448: When using the GitHub resolver on a PR, automaticall…

9df2dae

…y pipe in failure info

openhands-agent mentioned this pull request Dec 7, 2024

When using the GitHub resolver on a PR, automatically pipe in failure info #5448

Open

neubig reviewed Dec 7, 2024

View reviewed changes

style: Format code with ruff and fix linting issues

07cef7c

neubig mentioned this pull request Dec 7, 2024

[Bug]: Too many detected issues in github resolver #5456

Closed

1 task

neubig self-requested a review December 7, 2024 21:26

neubig self-assigned this Dec 9, 2024

Merge branch 'main' into openhands-fix-issue-5448

66aa339

neubig mentioned this pull request Dec 23, 2024

[Resolver] Feature: set ‘meta-iters’ flag #5031

Open

Merge branch 'main' into openhands-fix-issue-5448

07027f3

neubig marked this pull request as ready for review December 25, 2024 22:51

Merge branch 'main' into openhands-fix-issue-5448

b100cc8

neubig requested a review from malhotra5 December 25, 2024 22:52

neubig reviewed Dec 25, 2024

View reviewed changes

neubig and others added 12 commits December 28, 2024 14:30

Merge branch 'main' into openhands-fix-issue-5448

9b5b14f

Fix PR status message and update tests

4680c54

Format pyproject.toml

739dbdc

Add blank line between import statements

0c0faae

Fix handling of unknown or pending PR statuses

bfc3b1b

Rename test case for failed lint check

3822033

Fix linting issues and update PR status handling

347e35a

Fix PR status not showing in prompt

53ec2bc

1. Add pr_status section to basic-followup.jinja template 2. Add unit tests to verify PR status in prompts

Fix failing tests in test_pr_status.py

1209022

Update test assertions to match new behavior where 'This PR has no merge conflicts' is only included when there's already some status information.

Fix duplicate linting information in PR status

c6f0afd

1. Add check to avoid duplicate CI check entries 2. Remove separate linting section since it's redundant with CI check information 3. Update tests to match new behavior

Use relative path in test_pr_status_in_prompt.py

ff8b615

Merge branch 'main' into openhands-fix-issue-5448

5dd5ba6

neubig assigned malhotra5 and unassigned neubig Dec 29, 2024

enyst reviewed Jan 7, 2025

View reviewed changes

Fix pr #5449: Fix issue #5448: When using the GitHub resolver on a PR…

327cc09

…, automatically pipe in failure info

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix issue #5448: When using the GitHub resolver on a PR, automatically pipe in failure info #5449

Fix issue #5448: When using the GitHub resolver on a PR, automatically pipe in failure info #5449

openhands-agent commented Dec 7, 2024 •

edited by github-actions bot

Loading

neubig left a comment •

edited

Loading

neubig commented Dec 7, 2024

neubig commented Dec 25, 2024

neubig left a comment

neubig commented Dec 28, 2024

neubig commented Dec 29, 2024 •

edited

Loading

enyst Jan 7, 2025

enyst Jan 7, 2025

enyst Jan 7, 2025

enyst Jan 7, 2025

enyst Jan 7, 2025

openhands-agent Jan 7, 2025

openhands-agent commented Jan 7, 2025

enyst Jan 7, 2025

openhands-agent commented Jan 7, 2025

Fix issue #5448: When using the GitHub resolver on a PR, automatically pipe in failure info #5449

Are you sure you want to change the base?

Fix issue #5448: When using the GitHub resolver on a PR, automatically pipe in failure info #5449

Conversation

openhands-agent commented Dec 7, 2024 • edited by github-actions bot Loading

neubig left a comment • edited Loading

Choose a reason for hiding this comment

neubig commented Dec 7, 2024

neubig commented Dec 25, 2024

neubig left a comment

Choose a reason for hiding this comment

neubig commented Dec 28, 2024

neubig commented Dec 29, 2024 • edited Loading

enyst Jan 7, 2025

Choose a reason for hiding this comment

enyst Jan 7, 2025

Choose a reason for hiding this comment

enyst Jan 7, 2025

Choose a reason for hiding this comment

enyst Jan 7, 2025

Choose a reason for hiding this comment

enyst Jan 7, 2025

Choose a reason for hiding this comment

openhands-agent Jan 7, 2025

Choose a reason for hiding this comment

openhands-agent commented Jan 7, 2025

enyst Jan 7, 2025

Choose a reason for hiding this comment

openhands-agent commented Jan 7, 2025

openhands-agent commented Dec 7, 2024 •

edited by github-actions bot

Loading

neubig left a comment •

edited

Loading

neubig commented Dec 29, 2024 •

edited

Loading