Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Introduce CLI tests to ensure regression is caught in CI #401

Open
12 tasks
c12i opened this issue Oct 23, 2024 · 1 comment
Open
12 tasks

chore: Introduce CLI tests to ensure regression is caught in CI #401

c12i opened this issue Oct 23, 2024 · 1 comment

Comments

@c12i
Copy link
Collaborator

c12i commented Oct 23, 2024

Currently, the scaffolding CLI testing relies heavily on manual processes, which has led to several challenges:

  1. Issues with file/directory generation are only caught through manual testing
  2. The testing process is time-consuming
  3. Edge cases may be missed due to the manual nature of testing
  4. No automated verification of scaffolding output

Current Test Coverage Limitations

While we have integration tests using tryorama that verify the generated hApp's zome functions work as expected, these tests do not verify the CLI's code generation functionality. This has led to issues:

Recent example: The ui directory was completely omitted due to Crane lib filtering out .hbs files, but existing tests didn't catch this
Current tests focus on zome function behavior but don't verify the actual scaffolding output
No automated verification of template processing and file generation

Proposed Solution

Implement automated CLI tests in our CI pipeline to:

  • Verify correct file/directory generation
  • Ensure scaffolding creates expected structure
  • Validate command outputs
  • Catch regressions early

Scope

Phase 1: Basic CLI Testing

  • Set up CLI tests
  • Implement tests for basic file/directory generation
  • Add assertions for common scaffolding patterns
  • Verify command execution results
  • Add tests for error cases and edge scenarios

Phase 2 (Optional): UI Snapshot Testing

Technical Details

Test Coverage Should Include

  • Directory structure validation
  • File content verification
  • Command execution flows
  • Error handling scenarios

Example Test Cases

- Verify scaffold command creates expected directory structure
- Validate file permissions and contents
- Test command option combinations
- Verify error messages for invalid inputs

Success Criteria

  • All CLI commands have automated tests
  • CI pipeline includes CLI testing
  • Reduced time spent on manual testing
  • Improved confidence in CLI functionality
  • Documentation for writing new CLI tests
@c12i c12i self-assigned this Oct 23, 2024
@c12i c12i added this to Holochain Oct 23, 2024
@github-project-automation github-project-automation bot moved this to Backlog in Holochain Oct 23, 2024
@zippy zippy moved this from Backlog to Ready for refinement in Holochain Oct 23, 2024
@jost-s jost-s unassigned c12i Oct 29, 2024
@c12i c12i self-assigned this Nov 27, 2024
@c12i
Copy link
Collaborator Author

c12i commented Nov 28, 2024

Prioritizing #421

@c12i c12i closed this as completed Nov 28, 2024
@github-project-automation github-project-automation bot moved this from Ready for refinement to Done in Holochain Nov 28, 2024
@c12i c12i removed their assignment Dec 11, 2024
@c12i c12i reopened this Dec 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

No branches or pull requests

1 participant