Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Is this this ok? #12

Closed
wants to merge 66 commits into from
Closed

feat: Is this this ok? #12

wants to merge 66 commits into from

Conversation

phoevos
Copy link

@phoevos phoevos commented Jul 22, 2024

Did I fix the branch or did I just make it worse?

xiatianrui1110 and others added 8 commits July 22, 2024 11:49
@xiatianrui1110 @phoevos
rebase evaluation commits
94ac19f
@GrayNekoBean @phoevos
fix CLI import
77be524
@phoevos
eval: Update Python requirements and project config
5572663 
* Add requirements.in and generate requirements.txt for evaluation
  scripts
* Update requirements-dev.txt to include evaluation requirements
* Fix backend requirements to avoid conflicts with evaluation
* Update pyproject.toml to include evaluation scripts in linting

Signed-off-by: Phoevos Kalemkeris <[email protected]>
@binxuan39 @phoevos
Add a README.md and fix naming bug
3102f3b
@xiatianrui1110 @phoevos
deal with comments
b961e80
@xiatianrui1110 @phoevos
deal with comments and modify the readme file
f4a0d61
@xiatianrui1110 @phoevos
fix readme format
9c4c2dd
@xiatianrui1110 @phoevos
fix readme format
255518a
@github-actions GitHub Actions
Copy link

Check Broken URLs

We have automatically detected the following broken URLs in your files. Review and fix the paths to resolve this issue.

Check the file paths and associated broken URLs inside them. For more details, check our Contributing Guide.

File Full Path Issues
./docs/deploy_features.md 1. https://techcommunity.microsoft.com/t5/ai-azure-ai-services-blog/announcing-the-public-preview-of-integrated-vectorization-in/ba-p/3960809#:~:text=Integrated%20vectorization%20is%20a%20new%20feature%20of%20Azure,pull-indexers%2C%20and%20vectorization%20of%20text%20queries%20through%20vectorizers
./docs/data_ingestion.md 1. https://techcommunity.microsoft.com/t5/ai-azure-ai-services-blog/announcing-the-public-preview-of-integrated-vectorization-in/ba-p/3960809#:~:text=Integrated%20vectorization%20is%20a%20new%20feature%20of%20Azure,pull-indexers%2C%20and%20vectorization%20of%20text%20queries%20through%20vectorizers

binxuan39 and others added 12 commits July 22, 2024 16:07
@binxuan39
Update Red Teaming
1c05dda 
- Allow user to set the output directory
- Allow to run the redteaming on all the scorer in the directory in one time
- Upadte README.md regarding the running of red teaming
@binxuan39
Update README.md with the instruction of generating answers from ques…
6509182 
…tions
@binxuan39
Update red_teaming.py, adding validation of input yaml file
b539ab5
@phoevos
fix: Very Important Changes
cbe20a1 
Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos
fixup! fix: Very Important Changes
ba4a31a 
Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos
fixup! fix: Very Important Changes
532853f 
Fix `send_question_to_target` and relevant tests.

Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos
fix: Ignore PyRIT on unsupported Python versions
4094bb0 
Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos
fixup! fix: Ignore PyRIT on unsupported Python versions
3da2b3c 
Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos
fix: Pin azure-ai-generative[evaluate]
709ec31 
Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos
Merge remote-tracking branch 'origin/main' into evaluation-rb
3af3cfd
@phoevos
fixup! fix: Ignore PyRIT on unsupported Python versions
a83d19c 
Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos
feat: Add tests for evaluate.py
6064d48 
Signed-off-by: Phoevos Kalemkeris <[email protected]>
@github-actions GitHub Actions
Copy link

Check Broken URLs

We have automatically detected the following broken URLs in your files. Review and fix the paths to resolve this issue.

Check the file paths and associated broken URLs inside them. For more details, check our Contributing Guide.

File Full Path Issues
./docs/data_ingestion.md 1. https://techcommunity.microsoft.com/t5/ai-azure-ai-services-blog/announcing-the-public-preview-of-integrated-vectorization-in/ba-p/3960809#:~:text=Integrated%20vectorization%20is%20a%20new%20feature%20of%20Azure,pull-indexers%2C%20and%20vectorization%20of%20text%20queries%20through%20vectorizers

@github-actions GitHub Actions
Copy link

Check Broken URLs

We have automatically detected the following broken URLs in your files. Review and fix the paths to resolve this issue.

Check the file paths and associated broken URLs inside them. For more details, check our Contributing Guide.

File Full Path Issues
./README.md 1. https://techcommunity.microsoft.com/t5/ai-azure-ai-services-blog/access-control-in-generative-ai-applications-with-azure/ba-p/3956408
./samples/chat/README.md 1. https://aka.ms/entgptsearchblog

phoevos
phoevos commented Jul 26, 2024
View reviewed changes
Copy link
Author

@phoevos phoevos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PTAL, some changes are needed! Have we tested that with Hugging Face?

evaluation/app_chat_target.py Outdated Show resolved Hide resolved
evaluation/app_chat_target.py Outdated Show resolved Hide resolved
evaluation/app_chat_target.py Outdated Show resolved Hide resolved
evaluation/app_chat_target.py Outdated Show resolved Hide resolved
evaluation/app_chat_target.py Outdated Show resolved Hide resolved
evaluation/service_setup.py Outdated Show resolved Hide resolved
evaluation/evaluate.py Outdated Show resolved Hide resolved
evaluation/cli.py Outdated Show resolved Hide resolved
evaluation/app_chat_target.py Show resolved Hide resolved
evaluation/app_chat_target.py Show resolved Hide resolved
phoevos added 24 commits July 31, 2024 13:24
@phoevos
fix: Only load dotenv once
4559444 
Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos
fix: Default to using double quotes for strings
45d8ef4 
Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos
fix: Make endpoint_uri compulsory for chat target
bd9a8ff 
Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos
fix: Update docstrings in app_chat_target.py
c1773ef 
Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos
fix: Add retry decorator to app_chat_target.py
71b8508 
Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos
fix: Refactor list of mocked modules in conftest.py
d27b17f 
Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos
fix: Check if message content exists
2de460f 
Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos
fix: Pass config dict to get_app_target
e5a811b 
Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos
fix: Wrap target request with error handling
d2d9c44 
This is not ideal because we're using a PyRIT util to send out the
request which means that we subsequently have to catch PyRIT exceptions
as well as rely on httpx for the actual request.

Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos
fix: Update scorer definitions
4f71a27 
Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos
fix: Check if class_identifier exists for scorer
67ed3f4 
Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos
feat: Add util for saving JSONL files
fce53ce 
Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos
feat: Add helper for generating evaluation answers
a11fe81 
Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos
feat: Expose generate_answers function to the CLI
a67fa2b 
Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos
feat: Add scripts for creating evaluation .env
15083e6 
Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos
fix: Remove stale JMESPath refs from config.json
8101915 
Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos
fix: Update list of expected environment variables
e0fbd2f 
Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos
fix: Pretty dump scores.json
73b6a0e 
Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos
fix: Standardise handling of targeturl
b8dea7d 
Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos
fix: Convert docstring to imperative
a4961d0 
Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos
fix: Drop evaluate_parameters.json
e407122 
Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos
fix: Update README.md
e1daf58 
* Update README.md to reflect the changes in the project
* Rephrase some parts
* Standardise the formatting of the provided CLI commands
* Use 'create_eval_dotenv' scripts to set up the environment
* Remove '.env.sample' and 'load_env' scripts which are no longer needed

Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos
fixup! fix: Update README.md
ce052a6 
* Fix stupid markdown mistakes

Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos
fix: Update .gitignore for evaluation/output
3110886 
Signed-off-by: Phoevos Kalemkeris <[email protected]>
@binxuan39 binxuan39 marked this pull request as ready for review August 1, 2024 12:17
@phoevos phoevos closed this Aug 1, 2024
@phoevos phoevos deleted the evaluation-rb branch August 1, 2024 13:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xiatianrui1110 xiatianrui1110 xiatianrui1110 left review comments

@GrayNekoBean GrayNekoBean GrayNekoBean left review comments

Assignees

@binxuan39 binxuan39

@xiatianrui1110 xiatianrui1110

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@phoevos @GrayNekoBean @xiatianrui1110 @binxuan39