Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Changes for Rasa Pro #33

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

Changes for Rasa Pro #33

wants to merge 8 commits into from

Conversation

rgstephens
Copy link
Contributor

Changes to address PR #29 and #30 to get a basic Rasa Pro pipeline working.

Example repo can be found here.

This PR adds the following for the test/train action:

          test_type: none
          rasa_pro_license: ${{ secrets.RASA_PRO_LICENSE }}
          openai_api_key: ${{ secrets.OPENAI_API_KEY }}

@rgstephens rgstephens requested a review from soham96 November 29, 2024 20:14
@github-actions GitHub Actions
Copy link

Commit: e9f6b70
Data: default

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1 Story Recognition Micro F1
Rasa Demo - ALL 0.8875 (0.00) 0.8915 (0.00) 0.7750 (0.00) 1.0000 (0.00)

None of the test stories failed - all good!

@github-actions GitHub Actions
Copy link

Commit: e9f6b70
Data: default

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1 Story Recognition Micro F1
Rasa Demo - NLU 0.6737 (no data) 0.7167 (0.16) 0.6023 (-0.06) no data

Intent Cross-Validation Results

class support f1-score confused_with
macro avg 4995 0.43002415327665466 N/A
weighted avg 4995 0.62310015090761240 N/A
faq 880 0.80326197757390430 chitchat(45), enter_data(32)
chitchat 812 0.76458333333333340 faq(38), enter_data(12)
enter_data 759 0.76566125290023190 faq(29), contact_sales(14)
out_of_scope 410 0.09954751131221719 chitchat(133), enter_data(114), faq(41)
affirm 224 0.62857142857142860 enter_data(16), how_to_get_started(16)
technical_question 221 0.24175824175824173 faq(65), chitchat(30)
how_to_get_started 211 0.57494866529774120 chitchat(26), faq(15)
contact_sales 155 0.77142857142857160 faq(11), enter_data(2)
greet 147 0.69508196721311490 enter_data(13), chitchat(10)
signup_newsletter 141 0.92832764505119450 enter_data(2), how_to_get_started(1)
install_rasa 108 0.70848708487084880 faq(7), chitchat(3)
ask_which_events 107 0.87850467289719620 faq(9), install_rasa(2)
deny 100 0.54761904761904770 enter_data(14), how_to_get_started(9)
human_handoff 69 0.48000000000000000 chitchat(12), enter_data(11)
react_positive 65 0.30769230769230770 affirm(18), chitchat(17)
nlu_info 62 0.56074766355140190 chitchat(14), how_to_get_started(8)
switch 54 0.56097560975609760 affirm(13), enter_data(9)
ask_how_contribute 53 0.38235294117647060 contact_sales(9), enter_data(7)
react_negative 47 0.55555555555555550 chitchat(8), how_to_get_started(3)
why_rasa 45 0.23728813559322035 faq(15), affirm(5)
bye 42 0.64615384615384610 affirm(7), enter_data(5)
ask_question_in_forum 42 0.30508474576271183 faq(19), ask_which_events(3)
need_help_broad 41 0.12500000000000000 install_rasa(16), human_handoff(5), faq(4)
thank 39 0.70886075949367100 chitchat(2), enter_data(2)
source_code 34 0.37499999999999994 faq(8), chitchat(6)
canthelp 26 N/A chitchat(10), how_to_get_started(2), enter_data(2)
ask_why_contribute 21 0.08333333333333333 ask_question_in_forum(3), enter_data(3), faq(2)
pipeline_recommendation 19 N/A install_rasa(6), chitchat(4), how_to_get_started(3)
explain 16 0.11111111111111110 chitchat(8), greet(2), react_negative(1)
broken 15 0.21052631578947370 faq(6), enter_data(5)
nlu_generation_tool_recommendation 14 0.13333333333333333 enter_data(7), how_to_get_started(2), react_positive(2)
restart 10 N/A affirm(5), how_to_get_started(3), chitchat(2)
book_demo 6 N/A enter_data(2), faq(2), how_to_get_started(1)

Entity Cross-Validation Results

entity support f1-score precision recall
micro avg 1498 0.71668533034714450 0.81287044877222690 0.64085447263017360
macro avg 1498 0.49300432595120147 0.59644947339545220 0.45263374699363600
weighted avg 1498 0.67830838014538140 0.76438019543366290 0.64085447263017360
product 554 0.72796208530805700 0.76646706586826350 0.69314079422382670
language 297 0.96381578947368420 0.94212218649517690 0.98653198653198650
job_function 160 0.52549019607843140 0.70526315789473680 0.41875000000000000
name 155 0.41628959276018100 0.69696969696969700 0.29677419354838710
nlu_part 94 0.88888888888888880 0.84615384615384610 0.93617021276595750
company 87 0.08333333333333333 0.44444444444444440 0.04597701149425287
current_api 61 0.85470085470085480 0.89285714285714290 0.81967213114754100
location 42 0.72727272727272730 1 0.57142857142857140
user_type 19 0.23529411764705882 0.26666666666666666 0.21052631578947367
entity 16 N/A N/A N/A
install_type 13 N/A N/A N/A

@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 4, 2024

Commit: 14fa811
Data: default

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1 Story Recognition Micro F1
Rasa Demo - ALL 0.8551 (0.00) 0.8848 (0.00) 0.8240 (0.00) 1.0000 (0.00)

None of the test stories failed - all good!

@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 4, 2024

Commit: 14fa811
Data: default

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1 Story Recognition Micro F1
Rasa Demo - NLU 0.6669 (no data) 0.7264 (0.17) 0.6365 (-0.02) no data

Intent Cross-Validation Results

class support f1-score confused_with
macro avg 4995 0.42558221347075303 N/A
weighted avg 4995 0.62748178775493290 N/A
faq 880 0.79365079365079360 chitchat(39), enter_data(27)
chitchat 812 0.75176589303733600 faq(32), enter_data(13)
enter_data 759 0.73113207547169820 out_of_scope(33), faq(33)
out_of_scope 410 0.25500910746812383 chitchat(131), enter_data(82)
affirm 224 0.69565217391304350 enter_data(17), chitchat(10)
technical_question 221 0.27240143369175630 faq(62), chitchat(40)
how_to_get_started 211 0.67039106145251390 faq(39), chitchat(21)
contact_sales 155 0.73352435530085970 faq(10), enter_data(5)
greet 147 0.72549019607843150 enter_data(12), chitchat(6)
signup_newsletter 141 0.89320388349514560 enter_data(1), chitchat(1)
install_rasa 108 0.69879518072289160 faq(8), enter_data(3)
ask_which_events 107 0.80208333333333330 chitchat(18), faq(8)
deny 100 0.40875912408759124 enter_data(25), affirm(8)
human_handoff 69 0.53846153846153840 contact_sales(22), chitchat(10)
react_positive 65 0.19178082191780820 chitchat(27), affirm(11), enter_data(8)
nlu_info 62 0.71287128712871280 chitchat(9), enter_data(8)
switch 54 0.52272727272727280 enter_data(9), install_rasa(8)
ask_how_contribute 53 0.46153846153846156 chitchat(10), enter_data(8)
react_negative 47 0.32142857142857145 chitchat(12), enter_data(7)
why_rasa 45 0.12244897959183675 faq(16), switch(6), affirm(4)
bye 42 0.56666666666666670 enter_data(7), affirm(7)
ask_question_in_forum 42 0.18749999999999997 faq(30), broken(4)
need_help_broad 41 0.21052631578947367 install_rasa(11), faq(6)
thank 39 0.71428571428571420 affirm(3), enter_data(3)
source_code 34 0.38636363636363635 faq(7), chitchat(6)
canthelp 26 N/A chitchat(15), enter_data(5), deny(2)
ask_why_contribute 21 0.10810810810810810 ask_how_contribute(7), chitchat(4), enter_data(2)
pipeline_recommendation 19 0.25000000000000000 chitchat(5), greet(3)
explain 16 N/A chitchat(10), faq(3), affirm(1)
broken 15 0.11764705882352941 faq(5), enter_data(5)
nlu_generation_tool_recommendation 14 0.20000000000000000 enter_data(6), chitchat(3), ask_question_in_forum(2)
restart 10 N/A technical_question(2), enter_data(2), need_help_broad(2)
book_demo 6 N/A faq(3), need_help_broad(1), contact_sales(1)

Entity Cross-Validation Results

entity support f1-score precision recall
micro avg 1498 0.72638313887843430 0.8326143226919759 0.64419225634178900
macro avg 1498 0.54002328917499910 0.6502999188341375 0.50391378503683930
weighted avg 1498 0.68980424692787370 0.7966002996952575 0.64419225634178900
product 554 0.72674418604651160 0.7845188284518828 0.67689530685920570
language 297 0.95894909688013130 0.9358974358974359 0.98316498316498310
job_function 160 0.54771784232365140 0.8148148148148148 0.41250000000000000
name 155 0.44843049327354260 0.7352941176470589 0.32258064516129030
nlu_part 94 0.87562189054726360 0.8224299065420560 0.93617021276595750
company 87 0.12371134020618556 0.6000000000000000 0.06896551724137931
current_api 61 0.84955752212389390 0.9230769230769231 0.78688524590163930
location 42 0.74285714285714300 0.9285714285714286 0.61904761904761910
user_type 19 0.66666666666666660 0.6086956521739131 0.73684210526315790
entity 16 N/A N/A N/A
install_type 13 N/A N/A N/A

@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 4, 2024

Commit: 14fa811
Data: default

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1 Story Recognition Micro F1
Rasa Demo - Configuration 1 0.8809 (0.00) 0.8937 (0.00) 0.7797 (0.00) 1.0000 (0.00)
Rasa Demo - Configuration 2 0.8651 (0.00) 0.8906 (0.00) 0.8363 (0.00) 1.0000 (0.00)

None of the test stories failed - all good!

@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 4, 2024

Commit: c1b5132
Data: default

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1 Story Recognition Micro F1
Rasa Demo - ALL 0.8462 (0.00) 0.8858 (0.00) 0.7493 (0.00) 1.0000 (0.00)

None of the test stories failed - all good!

@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 4, 2024

Commit: c1b5132
Data: default

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1 Story Recognition Micro F1
Rasa Demo - NLU 0.6360 (no data) 0.7181 (0.16) 0.5723 (-0.09) no data

Intent Cross-Validation Results

class support f1-score confused_with
macro avg 4995 0.38311306806347510 N/A
weighted avg 4995 0.59590910477789050 N/A
faq 880 0.81466599698643890 chitchat(32), enter_data(24)
chitchat 812 0.70888468809073720 faq(28), enter_data(8)
enter_data 759 0.75265957446808510 affirm(74), faq(36)
out_of_scope 410 0.19841269841269843 chitchat(146), affirm(62), enter_data(54)
affirm 224 0.53312788906009260 chitchat(14), enter_data(9)
technical_question 221 0.26174496644295303 chitchat(60), faq(55)
how_to_get_started 211 0.62941176470588240 chitchat(29), faq(23)
contact_sales 155 0.71052631578947380 why_rasa(20), faq(9)
greet 147 0.74482758620689650 affirm(13), chitchat(11)
signup_newsletter 141 0.70801033591731270 chitchat(2), affirm(1)
install_rasa 108 0.56140350877192980 chitchat(24), faq(19)
ask_which_events 107 0.67816091954022980 chitchat(16), faq(12)
deny 100 0.39473684210526310 signup_newsletter(21), affirm(19)
human_handoff 69 0.25000000000000000 chitchat(30), contact_sales(8)
react_positive 65 0.25641025641025640 chitchat(25), affirm(13)
nlu_info 62 0.62000000000000000 faq(10), chitchat(5)
switch 54 0.45121951219512200 faq(6), enter_data(4)
ask_how_contribute 53 0.47500000000000003 chitchat(24), enter_data(3)
react_negative 47 0.52380952380952380 chitchat(9), affirm(3)
why_rasa 45 0.22680412371134020 chitchat(11), faq(11)
ask_question_in_forum 42 0.43835616438356160 faq(14), technical_question(4)
bye 42 0.41509433962264153 affirm(16), chitchat(8)
need_help_broad 41 0.13333333333333333 switch(6), install_rasa(5), affirm(4)
thank 39 0.39285714285714290 faq(6), affirm(5)
source_code 34 0.15384615384615385 faq(12), chitchat(10)
canthelp 26 N/A chitchat(13), enter_data(3), affirm(2)
ask_why_contribute 21 0.09090909090909090 chitchat(4), greet(4), why_rasa(3)
pipeline_recommendation 19 0.29629629629629630 chitchat(4), faq(3)
explain 16 N/A chitchat(10), why_rasa(2), switch(1)
broken 15 N/A faq(5), enter_data(3), chitchat(2)
nlu_generation_tool_recommendation 14 0.22222222222222224 chitchat(4), faq(4), technical_question(2)
restart 10 N/A chitchat(5), affirm(1), technical_question(1)
book_demo 6 N/A faq(2), affirm(1), ask_which_events(1)

Entity Cross-Validation Results

entity support f1-score precision recall
micro avg 1498 0.71806674338319900 0.8440036068530208 0.62483311081441930
macro avg 1498 0.52209563431202330 0.6850508976159667 0.48419378488909415
weighted avg 1498 0.67705592365985420 0.8281122859459960 0.62483311081441930
product 554 0.71636011616650530 0.7724425887265136 0.66787003610108310
language 297 0.96013289036544850 0.9475409836065574 0.97306397306397310
job_function 160 0.53448275862068970 0.8611111111111112 0.38750000000000000
name 155 0.39603960396039600 0.8510638297872340 0.25806451612903225
nlu_part 94 0.89583333333333330 0.8775510204081632 0.91489361702127660
company 87 0.08695652173913043 0.8000000000000000 0.04597701149425287
current_api 61 0.85454545454545450 0.9591836734693877 0.77049180327868850
location 42 0.72727272727272730 1 0.57142857142857140
user_type 19 0.57142857142857140 0.4666666666666667 0.73684210526315790
entity 16 N/A N/A N/A
install_type 13 N/A N/A N/A

@rgstephens rgstephens requested review from camattin and ai-adams and removed request for soham96 and ai-adams December 4, 2024 16:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@camattin camattin Awaiting requested review from camattin

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@rgstephens