Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ask smapi get-result-for-nlu-evaluations result is missing the values for expected slot names #385

Open
arisen1145 opened this issue Mar 9, 2021 · 1 comment

Comments

@arisen1145
Copy link

arisen1145 commented Mar 9, 2021

I'm submitting a...


[ ] Regression (a behavior that used to work and stopped working in a new release)
[x] Bug report  
[ ] Performance issue
[ ] Feature request
[ ] Documentation issue or request 
[ ] Other... Please describe: 

Expected Behavior

The API response of ask smapi get-result-for-nlu-evaluations includes the correct data for the property "expected" (same as defined in the NLU annotation).

Current Behavior

API response of ask smapi get-result-for-nlu-evaluations has a wrong (missing) data in the property "expected". The slot name and value which is defined in the NLU annotation is missing. The test is still passing, but the actual and expected are not matching.
Same behavior is in the UI of the in NLU Evaluation Results, see attached Screenshot

{
  "_links": {
        ...
    }
  },
  "paginationContext": {
    "totalCount": 1
  },
  "testCases": [
    {
      "actual": {
        "intent": {
          "confirmationStatus": "NONE",
          "name": "HelloWorldIntent",
          "slots": {
            "name": {
              "confirmationStatus": "NONE",
              "name": "name",
              "slotValue": {
                "type": "Simple",
                "value": "mike"
              }
            }
          }
        }
      },
      "expected": [
        {
          "intent": {
            "name": "HelloWorldIntent",
            "slots": {
              "name": {}
            }
          }
        }
      ],
      "inputs": {
        "utterance": "hello mike"
      },
      "status": "PASSED"
    }
  ],
  "totalFailed": 0
}

CLI Snapshot
If applicable, add screenshots to help explain your problem.

Steps to Reproduce (for bugs)

  • Create new sample skill.
  • Go to Build -> Intents -> HelloWorldIntent -> add an utterance "hello {name}" with slot Type Amazon.Firstname
  • Go to Build -> Intents -> Annotation Sets -> Create a NLU Annotation Set with
    Utterance: hello mike
    Expected Intent: HelloWorldIntent
    Expected slots: slot name=name, slot value=mike
  • Go to Evaluate Model -> NLU Evaluation -> Select your Annotation and click Run evaluation
  • Copy the evaluationId and do this cli request
    ask smapi get-result-for-nlu-evaluations -s amzn1.ask.skill.xyz--evaluation-id xyz

Possible Solution

Your Environment and Context

  • ask-cli version: 2.22.4
  • Operating System and version: macOS 11.2.2
  • Node.js version used for development: v12.16.1
  • NPM version used for development: 6.13.4

Screenshot 2021-03-09 at 17 42 59

@CamdenFoucht
Copy link
Contributor

Hi @arisen1145, thank you for creating this bug report. This seems like a problem with the SMAPI service, so we will try to investigate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants