Skip to content

Commit

Permalink
Add a skill to better understand bullet lists of items (#730)
Browse files Browse the repository at this point in the history
This is a re-submission of [**PR
652**](#652).

**Describe the contribution to the taxonomy**

Understand better when a bullet list of items means that the items are
needed jointly rather than separately.

- Documents often contain bullet lists of items that one must provide,
have available, complete, etc.
- Sometimes Merlinite 7b gets confused whether the listed items must be
provided jointly or separately
- Merlinite 7b tends to think that the items are needed separately, even
though they are needed jointly
- Here we add a few of such examples, with the correct answers

**Input given at the prompt**

In one example, Merlinite 7b was given the following (self-authored)
context document:
```
To apply for public benefit XYZ, provide the following:

* Your full name and date of birth

* Your vehicle license plate number
```
Then the following question was asked:
```
Which of the following is true:

(A) To apply for public benefit XYZ, you must provide both your full name and vehicle license plate number

(B) To apply for public benefit XYZ, you must provide either your full name or vehicle license plate number
```
Most humans would choose (A), but Merlinite 7b chose (B), responding as
follows:

**Response from the original model**

```
The correct answer is (B) To apply for public benefit XYZ, you must provide either your full name or vehicle license plate number.
The statement is true because the application process requires one of these pieces of information, but not necessarily both. 
```

**Response from the fine-tuned model**

(did not have an opportunity to fine tune the model)

**Contribution checklist**

- [x] The contribution was tested with `ilab generate`
- [x] No errors or warnings were produced by `ilab generate`
- [x] All [commits are signed
off](https://github.com/instructlab/taxonomy/blob/main/CONTRIBUTING.md#legal)
(DCO)
- [x] The `qna.yaml` file contains at least 5 `seed_examples`
- [x] The `qna.yaml` file was [linted](https://yamllint.com) and
[prettified](https://onlineyamltools.com/prettify-yaml)
([yaml-validator](https://jsonformatter.org/yaml-validator) can do both)
- [x] An `attribution.txt` file in the same folder as the `qna.yaml`
file.

---------

Signed-off-by: Alexandre Evfimievski <[email protected]>
Signed-off-by: BJ Hargrave <[email protected]>
Signed-off-by: Alexandre Evfimievski <[email protected]>
  • Loading branch information
ae2015 authored Jun 17, 2024
1 parent 3d05322 commit 0e3345c
Show file tree
Hide file tree
Showing 2 changed files with 189 additions and 0 deletions.
31 changes: 31 additions & 0 deletions compositional_skills/linguistics/bullet_lists/attribution.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
created_by: ae2015

Title of work: Understand bullet lists, context of example 1 (out of 5)
Link to work: N/A
Revision: N/A
License of the work: Apache-2.0
Creator names: self-authored

Title of work: Understand bullet lists, context of example 2 (out of 5)
Link to work: https://www.irs.gov/publications/p17
Revision: as of April 2, 2024
License of the work: public domain
Creator names: Internal Revenue Service Authors

Title of work: Understand bullet lists, context of example 3 (out of 5)
Link to work: https://www.irs.gov/publications/p17
Revision: as of April 2, 2024
License of the work: public domain
Creator names: Internal Revenue Service Authors

Title of work: Understand bullet lists, context of example 4 (out of 5)
Link to work: https://www.dmv.ca.gov/portal/dmv-virtual-office/title-transfers/
Revision: as of April 2, 2024
License of the work: public domain
Creator names: California Department of Motor Vehicles Authors

Title of work: Understand bullet lists, context of example 5 (out of 5)
Link to work: https://www.dmv.ca.gov/portal/driver-education-and-safety/educational-materials/fast-facts/requirements-for-a-california-identification-card-ffdl-6/
Revision: as of April 2, 2024
License of the work: public domain
Creator names: California Department of Motor Vehicles Authors
158 changes: 158 additions & 0 deletions compositional_skills/linguistics/bullet_lists/qna.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,158 @@
version: 2
created_by: ae2015
task_description: >
Understand when a bullet list of items means that the items are needed
jointly rather than separately.
seed_examples:
- context: >
To apply for public benefit XYZ, provide the following:
* Your full name and date of birth
* Your vehicle license plate number
question: >
Which of the following is true:
(A) To apply for public benefit XYZ, you must provide both your full name and vehicle license plate number
(B) To apply for public benefit XYZ, you must provide either your full name or vehicle license plate number
answer: >
(A) To apply for public benefit XYZ, you must provide both your full name and vehicle license plate number
- context: >
Form W-2. If you were an employee, you should receive Form W-2 from your employer.
You will need the information from this form to prepare your return. See Form W-2
under Credit for Withholding and Estimated Tax for 2023 in chapter 4. Your employer
is required to provide or send Form W-2 to you no later than January 31, 2024.
If it is mailed, you should allow adequate time to receive it before contacting
your employer. If you still don't get the form by early February, the IRS can help
you by requesting the form from your employer. When you request IRS help, be
prepared to provide the following information.
* Your name, address (including ZIP code), and phone number.
* Your SSN.
* Your dates of employment.
* Your employer's name, address (including ZIP code), and phone number.
question: >
Which of the following is true:
(A) When requesting IRS help, an employee must provide both name and SSN.
(B) When requesting IRS help, an employee must provide either name or SSN.
answer: >
(A) When requesting IRS help, an employee must provide both name and SSN.
- context: >
Foreign address. If your address is outside the United States or its territories,
enter the city name on the appropriate line of your Form 1040 or 1040-SR.
Don't enter any other information on that line, but also complete the spaces below
that line.
1. Foreign country name.
2. Foreign province/state/county.
3. Foreign postal code.
Don't abbreviate the country name. Follow the country's practice for entering
the postal code and the name of the province, county, or state.
question: >
Which of the following is true:
(A) If your address is outside the U.S., you must enter both foreign country name
and foreign postal code.
(B) If your address is outside the U.S., you must enter either foreign country name
or foreign postal code.
answer: >
(A) If your address is outside the U.S., you must enter both foreign country name
and foreign postal code.
- context: >
TITLE TRANSFERS
When you buy a vehicle, you need to transfer the vehicle's title to establish
yourself as the new legal owner.
Before you begin:
1. Have the California Certificate of Title with you. Make sure the title has
been signed by the buyer(s), seller(s), and lienholder (if applicable).
Note: If you do not have the title, complete an Application for Replacement
or Transfer of Title (REG 227).
2. Have the following:
* Your driver's license number
* Vehicle license plate number
* Vehicle identification number (VIN)
* Legal owner (or lienholder) name and address
* Vehicle make, model, and year
* Purchase date and price
3. Be able to pay the transfer fee. All transfer fees are the responsibility
of the buyer.
4. Allow 30 days from the day DMV receives your documents to process your
title transfer.
question: >
Which of the following is true:
(A) To transfer the vehicle's title, you must have both your driver's license
number and vehicle license plate number.
(B) To transfer the vehicle's title, you must have either your driver's license
number or vehicle license plate number.
answer: >
(A) To transfer the vehicle's title, you must have both your driver's license
number and vehicle license plate number.
- context: >
To apply for an ID, you must complete a Driver License and Identification Card
application and pay any fees. For current ID fees, visit www.dmv.ca.gov.
1. Before visiting a DMV office:
* Gather the identity, residency, and SSN documents you need to apply for an
ID card.
* Effective April 2018 DMV will begin offering an online DL and ID application
process. Applicants will complete their electronic application before visiting
DMV. Be sure to bring your application confirmation with you to your office visit.
* Make an appointment before visiting a DMV office. Appointments can be made
online at www.dmv.ca.gov or by calling 1-800-777-0133. You may also verify payment
options available at the DMV office.
2. When you arrive at a DMV office, you will need:
* Correct fee(s) for your application.
* Original or certified documents(s) establishing identity, birth date, true full name.
* Original document establishing proof of SSN if applying for a REAL ID Compliant ID.
* Residency Document.
The DMV representative may give you a number and/or direct you to wait. When it
is your turn, you will be called to a window to process your application.
question: >
Which of the following is true:
(A) When you arrive at a DMV office, you will need both correct fee(s) for your
application and residency document.
(B) When you arrive at a DMV office, you will need either correct fee(s) for your
application or residency document.
answer: >
(A) When you arrive at a DMV office, you will need both correct fee(s) for your
application and residency document.

0 comments on commit 0e3345c

Please sign in to comment.