From 0e3345c8350451be3c4c4967783c8cb3f518dcf7 Mon Sep 17 00:00:00 2001 From: Alexandre Evfimievski Date: Mon, 17 Jun 2024 15:11:51 -0700 Subject: [PATCH] Add a skill to better understand bullet lists of items (#730) This is a re-submission of [**PR 652**](https://github.com/instructlab/taxonomy/pull/652). **Describe the contribution to the taxonomy** Understand better when a bullet list of items means that the items are needed jointly rather than separately. - Documents often contain bullet lists of items that one must provide, have available, complete, etc. - Sometimes Merlinite 7b gets confused whether the listed items must be provided jointly or separately - Merlinite 7b tends to think that the items are needed separately, even though they are needed jointly - Here we add a few of such examples, with the correct answers **Input given at the prompt** In one example, Merlinite 7b was given the following (self-authored) context document: ``` To apply for public benefit XYZ, provide the following: * Your full name and date of birth * Your vehicle license plate number ``` Then the following question was asked: ``` Which of the following is true: (A) To apply for public benefit XYZ, you must provide both your full name and vehicle license plate number (B) To apply for public benefit XYZ, you must provide either your full name or vehicle license plate number ``` Most humans would choose (A), but Merlinite 7b chose (B), responding as follows: **Response from the original model** ``` The correct answer is (B) To apply for public benefit XYZ, you must provide either your full name or vehicle license plate number. The statement is true because the application process requires one of these pieces of information, but not necessarily both. ``` **Response from the fine-tuned model** (did not have an opportunity to fine tune the model) **Contribution checklist** - [x] The contribution was tested with `ilab generate` - [x] No errors or warnings were produced by `ilab generate` - [x] All [commits are signed off](https://github.com/instructlab/taxonomy/blob/main/CONTRIBUTING.md#legal) (DCO) - [x] The `qna.yaml` file contains at least 5 `seed_examples` - [x] The `qna.yaml` file was [linted](https://yamllint.com) and [prettified](https://onlineyamltools.com/prettify-yaml) ([yaml-validator](https://jsonformatter.org/yaml-validator) can do both) - [x] An `attribution.txt` file in the same folder as the `qna.yaml` file. --------- Signed-off-by: Alexandre Evfimievski Signed-off-by: BJ Hargrave Signed-off-by: Alexandre Evfimievski --- .../linguistics/bullet_lists/attribution.txt | 31 ++++ .../linguistics/bullet_lists/qna.yaml | 158 ++++++++++++++++++ 2 files changed, 189 insertions(+) create mode 100644 compositional_skills/linguistics/bullet_lists/attribution.txt create mode 100644 compositional_skills/linguistics/bullet_lists/qna.yaml diff --git a/compositional_skills/linguistics/bullet_lists/attribution.txt b/compositional_skills/linguistics/bullet_lists/attribution.txt new file mode 100644 index 000000000..2ea2ed3aa --- /dev/null +++ b/compositional_skills/linguistics/bullet_lists/attribution.txt @@ -0,0 +1,31 @@ +created_by: ae2015 + +Title of work: Understand bullet lists, context of example 1 (out of 5) +Link to work: N/A +Revision: N/A +License of the work: Apache-2.0 +Creator names: self-authored + +Title of work: Understand bullet lists, context of example 2 (out of 5) +Link to work: https://www.irs.gov/publications/p17 +Revision: as of April 2, 2024 +License of the work: public domain +Creator names: Internal Revenue Service Authors + +Title of work: Understand bullet lists, context of example 3 (out of 5) +Link to work: https://www.irs.gov/publications/p17 +Revision: as of April 2, 2024 +License of the work: public domain +Creator names: Internal Revenue Service Authors + +Title of work: Understand bullet lists, context of example 4 (out of 5) +Link to work: https://www.dmv.ca.gov/portal/dmv-virtual-office/title-transfers/ +Revision: as of April 2, 2024 +License of the work: public domain +Creator names: California Department of Motor Vehicles Authors + +Title of work: Understand bullet lists, context of example 5 (out of 5) +Link to work: https://www.dmv.ca.gov/portal/driver-education-and-safety/educational-materials/fast-facts/requirements-for-a-california-identification-card-ffdl-6/ +Revision: as of April 2, 2024 +License of the work: public domain +Creator names: California Department of Motor Vehicles Authors diff --git a/compositional_skills/linguistics/bullet_lists/qna.yaml b/compositional_skills/linguistics/bullet_lists/qna.yaml new file mode 100644 index 000000000..51e04f0c9 --- /dev/null +++ b/compositional_skills/linguistics/bullet_lists/qna.yaml @@ -0,0 +1,158 @@ +version: 2 +created_by: ae2015 +task_description: > + Understand when a bullet list of items means that the items are needed + jointly rather than separately. +seed_examples: +- context: > + To apply for public benefit XYZ, provide the following: + + * Your full name and date of birth + + * Your vehicle license plate number + question: > + Which of the following is true: + + (A) To apply for public benefit XYZ, you must provide both your full name and vehicle license plate number + + (B) To apply for public benefit XYZ, you must provide either your full name or vehicle license plate number + answer: > + (A) To apply for public benefit XYZ, you must provide both your full name and vehicle license plate number + +- context: > + Form W-2. If you were an employee, you should receive Form W-2 from your employer. + You will need the information from this form to prepare your return. See Form W-2 + under Credit for Withholding and Estimated Tax for 2023 in chapter 4. Your employer + is required to provide or send Form W-2 to you no later than January 31, 2024. + If it is mailed, you should allow adequate time to receive it before contacting + your employer. If you still don't get the form by early February, the IRS can help + you by requesting the form from your employer. When you request IRS help, be + prepared to provide the following information. + + * Your name, address (including ZIP code), and phone number. + + * Your SSN. + + * Your dates of employment. + + * Your employer's name, address (including ZIP code), and phone number. + question: > + Which of the following is true: + + (A) When requesting IRS help, an employee must provide both name and SSN. + + (B) When requesting IRS help, an employee must provide either name or SSN. + answer: > + (A) When requesting IRS help, an employee must provide both name and SSN. + +- context: > + Foreign address. If your address is outside the United States or its territories, + enter the city name on the appropriate line of your Form 1040 or 1040-SR. + Don't enter any other information on that line, but also complete the spaces below + that line. + + 1. Foreign country name. + + 2. Foreign province/state/county. + + 3. Foreign postal code. + + Don't abbreviate the country name. Follow the country's practice for entering + the postal code and the name of the province, county, or state. + question: > + Which of the following is true: + + (A) If your address is outside the U.S., you must enter both foreign country name + and foreign postal code. + + (B) If your address is outside the U.S., you must enter either foreign country name + or foreign postal code. + answer: > + (A) If your address is outside the U.S., you must enter both foreign country name + and foreign postal code. + +- context: > + TITLE TRANSFERS + + When you buy a vehicle, you need to transfer the vehicle's title to establish + yourself as the new legal owner. + + Before you begin: + + 1. Have the California Certificate of Title with you. Make sure the title has + been signed by the buyer(s), seller(s), and lienholder (if applicable). + + Note: If you do not have the title, complete an Application for Replacement + or Transfer of Title (REG 227). + + 2. Have the following: + + * Your driver's license number + + * Vehicle license plate number + + * Vehicle identification number (VIN) + + * Legal owner (or lienholder) name and address + + * Vehicle make, model, and year + + * Purchase date and price + + 3. Be able to pay the transfer fee. All transfer fees are the responsibility + of the buyer. + + 4. Allow 30 days from the day DMV receives your documents to process your + title transfer. + question: > + Which of the following is true: + + (A) To transfer the vehicle's title, you must have both your driver's license + number and vehicle license plate number. + + (B) To transfer the vehicle's title, you must have either your driver's license + number or vehicle license plate number. + answer: > + (A) To transfer the vehicle's title, you must have both your driver's license + number and vehicle license plate number. + +- context: > + To apply for an ID, you must complete a Driver License and Identification Card + application and pay any fees. For current ID fees, visit www.dmv.ca.gov. + + 1. Before visiting a DMV office: + + * Gather the identity, residency, and SSN documents you need to apply for an + ID card. + + * Effective April 2018 DMV will begin offering an online DL and ID application + process. Applicants will complete their electronic application before visiting + DMV. Be sure to bring your application confirmation with you to your office visit. + + * Make an appointment before visiting a DMV office. Appointments can be made + online at www.dmv.ca.gov or by calling 1-800-777-0133. You may also verify payment + options available at the DMV office. + + 2. When you arrive at a DMV office, you will need: + + * Correct fee(s) for your application. + + * Original or certified documents(s) establishing identity, birth date, true full name. + + * Original document establishing proof of SSN if applying for a REAL ID Compliant ID. + + * Residency Document. + + The DMV representative may give you a number and/or direct you to wait. When it + is your turn, you will be called to a window to process your application. + question: > + Which of the following is true: + + (A) When you arrive at a DMV office, you will need both correct fee(s) for your + application and residency document. + + (B) When you arrive at a DMV office, you will need either correct fee(s) for your + application or residency document. + answer: > + (A) When you arrive at a DMV office, you will need both correct fee(s) for your + application and residency document.