\\n
\\n\\n\"\n}\n[/block]\n\n\nThis notebook demonstrates how to create a chatbot (single turn) that answers user questions based on technical documentation made available to the model.\n\nWe use the `aws-documentation` dataset ([link](https://github.com/siagholami/aws-documentation/tree/main)) for representativeness. This dataset contains 26k+ AWS documentation pages, preprocessed into 120k+ chunks, and 100 questions based on real user questions.\n\nWe proceed as follows:\n1. Embed the AWS documentation into a vector database using Cohere embeddings and `llama_index`\n2. Build a retriever using Cohere's `rerank` for better accuracy, lower inference costs and lower latency\n3. Create model answers for the eval set of 100 questions\n4. Evaluate the model answers against the golden answers of the eval set\n\n\n## Setup\n\n\n```python\n%%capture\n!pip install cohere datasets llama_index llama-index-llms-cohere llama-index-embeddings-cohere\n```\n\n\n```python\nimport cohere\nimport datasets\nfrom llama_index.core import StorageContext, VectorStoreIndex, load_index_from_storage\nfrom llama_index.core.schema import TextNode\nfrom llama_index.embeddings.cohere import CohereEmbedding\nimport pandas as pd\n\nimport json\nfrom pathlib import Path\nfrom tqdm import tqdm\nfrom typing import List\n\n```\n\n\n```python\napi_key = \"\" # Creating a QA Bot From Technical Documentation
\\n\\n
\\n\\n\\nAdvanced Document Parsing For Enterprises
\\n\\n \\n
\\n\\n\"\n}\n[/block]\n\n\n## Introduction\n\nThe bread and butter of natural language processing technology is text. Once we can reduce a set of data into text, we can do all kinds of things with it: question answering, summarization, classification, sentiment analysis, searching and indexing, and more.\n\nIn the context of enterprise Retrieval Augmented Generation (RAG), the information is often locked in complex file types such as PDFs. These formats are made for sharing information between humans, but not so much with language models.\n\nIn this notebook, we will use a real-world pharmaceutical drug label to test out various performant approaches to parsing PDFs. This will allow us to use [Cohere's Command-R model](https://txt.cohere.com/command-r/) in a RAG setting to answer questions and asks about this label, such as \"I need a succinct summary of the compound name, indication, route of administration, and mechanism of action of\" a given pharmaceutical.\n\n[block:html]{\"html\":\"\\n ![\\\"Giannis](\\\"https://files.readme.io/73153cb-giannis.jpeg\\\")
\\n \\n
\\n \\n \\n \\n \\n ![\\\"Justin](\\\"https://files.readme.io/3678fac-Justin.jpg\\\")
\\n \\n
\\n \\n\\n \\n ![\\\"Document](\\\"https://github.com/cohere-ai/notebooks/raw/main/notebooks/images/document-parsing-2.png\\\")
\"}[/block]\n\n\n## PDF Parsing\n\nWe will go over five proprietary as well as open source options for processing PDFs. The parsing mechanisms demonstrated in the following sections are\n- [Google Document AI](#gcp)\n- [AWS Textract](#aws)\n- [Unstructured.io](#unstructured)\n- [LlamaParse](#llama)\n- [pdf2image + pytesseract](#pdf2image)\n\nBy way of example, we will be parsing a [21-page PDF](https://www.accessdata.fda.gov/drugsatfda_docs/label/2023/215500s000lbl.pdf) containing the label for a recent FDA drug approval, the beginning of which is shown below. Then, we will perform a series of basic RAG tasks with our different parsings and evaluate their performance.\n\n[block:html]{\"html\":\"![\\\"Drug](\\\"https://github.com/cohere-ai/notebooks/raw/main/notebooks/images/document-parsing-1.png\\\")
\"}[/block]\n\n## Getting Set Up\n\nBefore we dive into the technical weeds, we need to set up the notebook's runtime and filesystem environments. The code cells below do the following:\n- Install required libraries\n- Confirm that data dependencies from the GitHub repo have been downloaded. These will be under `data/document-parsing` and contain the following:\n - the PDF document that we will be working with, `fda-approved-drug.pdf` (this can also be found here: https://www.accessdata.fda.gov/drugsatfda_docs/label/2023/215500s000lbl.pdf)\n - precomputed parsed documents for each parsing solution. While the point of this notebook is to illustrate how this is done, we provide the parsed final results to allow readers to skip ahead to the RAG section without having to set up the required infrastructure for each solution.)\n- Add utility functions needed for later sections\n\n\n```python\n%%capture\n! sudo apt install tesseract-ocr poppler-utils\n! pip install \"cohere<5\" fsspec hnswlib google-cloud-documentai google-cloud-storage boto3 langchain-text-splitters llama_parse pytesseract pdf2image pandas\n\n```\n\n\n```python\ndata_dir = \"data/document-parsing\"\nsource_filename = \"example-drug-label\"\nextension = \"pdf\"\n```\n\n\n```python\nfrom pathlib import Path\n\nsources = [\"gcp\", \"aws\", \"unstructured-io\", \"llamaparse-text\", \"llamaparse-markdown\", \"pytesseract\"]\n\nfilenames = [\"{}-parsed-fda-approved-drug.txt\".format(source) for source in sources]\nfilenames.append(\"fda-approved-drug.pdf\")\n\nfor filename in filenames: \n file_path = Path(f\"{data_dir}/{filename}\")\n if file_path.is_file() == False:\n print(f\"File {filename} not found at {data_dir}!\")\n```\n\n### Utility Functions\nMake sure to include the notebook's utility functions in the runtime.\n\n\n```python\ndef store_document(path: str, doc_content: str):\n with open(path, 'w') as f:\n f.write(doc_content)\n```\n\n\n```python\nimport json\n\ndef insert_citations_in_order(text, citations, documents):\n \"\"\"\n A helper function to pretty print citations.\n \"\"\"\n\n citations_reference = {}\n for index, doc in enumerate(documents):\n citations_reference[index] = doc\n\n offset = 0\n # Process citations in the order they were provided\n for citation in citations:\n # Adjust start/end with offset\n start, end = citation['start'] + offset, citation['end'] + offset\n citation_numbers = []\n for doc_id in citation[\"document_ids\"]:\n for citation_index, doc in citations_reference.items():\n if doc[\"id\"] == doc_id:\n citation_numbers.append(citation_index)\n references = \"(\" + \", \".join(\"[{}]\".format(num) for num in citation_numbers) + \")\"\n modification = f'{text[start:end]} {references}'\n # Replace the cited text with its bolded version + placeholder\n text = text[:start] + modification + text[end:]\n # Update the offset for subsequent replacements\n offset += len(modification) - (end - start)\n\n # Add the citations at the bottom of the text\n text_with_citations = f'{text}'\n citations_reference = [\"[{}]: {}\".format(x[\"id\"], x[\"text\"]) for x in citations_reference.values()]\n\n return text_with_citations, \"\\n\".join(citations_reference)\n```\n\n\n```python\ndef format_docs_for_chat(documents):\n return [{\"id\": str(index), \"text\": x} for index, x in enumerate(documents)]\n```\n\n## Document Parsing Solutions\n\nFor demonstration purposes, we have collected and saved the parsed documents from each solution in this notebook. Skip to the [next section](#document-questions) to run RAG with Command-R on the pre-fetched versions. You can find all parsed resources in detail at the link [here](https://github.com/gchatz22/temp-cohere-resources/tree/main/data).\n\n\n### Solution 1: Google Cloud Document AI [[Back to Solutions]](#top)\n\nDocument AI helps developers create high-accuracy processors to extract, classify, and split documents.\n\nExternal documentation: https://cloud.google.com/document-ai\n\n#### Parsing the document\n\nThe following block can be executed in one of two ways:\n- Inside a Google Vertex AI environment\n - No authentication needed\n- From this notebook\n - Authentication is needed\n - There are pointers inside the code on which lines to uncomment in order to make this work\n\n**Note: You can skip to the next block if you want to use the pre-existing parsed version.**\n\n\n```python\n\"\"\"\nExtracted from https://cloud.google.com/document-ai/docs/samples/documentai-batch-process-document\n\"\"\"\n\nimport re\nfrom typing import Optional\n\nfrom google.api_core.client_options import ClientOptions\nfrom google.api_core.exceptions import InternalServerError\nfrom google.api_core.exceptions import RetryError\nfrom google.cloud import documentai # type: ignore\nfrom google.cloud import storage\n\nproject_id = \"\"\nlocation = \"\"\nprocessor_id = \"\"\ngcs_output_uri = \"\"\n# credentials_file = \"populate if you are running in a non Vertex AI environment.\"\ngcs_input_prefix = \"\"\n\n\ndef batch_process_documents(\n project_id: str,\n location: str,\n processor_id: str,\n gcs_output_uri: str,\n gcs_input_prefix: str,\n timeout: int = 400\n) -> None:\n parsed_documents = []\n\n # Client configs\n opts = ClientOptions(api_endpoint=f\"{location}-documentai.googleapis.com\")\n # With credentials\n # opts = ClientOptions(api_endpoint=f\"{location}-documentai.googleapis.com\", credentials_file=credentials_file)\n\n client = documentai.DocumentProcessorServiceClient(client_options=opts)\n processor_name = client.processor_path(project_id, location, processor_id)\n\n # Input storage configs\n gcs_prefix = documentai.GcsPrefix(gcs_uri_prefix=gcs_input_prefix)\n input_config = documentai.BatchDocumentsInputConfig(gcs_prefix=gcs_prefix)\n\n # Output storage configs\n gcs_output_config = documentai.DocumentOutputConfig.GcsOutputConfig(gcs_uri=gcs_output_uri, field_mask=None)\n output_config = documentai.DocumentOutputConfig(gcs_output_config=gcs_output_config)\n storage_client = storage.Client()\n # With credentials\n # storage_client = storage.Client.from_service_account_json(json_credentials_path=credentials_file)\n\n # Batch process docs request\n request = documentai.BatchProcessRequest(\n name=processor_name,\n input_documents=input_config,\n document_output_config=output_config,\n )\n\n # batch_process_documents returns a long running operation\n operation = client.batch_process_documents(request)\n\n # Continually polls the operation until it is complete.\n # This could take some time for larger files\n try:\n print(f\"Waiting for operation {operation.operation.name} to complete...\")\n operation.result(timeout=timeout)\n except (RetryError, InternalServerError) as e:\n print(e.message)\n\n # Get output document information from completed operation metadata\n metadata = documentai.BatchProcessMetadata(operation.metadata)\n if metadata.state != documentai.BatchProcessMetadata.State.SUCCEEDED:\n raise ValueError(f\"Batch Process Failed: {metadata.state_message}\")\n\n print(\"Output files:\")\n # One process per Input Document\n for process in list(metadata.individual_process_statuses):\n matches = re.match(r\"gs://(.*?)/(.*)\", process.output_gcs_destination)\n if not matches:\n print(\"Could not parse output GCS destination:\", process.output_gcs_destination)\n continue\n\n output_bucket, output_prefix = matches.groups()\n output_blobs = storage_client.list_blobs(output_bucket, prefix=output_prefix)\n\n # Document AI may output multiple JSON files per source file\n # (Large documents get split in multiple file \"versions\" doc --> parsed_doc_0 + parsed_doc_1 ...)\n for blob in output_blobs:\n # Document AI should only output JSON files to GCS\n if blob.content_type != \"application/json\":\n print(f\"Skipping non-supported file: {blob.name} - Mimetype: {blob.content_type}\")\n continue\n\n # Download JSON file as bytes object and convert to Document Object\n print(f\"Fetching {blob.name}\")\n document = documentai.Document.from_json(blob.download_as_bytes(), ignore_unknown_fields=True)\n # Store the filename and the parsed versioned document content as a tuple\n parsed_documents.append((blob.name.split(\"/\")[-1].split(\".\")[0], document.text))\n\n print(\"Finished document parsing process.\")\n return parsed_documents\n\n# Call service\n# versioned_parsed_documents = batch_process_documents(\n# project_id=project_id,\n# location=location,\n# processor_id=processor_id,\n# gcs_output_uri=gcs_output_uri,\n# gcs_input_prefix=gcs_input_prefix\n# )\n```\n\n\n```python\n\"\"\"\nPost process parsed document and store it locally.\nMake sure to run this in a Google Vertex AI environment or include a credentials file.\n\"\"\"\n\n\"\"\"\nfrom pathlib import Path\nfrom collections import defaultdict\n\nparsed_documents = []\ncombined_versioned_parsed_documents = defaultdict(list)\n\n# Assemble versioned documents together ({\"doc_name\": [(0, doc_content_0), (1, doc_content_1), ...]}).\nfor filename, doc_content in versioned_parsed_documents:\n filename, version = \"-\".join(filename.split(\"-\")[:-1]), filename.split(\"-\")[-1]\n combined_versioned_parsed_documents[filename].append((version, doc_content))\n\n# Sort documents by version and join the content together.\nfor filename, docs in combined_versioned_parsed_documents.items():\n doc_content = \" \".join([x[1] for x in sorted(docs, key=lambda x: x[0])])\n parsed_documents.append((filename, doc_content))\n\n# Store parsed documents in local storage.\nfor filename, doc_content in parsed_documents:\n file_path = \"{}/{}-parsed-{}.txt\".format(data_dir, \"gcp\", source_filename)\n store_document(file_path, doc_content)\n\"\"\"\n```\n\n#### Visualize the parsed document\n\n\n```python\nfilename = \"gcp-parsed-{}.txt\".format(source_filename)\nwith open(\"{}/{}\".format(data_dir, filename), \"r\") as doc:\n parsed_document = doc.read()\n\nprint(parsed_document[:1000])\n```\n\n\n### Solution 2: AWS Textract [[Back to Solutions]](#top)\n\n[Amazon Textract](https://aws.amazon.com/textract/) is an OCR service offered by AWS. It can detect text, forms, tables, and more in PDFs and images. In this section, we go over how to use Textract's asynchronous API.\n\n#### Parsing the document\n\nWe assume that you are working within the AWS ecosystem (from a SageMaker notebook, EC2 instance, a Lambda function, etc.) with valid credentials. Much of the code here is from supplemental materials created by AWS and offered here:\n\n- https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/python/example_code/textract\n- https://github.com/aws-samples/textract-paragraph-identification/tree/main\n\nAt minimum, you will need access to the following AWS resources to get started:\n\n- Textract\n- an S3 bucket containing the document(s) to process - in this case, our `example-drug-label.pdf` file\n- an SNS topic that Textract can publish to. This is used to send a notification that parsing is complete.\n- an IAM role that Textract will assume, granting access to the S3 bucket and SNS topic\n\nFirst, we bring in the `TextractWrapper` class provided in the [AWS Code Examples repository](https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/python/example_code/textract/textract_wrapper.py). This class makes it simpler to interface with the Textract service.\n\n\n```python\n# source: https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/python/example_code/textract\n\n# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# SPDX-License-Identifier: Apache-2.0\n\n\"\"\"\nPurpose\n\nShows how to use the AWS SDK for Python (Boto3) with Amazon Textract to\ndetect text, form, and table elements in document images.\n\"\"\"\n\nimport json\nimport logging\nfrom botocore.exceptions import ClientError\n\nlogger = logging.getLogger(__name__)\n\n\n# snippet-start:[python.example_code.textract.TextractWrapper]\nclass TextractWrapper:\n \"\"\"Encapsulates Textract functions.\"\"\"\n\n def __init__(self, textract_client, s3_resource, sqs_resource):\n \"\"\"\n :param textract_client: A Boto3 Textract client.\n :param s3_resource: A Boto3 Amazon S3 resource.\n :param sqs_resource: A Boto3 Amazon SQS resource.\n \"\"\"\n self.textract_client = textract_client\n self.s3_resource = s3_resource\n self.sqs_resource = sqs_resource\n\n # snippet-end:[python.example_code.textract.TextractWrapper]\n\n # snippet-start:[python.example_code.textract.DetectDocumentText]\n def detect_file_text(self, *, document_file_name=None, document_bytes=None):\n \"\"\"\n Detects text elements in a local image file or from in-memory byte data.\n The image must be in PNG or JPG format.\n\n :param document_file_name: The name of a document image file.\n :param document_bytes: In-memory byte data of a document image.\n :return: The response from Amazon Textract, including a list of blocks\n that describe elements detected in the image.\n \"\"\"\n if document_file_name is not None:\n with open(document_file_name, \"rb\") as document_file:\n document_bytes = document_file.read()\n try:\n response = self.textract_client.detect_document_text(\n Document={\"Bytes\": document_bytes}\n )\n logger.info(\"Detected %s blocks.\", len(response[\"Blocks\"]))\n except ClientError:\n logger.exception(\"Couldn't detect text.\")\n raise\n else:\n return response\n\n # snippet-end:[python.example_code.textract.DetectDocumentText]\n\n # snippet-start:[python.example_code.textract.AnalyzeDocument]\n def analyze_file(\n self, feature_types, *, document_file_name=None, document_bytes=None\n ):\n \"\"\"\n Detects text and additional elements, such as forms or tables, in a local image\n file or from in-memory byte data.\n The image must be in PNG or JPG format.\n\n :param feature_types: The types of additional document features to detect.\n :param document_file_name: The name of a document image file.\n :param document_bytes: In-memory byte data of a document image.\n :return: The response from Amazon Textract, including a list of blocks\n that describe elements detected in the image.\n \"\"\"\n if document_file_name is not None:\n with open(document_file_name, \"rb\") as document_file:\n document_bytes = document_file.read()\n try:\n response = self.textract_client.analyze_document(\n Document={\"Bytes\": document_bytes}, FeatureTypes=feature_types\n )\n logger.info(\"Detected %s blocks.\", len(response[\"Blocks\"]))\n except ClientError:\n logger.exception(\"Couldn't detect text.\")\n raise\n else:\n return response\n\n # snippet-end:[python.example_code.textract.AnalyzeDocument]\n\n # snippet-start:[python.example_code.textract.helper.prepare_job]\n def prepare_job(self, bucket_name, document_name, document_bytes):\n \"\"\"\n Prepares a document image for an asynchronous detection job by uploading\n the image bytes to an Amazon S3 bucket. Amazon Textract must have permission\n to read from the bucket to process the image.\n\n :param bucket_name: The name of the Amazon S3 bucket.\n :param document_name: The name of the image stored in Amazon S3.\n :param document_bytes: The image as byte data.\n \"\"\"\n try:\n bucket = self.s3_resource.Bucket(bucket_name)\n bucket.upload_fileobj(document_bytes, document_name)\n logger.info(\"Uploaded %s to %s.\", document_name, bucket_name)\n except ClientError:\n logger.exception(\"Couldn't upload %s to %s.\", document_name, bucket_name)\n raise\n\n # snippet-end:[python.example_code.textract.helper.prepare_job]\n\n # snippet-start:[python.example_code.textract.helper.check_job_queue]\n def check_job_queue(self, queue_url, job_id):\n \"\"\"\n Polls an Amazon SQS queue for messages that indicate a specified Textract\n job has completed.\n\n :param queue_url: The URL of the Amazon SQS queue to poll.\n :param job_id: The ID of the Textract job.\n :return: The status of the job.\n \"\"\"\n status = None\n try:\n queue = self.sqs_resource.Queue(queue_url)\n messages = queue.receive_messages()\n if messages:\n msg_body = json.loads(messages[0].body)\n msg = json.loads(msg_body[\"Message\"])\n if msg.get(\"JobId\") == job_id:\n messages[0].delete()\n status = msg.get(\"Status\")\n logger.info(\n \"Got message %s with status %s.\", messages[0].message_id, status\n )\n else:\n logger.info(\"No messages in queue %s.\", queue_url)\n except ClientError:\n logger.exception(\"Couldn't get messages from queue %s.\", queue_url)\n else:\n return status\n\n # snippet-end:[python.example_code.textract.helper.check_job_queue]\n\n # snippet-start:[python.example_code.textract.StartDocumentTextDetection]\n def start_detection_job(\n self, bucket_name, document_file_name, sns_topic_arn, sns_role_arn\n ):\n \"\"\"\n Starts an asynchronous job to detect text elements in an image stored in an\n Amazon S3 bucket. Textract publishes a notification to the specified Amazon SNS\n topic when the job completes.\n The image must be in PNG, JPG, or PDF format.\n\n :param bucket_name: The name of the Amazon S3 bucket that contains the image.\n :param document_file_name: The name of the document image stored in Amazon S3.\n :param sns_topic_arn: The Amazon Resource Name (ARN) of an Amazon SNS topic\n where the job completion notification is published.\n :param sns_role_arn: The ARN of an AWS Identity and Access Management (IAM)\n role that can be assumed by Textract and grants permission\n to publish to the Amazon SNS topic.\n :return: The ID of the job.\n \"\"\"\n try:\n response = self.textract_client.start_document_text_detection(\n DocumentLocation={\n \"S3Object\": {\"Bucket\": bucket_name, \"Name\": document_file_name}\n },\n NotificationChannel={\n \"SNSTopicArn\": sns_topic_arn,\n \"RoleArn\": sns_role_arn,\n },\n )\n job_id = response[\"JobId\"]\n logger.info(\n \"Started text detection job %s on %s.\", job_id, document_file_name\n )\n except ClientError:\n logger.exception(\"Couldn't detect text in %s.\", document_file_name)\n raise\n else:\n return job_id\n\n # snippet-end:[python.example_code.textract.StartDocumentTextDetection]\n\n # snippet-start:[python.example_code.textract.GetDocumentTextDetection]\n def get_detection_job(self, job_id):\n \"\"\"\n Gets data for a previously started text detection job.\n\n :param job_id: The ID of the job to retrieve.\n :return: The job data, including a list of blocks that describe elements\n detected in the image.\n \"\"\"\n try:\n response = self.textract_client.get_document_text_detection(JobId=job_id)\n job_status = response[\"JobStatus\"]\n logger.info(\"Job %s status is %s.\", job_id, job_status)\n except ClientError:\n logger.exception(\"Couldn't get data for job %s.\", job_id)\n raise\n else:\n return response\n\n # snippet-end:[python.example_code.textract.GetDocumentTextDetection]\n\n # snippet-start:[python.example_code.textract.StartDocumentAnalysis]\n def start_analysis_job(\n self,\n bucket_name,\n document_file_name,\n feature_types,\n sns_topic_arn,\n sns_role_arn,\n ):\n \"\"\"\n Starts an asynchronous job to detect text and additional elements, such as\n forms or tables, in an image stored in an Amazon S3 bucket. Textract publishes\n a notification to the specified Amazon SNS topic when the job completes.\n The image must be in PNG, JPG, or PDF format.\n\n :param bucket_name: The name of the Amazon S3 bucket that contains the image.\n :param document_file_name: The name of the document image stored in Amazon S3.\n :param feature_types: The types of additional document features to detect.\n :param sns_topic_arn: The Amazon Resource Name (ARN) of an Amazon SNS topic\n where job completion notification is published.\n :param sns_role_arn: The ARN of an AWS Identity and Access Management (IAM)\n role that can be assumed by Textract and grants permission\n to publish to the Amazon SNS topic.\n :return: The ID of the job.\n \"\"\"\n try:\n response = self.textract_client.start_document_analysis(\n DocumentLocation={\n \"S3Object\": {\"Bucket\": bucket_name, \"Name\": document_file_name}\n },\n NotificationChannel={\n \"SNSTopicArn\": sns_topic_arn,\n \"RoleArn\": sns_role_arn,\n },\n FeatureTypes=feature_types,\n )\n job_id = response[\"JobId\"]\n logger.info(\n \"Started text analysis job %s on %s.\", job_id, document_file_name\n )\n except ClientError:\n logger.exception(\"Couldn't analyze text in %s.\", document_file_name)\n raise\n else:\n return job_id\n\n # snippet-end:[python.example_code.textract.StartDocumentAnalysis]\n\n # snippet-start:[python.example_code.textract.GetDocumentAnalysis]\n def get_analysis_job(self, job_id):\n \"\"\"\n Gets data for a previously started detection job that includes additional\n elements.\n\n :param job_id: The ID of the job to retrieve.\n :return: The job data, including a list of blocks that describe elements\n detected in the image.\n \"\"\"\n try:\n response = self.textract_client.get_document_analysis(JobId=job_id)\n job_status = response[\"JobStatus\"]\n logger.info(\"Job %s status is %s.\", job_id, job_status)\n except ClientError:\n logger.exception(\"Couldn't get data for job %s.\", job_id)\n raise\n else:\n return response\n\n\n# snippet-end:[python.example_code.textract.GetDocumentAnalysis]\n```\n\nNext, we set up Textract and S3, and provide this to an instance of `TextractWrapper`.\n\n\n```python\nimport boto3\n\ntextract_client = boto3.client('textract')\ns3_client = boto3.client('s3')\n\ntextractWrapper = TextractWrapper(textract_client, s3_client, None)\n```\n\nWe are now ready to make calls to Textract. At a high level, Textract has two modes: synchronous and asynchronous. Synchronous calls return the parsed output once it is completed. As of the time of writing (March 2024), however, multipage PDF processing is only supported [asynchronously](https://docs.aws.amazon.com/textract/latest/dg/sync.html). So for our purposes here, we will only explore the asynchronous route.\n\nAsynchronous calls follow the below process:\n\n1. Send a request to Textract with an SNS topic, S3 bucket, and the name (key) of the document inside that bucket to process. Textract returns a Job ID that can be used to track the status of the request\n2. Textract fetches the document from S3 and processes it\n3. Once the request is complete, Textract sends out a message to the SNS topic. This can be used in conjunction with other services such as Lambda or SQS for downstream processes.\n4. The parsed results can be fetched from Textract in chunks via the job ID.\n\n\n```python\nbucket_name = \"your-bucket-name\"\nsns_topic_arn = \"your-sns-arn\" # this can be found under the topic you created in the Amazon SNS dashboard\nsns_role_arn = \"sns-role-arn\" # this is an IAM role that allows Textract to interact with SNS\n\nfile_name = \"example-drug-label.pdf\"\n```\n\n\n```python\n# kick off a text detection job. This returns a job ID.\njob_id = textractWrapper.start_detection_job(bucket_name=bucket_name, document_file_name=file_name,\n sns_topic_arn=sns_topic_arn, sns_role_arn=sns_role_arn)\n```\n\nOnce the job completes, this will return a dictionary with the following keys:\n\n```dict_keys(['DocumentMetadata', 'JobStatus', 'NextToken', 'Blocks', 'AnalyzeDocumentModelVersion', 'ResponseMetadata'])```\n\nThis response corresponds to one chunk of information parsed by Textract. The number of chunks a document is parsed into depends on the length of the document. The two keys we are most interested in are `Blocks` and `NextToken`. `Blocks` contains all of the information that was extracted from this chunk, while `NextToken` tells us what chunk comes next, if any.\n\nTextract returns an information-rich representation of the extracted text, such as their position on the page and hierarchical relationships with other entities, all the way down to the individual word level. Since we are only interested in the raw text, we need a way to parse through all of the chunks and their `Blocks`. Lucky for us, Amazon provides some [helper functions](https://github.com/aws-samples/textract-paragraph-identification/tree/main) for this purpose, which we utilize below.\n\n\n```python\ndef get_text_results_from_textract(job_id):\n response = textract_client.get_document_text_detection(JobId=job_id)\n collection_of_textract_responses = []\n pages = [response]\n\n collection_of_textract_responses.append(response)\n\n while 'NextToken' in response:\n next_token = response['NextToken']\n response = textract_client.get_document_text_detection(JobId=job_id, NextToken=next_token)\n pages.append(response)\n collection_of_textract_responses.append(response)\n return collection_of_textract_responses\n\ndef get_the_text_with_required_info(collection_of_textract_responses):\n total_text = []\n total_text_with_info = []\n running_sequence_number = 0\n\n font_sizes_and_line_numbers = {}\n for page in collection_of_textract_responses:\n per_page_text = []\n blocks = page['Blocks']\n for block in blocks:\n if block['BlockType'] == 'LINE':\n block_text_dict = {}\n running_sequence_number += 1\n block_text_dict.update(text=block['Text'])\n block_text_dict.update(page=block['Page'])\n block_text_dict.update(left_indent=round(block['Geometry']['BoundingBox']['Left'], 2))\n font_height = round(block['Geometry']['BoundingBox']['Height'], 3)\n line_number = running_sequence_number\n block_text_dict.update(font_height=round(block['Geometry']['BoundingBox']['Height'], 3))\n block_text_dict.update(indent_from_top=round(block['Geometry']['BoundingBox']['Top'], 2))\n block_text_dict.update(text_width=round(block['Geometry']['BoundingBox']['Width'], 2))\n block_text_dict.update(line_number=running_sequence_number)\n\n if font_height in font_sizes_and_line_numbers:\n line_numbers = font_sizes_and_line_numbers[font_height]\n line_numbers.append(line_number)\n font_sizes_and_line_numbers[font_height] = line_numbers\n else:\n line_numbers = []\n line_numbers.append(line_number)\n font_sizes_and_line_numbers[font_height] = line_numbers\n\n total_text.append(block['Text'])\n per_page_text.append(block['Text'])\n total_text_with_info.append(block_text_dict)\n\n return total_text, total_text_with_info, font_sizes_and_line_numbers\n\ndef get_text_with_line_spacing_info(total_text_with_info):\n i = 1\n text_info_with_line_spacing_info = []\n while (i < len(total_text_with_info) - 1):\n previous_line_info = total_text_with_info[i - 1]\n current_line_info = total_text_with_info[i]\n next_line_info = total_text_with_info[i + 1]\n if current_line_info['page'] == next_line_info['page'] and previous_line_info['page'] == current_line_info[\n 'page']:\n line_spacing_after = round((next_line_info['indent_from_top'] - current_line_info['indent_from_top']), 2)\n spacing_with_prev = round((current_line_info['indent_from_top'] - previous_line_info['indent_from_top']), 2)\n current_line_info.update(line_space_before=spacing_with_prev)\n current_line_info.update(line_space_after=line_spacing_after)\n text_info_with_line_spacing_info.append(current_line_info)\n else:\n text_info_with_line_spacing_info.append(None)\n i += 1\n return text_info_with_line_spacing_info\n```\n\nWe feed in the Job ID from before into the function `get_text_results_from_textract` to fetch all of the chunks associated with this job. Then, we pass the resulting list into `get_the_text_with_required_info` and `get_text_with_line_spacing_info` to organize the text into lines.\n\nFinally, we can concatenate the lines into one string to pass into our downstream RAG pipeline.\n\n\n```python\nall_text = \"\\n\".join([line[\"text\"] if line else \"\" for line in text_info_with_line_spacing])\n\nwith open(f\"aws-parsed-{source_filename}.txt\", \"w\") as f:\n f.write(all_text)\n```\n\n#### Visualize the parsed document\n\n\n```python\nfilename = \"aws-parsed-{}.txt\".format(source_filename)\nwith open(\"{}/{}\".format(data_dir, filename), \"r\") as doc:\n parsed_document = doc.read()\n\nprint(parsed_document[:1000])\n```\n\n\n### Solution 3: Unstructured.io [[Back to Solutions]](#top)\n\nUnstructured.io provides libraries with open-source components for pre-processing text documents such as PDFs, HTML and Word Documents.\n\nExternal documentation: https://github.com/Unstructured-IO/unstructured-api\n\n#### Parsing the document\n\nThe guide assumes an endpoint exists that hosts this service. The API is offered in two forms\n1. [a hosted version](https://unstructured.io/)\n2. [an OSS docker image](https://github.com/Unstructured-IO/unstructured-api?tab=readme-ov-file#dizzy-instructions-for-using-the-docker-image)\n\n**Note: You can skip to the next block if you want to use the pre-existing parsed version.**\n\n\n```python\nimport os\nimport requests\n\nUNSTRUCTURED_URL = \"\" # enter service endpoint\n\nparsed_documents = []\n\ninput_path = \"{}/{}.{}\".format(data_dir, source_filename, extension)\nwith open(input_path, 'rb') as file_data:\n response = requests.post(\n url=UNSTRUCTURED_URL,\n files={\"files\": (\"{}.{}\".format(source_filename, extension), file_data)},\n data={\n \"output_format\": (None, \"application/json\"),\n \"stratergy\": \"hi_res\",\n \"pdf_infer_table_structure\": \"true\",\n \"include_page_breaks\": \"true\"\n },\n headers={\"Accept\": \"application/json\"}\n )\n\nparsed_response = response.json()\n\nparsed_document = \" \".join([parsed_entry[\"text\"] for parsed_entry in parsed_response])\nprint(\"Parsed {}\".format(source_filename))\n```\n\n\n```python\n\"\"\"\nPost process parsed document and store it locally.\n\"\"\"\n\nfile_path = \"{}/{}-parsed-fda-approved-drug.txt\".format(data_dir, \"unstructured-io\")\nstore_document(file_path, parsed_document)\n```\n\n#### Visualize the parsed document\n\n\n```python\nfilename = \"unstructured-io-parsed-{}.txt\".format(source_filename)\nwith open(\"{}/{}\".format(data_dir, filename), \"r\") as doc:\n parsed_document = doc.read()\n\nprint(parsed_document[:1000])\n```\n\n\n\n### Solution 4: LlamaParse [[Back to Solutions]](#top)\n\nLlamaParse is an API created by LlamaIndex to efficiently parse and represent files for efficient retrieval and context augmentation using LlamaIndex frameworks.\n\nExternal documentation: https://github.com/run-llama/llama_parse\n\n#### Parsing the document\n\nThe following block uses the LlamaParse cloud offering. You can learn more and fetch a respective API key for the service [here](https://cloud.llamaindex.ai/parse).\n\nParsing documents with LlamaParse offers an option for two output modes both of which we will explore and compare below\n- Text\n- Markdown\n\n**Note: You can skip to the next block if you want to use the pre-existing parsed version.**\n\n\n```python\nimport os\nfrom llama_parse import LlamaParse\n\nimport nest_asyncio # needed to notebook env\nnest_asyncio.apply() # needed to notebook env\n\nllama_index_api_key = \"{API_KEY}\"\ninput_path = \"{}/{}.{}\".format(data_dir, source_filename, extension)\n```\n\n\n```python\n# Text mode\ntext_parser = LlamaParse(\n api_key=llama_index_api_key,\n result_type=\"text\"\n)\n\ntext_response = text_parser.load_data(input_path)\ntext_parsed_document = \" \".join([parsed_entry.text for parsed_entry in text_response])\n\nprint(\"Parsed {} to text\".format(source_filename))\n```\n\n\n```python\n\"\"\"\nPost process parsed document and store it locally.\n\"\"\"\n\nfile_path = \"{}/{}-text-parsed-fda-approved-drug.txt\".format(data_dir, \"llamaparse\")\nstore_document(file_path, text_parsed_document)\n```\n\n\n```python\n# Markdown mode\nmarkdown_parser = LlamaParse(\n api_key=llama_index_api_key,\n result_type=\"markdown\"\n)\n\nmarkdown_response = markdown_parser.load_data(input_path)\nmarkdown_parsed_document = \" \".join([parsed_entry.text for parsed_entry in markdown_response])\n\nprint(\"Parsed {} to markdown\".format(source_filename))\n```\n\n\n```python\n\"\"\"\nPost process parsed document and store it locally.\n\"\"\"\n\nfile_path = \"{}/{}-markdown-parsed-fda-approved-drug.txt\".format(data_dir, \"llamaparse\")\nstore_document(file_path, markdown_parsed_document)\n```\n\n#### Visualize the parsed document\n\n\n```python\n# Text parsing\n\nfilename = \"llamaparse-text-parsed-{}.txt\".format(source_filename)\n\nwith open(\"{}/{}\".format(data_dir, filename), \"r\") as doc:\n parsed_document = doc.read()\n \nprint(parsed_document[:1000])\n```\n\n\n```python\n# Markdown parsing\n\nfilename = \"llamaparse-markdown-parsed-fda-approved-drug.txt\"\nwith open(\"{}/{}\".format(data_dir, filename), \"r\") as doc:\n parsed_document = doc.read()\n \nprint(parsed_document[:1000])\n```\n\n\n\n### Solution 5: pdf2image + pytesseract [[Back to Solutions]](#top)\n\nThe final parsing method we examine does not rely on cloud services, but rather relies on two libraries: `pdf2image`, and `pytesseract`. `pytesseract` lets you perform OCR locally on images, but not PDF files. So, we first convert our PDF into a set of images via `pdf2image`.\n\n#### Parsing the document\n\n\n```python\nfrom matplotlib import pyplot as plt\nfrom pdf2image import convert_from_path\nimport pytesseract\n```\n\n\n```python\n# pdf2image extracts as a list of PIL.Image objects\npages = convert_from_path(filename)\n```\n\n\n```python\n# we look at the first page as a sanity check:\n\nplt.imshow(pages[0])\nplt.axis('off')\nplt.show()\n```\n\nNow, we can process the image of each page with `pytesseract` and concatenate the results to get our parsed document.\n\n\n```python\nlabel_ocr_pytesseract = \"\".join([pytesseract.image_to_string(page) for page in pages])\n```\n\n\n```python\nprint(label_ocr_pytesseract[:200])\n```\n\n HIGHLIGHTS OF PRESCRIBING INFORMATION\n \n These highlights do not include all the information needed to use\n IWILFIN™ safely and effectively. See full prescribing information for\n IWILFIN.\n \n IWILFIN™ (eflor\n\n\n\n```python\nlabel_ocr_pytesseract = \"\".join([pytesseract.image_to_string(page) for page in pages])\n\nwith open(f\"pytesseract-parsed-{source_filename}.txt\", \"w\") as f:\n f.write(label_ocr_pytesseract)\n```\n\n#### Visualize the parsed document\n\n\n```python\nfilename = \"pytesseract-parsed-{}.txt\".format(source_filename)\nwith open(\"{}/{}\".format(data_dir, filename), \"r\") as doc:\n parsed_document = doc.read()\n\nprint(parsed_document[:1000])\n```\n\n\n## Document Questions\n\nWe can now ask a set of simple + complex questions and see how each parsing solution performs with Command-R. The questions are\n- **What are the most common adverse reactions of Iwilfin?**\n - Task: Simple information extraction\n- **What is the recommended dosage of IWILFIN on body surface area between 0.5 and 0.75?**\n - Task: Tabular data extraction\n- **I need a succinct summary of the compound name, indication, route of administration, and mechanism of action of Iwilfin.**\n - Task: Overall document summary\n\n\n```python\nimport cohere\nco = cohere.Client(api_key=\"{API_KEY}\")\n```\n\n\n```python\n\"\"\"\nDocument Questions\n\"\"\"\nprompt = \"What are the most common adverse reactions of Iwilfin?\"\n# prompt = \"What is the recommended dosage of Iwilfin on body surface area between 0.5 m2 and 0.75 m2?\"\n# prompt = \"I need a succinct summary of the compound name, indication, route of administration, and mechanism of action of Iwilfin.\"\n\n\"\"\"\nChoose one of the above solutions\n\"\"\"\nsource = \"gcp\"\n# source = \"aws\"\n# source = \"unstructured-io\"\n# source = \"llamaparse-text\"\n# source = \"llamaparse-markdown\"\n# source = \"pytesseract\"\n```\n\n## Data Ingestion\n\n\nIn order to set up our RAG implementation, we need to separate the parsed text into chunks and load the chunks to an index. The index will allow us to retrieve relevant passages from the document for different queries. Here, we use a simple implementation of indexing using the `hnswlib` library. Note that there are many different indexing solutions that are appropriate for specific production use cases.\n\n\n```python\n\"\"\"\nRead parsed document content and chunk data\n\"\"\"\n\nimport os\nfrom langchain_text_splitters import RecursiveCharacterTextSplitter\n\ndocuments = []\n\nwith open(\"{}/{}-parsed-fda-approved-drug.txt\".format(data_dir, source), \"r\") as doc:\ndoc_content = doc.read()\n\n\"\"\"\nPersonal notes on chunking\nhttps://medium.com/@ayhamboucher/llm-based-context-splitter-for-large-documents-445d3f02b01b\n\"\"\"\n\n\n# Chunk doc content\ntext_splitter = RecursiveCharacterTextSplitter(\n chunk_size=512,\n chunk_overlap=200,\n length_function=len,\n is_separator_regex=False\n)\n\n# Split the text into chunks with some overlap\nchunks_ = text_splitter.create_documents([doc_content])\ndocuments = [c.page_content for c in chunks_]\n\nprint(\"Source document has been broken down to {} chunks\".format(len(documents)))\n```\n\n\n```python\n\"\"\"\nEmbed document chunks\n\"\"\"\ndocument_embeddings = co.embed(texts=documents, model=\"embed-english-v3.0\", input_type=\"search_document\").embeddings\n```\n\n\n```python\n\"\"\"\nCreate document index and add embedded chunks\n\"\"\"\n\nimport hnswlib\n\nindex = hnswlib.Index(space='ip', dim=1024) # space: inner product\nindex.init_index(max_elements=len(document_embeddings), ef_construction=512, M=64)\nindex.add_items(document_embeddings, list(range(len(document_embeddings))))\nprint(\"Count:\", index.element_count)\n```\n\n Count: 115\n\n\n## Retrieval\n\nIn this step, we use k-nearest neighbors to fetch the most relevant documents for our query. Once the nearest neighbors are retrieved, we use Cohere's reranker to reorder the documents in the most relevant order with regards to our input search query.\n\n\n```python\n\"\"\"\nEmbed search query\nFetch k nearest neighbors\n\"\"\"\n\nquery_emb = co.embed(texts=[prompt], model='embed-english-v3.0', input_type=\"search_query\").embeddings\ndefault_knn = 10\nknn = default_knn if default_knn <= index.element_count else index.element_count\nresult = index.knn_query(query_emb, k=knn)\nneighbors = [(result[0][0][i], result[1][0][i]) for i in range(len(result[0][0]))]\nrelevant_docs = [documents[x[0]] for x in sorted(neighbors, key=lambda x: x[1])]\n```\n\n\n```python\n\"\"\"\nRerank retrieved documents\n\"\"\"\n\nrerank_results = co.rerank(query=prompt, documents=relevant_docs, top_n=3, model='rerank-english-v2.0').results\nreranked_relevant_docs = format_docs_for_chat([x.document[\"text\"] for x in rerank_results])\n```\n\n## Final Step: Call Command-R + RAG!\n\n\n```python\n\"\"\"\nCall the /chat endpoint with command-r\n\"\"\"\n\nresponse = co.chat(\n message=prompt,\n model=\"command-r\",\n documents=reranked_relevant_docs\n)\n\ncited_response, citations_reference = insert_citations_in_order(response.text, response.citations, reranked_relevant_docs)\nprint(cited_response)\nprint(\"\\n\")\nprint(\"References:\")\nprint(citations_reference)\n```\n\n## Head-to-head Comparisons\n\nRun the code cells below to make head to head comparisons of the different parsing techniques across different questions.\n\n\n```python\nimport pandas as pd\nresults = pd.read_csv(\"{}/results-table.csv\".format(data_dir))\n```\n\n\n```python\nquestion = input(\"\"\"\nQuestion 1: What are the most common adverse reactions of Iwilfin?\nQuestion 2: What is the recommended dosage of Iwilfin on body surface area between 0.5 m2 and 0.75 m2?\nQuestion 3: I need a succinct summary of the compound name, indication, route of administration, and mechanism of action of Iwilfin.\n\nPick which question you want to see (1,2,3): \"\"\")\nreferences = input(\"Do you want to see the references as well? References are long and noisy (y/n): \")\nprint(\"\\n\\n\")\n\nindex = {\"1\": 0, \"2\": 3, \"3\": 6}[question]\n\nfor src in [\"gcp\", \"aws\", \"unstructured-io\", \"llamaparse-text\", \"llamaparse-markdown\", \"pytesseract\"]:\n print(\"| {} |\".format(src))\n print(\"\\n\")\n print(results[src][index])\n if references == \"y\":\n print(\"\\n\")\n print(\"References:\")\n print(results[src][index+1])\n print(\"\\n\")\n```\n\n \n Question 1: What are the most common adverse reactions of Iwilfin?\n Question 2: What is the recommended dosage of Iwilfin on body surface area between 0.5 m2 and 0.75 m2?\n Question 3: I need a succinct summary of the compound name, indication, route of administration, and mechanism of action of Iwilfin.\n \n Pick which question you want to see (1,2,3): 3\n Do you want to see the references as well? References are long and noisy (y/n): n\n \n \n \n | gcp |\n \n \n Compound Name: eflornithine hydrochloride ([0], [1], [2]) (IWILFIN ([1])™)\n \n Indication: used to reduce the risk of relapse in adult and paediatric patients with high-risk neuroblastoma (HRNB) ([1], [3]), who have responded at least partially to prior multiagent, multimodality therapy. ([1], [3], [4])\n \n Route of Administration: IWILFIN™ tablets ([1], [3], [4]) are taken orally twice daily ([3], [4]), with doses ranging from 192 to 768 mg based on body surface area. ([3], [4])\n \n Mechanism of Action: IWILFIN™ is an ornithine decarboxylase inhibitor. ([0], [2])\n \n \n \n | aws |\n \n \n Compound Name: eflornithine ([0], [1], [2], [3]) (IWILFIN ([0])™)\n \n Indication: used to reduce the risk of relapse ([0], [3]) in adults ([0], [3]) and paediatric patients ([0], [3]) with high-risk neuroblastoma (HRNB) ([0], [3]) who have responded to prior therapies. ([0], [3], [4])\n \n Route of Administration: Oral ([2], [4])\n \n Mechanism of Action: IWILFIN is an ornithine decarboxylase inhibitor. ([1])\n \n \n | unstructured-io |\n \n \n Compound Name: Iwilfin ([1], [2], [3], [4]) (eflornithine) ([0], [2], [3], [4])\n \n Indication: Iwilfin is indicated to reduce the risk of relapse ([1], [3]) in adult and paediatric patients ([1], [3]) with high-risk neuroblastoma (HRNB) ([1], [3]), who have responded to prior anti-GD2 ([1]) immunotherapy ([1], [4]) and multi-modality therapy. ([1])\n \n Route of Administration: Oral ([0], [3])\n \n Mechanism of Action: Iwilfin is an ornithine decarboxylase inhibitor. ([1], [2], [3], [4])\n \n \n | llamaparse-text |\n \n \n Compound Name: IWILFIN ([2], [3]) (eflornithine) ([3])\n \n Indication: IWILFIN is used to reduce the risk of relapse ([1], [2], [3]) in adult and paediatric patients ([1], [2], [3]) with high-risk neuroblastoma (HRNB) ([1], [2], [3]), who have responded at least partially to certain prior therapies. ([2], [3])\n \n Route of Administration: IWILFIN is administered as a tablet. ([2])\n \n Mechanism of Action: IWILFIN is an ornithine decarboxylase inhibitor. ([0], [1], [4])\n \n \n | llamaparse-markdown |\n \n \n Compound Name: IWILFIN ([1], [2]) (eflornithine) ([1])\n \n Indication: IWILFIN is indicated to reduce the risk of relapse ([1], [2]) in adult and paediatric patients ([1], [2]) with high-risk neuroblastoma (HRNB) ([1], [2]), who have responded at least partially ([1], [2], [3]) to prior anti-GD2 immunotherapy ([1], [2]) and multiagent, multimodality therapy. ([1], [2], [3])\n \n Route of Administration: Oral ([0], [1], [3], [4])\n \n Mechanism of Action: IWILFIN acts as an ornithine decarboxylase inhibitor. ([1])\n \n \n | pytesseract |\n \n \n Compound Name: IWILFIN™ ([0], [2]) (eflornithine) ([0], [2])\n \n Indication: IWILFIN is indicated to reduce the risk of relapse ([0], [2]) in adult and paediatric patients ([0], [2]) with high-risk neuroblastoma (HRNB) ([0], [2]), who have responded positively to prior anti-GD2 immunotherapy and multiagent, multimodality therapy. ([0], [2], [4])\n \n Route of Administration: IWILFIN is administered orally ([0], [1], [3], [4]), in the form of a tablet. ([1])\n \n Mechanism of Action: IWILFIN acts as an ornithine decarboxylase inhibitor. ([0])",
+ "body": "[block:html]\n{\n \"html\": \"\\n\\n\\n
\\n\\n\\nAdvanced Document Parsing For Enterprises
\\n\\n \\n
\\n\\n\"\n}\n[/block]\n\n\n## Introduction\n\nThe bread and butter of natural language processing technology is text. Once we can reduce a set of data into text, we can do all kinds of things with it: question answering, summarization, classification, sentiment analysis, searching and indexing, and more.\n\nIn the context of enterprise Retrieval Augmented Generation (RAG), the information is often locked in complex file types such as PDFs. These formats are made for sharing information between humans, but not so much with language models.\n\nIn this notebook, we will use a real-world pharmaceutical drug label to test out various performant approaches to parsing PDFs. This will allow us to use [Cohere's Command-R model](https://cohere.com/blog/command-r/) in a RAG setting to answer questions and asks about this label, such as \"I need a succinct summary of the compound name, indication, route of administration, and mechanism of action of\" a given pharmaceutical.\n\n[block:html]{\"html\":\"\\n \\n \\n
\\n \\n \\n \\n \\n \\n \\n
\\n \\n\\n \\n \"}[/block]\n\n\n## PDF Parsing\n\nWe will go over five proprietary as well as open source options for processing PDFs. The parsing mechanisms demonstrated in the following sections are\n- [Google Document AI](#gcp)\n- [AWS Textract](#aws)\n- [Unstructured.io](#unstructured)\n- [LlamaParse](#llama)\n- [pdf2image + pytesseract](#pdf2image)\n\nBy way of example, we will be parsing a [21-page PDF](https://www.accessdata.fda.gov/drugsatfda_docs/label/2023/215500s000lbl.pdf) containing the label for a recent FDA drug approval, the beginning of which is shown below. Then, we will perform a series of basic RAG tasks with our different parsings and evaluate their performance.\n\n[block:html]{\"html\":\"\"}[/block]\n\n## Getting Set Up\n\nBefore we dive into the technical weeds, we need to set up the notebook's runtime and filesystem environments. The code cells below do the following:\n- Install required libraries\n- Confirm that data dependencies from the GitHub repo have been downloaded. These will be under `data/document-parsing` and contain the following:\n - the PDF document that we will be working with, `fda-approved-drug.pdf` (this can also be found here: https://www.accessdata.fda.gov/drugsatfda_docs/label/2023/215500s000lbl.pdf)\n - precomputed parsed documents for each parsing solution. While the point of this notebook is to illustrate how this is done, we provide the parsed final results to allow readers to skip ahead to the RAG section without having to set up the required infrastructure for each solution.)\n- Add utility functions needed for later sections\n\n\n```python\n%%capture\n! sudo apt install tesseract-ocr poppler-utils\n! pip install \"cohere<5\" fsspec hnswlib google-cloud-documentai google-cloud-storage boto3 langchain-text-splitters llama_parse pytesseract pdf2image pandas\n\n```\n\n\n```python\ndata_dir = \"data/document-parsing\"\nsource_filename = \"example-drug-label\"\nextension = \"pdf\"\n```\n\n\n```python\nfrom pathlib import Path\n\nsources = [\"gcp\", \"aws\", \"unstructured-io\", \"llamaparse-text\", \"llamaparse-markdown\", \"pytesseract\"]\n\nfilenames = [\"{}-parsed-fda-approved-drug.txt\".format(source) for source in sources]\nfilenames.append(\"fda-approved-drug.pdf\")\n\nfor filename in filenames: \n file_path = Path(f\"{data_dir}/{filename}\")\n if file_path.is_file() == False:\n print(f\"File {filename} not found at {data_dir}!\")\n```\n\n### Utility Functions\nMake sure to include the notebook's utility functions in the runtime.\n\n\n```python\ndef store_document(path: str, doc_content: str):\n with open(path, 'w') as f:\n f.write(doc_content)\n```\n\n\n```python\nimport json\n\ndef insert_citations_in_order(text, citations, documents):\n \"\"\"\n A helper function to pretty print citations.\n \"\"\"\n\n citations_reference = {}\n for index, doc in enumerate(documents):\n citations_reference[index] = doc\n\n offset = 0\n # Process citations in the order they were provided\n for citation in citations:\n # Adjust start/end with offset\n start, end = citation['start'] + offset, citation['end'] + offset\n citation_numbers = []\n for doc_id in citation[\"document_ids\"]:\n for citation_index, doc in citations_reference.items():\n if doc[\"id\"] == doc_id:\n citation_numbers.append(citation_index)\n references = \"(\" + \", \".join(\"[{}]\".format(num) for num in citation_numbers) + \")\"\n modification = f'{text[start:end]} {references}'\n # Replace the cited text with its bolded version + placeholder\n text = text[:start] + modification + text[end:]\n # Update the offset for subsequent replacements\n offset += len(modification) - (end - start)\n\n # Add the citations at the bottom of the text\n text_with_citations = f'{text}'\n citations_reference = [\"[{}]: {}\".format(x[\"id\"], x[\"text\"]) for x in citations_reference.values()]\n\n return text_with_citations, \"\\n\".join(citations_reference)\n```\n\n\n```python\ndef format_docs_for_chat(documents):\n return [{\"id\": str(index), \"text\": x} for index, x in enumerate(documents)]\n```\n\n## Document Parsing Solutions\n\nFor demonstration purposes, we have collected and saved the parsed documents from each solution in this notebook. Skip to the [next section](#document-questions) to run RAG with Command-R on the pre-fetched versions. You can find all parsed resources in detail at the link [here](https://github.com/gchatz22/temp-cohere-resources/tree/main/data).\n\n\n### Solution 1: Google Cloud Document AI [[Back to Solutions]](#top)\n\nDocument AI helps developers create high-accuracy processors to extract, classify, and split documents.\n\nExternal documentation: https://cloud.google.com/document-ai\n\n#### Parsing the document\n\nThe following block can be executed in one of two ways:\n- Inside a Google Vertex AI environment\n - No authentication needed\n- From this notebook\n - Authentication is needed\n - There are pointers inside the code on which lines to uncomment in order to make this work\n\n**Note: You can skip to the next block if you want to use the pre-existing parsed version.**\n\n\n```python\n\"\"\"\nExtracted from https://cloud.google.com/document-ai/docs/samples/documentai-batch-process-document\n\"\"\"\n\nimport re\nfrom typing import Optional\n\nfrom google.api_core.client_options import ClientOptions\nfrom google.api_core.exceptions import InternalServerError\nfrom google.api_core.exceptions import RetryError\nfrom google.cloud import documentai # type: ignore\nfrom google.cloud import storage\n\nproject_id = \"\"\nlocation = \"\"\nprocessor_id = \"\"\ngcs_output_uri = \"\"\n# credentials_file = \"populate if you are running in a non Vertex AI environment.\"\ngcs_input_prefix = \"\"\n\n\ndef batch_process_documents(\n project_id: str,\n location: str,\n processor_id: str,\n gcs_output_uri: str,\n gcs_input_prefix: str,\n timeout: int = 400\n) -> None:\n parsed_documents = []\n\n # Client configs\n opts = ClientOptions(api_endpoint=f\"{location}-documentai.googleapis.com\")\n # With credentials\n # opts = ClientOptions(api_endpoint=f\"{location}-documentai.googleapis.com\", credentials_file=credentials_file)\n\n client = documentai.DocumentProcessorServiceClient(client_options=opts)\n processor_name = client.processor_path(project_id, location, processor_id)\n\n # Input storage configs\n gcs_prefix = documentai.GcsPrefix(gcs_uri_prefix=gcs_input_prefix)\n input_config = documentai.BatchDocumentsInputConfig(gcs_prefix=gcs_prefix)\n\n # Output storage configs\n gcs_output_config = documentai.DocumentOutputConfig.GcsOutputConfig(gcs_uri=gcs_output_uri, field_mask=None)\n output_config = documentai.DocumentOutputConfig(gcs_output_config=gcs_output_config)\n storage_client = storage.Client()\n # With credentials\n # storage_client = storage.Client.from_service_account_json(json_credentials_path=credentials_file)\n\n # Batch process docs request\n request = documentai.BatchProcessRequest(\n name=processor_name,\n input_documents=input_config,\n document_output_config=output_config,\n )\n\n # batch_process_documents returns a long running operation\n operation = client.batch_process_documents(request)\n\n # Continually polls the operation until it is complete.\n # This could take some time for larger files\n try:\n print(f\"Waiting for operation {operation.operation.name} to complete...\")\n operation.result(timeout=timeout)\n except (RetryError, InternalServerError) as e:\n print(e.message)\n\n # Get output document information from completed operation metadata\n metadata = documentai.BatchProcessMetadata(operation.metadata)\n if metadata.state != documentai.BatchProcessMetadata.State.SUCCEEDED:\n raise ValueError(f\"Batch Process Failed: {metadata.state_message}\")\n\n print(\"Output files:\")\n # One process per Input Document\n for process in list(metadata.individual_process_statuses):\n matches = re.match(r\"gs://(.*?)/(.*)\", process.output_gcs_destination)\n if not matches:\n print(\"Could not parse output GCS destination:\", process.output_gcs_destination)\n continue\n\n output_bucket, output_prefix = matches.groups()\n output_blobs = storage_client.list_blobs(output_bucket, prefix=output_prefix)\n\n # Document AI may output multiple JSON files per source file\n # (Large documents get split in multiple file \"versions\" doc --> parsed_doc_0 + parsed_doc_1 ...)\n for blob in output_blobs:\n # Document AI should only output JSON files to GCS\n if blob.content_type != \"application/json\":\n print(f\"Skipping non-supported file: {blob.name} - Mimetype: {blob.content_type}\")\n continue\n\n # Download JSON file as bytes object and convert to Document Object\n print(f\"Fetching {blob.name}\")\n document = documentai.Document.from_json(blob.download_as_bytes(), ignore_unknown_fields=True)\n # Store the filename and the parsed versioned document content as a tuple\n parsed_documents.append((blob.name.split(\"/\")[-1].split(\".\")[0], document.text))\n\n print(\"Finished document parsing process.\")\n return parsed_documents\n\n# Call service\n# versioned_parsed_documents = batch_process_documents(\n# project_id=project_id,\n# location=location,\n# processor_id=processor_id,\n# gcs_output_uri=gcs_output_uri,\n# gcs_input_prefix=gcs_input_prefix\n# )\n```\n\n\n```python\n\"\"\"\nPost process parsed document and store it locally.\nMake sure to run this in a Google Vertex AI environment or include a credentials file.\n\"\"\"\n\n\"\"\"\nfrom pathlib import Path\nfrom collections import defaultdict\n\nparsed_documents = []\ncombined_versioned_parsed_documents = defaultdict(list)\n\n# Assemble versioned documents together ({\"doc_name\": [(0, doc_content_0), (1, doc_content_1), ...]}).\nfor filename, doc_content in versioned_parsed_documents:\n filename, version = \"-\".join(filename.split(\"-\")[:-1]), filename.split(\"-\")[-1]\n combined_versioned_parsed_documents[filename].append((version, doc_content))\n\n# Sort documents by version and join the content together.\nfor filename, docs in combined_versioned_parsed_documents.items():\n doc_content = \" \".join([x[1] for x in sorted(docs, key=lambda x: x[0])])\n parsed_documents.append((filename, doc_content))\n\n# Store parsed documents in local storage.\nfor filename, doc_content in parsed_documents:\n file_path = \"{}/{}-parsed-{}.txt\".format(data_dir, \"gcp\", source_filename)\n store_document(file_path, doc_content)\n\"\"\"\n```\n\n#### Visualize the parsed document\n\n\n```python\nfilename = \"gcp-parsed-{}.txt\".format(source_filename)\nwith open(\"{}/{}\".format(data_dir, filename), \"r\") as doc:\n parsed_document = doc.read()\n\nprint(parsed_document[:1000])\n```\n\n\n### Solution 2: AWS Textract [[Back to Solutions]](#top)\n\n[Amazon Textract](https://aws.amazon.com/textract/) is an OCR service offered by AWS. It can detect text, forms, tables, and more in PDFs and images. In this section, we go over how to use Textract's asynchronous API.\n\n#### Parsing the document\n\nWe assume that you are working within the AWS ecosystem (from a SageMaker notebook, EC2 instance, a Lambda function, etc.) with valid credentials. Much of the code here is from supplemental materials created by AWS and offered here:\n\n- https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/python/example_code/textract\n- https://github.com/aws-samples/textract-paragraph-identification/tree/main\n\nAt minimum, you will need access to the following AWS resources to get started:\n\n- Textract\n- an S3 bucket containing the document(s) to process - in this case, our `example-drug-label.pdf` file\n- an SNS topic that Textract can publish to. This is used to send a notification that parsing is complete.\n- an IAM role that Textract will assume, granting access to the S3 bucket and SNS topic\n\nFirst, we bring in the `TextractWrapper` class provided in the [AWS Code Examples repository](https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/python/example_code/textract/textract_wrapper.py). This class makes it simpler to interface with the Textract service.\n\n\n```python\n# source: https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/python/example_code/textract\n\n# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# SPDX-License-Identifier: Apache-2.0\n\n\"\"\"\nPurpose\n\nShows how to use the AWS SDK for Python (Boto3) with Amazon Textract to\ndetect text, form, and table elements in document images.\n\"\"\"\n\nimport json\nimport logging\nfrom botocore.exceptions import ClientError\n\nlogger = logging.getLogger(__name__)\n\n\n# snippet-start:[python.example_code.textract.TextractWrapper]\nclass TextractWrapper:\n \"\"\"Encapsulates Textract functions.\"\"\"\n\n def __init__(self, textract_client, s3_resource, sqs_resource):\n \"\"\"\n :param textract_client: A Boto3 Textract client.\n :param s3_resource: A Boto3 Amazon S3 resource.\n :param sqs_resource: A Boto3 Amazon SQS resource.\n \"\"\"\n self.textract_client = textract_client\n self.s3_resource = s3_resource\n self.sqs_resource = sqs_resource\n\n # snippet-end:[python.example_code.textract.TextractWrapper]\n\n # snippet-start:[python.example_code.textract.DetectDocumentText]\n def detect_file_text(self, *, document_file_name=None, document_bytes=None):\n \"\"\"\n Detects text elements in a local image file or from in-memory byte data.\n The image must be in PNG or JPG format.\n\n :param document_file_name: The name of a document image file.\n :param document_bytes: In-memory byte data of a document image.\n :return: The response from Amazon Textract, including a list of blocks\n that describe elements detected in the image.\n \"\"\"\n if document_file_name is not None:\n with open(document_file_name, \"rb\") as document_file:\n document_bytes = document_file.read()\n try:\n response = self.textract_client.detect_document_text(\n Document={\"Bytes\": document_bytes}\n )\n logger.info(\"Detected %s blocks.\", len(response[\"Blocks\"]))\n except ClientError:\n logger.exception(\"Couldn't detect text.\")\n raise\n else:\n return response\n\n # snippet-end:[python.example_code.textract.DetectDocumentText]\n\n # snippet-start:[python.example_code.textract.AnalyzeDocument]\n def analyze_file(\n self, feature_types, *, document_file_name=None, document_bytes=None\n ):\n \"\"\"\n Detects text and additional elements, such as forms or tables, in a local image\n file or from in-memory byte data.\n The image must be in PNG or JPG format.\n\n :param feature_types: The types of additional document features to detect.\n :param document_file_name: The name of a document image file.\n :param document_bytes: In-memory byte data of a document image.\n :return: The response from Amazon Textract, including a list of blocks\n that describe elements detected in the image.\n \"\"\"\n if document_file_name is not None:\n with open(document_file_name, \"rb\") as document_file:\n document_bytes = document_file.read()\n try:\n response = self.textract_client.analyze_document(\n Document={\"Bytes\": document_bytes}, FeatureTypes=feature_types\n )\n logger.info(\"Detected %s blocks.\", len(response[\"Blocks\"]))\n except ClientError:\n logger.exception(\"Couldn't detect text.\")\n raise\n else:\n return response\n\n # snippet-end:[python.example_code.textract.AnalyzeDocument]\n\n # snippet-start:[python.example_code.textract.helper.prepare_job]\n def prepare_job(self, bucket_name, document_name, document_bytes):\n \"\"\"\n Prepares a document image for an asynchronous detection job by uploading\n the image bytes to an Amazon S3 bucket. Amazon Textract must have permission\n to read from the bucket to process the image.\n\n :param bucket_name: The name of the Amazon S3 bucket.\n :param document_name: The name of the image stored in Amazon S3.\n :param document_bytes: The image as byte data.\n \"\"\"\n try:\n bucket = self.s3_resource.Bucket(bucket_name)\n bucket.upload_fileobj(document_bytes, document_name)\n logger.info(\"Uploaded %s to %s.\", document_name, bucket_name)\n except ClientError:\n logger.exception(\"Couldn't upload %s to %s.\", document_name, bucket_name)\n raise\n\n # snippet-end:[python.example_code.textract.helper.prepare_job]\n\n # snippet-start:[python.example_code.textract.helper.check_job_queue]\n def check_job_queue(self, queue_url, job_id):\n \"\"\"\n Polls an Amazon SQS queue for messages that indicate a specified Textract\n job has completed.\n\n :param queue_url: The URL of the Amazon SQS queue to poll.\n :param job_id: The ID of the Textract job.\n :return: The status of the job.\n \"\"\"\n status = None\n try:\n queue = self.sqs_resource.Queue(queue_url)\n messages = queue.receive_messages()\n if messages:\n msg_body = json.loads(messages[0].body)\n msg = json.loads(msg_body[\"Message\"])\n if msg.get(\"JobId\") == job_id:\n messages[0].delete()\n status = msg.get(\"Status\")\n logger.info(\n \"Got message %s with status %s.\", messages[0].message_id, status\n )\n else:\n logger.info(\"No messages in queue %s.\", queue_url)\n except ClientError:\n logger.exception(\"Couldn't get messages from queue %s.\", queue_url)\n else:\n return status\n\n # snippet-end:[python.example_code.textract.helper.check_job_queue]\n\n # snippet-start:[python.example_code.textract.StartDocumentTextDetection]\n def start_detection_job(\n self, bucket_name, document_file_name, sns_topic_arn, sns_role_arn\n ):\n \"\"\"\n Starts an asynchronous job to detect text elements in an image stored in an\n Amazon S3 bucket. Textract publishes a notification to the specified Amazon SNS\n topic when the job completes.\n The image must be in PNG, JPG, or PDF format.\n\n :param bucket_name: The name of the Amazon S3 bucket that contains the image.\n :param document_file_name: The name of the document image stored in Amazon S3.\n :param sns_topic_arn: The Amazon Resource Name (ARN) of an Amazon SNS topic\n where the job completion notification is published.\n :param sns_role_arn: The ARN of an AWS Identity and Access Management (IAM)\n role that can be assumed by Textract and grants permission\n to publish to the Amazon SNS topic.\n :return: The ID of the job.\n \"\"\"\n try:\n response = self.textract_client.start_document_text_detection(\n DocumentLocation={\n \"S3Object\": {\"Bucket\": bucket_name, \"Name\": document_file_name}\n },\n NotificationChannel={\n \"SNSTopicArn\": sns_topic_arn,\n \"RoleArn\": sns_role_arn,\n },\n )\n job_id = response[\"JobId\"]\n logger.info(\n \"Started text detection job %s on %s.\", job_id, document_file_name\n )\n except ClientError:\n logger.exception(\"Couldn't detect text in %s.\", document_file_name)\n raise\n else:\n return job_id\n\n # snippet-end:[python.example_code.textract.StartDocumentTextDetection]\n\n # snippet-start:[python.example_code.textract.GetDocumentTextDetection]\n def get_detection_job(self, job_id):\n \"\"\"\n Gets data for a previously started text detection job.\n\n :param job_id: The ID of the job to retrieve.\n :return: The job data, including a list of blocks that describe elements\n detected in the image.\n \"\"\"\n try:\n response = self.textract_client.get_document_text_detection(JobId=job_id)\n job_status = response[\"JobStatus\"]\n logger.info(\"Job %s status is %s.\", job_id, job_status)\n except ClientError:\n logger.exception(\"Couldn't get data for job %s.\", job_id)\n raise\n else:\n return response\n\n # snippet-end:[python.example_code.textract.GetDocumentTextDetection]\n\n # snippet-start:[python.example_code.textract.StartDocumentAnalysis]\n def start_analysis_job(\n self,\n bucket_name,\n document_file_name,\n feature_types,\n sns_topic_arn,\n sns_role_arn,\n ):\n \"\"\"\n Starts an asynchronous job to detect text and additional elements, such as\n forms or tables, in an image stored in an Amazon S3 bucket. Textract publishes\n a notification to the specified Amazon SNS topic when the job completes.\n The image must be in PNG, JPG, or PDF format.\n\n :param bucket_name: The name of the Amazon S3 bucket that contains the image.\n :param document_file_name: The name of the document image stored in Amazon S3.\n :param feature_types: The types of additional document features to detect.\n :param sns_topic_arn: The Amazon Resource Name (ARN) of an Amazon SNS topic\n where job completion notification is published.\n :param sns_role_arn: The ARN of an AWS Identity and Access Management (IAM)\n role that can be assumed by Textract and grants permission\n to publish to the Amazon SNS topic.\n :return: The ID of the job.\n \"\"\"\n try:\n response = self.textract_client.start_document_analysis(\n DocumentLocation={\n \"S3Object\": {\"Bucket\": bucket_name, \"Name\": document_file_name}\n },\n NotificationChannel={\n \"SNSTopicArn\": sns_topic_arn,\n \"RoleArn\": sns_role_arn,\n },\n FeatureTypes=feature_types,\n )\n job_id = response[\"JobId\"]\n logger.info(\n \"Started text analysis job %s on %s.\", job_id, document_file_name\n )\n except ClientError:\n logger.exception(\"Couldn't analyze text in %s.\", document_file_name)\n raise\n else:\n return job_id\n\n # snippet-end:[python.example_code.textract.StartDocumentAnalysis]\n\n # snippet-start:[python.example_code.textract.GetDocumentAnalysis]\n def get_analysis_job(self, job_id):\n \"\"\"\n Gets data for a previously started detection job that includes additional\n elements.\n\n :param job_id: The ID of the job to retrieve.\n :return: The job data, including a list of blocks that describe elements\n detected in the image.\n \"\"\"\n try:\n response = self.textract_client.get_document_analysis(JobId=job_id)\n job_status = response[\"JobStatus\"]\n logger.info(\"Job %s status is %s.\", job_id, job_status)\n except ClientError:\n logger.exception(\"Couldn't get data for job %s.\", job_id)\n raise\n else:\n return response\n\n\n# snippet-end:[python.example_code.textract.GetDocumentAnalysis]\n```\n\nNext, we set up Textract and S3, and provide this to an instance of `TextractWrapper`.\n\n\n```python\nimport boto3\n\ntextract_client = boto3.client('textract')\ns3_client = boto3.client('s3')\n\ntextractWrapper = TextractWrapper(textract_client, s3_client, None)\n```\n\nWe are now ready to make calls to Textract. At a high level, Textract has two modes: synchronous and asynchronous. Synchronous calls return the parsed output once it is completed. As of the time of writing (March 2024), however, multipage PDF processing is only supported [asynchronously](https://docs.aws.amazon.com/textract/latest/dg/sync.html). So for our purposes here, we will only explore the asynchronous route.\n\nAsynchronous calls follow the below process:\n\n1. Send a request to Textract with an SNS topic, S3 bucket, and the name (key) of the document inside that bucket to process. Textract returns a Job ID that can be used to track the status of the request\n2. Textract fetches the document from S3 and processes it\n3. Once the request is complete, Textract sends out a message to the SNS topic. This can be used in conjunction with other services such as Lambda or SQS for downstream processes.\n4. The parsed results can be fetched from Textract in chunks via the job ID.\n\n\n```python\nbucket_name = \"your-bucket-name\"\nsns_topic_arn = \"your-sns-arn\" # this can be found under the topic you created in the Amazon SNS dashboard\nsns_role_arn = \"sns-role-arn\" # this is an IAM role that allows Textract to interact with SNS\n\nfile_name = \"example-drug-label.pdf\"\n```\n\n\n```python\n# kick off a text detection job. This returns a job ID.\njob_id = textractWrapper.start_detection_job(bucket_name=bucket_name, document_file_name=file_name,\n sns_topic_arn=sns_topic_arn, sns_role_arn=sns_role_arn)\n```\n\nOnce the job completes, this will return a dictionary with the following keys:\n\n```dict_keys(['DocumentMetadata', 'JobStatus', 'NextToken', 'Blocks', 'AnalyzeDocumentModelVersion', 'ResponseMetadata'])```\n\nThis response corresponds to one chunk of information parsed by Textract. The number of chunks a document is parsed into depends on the length of the document. The two keys we are most interested in are `Blocks` and `NextToken`. `Blocks` contains all of the information that was extracted from this chunk, while `NextToken` tells us what chunk comes next, if any.\n\nTextract returns an information-rich representation of the extracted text, such as their position on the page and hierarchical relationships with other entities, all the way down to the individual word level. Since we are only interested in the raw text, we need a way to parse through all of the chunks and their `Blocks`. Lucky for us, Amazon provides some [helper functions](https://github.com/aws-samples/textract-paragraph-identification/tree/main) for this purpose, which we utilize below.\n\n\n```python\ndef get_text_results_from_textract(job_id):\n response = textract_client.get_document_text_detection(JobId=job_id)\n collection_of_textract_responses = []\n pages = [response]\n\n collection_of_textract_responses.append(response)\n\n while 'NextToken' in response:\n next_token = response['NextToken']\n response = textract_client.get_document_text_detection(JobId=job_id, NextToken=next_token)\n pages.append(response)\n collection_of_textract_responses.append(response)\n return collection_of_textract_responses\n\ndef get_the_text_with_required_info(collection_of_textract_responses):\n total_text = []\n total_text_with_info = []\n running_sequence_number = 0\n\n font_sizes_and_line_numbers = {}\n for page in collection_of_textract_responses:\n per_page_text = []\n blocks = page['Blocks']\n for block in blocks:\n if block['BlockType'] == 'LINE':\n block_text_dict = {}\n running_sequence_number += 1\n block_text_dict.update(text=block['Text'])\n block_text_dict.update(page=block['Page'])\n block_text_dict.update(left_indent=round(block['Geometry']['BoundingBox']['Left'], 2))\n font_height = round(block['Geometry']['BoundingBox']['Height'], 3)\n line_number = running_sequence_number\n block_text_dict.update(font_height=round(block['Geometry']['BoundingBox']['Height'], 3))\n block_text_dict.update(indent_from_top=round(block['Geometry']['BoundingBox']['Top'], 2))\n block_text_dict.update(text_width=round(block['Geometry']['BoundingBox']['Width'], 2))\n block_text_dict.update(line_number=running_sequence_number)\n\n if font_height in font_sizes_and_line_numbers:\n line_numbers = font_sizes_and_line_numbers[font_height]\n line_numbers.append(line_number)\n font_sizes_and_line_numbers[font_height] = line_numbers\n else:\n line_numbers = []\n line_numbers.append(line_number)\n font_sizes_and_line_numbers[font_height] = line_numbers\n\n total_text.append(block['Text'])\n per_page_text.append(block['Text'])\n total_text_with_info.append(block_text_dict)\n\n return total_text, total_text_with_info, font_sizes_and_line_numbers\n\ndef get_text_with_line_spacing_info(total_text_with_info):\n i = 1\n text_info_with_line_spacing_info = []\n while (i < len(total_text_with_info) - 1):\n previous_line_info = total_text_with_info[i - 1]\n current_line_info = total_text_with_info[i]\n next_line_info = total_text_with_info[i + 1]\n if current_line_info['page'] == next_line_info['page'] and previous_line_info['page'] == current_line_info[\n 'page']:\n line_spacing_after = round((next_line_info['indent_from_top'] - current_line_info['indent_from_top']), 2)\n spacing_with_prev = round((current_line_info['indent_from_top'] - previous_line_info['indent_from_top']), 2)\n current_line_info.update(line_space_before=spacing_with_prev)\n current_line_info.update(line_space_after=line_spacing_after)\n text_info_with_line_spacing_info.append(current_line_info)\n else:\n text_info_with_line_spacing_info.append(None)\n i += 1\n return text_info_with_line_spacing_info\n```\n\nWe feed in the Job ID from before into the function `get_text_results_from_textract` to fetch all of the chunks associated with this job. Then, we pass the resulting list into `get_the_text_with_required_info` and `get_text_with_line_spacing_info` to organize the text into lines.\n\nFinally, we can concatenate the lines into one string to pass into our downstream RAG pipeline.\n\n\n```python\nall_text = \"\\n\".join([line[\"text\"] if line else \"\" for line in text_info_with_line_spacing])\n\nwith open(f\"aws-parsed-{source_filename}.txt\", \"w\") as f:\n f.write(all_text)\n```\n\n#### Visualize the parsed document\n\n\n```python\nfilename = \"aws-parsed-{}.txt\".format(source_filename)\nwith open(\"{}/{}\".format(data_dir, filename), \"r\") as doc:\n parsed_document = doc.read()\n\nprint(parsed_document[:1000])\n```\n\n\n### Solution 3: Unstructured.io [[Back to Solutions]](#top)\n\nUnstructured.io provides libraries with open-source components for pre-processing text documents such as PDFs, HTML and Word Documents.\n\nExternal documentation: https://github.com/Unstructured-IO/unstructured-api\n\n#### Parsing the document\n\nThe guide assumes an endpoint exists that hosts this service. The API is offered in two forms\n1. [a hosted version](https://unstructured.io/)\n2. [an OSS docker image](https://github.com/Unstructured-IO/unstructured-api?tab=readme-ov-file#dizzy-instructions-for-using-the-docker-image)\n\n**Note: You can skip to the next block if you want to use the pre-existing parsed version.**\n\n\n```python\nimport os\nimport requests\n\nUNSTRUCTURED_URL = \"\" # enter service endpoint\n\nparsed_documents = []\n\ninput_path = \"{}/{}.{}\".format(data_dir, source_filename, extension)\nwith open(input_path, 'rb') as file_data:\n response = requests.post(\n url=UNSTRUCTURED_URL,\n files={\"files\": (\"{}.{}\".format(source_filename, extension), file_data)},\n data={\n \"output_format\": (None, \"application/json\"),\n \"stratergy\": \"hi_res\",\n \"pdf_infer_table_structure\": \"true\",\n \"include_page_breaks\": \"true\"\n },\n headers={\"Accept\": \"application/json\"}\n )\n\nparsed_response = response.json()\n\nparsed_document = \" \".join([parsed_entry[\"text\"] for parsed_entry in parsed_response])\nprint(\"Parsed {}\".format(source_filename))\n```\n\n\n```python\n\"\"\"\nPost process parsed document and store it locally.\n\"\"\"\n\nfile_path = \"{}/{}-parsed-fda-approved-drug.txt\".format(data_dir, \"unstructured-io\")\nstore_document(file_path, parsed_document)\n```\n\n#### Visualize the parsed document\n\n\n```python\nfilename = \"unstructured-io-parsed-{}.txt\".format(source_filename)\nwith open(\"{}/{}\".format(data_dir, filename), \"r\") as doc:\n parsed_document = doc.read()\n\nprint(parsed_document[:1000])\n```\n\n\n\n### Solution 4: LlamaParse [[Back to Solutions]](#top)\n\nLlamaParse is an API created by LlamaIndex to efficiently parse and represent files for efficient retrieval and context augmentation using LlamaIndex frameworks.\n\nExternal documentation: https://github.com/run-llama/llama_parse\n\n#### Parsing the document\n\nThe following block uses the LlamaParse cloud offering. You can learn more and fetch a respective API key for the service [here](https://cloud.llamaindex.ai/parse).\n\nParsing documents with LlamaParse offers an option for two output modes both of which we will explore and compare below\n- Text\n- Markdown\n\n**Note: You can skip to the next block if you want to use the pre-existing parsed version.**\n\n\n```python\nimport os\nfrom llama_parse import LlamaParse\n\nimport nest_asyncio # needed to notebook env\nnest_asyncio.apply() # needed to notebook env\n\nllama_index_api_key = \"{API_KEY}\"\ninput_path = \"{}/{}.{}\".format(data_dir, source_filename, extension)\n```\n\n\n```python\n# Text mode\ntext_parser = LlamaParse(\n api_key=llama_index_api_key,\n result_type=\"text\"\n)\n\ntext_response = text_parser.load_data(input_path)\ntext_parsed_document = \" \".join([parsed_entry.text for parsed_entry in text_response])\n\nprint(\"Parsed {} to text\".format(source_filename))\n```\n\n\n```python\n\"\"\"\nPost process parsed document and store it locally.\n\"\"\"\n\nfile_path = \"{}/{}-text-parsed-fda-approved-drug.txt\".format(data_dir, \"llamaparse\")\nstore_document(file_path, text_parsed_document)\n```\n\n\n```python\n# Markdown mode\nmarkdown_parser = LlamaParse(\n api_key=llama_index_api_key,\n result_type=\"markdown\"\n)\n\nmarkdown_response = markdown_parser.load_data(input_path)\nmarkdown_parsed_document = \" \".join([parsed_entry.text for parsed_entry in markdown_response])\n\nprint(\"Parsed {} to markdown\".format(source_filename))\n```\n\n\n```python\n\"\"\"\nPost process parsed document and store it locally.\n\"\"\"\n\nfile_path = \"{}/{}-markdown-parsed-fda-approved-drug.txt\".format(data_dir, \"llamaparse\")\nstore_document(file_path, markdown_parsed_document)\n```\n\n#### Visualize the parsed document\n\n\n```python\n# Text parsing\n\nfilename = \"llamaparse-text-parsed-{}.txt\".format(source_filename)\n\nwith open(\"{}/{}\".format(data_dir, filename), \"r\") as doc:\n parsed_document = doc.read()\n \nprint(parsed_document[:1000])\n```\n\n\n```python\n# Markdown parsing\n\nfilename = \"llamaparse-markdown-parsed-fda-approved-drug.txt\"\nwith open(\"{}/{}\".format(data_dir, filename), \"r\") as doc:\n parsed_document = doc.read()\n \nprint(parsed_document[:1000])\n```\n\n\n\n### Solution 5: pdf2image + pytesseract [[Back to Solutions]](#top)\n\nThe final parsing method we examine does not rely on cloud services, but rather relies on two libraries: `pdf2image`, and `pytesseract`. `pytesseract` lets you perform OCR locally on images, but not PDF files. So, we first convert our PDF into a set of images via `pdf2image`.\n\n#### Parsing the document\n\n\n```python\nfrom matplotlib import pyplot as plt\nfrom pdf2image import convert_from_path\nimport pytesseract\n```\n\n\n```python\n# pdf2image extracts as a list of PIL.Image objects\npages = convert_from_path(filename)\n```\n\n\n```python\n# we look at the first page as a sanity check:\n\nplt.imshow(pages[0])\nplt.axis('off')\nplt.show()\n```\n\nNow, we can process the image of each page with `pytesseract` and concatenate the results to get our parsed document.\n\n\n```python\nlabel_ocr_pytesseract = \"\".join([pytesseract.image_to_string(page) for page in pages])\n```\n\n\n```python\nprint(label_ocr_pytesseract[:200])\n```\n\n HIGHLIGHTS OF PRESCRIBING INFORMATION\n \n These highlights do not include all the information needed to use\n IWILFIN™ safely and effectively. See full prescribing information for\n IWILFIN.\n \n IWILFIN™ (eflor\n\n\n\n```python\nlabel_ocr_pytesseract = \"\".join([pytesseract.image_to_string(page) for page in pages])\n\nwith open(f\"pytesseract-parsed-{source_filename}.txt\", \"w\") as f:\n f.write(label_ocr_pytesseract)\n```\n\n#### Visualize the parsed document\n\n\n```python\nfilename = \"pytesseract-parsed-{}.txt\".format(source_filename)\nwith open(\"{}/{}\".format(data_dir, filename), \"r\") as doc:\n parsed_document = doc.read()\n\nprint(parsed_document[:1000])\n```\n\n\n## Document Questions\n\nWe can now ask a set of simple + complex questions and see how each parsing solution performs with Command-R. The questions are\n- **What are the most common adverse reactions of Iwilfin?**\n - Task: Simple information extraction\n- **What is the recommended dosage of IWILFIN on body surface area between 0.5 and 0.75?**\n - Task: Tabular data extraction\n- **I need a succinct summary of the compound name, indication, route of administration, and mechanism of action of Iwilfin.**\n - Task: Overall document summary\n\n\n```python\nimport cohere\nco = cohere.Client(api_key=\"{API_KEY}\")\n```\n\n\n```python\n\"\"\"\nDocument Questions\n\"\"\"\nprompt = \"What are the most common adverse reactions of Iwilfin?\"\n# prompt = \"What is the recommended dosage of Iwilfin on body surface area between 0.5 m2 and 0.75 m2?\"\n# prompt = \"I need a succinct summary of the compound name, indication, route of administration, and mechanism of action of Iwilfin.\"\n\n\"\"\"\nChoose one of the above solutions\n\"\"\"\nsource = \"gcp\"\n# source = \"aws\"\n# source = \"unstructured-io\"\n# source = \"llamaparse-text\"\n# source = \"llamaparse-markdown\"\n# source = \"pytesseract\"\n```\n\n## Data Ingestion\n\n\nIn order to set up our RAG implementation, we need to separate the parsed text into chunks and load the chunks to an index. The index will allow us to retrieve relevant passages from the document for different queries. Here, we use a simple implementation of indexing using the `hnswlib` library. Note that there are many different indexing solutions that are appropriate for specific production use cases.\n\n\n```python\n\"\"\"\nRead parsed document content and chunk data\n\"\"\"\n\nimport os\nfrom langchain_text_splitters import RecursiveCharacterTextSplitter\n\ndocuments = []\n\nwith open(\"{}/{}-parsed-fda-approved-drug.txt\".format(data_dir, source), \"r\") as doc:\ndoc_content = doc.read()\n\n\"\"\"\nPersonal notes on chunking\nhttps://medium.com/@ayhamboucher/llm-based-context-splitter-for-large-documents-445d3f02b01b\n\"\"\"\n\n\n# Chunk doc content\ntext_splitter = RecursiveCharacterTextSplitter(\n chunk_size=512,\n chunk_overlap=200,\n length_function=len,\n is_separator_regex=False\n)\n\n# Split the text into chunks with some overlap\nchunks_ = text_splitter.create_documents([doc_content])\ndocuments = [c.page_content for c in chunks_]\n\nprint(\"Source document has been broken down to {} chunks\".format(len(documents)))\n```\n\n\n```python\n\"\"\"\nEmbed document chunks\n\"\"\"\ndocument_embeddings = co.embed(texts=documents, model=\"embed-english-v3.0\", input_type=\"search_document\").embeddings\n```\n\n\n```python\n\"\"\"\nCreate document index and add embedded chunks\n\"\"\"\n\nimport hnswlib\n\nindex = hnswlib.Index(space='ip', dim=1024) # space: inner product\nindex.init_index(max_elements=len(document_embeddings), ef_construction=512, M=64)\nindex.add_items(document_embeddings, list(range(len(document_embeddings))))\nprint(\"Count:\", index.element_count)\n```\n\n Count: 115\n\n\n## Retrieval\n\nIn this step, we use k-nearest neighbors to fetch the most relevant documents for our query. Once the nearest neighbors are retrieved, we use Cohere's reranker to reorder the documents in the most relevant order with regards to our input search query.\n\n\n```python\n\"\"\"\nEmbed search query\nFetch k nearest neighbors\n\"\"\"\n\nquery_emb = co.embed(texts=[prompt], model='embed-english-v3.0', input_type=\"search_query\").embeddings\ndefault_knn = 10\nknn = default_knn if default_knn <= index.element_count else index.element_count\nresult = index.knn_query(query_emb, k=knn)\nneighbors = [(result[0][0][i], result[1][0][i]) for i in range(len(result[0][0]))]\nrelevant_docs = [documents[x[0]] for x in sorted(neighbors, key=lambda x: x[1])]\n```\n\n\n```python\n\"\"\"\nRerank retrieved documents\n\"\"\"\n\nrerank_results = co.rerank(query=prompt, documents=relevant_docs, top_n=3, model='rerank-english-v2.0').results\nreranked_relevant_docs = format_docs_for_chat([x.document[\"text\"] for x in rerank_results])\n```\n\n## Final Step: Call Command-R + RAG!\n\n\n```python\n\"\"\"\nCall the /chat endpoint with command-r\n\"\"\"\n\nresponse = co.chat(\n message=prompt,\n model=\"command-r\",\n documents=reranked_relevant_docs\n)\n\ncited_response, citations_reference = insert_citations_in_order(response.text, response.citations, reranked_relevant_docs)\nprint(cited_response)\nprint(\"\\n\")\nprint(\"References:\")\nprint(citations_reference)\n```\n\n## Head-to-head Comparisons\n\nRun the code cells below to make head to head comparisons of the different parsing techniques across different questions.\n\n\n```python\nimport pandas as pd\nresults = pd.read_csv(\"{}/results-table.csv\".format(data_dir))\n```\n\n\n```python\nquestion = input(\"\"\"\nQuestion 1: What are the most common adverse reactions of Iwilfin?\nQuestion 2: What is the recommended dosage of Iwilfin on body surface area between 0.5 m2 and 0.75 m2?\nQuestion 3: I need a succinct summary of the compound name, indication, route of administration, and mechanism of action of Iwilfin.\n\nPick which question you want to see (1,2,3): \"\"\")\nreferences = input(\"Do you want to see the references as well? References are long and noisy (y/n): \")\nprint(\"\\n\\n\")\n\nindex = {\"1\": 0, \"2\": 3, \"3\": 6}[question]\n\nfor src in [\"gcp\", \"aws\", \"unstructured-io\", \"llamaparse-text\", \"llamaparse-markdown\", \"pytesseract\"]:\n print(\"| {} |\".format(src))\n print(\"\\n\")\n print(results[src][index])\n if references == \"y\":\n print(\"\\n\")\n print(\"References:\")\n print(results[src][index+1])\n print(\"\\n\")\n```\n\n \n Question 1: What are the most common adverse reactions of Iwilfin?\n Question 2: What is the recommended dosage of Iwilfin on body surface area between 0.5 m2 and 0.75 m2?\n Question 3: I need a succinct summary of the compound name, indication, route of administration, and mechanism of action of Iwilfin.\n \n Pick which question you want to see (1,2,3): 3\n Do you want to see the references as well? References are long and noisy (y/n): n\n \n \n \n | gcp |\n \n \n Compound Name: eflornithine hydrochloride ([0], [1], [2]) (IWILFIN ([1])™)\n \n Indication: used to reduce the risk of relapse in adult and paediatric patients with high-risk neuroblastoma (HRNB) ([1], [3]), who have responded at least partially to prior multiagent, multimodality therapy. ([1], [3], [4])\n \n Route of Administration: IWILFIN™ tablets ([1], [3], [4]) are taken orally twice daily ([3], [4]), with doses ranging from 192 to 768 mg based on body surface area. ([3], [4])\n \n Mechanism of Action: IWILFIN™ is an ornithine decarboxylase inhibitor. ([0], [2])\n \n \n \n | aws |\n \n \n Compound Name: eflornithine ([0], [1], [2], [3]) (IWILFIN ([0])™)\n \n Indication: used to reduce the risk of relapse ([0], [3]) in adults ([0], [3]) and paediatric patients ([0], [3]) with high-risk neuroblastoma (HRNB) ([0], [3]) who have responded to prior therapies. ([0], [3], [4])\n \n Route of Administration: Oral ([2], [4])\n \n Mechanism of Action: IWILFIN is an ornithine decarboxylase inhibitor. ([1])\n \n \n | unstructured-io |\n \n \n Compound Name: Iwilfin ([1], [2], [3], [4]) (eflornithine) ([0], [2], [3], [4])\n \n Indication: Iwilfin is indicated to reduce the risk of relapse ([1], [3]) in adult and paediatric patients ([1], [3]) with high-risk neuroblastoma (HRNB) ([1], [3]), who have responded to prior anti-GD2 ([1]) immunotherapy ([1], [4]) and multi-modality therapy. ([1])\n \n Route of Administration: Oral ([0], [3])\n \n Mechanism of Action: Iwilfin is an ornithine decarboxylase inhibitor. ([1], [2], [3], [4])\n \n \n | llamaparse-text |\n \n \n Compound Name: IWILFIN ([2], [3]) (eflornithine) ([3])\n \n Indication: IWILFIN is used to reduce the risk of relapse ([1], [2], [3]) in adult and paediatric patients ([1], [2], [3]) with high-risk neuroblastoma (HRNB) ([1], [2], [3]), who have responded at least partially to certain prior therapies. ([2], [3])\n \n Route of Administration: IWILFIN is administered as a tablet. ([2])\n \n Mechanism of Action: IWILFIN is an ornithine decarboxylase inhibitor. ([0], [1], [4])\n \n \n | llamaparse-markdown |\n \n \n Compound Name: IWILFIN ([1], [2]) (eflornithine) ([1])\n \n Indication: IWILFIN is indicated to reduce the risk of relapse ([1], [2]) in adult and paediatric patients ([1], [2]) with high-risk neuroblastoma (HRNB) ([1], [2]), who have responded at least partially ([1], [2], [3]) to prior anti-GD2 immunotherapy ([1], [2]) and multiagent, multimodality therapy. ([1], [2], [3])\n \n Route of Administration: Oral ([0], [1], [3], [4])\n \n Mechanism of Action: IWILFIN acts as an ornithine decarboxylase inhibitor. ([1])\n \n \n | pytesseract |\n \n \n Compound Name: IWILFIN™ ([0], [2]) (eflornithine) ([0], [2])\n \n Indication: IWILFIN is indicated to reduce the risk of relapse ([0], [2]) in adult and paediatric patients ([0], [2]) with high-risk neuroblastoma (HRNB) ([0], [2]), who have responded positively to prior anti-GD2 immunotherapy and multiagent, multimodality therapy. ([0], [2], [4])\n \n Route of Administration: IWILFIN is administered orally ([0], [1], [3], [4]), in the form of a tablet. ([1])\n \n Mechanism of Action: IWILFIN acts as an ornithine decarboxylase inhibitor. ([0])",
"html": "",
"htmlmode": false,
"fullscreen": false,
diff --git a/scripts/cookbooks-json/fueling-generative-content.json b/scripts/cookbooks-json/fueling-generative-content.json
index 792ef656..fb6d4b8b 100644
--- a/scripts/cookbooks-json/fueling-generative-content.json
+++ b/scripts/cookbooks-json/fueling-generative-content.json
@@ -13,7 +13,7 @@
},
"title": "Fueling Generative Content with Keyword Research",
"slug": "fueling-generative-content",
- "body": "[block:html]\n{\n \"html\": \"\\n\\n\\n
\\n\\n\"\n}\n[/block]\n\n\nGenerative models have proven extremely useful in content idea generation. But they don’t take into account user search demand and trends. In this notebook, let’s see how we can solve that by adding keyword research into the equation.\n\nRead the accompanying [blog post here](https://txt.cohere.ai/generative-content-keyword-research/).\n\n```python\n! pip install cohere -q\n```\n\n```python\nimport cohere\nimport numpy as np\nimport pandas as pd\nfrom sklearn.cluster import KMeans\n\nimport cohere\nco = cohere.Client(\"COHERE_API_KEY\") # Get your API key: https://dashboard.cohere.com/api-keys\n```\n\n\n\n```python\n#@title Enable text wrapping in Google Colab\n\nfrom IPython.display import HTML, display\n\ndef set_css():\n display(HTML('''\n \n '''))\nget_ipython().events.register('pre_run_cell', set_css)\n```\n\nFirst, we need to get a supply of high-traffic keywords for a given topic. We can get this via keyword research tools, of which are many available. We’ll use Google Keyword Planner, which is free to use.\n\n```python\n\nimport wget\nwget.download(\"https://raw.githubusercontent.com/cohere-ai/notebooks/main/notebooks/data/remote_teams.csv\", \"remote_teams.csv\")\n```\n\n\n\n```\n'remote_teams.csv'\n```\n\n```python\ndf = pd.read_csv('remote_teams.csv')\ndf.columns = [\"keyword\",\"volume\"]\ndf.head()\n```\n\n\n\nFueling Generative Content with Keyword Research
\\n\n\n\n \n
\n
\n\nWe now have a list of keywords, but this list is still raw. For example, “managing remote teams” is the top-ranking keyword in this list. But at the same time, there are many similar keywords further down in the list, such as “how to effectively manage remote teams.”\n\nWe can do that by clustering them into topics. For this, we’ll leverage Cohere’s Embed endpoint and scikit-learn.\n\n### Embed the Keywords with Cohere Embed\n\nThe Cohere Embed endpoint turns a text input into a text embedding.\n\n```python\ndef embed_text(texts):\n output = co.embed(\n texts=texts,\n model='embed-english-v3.0',\n input_type=\"search_document\",\n )\n return output.embeddings\n\nembeds = np.array(embed_text(df['keyword'].tolist()))\n```\n\n\n\n### Cluster the Keywords into Topics with scikit-learn\n\nWe then use these embeddings to cluster the keywords. A common term used for this exercise is “topic modeling.” Here, we can leverage scikit-learn’s KMeans module, a machine learning algorithm for clustering.\n\n```python\nNUM_TOPICS = 4\nkmeans = KMeans(n_clusters=NUM_TOPICS, random_state=21, n_init=\"auto\").fit(embeds)\ndf['topic'] = list(kmeans.labels_)\ndf.head()\n```\n\n\n\n| \n | keyword | \nvolume | \n
|---|---|---|
| 0 | \nmanaging remote teams | \n1000 | \n
| 1 | \nremote teams | \n390 | \n
| 2 | \ncollaboration tools for remote teams | \n320 | \n
| 3 | \nonline games for remote teams | \n320 | \n
| 4 | \nhow to manage remote teams | \n260 | \n
\n\n\n \n
\n
\n\n### Generate Topic Names with Cohere Chat\n\nWe use the Chat to generate a topic name for that cluster.\n\n```python\ntopic_keywords_dict = {topic: list(set(group['keyword'])) for topic, group in df.groupby('topic')}\n```\n\n\n\n```python\ndef generate_topic_name(keywords):\n # Construct the prompt\n prompt = f\"\"\"Generate a concise topic name that best represents these keywords.\\\nProvide just the topic name and not any additional details.\n\nKeywords: {', '.join(keywords)}\"\"\"\n \n # Call the Cohere API\n response = co.chat(\n model='command-r', # Choose the model size\n message=prompt,\n preamble=\"\")\n \n # Return the generated text\n return response.text\n```\n\n\n\n```python\ntopic_name_mapping = {topic: generate_topic_name(keywords) for topic, keywords in topic_keywords_dict.items()}\n\ndf['topic_name'] = df['topic'].map(topic_name_mapping)\n\ndf.head()\n```\n\n\n\n| \n | keyword | \nvolume | \ntopic | \n
|---|---|---|---|
| 0 | \nmanaging remote teams | \n1000 | \n0 | \n
| 1 | \nremote teams | \n390 | \n1 | \n
| 2 | \ncollaboration tools for remote teams | \n320 | \n1 | \n
| 3 | \nonline games for remote teams | \n320 | \n3 | \n
| 4 | \nhow to manage remote teams | \n260 | \n0 | \n
\n\n\n \n
\n
\n\n```python\nfor topic, name in topic_name_mapping.items():\n print(f\"Topic {topic}: {name}\")\n```\n\n\n\n```\nTopic 0: Remote Team Management\nTopic 1: Remote Team Tools and Tips\nTopic 2: Remote Team Resources\nTopic 3: Remote Team Fun\n```\n\nNow that we have the keywords nicely grouped into topics, we can proceed to generate the content ideas.\n\n### Take the Top Keywords from Each Topic\n\nHere we can implement a filter to take just the top N keywords from each topic, sorted by the search volume. In our case, we use 10.\n\n```python\nTOP_N = 10\n\ntop_keywords = (df.groupby('topic')\n .apply(lambda x: x.nlargest(TOP_N, 'volume'))\n .reset_index(drop=True))\n\n\ncontent_by_topic = {}\nfor topic, group in top_keywords.groupby('topic'):\n keywords = ', '.join(list(group['keyword']))\n topic2name = topic2name = dict(df.groupby('topic')['topic_name'].first())\n topic_name = topic2name[topic]\n content_by_topic[topic] = {'topic_name': topic_name, 'keywords': keywords}\n```\n\n\n\n```python\ncontent_by_topic\n```\n\n\n\n```\n{0: {'topic_name': 'Remote Team Management',\n 'keywords': 'managing remote teams, how to manage remote teams, leading remote teams, managing remote teams best practices, remote teams best practices, best practices for managing remote teams, manage remote teams, building culture in remote teams, culture building for remote teams, managing remote teams training'},\n 1: {'topic_name': 'Remote Team Tools and Tips',\n 'keywords': 'remote teams, collaboration tools for remote teams, team building for remote teams, scrum remote teams, tools for remote teams, zapier remote teams, working agreements for remote teams, working with remote teams, free collaboration tools for remote teams, free retrospective tools for remote teams'},\n 2: {'topic_name': 'Remote Team Resources',\n 'keywords': 'best collaboration tools for remote teams, slack best practices for remote teams, best communication tools for remote teams, best tools for remote teams, always on video for remote teams, best apps for remote teams, best free collaboration tools for remote teams, best games for remote teams, best gifts for remote teams, best ice breaker questions for remote teams'},\n 3: {'topic_name': 'Remote Team Fun',\n 'keywords': 'online games for remote teams, team building activities for remote teams, games for remote teams, retrospective ideas for remote teams, team building ideas for remote teams, fun retrospective ideas for remote teams, retro ideas for remote teams, team building exercises for remote teams, trust building exercises for remote teams, activities for remote teams'}}\n```\n\n### Create a Prompt with These Keywords\n\nNext, we use the Chat endpoint to produce the content ideas. The prompt we’ll use is as follows\n\n```python\ndef generate_blog_ideas(keywords):\n prompt = f\"\"\"{keywords}\\n\\nThe above is a list of high-traffic keywords obtained from a keyword research tool. \nSuggest three blog post ideas that are highly relevant to these keywords. \nFor each idea, write a one paragraph abstract about the topic. \nUse this format:\nBlog title: | \n | keyword | \nvolume | \ntopic | \ntopic_name | \n
|---|---|---|---|---|
| 0 | \nmanaging remote teams | \n1000 | \n0 | \nRemote Team Management | \n
| 1 | \nremote teams | \n390 | \n1 | \nRemote Team Tools and Tips | \n
| 2 | \ncollaboration tools for remote teams | \n320 | \n1 | \nRemote Team Tools and Tips | \n
| 3 | \nonline games for remote teams | \n320 | \n3 | \nRemote Team Fun | \n
| 4 | \nhow to manage remote teams | \n260 | \n0 | \nRemote Team Management | \n
\\n
\\n\\n\"\n}\n[/block]\n\n\nGenerative models have proven extremely useful in content idea generation. But they don’t take into account user search demand and trends. In this notebook, let’s see how we can solve that by adding keyword research into the equation.\n\nRead the accompanying [blog post here](https://cohere.com/blog/generative-content-keyword-research/).\n\n```python\n! pip install cohere -q\n```\n\n```python\nimport cohere\nimport numpy as np\nimport pandas as pd\nfrom sklearn.cluster import KMeans\n\nimport cohere\nco = cohere.Client(\"COHERE_API_KEY\") # Get your API key: https://dashboard.cohere.com/api-keys\n```\n\n\n\n```python\n#@title Enable text wrapping in Google Colab\n\nfrom IPython.display import HTML, display\n\ndef set_css():\n display(HTML('''\n \n '''))\nget_ipython().events.register('pre_run_cell', set_css)\n```\n\nFirst, we need to get a supply of high-traffic keywords for a given topic. We can get this via keyword research tools, of which are many available. We’ll use Google Keyword Planner, which is free to use.\n\n```python\n\nimport wget\nwget.download(\"https://raw.githubusercontent.com/cohere-ai/notebooks/main/notebooks/data/remote_teams.csv\", \"remote_teams.csv\")\n```\n\n\n\n```\n'remote_teams.csv'\n```\n\n```python\ndf = pd.read_csv('remote_teams.csv')\ndf.columns = [\"keyword\",\"volume\"]\ndf.head()\n```\n\n\n\nFueling Generative Content with Keyword Research
\\n\n\n\n \n
\n
\n\nWe now have a list of keywords, but this list is still raw. For example, “managing remote teams” is the top-ranking keyword in this list. But at the same time, there are many similar keywords further down in the list, such as “how to effectively manage remote teams.”\n\nWe can do that by clustering them into topics. For this, we’ll leverage Cohere’s Embed endpoint and scikit-learn.\n\n### Embed the Keywords with Cohere Embed\n\nThe Cohere Embed endpoint turns a text input into a text embedding.\n\n```python\ndef embed_text(texts):\n output = co.embed(\n texts=texts,\n model='embed-english-v3.0',\n input_type=\"search_document\",\n )\n return output.embeddings\n\nembeds = np.array(embed_text(df['keyword'].tolist()))\n```\n\n\n\n### Cluster the Keywords into Topics with scikit-learn\n\nWe then use these embeddings to cluster the keywords. A common term used for this exercise is “topic modeling.” Here, we can leverage scikit-learn’s KMeans module, a machine learning algorithm for clustering.\n\n```python\nNUM_TOPICS = 4\nkmeans = KMeans(n_clusters=NUM_TOPICS, random_state=21, n_init=\"auto\").fit(embeds)\ndf['topic'] = list(kmeans.labels_)\ndf.head()\n```\n\n\n\n| \n | keyword | \nvolume | \n
|---|---|---|
| 0 | \nmanaging remote teams | \n1000 | \n
| 1 | \nremote teams | \n390 | \n
| 2 | \ncollaboration tools for remote teams | \n320 | \n
| 3 | \nonline games for remote teams | \n320 | \n
| 4 | \nhow to manage remote teams | \n260 | \n
\n\n\n \n
\n
\n\n### Generate Topic Names with Cohere Chat\n\nWe use the Chat to generate a topic name for that cluster.\n\n```python\ntopic_keywords_dict = {topic: list(set(group['keyword'])) for topic, group in df.groupby('topic')}\n```\n\n\n\n```python\ndef generate_topic_name(keywords):\n # Construct the prompt\n prompt = f\"\"\"Generate a concise topic name that best represents these keywords.\\\nProvide just the topic name and not any additional details.\n\nKeywords: {', '.join(keywords)}\"\"\"\n \n # Call the Cohere API\n response = co.chat(\n model='command-r', # Choose the model size\n message=prompt,\n preamble=\"\")\n \n # Return the generated text\n return response.text\n```\n\n\n\n```python\ntopic_name_mapping = {topic: generate_topic_name(keywords) for topic, keywords in topic_keywords_dict.items()}\n\ndf['topic_name'] = df['topic'].map(topic_name_mapping)\n\ndf.head()\n```\n\n\n\n| \n | keyword | \nvolume | \ntopic | \n
|---|---|---|---|
| 0 | \nmanaging remote teams | \n1000 | \n0 | \n
| 1 | \nremote teams | \n390 | \n1 | \n
| 2 | \ncollaboration tools for remote teams | \n320 | \n1 | \n
| 3 | \nonline games for remote teams | \n320 | \n3 | \n
| 4 | \nhow to manage remote teams | \n260 | \n0 | \n
\n\n\n \n
\n
\n\n```python\nfor topic, name in topic_name_mapping.items():\n print(f\"Topic {topic}: {name}\")\n```\n\n\n\n```\nTopic 0: Remote Team Management\nTopic 1: Remote Team Tools and Tips\nTopic 2: Remote Team Resources\nTopic 3: Remote Team Fun\n```\n\nNow that we have the keywords nicely grouped into topics, we can proceed to generate the content ideas.\n\n### Take the Top Keywords from Each Topic\n\nHere we can implement a filter to take just the top N keywords from each topic, sorted by the search volume. In our case, we use 10.\n\n```python\nTOP_N = 10\n\ntop_keywords = (df.groupby('topic')\n .apply(lambda x: x.nlargest(TOP_N, 'volume'))\n .reset_index(drop=True))\n\n\ncontent_by_topic = {}\nfor topic, group in top_keywords.groupby('topic'):\n keywords = ', '.join(list(group['keyword']))\n topic2name = topic2name = dict(df.groupby('topic')['topic_name'].first())\n topic_name = topic2name[topic]\n content_by_topic[topic] = {'topic_name': topic_name, 'keywords': keywords}\n```\n\n\n\n```python\ncontent_by_topic\n```\n\n\n\n```\n{0: {'topic_name': 'Remote Team Management',\n 'keywords': 'managing remote teams, how to manage remote teams, leading remote teams, managing remote teams best practices, remote teams best practices, best practices for managing remote teams, manage remote teams, building culture in remote teams, culture building for remote teams, managing remote teams training'},\n 1: {'topic_name': 'Remote Team Tools and Tips',\n 'keywords': 'remote teams, collaboration tools for remote teams, team building for remote teams, scrum remote teams, tools for remote teams, zapier remote teams, working agreements for remote teams, working with remote teams, free collaboration tools for remote teams, free retrospective tools for remote teams'},\n 2: {'topic_name': 'Remote Team Resources',\n 'keywords': 'best collaboration tools for remote teams, slack best practices for remote teams, best communication tools for remote teams, best tools for remote teams, always on video for remote teams, best apps for remote teams, best free collaboration tools for remote teams, best games for remote teams, best gifts for remote teams, best ice breaker questions for remote teams'},\n 3: {'topic_name': 'Remote Team Fun',\n 'keywords': 'online games for remote teams, team building activities for remote teams, games for remote teams, retrospective ideas for remote teams, team building ideas for remote teams, fun retrospective ideas for remote teams, retro ideas for remote teams, team building exercises for remote teams, trust building exercises for remote teams, activities for remote teams'}}\n```\n\n### Create a Prompt with These Keywords\n\nNext, we use the Chat endpoint to produce the content ideas. The prompt we’ll use is as follows\n\n```python\ndef generate_blog_ideas(keywords):\n prompt = f\"\"\"{keywords}\\n\\nThe above is a list of high-traffic keywords obtained from a keyword research tool. \nSuggest three blog post ideas that are highly relevant to these keywords. \nFor each idea, write a one paragraph abstract about the topic. \nUse this format:\nBlog title: | \n | keyword | \nvolume | \ntopic | \ntopic_name | \n
|---|---|---|---|---|
| 0 | \nmanaging remote teams | \n1000 | \n0 | \nRemote Team Management | \n
| 1 | \nremote teams | \n390 | \n1 | \nRemote Team Tools and Tips | \n
| 2 | \ncollaboration tools for remote teams | \n320 | \n1 | \nRemote Team Tools and Tips | \n
| 3 | \nonline games for remote teams | \n320 | \n3 | \nRemote Team Fun | \n
| 4 | \nhow to manage remote teams | \n260 | \n0 | \nRemote Team Management | \n
\\n
\\n\\n\"\n}\n[/block]\n\n\nHere we take a quick tour of what’s possible with language AI via Cohere’s Large Language Model (LLM) API. This is the Hello, World! of language AI, written for developers with little or no background in AI. In fact, we’ll do that by exploring the Hello, World! phrase itself.\n\nRead the accompanying [blog post here](https://txt.cohere.ai/hello-world-p1/).\n\n[block:html]\n{\n \"html\": \"Hello World! Meet Language AI
\\n![\\\"Hello](\\\"https://github.com/cohere-ai/notebooks/raw/main/notebooks/images/hello-world/hello-world-feat.png\\\")
\"\n}\n[/block]\n\n\nWe’ll cover three groups of tasks that you will typically work on when dealing with language data, including:\n\n- Generating text\n- Classifying text\n- Analyzing text\n\nThe first step is to install the Cohere Python SDK. Next, create an API key, which you can generate from the Cohere [dashboard](https://os.cohere.ai/register) or [CLI tool](https://docs.cohere.ai/cli-key).\n\n```python\n! pip install cohere altair umap-learn -q\n```\n\n```python\nimport cohere\nimport pandas as pd\nimport numpy as np\nimport altair as alt\n\nco = cohere.Client(\"COHERE_API_KEY\") # Get your API key: https://dashboard.cohere.com/api-keys\n```\n\nThe Cohere Generate endpoint generates text given an input, called “prompt”. The prompt provides a context of what we want the model to generate text. To illustrate this, let’s start with a simple prompt as the input. \n\n### Try a Simple Prompt\n\n```python\nprompt = \"What is a Hello World program.\"\n\nresponse = co.chat(\n message=prompt,\n model='command-r')\n\nprint(response.text)\n```\n\n````\nA \"Hello World\" program is a traditional and simple program that is often used as an introduction to a new programming language. The program typically displays the message \"Hello World\" as its output. The concept of a \"Hello World\" program originated from the book *The C Programming Language* written by Kernighan and Ritchie, where the example program in the book displayed the message using the C programming language. \n\nThe \"Hello World\" program serves as a basic and straightforward way to verify that your development environment is set up correctly and to familiarize yourself with the syntax and fundamentals of the programming language. It's a starting point for learning how to write and run programs in a new language.\n\nThe program's simplicity makes it accessible to programmers of all skill levels, and it's often one of the first programs beginners write when learning to code. The exact implementation of a \"Hello World\" program varies depending on the programming language being used, but the core idea remains the same—to display the \"Hello World\" message. \n\nHere's how a \"Hello World\" program can be written in a few select languages:\n1. **C**:\n```c\n#include \n\n\n \n
\n
\n\nWe use the Embed endpoint to get the embeddings for each of these keywords.\n\n```python\ndef embed_text(texts, input_type):\n \"\"\"\n Turns a piece of text into embeddings\n Arguments:\n text(str): the text to be turned into embeddings\n Returns:\n embedding(list): the embeddings\n \"\"\"\n # Embed text by calling the Embed endpoint\n response = co.embed(\n model=\"embed-english-v3.0\",\n input_type=input_type,\n texts=texts)\n \n return response.embeddings\n```\n\n```python\ndf[\"search_term_embeds\"] = embed_text(texts=df[\"search_term\"].tolist(),\n input_type=\"search_document\")\ndoc_embeds = np.array(df[\"search_term_embeds\"].tolist())\n```\n\n### Semantic Search\n\nWe’ll look at a couple of example applications. The first example is semantic search. Given a new query, our \"search engine\" must return the most similar FAQs, where the FAQs are the 50 search terms we uploaded earlier.\n\n```python\nquery = \"what is the history of hello world\"\n\nquery_embeds = embed_text(texts=[query],\n input_type=\"search_query\")[0]\n```\n\nWe use cosine similarity to compare the similarity of the new query with each of the FAQs\n\n```python\n\nfrom sklearn.metrics.pairwise import cosine_similarity\n\ndef get_similarity(target, candidates):\n \"\"\"\n Computes the similarity between a target text and a list of other texts\n Arguments:\n target(list[float]): the target text\n candidates(list[list[float]]): a list of other texts, or candidates\n Returns:\n sim(list[tuple]): candidate IDs and the similarity scores\n \"\"\"\n # Turn list into array\n candidates = np.array(candidates)\n target = np.expand_dims(np.array(target),axis=0)\n\n # Calculate cosine similarity\n sim = cosine_similarity(target,candidates)\n sim = np.squeeze(sim).tolist()\n\n # Sort by descending order in similarity\n sim = list(enumerate(sim))\n sim = sorted(sim, key=lambda x:x[1], reverse=True)\n\n # Return similarity scores\n return sim\n```\n\nFinally, we display the top 5 FAQs that match the new query\n\n```python\nsimilarity = get_similarity(query_embeds,doc_embeds)\n\nprint(\"New query:\")\nprint(new_query,'\\n')\n\nprint(\"Similar queries:\")\nfor idx,score in similarity[:5]:\n print(f\"Similarity: {score:.2f};\", df.iloc[idx][\"search_term\"])\n```\n\n```\nNew query:\nwhat is the history of hello world \n\nSimilar queries:\nSimilarity: 0.58; how did hello world originate\nSimilarity: 0.56; where did hello world come from\nSimilarity: 0.54; why hello world\nSimilarity: 0.53; why is hello world so famous\nSimilarity: 0.53; what is hello world\n```\n\n### Semantic Exploration\n\nIn the second example, we take the same idea as semantic search and take a broader look, which is exploring huge volumes of text and analyzing their semantic relationships.\n\nWe'll use the same 50 top web search terms about Hello, World! There are different techniques we can use to compress the embeddings down to just 2 dimensions while retaining as much information as possible. We'll use a technique called UMAP. And once we can get it down to 2 dimensions, we can plot these embeddings on a 2D chart.\n\n```python\nimport umap\nreducer = umap.UMAP(n_neighbors=49) \numap_embeds = reducer.fit_transform(doc_embeds)\n\ndf['x'] = umap_embeds[:,0]\ndf['y'] = umap_embeds[:,1]\n```\n\n```python\nchart = alt.Chart(df).mark_circle(size=500).encode(\n x=\n alt.X('x',\n scale=alt.Scale(zero=False),\n axis=alt.Axis(labels=False, ticks=False, domain=False)\n ),\n\n y=\n alt.Y('y',\n scale=alt.Scale(zero=False),\n axis=alt.Axis(labels=False, ticks=False, domain=False)\n ),\n \n tooltip=['search_term']\n )\n\ntext = chart.mark_text(align='left', dx=15, size=12, color='black'\n ).encode(text='search_term', color= alt.value('black'))\n\nresult = (chart + text).configure(background=\"#FDF7F0\"\n ).properties(\n width=1000,\n height=700,\n title=\"2D Embeddings\"\n )\n\nresult\n```\n\n\n\n",
+ "body": "[block:html]\n{\n \"html\": \"\\n\\n| \n | search_term | \n
|---|---|
| 0 | \nhow to print hello world in python | \n
| 1 | \nwhat is hello world | \n
| 2 | \nhow do you write hello world in an alert box | \n
| 3 | \nhow to print hello world in java | \n
| 4 | \nhow to write hello world in eclipse | \n
\\n
\\n\\n\"\n}\n[/block]\n\n\nHere we take a quick tour of what’s possible with language AI via Cohere’s Large Language Model (LLM) API. This is the Hello, World! of language AI, written for developers with little or no background in AI. In fact, we’ll do that by exploring the Hello, World! phrase itself.\n\nRead the accompanying [blog post here](https://cohere.com/blog/hello-world-p1/).\n\n[block:html]\n{\n \"html\": \"Hello World! Meet Language AI
\\n\"\n}\n[/block]\n\n\nWe’ll cover three groups of tasks that you will typically work on when dealing with language data, including:\n\n- Generating text\n- Classifying text\n- Analyzing text\n\nThe first step is to install the Cohere Python SDK. Next, create an API key, which you can generate from the Cohere [dashboard](https://os.cohere.ai/register) or [CLI tool](https://docs.cohere.ai/cli-key).\n\n```python\n! pip install cohere altair umap-learn -q\n```\n\n```python\nimport cohere\nimport pandas as pd\nimport numpy as np\nimport altair as alt\n\nco = cohere.Client(\"COHERE_API_KEY\") # Get your API key: https://dashboard.cohere.com/api-keys\n```\n\nThe Cohere Generate endpoint generates text given an input, called “prompt”. The prompt provides a context of what we want the model to generate text. To illustrate this, let’s start with a simple prompt as the input. \n\n### Try a Simple Prompt\n\n```python\nprompt = \"What is a Hello World program.\"\n\nresponse = co.chat(\n message=prompt,\n model='command-r')\n\nprint(response.text)\n```\n\n````\nA \"Hello World\" program is a traditional and simple program that is often used as an introduction to a new programming language. The program typically displays the message \"Hello World\" as its output. The concept of a \"Hello World\" program originated from the book *The C Programming Language* written by Kernighan and Ritchie, where the example program in the book displayed the message using the C programming language. \n\nThe \"Hello World\" program serves as a basic and straightforward way to verify that your development environment is set up correctly and to familiarize yourself with the syntax and fundamentals of the programming language. It's a starting point for learning how to write and run programs in a new language.\n\nThe program's simplicity makes it accessible to programmers of all skill levels, and it's often one of the first programs beginners write when learning to code. The exact implementation of a \"Hello World\" program varies depending on the programming language being used, but the core idea remains the same—to display the \"Hello World\" message. \n\nHere's how a \"Hello World\" program can be written in a few select languages:\n1. **C**:\n```c\n#include \n\n\n \n
\n
\n\nWe use the Embed endpoint to get the embeddings for each of these keywords.\n\n```python\ndef embed_text(texts, input_type):\n \"\"\"\n Turns a piece of text into embeddings\n Arguments:\n text(str): the text to be turned into embeddings\n Returns:\n embedding(list): the embeddings\n \"\"\"\n # Embed text by calling the Embed endpoint\n response = co.embed(\n model=\"embed-english-v3.0\",\n input_type=input_type,\n texts=texts)\n \n return response.embeddings\n```\n\n```python\ndf[\"search_term_embeds\"] = embed_text(texts=df[\"search_term\"].tolist(),\n input_type=\"search_document\")\ndoc_embeds = np.array(df[\"search_term_embeds\"].tolist())\n```\n\n### Semantic Search\n\nWe’ll look at a couple of example applications. The first example is semantic search. Given a new query, our \"search engine\" must return the most similar FAQs, where the FAQs are the 50 search terms we uploaded earlier.\n\n```python\nquery = \"what is the history of hello world\"\n\nquery_embeds = embed_text(texts=[query],\n input_type=\"search_query\")[0]\n```\n\nWe use cosine similarity to compare the similarity of the new query with each of the FAQs\n\n```python\n\nfrom sklearn.metrics.pairwise import cosine_similarity\n\ndef get_similarity(target, candidates):\n \"\"\"\n Computes the similarity between a target text and a list of other texts\n Arguments:\n target(list[float]): the target text\n candidates(list[list[float]]): a list of other texts, or candidates\n Returns:\n sim(list[tuple]): candidate IDs and the similarity scores\n \"\"\"\n # Turn list into array\n candidates = np.array(candidates)\n target = np.expand_dims(np.array(target),axis=0)\n\n # Calculate cosine similarity\n sim = cosine_similarity(target,candidates)\n sim = np.squeeze(sim).tolist()\n\n # Sort by descending order in similarity\n sim = list(enumerate(sim))\n sim = sorted(sim, key=lambda x:x[1], reverse=True)\n\n # Return similarity scores\n return sim\n```\n\nFinally, we display the top 5 FAQs that match the new query\n\n```python\nsimilarity = get_similarity(query_embeds,doc_embeds)\n\nprint(\"New query:\")\nprint(new_query,'\\n')\n\nprint(\"Similar queries:\")\nfor idx,score in similarity[:5]:\n print(f\"Similarity: {score:.2f};\", df.iloc[idx][\"search_term\"])\n```\n\n```\nNew query:\nwhat is the history of hello world \n\nSimilar queries:\nSimilarity: 0.58; how did hello world originate\nSimilarity: 0.56; where did hello world come from\nSimilarity: 0.54; why hello world\nSimilarity: 0.53; why is hello world so famous\nSimilarity: 0.53; what is hello world\n```\n\n### Semantic Exploration\n\nIn the second example, we take the same idea as semantic search and take a broader look, which is exploring huge volumes of text and analyzing their semantic relationships.\n\nWe'll use the same 50 top web search terms about Hello, World! There are different techniques we can use to compress the embeddings down to just 2 dimensions while retaining as much information as possible. We'll use a technique called UMAP. And once we can get it down to 2 dimensions, we can plot these embeddings on a 2D chart.\n\n```python\nimport umap\nreducer = umap.UMAP(n_neighbors=49) \numap_embeds = reducer.fit_transform(doc_embeds)\n\ndf['x'] = umap_embeds[:,0]\ndf['y'] = umap_embeds[:,1]\n```\n\n```python\nchart = alt.Chart(df).mark_circle(size=500).encode(\n x=\n alt.X('x',\n scale=alt.Scale(zero=False),\n axis=alt.Axis(labels=False, ticks=False, domain=False)\n ),\n\n y=\n alt.Y('y',\n scale=alt.Scale(zero=False),\n axis=alt.Axis(labels=False, ticks=False, domain=False)\n ),\n \n tooltip=['search_term']\n )\n\ntext = chart.mark_text(align='left', dx=15, size=12, color='black'\n ).encode(text='search_term', color= alt.value('black'))\n\nresult = (chart + text).configure(background=\"#FDF7F0\"\n ).properties(\n width=1000,\n height=700,\n title=\"2D Embeddings\"\n )\n\nresult\n```\n\n\n\n",
"html": "",
"htmlmode": false,
"fullscreen": false,
diff --git a/scripts/cookbooks-json/multilingual-search.json b/scripts/cookbooks-json/multilingual-search.json
index 806fc472..25da2d8c 100644
--- a/scripts/cookbooks-json/multilingual-search.json
+++ b/scripts/cookbooks-json/multilingual-search.json
@@ -13,7 +13,7 @@
},
"title": "Multilingual Search with Cohere and Langchain",
"slug": "multilingual-search",
- "body": "[block:html]\n{\n \"html\": \"\\n\\n| \n | search_term | \n
|---|---|
| 0 | \nhow to print hello world in python | \n
| 1 | \nwhat is hello world | \n
| 2 | \nhow do you write hello world in an alert box | \n
| 3 | \nhow to print hello world in java | \n
| 4 | \nhow to write hello world in eclipse | \n
\\n
\\n\\n\"\n}\n[/block]\n\n\n***Read the accompanying [blog post here](https://txt.cohere.ai/search-cohere-langchain/).***\n\nThis notebook contains two examples for performing multilingual search using Cohere and Langchain. Langchain is a library that assists the development of applications built on top of large language models (LLMs), such as Cohere's models.\n\nIn short, Cohere makes it easy for developers to leverage LLMs and Langchain makes it easy to build applications with these models.\n\nWe'll go through the following examples:\n- **Example 1 - Basic Multilingual Search**\n\n This is a simple example of multilingual search over a list of documents.\n\n The steps in summary:\n - Import a list of documents\n - Embed the documents and store them in an index\n - Enter a query\n - Return the document most similar to the query\n- **Example 2 - Search-Based Question Answering**\n\n This example shows a more involved example where search is combined with text generation to answer questions about long-form documents.\n\n The steps in summary:\n - Add an article and chunk it into smaller passages\n - Embed the passages and store them in an index\n - Enter a question\n - Answer the question based on the most relevant documents\n\n\n```python\nfrom langchain.embeddings.cohere import CohereEmbeddings\nfrom langchain.llms import Cohere\nfrom langchain.prompts import PromptTemplate\nfrom langchain.text_splitter import RecursiveCharacterTextSplitter\nfrom langchain.chains.question_answering import load_qa_chain\nfrom langchain.chains import RetrievalQA\nfrom langchain.vectorstores import Qdrant\nfrom langchain.document_loaders import TextLoader\nimport textwrap as tr\nimport random\nimport dotenv\nimport os\n\ndotenv.load_dotenv(\".env\") # Upload an '.env' file containing an environment variable named 'COHERE_API_KEY' using your Cohere API Key\n```\n\n\n\n\n True\n\n\n\n\n[block:html]{\"html\":\"Multilingual Search with Cohere and Langchain
\\n\\n
\\n\\n\"\n}\n[/block]\n\n\n***Read the accompanying [blog post here](https://cohere.com/blog/search-cohere-langchain/).***\n\nThis notebook contains two examples for performing multilingual search using Cohere and Langchain. Langchain is a library that assists the development of applications built on top of large language models (LLMs), such as Cohere's models.\n\nIn short, Cohere makes it easy for developers to leverage LLMs and Langchain makes it easy to build applications with these models.\n\nWe'll go through the following examples:\n- **Example 1 - Basic Multilingual Search**\n\n This is a simple example of multilingual search over a list of documents.\n\n The steps in summary:\n - Import a list of documents\n - Embed the documents and store them in an index\n - Enter a query\n - Return the document most similar to the query\n- **Example 2 - Search-Based Question Answering**\n\n This example shows a more involved example where search is combined with text generation to answer questions about long-form documents.\n\n The steps in summary:\n - Add an article and chunk it into smaller passages\n - Embed the passages and store them in an index\n - Enter a question\n - Answer the question based on the most relevant documents\n\n\n```python\nfrom langchain.embeddings.cohere import CohereEmbeddings\nfrom langchain.llms import Cohere\nfrom langchain.prompts import PromptTemplate\nfrom langchain.text_splitter import RecursiveCharacterTextSplitter\nfrom langchain.chains.question_answering import load_qa_chain\nfrom langchain.chains import RetrievalQA\nfrom langchain.vectorstores import Qdrant\nfrom langchain.document_loaders import TextLoader\nimport textwrap as tr\nimport random\nimport dotenv\nimport os\n\ndotenv.load_dotenv(\".env\") # Upload an '.env' file containing an environment variable named 'COHERE_API_KEY' using your Cohere API Key\n```\n\n\n\n\n True\n\n\n\n\n[block:html]{\"html\":\"Multilingual Search with Cohere and Langchain
\\n\\n
\\n\\n\"\n}\n[/block]\n\nThis notebook contains the starter code to do simple [semantic search](https://txt.cohere.ai/what-is-semantic-search/) on the [Wikipedia embeddings archives](https://txt.cohere.ai/embedding-archives-wikipedia/) published by Cohere. These archives embed Wikipedia sites in multiple languages. In this example, we'll use [Wikipedia Simple English](https://huggingface.co/datasets/Cohere/wikipedia-22-12-simple-embeddings). \n\nLet's now download 1,000 records from the English Wikipedia embeddings archive so we can search it afterwards.\n\n\n```python\nfrom datasets import load_dataset\nimport torch\nimport cohere\n\nco = cohere.Client(\"\") \n\n#Load at max 1000 documents + embeddings\nmax_docs = 1000\ndocs_stream = load_dataset(f\"Cohere/wikipedia-22-12-simple-embeddings\", split=\"train\", streaming=True)\n\ndocs = []\ndoc_embeddings = []\n\nfor doc in docs_stream:\n docs.append(doc)\n doc_embeddings.append(doc['emb'])\n if len(docs) >= max_docs:\n break\n\ndoc_embeddings = torch.tensor(doc_embeddings)\n```\n\n\n Downloading: 0%| | 0.00/1.29k [00:00\\n \\n Wikipedia Semantic Search with Cohere Embedding Archives
\\n\\n \\n
\\n Back to Cookbooks\\n \\n\\n \\n Open in GitHub\\n \\n \\n
\\n \\n\\n\\n\\n
\\n\\n\"\n}\n[/block]\n\nThis notebook contains the starter code to do simple [semantic search](https://cohere.com/llmu/what-is-semantic-search/) on the [Wikipedia embeddings archives](https://cohere.com/blog/embedding-archives-wikipedia/) published by Cohere. These archives embed Wikipedia sites in multiple languages. In this example, we'll use [Wikipedia Simple English](https://huggingface.co/datasets/Cohere/wikipedia-22-12-simple-embeddings). \n\nLet's now download 1,000 records from the English Wikipedia embeddings archive so we can search it afterwards.\n\n\n```python\nfrom datasets import load_dataset\nimport torch\nimport cohere\n\nco = cohere.Client(\"\") \n\n#Load at max 1000 documents + embeddings\nmax_docs = 1000\ndocs_stream = load_dataset(f\"Cohere/wikipedia-22-12-simple-embeddings\", split=\"train\", streaming=True)\n\ndocs = []\ndoc_embeddings = []\n\nfor doc in docs_stream:\n docs.append(doc)\n doc_embeddings.append(doc['emb'])\n if len(docs) >= max_docs:\n break\n\ndoc_embeddings = torch.tensor(doc_embeddings)\n```\n\n\n Downloading: 0%| | 0.00/1.29k [00:00
diff --git a/scripts/cookbooks-mdx/fueling-generative-content.mdx b/scripts/cookbooks-mdx/fueling-generative-content.mdx
index 11d1d1d5..b2316064 100644
--- a/scripts/cookbooks-mdx/fueling-generative-content.mdx
+++ b/scripts/cookbooks-mdx/fueling-generative-content.mdx
@@ -135,7 +135,7 @@ slug: /page/fueling-generative-content
Generative models have proven extremely useful in content idea generation. But they don’t take into account user search demand and trends. In this notebook, let’s see how we can solve that by adding keyword research into the equation.
-Read the accompanying [blog post here](https://txt.cohere.ai/generative-content-keyword-research/).
+Read the accompanying [blog post here](https://cohere.com/blog/generative-content-keyword-research/).
```python PYTHON
! pip install cohere -q
diff --git a/scripts/cookbooks-mdx/hello-world-meet-ai.mdx b/scripts/cookbooks-mdx/hello-world-meet-ai.mdx
index b7a4b089..de119e06 100644
--- a/scripts/cookbooks-mdx/hello-world-meet-ai.mdx
+++ b/scripts/cookbooks-mdx/hello-world-meet-ai.mdx
@@ -135,7 +135,7 @@ slug: /page/hello-world-meet-ai
Here we take a quick tour of what’s possible with language AI via Cohere’s Large Language Model (LLM) API. This is the Hello, World! of language AI, written for developers with little or no background in AI. In fact, we’ll do that by exploring the Hello, World! phrase itself.
-Read the accompanying [blog post here](https://txt.cohere.ai/hello-world-p1/).
+Read the accompanying [blog post here](https://cohere.com/blog/hello-world-p1/).
Wikipedia Semantic Search with Cohere Embedding Archives
\\n
diff --git a/scripts/cookbooks-mdx/multilingual-search.mdx b/scripts/cookbooks-mdx/multilingual-search.mdx
index b9d704e4..75de0baf 100644
--- a/scripts/cookbooks-mdx/multilingual-search.mdx
+++ b/scripts/cookbooks-mdx/multilingual-search.mdx
@@ -133,7 +133,7 @@ slug: /page/multilingual-search
-***Read the accompanying [blog post here](https://txt.cohere.ai/search-cohere-langchain/).***
+***Read the accompanying [blog post here](https://cohere.com/blog/search-cohere-langchain/).***
This notebook contains two examples for performing multilingual search using Cohere and Langchain. Langchain is a library that assists the development of applications built on top of large language models (LLMs), such as Cohere's models.
diff --git a/scripts/cookbooks-mdx/wikipedia-semantic-search.mdx b/scripts/cookbooks-mdx/wikipedia-semantic-search.mdx
index fc1c6a26..7b456d05 100644
--- a/scripts/cookbooks-mdx/wikipedia-semantic-search.mdx
+++ b/scripts/cookbooks-mdx/wikipedia-semantic-search.mdx
@@ -132,7 +132,7 @@ slug: /page/wikipedia-semantic-search
}
-This notebook contains the starter code to do simple [semantic search](https://txt.cohere.ai/what-is-semantic-search/) on the [Wikipedia embeddings archives](https://txt.cohere.ai/embedding-archives-wikipedia/) published by Cohere. These archives embed Wikipedia sites in multiple languages. In this example, we'll use [Wikipedia Simple English](https://huggingface.co/datasets/Cohere/wikipedia-22-12-simple-embeddings).
+This notebook contains the starter code to do simple [semantic search](https://cohere.com/llmu/what-is-semantic-search/) on the [Wikipedia embeddings archives](https://cohere.com/blog/embedding-archives-wikipedia/) published by Cohere. These archives embed Wikipedia sites in multiple languages. In this example, we'll use [Wikipedia Simple English](https://huggingface.co/datasets/Cohere/wikipedia-22-12-simple-embeddings).
Let's now download 1,000 records from the English Wikipedia embeddings archive so we can search it afterwards.
@@ -167,7 +167,7 @@ doc_embeddings = torch.tensor(doc_embeddings)
Using custom data configuration Cohere--wikipedia-22-12-simple-embeddings-94deea3d55a22093
-Now, `doc_embeddings` holds the embeddings of the first 1,000 documents in the dataset. Each document is represented as an [embeddings vector](https://txt.cohere.ai/sentence-word-embeddings/) of 768 values.
+Now, `doc_embeddings` holds the embeddings of the first 1,000 documents in the dataset. Each document is represented as an [embeddings vector](https://cohere.com/llmu/sentence-word-embeddings/) of 768 values.
```python PYTHON