Merge remote-tracking branch 'origin' into prod

README.md

@@ -2,32 +2,32 @@
 
 Work in Progress
 
-Sawt is a tool designed to bridge the communication gap between New Orleanians and their city council representatives. 
+Sawt is a tool designed to bridge the communication gap between New Orleanians and their city council representatives.
 
-## Quickstart (no web)
-1. `cd packages/backend && pip install -r requirements && python src`
-2. `cd packages/googlecloud/functions && pip install -r requirements && python getanswer`
+## Prerequisites
 
-## Project structure
-- `backend`: Preprocessor, only needs to be run once
-- `web`: NextJS web site
-- `googlecloud`: Google Cloud Function invoked for every query
+- Install [DVC](https://dvc.org/doc/install)
 
-## Getting started 
-Follow the instructions below to run test locally 
-```bash
-# Pull data (transcripts, minutes, agendas) from the preprocessed stage using DVC
-dvc pull
+## Quickstart
 
-# Navigate to the 'src' directory inside 'packages/backend'
-cd packages/backend/src
+1. Pull data: `dvc pull`
+1. Start cloud function locally:
 
-# Install the required Python dependencies
+```
+cd packages/googlecloud/functions
 pip install -r requirements.txt
+OPENAI_API_KEY=sk-XXX functions-framework --target=getanswer --debug
+```
 
-# Navigate to the 'getanswer' function directory inside 'packages/googlecloud/functions'
-cd ../../googlecloud/functions/getanswer 
+1. Get answer:
 
-# Run the main Python script
-python __main__.py
 ```
+curl -XPOST http://localhost:8080 -v -H "Content-Type: application/json" \
+    -d '{"query":"Is surveillance effective?","response_type":"in_depth"}'
+```
+
+## Project structure
+
+- `packages/backend`: Preprocessor, only needs to be run once
+- `packages/googlecloud`: Google Cloud Function invoked for every query
+- `packages/web`: NextJS web site

packages/backend/.env.example

@@ -0,0 +1 @@
+OPENAI_API_KEY=

packages/backend/src/__main__.py

@@ -15,6 +15,7 @@ def main():
     cj_json_directory = "json_cj_directory"
     fc_json_directory = "json_fc_directory"
     pc_json_directory = "json_public_comment_directory"
+    news_json_directory = "news_directory"
     print(f"Preprocessing videos, agendas, and minutes to generate a cache.")
     general_embeddings, in_depth_embeddings = create_embeddings()
 
@@ -24,6 +25,7 @@ def main():
         cj_json_directory,
         pdf_directory,
         pc_json_directory,
+        news_json_directory,
         general_embeddings,
         in_depth_embeddings,
     )

packages/backend/src/cache/faiss_index_general.dvc

@@ -1,6 +1,6 @@
 outs:
-- md5: e50dbff10213e5c44c74e31317370fea.dir
-  size: 59806708
+- md5: f86af62933bae171911ae891560b2407.dir
+  size: 61663926
   nfiles: 2
   hash: md5
   path: faiss_index_general

packages/backend/src/cache/faiss_index_in_depth.dvc

@@ -1,6 +1,6 @@
 outs:
-- md5: 048b83157f31c250755ba73dc6b08f69.dir
-  size: 59806708
+- md5: ed5520e1a6f903d4d38105dd3d9d034b.dir
+  size: 61663926
   nfiles: 2
   hash: md5
   path: faiss_index_in_depth

packages/backend/src/news_directory/Frequent_outages_could_lead_to_big_fines_for_Entergy_New_Orleans_under_new_standards.json

@@ -0,0 +1,9 @@
+{
+    "messages": [
+        {
+            "page_content": "HOME\nABOUT\nSUBSCRIBE\nDONATE\nELECTION 2023\nGOVT + POLITICS\nENVIRONMENT\nHEALTH\nCRIMINAL JUSTICE\nEDUCATION\nBUSINESS + LABOR\nGOVT + POLITICS\nWEATHER\nFrequent outages could lead to big fines for Entergy New Orleans under new standards\nBY: MICHAEL ISAAC STEIN, VERITE - FEBRUARY 18, 2023 10:00 AM\nEntergy\u2019s New Orleans Power Station. (Photo by Julie Dermansky)\nThe New Orleans City Council established a new set of reliability standards Thursday for Entergy New Orleans that could lead to fines up to $3.7 million per year if the number of annual blackouts exceeds those standards. The council is the regulator of Entergy New Orleans.\n\u201cEntergy New Orleans must meet our standards or owe the people of New Orleans, their customers, millions of dollars in fines,\u201d Councilwoman Helena Moreno said at a utility committee meeting this month. \u201cI believe we\u2019re one of the only regulators in the southern region that\u2019s set up these types of reliability standards.\u201d\nThe council hopes the new standards will finally allow it to collect a $1 million fine it imposed on Entergy in 2019 over frequent outages. The fine was levied after a council investigation found that the grid\u2019s reliability fell sharply shortly after the company cut millions of dollars from its budget to maintain it.\nEntergy challenged the fine in court, arguing it was unfair because the council hadn\u2019t established reliability standards. An Orleans Parish Civil District Court judge ruled in Entergy\u2019s favor in June.\nMoreno\u2019s chief of staff Andrew Tuozzolo told Verite that the council\u2019s plan was now to re-levy the $1 million fine, in the hopes that the new reliability standards will make it easier to defend in court.\nThe new standards only apply to \u201cfair-weather day\u201d outages, and exclude outages caused by major events like hurricanes. The council is currently conducting a separate process to create new rules and improvement plans for the grid\u2019s storm resilience.\nWhile hurricane-related outages are a well-known and deadly occurrence in New Orleans, the city has also struggled for years with frequent outages on sunny days. In fact, the process to pass these new rules began in 2017, when complaints grew about the number of blackouts happening on clear days, when the grid should have been operating normally.\nIn 2019, a council investigation found that Entergy failed in its responsibility to maintain an adequate grid. It found the city experienced 2,599 outages between June 1, 2016 and May 31, 2017 alone, the majority of which occurred on fair weather days. The report said the main cause of the outages were equipment failures in the local distribution grid \u2014 the poles and lines that run down every street and deliver electricity directly to buildings.\nThe investigation also found that the grid\u2019s reliability started to fall shortly after Entergy New Orleans made multi-million dollar cuts to investing in distribution system maintenance and improvement. That investigation led to the council\u2019s decision to fine Entergy $1 million.\nUnder the measure passed Thursday, Entergy will have three new reliability standards that could lead to the fine.\nThe first standard is how many fair weather outages New Orleans customers face each year. That\u2019s based on an industry standard called the system average interruption frequency index (SAIFI). The minimum SAIFI established by the council is 1.53 \u2014 meaning that a customer in New Orleans should experience, on average, 1.53 outages per year. The exact fine depends on how off that mark Entergy was in a given year, with a maximum penalty of $2.7 million.\nSUPPORT NEWS YOU TRUST.\nDONATE\nThe second standard is about the duration of fair weather outages every year, based on the system average interruption duration index (SAIDI). The maximum baseline SAIDI score for Entergy was set at 178.2, meaning New Orleans customers on average would experience 178.2 minutes \u2014 roughly three hours \u2014 of outages each year. Entergy could face an annual fine of $500,000 for failing to achieve that.\nThe last standard could lead to fines up to $500,000 if Entergy fails to adequately improve the grid\u2019s worst performing feeders \u2014 a piece of equipment in the distribution system that has been blamed for outages in the past.\nThe council\u2019s ordinance notes that under the newly established fine schedule, Entergy\u2019s poor reliability in 2016 and 2017 \u2014 for which the council originally levied a $1 million fine \u2014 would have resulted in a fine of almost the exact same amount \u2014 $1,005,000.\nRepresentatives of the Alliance for Affordable Energy, which advocates on behalf of customers,  applauded the move. But they also said they believe the council should go farther and use these new standards as a starting point.\n\u201cWe\u2019re certainly supportive of having them in place, but we\u2019d like to note this is merely a first step,\u201d Alliance policy director Jesse George said at the utility meeting this month. \u201cWe encourage the council to continue making these standards more stringent over time.\u201d\nThe council\u2019s utility advisers noted that from 2013 through 2021, Entergy New Orleans would have only run afoul of the new standards in two years, 2016 and 2017.\n\u201cI think if you ask the average New Orleanian whether their electric service was reliable over the last five years they would probably say no,\u201d George said in a recent Facebook video.\nMoreno said during the committee meeting that because of pressure from the council, Entergy New Orleans had improved its reliability.\n\u201cTheir reliability is definitely not perfect, but it has improved due to insistence from the City Council,\u201d Moreno said. \u201cEntergy New Orleans has now increased its reliability performance to just above the national average. This is progress, but we are certainly far from done.\u201d\nEntergy did not respond to a request for comment.\nThis article first appeared on Verite and is republished here under a Creative Commons license.\nREPUBLISH\nOur stories may be republished online or in print under Creative Commons license CC BY-NC-ND 4.0. We ask that you edit only for style or to shorten, provide proper attribution and link to our web site. Please see our republishing guidelines for use of photos and graphics.\nMICHAEL ISAAC STEIN, VERITE\nBefore joining Verite, Michael Isaac Stein spent five years as an investigative reporter at The Lens, a nonprofit New Orleans news publication, covering local government, housing and labor issues. Before working at The Lens, Stein was a reporter for WWNO New Orleans Public Radio and freelanced for various national publications including The Intercept, The New Republic and Bloomberg\u2019s CityLab. He holds a bachelor\u2019s degree from the University of Michigan\u2019s Ford School of Public Policy.\nMORE FROM AUTHOR\nRELATED NEWS\nEntergy threatens \u2018expensive litigation\u2019 over $1 million\u2026\nBY MICHAEL ISAAC STEIN, VERITE\nMay 13, 2023\nAfter strain from another winter storm, experts say it\u2019s\u2026\nBY ROBERT ZULLO\nDecember 31, 2022\nSHINING A LIGHT ON THE BAYOU STATE\nDemocracy Toolkit //\nRegister to vote\n|\nFind your voting precinct\n|\nBecome an election worker\n|\nConduct a voter registration drive\n|\nContact your state and federal lawmakers\nABOUT US\nThe Louisiana Illuminator is an independent, nonprofit, nonpartisan news organization with a mission to cast light on how decisions in Baton Rouge are made and how they affect the lives of everyday Louisianians. Our in-depth investigations and news stories, news briefs and commentary help residents make sense of how state policies help or hurt them and their neighbors statewide.\nOur stories may be republished online or in print under Creative Commons license CC BY-NC-ND 4.0. We ask that you edit only for style or to shorten, provide proper attribution and link to our web site.\nDEIJ Policy | Ethics Policy | Privacy Policy\n\u00a9 Louisiana Illuminator, 2023",
+            "url": "https://lailluminator.com/2023/02/18/frequent-outages-could-lead-to-big-fines-for-entergy-new-orleans-under-new-standards/",
+            "title": "Frequent outages could lead to big fines for Entergy New Orleans under new standards"
+        }
+    ]
+}

packages/backend/src/news_directory/How_should_New_Orleans_spend_the_millions_left_in_pandemic_relief_aid_A_new_debate_begins.json

@@ -0,0 +1,9 @@
+{
+    "messages": [
+        {
+            "page_content": "Skip to main content\nHomes\nCalendar\nStore\nPublic Notices\nE-Edition\nNewsletters\nSubscribe for $1\nHow should New Orleans spend the millions left in pandemic relief aid? A new debate begins\nBY MATT SLEDGE | Staff writer\nApr 4, 2023\n3 min to read\n1 of 2\nPeople who live on the streets are fed a hot meal in New Orleans, Sunday, Dec. 25, 2022.\nSTAFF PHOTO BY DAVID GRUNFELD\nFacebook\nTwitter\nEmail\nPrint\nCopy article link\nSave\nSolutions for crime and homelessness would claim tens of millions of the city\u2019s remaining federal relief dollars under a plan unveiled by Mayor LaToya Cantrell's administration on Tuesday, while progressive groups made a much larger pitch that would also call for the use of money from the general fund.\nIn the city\u2019s vision, the New Orleans Health Department would quarterback programs designed to prevent violence, such as mental health services for school children and a rejuvenated street team of violence \u201cinterrupters.\u201d\nAlong with a $10 million plan targeting homelessness, the plan would use $38.4 million of the remaining $53.6 million in pandemic relief funds from the American Rescue Plan Act.\nA coalition of progressive groups, by contrast, told the City Council that the city\u2019s one-time spending should rise much higher. They proposed a $147 million package, including $107 million for housing, that would also draw from a growing pot of unspent general fund dollars.\nThe council members who will have the final say sounded receptive notes but didn\u2019t make firm commitments. Next week, they will hear from the city\u2019s chief administrative officer, Gilbert Monta\u00f1o, who prefers keeping more money socked away.\nThe debate that will play out in the coming weeks is in many ways a repeat of the 2023 budget process, which culminated in a last-minute, $262 million amendment to use one-time funds. Advocates said they were caught off guard in December, and this time around, they\u2019re keen to make their voices heard.\n\u201cNow the question is, how do you reconcile what the city wants to do with the (relief) money with what we heard today from advocacy groups and from people interested in how that money is spent?\u201d said City Council member Joe Giarrusso, who chairs the budget committee.\nCity plan\nThe late-breaking budget amendment last year aimed lots of one-time money at the most urgent issue for many voters -- violent crime. Big spends included $22.5 million for police recruitment and retention, $30 million for new criminal justice software and $4.9 million for juvenile probation and parole.\nBut some advocates complained that far too many of those funds were directed at punitive measures instead of social services. The Cantrell administration is now proposing a package aimed at preventing and interrupting violence through non-policing measures.\nThe city\u2019s plan includes $10 million for a joint program between NOLA Public Schools and Children\u2019s Hospital to place mental health professionals, case managers and expanded services in the \u201chighest risk\u201d schools where kids need help to address trauma. Separately, the school system would partner with the city's Office of Workforce Development for an $8.1 million program to put at-risk high schoolers on career paths.\nMayor LaToya Cantrell and Dr. Jennifer Avegno, pictured in 2020 \nPHOTO BY MAX BECHERER / THE TIMES-PICAYUNE/NEW ORLEANS ADVOCATE\nOther expenditures would include $500,000 for a rebooted violence interruption street team, $1 million for a blight remediation partnership between Tulane public health researchers and city code enforcement, $2.5 million for pocket parks, $5 million for financial assistance programs and $1 million for food security programs.\nThe city\u2019s package is designed to put a \u201cdown payment\u201d on projects that have been proven to reduce crime, said Dr. Jennifer Avegno, health department director. The City Council in January passed a motion mandating that her department create a violence reduction program.\n\u201cIf you don\u2019t make significant investments in the root causes of crime and violence \u2013 and certainly, from the public health perspective, shootings and killings, which are a major cause of mortality in this city \u2013 then you\u2019re not getting a lot for your money,\u201d she said.\nSeparately, the city would spend $10 million on reducing homelessness.\nThe city\u2019s proposals draw only from American Rescue Plan Act funds. However, advocates note that the city also has a growing pot of unspent general fund dollars \u2013 pegged at $274 million in a recent city report.\nCAO Gilbert Montano, right \nPHOTO BY BRETT DUKE / THE TIMES-PICAYUNE\nMonta\u00f1o said that number represents an unaudited point-in-time figure. He acknowledged that the city\u2019s fund balance is \u201csizable\u201d but said he wants to keep a healthy reserve in place for emergencies and economic downturns.\n\u201cIf we don\u2019t protect the city for the future, then I don't think we\u2019re being good stewards,\u201d he said.\nAdvocates\u2019 plan\nThe community groups pitching the council on Tuesday proposed a much more aggressive spending plan. They said it would be a mistake to continue using the federal pandemic relief dollars to build up the city\u2019s fund balance.\nPresenters from the Louisiana Fair Housing Action Center, New Orleans Workers Center for Racial Justice and the Vera Institute said they had contacted dozens of community groups for input and conducted their own vetting project to come up with a wish-list of mostly existing projects that could benefit from infusions of one-time cash.\nThe top item on that list was $107 million for housing, including $70 million for the construction of affordable rental housing, $15 million for reducing street homelessness and $10 million for the renovation of existing but substandard rental units.\nOther proposals include a $20 million investment in youth development through community organizations and $18 million for \"community equity\" projects including $5 million for free public transit fares, $5 million for food banks and food producers, and $5.2 million in direct cash assistance for people who were left out of earlier rounds of government assistance during the pandemic.\nWill Snowden, director of Vera Institute of Justice's New Orleans office\nPROVIDED PHOTO\nWhile last year\u2019s budget appropriated tens of millions to agencies that respond to crime, Will Snowden, the director of the Vera Institute in New Orleans, said it was time to pivot to addressing root causes. He saw the groups\u2019 proposal as a floor, rather than a ceiling.\n\u201cThere can be more. When we\u2019re having conversations in New Orleans about what provides public safety and what creates public safety, it is these kinds of investments,\u201d he said.\nEmail Matt Sledge at [email protected].\nFacebook\nTwitter\nEmail\nPrint\nCopy article link\nSave\nCity ARPA plan\nCity of New Orleans Apr 5, 2023\nMORE INFORMATION\nThe Nightly Top 5: Catch up on today's top stories\nDebate continues over how New Orleans should spend millions in pandemic relief aid, S&WB walks back its threat to cut water to tenants wit\u2026\nTags\nHardwall\nThis Day in History\nSponsored by Connatix\nRecommended for you\nRecommended by\nSECTIONS\nHOME\nNEWS\nOPINION\nSPORTS\nENTERTAINMENT/LIFE\nNEWSLETTERS\nGAMES\nSERVICES\nCLASSIFIEDS\nSEARCH\nSUBSCRIBE | GROUPS\nDIGITAL ADVERTISING\nHELP/CONTACT US\nRSS FEEDS\nMEDIA KIT\nEEDITION\nCAREERS\nTEACHER'S LINK\nREVIEWING THE RECORD\nOUR SITES\nOBITUARIES\nJOBS\nCELEBRATIONS\nNIE\nCLASSIFIEDS\nHOMES\nPETS\nARCHIVES\nSTORE\nCONTACT INFORMATION\nnola.com\n840 St. Charles Avenue\nNew Orleans, LA 70130\nPhone: 504-529-0522\n\nNews Tips:\[email protected]\nOther questions:\[email protected]\nNeed help?\nReport a delivery issue\nCreate a temporary stop\nSign up for recurring payments\nPay your bill\nUpdate your billing info\n\n\u00a9 Copyright 2023 NOLA.com 840 St. Charles Avenue, New Orleans, LA  | Terms of Use | Privacy Policy\nPowered by BLOX Content Management System from BLOX Digital.\n  This website stores data such as cookies to enable essential website functionality, marketing, personalization and analytics. By remaining on this website you indicate your consent. See updated terms and conditions.",
+            "url": "https://www.nola.com/news/politics/how-should-new-orleans-spend-millions-of-one-time-funds-new-debate-begins/article_ef36af06-d320-11ed-b9fe-978f03c2909b.html",
+            "title": "How should New Orleans spend the millions left in pandemic relief aid? A new debate begins"
+        }
+    ]
+}