scraping opinions

[legaltextai]

Is there a way I could help scraping court opinions from Indiana into CL database? The latest CL has is from 2021. I saw somewhere an extension. Is the process semi-automatic or will I need to do heavy adjustments?
The opinions are publicly accessible.

[mlissner]

We do need help maintaining our scrapers. Check out Juriscraper and see if you can sort out the problem?

[flooie]

@legaltextai - we updated the Indiana scraper recently and the most recent opinion is from November 21st, 2023 - im sure we could use a back scraper to collect all the missing opinions still though.

[legaltextai]

thanks! is it now within bulk downloads? how can i help?

[mlissner]

It should show up in the bulk data files when they're next generated. Bill, what's the best way for a volunteer to contribute a back scraper? Is it best for them to do the scrape and contribute the data or to contribute the code and we do the scrape or something else?

[flooie]

I see 33 opinions from 2022 on from Indiana Supreme Court. Do you see that too?

[flooie]

https://www.courtlistener.com/?q=court_id%3Aind&type=o&order_by=dateFiled%20desc&stat_Precedential=on&filed_after=12%2F31%2F2021&filed_before=01%2F21%2F2024

[legaltextai]

I am using bulk updates -> database. am trying to figure out at the moment at which step i missed those.

[legaltextai]

SELECT MAX(date_filed) AS latest_date, court_short_name FROM search_opinion_cluster_docket_court WHERE court_short_name = 'Indiana Court of Appeals' GROUP BY court_short_name;

After I join the table with citations file, the latest decision is from 2019

SELECT MAX(date_filed) AS latest_date, court_short_name FROM search_opinion_cluster_docket_court_citation WHERE court_short_name = 'Indiana Court of Appeals' GROUP BY court_short_name;

Could it be because the citations file has not been updated?

[legaltextai]

@mlissner @flooie the citation file is not updated, and therefore, when joined, ignores lots of decisions with dates after 01-2023, if not earlier. that's at least the case for Indiana
For example,
None of the 'cluster_id' from these decisions:
SELECT * FROM search_opinion_cluster_docket_court WHERE court_short_name = 'Indiana Court of Appeals' AND date_filed > '2023-01-01' ORDER BY date_filed ASC LIMIT 10;
is in the citation file:
select * from search_citation where cluster_id = 9439621 [as an example];
They are only joinable on cluster_id, correct?

[mlissner]

Hm, you've got the data in your own schema, so I'm not feeling particularly confident to reply knowledgeably, but looking at the bulk data files, it sure looks like citations are being exported OK (the last one was in early december though, hm).

I suspect what you're seeing is that we haven't added any new citations to the citations table for that court, which is entirely possible since citations are difficult to come by.

[mlissner]

Just like any other endpoint. Do an OPTIONS request and it'll tell you which filters it has. From there, you can filter accordingly (I think you'll wind up using the opinion ID).

[legaltextai]

I am very sorry but I don't see 'citation' in the options.
Can you pls show how to find a citation (volume, reporter, page) for this opinion_id 9897446.
Thanks.

[mlissner]

Sorry, I think I confused your request with a different one I was fielding. If you want to get the citation for an opinion you can just look at the cluster ID:

curl 'https://www.courtlistener.com/api/rest/v3/clusters/?sub_opinions=9897446&fields=citations'

That would work generally, but unfortunately we don't have any citations associated with this case.

Did I understand correctly this time? :)

[legaltextai]

You did. Thank you. I take it that API is not different from bulk files then (citations-2023-12-04.csv.bz2 in this case). I was hoping there was a way to pull missing citations from API.

[mlissner]

Well, the bulk files are once a month exports from our DB and the API is real time access to the DB, but other than that, no, it's the same underlying data.