-
Notifications
You must be signed in to change notification settings - Fork 14
Home
This is a public-private partnership with the City of New York and civic technologists looking to unlock 15 years of PDFs. In these PDFs, you will find the canonical data of City solicitation procurement notices and awards, public hearings, meetings, court notices, property dispositions, agency public hearings, agency rules, and changes in personnel.
The goal of this wiki is to document schema, methodology, and tools.
- Proposed Project Phases
- Similar Schemas
- Proposed Schema
- How to convert PDFs
- Listing of potential tools
- CityRecord Online User Stories
- Open Source Document Management Tools
Currently, there are three things you can do.
- Join the City Record Online Working Group, aka CROW discussion list.
- Help download and share PDFs
- Help document tools to help scrape
- Help document schema
This project's main discussion list is located on a Google Group hosted by BetaNYC.
Currently, we are working on developing a number of outlets to download this treasure trove of information. In total, there are 16.3 gigs of archival PDFs. If you have any problems downloading these files, report an issue on GitHub or the Discussion List
- 1998 to 2008 are scaned documents
- March 2008 till present are 'text selectable'
If you have a tool that will crawl and download websites, you can download all of the PDFs from the City.
- Browse All City Records
- Download XML Listing for All City Records
- Suggested tools - SiteSucker, a Mac OSX, Windows Tool, SUGGESTION NEEDED
We have shared the complete collection of files via Dropbox. You can download them individually or you can add the primary folder and sync to a local storage device. (shareable link bit.ly/dropbox-crow)
As a bit of an experiment we are using BitTorrent protocol. This is the complete 16.3 gigs. Please help by downloading and socializing these PDFs. You will need to download BitTorrent Sync and use the following read-only 'secret' passcode "BDGM4KAQHZ6XII2JNJREDX6VDN3QTLI7G" (shareable link bit.ly/bts-crow)
BetaNYC is hosting an FTP server with all of the PDFs. These files can be fetched anonymously via files.betanyc.us.
If you have the Google Drive for your computer, you can download all of the PDFs to your local computer. We are in the process of uploading all of the documents. For now, you can only access the text selectable documents (March 2008 till present) via http://bit.ly/gdocs-crow
On 24 March 2009, Intro 952-2009 was introduced to the City Council to amend the charter and create a website and effectively place the New York City Record online and free of charge. On 28 April, the Committee on Governmental Operations held a hearing, at which good government groups such as Citizens Union testified in support of the proposal. The bill never made it out of committee and died on 31 December 2009.
Last year, members of BetaNYC penned The People’s Guide to a Digital New York City and outlined 32 issues future New York City governments should address. Recommendation #14 requested the publication of the City Record “online” and in “more useful ways.” We argued: “As the city record is the canonical database of solicitation procurement notices and awards, public hearings, meetings, court notices, property dispositions, agency public hearings, agency rules, and changes in personnel, the City Record needs to be envisioned as a portal accessible via all types of devices for all New Yorkers. Starting with real-time data feeds or APIs, reimagining this site would unlock opportunities for small business to know when and where they could bid on services and engage in government. Finally, by producing this tool in a more open format, the people of New York would finally get access to the true inner workings of government.”
On 29 May 2014, the New York City Council introduced Intro 0363-2014. This bill states, “[a]ll information published in the City Record after the effective date of the local law that created this subdivision shall be available as soon as possible, but no later than 24 hours of publishing, at no charge on a website maintained by or on behalf of the city of New York as well as on a single web portal that is linked to nyc.gov or any successor website maintained by, or on behalf of, the city of New York created pursuant to section 23-502 of the administrative code. Such information shall be available in both a non-proprietary, machine-readable format and a human-readable format and shall be capable of being downloaded in bulk. Such information shall be searchable by, at minimum, date of publication, relevant agency, keyword, and category, such as public hearings, procurement notices, and changes in personnel… This local law shall take effect one year after its enactment, provided, however, that the department of citywide administrative services shall take such actions prior to such time as are necessary for timely implementation of this local law.”
The drafting of this legislation was undertaken by Manhattan Borough President Gale Brewer, Council Member Brad Lander, Council Member Ben Kallos, Chair of the Government Operations Committee, and Council Member James Vacca, Chair of the Technology Committee. We thank them, their staff, and Citizens Union in supporting the vision of placing the City’s Record online, free of charge, and in machine readable format.
Intro 363-2014, the City Record Online Act, was passed on 24 June 2014, and the Mayor is scheduled to sign it on 7 August 2014.
The drafting of this legislation was undertaken by Manhattan Borough President Gale Brewer, Council Member Brad Lander, Council Member Ben Kallos, Chair of the Government Operations Committee, and Council Member James Vacca, Chair of the Technology Committee. We thank them, their staff, and Citizens Union in supporting the vision of placing the City’s Record online, free of charge, and in machine readable format.