Skip to content

Commit

Permalink
Device event listing (#98)
Browse files Browse the repository at this point in the history
* - Updated bootstrap-vue-next lib
- Updated Vue3
- Reworking mask regions component to disallow self-intersecting regions, and allow an apron so regions can be created slightly outside the camera frame.
- Creating wizard steps for manual uploads
- Moving device status out of modal

* - Fix mask regions tests

* - Fix mask regions tests (2, linting)

* - Fix sunset/sunrise in visits breakdown
- Add more info to mobile device cards in DevicesView

* - Fix reference photo component, make it work on mobile.
- More responsive layouts for device setup.

* - Reference photos are properly scaled when overlaying on CptvPlayer on mobile.
- Overflowing tab list component to make longer tabbed nav lists work better on mobile.

* - Trail cameras use an appropriate daytime image as an automatic reference image.
- Masked regions are now used when adding tracks via the API to ignore tracks that are exclusively inside the masked region.
- Use common track rect drawing code for showing tracks for trailcam images.

* - Added a dummy track to trailcam recordings where there is no track – the actual track is created the first time a user classification is added.  This could be extended to empty CPTV files.

* - Styling dummy track

* - Fix tests

* - Added ability to add a dummy track to a CPTV file that has no tracks.
- Automatically add "missed track" tag to CPTV recordings that get manual IDs when they had no tracks.
- Poll every 30 seconds when an open recording is still processing, and update UI when processing is complete.

* - Improved behaviour when mashing on next/prev buttons in CPTV player very quickly :)

* - Fix null settings read in mask-regions API

* - Remove stations wording on Dashboard

* - API Documentation fix
- Add "dummy-track" to tracker version to make it available to processing.

* - Don't send algorithm with tracks, use default

* - Fix mask-regions api read

* - Clone rather than always update in place DeviceHistory entries for adding reference images and mask regions in the same location.
- Fix route for Visits modal from activity view to include selected track.

* - Clone rather than always update in place DeviceHistory entries for adding reference images and mask regions in the same location.
- Fix route for Visits modal from activity view to include selected track. (2)

* - Improved responsive layout for activity search
- Make search url query driven (WIP)

* - Improved responsive layout for activity search
- Make search url query driven (WIP)
- Fix mask-regions api

* - Improved responsive layout for activity search
- Make search url query driven
- Hack in support for "Animal" class from Megadetector
- Show processing progress for visits in API
- Reflect processing progress in the UI.

* - Add UI to visits when recordings are still processing
- Handle scaling DPI when drawing track boxes on trailcam images

* - Make sure player inherits query params from parent component
- Make recording/visits exports work.

* - Don't delete previous reference images

* Scale UP:
- New bulk querying/reporting API which is *many*, *many* times faster for tag filtering queries over longer periods, esp. when the number of recordings in a project grows.
- Thumbnail images on the front-end only get queried when visible.

* Scale UP:
- CI Tests for new bulk tag mode.

* Merge main

* Merge main, add all tagModes to improved bulk query.

* Address review feedback.

* Small fixes

* Small fixes (2)

* Small fixes (3)

* Small fixes (3)

* Small fixes (4)

* Get rid of legacy bulk recordings query in browse-next

* Rework bulk-query, fixing edge-cases and adding more comprehensive tests.
Improve UI, lazy loading of activity page.

* Fix tests

* Fix tests (2)

* Make sure we return recordings in the correct order.

* Auth.

* Make sure we send recording type when querying!

* Auth again

* Fix recordings export report

* Export larger pages to increase export speed, increase
limit on server-side

* Fix issue where we try to request things from inactive devices and get 403 rejected

* Make sure deleted recordings are not included by default in new bulk queries, and add a test for that.
Display "leporadae" as "rabbit or hare"

* Make sure deleted recordings are not included by default in new bulk queries, and add a test for that.
Display "leporadae" as "rabbit or hare"

* add recording window and low power mode settings

* fix lint

* fix for device extract, move settings up

* readd location check

* Misc fixes and improvements, starting on non-paginated visits API.

* Linting

* Fix tests

* Fix tests (2)

* Fix error event logging

* Fix cron script permissions for platform usage report

* Update docker-compose.yml

* Add alert relationship for project-wide alerts

* Link to device recordings search from device page

* Link to device recordings search from device page
- Fix earliest time detection for location recording results

* Add ability for the user to manage their own email alerts via browse-next UI.

* Linting

* Fix tests

* Improve device -> recordings button appearance on mobile.

* Improve service report email, improve alert email

* Service email should only be for the last 24 hours

* Improve formatting of service errors.
Make audio recordings export include cacophony index column.

* Make stopped alerts work for thermal devices we didn't hear from in over 24 hours.
Improve formatting of service error emails (stop gmail stripping things?)
Fix email link for new-style alert emails.

* Fix latest recording times update for groups.

* Fix latest recording times update for groups. (2)

* Update stopped device email text

* Update stopped device email text (2)

* Revamp stopped devices notification email.

* Skip heartbeat tests (no longer rely on heartbeats for stops)

* Make alert animal image link to recording.

* Tweak stopped devices logic

* Tweak stopped devices logic again

* Fleshing out daily/weekly digest emails + user preferences.

* add fox tag

* Daily+weekly digest emails.
Store user timezone info

* Add missing tag icons.

* Add missing tag icons. (2)

* Add fox tag

* Add ability to hide false-trigger tracks in Recording View.

* White outlines around email tag classification icons so that they show up in dark-mode contexts.

* - Improve thumbnail selection in recordings view.
- Fix stopped devices query
- Add aussie tags to classifications.

* - Diagnostics

* fix device settings

* fix time format in settings

* - Adjust stopped device query

* - Add ability for super-users to view any group they're not already part of (as an admin user)

* - Implement deletion of recordings in browse-next and tweak deletion API to update book-keeping for group and stations lastRecordingTimes.

* - Modify locations query to be able to return only locations that don't have zero recordings associated with them, so that deleting the last recording in a location allows that location to be hidden.

* - Do latest time fixups for bulkDelete API too.

* - Fix tooltips for activity search

* - Fix local saving of view false-triggers setting when viewing as a super-user

* - Fix hiding of AI false-trigger tags for tracks where there is a user non-false-trigger tag.

* - Fix requests for tracks that are user-created and don't have thumbnails.

* - Fix user settings persistence

* - Fix issue with removing track tags twice.

* - Cosmetic tweak – images that fail to load.

* - Fix thumbnail selection code.

* - Fix display of sunset/sunrise times in visits breakdown for 24/7 cameras.

* - Fix date offsets for digests

* - Fix date offsets for digests

* - Fix date offsets for digests

* - Fix date offsets for digests (2)

* - Remove beta version wording, add additional filter events

* - Fixes for digest email timings and thumbnail selection.

* - Allow viewing live thumbnails and cptv recordings in local dev mode.
- Add track replay button to tagging view.
- Hide audio mode stuff for now.

* - Fix unitVersion grouping in daily service error emails.

* - Linting

* - Additional options for finding a project to view as a super-user

* fix to make update device settings invariant

* only update if changed

* - Complete reset password flow for browse-next.

* - Complete reset password flow for browse-next.
- Add cypress ui tests for reset password flow.

* - Make project switcher work better
- Properly reset activity search when projects are switched (WIP)

* - Fix linting errors

* - Fix broken activity lists
- Fix weirdness with project switching (when on activity page)

* - Fix CPTV player resizing when browser window is very short vertically, but has a desktop width breakpoint.
- Fix date field not being populated when on activity page and page refreshed.

* - Fix type import

* - Updated device view, device "dashboard" WIP

* - Updated device view, device "dashboard" WIP (2)

* - Fix visitContext filtering on dashboard.

* - Fix recording window length calulation for absolute recordindg windows.
- Fix display of visits activity search description if a device was previously filtered on.

* - In daily/weekly digest emails, try to fall back to another parent class icon if species icon for tag is not found.

* - Improve thumbnail selection for recordings list.

* - Improve pathological cases of recording search where there are many many recordings in a project over a long time span, and the user wants to search over "all time".  To handle this case, start searching over a max time window of 3 months on the API side, and progressively widen that search until there are no results left or the limit has been reached.

* - Handle recordings export where there are (legacy) recordings without a location.

* - Abort in-flight requests if multiple searches are done in quick succession to keep state correct.
- Script to fixup latest recording times
- Fix locations table view
- Fix incorrect thumbnail selection in recordings activity search
- Don't sync activity search context when we enter a recording.
- Don't try to load tag-icons in track tagger view unless the tagging interface is actually expanded.

* - Fix station-related tests to not use status-recording length test recordings
- Add missing recording labels.

* - When we delete the last recording that isn't a status recording, properly fixup last recording times.

* - Handle aborted activity search queries more gracefully in the UI.

* - Handle aborted activity search queries more gracefully in the UI. (2)

* - Fix recording export length truncation when using non-advanced export

* - Fix recording export length truncation when using non-advanced export (2)

* add device sync-settings to browse-next

* - Fixes to dashboard showing non-predator visits and locations
- Fix for unselect locations in dashboard
- Remove unused dashboard view options.
- Add a 'time-sensitive' param to the recordings query, to return from slower queries after a time-out, as long as there are some results gathered.  This will make the UI appear more responsive for sparse queries over longer time periods.
- Add ability to go to activity for locations from locations page for both visits and recordings scoped to the given location.

* - Make max-zoom on maps be not quite as zoomed in (15), since sometimes tiles aren't available for the max zoom level of 16.

* - Tidy locations page some more

* fix custom recording windows

* - Change page size calculation for recordings search.

* - Add a script to fixup device history settings corruption by ratThresh script, and fix ratThresh script so that it doesn't nuke settings.

* - Make ratThresh script only update device settings if there are new human tagged rodents.

* - Make ratThresh script only update device settings if there are new human tagged rodents (2)

* remove loction from device history settings. Remove poweron/off for tc2

* - Adding test coverage for device-settings

* - Make sure location specific device settings get purged when creating a new station for a device, while previously set device recording window and recording settings get retained.

* minor fix on settings api, handle no settings in browse next

* - Fix tagged recording query
- Set lastConnectionTime to null if newer recordings are uploaded via sidekick so that we see a device as being "offline" from that point.
- Fix device settings update in place issue.

* - Sync changes with upstream.

* - Remove ratthreshold debug code

* - Fix UI tests

* - Moving settings sync UI to device setup tab (WIP)
- Adding flag to get latest *synced* settings
- Add tests to confirm/sync settings, and to test getting latest synced settings.

* - Moving settings sync UI to device setup tab
- Adding flag to get latest *synced* settings
- Add latest battery info to devices listing, device diagnostic page, and link to device page from recording view.

* - Moving settings sync UI to device setup tab
- Adding flag to get latest *synced* settings
- Add latest battery info to devices listing, device diagnostic page, and link to device page from recording view.

* - Add spinner while saving window settings

* - Fix lastActiveTime for stations when recordings are uploaded from Devices

* - Better relative time and layout formatting for Locations view.
- Update to Vue 3.5

* - Add device events listing for super-users.
- Add API end-points for getting all known event types, and for all known event types for a given device in the last month.

* - Fix clearing event filter

* - Better format nested event item details.

* - Fix battery level caching bug for aborted api requests

* - Add ability to rename locations that need renaming if the current user is an admin of the project.

* - Add ability to rename locations that need renaming if the current user is an admin of the project (2).

* update to match browse-next, fix getting settings

* - Add known device locations to audio files that are missing location until we get that information embedded in audio metadata

* - Fix UI tests

* - Fix merge

* - Fix CptvPlayer overlay when browser has pixelRatio < 1.0 (zoomed out)

* - Various cosmetic fixes to browse next for mobile viewports.

* Use github actions instead of travis

* Update dockerhub login

* Install docker compose

* - Added new versions of cptv-decoder
- Added m4a metadata extraction to uploads

* - Added new versions of cptv-decoder
- Added m4a metadata extraction to uploads (2)

* - Fix test

* - Remove vestiges of unused vue-multiselect.

* - Bump cptv-player-vue version

* - Bump cptv-player-vue version (2)

* - Only set device active when getting a new recording uploaded via sidekick if it's currently inactive, and there's no active instance of that device elsewhere in the system.

* - Fix log warning.

* - Fix vie build error?

* - Update eslint on browse-next
- Add tests for verifying uploaded audio files.

* - Parallelize github actions jobs
- Fix date range picker
- Hopefully fix Vite build errors

* - Add logging to release build script

* - Add logging to release build script (2)

* - Add IS_CI_ENV to release job.

* - Fix sub-species use of icons in digest emails

* - Clicking a visit should always select+display the first recording in the visit with a track matching the visit classification.

* - Split api integration tests into two batches of roughly equal time to enable us to parallelise the test run into two jobs in CI, decreasing total CI time.

* - Split api integration tests into two batches of roughly equal time to enable us to parallelise the test run into two jobs in CI, decreasing total CI time. (2)

* - Check pwd for tests.

* - Actually add moved spec files!

* - Re-balance test batches

* - Clean up test external type imports
- Remove unused tests
- Use matrix strategy to run test batches

* - Use npm cache

* - Use npm ci instead of npm install to take advantage of github actions cache (once run against `main`)

* - More comprehensive testing of deleted, inactive and reassigned devices.
- Fix behaviour when re-assigning a device, so we don't orphan recordings.
- Fix handling of inactive devices in browse-next.

* - Before re-registering a device in tests, make sure it has at least one recording so that the previous device is set inactive rather than deleted.

* - Fix routing

* - If moving a device from the new group, soft delete all recordings made in new group, and also delete old device.

* - Show when there are no events for device in the last month

* - Only check battery levels on devices we think aren't stopped or offline.

* - Fix device status image retrieval.

* - Sometimes we need to keep asking for frames till we get a good one on cptv single image component.

* - Allow alerts on `unidientified` tags.

* - Resolving review comments

* - Resolving review comments (2)

---------

Co-authored-by: Abra <[email protected]>
Co-authored-by: Patrick Baxter <[email protected]>
Co-authored-by: cam <[email protected]>
  • Loading branch information
4 people authored Oct 22, 2024
1 parent 17cde07 commit 4a6a6e9
Show file tree
Hide file tree
Showing 207 changed files with 7,389 additions and 4,441 deletions.
132 changes: 132 additions & 0 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
name: Node CI and Release

on:
push:
branches:
- '**'
tags:
- '*'
pull_request:
branches:
- '**'

jobs:
lint:
runs-on: ubuntu-20.04
env:
IS_CI_ENV: true
steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: 18
cache: 'npm'
cache-dependency-path: '**/package-lock.json'

- name: Run linting checks
run: |
cd types && npm ci --loglevel=error && npm run lint && cd ..
cd browse && npm ci --loglevel=error && npm run lint && cd ..
cd api && npm ci --loglevel=error && npm run lint && cd ..
cd integration-tests && npm ci --loglevel=error && npm run lint && cd ..
test:
runs-on: ubuntu-20.04
strategy:
matrix:
batch: [ci-batch-1, ci-batch-2]
services:
docker:
image: docker:19.03.12
options: --privileged
ports:
- 2375:2375
env:
DOCKER_TLS_CERTDIR: ""

env:
IS_CI_ENV: true

steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Login to DockerHub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USER }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: Install Docker Compose
run: |
sudo curl -L "https://github.com/docker/compose/releases/download/v2.21.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker-compose --version # Verify installation
- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: 18
cache: 'npm'
cache-dependency-path: '**/package-lock.json'

- name: Set up Docker environment
run: |
docker-compose build
docker-compose up --force-recreate &> dockerstart.log &
node ./api/waitForApi.js || { docker ps; docker-compose logs; exit 1; }
- name: Apply migration undo and redo to test undo migration
run: |
docker-compose exec server /app/api/node_modules/.bin/sequelize db:migrate:undo --config /app/api/config/app_test_default.js --migrations-path /app/api/migrations
docker-compose exec server /app/api/node_modules/.bin/sequelize db:migrate --config /app/api/config/app_test_default.js --migrations-path /app/api/migrations
- name: Run Cypress tests
env:
BATCH: ${{ matrix.batch }}
run: |
cd integration-tests
npm ci --loglevel=error
npm run $BATCH
build:
if: startsWith(github.ref, 'refs/tags/')
runs-on: ubuntu-20.04
env:
IS_CI_ENV: true
steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: 18
cache: 'npm'
cache-dependency-path: '**/package-lock.json'

- name: Install global npm packages
run: npm install -g --loglevel=error npm json [email protected]

- name: Generate API docs
run: |
cd api
npm update --loglevel=error apidoc-plugin-ts
npm run apidoc
- name: Prepare for deployment
run: |
version=${GITHUB_REF#refs/tags/v}
mkdir /tmp/nfpm
curl -sL https://github.com/goreleaser/nfpm/releases/download/v0.9.5/nfpm_0.9.5_Linux_x86_64.tar.gz | tar -C /tmp/nfpm -xzf -
export PATH=/tmp/nfpm:$PATH
_release/build.sh ${version}
- name: Upload release assets
uses: ncipollo/release-action@v1
with:
artifacts: "dist/*.deb"
allowUpdates: true
File renamed without changes.
20 changes: 9 additions & 11 deletions _release/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -32,15 +32,15 @@ cd ${build_dir}
echo "Building Cacophony Browse-Next"
cd browse-next
npm version --no-git-tag-version ${version}
rm -rf node_modules
npm install
echo "Installing dependencies"
npm ci
npm run build
rm -rf node_modules

echo "Installing shared type definitions"
cd ../types
rm -rf node_modules
npm install
echo "Installing dependencies"
npm ci
echo "Compiling TypeScript..."
./node_modules/.bin/tsc
npm run generate-schemas
Expand All @@ -49,20 +49,16 @@ cd ..

echo "Installing dependencies for build..."
cd api

rm -rf node_modules
npm install
npm ci

echo "Compiling TypeScript..."
./node_modules/.bin/tsc

echo "Creating API docs..."
npm update apidoc-plugin-ts
npm run apidoc

echo "Removing external dependencies..."
rm -rf node_modules

echo "Removing TypeScript files..."
find -name '*.ts' -print0 | xargs -0 rm

Expand All @@ -72,9 +68,11 @@ find -name '*.ts' -print0 | xargs -0 rm
echo "Building Cacophony Browse"
cd ../browse
npm version --no-git-tag-version ${version}
rm -rf node_modules
npm install
echo "Installing dependencies"
npm ci
echo "Running build with webpack"
npm run release
echo "Cleaning up node_modules"
rm -rf node_modules

cd ../types
Expand Down
1 change: 1 addition & 0 deletions api/api/V1/Admin.ts
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ import { SuperUsers } from "@/Globals.js";
interface ApiUpdateGlobalPermissionRequestBody {
permission: UserGlobalPermission; // Permission to apply for user
}

export default function (app: Application, baseUrl: string) {
const apiUrl = `${baseUrl}/admin`;
{
Expand Down
Loading

0 comments on commit 4a6a6e9

Please sign in to comment.