Integration: Google Maps Timeline / Location History (meta: what about backfill-only integrations?)

[steren]

• App: "Google Maps"
• Website: https://www.google.com/maps/timeline

Google Maps allows to store one's location history. Personally, I have been recording my position since 2009. The data is entirely private to the individual.

API
As far as I know, there is no public API to consume the data. However, users can download their entire location history via Google Takeout.

When doing so, one currently gets two feeds:

1. a JSON with all the recorded raw positions
2. a folder with many JSON files containing a semantic analysis of the above positions

This second one is the most interesting, as it includes items like this:

  }, {
    "activitySegment" : {
      "startLocation" : {
        "latitudeE7" : 376170755,
        "longitudeE7" : -1223805551
      },
      "endLocation" : {
        "latitudeE7" : 474489486,
        "longitudeE7" : -1223033108
      },
      "duration" : {
        "startTimestampMs" : "1579623399874",
        "endTimestampMs" : "1579632555789"
      },
      "distance" : 1093383,
      "activityType" : "FLYING",
      "confidence" : "HIGH",
      [...]
      }
    }
  }, {

This means that by parsing these JSON files, North app would be able to get all my past flights, as well as all my past car, cycling, subway train trips... But not the future ones.

Parsing these JSON doesn't seem hard. For a better user experience, there is probably also the need for unzipping what users download from takeout, which can be done with jszip. I might be able to give a hand.

This leads to a meta question: Should the North app support such "backfill" only integrations? or is the idea to only build integrations that constantly fetch data.

My opinion is that such backfills are valuable:

• to be able to compare my past impact with my current impact
• to have an estimate of how much CO2 I previously released per year / past months
• users can do periodic import of the data to keep things roughly up to date.

[corradio]

cc'ing @sorensvejstrup (our product man).
This is very interesting. I believe we could look into a "data import" feature where one could import Google Maps Timeline data.
Maybe this should be put as an idea in product board? https://north-app.com/roadmap

[steren]

By looking at how integrations work, I also realize that because this integration is neither username/password nor Oauth, it would require a new type of integration input: "file selector", where the user would select a local file from the phone to be parsed by the app.

[sorensvejstrup]

Yes, this is extremely interesting. But as you also mention, @steren, there's a bigger conceptual question here regarding backfill and not having real-time data. I guess this could be solved be reminding the users to add data every now and then. The amount of manual work worries me a bit though.
As you're mentioning there's quite a few changes that need to be taken into account;

1. File, selector
2. JSON parser
3. Reminder (push notification) to upload data.
4. Backfill UX concept

As I see it the task is pretty big. So if you can help out in any way, it'd definitely speed up the process. And then we of course also need to consider other tasks at hand.

[steren]

I agree.

Regarding 3. (Reminder to upload data), I don't think that's needed. In my opinion, users should use the Location History backfill only once. Once they have back-filled the past, they should just populate new activities manually, and more importantly flights.

For me, Parsing Location History is a way to do an audit of my CO2 footprint over the past 10 years. Maybe that's not the role of NorthApp, but it's definitely a good way for me to see where I am regarding my CO2 monthly target based on a very large amount of existing data.

The unzip and JSON parser is the easy part, I have a working prototype already. I'll share that soon.

[steren]

Here is a proof of concept of unzipping and parsing the exported Location History data: https://observablehq.com/@steren/your-co2-emissions-from-location-history

Which generates a simple viz of the extracted activities:

[sorensvejstrup]

Wow, so cool, @steren! I'm not finding the "Location History" though?
This is what I got under Maps 👇

[steren]

It's not "Maps", it's "Location History". If you do not see it, it might be because you have not opted into the feature?

[sorensvejstrup]

It's not "Maps", it's "Location History". If you do not see it, it might be because you have not opted into the feature?

Ahh, sure. That's probably it. I did that some years ago but forgot about it when testing. I'll try to find an account I can test with.

[ynegve]

@corradio I'm seeing this has been open for some time and it's the feature that I am missing most in the app. Is there any technical/product reason this has not been integrated? Would be happy to give it a go otherwise.

[corradio]

@corradio I'm seeing this has been open for some time and it's the feature that I am missing most in the app. Is there any technical/product reason this has not been integrated? Would be happy to give it a go otherwise.

Unfortunately we didn't find an easy way to make the data transfer seamless. The reason is the lack of open APIs from Google's side. The manual export is unfortunately quite cumbersome for most users.