21-getting-started-with-bigquery.md

21. Getting started with BigQuery

We will stream our App Engine logs into BigQuery and query the data.

We will use the App Engine project created in #14 to store queries.

Setup Cloud Logging

1. Confirm that you have a running instance of your app at App Engine
2. Visit Cloud Logging and select your currently running application
3. Click 'Create Export' to create a dataset to write to
   1. Add a meaningful name
   2. Select 'BigQuery' for sink service
   3. Create a sink destination (or select one if already created)
   4. Create sink
4. Visit Exports from the sidebar

Making queries

1. Visit BigQuery where the dataset should be visible (you may need to select the correct project)
2. Create a few requests in your application
3. Back in BigQuery, click on the table generated by Google below the export name in the sidebar
4. In the table details click a few fields (only leaf nodes can be selected for the query) and make the query

Make a custom query

1. Get the top 10 requests ordered by latency

   SELECT protoPayload.resource, protoPayload.latency
   FROM <your-table-id>
   ORDER BY protoPayload.latency DESC
   LIMIT 10

2. Get all status codes and the number of times those status codes were returned in a request:

   SELECT
     protoPayload.status,
     COUNT(protoPayload.status) AS COUNT
   FROM
     [<table-di>]
   ORDER BY
     protoPayload.latency DESC
   GROUP BY
     protoPayload.status

Cleanup

1. Under Log Exports delete the export
2. In BigQuery delete your dataset

Make sure to also delete the compute instance and storage created in the earlier demos.