Skip to content

Commit

Permalink
Merge pull request #19 from MarkEdmondson1234/master
Browse files Browse the repository at this point in the history 
CRAN 0.2.0 checks
  • Loading branch information
@MarkEdmondson1234
MarkEdmondson1234 authored Aug 11, 2016
2 parents 949262c + 6c5cbef commit 4da26f3
Show file tree
Hide file tree
Showing 6 changed files with 211 additions and 16 deletions.
3 changes: 2 additions & 1 deletion .Rbuildignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,5 @@
^\.httr-oauth$
^\.travis\.yml$
^CONTRIBUTING\.md$
^cran-comments\.md$
^cran-comments\.md$
^revdep
8 changes: 4 additions & 4 deletions DESCRIPTION
View file
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
Package: bigQueryR
Title: Interface with Google BigQuery with Shiny Compatibility
Version: 0.1.0.9000
Version: 0.2.0
Authors@R: c(person("Mark", "Edmondson",email = "[email protected]",
role = c("aut", "cre")),
person("Hadley", "Wickham", , "[email protected]", role = "ctb")
)
Description: Interface with Google BigQuery,
Description: Interface with 'Google BigQuery',
see <https://cloud.google.com/bigquery/> for more information.
This package uses 'googleAuthR' so is compatible with similar packages,
including Google Cloud Storage (<https://cloud.google.com/storage/>) for result extracts.
including 'Google Cloud Storage' (<https://cloud.google.com/storage/>) for result extracts.
URL: http://code.markedmondson.me/bigQueryR/
BugReports: https://github.com/cloudyr/bigQueryR/issues
License: MIT + file LICENSE
Expand All @@ -17,7 +17,7 @@ Depends:
R (>= 3.2.0)
Imports:
googleAuthR (>= 0.3.1),
googleCloudStorageR (>= 0.1.0)
googleCloudStorageR
Suggests:
shiny (>= 0.12.1),
jsonlite (>= 1.0),
Expand Down
8 changes: 2 additions & 6 deletions NEWS.md
View file
Original file line number Diff line number Diff line change
@@ -1,12 +1,8 @@
# bigQueryR 0.1.0.9000
# bigQueryR 0.2.0 (CRAN)

* Download asynch queries straight to disk via googleCloudStorageR


# bigQueryR 0.1.0 (CRAN)
# bigQueryR 0.1.0

* Added a `NEWS.md` file to track changes to the package.
* Initial release



12 changes: 7 additions & 5 deletions cran-comments.md
View file
Original file line number Diff line number Diff line change
@@ -1,17 +1,19 @@
## Test environments
* local OS X install, R 3.3.0
* ubuntu 12.04 (on travis-ci), R 3.3.0
* ubuntu 12.04 (on travis-ci), R 3.3.1
* win-builder (devel and release)

## R CMD check results

0 errors | 0 warnings | 2 notes
0 errors | 0 warnings | 1 notes

* This is a new release.
* Possibly mis-spelled words in DESCRIPTION: BigQuery - this is spelt correctly.
* Possibly mis-spelled words in DESCRIPTION:
BigQuery (2:30)

This is spelt correctly

## Reverse dependencies

This is a new release, so there are no reverse dependencies.
googleAnalyticsR is a dependency, that when checked had 0 errors.

---
Binary file added revdep/checks.rds
View file
Binary file not shown.
196 changes: 196 additions & 0 deletions vignettes/bigQueryR.html
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,196 @@
<!DOCTYPE html>

<html xmlns="http://www.w3.org/1999/xhtml">

<head>

<meta charset="utf-8">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="pandoc" />

<meta name="viewport" content="width=device-width, initial-scale=1">

<meta name="author" content="Mark Edmondson" />

<meta name="date" content="2016-08-11" />

<title>bigQueryR</title>



<style type="text/css">code{white-space: pre;}</style>
<style type="text/css">
div.sourceCode { overflow-x: auto; }
table.sourceCode, tr.sourceCode, td.lineNumbers, td.sourceCode {
margin: 0; padding: 0; vertical-align: baseline; border: none; }
table.sourceCode { width: 100%; line-height: 100%; }
td.lineNumbers { text-align: right; padding-right: 4px; padding-left: 4px; color: #aaaaaa; border-right: 1px solid #aaaaaa; }
td.sourceCode { padding-left: 5px; }
code > span.kw { color: #007020; font-weight: bold; } /* Keyword */
code > span.dt { color: #902000; } /* DataType */
code > span.dv { color: #40a070; } /* DecVal */
code > span.bn { color: #40a070; } /* BaseN */
code > span.fl { color: #40a070; } /* Float */
code > span.ch { color: #4070a0; } /* Char */
code > span.st { color: #4070a0; } /* String */
code > span.co { color: #60a0b0; font-style: italic; } /* Comment */
code > span.ot { color: #007020; } /* Other */
code > span.al { color: #ff0000; font-weight: bold; } /* Alert */
code > span.fu { color: #06287e; } /* Function */
code > span.er { color: #ff0000; font-weight: bold; } /* Error */
code > span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */
code > span.cn { color: #880000; } /* Constant */
code > span.sc { color: #4070a0; } /* SpecialChar */
code > span.vs { color: #4070a0; } /* VerbatimString */
code > span.ss { color: #bb6688; } /* SpecialString */
code > span.im { } /* Import */
code > span.va { color: #19177c; } /* Variable */
code > span.cf { color: #007020; font-weight: bold; } /* ControlFlow */
code > span.op { color: #666666; } /* Operator */
code > span.bu { } /* BuiltIn */
code > span.ex { } /* Extension */
code > span.pp { color: #bc7a00; } /* Preprocessor */
code > span.at { color: #7d9029; } /* Attribute */
code > span.do { color: #ba2121; font-style: italic; } /* Documentation */
code > span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* Annotation */
code > span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* CommentVar */
code > span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* Information */
</style>



<link href="data:text/css;charset=utf-8,body%20%7B%0Abackground%2Dcolor%3A%20%23fff%3B%0Amargin%3A%201em%20auto%3B%0Amax%2Dwidth%3A%20700px%3B%0Aoverflow%3A%20visible%3B%0Apadding%2Dleft%3A%202em%3B%0Apadding%2Dright%3A%202em%3B%0Afont%2Dfamily%3A%20%22Open%20Sans%22%2C%20%22Helvetica%20Neue%22%2C%20Helvetica%2C%20Arial%2C%20sans%2Dserif%3B%0Afont%2Dsize%3A%2014px%3B%0Aline%2Dheight%3A%201%2E35%3B%0A%7D%0A%23header%20%7B%0Atext%2Dalign%3A%20center%3B%0A%7D%0A%23TOC%20%7B%0Aclear%3A%20both%3B%0Amargin%3A%200%200%2010px%2010px%3B%0Apadding%3A%204px%3B%0Awidth%3A%20400px%3B%0Aborder%3A%201px%20solid%20%23CCCCCC%3B%0Aborder%2Dradius%3A%205px%3B%0Abackground%2Dcolor%3A%20%23f6f6f6%3B%0Afont%2Dsize%3A%2013px%3B%0Aline%2Dheight%3A%201%2E3%3B%0A%7D%0A%23TOC%20%2Etoctitle%20%7B%0Afont%2Dweight%3A%20bold%3B%0Afont%2Dsize%3A%2015px%3B%0Amargin%2Dleft%3A%205px%3B%0A%7D%0A%23TOC%20ul%20%7B%0Apadding%2Dleft%3A%2040px%3B%0Amargin%2Dleft%3A%20%2D1%2E5em%3B%0Amargin%2Dtop%3A%205px%3B%0Amargin%2Dbottom%3A%205px%3B%0A%7D%0A%23TOC%20ul%20ul%20%7B%0Amargin%2Dleft%3A%20%2D2em%3B%0A%7D%0A%23TOC%20li%20%7B%0Aline%2Dheight%3A%2016px%3B%0A%7D%0Atable%20%7B%0Amargin%3A%201em%20auto%3B%0Aborder%2Dwidth%3A%201px%3B%0Aborder%2Dcolor%3A%20%23DDDDDD%3B%0Aborder%2Dstyle%3A%20outset%3B%0Aborder%2Dcollapse%3A%20collapse%3B%0A%7D%0Atable%20th%20%7B%0Aborder%2Dwidth%3A%202px%3B%0Apadding%3A%205px%3B%0Aborder%2Dstyle%3A%20inset%3B%0A%7D%0Atable%20td%20%7B%0Aborder%2Dwidth%3A%201px%3B%0Aborder%2Dstyle%3A%20inset%3B%0Aline%2Dheight%3A%2018px%3B%0Apadding%3A%205px%205px%3B%0A%7D%0Atable%2C%20table%20th%2C%20table%20td%20%7B%0Aborder%2Dleft%2Dstyle%3A%20none%3B%0Aborder%2Dright%2Dstyle%3A%20none%3B%0A%7D%0Atable%20thead%2C%20table%20tr%2Eeven%20%7B%0Abackground%2Dcolor%3A%20%23f7f7f7%3B%0A%7D%0Ap%20%7B%0Amargin%3A%200%2E5em%200%3B%0A%7D%0Ablockquote%20%7B%0Abackground%2Dcolor%3A%20%23f6f6f6%3B%0Apadding%3A%200%2E25em%200%2E75em%3B%0A%7D%0Ahr%20%7B%0Aborder%2Dstyle%3A%20solid%3B%0Aborder%3A%20none%3B%0Aborder%2Dtop%3A%201px%20solid%20%23777%3B%0Amargin%3A%2028px%200%3B%0A%7D%0Adl%20%7B%0Amargin%2Dleft%3A%200%3B%0A%7D%0Adl%20dd%20%7B%0Amargin%2Dbottom%3A%2013px%3B%0Amargin%2Dleft%3A%2013px%3B%0A%7D%0Adl%20dt%20%7B%0Afont%2Dweight%3A%20bold%3B%0A%7D%0Aul%20%7B%0Amargin%2Dtop%3A%200%3B%0A%7D%0Aul%20li%20%7B%0Alist%2Dstyle%3A%20circle%20outside%3B%0A%7D%0Aul%20ul%20%7B%0Amargin%2Dbottom%3A%200%3B%0A%7D%0Apre%2C%20code%20%7B%0Abackground%2Dcolor%3A%20%23f7f7f7%3B%0Aborder%2Dradius%3A%203px%3B%0Acolor%3A%20%23333%3B%0Awhite%2Dspace%3A%20pre%2Dwrap%3B%20%0A%7D%0Apre%20%7B%0Aborder%2Dradius%3A%203px%3B%0Amargin%3A%205px%200px%2010px%200px%3B%0Apadding%3A%2010px%3B%0A%7D%0Apre%3Anot%28%5Bclass%5D%29%20%7B%0Abackground%2Dcolor%3A%20%23f7f7f7%3B%0A%7D%0Acode%20%7B%0Afont%2Dfamily%3A%20Consolas%2C%20Monaco%2C%20%27Courier%20New%27%2C%20monospace%3B%0Afont%2Dsize%3A%2085%25%3B%0A%7D%0Ap%20%3E%20code%2C%20li%20%3E%20code%20%7B%0Apadding%3A%202px%200px%3B%0A%7D%0Adiv%2Efigure%20%7B%0Atext%2Dalign%3A%20center%3B%0A%7D%0Aimg%20%7B%0Abackground%2Dcolor%3A%20%23FFFFFF%3B%0Apadding%3A%202px%3B%0Aborder%3A%201px%20solid%20%23DDDDDD%3B%0Aborder%2Dradius%3A%203px%3B%0Aborder%3A%201px%20solid%20%23CCCCCC%3B%0Amargin%3A%200%205px%3B%0A%7D%0Ah1%20%7B%0Amargin%2Dtop%3A%200%3B%0Afont%2Dsize%3A%2035px%3B%0Aline%2Dheight%3A%2040px%3B%0A%7D%0Ah2%20%7B%0Aborder%2Dbottom%3A%204px%20solid%20%23f7f7f7%3B%0Apadding%2Dtop%3A%2010px%3B%0Apadding%2Dbottom%3A%202px%3B%0Afont%2Dsize%3A%20145%25%3B%0A%7D%0Ah3%20%7B%0Aborder%2Dbottom%3A%202px%20solid%20%23f7f7f7%3B%0Apadding%2Dtop%3A%2010px%3B%0Afont%2Dsize%3A%20120%25%3B%0A%7D%0Ah4%20%7B%0Aborder%2Dbottom%3A%201px%20solid%20%23f7f7f7%3B%0Amargin%2Dleft%3A%208px%3B%0Afont%2Dsize%3A%20105%25%3B%0A%7D%0Ah5%2C%20h6%20%7B%0Aborder%2Dbottom%3A%201px%20solid%20%23ccc%3B%0Afont%2Dsize%3A%20105%25%3B%0A%7D%0Aa%20%7B%0Acolor%3A%20%230033dd%3B%0Atext%2Ddecoration%3A%20none%3B%0A%7D%0Aa%3Ahover%20%7B%0Acolor%3A%20%236666ff%3B%20%7D%0Aa%3Avisited%20%7B%0Acolor%3A%20%23800080%3B%20%7D%0Aa%3Avisited%3Ahover%20%7B%0Acolor%3A%20%23BB00BB%3B%20%7D%0Aa%5Bhref%5E%3D%22http%3A%22%5D%20%7B%0Atext%2Ddecoration%3A%20underline%3B%20%7D%0Aa%5Bhref%5E%3D%22https%3A%22%5D%20%7B%0Atext%2Ddecoration%3A%20underline%3B%20%7D%0A%0Acode%20%3E%20span%2Ekw%20%7B%20color%3A%20%23555%3B%20font%2Dweight%3A%20bold%3B%20%7D%20%0Acode%20%3E%20span%2Edt%20%7B%20color%3A%20%23902000%3B%20%7D%20%0Acode%20%3E%20span%2Edv%20%7B%20color%3A%20%2340a070%3B%20%7D%20%0Acode%20%3E%20span%2Ebn%20%7B%20color%3A%20%23d14%3B%20%7D%20%0Acode%20%3E%20span%2Efl%20%7B%20color%3A%20%23d14%3B%20%7D%20%0Acode%20%3E%20span%2Ech%20%7B%20color%3A%20%23d14%3B%20%7D%20%0Acode%20%3E%20span%2Est%20%7B%20color%3A%20%23d14%3B%20%7D%20%0Acode%20%3E%20span%2Eco%20%7B%20color%3A%20%23888888%3B%20font%2Dstyle%3A%20italic%3B%20%7D%20%0Acode%20%3E%20span%2Eot%20%7B%20color%3A%20%23007020%3B%20%7D%20%0Acode%20%3E%20span%2Eal%20%7B%20color%3A%20%23ff0000%3B%20font%2Dweight%3A%20bold%3B%20%7D%20%0Acode%20%3E%20span%2Efu%20%7B%20color%3A%20%23900%3B%20font%2Dweight%3A%20bold%3B%20%7D%20%20code%20%3E%20span%2Eer%20%7B%20color%3A%20%23a61717%3B%20background%2Dcolor%3A%20%23e3d2d2%3B%20%7D%20%0A" rel="stylesheet" type="text/css" />

</head>

<body>




<h1 class="title toc-ignore">bigQueryR</h1>
<h4 class="author"><em>Mark Edmondson</em></h4>
<h4 class="date"><em>2016-08-11</em></h4>



<div id="introduction" class="section level2">
<h2>Introduction</h2>
<p>This is a package for interating with <a href="https://cloud.google.com/bigquery/">BigQuery</a> from within R.</p>
<p>You may want instead to use <a href="https://github.com/hadley/bigrquery">bigrquery</a> which is more developed with integration with <code>dplyr</code> etc. Some functions from <code>bigrquery</code> are used in this package.</p>
<div id="why-this-package-then" class="section level3">
<h3>Why this package then?</h3>
<p>This package is here as it uses <a href="https://github.com/MarkEdmondson1234/googleAuthR">googleAuthR</a> as backend, so has Shiny support, and compatibility with other googleAuthR dependent packages.</p>
<p>It also has support for data extracts to Google Cloud Storage, meaning you can download data and make the download URL available to a user via their Google email. If you do a query normally with over 100000 results it hangs and errors.</p>
<p>An example of a BigQuery Shiny app running OAuth2 is here, the <a href="https://mark.shinyapps.io/bigquery-viz/">BigQuery Visualiser</a></p>
</div>
</div>
<div id="authentication" class="section level2">
<h2>Authentication</h2>
<p>Authentication is as used in other <code>googleAuthR</code> libraries:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">library</span>(bigQueryR)

## this will open your browser
## Authenticate with an email that has access to the BigQuery project you need
<span class="kw">bqr_auth</span>()

## verify under a new user
<span class="kw">bqr_auth</span>(<span class="dt">new_user=</span><span class="ot">TRUE</span>)</code></pre></div>
<p>If you are authenticating under several APIs via <code>googleAuthR</code>then use <code>gar_auth()</code> instead with the appropriate scopes set.</p>
<p>You can also use service-to-service JSON files and multi-user authentication under Shiny, see the <code>googleAuthR</code> readme for details.</p>
</div>
<div id="listing-bigquery-meta-data" class="section level2">
<h2>Listing BigQuery meta data</h2>
<p>Various functions for listing what is in your BigQuery account.</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">library</span>(bigQueryR)

## this will open your browser
## Authenticate with an email that has access to the BigQuery project you need
<span class="kw">bqr_auth</span>()

## verify under a new user
<span class="kw">bqr_auth</span>(<span class="dt">new_user=</span><span class="ot">TRUE</span>)

## get projects
projects &lt;-<span class="st"> </span><span class="kw">bqr_list_projects</span>()

my_project &lt;-<span class="st"> </span>projects[<span class="dv">1</span>]

## for first project, get datasets
datasets &lt;-<span class="st"> </span>bqr_list_datasets[my_project]

my_dataset &lt;-<span class="st"> </span>datasets[<span class="dv">1</span>]
## list tables
my_table &lt;-<span class="st"> </span><span class="kw">bqr_list_tables</span>(my_project, my_dataset)

## get metadata for table
meta_table &lt;-<span class="st"> </span><span class="kw">bqr_table_meta</span>(my_project, my_dataset, my_table)</code></pre></div>
</div>
<div id="simple-queries" class="section level2">
<h2>Simple Queries</h2>
<p>You can pass in queries that have results under ~ 100000 rows using this command:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">bqr_query</span>(<span class="st">&quot;big-query-r&quot;</span>,<span class="st">&quot;samples&quot;</span>,
<span class="st">&quot;SELECT COUNT(repository.url) FROM [publicdata:samples.github_nested]&quot;</span>)</code></pre></div>
<p>More than that, and the API starts to hang and you are limited by your download bandwidth.</p>
</div>
<div id="asynchronous-queries" class="section level2">
<h2>Asynchronous Queries</h2>
<p>For bigger queries, asynchronous queries save the results to another BigQuery table. You can check the progress of the job via <code>bqr_get_job</code></p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">library</span>(bigQueryR)

## Auth with a project that has at least BigQuery and Google Cloud Storage scope
<span class="kw">bqr_auth</span>()

## make a big query
job &lt;-<span class="st"> </span><span class="kw">bqr_query_asynch</span>(<span class="st">&quot;your_project&quot;</span>,
<span class="st">&quot;your_dataset&quot;</span>,
<span class="st">&quot;SELECT * FROM blah LIMIT 9999999&quot;</span>,
<span class="dt">destinationTableId =</span> <span class="st">&quot;bigResultTable&quot;</span>)

## poll the job to check its status
## its done when job$status$state == &quot;DONE&quot;
<span class="kw">bqr_get_job</span>(<span class="st">&quot;your_project&quot;</span>, job$jobReference$jobId)

##once done, the query results are in &quot;bigResultTable&quot;</code></pre></div>
<p>You may now want to download this data. For large datasets, this is best done via extracting the BigQuery result to Google Cloud Storage, then downloading the data from there.</p>
<p>You can create a bucket at Google Cloud Storage see <a href="https://cloud.google.com/storage/docs/cloud-console" class="uri">https://cloud.google.com/storage/docs/cloud-console</a>, or you can use <a href="https://github.com/cloudyr/googleCloudStorageR">library(googleCloudStorageR)</a></p>
<p>Once created, you can extract your data via the below:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">## Create the data extract from BigQuery to Cloud Storage
job_extract &lt;-<span class="st"> </span><span class="kw">bqr_extract_data</span>(<span class="st">&quot;your_project&quot;</span>,
<span class="st">&quot;your_dataset&quot;</span>,
<span class="st">&quot;bigResultTable&quot;</span>,
<span class="st">&quot;your_cloud_storage_bucket_name&quot;</span>)

## poll the extract job to check its status
## its done when job$status$state == &quot;DONE&quot;
<span class="kw">bqr_get_job</span>(<span class="st">&quot;your_project&quot;</span>, job_extract$jobReference$jobId)

## to download via a URL and not logging in via Google Cloud Storage interface:
## Use an email that is Google account enabled
## Requires scopes:
## https://www.googleapis.com/auth/devstorage.full_control
## https://www.googleapis.com/auth/cloud-platform
## set via options(&quot;bigQueryR.scopes&quot;) and reauthenticate if needed

download_url &lt;-<span class="st"> </span><span class="kw">bqr_grant_extract_access</span>(job_extract, <span class="st">&quot;[email protected]&quot;</span>)

## download_url may be multiple if the data is &gt; 1GB
&gt;<span class="st"> </span>[<span class="dv">1</span>] <span class="st">&quot;https://storage.cloud.google.com/big-query-r-extracts/extract-20160311112410-000000000000.csv&quot;</span>
&gt;<span class="st"> </span>[<span class="dv">2</span>] <span class="st">&quot;https://storage.cloud.google.com/big-query-r-extracts/extract-20160311112410-000000000001.csv&quot;</span>
&gt;<span class="st"> </span>[<span class="dv">3</span>] <span class="st">&quot;https://storage.cloud.google.com/big-query-r-extracts/extract-20160311112410-000000000002.csv&quot;</span></code></pre></div>
</div>



<!-- dynamically load mathjax for compatibility with self-contained -->
<script>
(function () {
var script = document.createElement("script");
script.type = "text/javascript";
script.src = "https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML";
document.getElementsByTagName("head")[0].appendChild(script);
})();
</script>

</body>
</html>

0 comments on commit 4da26f3

Please sign in to comment.