From c5bc1ff69f1087ef7d9185b0117ac5a1329bdc04 Mon Sep 17 00:00:00 2001 From: Phil Marshall Date: Wed, 11 Jul 2018 09:45:14 -0700 Subject: [PATCH 1/9] DIA project set-up --- DIA/README.md | 7 +++++++ README.md | 23 ++++++++++++----------- 2 files changed, 19 insertions(+), 11 deletions(-) create mode 100644 DIA/README.md diff --git a/DIA/README.md b/DIA/README.md new file mode 100644 index 00000000..84eec337 --- /dev/null +++ b/DIA/README.md @@ -0,0 +1,7 @@ +# Difference Image Analysis + +This folder contains a set of tutorial notebooks exploring the Difference Image Analysis (DIA) parts of the LSST science pipelines. To join the discussion of this project, please see [issue #48](https://github.com/LSSTScienceCollaborations/DMStackClub/issues/48) + +| Notebook | Description | Owner | +|---|---|---| +| [DIA_How_To_Generate_a_Template_Image]() | Not yet started. | Phil Marshall (@drphilmarshall) | diff --git a/README.md b/README.md index b46901d5..8e4ed1ac 100644 --- a/README.md +++ b/README.md @@ -1,21 +1,22 @@ -This is an LSSTC-supported project, to form a small community committed to learning how to use, and explain, the LSST DM Stack. +This is an LSSTC-supported project, to form a small community committed to learning how to use, and explain, the LSST DM Stack. We're following a 3-phase plan, which you can read about in more detail [here](https://docs.google.com/document/d/103kzjOklSUWo5MJP9B-EsnAdO7V6bstTC_mzBvd0NIk/edit#). Phase 0 is collecting existing tutorials and identifying potential club members from around the LSST Science Collaborations. Then, in Phase 1 (late May 2018 to mid August 2018) we'll work together to turn a subset of those existing "seed" tutorials into community-maintained jupyter notebooks, for display at the August LSST 2018 Project and Community Workshop. Then, we plan to open up to a larger group from the science collaborations starting at the PCW, extending and spinning off the initial set of notebooks. The idea is that the best way to learn something is to try and teach it: if you can write a useful tutorial on part of the DM Stack, you have to understand that part first. -If you have questions, please [write us an issue](https://github.com/LSSTScienceCollaborations/DMStackClub/issues/new)! +If you have questions, please [write us an issue](https://github.com/LSSTScienceCollaborations/DMStackClub/issues/new)! ## Community Tutorials -In progress! Our goal is to build on the existing LSST DM Stack demo and tutorial notebooks to create a set of -_community-generated, community-oriented_ notebooks that reflect the science interests and expected analyses of -the LSST Science Collaborations. -* [List of Topics, with links to tutorial seeds](https://docs.google.com/document/d/1PSA1uWwTfs9CweatpxF8CEPGBYRY5ZaXB39JzXYE7_U/edit#). Includes comprehensive collection of existing project and community tutorial web pages and demo notebooks, from which seeds are drawn. +In progress! Our goal is to build on the existing LSST DM Stack demo and tutorial notebooks to create a set of +_community-generated, community-oriented_ notebooks that reflect the science interests and expected analyses of +the LSST Science Collaborations. +* [List of Topics, with links to tutorial seeds](https://docs.google.com/document/d/1PSA1uWwTfs9CweatpxF8CEPGBYRY5ZaXB39JzXYE7_U/edit#). Includes comprehensive collection of existing project and community tutorial web pages and demo notebooks, from which seeds are drawn. | Topic | Description | Notebook Location | |---|---|---| | Getting Started | Basics of using JupyterLab, and contributing to the DM Stack Club repo. | [DMStackClub/GettingStarted](GettingStarted) | | Visualization | Displaying images and catalogs. | [DMStackClub/Visualization](Visualization) | | Image Processing | From raw images to `calexp`s and `coadd`s. | [DMStackClub/ImageProcessing](ImageProcessing) | +| Difference Image Analysis | Walkthroughs of the DIA parts of the science pipeline | [DMStackClub/DIA](DIA) | ## Contributing New Stack Club members: please see the [notes on getting started](GettingStarted/GettingStarted.md) - they'll walk you onto you new LSST Science Platform account, and show you how to work on your tutorial notebooks. @@ -34,10 +35,10 @@ The Club meets periodically via Zoom, but you can find us on LSSTC Slack at [#dm ## License -The text in this repository is Copyright The Authors, and licensed under Creative Commons [CC-BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0/), which means +The text in this repository is Copyright The Authors, and licensed under Creative Commons [CC-BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0/), which means you can copy and redistribute the material in any medium or format -for any purpose, as long as you give appropriate credit, provide a link to the license, and indicate if changes were made. -If you remix, transform, or build upon the material, you may not distribute the modified material - this is to prevent incorrect +for any purpose, as long as you give appropriate credit, provide a link to the license, and indicate if changes were made. +If you remix, transform, or build upon the material, you may not distribute the modified material - this is to prevent incorrect information about the Stack getting out there, or at least, take responsibility ourselves if it does. -All the code in this repository is available for re-use under the [MIT License](https://github.com/LSSTScienceCollaborations/DMStackClub/blob/master/LICENSE), which means you can do anything you like with it -but you can't blame us if it doesn't do what you want. +All the code in this repository is available for re-use under the [MIT License](https://github.com/LSSTScienceCollaborations/DMStackClub/blob/master/LICENSE), which means you can do anything you like with it +but you can't blame us if it doesn't do what you want. From fca437b3a4d49a4825fc25183534557d78330e32 Mon Sep 17 00:00:00 2001 From: Phil Marshall Date: Fri, 13 Jul 2018 17:41:11 +0000 Subject: [PATCH 2/9] Started template generation how-to --- ...DIA_How_To_Generate_a_Template_Image.ipynb | 87 +++++++++++++++++++ DIA/README.md | 2 +- 2 files changed, 88 insertions(+), 1 deletion(-) create mode 100644 DIA/DIA_How_To_Generate_a_Template_Image.ipynb diff --git a/DIA/DIA_How_To_Generate_a_Template_Image.ipynb b/DIA/DIA_How_To_Generate_a_Template_Image.ipynb new file mode 100644 index 00000000..4eb11deb --- /dev/null +++ b/DIA/DIA_How_To_Generate_a_Template_Image.ipynb @@ -0,0 +1,87 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# DIA: How To Generate a Template Image" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
**Owner:** Phil Marshall (@drphilmarshall)\n", + "
**Last Run:** 2018-07-13\n", + "
**Stack Version:** v16\n", + "
**Discussion:** [issue #48](https://github.com/LSSTScienceCollaborations/DMStackClub/issues/48)\n", + "\n", + "## Goals\n", + "In this tutorial we will understand the possible algorithm choices for DIA template image generation, see how these are currently implemented in the science pipelines, and demonstrate them on the Twinkles images." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Requirements" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Background\n", + "\n", + "The DRP pipelines are summarized in Figure 9 of [LDM-151](http://ls.st/ldm-151), reproduced here:\n", + "\n", + " \n", + "\n", + "Difference imaging is shown in block 4, and includes creation of `TemplateCoadd` images, image differencing, and `DIASource` creation. The relevant pipeline design text on template generation, for the data release processing, is in Section 5.2:\n", + "\n", + "> \"4. We run the WarpTemplates, CoaddTemplates, and DiffIm pipelines to generate the DI- ASource and DiffExp datasets. We may then be able to generate better CalExp Masks than we can obtain from BackgroundMatchAndReject by comparing the DiffExp masks across visits in the UpdateMasks pipeline.\n", + "> 5. After all CalExp components have been finalized, we run the WarpRemaining and CoaddRemaining to build additional coadd data products.\"\n", + "\n", + "These steps assume that the final visit CalExp images have already been made. The main coadd data product that is needed for DIA is the TemplateCoadd:\n", + "\n", + "> \"**TemplateCoadd**: A coadd data product used for difference imaging in both DRP and AP. In order to produce templates appropriate for the level of DCR in a given science image, these coadds may require a third dimension in addition to the usual two image dimen- sions (likely either wavelength or a quantity that is a function of airmass).\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Exactly how this TemplateCoadd is made is still under development: LDM-151 lists several possible scenarios, in which different algorithms for the TemplateCoadd generation are employed. The four choices for TemplateCoadd generation seem to be as follows:\n", + "\n", + "* **Standard decorrelated coadds**\n", + "\n", + "* **Constant-PSF partially-decorrelated coadds**\n", + "\n", + "* **PSF-matched coadds**\n", + "\n", + "* **\"The result of inference on resampled exposures with no PSF-matching\"**\n" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "LSST", + "language": "python", + "name": "lsst" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.2" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/DIA/README.md b/DIA/README.md index 84eec337..42392380 100644 --- a/DIA/README.md +++ b/DIA/README.md @@ -4,4 +4,4 @@ This folder contains a set of tutorial notebooks exploring the Difference Image | Notebook | Description | Owner | |---|---|---| -| [DIA_How_To_Generate_a_Template_Image]() | Not yet started. | Phil Marshall (@drphilmarshall) | +| [DIA_How_To_Generate_a_Template_Image](DIA_How_To_Generate_a_Template_Image.ipynb) | Options for template coadd generation, demonstration on Twinkles images. | Phil Marshall (@drphilmarshall) | From 713e188ae0b5c65c5f5b80da410ebe8cdbba1ef9 Mon Sep 17 00:00:00 2001 From: Phil Marshall Date: Fri, 13 Jul 2018 17:46:11 +0000 Subject: [PATCH 3/9] Trying to show image... --- ...DIA_How_To_Generate_a_Template_Image.ipynb | 21 ++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/DIA/DIA_How_To_Generate_a_Template_Image.ipynb b/DIA/DIA_How_To_Generate_a_Template_Image.ipynb index 4eb11deb..eb6fa114 100644 --- a/DIA/DIA_How_To_Generate_a_Template_Image.ipynb +++ b/DIA/DIA_How_To_Generate_a_Template_Image.ipynb @@ -33,14 +33,25 @@ "source": [ "## Background\n", "\n", - "The DRP pipelines are summarized in Figure 9 of [LDM-151](http://ls.st/ldm-151), reproduced here:\n", - "\n", - " \n", - "\n", + "The DRP pipelines are summarized in Figure 9 of [LDM-151](http://ls.st/ldm-151), reproduced here:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "![](https://github.com/lsst/LDM-151/blob/master/figures/drp_summary.png)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ "Difference imaging is shown in block 4, and includes creation of `TemplateCoadd` images, image differencing, and `DIASource` creation. The relevant pipeline design text on template generation, for the data release processing, is in Section 5.2:\n", "\n", "> \"4. We run the WarpTemplates, CoaddTemplates, and DiffIm pipelines to generate the DI- ASource and DiffExp datasets. We may then be able to generate better CalExp Masks than we can obtain from BackgroundMatchAndReject by comparing the DiffExp masks across visits in the UpdateMasks pipeline.\n", - "> 5. After all CalExp components have been finalized, we run the WarpRemaining and CoaddRemaining to build additional coadd data products.\"\n", + ">\n", + "> \"5. After all CalExp components have been finalized, we run the WarpRemaining and CoaddRemaining to build additional coadd data products.\"\n", "\n", "These steps assume that the final visit CalExp images have already been made. The main coadd data product that is needed for DIA is the TemplateCoadd:\n", "\n", From 72d3f1d71d045eeda912be2f2def229a8d05bcf0 Mon Sep 17 00:00:00 2001 From: Phil Marshall Date: Fri, 13 Jul 2018 18:12:07 +0000 Subject: [PATCH 4/9] Fixed figure, guessed section headings --- ...DIA_How_To_Generate_a_Template_Image.ipynb | 41 ++++++++++++++----- 1 file changed, 30 insertions(+), 11 deletions(-) diff --git a/DIA/DIA_How_To_Generate_a_Template_Image.ipynb b/DIA/DIA_How_To_Generate_a_Template_Image.ipynb index eb6fa114..21a54a21 100644 --- a/DIA/DIA_How_To_Generate_a_Template_Image.ipynb +++ b/DIA/DIA_How_To_Generate_a_Template_Image.ipynb @@ -17,14 +17,12 @@ "
**Discussion:** [issue #48](https://github.com/LSSTScienceCollaborations/DMStackClub/issues/48)\n", "\n", "## Goals\n", - "In this tutorial we will understand the possible algorithm choices for DIA template image generation, see how these are currently implemented in the science pipelines, and demonstrate them on the Twinkles images." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Requirements" + "In this tutorial we will understand the possible algorithm choices for DIA template image generation, see how these are currently implemented in the science pipelines, and demonstrate them on the Twinkles images.\n", + "\n", + "#### Requirements\n", + "No special packages are needed, just the DM Stack.\n", + "\n", + "----" ] }, { @@ -40,16 +38,16 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "![](https://github.com/lsst/LDM-151/blob/master/figures/drp_summary.png)" + "![](https://github.com/lsst/LDM-151/raw/master/figures/drp_coaddition_and_diffim.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "Difference imaging is shown in block 4, and includes creation of `TemplateCoadd` images, image differencing, and `DIASource` creation. The relevant pipeline design text on template generation, for the data release processing, is in Section 5.2:\n", + "Difference imaging is shown in block 4, and includes creation of TemplateCoadd images, image differencing, and DIASource creation. The relevant pipeline design text on template generation, for the data release processing, is in Section 5.2:\n", "\n", - "> \"4. We run the WarpTemplates, CoaddTemplates, and DiffIm pipelines to generate the DI- ASource and DiffExp datasets. We may then be able to generate better CalExp Masks than we can obtain from BackgroundMatchAndReject by comparing the DiffExp masks across visits in the UpdateMasks pipeline.\n", + "> \"4. We run the WarpTemplates, CoaddTemplates, and DiffIm pipelines to generate the DIASource and DiffExp datasets. We may then be able to generate better CalExp Masks than we can obtain from BackgroundMatchAndReject by comparing the DiffExp masks across visits in the UpdateMasks pipeline.\n", ">\n", "> \"5. After all CalExp components have been finalized, we run the WarpRemaining and CoaddRemaining to build additional coadd data products.\"\n", "\n", @@ -72,6 +70,27 @@ "\n", "* **\"The result of inference on resampled exposures with no PSF-matching\"**\n" ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Selecting the Input Visit Images" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Warping the Input Images" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Making a Standard Decorrelated TemplateCoadd" + ] } ], "metadata": { From 5d1dc32e927552867b50c912b4f240fa36b75010 Mon Sep 17 00:00:00 2001 From: Phil Marshall Date: Fri, 24 Aug 2018 18:05:37 +0000 Subject: [PATCH 5/9] Started looking at Twinkles data --- ...DIA_How_To_Generate_a_Template_Image.ipynb | 191 ++++++++++++++++-- 1 file changed, 173 insertions(+), 18 deletions(-) diff --git a/DIA/DIA_How_To_Generate_a_Template_Image.ipynb b/DIA/DIA_How_To_Generate_a_Template_Image.ipynb index 21a54a21..2486325b 100644 --- a/DIA/DIA_How_To_Generate_a_Template_Image.ipynb +++ b/DIA/DIA_How_To_Generate_a_Template_Image.ipynb @@ -4,25 +4,35 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# DIA: How To Generate a Template Image" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "
**Owner:** Phil Marshall (@drphilmarshall)\n", - "
**Last Run:** 2018-07-13\n", - "
**Stack Version:** v16\n", + "# DIA: How To Generate a Template Image\n", + "
Owner(s): **Phil Marshall** ([@drphilmarshall](https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@drphilmarshall))\n", + "
Last Verified to Run: **2018-08-24**\n", + "
Verified Stack Release: **16.0**\n", "
**Discussion:** [issue #48](https://github.com/LSSTScienceCollaborations/DMStackClub/issues/48)\n", "\n", - "## Goals\n", "In this tutorial we will understand the possible algorithm choices for DIA template image generation, see how these are currently implemented in the science pipelines, and demonstrate them on the Twinkles images.\n", "\n", - "#### Requirements\n", - "No special packages are needed, just the DM Stack.\n", + "### Learning Objectives:\n", "\n", - "----" + "After working through this tutorial you should be able to: \n", + "1. Use a SkyMap to identify all the visit images in a given filter you want to coadd ;\n", + "2. Combine these images into a decorrelated template image.\n", + "\n", + "### Logistics\n", + "This notebook is intended to be runnable on `lsst-lspdev.ncsa.illinois.edu` from a local git clone of https://github.com/LSSTScienceCollaborations/StackClub.\n", + "\n", + "## Set-up" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# What version of the Stack am I using?\n", + "! echo $HOSTNAME\n", + "! eups list -s | grep lsst_distrib" ] }, { @@ -31,7 +41,8 @@ "source": [ "## Background\n", "\n", - "The DRP pipelines are summarized in Figure 9 of [LDM-151](http://ls.st/ldm-151), reproduced here:" + "The DRP pipelines are summarized in Figure 9 of the _Data Management Science Pipelines\n", + "Design Document_, [LDM-151](http://ls.st/ldm-151), reproduced here:" ] }, { @@ -75,21 +86,165 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Selecting the Input Visit Images" + "## The Twinkles_subset Dataset\n", + "\n", + "We'll use the `Twinkles_subset` data, in a local data \"repository\" copied from the shared project space:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%%bash\n", + "if [ ! -d TWINKLES_DATA ]; then cp -r /project/shared/data/Twinkles_subset/input_data_v2 TWINKLES_DATA; fi" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Here's what this data repository contains:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!ls -lh TWINKLES_DATA/" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The Butler uses a mapper to find and organize data in a format specific to each camera. Here we're using `lsst.obs.lsstSim.LsstSimMapper` mapper for the Twinkles simulated data:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!cat TWINKLES_DATA/_mapper" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "All of the relavent images and calibrations have already been ingested into the Butler for this data set." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Warping the Input Images" + "## Making a SkyMap\n", + "\n", + "We need to make a SkyMap, to define the _tracts_ and _patches_ of sky that will be covered by our coadd image. For this, we could use the `makeSkyMap.py` command line task, as described in the [pipelines.lsst.io Getting started part 4](https://pipelines.lsst.io/getting-started/coaddition.html) tutorial." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "! makeDiscreteSkyMap.py TWINKLES_DATA --id --rerun coadd --config skyMap.projection=\"TAN\"\n", + "\n", + "# cf the following line from Dominique Fouchez's minicookbook:\n", + "# makeSkyMap.py input --output output --configfile makeSkyMapConfig.py" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Making a Standard Decorrelated TemplateCoadd" + "The above command fails with the following error:\n", + "```\n", + "makeDiscreteSkyMap INFO: Extracting bounding boxes of 0 images\n", + "makeDiscreteSkyMap FATAL: Failed: No data found from which to compute convex hull\n", + "```\n", + "\n", + "We need to somehow specify which images in the Twinkles_subset to use - and these are e-images, rather than outputs from `processCcd`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%load_ext autoreload\n", + "%autoreload 2" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from stackclub import where_is" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# where_is(\"makeDiscreteSkyMap.py\", in_the=\"repo\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The above command does not work yet: to find information on the command line tasks, we need to upgrade the `where_is` function." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Making the Coadd Images" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "\"\"\"\n", + "#find the tract/patches overlapping visit 182014\n", + "./reportPatchesWithImages.py output --visits 182014 --filt r\n", + "sed -e 's/^/--id filter=r /' r_182014_patches.list > patches_r.txt\n", + "\n", + "#make coadd on these patches to create template\n", + "makeCoaddTempExp.py input --output output @patches_r.txt --selectId filter=r visit=182014^452599 -j 32\n", + "assembleCoadd.py output --output output @patches_r.txt --selectId filter=r visit=182014^452599 -j 32 \n", + "\"\"\";" ] } ], From 6b009b30a8489a1a82a81dd51c860455891f98bd Mon Sep 17 00:00:00 2001 From: Phil Marshall Date: Fri, 24 Aug 2018 19:58:18 +0000 Subject: [PATCH 6/9] Upgrade to enable command line tasks to be found --- stackclub/where_is.py | 38 +++++++++++++++++++++++++------------- 1 file changed, 25 insertions(+), 13 deletions(-) diff --git a/stackclub/where_is.py b/stackclub/where_is.py index 9ff5ae45..ab8f7ab0 100644 --- a/stackclub/where_is.py +++ b/stackclub/where_is.py @@ -1,13 +1,15 @@ -def where_is(object, in_the=None): +def where_is(object, in_the='source', assuming_its_a=None): """ Print a markdown hyperlink to the source code of `object`. Parameters ---------- - object: python object - The class or function you are looking for. + object: python object or string + The class or function you are looking for, or the name of a python object or file. in_the: string, optional The kind of place you want to look in: `['source', 'repo', 'technotes']` + assuming_its_a: string, optional + The kind of object you think you have: `['cmdlinetask'], default=None Examples -------- @@ -17,32 +19,42 @@ def where_is(object, in_the=None): >>> where_is(Butler.get, in_the='source') >>> where_is(Butler, in_the='repo') >>> where_is(Butler, in_the='technotes') + >>> where_is("makeDiscreteSkyMap.py", in_the="source", assuming_its_a="cmdlinetask") Notes ----- See also the `FindingDocs tutorial notebook `_ for a working demo. """ + # Deal with string object names - useful for locating command line tasks: + if isinstance(object, str): + objectname = object + if in_the == 'source' and assuming_its_a == None: + raise ValueError('Cannot locate task/object `'+object+'` in the source by name. Either pass in an object, or use the "assuming_its_a" kwarg to guess what kind of object it is.') + if assuming_its_a == "cmdlinetask": + modulename = 'lsst.pipe.tasks.'+objectname + + elif isinstance(object, module): + # Locate the module that contains the desired object, and break its name into pieces: + modulename = object.__module__ + objectname = object.__name__ - # Locate the module that contains the desired object, and break its name into pieces: - modulename = object.__module__ - pieces = str.split(modulename,'.') - objectname = object.__name__ + else: + raise TypeError('Expecting objects of type "string" or "module"') - # Form the URL, and a useful markdown representation of it: - if in_the is None: in_the = 'source' - + # Form the URL, and a useful markdown representation of it: if in_the == 'source': + pieces = str.split(modulename,'.') URL = 'https://github.com/'+pieces[0]+'/'+pieces[1]+'_'+pieces[2] \ + '/blob/master/python/'+pieces[0]+'/'+pieces[1]+'/'+pieces[2]+'/'+pieces[3]+'.py' - link = '['+modulename+']('+URL+')' + link = '[`'+modulename+'`]('+URL+')' elif in_the == 'repo': - URL = 'https://github.com/search?l=Python&q=user%3Alsst+'+objectname+'&type=Code' + URL = 'https://github.com/search?l=Python&q=org%3Alsst+'+objectname+'&type=Code' link = '[searching for `'+objectname+'` in the `lsst` repo]('+URL+')' elif in_the == 'technotes': - URL = 'https://github.com/search?l=reStructuredText&q=user%3Alsst-dm+'+objectname+'&type=Code' + URL = 'https://github.com/search?l=reStructuredText&q=org%3Alsst-dm+'+objectname+'&type=Code' link = '[searching for `'+objectname+'` in the `lsst-dm` technotes]('+URL+')' else: From 4f19e8cc75ed93cc2d87849cdd1685783fadafd3 Mon Sep 17 00:00:00 2001 From: Phil Marshall Date: Fri, 24 Aug 2018 19:58:55 +0000 Subject: [PATCH 7/9] Finding task docs --- ...DIA_How_To_Generate_a_Template_Image.ipynb | 57 ++++++++++--------- .../templates/template_Notebook.ipynb | 34 ++++++----- 2 files changed, 51 insertions(+), 40 deletions(-) diff --git a/DIA/DIA_How_To_Generate_a_Template_Image.ipynb b/DIA/DIA_How_To_Generate_a_Template_Image.ipynb index 2486325b..c7fb07a4 100644 --- a/DIA/DIA_How_To_Generate_a_Template_Image.ipynb +++ b/DIA/DIA_How_To_Generate_a_Template_Image.ipynb @@ -8,7 +8,7 @@ "
Owner(s): **Phil Marshall** ([@drphilmarshall](https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@drphilmarshall))\n", "
Last Verified to Run: **2018-08-24**\n", "
Verified Stack Release: **16.0**\n", - "
**Discussion:** [issue #48](https://github.com/LSSTScienceCollaborations/DMStackClub/issues/48)\n", + "
Discussion: **[issue #48](https://github.com/LSSTScienceCollaborations/DMStackClub/issues/48)**\n", "\n", "In this tutorial we will understand the possible algorithm choices for DIA template image generation, see how these are currently implemented in the science pipelines, and demonstrate them on the Twinkles images.\n", "\n", @@ -35,6 +35,31 @@ "! eups list -s | grep lsst_distrib" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "During development, it's helpful to have the `stackclub` library available - and the `where_is` documentation finder in particular." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "! cd .. && python setup.py -q develop --user && cd -" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from stackclub import where_is" + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -146,7 +171,7 @@ "source": [ "## Making a SkyMap\n", "\n", - "We need to make a SkyMap, to define the _tracts_ and _patches_ of sky that will be covered by our coadd image. For this, we could use the `makeSkyMap.py` command line task, as described in the [pipelines.lsst.io Getting started part 4](https://pipelines.lsst.io/getting-started/coaddition.html) tutorial." + "We need to make a SkyMap, to define the _tracts_ and _patches_ of sky that will be covered by our coadd image. For this, we could use the [`lsst.pipe.tasks.makeDiscreteSkyMap.py`](https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/makeDiscreteSkyMap.py) command line task, as described in the [pipelines.lsst.io Getting started part 4](https://pipelines.lsst.io/getting-started/coaddition.html) tutorial." ] }, { @@ -180,8 +205,7 @@ "metadata": {}, "outputs": [], "source": [ - "%load_ext autoreload\n", - "%autoreload 2" + "where_is(\"makeDiscreteSkyMap.py\", in_the=\"source\", assuming_its_a=\"cmdlinetask\")" ] }, { @@ -190,7 +214,7 @@ "metadata": {}, "outputs": [], "source": [ - "from stackclub import where_is" + "from lsst.pipe.tasks.makeDiscreteSkyMap import MakeDiscreteSkyMapTask" ] }, { @@ -199,30 +223,9 @@ "metadata": {}, "outputs": [], "source": [ - "# where_is(\"makeDiscreteSkyMap.py\", in_the=\"repo\")" + "help(MakeDiscreteSkyMapTask)" ] }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The above command does not work yet: to find information on the command line tasks, we need to upgrade the `where_is` function." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - }, { "cell_type": "markdown", "metadata": {}, diff --git a/GettingStarted/templates/template_Notebook.ipynb b/GettingStarted/templates/template_Notebook.ipynb index db8831b9..218b9ea1 100644 --- a/GettingStarted/templates/template_Notebook.ipynb +++ b/GettingStarted/templates/template_Notebook.ipynb @@ -42,24 +42,16 @@ "```\n", "pip install git+git://github.com/LSSTScienceCollaborations/StackClub.git#egg=stackclub\n", "```\n", - "If you are developing the `stackclub` package (eg by adding modules to it to support the Stack Club tutorial that you are writing, you'll need to make a local, editable installation. In the top level folder of the `StackClub` repo, do:" + "If you are developing the `stackclub` package (eg by adding modules to it to support the Stack Club tutorial that you are writing, you'll need to make a local, editable installation, like this:" ] }, { "cell_type": "code", - "execution_count": 1, + "execution_count": null, "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "python: can't open file 'setup.py': [Errno 2] No such file or directory\n" - ] - } - ], + "outputs": [], "source": [ - "! cd .. && python setup.py -q develop --user && cd -" + "! cd $HOME/notebooks/StackClub && python setup.py -q develop --user && cd -" ] }, { @@ -71,7 +63,7 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": null, "metadata": {}, "outputs": [], "source": [ @@ -79,6 +71,22 @@ "%autoreload 2" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "During development, it's helpful to have the `where_is` documentation finder loaded in:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from stackclub import where_is" + ] + }, { "cell_type": "markdown", "metadata": { From eac65bdf6ecd24cc802a2b0a4bc73e044a9902c9 Mon Sep 17 00:00:00 2001 From: Phil Marshall Date: Fri, 31 Aug 2018 10:48:01 -0700 Subject: [PATCH 8/9] Guessed MakeDiscreteSkyMap task set-up --- ...DIA_How_To_Generate_a_Template_Image.ipynb | 37 ++++++++++++++++--- 1 file changed, 31 insertions(+), 6 deletions(-) diff --git a/DIA/DIA_How_To_Generate_a_Template_Image.ipynb b/DIA/DIA_How_To_Generate_a_Template_Image.ipynb index c7fb07a4..d753b077 100644 --- a/DIA/DIA_How_To_Generate_a_Template_Image.ipynb +++ b/DIA/DIA_How_To_Generate_a_Template_Image.ipynb @@ -208,13 +208,20 @@ "where_is(\"makeDiscreteSkyMap.py\", in_the=\"source\", assuming_its_a=\"cmdlinetask\")" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's unpack the `makeDiscreteSkyMap` command line task, and configure and run it from python. First we need to import the task and its configuration object." + ] + }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ - "from lsst.pipe.tasks.makeDiscreteSkyMap import MakeDiscreteSkyMapTask" + "from lsst.pipe.tasks.makeDiscreteSkyMap import MakeDiscreteSkyMapTask, MakeDiscreteSkyMapConfig" ] }, { @@ -226,6 +233,24 @@ "help(MakeDiscreteSkyMapTask)" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Setting up the task with the default configuration, let's run it in this vanilla mode before reconfiguring." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "skyMapConfig = MakeDiscreteSkyMapConfig()\n", + "skyMapTask = MakeDiscreteSkyMapTask(config=skyMapConfig)\n", + "skyMapTask.run()" + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -253,21 +278,21 @@ ], "metadata": { "kernelspec": { - "display_name": "LSST", + "display_name": "Python 2", "language": "python", - "name": "lsst" + "name": "python2" }, "language_info": { "codemirror_mode": { "name": "ipython", - "version": 3 + "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.2" + "pygments_lexer": "ipython2", + "version": "2.7.12" } }, "nbformat": 4, From a6c06adaf6953540796c54a539ce1d9b7764b3da Mon Sep 17 00:00:00 2001 From: Phil Marshall Date: Fri, 25 Jan 2019 10:39:12 -0800 Subject: [PATCH 9/9] Replaced markdown with rst --- DIA/README.md | 7 ------- DIA/README.rst | 29 +++++++++++++++++++++++++++++ 2 files changed, 29 insertions(+), 7 deletions(-) delete mode 100644 DIA/README.md create mode 100644 DIA/README.rst diff --git a/DIA/README.md b/DIA/README.md deleted file mode 100644 index 42392380..00000000 --- a/DIA/README.md +++ /dev/null @@ -1,7 +0,0 @@ -# Difference Image Analysis - -This folder contains a set of tutorial notebooks exploring the Difference Image Analysis (DIA) parts of the LSST science pipelines. To join the discussion of this project, please see [issue #48](https://github.com/LSSTScienceCollaborations/DMStackClub/issues/48) - -| Notebook | Description | Owner | -|---|---|---| -| [DIA_How_To_Generate_a_Template_Image](DIA_How_To_Generate_a_Template_Image.ipynb) | Options for template coadd generation, demonstration on Twinkles images. | Phil Marshall (@drphilmarshall) | diff --git a/DIA/README.rst b/DIA/README.rst new file mode 100644 index 00000000..42332fe1 --- /dev/null +++ b/DIA/README.rst @@ -0,0 +1,29 @@ +Difference Image Analysis +------------------------- + +This folder contains a set of tutorial notebooks exploring the Difference Image Analysis (DIA) parts of the LSST science pipelines. See the index table below for links to the notebook code, and an auto-rendered view of the notebook with outputs. +To join the discussion of this project, please see [issue #48](https://github.com/LSSTScienceCollaborations/DMStackClub/issues/48) + +.. list-table:: + :widths: 10 20 10 10 + :header-rows: 1 + + * - Notebook + - Short description + - Links + - Owner + + + * - **How_To_Generate_a_Template_Image** + - Options for template coadd generation, demonstration on Twinkles images. + - `ipynb `__, + `rendered `__ + + .. raw:: html + + + + + + + - `Phil Marshall `__