Changes to include the possibility of a notebook computing the physical parameters

docs/notebooks/README.md

@@ -3,7 +3,10 @@
 This repository contains a number of notebooks designed to introduce key aspects of the Python interface to lephare. 
 Here is a biref outline of each notebook and its purpose:
 
-- [Example_full_run.ipynb](Example_full_run.ipynb) This is an example of how to conduct a full photomoetric redshift run over an input catalogue. All intermediate stages are performed.
+- [Minimal_photoz_run.ipynb] (Minimal_photoz_run.ipynb) This is a minimal run of photo-z.
+- [Typical_use_case.ipynb] (Typical_use_case.ipynb) This is a run showing how to compute the photo-z for a reduced part of the COSMOS2020 catalogue.
+- [Typical_use_case_physicalParameters.ipynb] (Typical_use_case_physicalParameters.ipynb) This is a run showing how to compute the physical parameters for a reduced part of the COSMOS2020 catalogue.
+- [Example_full_run.ipynb](Example_full_run.ipynb) This is an example of how to conduct a full photometric redshift run over an input catalogue. All intermediate stages are performed.
 - [Building_list_of_onesources.ipynb](Building_list_of_onesources.ipynb) This notebook creates a list of sources with similar photometric data which can be looped over to compute photometric redshift estimates.
 - [Example_of_usage_of_magSvc.ipynb](Example_of_usage_of_magSvc.ipynb) This notebook demonstrates the magSvc class which can return magnitudes for a given SED template under given conditions.
 - [Testing_fit_of_one_object_of_the_catalogue.ipynb](Testing_fit_of_one_object_of_the_catalogue.ipynb) Demonstrating a simple fit and plotting functions.

docs/notebooks/Typical_use_case_physicalParameters.ipynb

@@ -0,0 +1,298 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "da7fc884-bc5d-4758-aede-087ac6cd18c3",
+   "metadata": {},
+   "source": [
+    "# Typical use case for physical parameters\n",
+    "\n",
+    "This example notebook shows a run on the COSMOS2020 (Weaver et al. 2022) data set in order to estimate the physical parameters.\n",
+    "\n",
+    "In this notebook we follow what has been done for photo-z. We will be looking at the same COSMOS data set but only use the ugrizy bands. The main difference is the use of BC03 templates to compute physical parameters, and we set the redshift to its spectrosocpic value."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "37a53404-5eca-4634-8a76-24617cd42af7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import lephare as lp\n",
+    "from astropy.table import Table\n",
+    "import numpy as np\n",
+    "import os\n",
+    "from matplotlib import pylab as plt\n",
+    "import time\n",
+    "\n",
+    "%matplotlib inline"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "25dd1907-2e49-45f2-b56b-3bdd220cb887",
+   "metadata": {},
+   "source": [
+    "## Update the config\n",
+    "We will start with the COSMOS configuration as a basis. We will update the various keywords needed for this example. We use the default which is shipped with lephare. You could also download the example text file config from [here](https://github.com/lephare-photoz/lephare-data/blob/main/examples/COSMOS.para) or write it completely from scratch."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1f10ab2c-5d4c-42c6-bfab-340c7316ae5b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "config = lp.default_cosmos_config.copy()\n",
+    "\n",
+    "# You could also load this from a local text file:\n",
+    "# !curl -s -o https://raw.githubusercontent.com/lephare-photoz/lephare-data/refs/heads/main/examples/COSMOS.para\n",
+    "# config = lp.read_config(\"./COSMOS.para\")\n",
+    "\n",
+    "config.update(\n",
+    "    {\n",
+    "        # For a quick demonstration we use a very sparse redshift grid. DO NOT USE FOR SCIENCE!\n",
+    "        # Comment out the following line to improve results.\n",
+    "        \"Z_STEP\": \"0.1,0.,3.\",\n",
+    "        # SED\n",
+    "        # In order to get the physical parameters you need to use\n",
+    "        # Composite Stellar Population synthesis models. Here Bruzual & Charlot (2003).\n",
+    "        # This can be done only for galaxies.\n",
+    "        \"GAL_SED\": \"$LEPHAREDIR/sed/GAL/BC03_CHAB/BC03COMB_MOD.list\",\n",
+    "        # Limit the number of ages\n",
+    "        \"SEL_AGE\": \"$LEPHAREDIR/sed/GAL/BC03_CHAB/AGE_BC03COMB.dat\",\n",
+    "        \"MOD_EXTINC\": \"0,12,0,12\",\n",
+    "        \"EXTINC_LAW\": \"SB_calzetti.dat,SMC_prevot.dat\",\n",
+    "        \"EM_LINES\": \"PHYS\",\n",
+    "        \"EM_DISPERSION\": \"1.\",\n",
+    "        # FILTERS\n",
+    "        # A reduced list of filters:\n",
+    "        \"FILTER_LIST\": \"cosmos/u_new.pb,hsc/gHSC.pb,hsc/rHSC.pb,hsc/iHSC.pb,hsc/zHSC.pb,hsc/yHSC.pb\",\n",
+    "        \"FILTER_CALIB\": \"0\",\n",
+    "        \"FILTER_FILE\": \"filter_test\",\n",
+    "        # FIT\n",
+    "        # We set the redshift to the spec-z value\n",
+    "        \"ZFIX\": \"YES\",\n",
+    "        \"ERR_SCALE\": \"0.02\",\n",
+    "        \"ERR_FACTOR\": \"1.5\",\n",
+    "        \"SPEC_OUT\": \"NO\",  # We would like to see the output\n",
+    "    }\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b1c8ca77-639f-41e3-9b40-2ac27ef26722",
+   "metadata": {},
+   "source": [
+    "## Download the missing data\n",
+    "One does not need to use this functionality if already cloned the full auxiliary data.\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5e666b41-e71e-4174-b939-17ca828e3197",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "lp.data_retrieval.get_auxiliary_data(\n",
+    "    keymap=config,\n",
+    "    # The additional extinction laws for galaxies are not in the principle config\n",
+    "    # so we must add them to be downloaded:\n",
+    "    additional_files=[\n",
+    "        # We also want the example cosmos catalogue to experiment with\n",
+    "        \"examples/COSMOS.in\",\n",
+    "        \"ext/SMC_prevot.dat\",\n",
+    "        \"ext/SB_calzetti.dat\",\n",
+    "    ],\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "681eb30e-c277-4fde-a1c6-4579a8d0ca1b",
+   "metadata": {},
+   "source": [
+    "## Run prepare\n",
+    "\n",
+    "These are the key preparatory stages that calculate the filters in the LePHARE format, calculate the library of SEDs and finally calculate the library of magnitudes for all the models. The prepare method runs *filter*, *sedtolib*, and *mag_gal* that would be run independently at the command line. These are all explained in detail in the [documentation](https://lephare.readthedocs.io/en/latest/original.html#detailed-lephare-user-manual)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b20daf8d-8881-4dc5-a338-9f86121ec2be",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "lp.prepare(\n",
+    "    config,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7f91e803-33ce-4ba4-837f-9a6c52625f68",
+   "metadata": {},
+   "source": [
+    "### Creating the input table\n",
+    "\n",
+    "We need to make an astropy table as input. This can be done using the standard column order:\n",
+    "id, flux0, err0, flux1, err1,..., context, zspec, arbitrary_string. A simple example table with two filters might look like this:\n",
+    "|  id | flux_filt1  |  fluxerr_filt1 |  flux_filt2  |  fluxerr_filt2 | context | zspec | string_data |\n",
+    "|---|---|---|---|---|---|---|---|\n",
+    "|  0 | 1.e-31  | 1.e-32  | 1.e-31  | 2.e-32  | 3 | NaN | \"This is just a note\" |\n",
+    "|  1 | 2.e-31  |  1.e-32 | 1.e-31  | 2.e-32  |3 | 1. | \"This has a specz\" |\n",
+    "|  2 | 2.e-31 | 1.e-32  | 2.e-31  | 2.e-32  | 2 | NaN| \"This context only uses the second filter\" |\n",
+    "\n",
+    "The context detemermines which bands are used but can be -99 or a numpy.nan. We do not need to have units on the flux columns but LePHARE assumes they are in erg /s /cm**2 / Hz if we are using fluxes. The number of columns must be two times the number of filters plus the four additional columns.\n",
+    "\n",
+    "This input table **must use** the standard column ordering to determine column meaning. This odering depends on the filter order in the config FILTER_LIST value. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c43e9307-4dc7-4b26-bb8b-5126f29874f7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Load the full cosmos example we downloaded at the start\n",
+    "cosmos_full = Table.read(f\"{lp.LEPHAREDIR}/examples/COSMOS.in\", format=\"ascii\")\n",
+    "# Lets just look at the first 1000 specz between 0 and 3 to be fast and have a small sample to test\n",
+    "specz_colname = cosmos_full.colnames[-2]\n",
+    "mask = cosmos_full[specz_colname] > 0\n",
+    "mask &= cosmos_full[specz_colname] < 3\n",
+    "cosmos_full = cosmos_full[mask][:1000]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "391d66ef-1108-47d7-8ca0-15c1eab04065",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "input_table = Table()\n",
+    "# The id is in the first column\n",
+    "input_table[\"id\"] = cosmos_full[cosmos_full.colnames[0]]\n",
+    "# Loop over the filters we want to keep to get the number of the filter, n, and the name, b,\n",
+    "filter_names = config[\"FILTER_LIST\"].split(\",\")\n",
+    "for n, filter_name in enumerate(filter_names):\n",
+    "    # The ugrizy fluxes and errors are in cols 3 to 14\n",
+    "    f_col = cosmos_full.colnames[2 * n + 3]\n",
+    "    ferr_col = cosmos_full.colnames[2 * n + 4]\n",
+    "    # By default lephare uses column order so names are irrelevant\n",
+    "    input_table[f\"f_{filter_name}\"] = cosmos_full[f_col]\n",
+    "    input_table[f\"ferr_{filter_name}\"] = cosmos_full[ferr_col]\n",
+    "# The context is a binary flag. Here we set it to use all filters.\n",
+    "input_table[\"context\"] = np.sum(2 ** np.arange(len(filter_names)))\n",
+    "input_table[\"zspec\"] = cosmos_full[specz_colname]\n",
+    "input_table[\"string_data\"] = \"arbitrary_info\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "13754ace-2852-4073-a36b-62609645759b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Look at the first 5 lines of the input table\n",
+    "input_table[:5]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d80599b6-529e-4565-be0e-2dc51445c54c",
+   "metadata": {},
+   "source": [
+    "## Run process\n",
+    "\n",
+    "Finally we run the main fitting process which is equivalent to *zphota* when using the command line. We also need to update some of the config values to make them consistent with the number of filters."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3ca129de-1bc1-479f-b1c7-d0b0a3e041a0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Compute physical parameters\n",
+    "output, _ = lp.process(config, input_table)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "495ca7bf-b03e-4102-8381-5185772cf2f5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# the output is an astropy tabel that can be manipulated in the standard ways.\n",
+    "output[:5]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8f56984e-b3cb-4cde-9ef1-1b13e157dc20",
+   "metadata": {},
+   "source": [
+    "Next we can perform some simple plots to check the output"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a2335825-4cba-492f-a277-8349a43f88c4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "logmass = output[\"MASS_MED\"]\n",
+    "logSFR = output[\"SFR_MED\"]\n",
+    "z = output[\"Z_BEST\"]\n",
+    "cond = (z > 0.5) & (z < 1) & (logmass > 0)\n",
+    "plt.hist(logmass[cond], bins=20)\n",
+    "plt.xlabel(\"log(stellar mass)\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8f1fb5cd-bbdb-4574-85b6-7f209bc65736",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "plt.scatter(logmass[cond], logSFR[cond], s=2.0)\n",
+    "plt.xlabel(\"log(stellar mass)\")\n",
+    "plt.ylabel(\"log(SFR)\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/python_usage.rst

@@ -30,6 +30,7 @@ They should all be able to run independently following installation.
 
     Minimal full run <notebooks/Minimal_photoz_run>
     Typical use case <notebooks/Typical_use_case>
+    Typical use case: physical parameters <notebooks/Typical_use_case_physicalParameters>
     Detailed run <notebooks/Detailed_run>
     Developer example: Building a list of onesources <notebooks/Building_list_of_onesources>
     Developer example: Example of usage of magSvc <notebooks/Example_of_usage_of_magSvc>

src/lephare/data_retrieval.py

@@ -409,8 +409,15 @@ def config_to_required_files(keymap, base_url=None):
     sed_keys = ["STAR_SED", "GAL_SED", "QSO_SED"]
     for key in sed_keys:
         try:
-            list_file = base_url + keymap[key].value
-            required_files += [keymap[key].value]
+            # If find sed/ in the path, assume the list is present in lephare-data
+            # and try to retreive the files
+            list_file = keymap[key].value
+            # Remove the beginning of the path before sed/
+            if list_file.find("sed/") > 0:
+                list_file = (list_file[list_file.find("sed/") :]).strip()
+            required_files += [list_file]
+            # Add the url to retrieve the files
+            list_file = base_url + list_file
             file_names = read_list_file(list_file, prefix=f"sed/{key.split('_')[0]}/")
             required_files += file_names
         except KeyError:

src/lephare/default_cosmos_config.py

@@ -45,7 +45,7 @@
     "GAL_LIB": "LIB_CE",
     "GAL_LIB_IN": "LIB_CE",
     "GAL_LIB_OUT": "CE_COSMOS",
-    "GAL_SED": "examples/COSMOS_MOD.list",
+    "GAL_SED": f"{lp.LEPHAREDIR}/examples/COSMOS_MOD.list",
     "GLB_CONTEXT": "0",
     "INP_TYPE": "F",
     "LIB_ASCII": "NO",
@@ -71,15 +71,15 @@
     "QSO_LIB": "LIB_QSO",
     "QSO_LIB_IN": "LIB_QSO",
     "QSO_LIB_OUT": "QSO_COSMOS",
-    "QSO_SED": "sed/QSO/SALVATO09/AGN_MOD.list",
+    "QSO_SED": f"{lp.LEPHAREDIR}/sed/QSO/SALVATO09/AGN_MOD.list",
     "RF_COLORS": "32,4,4,13",
     "RM_DISCREPANT_BD": "500",
     "SPEC_OUT": "NO",
     "STAR_FSCALE": "3.432E-09",
     "STAR_LIB": "LIB_STAR",
     "STAR_LIB_IN": "LIB_STAR",
     "STAR_LIB_OUT": "STAR_COSMOS",
-    "STAR_SED": "sed/STAR/STAR_MOD_ALL.list",
+    "STAR_SED": f"{lp.LEPHAREDIR}/sed/STAR/STAR_MOD_ALL.list",
     "TRANS_TYPE": "1",
     "VERBOSE": "NO",
     "ZFIX": "NO",

src/lephare/prepare.py

@@ -72,7 +72,10 @@ def prepare(config, star_config=None, gal_config=None, qso_config=None):
         mag_output = os.path.join(os.environ["LEPHAREWORK"], "lib_mag", mag_out_name)
         # Run sedtolib
         sedlib = lp.Sedtolib(config_keymap=updated_config)
-        list_loc = os.path.join(lp.LEPHAREDIR, updated_config[f"{object_type}_SED"].value)
+        list_loc = updated_config[f"{object_type}_SED"].value
+        # if find sed/ in the path, assume it is in lephare-data and not an absolute path
+        if list_loc.find("sed/") >= 0:
+            list_loc = os.path.join(lp.LEPHAREDIR, list_loc)
         sedtolib_kwargs = {f"{object_type.lower()}_sed": list_loc}
         print(sedtolib_kwargs)
         sedlib.run(typ=object_type, **sedtolib_kwargs)

src/lephare/zphota.py

@@ -1,7 +1,5 @@
 import time
 
-import numpy as np
-
 from lephare._lephare import (  # , read_lib, read_doc_filters, readOutKeywords, GalMag, bestFilter, maxkcolor
     GalMag,
     PhotoZ,
@@ -102,9 +100,6 @@ def run(self, **kwargs):
         if autoadapt:
             adapt_srcs = photoz.read_autoadapt_sources()
             a0, a1 = photoz.run_autoadapt(adapt_srcs)
-            offsets = ",".join(np.array(a0).astype(str))
-            offsets = "# Offsets from auto-adapt: " + offsets + "\n"
-            photoz.outputHeader += offsets
         else:
             a0 = []
             a1 = []