DataCheckpoint_Group019_WI24.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**If you lost points on the last checkpoint you can get them back by responding to TA/IA feedback**  \n",
    "\n",
    "Update/change the relevant sections where you lost those points, make sure you respond on GitHub Issues to your TA/IA to call their attention to the changes you made here.\n",
    "\n",
    "Please update your Timeline... no battle plan survives contact with the enemy, so make sure we understand how your plans have changed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# COGS 108 - Data Checkpoint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Names\n",
    "\n",
    "- McKayla David\n",
    "- Sebastian Modafferi\n",
    "- Anna Potapenko\n",
    "- Matthew Chan\n",
    "- Kirthin Rajkumar\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Research Question"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-  Include a specific, clear data science question.\n",
    "-  Make sure what you're measuring (variables) to answer the question is clear\n",
    "\n",
    "What is your research question? Include the specific question you're setting out to answer. This question should be specific, answerable with data, and clear. A general question with specific subquestions is permitted. (1-2 sentences)\n",
    "\n",
    "On a global economic scale, is a larger (on the basis of funding and quantity of employees) company more likely to lay off employees than a smaller company? If not, what indicators can be viewed to analyze what influences company layoffs?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Background and Prior Work"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Layoffs refer to economic and organizational changes, and are a significant indicator of the success and development of companies. While they affect employees, layoff rates have a broader implication on the health of the economy, industry trends, and the state of the workforce. Understanding the trends and impacts of layoffs is vital not only on a global economic scale, but within communities of stakeholders and individual communities.\n",
    "\n",
    "Layoffs are important to study due to their relevance to both companies and employees. For a company, understanding indicators which can predict imminent layoffs can help them course correct before reaching a point of no return. On the other hand, employees understanding layoff indicators can help them in choosing the correct company for their next role, ensuring job safety. \n",
    "\n",
    "Research published in the Journal of the European Economic Association [1^](#https://academic.oup.com/jeea/article-abstract/18/1/427/5247011) explored the economic influences that cause layoffs, and inquired into how financial health and market factors influence layoff decisions. A similar study published in Journal of Labor Economics looks into the effects of layoffs on unemployment rates, and found that layoffs can have lasting impact on the job market and employee career trajectory.\n",
    "\n",
    "The journal of Labor Empirical Finance [3^](#https://doi.org/10.1016/s0927-5398\\(01\\)00024-x) also looks into the different firms and what caused their layoffs, giving insight into company restructuring and different technologies that help to modify the workforce requirements. Additionally, past precedent reviewed by JSTOR [2^](#https://www.jstor.org/stable/117002?casa_token=m7s1bFw7mY4AAAAA%3AhaYXwJWsj5E0Xo7vbnjns6omvUnSFYlenLVZ99nBhONKkQRCLyfLIdEk3ZJycob9If4HtLaMga7y7cQzrzAO6QfJYXTkccHfVciVYhTXREH7HSHuGN4) article explains the repetition of layoffs and how it correlates with economic cycles. This suggests that layoffs are an essential part of economic growth.\n",
    "\n",
    "Research on layoffs adopts an interdisciplinary approach, using economic theories, organizational behavior, and societal impacts. Overall, it is imperative to understand the factors that influence layoffs because knowledge about these factors can help researchers develop strategies to mitigate the negative effects of layoffs on employees and the economy at large. Existing work does not provide internal indicators for when a company is about to execute layoffs, so our research seeks to identify a correlation between company size and layoffs.\n",
    "\n",
    "1. <a name=\"Journal of the European Economic Association\"></a> [^](#https://academic.oup.com/jeea/article-abstract/18/1/427/5247011) Gathmann, C., Helm, I., & Schönberg, U. (2018). Spillover effects of mass layoffs. Journal of the European Economic Association, 18(1), 427–468. https://doi.org/10.1093/jeea/jvy045\n",
    "2. <a name=\"JSTOR\"></a> [^](#cite_ref-2) Hallock, Kevin, (1998). Layoffs, top executive pay, and firm performance on JSTOR. (n.d.). www.jstor.org. https://www.jstor.org/stable/117002\n",
    "3. <a name=\"Journal of Empirical Finance\"></a> [^](#https://doi.org/10.1016/s0927-5398\\(01\\)00024-x) Chen, P., Mehrotra, V., Sivakumar, R., & Yu, W. (2001). Layoffs, shareholders’ wealth, and corporate performance. Journal of Empirical Finance, 8(2), 171–199. https://doi.org/10.1016/s0927-5398(01)00024-x\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hypothesis\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We hypothesize that larger companies are more likely to lay off employees (especially amidst a recession) than smaller companies. We are inclined to believe this due to the fact that smaller companies already have less employees, so lay-offs are more likely to harm the business than benefit it. Additionally, larger companies are able to withstand more financial pressure, allowing them to perform large layoffs despite the impact on company performance given that they have enough capital with withstand the losses.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data overview\n",
    "\n",
    "For each dataset include the following information\n",
    "- Dataset #1 - Kaggle\n",
    "  - Dataset Name: \"Tech Layoffs 2020-2024\"\n",
    "  - https://www.kaggle.com/datasets/ulrikeherold/tech-layoffs-2020-2024\n",
    "  - Number of observations: 1418\n",
    "  - Number of variables: 16\n",
    "\n",
    "This dataset was webscraped from layoffs.fyi. It contains layoff data over the past 4 years which was webscraped from news articles. The key data variables we will be using are `Money_Raised_in_$_mil`, `Percentage`, `Laid_Off`, `Funding`, and `Stage`. We are focusing analysis on these columns because they contain vital information about layoffs and how the company is performing. It comes fairly clean, and the only correction required is the `Money_Raised_in_$_mil` column, as it initally was stored as a string containing a dollar sign character."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset #1 (Layoffs.fyi)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>#</th>\n",
       "      <th>Company</th>\n",
       "      <th>Location_HQ</th>\n",
       "      <th>Country</th>\n",
       "      <th>Continent</th>\n",
       "      <th>Laid_Off</th>\n",
       "      <th>Date_layoffs</th>\n",
       "      <th>Percentage</th>\n",
       "      <th>Company_Size_before_Layoffs</th>\n",
       "      <th>Company_Size_after_layoffs</th>\n",
       "      <th>Industry</th>\n",
       "      <th>Stage</th>\n",
       "      <th>Money_Raised_in_$_mil</th>\n",
       "      <th>Year</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>ShareChat</td>\n",
       "      <td>Bengaluru</td>\n",
       "      <td>India</td>\n",
       "      <td>Asia</td>\n",
       "      <td>200</td>\n",
       "      <td>2023-12-20</td>\n",
       "      <td>15.0</td>\n",
       "      <td>1333</td>\n",
       "      <td>1133</td>\n",
       "      <td>Consumer</td>\n",
       "      <td>Series H</td>\n",
       "      <td>$1700</td>\n",
       "      <td>2023</td>\n",
       "      <td>12.97194</td>\n",
       "      <td>77.59369</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>InSightec</td>\n",
       "      <td>Haifa</td>\n",
       "      <td>Israel</td>\n",
       "      <td>Asia</td>\n",
       "      <td>100</td>\n",
       "      <td>2023-12-19</td>\n",
       "      <td>20.0</td>\n",
       "      <td>500</td>\n",
       "      <td>400</td>\n",
       "      <td>Healthcare</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>$733</td>\n",
       "      <td>2023</td>\n",
       "      <td>32.81841</td>\n",
       "      <td>34.98850</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6</td>\n",
       "      <td>Enphase Energy</td>\n",
       "      <td>San Francisco Bay Area</td>\n",
       "      <td>USA</td>\n",
       "      <td>North America</td>\n",
       "      <td>350</td>\n",
       "      <td>2023-12-18</td>\n",
       "      <td>10.0</td>\n",
       "      <td>3500</td>\n",
       "      <td>3150</td>\n",
       "      <td>Energy</td>\n",
       "      <td>Post-IPO</td>\n",
       "      <td>$116</td>\n",
       "      <td>2023</td>\n",
       "      <td>37.54827</td>\n",
       "      <td>-121.98857</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7</td>\n",
       "      <td>Udaan</td>\n",
       "      <td>Bengaluru</td>\n",
       "      <td>India</td>\n",
       "      <td>Asia</td>\n",
       "      <td>100</td>\n",
       "      <td>2023-12-18</td>\n",
       "      <td>10.0</td>\n",
       "      <td>1000</td>\n",
       "      <td>900</td>\n",
       "      <td>Retail</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>1500</td>\n",
       "      <td>2023</td>\n",
       "      <td>12.97194</td>\n",
       "      <td>77.59369</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>14</td>\n",
       "      <td>Cruise</td>\n",
       "      <td>San Francisco Bay Area</td>\n",
       "      <td>USA</td>\n",
       "      <td>North America</td>\n",
       "      <td>900</td>\n",
       "      <td>2023-12-14</td>\n",
       "      <td>24.0</td>\n",
       "      <td>3750</td>\n",
       "      <td>2850</td>\n",
       "      <td>Transportation</td>\n",
       "      <td>Acquired</td>\n",
       "      <td>$15000</td>\n",
       "      <td>2023</td>\n",
       "      <td>37.77493</td>\n",
       "      <td>-122.41942</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    #         Company             Location_HQ Country      Continent  \\\n",
       "0   3       ShareChat               Bengaluru   India           Asia   \n",
       "1   4       InSightec                   Haifa  Israel           Asia   \n",
       "2   6  Enphase Energy  San Francisco Bay Area     USA  North America   \n",
       "3   7           Udaan               Bengaluru   India           Asia   \n",
       "4  14          Cruise  San Francisco Bay Area     USA  North America   \n",
       "\n",
       "   Laid_Off Date_layoffs  Percentage  Company_Size_before_Layoffs  \\\n",
       "0       200   2023-12-20        15.0                         1333   \n",
       "1       100   2023-12-19        20.0                          500   \n",
       "2       350   2023-12-18        10.0                         3500   \n",
       "3       100   2023-12-18        10.0                         1000   \n",
       "4       900   2023-12-14        24.0                         3750   \n",
       "\n",
       "   Company_Size_after_layoffs        Industry     Stage Money_Raised_in_$_mil  \\\n",
       "0                        1133        Consumer  Series H                 $1700   \n",
       "1                         400      Healthcare   Unknown                  $733   \n",
       "2                        3150          Energy  Post-IPO                  $116   \n",
       "3                         900          Retail   Unknown                  1500   \n",
       "4                        2850  Transportation  Acquired                $15000   \n",
       "\n",
       "   Year       lat        lng  \n",
       "0  2023  12.97194   77.59369  \n",
       "1  2023  32.81841   34.98850  \n",
       "2  2023  37.54827 -121.98857  \n",
       "3  2023  12.97194   77.59369  \n",
       "4  2023  37.77493 -122.41942  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "df = pd.read_excel('./data/tech_layoffs.xlsx')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "#remove company name\n",
    "df = df.drop(columns=['Company'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('O')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Money_Raised_in_$_mil'].dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1418, 16)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#clean the Money_Raised_in_$_mil column to be float instead of string\n",
    "df['Funding'] = df['Money_Raised_in_$_mil'].apply(lambda s: np.float64(s[1:])) \n",
    "df['Funding'].head()\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ethics & Privacy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Potential ethical concerns and how we plan to address them: \n",
    "Our dataset is webscraped from Layoffs.fyi, which contains explicit personal information on individuals who were laid off. Without explicit documentation of informed consent, for the sake of privacy conservation, we will be omitting this information and focusing on the metadata (corporations over the individual). Additionally, Layoffs.fyi only pulls data from news articles, so it is a biased sample that is pulled from data that is only accessible to the public. This dataset is primarily constructed by data contained to the USA, which effectively neglects layoffs that occur in other regions of the world, leading to potentially biased analysis and results. As a result of unsatisfactory observations from foreign countries, we will be orienting our data analysis in the context of the USA's economy. However, we will still include models and representations of non-US observations to provide scope and a point of reference to our data. The timeframe of our data is 2020-2024, which unfortunately excludes a larger historical context regarding layoffs, compounding potential bias and lack of scope. Due to this, our analysis will be further oriented towards a COVID and post-COVID economy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Team Expectations "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We expect our team members to be reliable in terms of completing one’s own work/contributions. They should maintain open communication between team members and are expected to communicate any scheduling conflicts for team meetings. They are still expected to complete their work before the meeting even if they are not able to make it. During team meetings, we expect all members to be actively contributing to discussion, and to be professional when discussing conflicts between ideas. Each member will be assigned tasks by the end of the team meeting, and they are expected to arrive to the next team meeting with their task completed sufficiently, and uploaded to the repository, such that we are able to discuss progress and any issues we ran into."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Project Timeline Proposal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Meeting Date  | Meeting Time| Completed Before Meeting  | Discuss at Meeting |\n",
    "|---|---|---|---|\n",
    "| 02/04 | 1 PM | Read previous COGS 108 Final Projects | Complete previous quarters’ COGS 108 Final Project Analysis, plan meeting times, begin discussing project topics. | \n",
    "| 02/05  | 1 PM | Brainstorm project topics, potential data sources, and viability of research questions | Discuss and decide on final project topic; discuss hypothesis; begin background research Discuss ideal dataset(s) and ethics; draft project proposal| \n",
    "| 02/11  | 1 PM | Edit, finalize, and submit proposal; Search for datasets | Discuss Wrangling and possible analytical approaches; Assign group members to lead each specific part |\n",
    "| 02/18  | 1 PM  | Delegate Tasks and start wrangling | Go over what everyone has done. Make edits or revise things before. Also go over revisions and feedback from the proposal. |\n",
    "| 02/25  | 1 PM  | Finalize wrangling/EDA; Begin Analysis | Meet for Checkpoint #1 |\n",
    "| 03/03  | 1 PM  | Finalize Data Viz and EDA; Begin Analysis | Meet for Checkpoint #2 |\n",
    "| 03/10  | 1 PM  | Finalize quantitative analysis; Discuss approach to final video submission | Meet for video and final submission semantics |\n",
    "| 03/13 | 12 PM  | Complete analysis; Draft results/conclusion/discussion | Discuss/edit full project |\n",
    "| 03/20  | Before 11:59 PM  | NA | Turn in Final Project & Group Project Surveys |"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}