[Chap02]Things_All_About_Data.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": " [Chap02]Things All About Data.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "metadata": {
        "id": "UF-Z_DGTEKfq",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#  Things All About Data\n",
        "\n",
        "从 莎草纸 到 互联网.  信息的生产, 传递, 交换... 从未变得如此的便宜  : - )  \n",
        "\n",
        "\n",
        "每个人都有被连接的渴望.  无论是 丝绸之路 还是 微博社交. \n",
        "\n",
        "信息就像水流,  从 高的地方流向低的地方. 从最关心ta 的人群那里, 传递到外缘. \n",
        "\n",
        "炒股的人最先关心哪只股票. 追星的人关心明星的动向. \n",
        "吃货会告诉你哪里有好吃的. ..\n",
        "\n",
        "\n",
        "我们就像骆驼商队, 在孤独的沙漠里,  寻找丝绸之路. \n",
        "\n",
        "而这也是互联网人 或 互联网的程序员的意义.\n",
        "\n",
        "而如果你是经济学家, 你会说, 熙熙攘攘, 皆为利往 :-) \n",
        "\n",
        "\n",
        "So ... This's how a programmer think about the internet and data . \n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "Qhh-y-XqP7a9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## 插播一则童话\n",
        "\n",
        "（-（+（+ 1 2）3）2）1和2就像一个瘦的河流和一个胖的河流在某个节点（+）不期而遇，汇聚一起变成了一条又胖一点的小溪3，然后继续向前走，某天又在某个节点和另一个长得和自己一样的3，在一个鸟语花香的地点相遇，结伴前行变成了6，然而生活不会一直没好，也不会一直只是遇见，可能在某个电闪雷鸣的夜晚，瑟瑟发抖的小6，来到一个岔路口，将自己变身为 4 和 2 ，4 挥着泪 ，和 2告别，继续前行。maybe 心中还有所期待，期待在下个路口遇见一个新朋友。当然，也可能在下个路口即将发生的又是一次离别。\n"
      ]
    },
    {
      "metadata": {
        "id": "lownEC9mRr7p",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# What happend when I open google.com ?\n",
        "\n",
        "\n",
        "\n",
        "## UI & UX\n",
        "\n",
        "啥是 UI?  简而言之, 所见即所得.   也就是使用浏览器 打开  [Google](https://www.google.com/)  所看见的 :-) \n",
        "\n",
        "It's really simple. 只有一个输入框.  整个页面只有这一个可以进行交互的地方.\n",
        "\n",
        "就像是你看见一扇门, 上面 只有一个 大大的按钮. “按我”  :-) \n",
        "\n",
        "\n",
        "当你进行输入的时候,  它会出现一些联想词.  这是怎么产生的? \n",
        "\n",
        "Now here is the magic happens. \n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "Q7BSwYJhyF8_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## API (Application Programming Interface)\n",
        "\n",
        "如果你使用 Chrome. 那么右键审查元素, 在 network tab 下,  你就能拆穿这个自动联想关键词的西洋镜了.\n",
        "你会发现一系列的 HTTP 请求.( Wait.. 啥是 HTTP 请求.. 你把它想象成你在给热心的朋友寄信, 你写一封, 他就立刻回复你一封. 信封上的 URL 相当于邮寄地址. )\n",
        "\n",
        "HTTP 有 Headers 和 Body. \n",
        "为啥需要有 Headers 和 Body 呢?\n",
        "你可以想象成 信封和里面的信纸的关系. \n",
        "邮递员可不需要关心里面内容是啥, 他只需要送往哪里去, 有没有邮戳 就可以寄信了.\n",
        "\n",
        "当然 Body 内容千变万化, 正如信件有 官方文档似的咬文嚼字, 也有如情人间的通信, 充满了胡言乱语. \n",
        "有的 Body 是 工整的 Json, 有的是 漂亮的 HTML, 有的是五彩斑斓的 CSS.  \n",
        "浏览器就像一个一个喜欢乐高的程序员, 把 五彩斑斓的 CSS 披在 HTML 身上, 然后, 往里面吹一口仙气似的, 把 Json 数据注入进去.\n",
        "\n",
        "于是, 你就看见一支栩栩如生的花.\n"
      ]
    },
    {
      "metadata": {
        "id": "LkKKJclfywLH",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "ec2b3a73-bb76-4571-8d13-3d9a058a45c0"
      },
      "cell_type": "code",
      "source": [
        "import requests #引入 requests 模块\n",
        "\n",
        "# open https://finance.yahoo.com/quote/GOOG?p=GOOG&guccounter=1\n",
        "# 浏览器打开这个页面后发起了很多 http 请求. 比如下面是其中一个.\n",
        "resp = requests.get(\"https://query1.finance.yahoo.com/v8/finance/chart/GOOG\") #获取这个地址的页面\n",
        "\n",
        "data = resp.json() #以 json 的格式把获取的内容传入到 data 参数中\n",
        "print(data['chart']['result'][0]['meta']) #打印 Google 股票的数据（表示不理解，，）\n",
        "\n",
        "\n"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{'currency': 'USD', 'symbol': 'GOOG', 'exchangeName': 'NMS', 'instrumentType': 'EQUITY', 'firstTradeDate': 1092902400, 'gmtoffset': -14400, 'timezone': 'EDT', 'exchangeTimezoneName': 'America/New_York', 'chartPreviousClose': 1079.69, 'previousClose': 1079.69, 'scale': 3, 'currentTradingPeriod': {'pre': {'timezone': 'EDT', 'start': 1527235200, 'end': 1527255000, 'gmtoffset': -14400}, 'regular': {'timezone': 'EDT', 'start': 1527255000, 'end': 1527278400, 'gmtoffset': -14400}, 'post': {'timezone': 'EDT', 'start': 1527278400, 'end': 1527292800, 'gmtoffset': -14400}}, 'tradingPeriods': [[{'timezone': 'EDT', 'start': 1527168600, 'end': 1527192000, 'gmtoffset': -14400}]], 'dataGranularity': '1m', 'validRanges': ['1d', '5d', '1mo', '3mo', '6mo', '1y', '2y', '5y', '10y', 'ytd', 'max']}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "DpgvZZUeyRUk",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## DataBase\n",
        "\n",
        "Ooook, 作为一个后端程序员, 关心的当然还是数据了. \n",
        "\n",
        "So, 你在发送 HTTP 请求, Google 这个据说精通字母表的家伙, 从哪搞到的数据呢?\n",
        "\n",
        "Google 可不简单, 假如互联网的数据是金子的话, Google 一家就占据了全球 一半左右的金子 :-)  Huh,  我是个视金子如粪土的 伪君子 :-) \n",
        "\n",
        "跑偏了...\n",
        "\n",
        "\n",
        "上回说到,  你的浏览器和 Google 的 服务器 就像两个热情的好友一样, 总是迅速即时的响应对方. \n",
        "浏览器就像一个胖子, Google 服务器就像一个零食店的老板, 面对 胖子的请求, 总是不断给出数据.\n",
        "\n",
        "那么,  Google 服务器 是怎么存储和使用这些数据呢?\n",
        "\n",
        "正如书需要放在书架, 要把它编好类, 要找书的话, 就按照分类号找书架, 然后就可以快速的找到对应的书.\n",
        "\n",
        "\n",
        "Google 的数据库就像 图书馆一样, 把各种数据, 分类放好. \n",
        "当然有新数据的时候, 需要把书先分类, 才能放到书架. 这在一定程度上降低了 加书的速度, 但为了找起来方便, 又有何不可呢 ? \n",
        "\n",
        "当然, 图书馆员可不是好当的 :-)\n",
        "\n",
        "你需要应对火灾, 雷电, 小偷, 地震... 墨菲定律告诉我们, 凡是有可能发生的, 那就一定会发生.\n",
        "\n",
        "面对日益增长的图书, 一个图书馆是不够的. 而且为了避免各种灾难, 需要把书存多个副本, 存在不同地方. \n",
        "\n",
        "当然一旦把书存多个副本.. 万一有本书有新的版本出现.. 来借阅的人在不同图书馆就可能借到不同版本的书... 真是头疼的事啊 :-)\n",
        "\n",
        "So.. 珍惜这些图书馆员吧. 他们是耐心的, 可爱的, 能应付一切生活的苦.\n",
        "\n",
        "\n",
        "And  据说, 要 把 Google 的 图书完全消灭.  需要在地球上远隔千里的六个城市同时惊现恐怖袭击, 并且一击必杀....."
      ]
    },
    {
      "metadata": {
        "id": "Wa3gTyaCWv6z",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# A really simple database using a file :-)\n",
        "# your code here\n",
        "# Database is a class. it provides a method `query`. \n",
        "# db = Database(fpath)\n",
        "# db.query(\"XXX\") return a list of results. \n",
        "# CRUD create, update, delete, read\n",
        "\n",
        "\n",
        "class Database(object):\n",
        "  \n",
        "  def __init__(self, fpath):\n",
        "    self.fpath = fpath\n",
        "    \n",
        "  \n",
        "  def query(self, q):\n",
        "    # your code here \n",
        "    pass \n",
        "  \n",
        "  def insert(self, row):\n",
        "    # your code here\n",
        "    pass \n",
        "  \n",
        "  \n",
        "  \n",
        "   \n",
        " \n",
        "  "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "UOw7AAfvyU9z",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "\n",
        "## Platform\n",
        "\n",
        "Oookay,  平台这个概念, 很容易让人摸不着头脑, 感觉是天台呢, 还是啥?\n",
        "At first, 先从经济学理念上理解的话.  平台就是商品聚集的地方. 就像一个古代的市集.\n",
        "消费者和供应者在此交换 :-)\n",
        "\n",
        "So, 当我们说一个平台的时候, 你应该脑补, 这个平台的消费者和供应者是啥? \n",
        "\n",
        "比如 阿里云平台, 企业的服务和 企业的用户分别是供应者和消费者.  \n",
        "阿里云平台相当于是个包租婆.. in .. fact \n",
        "\n",
        "当我们理解, 租和共享的概念时, 我们就明白了云计算是什么个意思了.\n",
        "\n",
        "比如国定路有一家店铺, 早上是卖煎饼果子, 中午变成了黄焖鸡米饭, 晚上的时候又变成了烧烤店.\n",
        "\n",
        "这个共享叫做分时共享.\n",
        "\n",
        "又比如, 一个路由器, 100M, 可以支持 10个人 同时上网, 这个叫 分段共享. 就像 一条很宽的马路上, 并排行驶的汽车.\n",
        "\n",
        "所有权和使用权的分离,  让使用者免去维护和运营资源的烦恼, 也减少了资源受损的风险.\n",
        "\n",
        "所以, 云平台提供了资源, 承担了机器可能被雷劈, 网线被挖掘机搞断的风险. \n",
        "\n",
        "云服务的收费, 不仅包含了资源的使用费, 还包含了其风险..\n",
        "\n",
        "正如老板承担了企业倒闭的风险, 利润受损的风险,  So .. 公司里谁承担的风险大, 就是说了算.\n",
        "\n",
        "那么, 你应该明白,  股东和傻呼呼的投资人承担的风险有多大了吧 :-)\n",
        "\n",
        "这种做法, 本质上是分离. 但会经常换一种面目出现. \n",
        "\n",
        "比如, \n",
        "\n",
        "编程时, 把 How(怎么实现) 和 What (做什么) 分离, 叫做解耦 . \n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "WKfX7FnipqVB",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Reference\n",
        "\n",
        "参考书目\n",
        "\n",
        "*   《 Computer Networks  Andrew S. Tanenbaum》\n",
        "*   《 Search Engines : Information Retrieval in Practice 》\n",
        "*   《 Designing Data-Intensive Applications 》\n",
        "  \n"
      ]
    },
    {
      "metadata": {
        "id": "N7PlZSxUpSq5",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Exercise \n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "Wany1lxm28lw",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## 图书分类\n",
        "\n",
        "了解一下中国图书分类法, 分别使用各大高校图书馆(交通大学, 上海图书馆, 华师大图书馆)的搜索 找到\n",
        "《计算机程序的构造和解释》\n",
        "\n",
        "查看有哪些字段. 并尝试使用不同字段, 如 作者名, 题名, 关键词 等搜索.\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "gx-jw-Mw25rM",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "\n",
        "## Simple 爬虫\n",
        "\n",
        "\n",
        "使用 360 搜索引擎, 观察输入 关键词 产生的 HTTP 请求. 并尝试使用你最喜欢的编程语言 模拟这一过程 \n",
        "\n",
        "Tips: python requests : http://docs.python-requests.org/zh_CN/latest/user/quickstart.html\n"
      ]
    },
    {
      "metadata": {
        "id": "XF0iYmlT3ECH",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import requests\n",
        "\n",
        "url = \"\"\n",
        "\n",
        "# your code here :-)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "g5km-pDV2znk",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Project: Find A Song  \n",
        "\n",
        "需要大约 一个星期\n",
        "\n",
        "使用 Python 写一个 简单的 歌曲搜索引擎 .\n",
        "能够提供按歌名, 歌手 等字段进行搜索.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "文本文件如下所示:\n",
        "\n",
        "```\n",
        "歌名, 演唱者, 词作者, 专辑, 链接\n",
        "遍地开花, 棉花糖, 棉花糖, 幸运儿, https://www.xiami.com/song/mQGMDV58aa1\n",
        "\n",
        "```\n",
        "\n",
        "Tips: Python flask : http://docs.jinkan.org/docs/flask/\n"
      ]
    },
    {
      "metadata": {
        "id": "bOhpDIEX2rNh",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# save this to your computer as find_a_song.py \n",
        "# run using  >> python3  find_a_song.py \n",
        "\n",
        "\n",
        "!pip install -q flask # comment this line when save to your computer \n",
        "\n",
        "\n",
        "from datetime import datetime\n",
        "from flask import Flask\n",
        "from flask import request\n",
        "\n",
        "app = Flask(__name__)\n",
        "\n",
        "@app.route(\"/\")\n",
        "def index():\n",
        "    return \"Hello World! It's `%s` \"%(datetime.now())\n",
        "\n",
        "@app.route('/search')\n",
        "def search():\n",
        "\tkeyword = request.args.get('keyword', '')\n",
        "\tsong =  find_a_song(keyword)\n",
        "\treturn \"Oh, I find a song maybe you like `%s` :-) \"%song\n",
        "\n",
        "\n",
        "\n",
        "class Database(object):\n",
        "  def __init__(self, fpath):\n",
        "    self.fpath = fpath\n",
        "   \n",
        "  def query(self, q, column):\n",
        "    \n",
        "    # your code here \n",
        "    # read a file, query by column name \n",
        "    \n",
        "    pass \n",
        "  \n",
        "\n",
        "def find_a_song(keyword):\n",
        "\t\"\"\"\n",
        "\tyour code here :-)\n",
        "\t\"\"\"\n",
        "\t# open song.txt  https://docs.python.org/3/tutorial/inputoutput.html\n",
        "\t# read data\n",
        "\t# find a match \n",
        "  #   fpath = \"XXX\"\n",
        "  #   db = Database(fpath)\n",
        "  #   result = db.query(keyword, 'title')\n",
        "  \n",
        "  \n",
        "\treturn \"Not Found\"\n",
        "\n",
        "\n",
        "# uncomment lines below when you run it :-)\n",
        "\n",
        "# if __name__ == '__main__':\n",
        "# \tapp.run(debug=True)\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}