Skip to content

Latest commit

 

History

History
91 lines (60 loc) · 2.56 KB

README.md

File metadata and controls

91 lines (60 loc) · 2.56 KB
Raw

Full Stack Web Application

Purpose

This is a demo project for showing how one full stack web application should be like.

Introduction

features:

  • scrawler get data into database
  • webserver push data to browser periodically

Demo

page

Architecture

+-----------+                 +------------+              +------------+               +--------------+
|           |      frontend   |            |              |            |      data     |              |
|   Browser +<----------------+  Vue Server|              |   RestAPI  +<--------------+    Scrapy    |
|           |                 |            |              |            |               |              |
+------+----+                 +------------+              +------+-----+               +--------------+
       ^                                                         |
       |                                                         |
       |                                                         |data
       |                                                         |
       |                                                         |
       |                                                         v
       |                     +-------------+              +------+------+
       |     data            |             |    data      |             |
       +---------------------+  Websocket  +<-------------+   MongoDB   |
                             |             |              |             |
                             +-------------+              +-------------+

Workflow

  1. scratch quotes from scrapy offical demo site(quotes), save to database(MongoDB), and show them through web.
  2. Front end is vue.js project, including WebSocket client(socketio) component.
  3. Back end is websocket server, which query data form MongoDB through ORM mongoose every period(1s), pushing quotes data to front end.
  4. data picker use scrapy to crawl quotes to save to MongoDB.

Technology Stack

Scrapy + webservice[restful] + websocket + VueJS + MongoDB

Component Description
Scrapy scratch data from web
webservice serve restapi for data storation and fetch
websocket push data to browser
VueJS frontend
MongoDB DB

Install

./bin/install.sh

Run

# run all services
./bin/start.sh

# stop all services
./bin/stop.sh

# start scrapy
./bin/start_scrapy.sh

Reference