Full Stack Web Application

Purpose

This is a demo project for showing how one full stack web application should be like.

Introduction

features:

• scrawler get data into database
• webserver push data to browser periodically

---

Demo

Architecture

+-----------+                 +------------+              +------------+               +--------------+
|           |      frontend   |            |              |            |      data     |              |
|   Browser +<----------------+  Vue Server|              |   RestAPI  +<--------------+    Scrapy    |
|           |                 |            |              |            |               |              |
+------+----+                 +------------+              +------+-----+               +--------------+
       ^                                                         |
       |                                                         |
       |                                                         |data
       |                                                         |
       |                                                         |
       |                                                         v
       |                     +-------------+              +------+------+
       |     data            |             |    data      |             |
       +---------------------+  Websocket  +<-------------+   MongoDB   |
                             |             |              |             |
                             +-------------+              +-------------+

Workflow

1. scratch quotes from scrapy offical demo site(quotes), save to database(MongoDB), and show them through web.
2. Front end is vue.js project, including WebSocket client(socketio) component.
3. Back end is websocket server, which query data form MongoDB through ORM mongoose every period(1s), pushing quotes data to front end.
4. data picker use scrapy to crawl quotes to save to MongoDB.

---

Technology Stack

Scrapy + webservice[restful] + websocket + VueJS + MongoDB

Component	Description
Scrapy	scratch data from web
webservice	serve restapi for data storation and fetch
websocket	push data to browser
VueJS	frontend
MongoDB	DB

---

Install

./bin/install.sh

Run

# run all services
./bin/start.sh

# stop all services
./bin/stop.sh

# start scrapy
./bin/start_scrapy.sh

Reference

• mongodb install