This guided tutorial will introduce HSC Software Engineering to the basics of developing websites with the node.JS framework. The tutorial has been specifically designed for requirements in the NESA Software Engineering Syllabus and students in NSW Department of Education schools using eT4L computers.
A list of popular PWA's (including Ube, Spotify, Facebook and Google Maps)
A Progressive Web Apps (PWAs) is an app that is built using web platform technologies, but that provides a user experience like that of a platform-specific app. Like a website, a PWA can run on multiple platforms and devices from a single codebase. Like a platform-specific app, it can be installed on the device, can operate while offline and in the background, and can integrate with the device and with other installed apps.
Because PWAs are websites, they have the same basic features as any other website: at least one HTML page, which loads CSS and JavaScript. Javascript is the language of the web and is exclusively used for the client-side front end; python, in the web context, can only be used in the back end. Like a normal website, the JavaScript loaded by the page has a global Window object and can access all the Web APIs that are available through that object. The PWA standard as defined by W3C Standards has some specific features additional to a website:
Feature | Purpose |
---|---|
manifest.json | An app manifest file, which, at a minimum, provides information that the operating system needs to install the PWA, such as the app name, screen orientation and icon set for different-sized viewports. |
serviceworker.js | A service worker, which, at a minimum, manages the caching that enables an online and offline experience whilst also interfacing with API's such as the notification web API. It's important to understand that this JS file cannot control the DOM of the application for security reasons. |
Icons & screenshots | A set of icons and screenshots that are used when uploading to an app store and when installing it as a native application. It is these icons that will be used in the desktop or app launcher when installed. |
Installable | Because of the information contained in the manifest.json all PWA's can be installed like a native app. They can also be packaged and uploaded to the Google, Microsoft & Apple app stores. |
Cached locally | Because the service worker details all apps and pages to be cached (all pages must have a *.html name), the app and its resources can be cached locally for quick load times. Note backend apps where the web server serves all pages from the DNS root do not meet the PWA specification. |
The below image illustrates how the servicework manages online and offline behaviour.
This screen capture shows how the final PWA will be rendered to the user.
Important
MacOS and Linux users may have a pip3
soft link instead of pip
, run the below commands to see what path your system is configured with and use that command through the project. If neither command returns a version, then likely Python 3.x needs to be installed.
pip show pip
pip3 show pip
- Bash basics & using the GIT Bash shell in VSCode
- SQL
- HTML Basics
- CSS Basics
- Python
Note
Helpful VSCode settings are configured in .vscode/settings.json, which will automatically apply if you are not using a custom profile. If you are using a custom profile, it is suggested you manually apply those settings to your profile, especially the *.md file association, so the README.md default opens in preview mode and setting bash as your default terminal.
- Install the necessary extensions for this tutorial.
Required Extensions | Suggested nodeJS Extensions |
---|---|
McCarter.start-git-bash | ecmel.vscode-html-css |
yy0931.vscode-sqlite3-editor | ms-vscode.js-debug |
medo64.render-crlf | esbenp.prettier-vscode* |
oderwat.indent-rainbow |
*You will need to configure esbenp.prettier-vscode as your default formatter
- Open a GIT BASH terminal
Note
From now on, you should aim to run all commands from the CLI. You are discouraged from left/right clicking the GUI. You will find it feels slow at first, but through disciplined use, you will become much quicker and more accurate with CLI commands than GUI controls.
Make sure you open a new terminal with the keys Ctrl + ` and choose Git Bash from the menu option in the top right of the terminal shell.
-
Get the working files, which include this README.md
- Open a new window in VSCode
- Choose your working directory
git clone https://github.com/TempeHS/NodeJS_PWA_Programming_For_The_Web_Task_Template.git
cd NodeJS_PWA_Programming_For_The_Web_Task_Template
Tip
Alternatively, you can fork the template repository to your own GitHub account and open it in a Codespace in which all dependencies and extensions will be automatically installed.
- Inititalise a node application
npm init -y
- Install necessary dependencies.
npm install sqlite3
npm install express
- Make a folder for all your working documents like photoshop *.psd files, developer documentation etc.
mkdir working_documents
- Create a license file.
touch LICENSE
code LICENSE
Copy the GNU GPL license text into the file. GNU GPL is a free software license, or copyleft license, that guarantees end users the freedom to run, study, share, and modify the software.
- Create your directory structure and some base files using BASH scripts reading text files.
├── database
├── working_documents
├── public
│ ├── css
│ ├── icons
│ ├── images
│ ├── js
│ ├── index.html
│ ├── about.html
│ ├── manifest.json
│ ├── serviceworker.js
├── LICENSE
└── index.js
- Create a text file with a list of folders you need in the public folder of your project. The web server will serve the contents of the public folder. This folder is the 'FRONT END,' while all folders behind it are the 'BACK END.'
mkdir public
cd public
touch folders.txt
code folder.txt
- Run a BASH script to read the text file and create the folders listed in it.
while read -r line; do
echo $line
mkdir -p $line
done < folders.txt
- Populate the file with a list of files you need at the root of your project.
touch files.txt
code files.txt
- Run a BASH script to read the text file and create the files listed in it.
while read -r line; do
echo $line
touch -p $line
done < files.txt\
Important
—The last list item needs a line ending, so make sure the last line in the file is blank.
- You will find that all file and folder names have an unwanted
space
character at the end. This is because you are using a BASH emulator on the Windows operating system. Bash is a Unix language that uses LF Unicode character 000A while Windows uses CRLF Unicode characters 000D and 000A. Because you have installed the medo64.render-crlf extension, click onCRLF
in the bottom bar of VSCode and chooseLF
to change the line ending before running your BASH script.
cd ..
mkdir database
cd database
touch data_source.db
Note
The following SQL queries are provided as an example only. Students are encouraged to select their content and design a database schema for it; ideas include:
- Favourite bands
- Favourite movies
- Favourite games
- Favourite books
- etc
- To run SQLite3 SQL queries in VSCode Open the DB file, then choose "Query Editor" from the top menu.
code data_source.db
CREATE TABLE extension(extID INTEGER NOT NULL PRIMARY KEY,name TEXT NOT NULL, hyperlink TEXT NOT NULL,about TEXT NOT NULL,image TEXT NOT NULL,language TEXT NOT NULL);
- After running each query put
--
infront of the query to turn it into a comment so it doesn't run again and error. - Run SQL queries to populate your table.
INSERT INTO extension(extID,name,hyperlink,about,image,language) VALUES (1,"Live Server","https://marketplace.visualstudio.com/items?itemName=ritwickdey.LiveServer","Launch a development local Server with live reload feature for static & dynamic pages","https://ritwickdey.gallerycdn.vsassets.io/extensions/ritwickdey/liveserver/5.7.9/1661914858952/Microsoft.VisualStudio.Services.Icons.Default","HTML CSS JS");
INSERT INTO extension(extID,name,hyperlink,about,image,language) VALUES (2,"Render CR LF","https://marketplace.visualstudio.com/items?itemName=medo64.render-crlf","Displays the line ending symbol and optionally extra whitespace when 'Render whitespace' is turned on.","https://medo64.gallerycdn.vsassets.io/extensions/medo64/render-crlf/1.7.1/1689315206970/Microsoft.VisualStudio.Services.Icons.Default","#BASH");
INSERT INTO extension(extID,name,hyperlink,about,image,language) VALUES (3,"Start GIT BASH","https://marketplace.visualstudio.com/items?itemName=McCarter.start-git-bash","Adds a bash command to VSCode that allows you to start git-bash in the current workspace's root folder.","https://mccarter.gallerycdn.vsassets.io/extensions/mccarter/start-git-bash/1.2.1/1499505567572/Microsoft.VisualStudio.Services.Icons.Default","#BASH");
INSERT INTO extension(extID,name,hyperlink,about,image,language) VALUES (4,"SQLite3 Editor","https://marketplace.visualstudio.com/items?itemName=yy0931.vscode-sqlite3-editor","Edit SQLite3 files like you would in spreadsheet applications.","https://yy0931.gallerycdn.vsassets.io/extensions/yy0931/vscode-sqlite3-editor/1.0.85/1690893830873/Microsoft.VisualStudio.Services.Icons.Default","SQL");
- Run some SQL queries to test your database.
SELECT * FROM extension;
SELECT * FROM extension WHERE language LIKE '#BASH';
Note
Graphic design is not the focus of this course. It is suggested that you do not spend excessive time designing logos and icons.
- Use Photoshop or Canva to design a simple square logo 1080px X 1080px named logo.png. Save all working files (*.psd, pre-optimised originals, etc) into the working_documents directory.
- Design a simplified app icon 512px X 512px named favicon.png.
- Web optimise the images using TinyPNG.
- Save the files into the public/images folder.
- Rename the 512x512 icon to icon-512x512.png, then resize and rename it as follows:
- icon-128x128.png
- icon-192x192.png
- icon-384x384.png
- icon-512x512.png
- Web optimise the images using TinyPNG.
- Save the optimised icons to public/icons.
- Save the optimised logo and favicon to public/images.
Note
Adjust titles, headings and content to match your concept.
cd ../templates
touch layout.html
code layout.html
- Insert the basic HTML structure in your index.html file.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta http-equiv="X-UA-Compatible" content="ie=edge" />
<meta http-equiv="Content-Security-Policy" content="script-src 'self';" />
<link rel="stylesheet" href="css/style.css" />
<title>VSCode Extension Catalogue</title>
<link rel="manifest" href="manifest.json" />
<link rel="icon" type="image/x-icon" href="images/favicon.png" />
</head>
<body>
<main>
<!-- NAV START -->
<!-- NAV END -->
<div class="container"></div>
</main>
<script src="js/app.js"></script>
</body>
</html>
cd css
touch style.css
code style.css
- Insert the css code into css/style.css.
@import url("https://fonts.googleapis.com/css?family=Nunito:400,700&display=swap");
* {
margin: 0;
padding: 0;
box-sizing: border-box;
}
body {
background: #fdfdfd;
font-family: "Nunito", sans-serif;
font-size: 1rem;
}
main {
max-width: 900px;
margin: auto;
padding: 0.5rem;
text-align: center;
}
cd ..
code index.html
- Insert the menu HTML into index.html between the comment placeholders.
<nav>
<img src="images\logo.png" alt="VSCode Extensions site logo." />
<h1>VSCode Extensions</h1>
<ul class="topnav">
<li><a href="#">Home</a></li>
<li><a href="add.html">Add me</a></li>
<li><a href="about.html">About</a></li>
</ul>
</nav>
`cd ../../public/css`
`code style.css`
- Style the menu by inserting this below your existing CSS in public/css/style.css.
nav {
display: flex;
justify-content: space-between;
align-items: center;
}
nav img {
height: 100px;
}
nav ul {
list-style: none;
display: flex;
}
nav li {
margin-right: 1rem;
}
nav ul li a {
text-decoration-line: none;
text-transform: uppercase;
color: #393b45;
}
nav ul li a:hover {
color: #14e6dd;
}
nav h1 {
color: #106d69;
margin-bottom: 0.5rem;
}
Render your website
Express is a light weight webserver designed specifically for Node.js web applications. You have already installed it when you set up your environment.
cd ../..
code index.js
- Insert the node.js to the backend index.js.
// Insert additional backend js above the express server configuration
const express = require("express");
const path = require("path");
const app = express();
app.use(express.static(path.join(__dirname, "public")));
app.get("/", function (req, res) {
res.sendFile(path.join(__dirname, "public/index.html"));
});
app.listen(5000, () =>
console.log(
"Server is running on Port 5000, visit http://localhost:5000/ or http://127.0.0.1:5000 to access your website"
)
);
- Run the built-in webserver.
node index.js
- Visit your website and look at the source in developer tools to see how the page has been rendered.
Note
From here students have two choices, they can use their existing Python skills or new JS skills. Either way, students will be querying a table in data_source.db and then constructing a JSON file that will be pushed to the frontend, ready for rending by a frontend JS script. If you choose the JS method, you should refer to the Python method in the future as a helpful way to have more complex Python programs in the backend and create a simple responsive GUI using HTML/CSS/JS.
JSON (JavaScript Object Notation) is a lightweight data-interchange format. It is easy for humans to read and write. It is easy for machines to parse and generate. It is also very secure and the worflow used in this application ensures data integrity of the backend.
- Install the Python SQLite3 requirements
pip install sqlite3
touch database_manager.py
code database_manager.py
- Write the Python Script to query the SQL database and construct the JSON file.
import sqlite3 as sql
con = sql.connect("database/data_source.db")
cur = con.cursor()
data = cur.execute('SELECT * FROM extension').fetchall()
con.close()
f = open("public/frontEndData.json", "w")
f.write("[\n")
for row in data:
f.write('{\n')
f.write(f'"extID":{row[0]},\n"name":"{row[1]}",\n"hyperlink":"{row[2]}",\n"about":"{row[3]}",\n"image":"{row[4]}",\n"language":"{row[5]}"\n')
if row == data[len(data)-1]:
f.write("}\n")
else:
f.write("},\n")
f.write("]\n")
f.close()
[!Note] This approach is different from the Pythonic way to generate a JSON file. Because this approach is about algorithm design, it models how an algorithm can easily migrate data from one format/structure to another. If you know the Pythonic way, you can just implement it. However, software engineers should understand and be able to replicate data migration algorithms.
code index.js
- Insert this Javascript above the express setup in index.js to execute the Python program.
[!Important] This code must be above the Express server configuration and instantiation.
const spawn = require("child_process").spawn;
// you can add arguments with spawn('python',["path/to/script.py", arg1, arg2, ...])
const pythonProcess = spawn("python", ["database_manager.py"]);
- Test your application. The expected behaviour is a file called frontEndData.json filled with the table from data_source is saved in the /public folder. The JSON file should validate with jsonlint.
node index.js
code index.js
- Insert this Javascript above the express setup in index.js to query the SQL database and construct the JSON file.
[!Important] This code must be above the Express server configuration and instantiation.
const sqlite3 = require("sqlite3").verbose();
const db = new sqlite3.Database("database/data_source.db");
let myString = "[\n";
db.all("SELECT * FROM extension", function (err, rows) {
let myCounter = 0;
rows.forEach(function (row) {
// for debugging
// console.log(row.extID + ": " + row.name + ": " + row.hyperlink + ": " + row.about + ": " + row.image + ": " + row.language);
myString =
myString +
'{\n"extID":' +
row.extID +
',\n"name":"' +
row.name +
'",\n"hyperlink":"' +
row.hyperlink +
'",\n"about":"' +
row.about +
'",\n"image":"' +
row.image +
'",\n"language":"' +
row.language;
myCounter++;
if (myCounter == rows.length) {
myString = myString + '"\n}\n';
} else {
myString = myString + '"\n},\n';
}
});
// console.log(myString);
var fs = require("fs");
fs.writeFile("public/frontEndData.json", myString + "]", function (err) {
if (err) {
console.log(err);
}
});
});
- Test your application. The expected behaviour is a file called frontEndData.json filled with the table from data_source is saved in the /public folder. The JSON file should validate with jsonlint.
node index.js
cd public/js
touch app.js
code app.js
- Insert the js into public/js/app.js; this JS reads the JSON file and inserts it as HTML into the .container class
<DIV>
.
let result = "";
fetch("./frontEndData.json")
.then(function (response) {
return response.json();
})
.then(function (data) {
appendData(data);
})
.catch(function (err) {
console.log("error: " + err);
});
function appendData(data) {
data.forEach(({ name, image, hyperlink, about, language } = rows) => {
result += `
<div class="card">
<img class="card-image" src="${image}" alt="Product image for the ${name} VSCode extension."/>
<h1 class="card-name">${name}</h1>
<p class="card-about">${about}</p>
<a class="card-link" href="${hyperlink}"><button class="btn">Read More</button></a>
</div>
`;
});
document.querySelector(".container").innerHTML = result;
}
cd ../css
code style.css
- Style the cards by inserting this below your existing CSS in public/css/style.css.
.container {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(15rem, 1fr));
grid-gap: 1rem;
justify-content: center;
align-items: center;
margin: auto;
padding: 1rem 0;
}
.card {
display: flex;
align-items: center;
flex-direction: column;
width: 17rem;
background: #fff;
box-shadow: 0 10px 20px rgba(0, 0, 0, 0.19), 0 6px 6px rgba(0, 0, 0, 0.23);
border-radius: 10px;
margin: auto;
overflow: hidden;
}
.card-image {
width: 100%;
height: 15rem;
object-fit: cover;
}
.card-name {
color: #222;
font-weight: 700;
text-transform: capitalize;
font-size: 1.1rem;
margin-top: 0.5rem;
}
.card-about {
text-overflow: ellipsis;
width: 15rem;
white-space: nowrap;
overflow: hidden;
margin-bottom: 1rem;
}
.btn {
border: none;
background: none;
border-radius: 5px;
box-shadow: 1px 1px 2px rgba(21, 21, 21, 0.1);
cursor: pointer;
font-size: 1.25rem;
margin: 0 1rem;
padding: 0.25rem 2rem;
transition: all 0.25s ease-in-out;
background: hsl(110, 21%, 93%);
color: hsl(141, 100%, 22%);
margin-bottom: 1rem;
}
.btn:focus,
.btn:hover {
box-shadow: 1px 1px 2px rgba(21, 21, 21, 0.2);
background: hsl(111, 21%, 86%);
}
.about-container {
font-size: 1.25rem;
margin-top: 2rem;
text-align: justify;
text-justify: inter-word;
}
- Take a screenshot of the website. Then size the image to 1080px X 1920px, web optimise the images using TinyPNG and save it to public/icons.
cd ..
code manifest.json
- Configure the manifest.json to the PWA standard by inserting the JSON below and validating the JSON with jsonlint. The manifest.json sets the configuration for the installation and caching of the PWA.
{
"name": "VSCode Extension Catalogue",
"short_name": "vscodeextcat",
"start_url": "/",
"display": "standalone",
"background_color": "#fdfdfd",
"theme_color": "#14E6DD",
"orientation": "landscape-primary",
"icons": [
{
"src": "icons/icon-128x128.png",
"type": "image/png",
"sizes": "128x128",
"purpose": "maskable"
},
{
"src": "icons/icon-128x128.png",
"type": "image/png",
"sizes": "128x128",
"purpose": "any"
},
{
"src": "icons/icon-192x192.png",
"type": "image/png",
"sizes": "192x192",
"purpose": "maskable"
},
{
"src": "icons/icon-192x192.png",
"type": "image/png",
"sizes": "192x192",
"purpose": "any"
},
{
"src": "icons/icon-384x384.png",
"type": "image/png",
"sizes": "384x384",
"purpose": "maskable"
},
{
"src": "icons/icon-384x384.png",
"type": "image/png",
"sizes": "384x384",
"purpose": "any"
},
{
"src": "icons/icon-512x512.png",
"type": "image/png",
"sizes": "512x512",
"purpose": "maskable"
},
{
"src": "icons/icon-512x512.png",
"type": "image/png",
"sizes": "512x512",
"purpose": "any"
}
],
"screenshots": [
{
"src": "icons/desktop_screenshot.png",
"sizes": "1920x1080",
"type": "image/png",
"label": ""
},
{
"src": "icons/mobile_screenshot.png",
"sizes": "1080x1920",
"type": "image/png",
"form_factor": "wide",
"label": ""
}
]
}
cd js
code app.js
- Configure the app.js to initiate the servicework.js ny inserting the JS. This ensures that when the window (app) loads, the serviceworker.js is called to memory.
if ("serviceworker" in navigator) {
window.addEventListener("load", function () {
navigator.serviceworker
.register("js/serviceworker.js")
.then((res) => console.log("service worker registered"))
.catch((err) => console.log("service worker not registered", err));
});
}
cd js
code serviceworker.js
- Configure the serviceworker.js by inserting the JS. The serviceworker.js, as the name suggests, is the file that does all the work in a PWA, including caching and API integration for the WEB APIs.
const assets = [
"/",
"css/style.css",
"js/app.js",
"images/logo.png",
"images/favicon.png",
"icons/icon-128x128.png",
"icons/icon-192x192.png",
"icons/icon-384x384.png",
"icons/icon-512x512.png",
"icons/desktop_screenshot.png",
"icons/mobile_screenshot.png",
];
const CATALOGUE_ASSETS = "catalogue-assets";
self.addEventListener("install", (installEvt) => {
installEvt.waitUntil(
caches
.open(CATALOGUE_ASSETS)
.then((cache) => {
console.log(cache);
cache.addAll(assets);
})
.then(self.skipWaiting())
.catch((e) => {
console.log(e);
})
);
});
self.addEventListener("activate", function (evt) {
evt.waitUntil(
caches
.keys()
.then((keyList) => {
return Promise.all(
keyList.map((key) => {
if (key === CATALOGUE_ASSETS) {
console.log("Removed old cache from", key);
return caches.delete(key);
}
})
);
})
.then(() => self.clients.claim())
);
});
self.addEventListener("fetch", function (evt) {
evt.respondWith(
fetch(evt.request).catch(() => {
return caches.open(CATALOGUE_ASSETS).then((cache) => {
return cache.match(evt.request);
});
})
);
});
Validation is important to ensure the app is compliant to W3 web standards.
- Open your website in Chrome, open developer tools (F12), and run a Lighthouse report. .
- Open your website in Edge, open developer tools (F12), and look at the application report. .
The following code snippets will help you create a simple form on the add.html page. This form allows people to add their details to an email database for updates on your catalogue. Less explicit instructions have been provided; students are encouraged to practice their BASH, SQL, HTML, CSS, and JS to bring it all together. The screenshot below shows what the page should look like, and when users submit, the database is updated.
- Page specifications:
- Simple form where the user inserts their name and email address
- When they click submit the database is updated
- The input form must be styled to be consistent with the rest of the website
- A message confirming submission is returned to the user
- SQL schema specifications:
- A new table called contact_list
- 3 columns
- id is the primary key and should increment automatically
- email must be unique
- name
Note
You will need to catch the expectation of a duplicate email
npm npm install body-parser
let bodyParser = require("body-parser");
app.use(bodyParser.urlencoded({ extended: false }));
res.sendFile(path.join(__dirname, "public/add.html"));
app.post("/add.html", function (req, res) {
db.serialize(() => {
db.run(
"INSERT INTO contact_list(email,name) VALUES(?,?)",
[req.body.email, req.body.name],
function (err) {
if (err) {
return console.log(err.message);
}
res.send(
"Thank you " +
req.body.name +
" we have added your email " +
req.body.email +
" to our distribution list."
);
}
);
});
});
<form action="/app.html" method="POST" class="box">
<div>
<label class="form-label">Email address</label>
<input
name="email"
type="email"
class="form-control"
id="email"
pattern="[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2, 4}$"
placeholder="[email protected]"
/>
</div>
<div>
<label class="form-label">Name</label>
<textarea class="form-control" name="name" id="name" rows="1"></textarea>
</div>
<br />
<div>
<button type="submit" class="btn">Submit</button>
</div>
</form>
.form-control {
}
Node.js PWA Programming For The Web Task Source and Node.js PWA Programming For The Web Task Template by Ben Jones is licensed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International