Technical architecture

SnowPlow has a very different architecture from conventional open-source web analytics packages such as Piwik piwik or [Open Web Analytics] owa. Where those packages are built on a tightly-coupled LAMP stack, SnowPlow has a loosely-coupled architecture which consists of five sub-systems:

![architecture] conceptual-architecture

To briefly explain these five sub-systems:

• Trackers fire SnowPlow events. Currently we have a JavaScript tracker; iOS and Android trackers are on the roadmap
• Collectors receive SnowPlow events from trackers. Currently we have a simple CDN-based collector on [Amazon CloudFront] cloudfront
• ETL (extract, transform and load) cleans up the raw SnowPlow events, enriches them and puts them into storage. Currently we have an ETL process using [Apache Hive] hive
• Storage is where the SnowPlow events live. Currently we store the SnowPlow events in an immutable Hive-format flatfile structure on S3
• Analytics are performed on the SnowPlow events. Currently we have a set of ad hoc analyses written in Hive

In the rest of this page we explain our rationale for this architecture, map out the specific technical components and finally flag up the strengths and limitations of this architecture.

Rationale for architecture

SnowPlow's distinctive architecture has been informed by a set of key design principles:

1. Extreme scalability - SnowPlow should be able to scale to tracking billions of customer events without affecting the performance of your client (e.g. website) or making it difficult to subsequently analyse all of those events
2. Permanent event history - SnowPlow events should be stored in a simple, non-relational, immutable data store
3. Direct access to individual events - you should have direct access to your raw SnowPlow event data at the atomic level
4. Separation of concerns - event tracking and event analysis should be two separate systems, only loosely-coupled
5. Support any analysis - SnowPlow should make it easy for business analysts, data scientists and engineers to answer any business question they want, using as wide a range of analytical tools as possible

Architectural diagram

The current technical architecture for SnowPlow looks like this:

![SnowPlow Technical Architecture] tech-architecture

This architecture diagram will be updated shortly with the new ETL control tool, written in Ruby.

Technical strengths

The SnowPlow approach has several technical advantages over more conventional web analytics approaches. In no particular order, these advantages are:

• Scalable, fast tracking - using CloudFront for event tracking reduces complexity and minimizes client slowdown worldwide
• Never lose your raw data - your raw event data is never compacted, overwritten or otherwise corrupted by SnowPlow
• Direct access to events - not intermediated by a third-party vendor, or a slow API, or an interface offering aggregates only
• Analysis tool agnostic - SnowPlow can be used to feed whatever analytics process you want (e.g. Hive, R, Pig, Sky EQL)
• Integrable with other data sources - join SnowPlow data into your other data sources (e.g. ecommerce, CRM) at the event level
• Clean separation of tracking and analysis - new analyses will not require re-tagging of your site or app

For a wider discussion of the strengths of SnowPlow from a business or product perspective, please see the Features and benefits page.

Technical limitations

The current SnowPlow architecture, tightly coupled as it is to Amazon CloudFront and S3, has some specific limitations to consider:

• Not real-time - CloudFront takes 20-60 minutes to collate logs from its edge nodes, so real-time analytics are not feasible
• Cannot track users across multiple domains - the user tracking cookie is a first-party cookie set by SnowPlow.js, so the user will be assigned a different ID on each domain
• Data payload limited by querystring length - SnowPlow data is logged via a GET querystring - which of course could potentially hit the de facto [2000 character] 2000char URL length limit

For more information on these limitations, please see the Technical FAQ.