versie 0.9, een begin

README.md

@@ -0,0 +1,15 @@
+# SCART Documentation
+
+## Installation
+
+DOCS are based on the https://www.mkdocs.org/ tool. This is a simple setup with
+.MD files.
+
+Linux VM:
+- install mkdocs
+- put repro in map (eg /var/scartdocs/)
+- go /var/scartdocs/
+- run run.sh
+- open in browser http://localhost:8000 
+
+

docs/basic/classification.md

@@ -0,0 +1,92 @@
+# Classification
+
+---
+
+Within the classify SCARt function reports can be classified. The function 
+starts with a list of reports which can be selected and opened by the analyst.
+
+In this list you can see if a report already is locked by an analyst. And you 
+can select one or more reports to work on.
+
+## Classify
+
+**Note: the CLASSIFY function is the most "important" user screen with a lot of options 
+to control everything. Advicable is to use this function on a large (bigger) screen.** 
+
+When you open a report you find the following screen:
+
+![img_5.png](../images/img_5.png)
+
+You find the following general controls:
+
+| Name                                            | Description                                          |
+|:------------------------------------------------|:-----------------------------------------------------|
+| ![img_6.png](../images/img_6.png) ![img_7.png](../images/img_7.png) | Switch between list and grid view                    
+| Records...                                      | Number of records to display                         |
+| Sort...                                         | Sort on                                              |
+| Hide...                                         | Hide records or not                                  |
+| Select...                                       | Select records based on status and current selection |
+| Bulk...                                         | Do an action on the selected records                 | 
+| ![img_8.png](../images/img_8.png)                                         | Regresh the screen                                   |
+
+Special controls:
+
+| Name                                            | Description                                                        |
+|:------------------------------------------------|:-------------------------------------------------------------------|
+| Rules...| Rules maintenance for the domains of the current records |
+| Done| Used to finalize everything (done)                                 | 
+
+Work area:
+
+In this area you have two different views; the LIST and the GRID view. 
+
+In the GRID view you can specify how many columns are on the screen and you can scroll 
+to the bottom till all the images are showed. 
+
+In the LIST view you see only a group of records (5, 10, 25, or 50) and you can page 
+through all the records. This view is handy when there are al lot of records (>100) 
+so the display of the classify screen is very buys with loading the images.
+
+In each view the following buttons for each record are supported:
+
+| Button                                  | Action             | Description |
+|:----------------------------------------|:-------------------|:-|
+| ![img_11.png](../images/img_11.png) | Set ILLEGAL        |Illegal classify|
+| ![img_10.png](../images/img_10.png) | Set NOT ILLEGAL    |Not illegal classify|
+| ![img_12.png](../images/img_12.png)               | Set IGNORE         |Ignore (eg icon)|
+| ![img_13.png](../images/img_13.png)               | Set FIRST POLICE   |Send to police and wait|
+| ![img_14.png](../images/img_14.png)               | Set MANUAL         |Manual check if online|
+| ![img_15.png](../images/img_15.png)               | Edit record fields |Edit different fields| 
+
+
+The FIRST POLICE is only possible when the record is classified as ILLEGAL. The 
+abusecontact with POLICE marked (on) will be informed by email with all the 
+records (urls) marked with FIRST POLICE.
+
+Set MANUAL is also only possible when the record is ILLEGAL. When this is set 
+SCARt will  automatically only check the WhoIs information and not if the 
+url (image) is online. In the function CHECKONLINE records can be set 
+offline. 
+
+With EDIT a number of fields can be updated:
+
+![img_16.png](../images/img_16.png)
+
+## Rules
+
+With rules the flow and (eg) hosting setting of records can be overruled. You can 
+set the hoster or site owner based on the domain or set a proxy service (like 
+Cloudfest) for determining the real IP. 
+
+The RULES function is available within the classify function with only the options
+(domains) valid for the records and as general function with all the possiblities.
+
+See [Rules](../details/rules.md) for more information.
+
+
+
+
+
+
+
+

docs/basic/import.md

@@ -0,0 +1,94 @@
+# Import 
+
+---
+
+SCARt has several ways for importing reports:
+
+- Manual input
+- ICCAM (Inhope)
+- By uploading 
+- By email
+
+
+
+## Manual input
+
+In the INPUTS function you can use CREATE to manual add an input. Required is a
+valid url, workuser, source and type. Default this input will set on the  scrape
+status so the input (URL) will automatically picked up by the SCARt background
+process to analyze and scrape.
+
+## ICCAM
+
+SCARt supports ICCAM API version v2 and v3. 
+
+The main difference With version v3 and v2 is that in v3 the synchronization is done 
+on a much directer one-to-one level between SCARt and ICCAM.
+
+System admin has to configure:
+- ICCAM certificate (one time each year)
+- ICCAM API user and password
+
+## By upload
+
+In the SCARt functionm INPUTS you can import a CSV file with on every line:
+
+`
+URL;REFERER;WORKUSER-EMAIL;REFERENCE;SOURCE;TYPE
+`
+
+Note: URL is **required**, the other fields are optional.
+
+## By email
+
+SCARt can read a mailbox to receive webform input. The admin has to configure 
+the mailbox account. After this, all email in this mailbox will automatically 
+process by SCART.
+
+Note that for protecting reasons, a whitelist access policy is setup. Within SCARt 
+the admin can put sender email address on the whitelist (SETTINGS -> WHITE LIST). 
+Is a sender from an email not on this list then the import email will not be accepted.
+
+### Email import of reports
+
+You can send an email with as subject "SCART-INPUT" to the mailbox. Each line can 
+hold:
+
+`
+URL;REFERER;NOTE
+`
+
+Note: 
+
+1. URL is **required**, the other fields are optional.
+2. The system admin can set if the status is set on OPEN or SCRAPE
+
+
+### Email import with source specification 
+
+You can send an email with as subject "SCART-INPUT-SOURCE [source]" to the mailbox 
+with "[source]" replaced with the source you want to be set. if the source is not 
+found in SCARt, then the source is automatically added.
+
+The body format is the same as above (including the note).
+
+### Email import Content Removed
+
+You can send an email with as subject "SCART-CONTENTREMOVED" to the mailbox. Each 
+line can hold the url of the report in SCARt to be set on CLOSE, including 
+sending to ICCAM the CR action.
+
+### Email import Content Unavailable 
+
+You can send an email with as subject "SCART-CONTENTUNAVAILABLE" to the mailbox. Each
+line can hold the url of the report in SCARt to be set on CLOSE, including
+sending to ICCAM the CU action.
+
+### Custom webform email import
+
+A special function is made for importing special webform data, including importing 
+data without an url. 
+
+See [custom webforms](../details/custom_webforms.md) for more information.
+
+

docs/basic/ntd.md

@@ -0,0 +1,35 @@
+# Notice & Take Down
+
+---
+
+Within the NTD function you can find the Notice & Take Down (NTD) messages send 
+and which are waiting te be send.
+
+## Sending NTD's
+
+There are a number of moments when SCARt starts a NTD:
+
+- after classify and when the hoster of illegal material is local (country)
+- after 24 hours when the illegal content is still online
+- when marked as POLICE in classify (ntd to police contact)
+- when send LEA in POLICE (ntd to LEA contact)
+
+After "starting" a NTD, SCARt will group urls for the samen abusecontact until the 
+hour-threshold is reached. 
+
+Default hour-thresholds:
+
+- FIRST NTD hoster; 1 hour
+- FIRST POLICE; 1 hour
+- SEND LEA; 1 hour
+- NEXT NTD hoster; 24 hour
+
+Note: these thresholds can be set invidual for each SCARt environment.
+
+Before actual sending a NTD, SCARt will last minute check for each attached url 
+if the hoster is still the same. If not, the url will be removed from the NTD.
+
+## NTD email template 
+
+
+

docs/basic/report.md

@@ -0,0 +1,27 @@
+# Reports
+
+---
+
+In SCARt a basic reporting module is include which export the data into a CSV
+(comma seperated values) file.
+
+You can create a report in SCARt. After this, the report will be run in the background 
+because of the possible time needed to generate the report. After finishing, an email 
+is send to the report notify email address and the report can be downloaded when you 
+open the report.
+
+## Columns
+
+On the COLUMNS tab you can specify which columns should be included in the export 
+and also in which order.
+
+## Export format
+
+The export format is CSV with the field content enclosed in quotes like:
+
+"field 1";"field 2";.."Field n";
+
+This file can directly be opened by spreadsheet programmas like Microsoft Excel or  
+Libreoffice Calc. 
+
+

docs/details/contacts.md

@@ -0,0 +1,44 @@
+# Abuse contacts
+
+---
+
+Abuse contacts in SCARt are used for:
+
+- hoster contact information
+- registrar contact information
+- site owner contact information
+- police contact information
+- LEA contact information
+
+## Creation 
+
+Contact records are automatically created by SCARt when a contact is not already found 
+based on:
+
+- name
+- aliases
+- abuse email address
+
+When a contact is created by SCARt, The GDPR approved is NOT set and reports with this 
+contact will be placed in the CHANGED status. 
+
+First an analyst has to check the contact information such as email address, country and 
+GDPR approved. if checked, he or she can set these fields so this abuse contact can be 
+used.
+
+## Specific NTD email template
+
+An example of an abuse contact detail screen:
+![img.png](img.png)
+
+For each contact a specific NTD (LEA) email template can be set. You can use the same 
+syntax for the body as specified in [NTD email templates](../basic/ntd.md). 
+
+## API linker check
+
+Some hosters supports an API to support automatically pushing illegal URLS so these are 
+directly removed by there services.
+
+Contact your SCARt provider for adding a specific API for a hoster to SCARt. 
+
+