See the changelog for a full list of changes.
The documentation is now available at intelmq.readthedocs.io (FIXME: Link to stable version?).
IntelMQ no longer supports Python 3.5 (and thus Debian 9 and Ubuntu 16.04), the minimum supported Python version is 3.6.
CentOS 7 (with EPEL) provides both Python 3.4 and Python 3.6. If IntelMQ was installed with Python 3.4, the code needs to be re-installed with Python 3.6 and removed for Python 3.4. Application data is compatible. To install the Python 3.6 packages, use: yum install python36 python36-devel python36-requests
.
-
Bots that require a database file (such as
maxmind_geoip
,asn_lookup
,tor_nodes
andrecordedfuture_iprisk
) have new command line option--update-database
. It is not necessary to specify a bot ID, the function automatically updates the database for all the bots of the same type and reloads them afterwards. Removes any external dependencies (such as curl or wget). This is a replacement for shell scripts such asupdate-tor-nodes
,update-asn-data
,update-geoip-data
,update-rfiprisk-data
.Usage:
intelmq.bots.experts.asn_lookup.expert --update-database intelmq.bots.experts.maxmind_geoip.expert --update-database intelmq.bots.experts.recordedfuture_iprisk.expert --update-database intelmq.bots.experts.tor_nodes.expert --update-database
The provided shell scripts use these new commands, however they are now deprecated and will be removed in version 3.0.
Both the XMPP output bot and the XMPP collector bot are deprecated. The bots need to be migrate to another XMPP library (see #1614 for details) and a survey on the mailing listed revealed no users. If you depend on this bot, please reach out to us via the mailing list or GitHub. The bots are logging a deprecation warning now and the current plan is to remove them in IntelMQ version 3.0.
The feed template for the URLHaus feed contained a spelling error:
The correct name for the parameter "delimeter" is "delimiter". Please fix your configured bots.
The intelmqctl upgrade-config
command automatically fixes a configuration if the misspelling is detected.
There was a spelling error in the Spamhaus CERT parser's "event_description.text" texts. The following statements optionally update existing data. Please check if you did use these feed names and eventually adapt them for your setup!
UPDATE events
SET "event_description.text" = 'The malicious client used a honeypot as proxy.'
WHERE "event_description.text" = 'The malicous client used a honeypot as proxy.' AND "classification.taxonomy" = 'other' AND "classification.type" = 'other' AND "classification.identifier" = 'proxyget' AND "feed.name" = 'Spamhaus CERT';
UPDATE events
SET "event_description.text" = 'The infected iot device logged in to a honeypot and issued malicious commands.'
WHERE "event_description.text" = 'The infected iot device logged in to a honeypot and issued malicous commands.' AND "classification.taxonomy" = 'intrusions' AND "classification.type" = 'unauthorized-command' AND "classification.identifier" = 'iot' AND "feed.name" = 'Spamhaus CERT';
A bug in the taxonomy expert did set the Taxonomy for the type scanning
to information gathering
whereas for the type sniffing
and social-engineering
, the taxonomy was correctly set to information-gathering
.
This inconsistency for the taxonomy information-gathering
is now fixed, but the data eventually needs to fixed in data output (databases) as well.
There are still some inconsistencies in the naming of the classification taxonomies and types, more fixes will come in version 3.0.0. See issue #1409.
The following statements optionally update existing data. Please check if you did use these feed names and eventually adapt them for your setup!
UPDATE events
SET "classification.taxonomy" = 'information-gathering'
WHERE "classification.taxonomy" = 'information gathering';
The cache key calculation has been fixed. It previously led to duplicate keys for different IP addresses and therefore wrong results in rare cases. The cache key calculation is intentionally not backwards-compatible. Therefore, this bot may take longer processing events than usual after applying this update. More details can be found in issue #1592.
The feed "Blacklisted-IP" has been renamed by ShadowServer to "Blocklist". In IntelMQ, the old name can still be used in IntelMQ until version 3.0.
The current python library versions of geoip (version 4) and maxminddb (version 2) no longer support Python 3.5. Keep older versions of these libraries if you are using this Python version.
The current documented value for the column
parameter was:
['time.source', 'source.url', 'status', 'extra.urlhaus.threat_type', 'source.fqdn', 'source.ip', 'source.asn', 'source.geolocation.cc']
Better is:
['time.source', 'source.url', 'status', 'classification.type|__IGNORE__', 'source.fqdn|__IGNORE__', 'source.ip', 'source.asn', 'source.geolocation.cc']
- IntelMQ no longer supports Python 3.4, Python
>=
3.5 is required. CentOS 7 (with EPEL) provides both Python 3.4 and Python 3.6. If IntelMQ was installed with Python 3.4, the code needs to be re-installed with Python 3.6 and removed for Python 3.4. Application data is compatible. To install needed packages:yum install python36 python36-devel python36-requests
. - The AMQP collector requires the
pika
library minimum version 1.0.
The ElasticSearch Output bot does no longer support (only) ElasticSearch version 5, but only version 7 (#1513).
The Bot has been majorly changed to support the current Azure Python library azure-storage-blob>=12.0.0
.
This also changes the required configuration parameters. The new required parameters are:
redis_cache_db
: 5redis_cache_host
:"127.0.0.1"
redis_cache_password
: null, depending on your Redis server configurationredis_cache_port
: 6379, depending on your Redis server configurationredis_cache_ttl
: 864000 (10 days), depending on how old fast the data you are fetching is removed from the storageconnection_string
: connection string as given by Microsoft, includes endpoint and authentication informationcontainer_name
: name of the container to connect to
The previous parameters account_name
, account_key
and delete
are not supported anymore.
All URLVir feeds have been discontinued. The URLVir Parser has been removed.
The intelmqctl upgrade-config
command warns if you have these feed and the bot in use.
The python library requests
is required by the core.
The regular expression of the field protocol.transport
has been updated to accommodate the value nvp-ii
.
intelmqctl upgrade-config
handles the change to automatically upgrade your configuration.
The Taichung feed "Netflow (Recent 30)" with URL https://www.tc.edu.tw/net/netflow/lkout/recent/30
is no longer available and gives an error 500.
As a drop-in replacement the Parser as well as the Feed documentation are now adapted to the full feed available at https://www.tc.edu.tw/net/netflow/lkout/recent/
.
The intelmqctl upgrade-config
command takes care of this change.
The Abuse.ch Zeus Tracker has been discontinued on 2019-07-08. The intelmqctl upgrade-config
command warns if you have this feed in use.
The Abuse.ch Ransomware Tracker has been discontinued on 2019-12-08. The intelmqctl upgrade-config
command warns if you have this feed in use.
The Bitcash.cz Banned IPs / Blocklist feed previously available under https://bitcash.cz/misc/log/blacklist
is no longer available. The intelmqctl upgrade-config
command warns if you have this feed in use.
The Fraunhofer DDoS Attack feed previously available under https://feed.caad.fkie.fraunhofer.de/ddosattackfeed/
is no longer available. The intelmqctl upgrade-config
command warns if you have this feed in use.
Many Bambenek feeds require a license now and URLs have changed. See https://osint.bambenekconsulting.com/feeds/ for more information. The intelmqctl upgrade-config
command also warns if you have previously documents feeds in use and migrates the URL for the DGA domain feed.
All Nothink Honeypot feeds have been discontinued and current the data available covers the time until 2019. The Nothink Parser has been removed.
The intelmqctl upgrade-config
command warns if you have these feed and the bot in use.
MaxMind requires a registration before being able to download the GeoLite2 database starting with 2019-12-30: https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/
If the provided update-geoip-data
script is used, the license key can now be set second parameter.
When using MISP integrations, make sure your currently installed PyMISP version supports the installed Python version. Any PyMISP version newer than 2.4.119.1 requires Python 3.6 or newer.
intelmqctl check
and intelmqctl upgrade-config
checks/applies a generic harmonization upgrade, checking for it's completeness.
In version 2.1.0 a new field extra
has been added to Reports. You need to add this to the report section in your harmonization configuration, or run intelmqctl upgrade-config
:
"extra": {
"description": "All anecdotal information of the report, which cannot be parsed into the data harmonization elements. E.g. subject of mails, etc. This is data is not automatically propagated to the events.",
"type": "JSONDict"
},
For AMQP brokers, the port 15671 was used as default for connecting to the management interface. But RabbitMQ's default is 15672. This was corrected in this version. If you changed the port in RabbitMQ or IntelMQ (intelmqctl_rabbitmq_monitoring_url
), the settings needs to be adapted.
Run intelmqctl upgrade-config
and intelmqctl check
after the upgrade.
The Shadowserver Parser is now able to detect the feed base on the report's field extra.file_name
, added by collectors. Have a look at their documentation for more details.
The PostgreSQL Output Bot has been integrated into the new and generic SQL Output Bot
- module name:
- old:
intelmq.bots.outputs.postgresql.output
- new:
intelmq.bots.outputs.sql.output
- old:
- parameters:
- new:
engine
=postgresql
IntelMQ versions 2.x will be compatible with previous configurations.intelmqctl upgrade-config
migrates configurations.
- new:
Run intelmqctl upgrade-config
and intelmqctl check
after the upgrade.
The deprecated parameter feed
for collectors is again supported as the documentation as not properly updated. The support will be removed before version 2.2.
In the upgrade function for version 1.1.0 (in effect in version 2.0.1) the addition of the parameter query_ripe_stat_ip
was not correctly done and is maybe missing. A new upgrade function re-adds it with the value of query_ripe_stat_ip
.
The Cymru CAP Feed is (being) migrated to a new URL with a different format and more data. Look at the feed's documentation for more information.
These bots overwrite existing fields by default. A parameter overwrite
has been added to make this optional, with the default value of True
for backwards compatibility. If the parameter is not set, a warning is logged. The default value will change to False
in version 3.0.0. The default for all new bots (in BOTS
) is False
already.
intelmqctl has a new function intelmqctl upgrade-config
to upgrade the configuration from previous installations. It is recommended to call this function after every upgrade.
See also the news for 2.0.0.beta1 below.
The allowed values for the classification.type
field have been updated to the RSIT mapping. These values have changed and are automatically mapped:
botnet drone
withinfected-system
infected system
withinfected-system
ids alert
withids-alert
c&c
withc2server
malware configuration
withmalware-configuration
Four new values have been introduced to configure the statistics database. Add them to your defaults.conf
file:
statistics_database
:3
,statistics_host
:"127.0.0.1"
,statistics_password
:null
,statistics_port
:6379
,
Version 1.1.2 broke the compatibility of the TCP Output with third-party counterparts like filebeat, but is more stable for a TCP Collector counterpart. A new parameter counterpart_is_intelmq
has been introduced, it's default is false
for backwards compatibility. If you use a TCP collector, set this to true
, otherwise to false
.
The following statements optionally update existing data. Please check if you did use these feed names and eventually adapt them for your setup!
UPDATE events
SET "classification.type" = 'infected-system'
WHERE "classification.type" = 'botnet drone';
UPDATE events
SET "classification.type" = 'infected-system'
WHERE "classification.type" = 'infected system';
UPDATE events
SET "classification.type" = 'ids-alert'
WHERE "classification.type" = 'ids alert';
UPDATE events
SET "classification.type" = 'c2server'
WHERE "classification.type" = 'c&c';
UPDATE events
SET "classification.type" = 'malware-configuration'
WHERE "classification.type" = 'malware configuration';
There are some features considered as beta and marked as such in the documentation, do not use them in production yet.
The bot intelmq.bots.experts.ripencc_abuse_contact.expert
has been renamed to intelmq.bots.experts.ripe.expert
, the compatibility shim will be removed in version 3.0. Adapt your runtime.conf
accordingly.
- The URL of the "Feodo Tracker IPs" feed has changed. The new one is
https://feodotracker.abuse.ch/downloads/ipblocklist.csv
. If you are using this feed, adapt your configuration accordingly. The parser has been updated to support the new format. - The feed "Feodo Tracker Domains" has been discontinued.
In 1.1.0 the default value for the parameter error_dump_message
was set to false
. The recommended value, used in previous and future release is true
to not loose any data in case of errors. Users are advised to check the values configured in their defaults.conf
file.
The following statements optionally update existing data. Please check if you did use these feed names and eventually adapt them for your setup!
UPDATE events
SET "classification.taxonomy" = 'abusive content', "classification.type" = 'spam', "classification.identifier" = 'spam', "malware.name" = NULL, "source.fqdn" = "source.reverse_dns", "source.reverse_dns" = NULL, "source.url" = "destination.url", "destination.url" = NULL
WHERE "malware.name" = 'spam' AND "feed.name" = 'Drone';
In the section for 1.1.0 there was this command:
UPDATE events
SET "classification.identifier" = 'open-portmapper',
"protocol.application" = 'portmap'
WHERE "classification.identifier" = 'openportmapper' AND "feed.name" = 'Open-Portmapper' AND "protocol.application" = 'portmapper';
protocol.application
was incorrect. To fix it you can use:
UPDATE events
SET "protocol.application" = 'portmapper'
WHERE "classification.identifier" = 'open-portmapper' AND "feed.name" = 'Open-Portmapper' AND "protocol.application" = 'portmap';
In previous version the MongoDB Output Bot saved the fields time.observation
and time.source
as strings in ISO format. But MongoDB does support saving datetime objects directly which are converted to its native date format, enabling certain optimizations and features. The MongoDB Output Bot now saves these values as datetime objects.
- Python 3.4 or newer is required.
intelmqctl start
prints bot's error messages in stderr if it failed to start.intelmqctl check
checks if all keys in the packaged defaults.conf are present in the current configuration.
The malware name rules of the modify expert have been migrated to the Malware Name Mapping repository.
See contrib/malware_name_mapping/
for download and conversion scripts as well as documentation.
The classification type for malware has been changed from "botnet drone" to the more generic "infected system". The classification identifiers have been harmonized too:
old identifier | new identifier |
---|---|
openmdns | open-mdns |
openchargen | open-chargen |
opentftp | open-tftp |
openredis | open-redis |
openportmapper | open-portmapper |
openipmi | open-ipmi |
openqotd | open-qotd |
openssdp | open-ssdp |
opensnmp | open-snmp |
openmssql | open-mssql |
openmongodb | open-mongodb |
opennetbios | open-netbios-nameservice |
openelasticsearch | open-elasticsearch |
opendns | dns-open-resolver |
openntp | ntp-monitor |
SSL-FREAK | ssl-freak |
SSL-Poodle | ssl-poodle |
openmemcached | open-memcached |
openxdmcp | open-xdmcp |
opennatpmp | open-natpmp |
opennetis | open-netis |
openntpversion | ntp-version |
sandboxurl | sandbox-url |
spamurl | spam-url |
openike | open-ike |
openrdp | open-rdp |
opensmb | open-smb |
openldap | open-ldap |
blacklisted | blacklisted-ip |
opentelnet | open-telnet |
opencwmp | open-cwmp |
accessiblevnc | open-vnc |
In the section Postgres databases you can find SQL statements for these changes.
Some feed names have changed, see the comment below in the section Configuration.
You may want to update your harmonization configuration
- Newly added fields:
destination.urlpath
andsource.urlpath
.destination.domain_suffix
andsource.domain_suffix
.tlp
with a new type TLP.
- Changed fields:
- ASN fields now have a new type
ASN
.
- ASN fields now have a new type
- Classification:
- New value for
classification.type
:vulnerable client
with taxonomyvulnerable
. - New value for
classification.type
:infected system
with taxonomymalicious code
as replacement forbotnet drone
.
- New value for
- Renamed
JSON
toJSONDict
and added a new typeJSON
.JSONDict
saves data internally as JSON, but acts like a dictionary.JSON
accepts any valid JSON.
Some bots depend on the three new harmonization fields.
A new harmonization type JSONDict
has been added specifically for the extra
field. It is highly recommended to change the type of this field. The change is backwards compatible and the change is not yet necessary, IntelMQ 1.x.x works with the old configuration too.
The feed names in the shadowserver parser have been adapted to the current subjects. Old subjects will still work in IntelMQ 1.x.x. Change your configuration accordingly:
Botnet-Drone-Hadoop
toDrone
DNS-open-resolvers
toDNS-Open-Resolvers
Open-NetBIOS
toOpen-NetBIOS-Nameservice
Ssl-Freak-Scan
toSSL-FREAK-Vulnerable-Servers
Ssl-Scan
toSSL-POODLE-Vulnerable-Servers
The Maxmind GeoIP expert did previously always overwrite existing data. A new parameter overwrite
has been added,
which is by default set to false
to be consistent with other bots.
The bot bots.collectors.n6.collector_stomp
has been renamed to the new module bots.collectors.stomp.collector
. Adapt your runtime.conf
accordingly.
The parameter feed
for collectors has been renamed to name
, as it results in feed.name
. Backwards compatibility is ensured until 2.0.
The following statements optionally update existing data. Please check if you did use these feed names and eventually adapt them for your setup!
ALTER TABLE events
ADD COLUMN "destination.urlpath" text,
ADD COLUMN "source.urlpath" text;
ALTER TABLE events
ADD COLUMN "destination.domain_suffix" text,
ADD COLUMN "source.domain_suffix" text;
ALTER TABLE events
ADD COLUMN "tlp" text;
UPDATE events
SET "classification.type" = 'infected system'
WHERE "classification.type" = 'botnet drone';
UPDATE events
SET "classification.identifier" = 'open-mdns'
WHERE "classification.identifier" = 'openmdns' AND "feed.name" = 'Open-mDNS';
UPDATE events
SET "classification.identifier" = 'open-chargen'
WHERE "classification.identifier" = 'openchargen' AND "feed.name" = 'Open-Chargen';
UPDATE events
SET "classification.identifier" = 'open-tftp'
WHERE "classification.identifier" = 'opentftp' AND "feed.name" = 'Open-TFTP';
UPDATE events
SET "classification.identifier" = 'open-redis'
WHERE "classification.identifier" = 'openredis' AND "feed.name" = 'Open-Redis';
UPDATE events
SET "classification.identifier" = 'open-ipmi'
WHERE "classification.identifier" = 'openipmi' AND "feed.name" = 'Open-IPMI';
UPDATE events
SET "classification.identifier" = 'open-qotd'
WHERE "classification.identifier" = 'openqotd' AND "feed.name" = 'Open-QOTD';
UPDATE events
SET "classification.identifier" = 'open-snmp'
WHERE "classification.identifier" = 'opensnmp' AND "feed.name" = 'Open-SNMP';
UPDATE events
SET "classification.identifier" = 'open-mssql'
WHERE "classification.identifier" = 'openmssql' AND "feed.name" = 'Open-MSSQL';
UPDATE events
SET "classification.identifier" = 'open-mongodb'
WHERE "classification.identifier" = 'openmongodb' AND "feed.name" = 'Open-MongoDB';
UPDATE events
SET "classification.identifier" = 'open-netbios-nameservice', "feed.name" = 'Open-NetBIOS-Nameservice'
WHERE "classification.identifier" = 'opennetbios' AND "feed.name" = 'Open-NetBIOS';
UPDATE events
SET "classification.identifier" = 'open-elasticsearch'
WHERE "classification.identifier" = 'openelasticsearch' AND "feed.name" = 'Open-Elasticsearch';
UPDATE events
SET "classification.identifier" = 'dns-open-resolver', "feed.name" = 'DNS-Open-Resolvers'
WHERE "classification.identifier" = 'opendns' AND "feed.name" = 'DNS-open-resolvers';
UPDATE events
SET "classification.identifier" = 'ntp-monitor'
WHERE "classification.identifier" = 'openntp' AND "feed.name" = 'NTP-Monitor';
UPDATE events
SET "classification.identifier" = 'ssl-poodle', "feed.name" = 'SSL-POODLE-Vulnerable-Servers'
WHERE "classification.identifier" = 'SSL-Poodle' AND "feed.name" = 'Ssl-Scan';
UPDATE events
SET "classification.identifier" = 'ssl-freak', "feed.name" = 'SSL-FREAK-Vulnerable-Servers'
WHERE "classification.identifier" = 'SSL-FREAK' AND "feed.name" = 'Ssl-Freak-Scan';
UPDATE events
SET "classification.identifier" = 'open-memcached'
WHERE "classification.identifier" = 'openmemcached' AND "feed.name" = 'Open-Memcached';
UPDATE events
SET "classification.identifier" = 'open-xdmcp'
WHERE "classification.identifier" = 'openxdmcp' AND "feed.name" = 'Open-XDMCP';
UPDATE events
SET "classification.identifier" = 'open-natpmp', "protocol.application" = 'natpmp'
WHERE "classification.identifier" = 'opennatpmp' AND "feed.name" = 'Open-NATPMP' AND "protocol.application" = 'nat-pmp';
UPDATE events
SET "classification.identifier" = 'open-netis'
WHERE "classification.identifier" = 'opennetis' AND "feed.name" = 'Open-Netis';
UPDATE events
SET "classification.identifier" = 'ntp-version'
WHERE "classification.identifier" = 'openntpversion' AND "feed.name" = 'NTP-Version';
UPDATE events
SET "classification.identifier" = 'sandbox-url'
WHERE "classification.identifier" = 'sandboxurl' AND "feed.name" = 'Sandbox-URL';
UPDATE events
SET "classification.identifier" = 'spam-url'
WHERE "classification.identifier" = 'spamurl' AND "feed.name" = 'Spam-URL';
UPDATE events
SET "classification.identifier" = 'open-ike'
WHERE "classification.identifier" = 'openike' AND "feed.name" = 'Vulnerable-ISAKMP';
UPDATE events
SET "classification.identifier" = 'open-rdp'
WHERE "classification.identifier" = 'openrdp' AND "feed.name" = 'Accessible-RDP';
UPDATE events
SET "classification.identifier" = 'open-smb'
WHERE "classification.identifier" = 'opensmb' AND "feed.name" = 'Accessible-SMB';
UPDATE events
SET "classification.identifier" = 'open-ldap'
WHERE "classification.identifier" = 'openldap' AND "feed.name" = 'Open-LDAP';
UPDATE events
SET "classification.identifier" = 'blacklisted-ip'
WHERE "classification.identifier" = 'blacklisted' AND "feed.name" = 'Blacklisted-IP';
UPDATE events
SET "classification.identifier" = 'open-telnet'
WHERE "classification.identifier" = 'opentelnet' AND "feed.name" = 'Accessible-Telnet';
UPDATE events
SET "classification.identifier" = 'open-cwmp'
WHERE "classification.identifier" = 'opencwmp' AND "feed.name" = 'Accessbile-CWMP';
UPDATE events
SET "classification.identifier" = 'open-vnc'
WHERE "classification.identifier" = 'accessiblevnc' AND "feed.name" = 'Accessible-VNC';
- Some optional dependencies do not support Python 3.3 anymore. If your are still using this unsupported version consider upgrading. IntelMQ 1.0.x itself is compatible with Python 3.3.
Use the following statement carefully to upgrade your database. Adapt your feedname in the query to the one used in your setup.
UPDATE events
SET "classification.taxonomy" = 'abusive content', "classification.type" = 'spam', "classification.identifier" = 'spamlink', "malware.name" = NULL, "event_description.text" = 'The URL appeared in a spam email sent by extra.spam_ip.', "source.url" = "destination.ip", "destination.ip" = NULL
WHERE "malware.name" = 'l_spamlink' AND "feed.name" = 'Spamhaus CERT';
UPDATE events
SET "classification.taxonomy" = 'other', "classification.type" = 'other', "classification.identifier" = 'proxyget', "malware.name" = NULL, "event_description.text" = 'The malicious client used a honeypot as proxy.'
WHERE "malware.name" = 'proxyget' AND "feed.name" = 'Spamhaus CERT';
Use the following statement carefully to upgrade your database. Adapt your feedname in the query to the one used in your setup.
UPDATE events
SET "extra" = json_build_object('source.local_port', "extra"->'destination.local_port')
WHERE "feed.name" = 'Spamhaus CERT' AND "classification.type" = 'brute-force' AND "classification.identifier" = 'telnet';
Use the following statement carefully to upgrade your database. Adapt your feedname in the query to the one used in your setup.
UPDATE events
SET "classification.taxonomy" = 'intrusion attempts', "classification.type" = 'brute-force', "classification.identifier" = 'rdp', "protocol.application" = 'rdp', "malware.name" = NULL
WHERE "malware.name" = 'iotrdp' AND "feed.name" = 'Spamhaus CERT';
UPDATE events
SET "classification.taxonomy" = 'vulnerable', "classification.type" = 'vulnerable service', "classification.identifier" = 'openrelay', "protocol.application" = 'smtp', "malware.name" = NULL
WHERE "malware.name" = 'openrelay' AND "feed.name" = 'Spamhaus CERT';
UPDATE events
SET "protocol.application" = 'portmapper'
WHERE "classification.identifier" = 'openportmapper' AND "feed.name" = 'Open-Portmapper';
UPDATE events
SET "protocol.application" = 'netbios-nameservice'
WHERE "classification.identifier" = 'opennetbios' AND "feed.name" = 'Open-NetBIOS-Nameservice';
UPDATE events
SET "protocol.application" = 'ipsec'
WHERE "classification.identifier" = 'openike' AND "feed.name" = 'Vulnerable-ISAKMP';
UPDATE events
SET "classification.taxonomy" = 'intrusion attempts', "classification.type" = 'brute-force', "classification.identifier" = 'ssh', "malware.name" = NULL, "protocol.application" = 'ssh'
WHERE "malware.name" = 'sshauth' AND "feed.name" = 'Spamhaus CERT';
UPDATE events
SET "classification.taxonomy" = 'intrusion attempts', "classification.type" = 'brute-force', "classification.identifier" = 'telnet', "malware.name" = NULL, "protocol.application" = 'telnet'
WHERE ("malware.name" = 'telnetauth' OR "malware.name" = 'iotcmd' OR "malware.name" = 'iotuser') AND "feed.name" = 'Spamhaus CERT';
UPDATE events
SET "classification.taxonomy" = 'information gathering', "classification.type" = 'scanner', "classification.identifier" = 'wordpress-vulnerabilities', "malware.name" = NULL, "event_description.text" = 'scanning for wordpress vulnerabilities', "protocol.application" = 'http'
WHERE "malware.name" = 'wpscanner' AND "feed.name" = 'Spamhaus CERT';
UPDATE events
SET "classification.taxonomy" = 'information gathering', "classification.type" = 'scanner', "classification.identifier" = 'wordpress-login', "malware.name" = NULL, "event_description.text" = 'scanning for wordpress login pages', "protocol.application" = 'http'
WHERE "malware.name" = 'w_wplogin' AND "feed.name" = 'Spamhaus CERT';
UPDATE events
SET "classification.taxonomy" = 'intrusion attempts', "classification.type" = 'scanner', "classification.identifier" = 'scanner-generic', "malware.name" = NULL, "event_description.text" = 'infected IoT device scanning for other vulnerable IoT devices'
WHERE "malware.name" = 'iotscan' AND "feed.name" = 'Spamhaus CERT';
bots.parsers.cleanmx
removed CSV format support and now only supports XML format. Therefore, CleanMX collectors must define thehttp_url
parameter with the feed url which points to XML format. See Feeds.md file on documentation section to get the correct URLs. Also, downloading the data from CleanMX feed can take a while, therefore, CleanMX collectors must overwrite thehttp_timeout_sec
parameter with the value120
.- The classification mappings for the n6 parser have been corrected:
n6 classification | Previous classification | Current classification | Notes | ||||
---|---|---|---|---|---|---|---|
taxonomy | type | identifier | taxonomy | type | identifier | ||
dns-query | other | other | ignore me | other | other | dns-query | |
proxy | vulnerable | proxy | open proxy | other | proxy | openproxy | |
sandbox-url | ignore | ignore | ignore me | malicious code | malware | sandboxurl | As this previous taxonomy did not exist, these events have been rejected |
other | vulnerable | unknown | unknown | other | other | other |
Use the following statement carefully to upgrade your database. Adapt your feedname in the query to the one used in your setup.
UPDATE events
SET "classification.identifier" = 'dns-query'
WHERE "feed.name" = 'n6' AND "classification.taxonomy" = 'other' AND "classification.type" = 'other' AND "classification.identifier" = 'ignore me';
UPDATE events
SET "classification.taxonomy" = 'malicious code' AND "classification.type" = 'malware' AND "classification.identifier" = 'sandboxurl'
WHERE "feed.name" = 'n6' AND "classification.taxonomy" = 'vulnerable' AND "classification.type" = 'ignore' AND "classification.identifier" = 'ignore me';
UPDATE events
SET "classification.taxonomy" = 'other' AND "classification.type" = 'other' AND "classification.identifier" = 'other'
WHERE "feed.name" = 'n6' AND "classification.taxonomy" = 'vulnerable' AND "classification.type" = 'unknow' AND "classification.identifier" = 'unknow';
No changes needed.
No changes needed.
bots.experts.ripencc_abuse_contact
now has the two additional parametersquery_ripe_stat_asn
andquery_ripe_stat_ip
instead ofquery_ripe_stat
. The old parameter will be supported until version 1.1. An additional parametermode
has been introduced. See the bot's documentation for more details: docs/Bots.md#ripencc-abuse-contactbots.experts.certat_contact
has been renamed tobots.experts.national_cert_contact_certat
(#995)bots.collectors.ftp
has been dropped (unused, unmaintained, #842)- system.conf and startup.conf have been dropped entirely, use defaults.conf and runtime.conf instead
- Many bots have new/changed parameters
- Many bots have been renamed/moved or deleted. Please read the Bots section in the changelog and upgrade your configuration accordingly.
http_timeout
has been renamed tohttp_timeout_sec
andhttp_timeout_max_tries
has been added.
Two new fields have been added to defaults.conf
which are expected by the bots:
"log_processed_messages_count": 500
and'log_processed_messages_seconds": 900
Configure them in your setup and optionally adapt the values to your needs.
Use the following statement carefully to upgrade your database.
ALTER TABLE events
ADD COLUMN "output" json
-
The deduplicator expert requires a new parameter
filter_type
, the old previous default wasblacklist
. The keyignore_keys
has been renamed tofilter_keys
. -
The tor_nodes expert has a new parameter
overwrite
, which is by defaultfalse
. -
The configuration format of the modify expert has been change to a list-based syntax. Old format:
{ "Blocklist.de": { "__default": [{ "feed.name": "^BlockList\\.de$", "classification.identifier": "" }, { }] }, ... }
new format:
[ { "rulename": "Blocklist.de __default", "if": { "classification.identifier": "", "feed.name": "^BlockList\\.de$" }, "then": {} }, ... ]
The built-in Alienvault OTX API library has been removed, install the library from github instead. See the README.md for details.
Use the following statement carefully to upgrade your database. Take care that no data will be lost, the statement may not be complete!
Also note that size constraints have changed!
ALTER TABLE events
ADD COLUMN "feed.documentation" text;
UPDATE events
SET "source.local_hostname"="destination.local_hostname",
"destination.local_hostname"=DEFAULT
WHERE "feed.name"='Open-LDAP' AND "source.local_hostname" IS NULL;
UPDATE events
SET "feed.url" = substring("feed.url" from 1 for 37)
WHERE SUBSTRING("feed.url" from 1 for 38) = 'https://prod.cyberfeed.net/stream?key='
UPDATE events
SET "feed.url" = regexp_replace("feed.url", 'receipt=([^&])*', '')
WHERE substring("feed.url" from 1 for 43) = 'https://lists.malwarepatrol.net/cgi/getfile'
UPDATE events
SET "feed.url" = substring("feed.url" from 1 for 36)
WHERE SUBSTRING("feed.url" from 1 for 37) = 'https://data.phishtank.com/data/'
UPDATE events
SET "classification.taxonomy" = lower("classification.taxonomy")
WHERE "classification.taxonomy" IS NOT NULL;
ALTER TABLE events
ADD COLUMN "feed.provider" text
Syntax of runtime.conf has changed
ALTER TABLE events
ADD COLUMN "misp.attribute_uuid" varchar(36),
ADD COLUMN "malware.hash.sha256" text,
ALTER COLUMN "misp.event_uuid" SET DATA TYPE varchar(36);
ALTER TABLE events RENAME COLUMN "misp_uuid" TO "misp.event_uuid";
UPDATE events
SET "protocol.application" = lower("protocol.application")
WHERE "protocol.application" IS NOT NULL;
UPDATE events
SET "source.abuse_contact" = lower("source.abuse_contact")
WHERE "source.abuse_contact" IS NOT NULL;
UPDATE events
SET "destination.abuse_contact" = lower("destination.abuse_contact")
WHERE "destination.abuse_contact" IS NOT NULL;
UPDATE events
SET "event_hash" = lower("event_hash")
WHERE "event_hash" IS NOT NULL;
UPDATE events
SET "malware.hash.md5" = lower("malware.hash.md5");
UPDATE events
SET "malware.hash.sha1" = lower("malware.hash.sha1");
UPDATE events
SET "malware.hash.sha256" = lower("malware.hash.sha256");
UPDATE events
SET "malware.hash.md5" = lower(substring("malware.hash" from 4))
WHERE substring("malware.hash" from 1 for 3) = '$1$';
UPDATE events
SET "malware.hash.sha1" = lower(substring("malware.hash" from 7))
WHERE substring("malware.hash" from 1 for 6) = '$sha1$';
UPDATE events
SET "malware.hash.sha256" = lower(substring("malware.hash" from 4))
WHERE substring("malware.hash" from 1 for 3) = '$5$';
UPDATE events
SET "malware.hash.md5" = lower("malware.hash.md5")
WHERE "malware.hash.md5" IS NOT NULL;
UPDATE events
SET "malware.hash.sha1" = lower("malware.hash.sha1")
WHERE "malware.hash.sha1" IS NOT NULL;
ALTER TABLE events
ADD COLUMN "classification.identifier" text,
ADD COLUMN "feed.accuracy" text,
ADD COLUMN "feed.code" text,
ADD COLUMN "malware.hash.md5" text,
ADD COLUMN "malware.hash.sha1" text,
ADD COLUMN "protocol.transport" text,
ALTER COLUMN "extra" SET DATA TYPE json,
RENAME COLUMN "additional_information" TO "extra",
RENAME COLUMN "description.target" TO "event_description.target",
RENAME COLUMN "description.text" TO "event_description.text",
RENAME COLUMN "destination.bgp_prefix" TO "destination.network" text,
RENAME COLUMN "destination.cc" TO "destination.geolocation.cc" text,
RENAME COLUMN "destination.email_address" TO "destination.account" text,
RENAME COLUMN "destination.reverse_domain_name" TO "destination.reverse_dns" text,
RENAME COLUMN "misp_id" TO "misp_uuid",
RENAME COLUMN "source.bgp_prefix" TO "source.network" text,
RENAME COLUMN "source.cc" TO "source.geolocation.cc" text,
RENAME COLUMN "source.email_address" TO "source.account" text,
RENAME COLUMN "source.reverse_domain_name" TO "source.reverse_dns" text,
RENAME COLUMN "webshot_url" TO "screenshot_url" text;
UPDATE events
SET "extra"=json_build_object('os.name', "os.name", 'os.version', "os.version", 'user_agent', "user_agent")
WHERE "os.name" IS NOT NULL AND "os.version" IS NOT NULL AND "user_agent" IS NOT NULL AND "extra" IS NULL;
ALTER TABLE events
DROP COLUMN "os.name",
DROP COLUMN "os.version",
DROP COLUMN "user_agent",
DROP COLUMN "malware.hash";