IntelMQ is a solution for IT security teams (CERTs, CSIRTs, abuse departments,…) for collecting and processing security feeds (such as log files) using a message queuing protocol. It’s a community driven initiative called IHAP (Incident Handling Automation Project) which was conceptually designed by European CERTs/CSIRTs during several InfoSec events. Its main goal is to give to incident responders an easy way to collect & process threat intelligence thus improving the incident handling processes of CERTs.
IntelMQ’s design was influenced by AbuseHelper, however it was re-written from scratch and aims at:
+ Reduce the complexity of system administration
+ Reduce the complexity of writing new bots for new data feeds
+ Reduce the probability of events lost in all process with persistence functionality (even system crash)
+ Use and improve the existing Data Harmonization Ontology
+ Use JSON format for all messages
+ Integration of the existing tools (AbuseHelper, CIF)
+ Provide easy way to store data into Log Collectors like ElasticSearch, Splunk, databases (such as PostgreSQL)
+ Provide easy way to create your own black-lists
+ Provide easy communication with other systems via HTTP RESTFUL API
intelmqdump
It follows the following basic meta-guidelines:
+ Don’t break simplicity – KISS
+ Keep it open source – forever
+ Strive for perfection while keeping a deadline
+ Reduce complexity/avoid feature bloat
+ Embrace unit testing
+ Code readability: test with unexperienced programmers
+ Communicate clearly
Changelog intelmq v1.0.0.dev6:
### General changes
– Dropped support for Python 2, Python >= 3.3 is needed
– Dropped startup.conf and system.conf. Sections in BOTS can be copied directly yo runtime.conf now.
### Bot changes
– ENH: added bots.collectors.rt.collector
– ENH: added bots.parsers.spamhaus.parser_cert
– ENH: added bots.parsers.fraunhofer.parser_dga
– ENH: added bots.experts.certat_contact.expert
– MAINT: renamed bots.parsers.spamhaus.parser to bots.parsers.spamhaus.parser_drop
– Dropped dragon research group feeds: discontinued
– changed configuration syntax for bots.experts.modify
– dropped bots.collectors.bitsight.collector in favor of bots.collectors.http.collector_http_stream
### Bug fixes
– FIX: all bots handle message which are None
– FIX: various encoding issues resolved in core and bots
– FIX: time.observation is generated in collectors, not in parsers
### Other enhancements and changes
– TST: testing framework for core and tests. Newly introduced components should always come with proper unit tests.
– ENH: intelmqctl has shortcut parameters and can clear queues
– STY: code obeys PEP8, new code should always be properly formatted
– ENH: More code is Python 3 compatible
– DOC: Updated user and dev guide
– Removed Message.contains, Message.update methods Message.add ignore parameter
###Configuration
– ENH: New parameter and field named accuracy to represent the accuracy of each feed
– Consistent naming “overwrite” to switch overwriting capabilities of bots (as opposed to override)
– Renamed http_ssl_proxy to https_proxy
– deduplicator bot has a new parameter to configure deduplication mode
– deduplicator bot key ignore_keys was renamed to filter_keys
### Harmonization
– ENH: Additional data types: integer, float and Boolean
– ENH: Added descriptions and matching types to all fields
– DOC: harmonization documentation has same fields as configuration, docs are generated from configuration
– ENH: New type LowercaseString and UppercaseString
– BUG: FQDNs are only allowed in IDN representation
– ENH: New fields feed.documentation and feed.provider
#### Most important changes:
– (source|destination).bgp_prefix is now (source|destination).network
– (source|destination).cc is now (source|destination).geolocation.cc
– (source|destination).reverse_domain_name is (source|destination).reverse_dns
– misp_id changed to misp_uuid
– protocol.transport added
– webshot_url removed
– additional_information renamed to extra, must be JSON
– os.name, os.version, user_agent removed in favor of extra
Install from source:
CentOS 7 yum install epel-release yum install python34 python34-devel yum install git gcc gcc-c++ yum install redis Ubuntu 14.04 / Debian 8 apt-get install python3 python3-pip apt-get install git build-essential libffi-dev apt-get install python3-dev apt-get install redis-server git clone https://github.com/certtools/intelmq.git /tmp/intelmq cd /tmp/intelmq sudo -s pip3 install -r REQUIREMENTS pip3 install . useradd -d /opt/intelmq -U -s /bin/bash intelmq chmod -R 0770 /opt/intelmq chown -R intelmq.intelmq /opt/intelmq Upgrade: cd intelmq/ git pull pip install -U intelmq
Source: https://github.com/certtools