Using OSINT to create CDB lists and block malicious IPs

Today’s post is about using Open Source Intelligence, OSINT, and CDB lists. Let’s start with the latter. If you have been using Wazuh for a while, you, for instance, might be receiving some alerts related to legitime users and might be wondering how to ignore them. A perfect example for CDB lists. Another one and this is the case we are going to focus on, you might have a reliable blacklist containing malicious IPs that you want to block. Both are good scenarios for using CDB lists. What are they? They are a list of values that are checked against a particular field extracted by a decoder. What we need to do is to create a custom rule that checks if the IP, the user or any other field extracted, are included in the list. Depending on the result, we can decide whether to generate an alert or not.

To show how they work, we are going to block all IPs that Wazuh finds in our CDB list: blacklist-ip. To create this list we will use OSINT, this term is used to refer to the data collected from publicly available sources to be used in an intelligence context.

Getting our blacklist

First of all, we need to create our blacklist. To do it we are using FIREHOL, a service provided by OSINT. It analyses security IP Feeds, mainly related to online attacks, on-line service abuse, malware, botnets and other cybercrime activities. It has several lists. In our case we are going to choose the alienvault_reputation list.

The format of an item in a CDB list is key:value and allows the following subnets: 32, 24, 16 and 8. Visit our documentation for further details. First step will be to convert the blacklist format into the CDB list format. We can create a python script, shown below, to do the work. As a result, the key will be the IP, the value will be empty and empty lines or unsupported subnets will be removed.

#!/usr/bin/env python
#
# Convert IP list to CDB list
# Copyright (C) 2016 Wazuh Inc.
#
# This program is a free software; you can redistribute it
# and/or modify it under the terms of the GNU General Public
# License (version 2) as published by the FSF - Free Software
# Foundation.
#
import re
from sys import exit, argv
try:
    if len(argv) != 3:
        print("Bad arguments. Try: iplist-to-cdblist.py input output")
        exit(1)
    ip_regex = re.compile("^((?:[0-9]{1,3}\.){3}[0-9]{1,3})(?:/(\d{1,2}){0,1}|)")
    first_time = True
    cdir_conversion = {"32": 4, "24": 3, "16": 2, "8": 1}
    fo = open(argv[2], 'w')
    with open(argv[1]) as f:
        for line in f:
            match = ip_regex.match(line.rstrip('\r\n'))
            if not match:  # Read just lines that start with an IP
                continue
            ip = match.group(1)
            mask = match.group(2)
            if mask:  # Convert allowed masks (32, 24, 16, 8)
                ip = ip.split('.')
                if mask in cdir_conversion:
                    ip = '.'.join(ip[:cdir_conversion[mask]])
                    if mask != "32":
                        ip += "."
                else:
                    continue
            ip += ":"  # CDB List format
            if first_time:
                fo.write(ip)
                first_time = False
            else:
                fo.write("\n" + ip)
    fo.close()
    print("[{0}] -> [{1}]".format(argv[1], argv[2]))
except Exception as e:
    print("Error:\n{0}\nExiting".format(e))
    exit(1)

Configuring the CDB lists

Let’s assume that we are using the default directory for CDB lists: /var/ossec/lists.
Now, it’s time to download the blacklist and use the python script to convert it into a CDB list format:

sudo wget https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/alienvault_reputation.ipset -O /var/ossec/etc/lists/alienvault_reputation.ipset
sudo wget https://wazuh.com/resources/iplist-to-cdblist.py -O /var/ossec/etc/lists/iplist-to-cdblist.py
sudo chmod +x /var/ossec/etc/lists/iplist-to-cdblist.py
sudo /var/ossec/etc/lists/iplist-to-cdblist.py /var/ossec/etc/lists/alienvault_reputation.ipset /var/ossec/etc/lists/blacklist-alienvault
sudo rm -f /var/ossec/etc/lists/alienvault_reputation.ipset

The script will generate the list blacklist-alienvault. Now, we need to add it to our configuration as follows:

...
<ruleset>
  <list>etc/lists/blacklist-alienvault</list>
...

Wazuh needs to compile the lists before using them, so remember to compile your list every time you make a change:

sudo /var/ossec/bin/ossec-makelists
* File etc/lists/blacklist-alienvault.cdb needs to be updated

After compiling the list the file blacklist-alienvault.cdb will be generated.

Using the CDB lists

We can start using our list in custom rules. In this case, we have created a rule that will fire an alert when the “group” is web, attack or attacks, and the IP is in our blacklist. Remember that the IP must be extracted as srcip by the decoder. Below is the custom rule:

<group name="attack,">
    <rule id="100100" level="10">
      <if_group>web|attack|attacks</if_group>
      <list field="srcip" lookup="address_match_key">etc/lists/blacklist-alienvault</list>
      <description>IP in black list.</description>
    </rule>
</group>

To apply changes, we need to restart OSSEC:

$ sudo /var/ossec/bin/ossec-control restart

Time to test our new rule using an event with an IP in the blacklist and with group web, for example 223.229.189.193:

223.229.189.193 - - [09/Jun/2017:11:17:03 +0000] "POST /command.php HTTP/1.0" 404 464 "-" "Wget(linux)"
**Phase 1: Completed pre-decoding.
       full event: '223.229.189.193 - - [09/Jun/2017:11:17:03 +0000] "POST /command.php HTTP/1.0" 404 464 "-" "Wget(linux)"'
       hostname: 'ip-10-0-0-10'
       program_name: '(null)'
       log: '223.229.189.193 - - [09/Jun/2017:11:17:03 +0000] "POST /command.php HTTP/1.0" 404 464 "-" "Wget(linux)"'
**Phase 2: Completed decoding.
       decoder: 'web-accesslog'
       srcip: '223.229.189.193'
       url: '/command.php'
       id: '404'
**Phase 3: Completed filtering (rules).
       Rule id: '100100'
       Level: '10'
       Description: 'IP in black list.'
**Alert to be generated.

Blocking IPs

In a previous post we explained how to block attacks using active response. Just follow the same steps but this time use the id of the custom rule we have just created, 100100:

<active-response>
    <command>firewall-drop</command>
    <location>local</location>
    <rules_id>100100</rules_id>
    <timeout>1800</timeout>
</active-response>

Conclusion

This is just an example that shows the potential of CDB lists. There are many other use cases where these lists are useful, for instance, to create your own black/white lists of users, IPs, URLs, processes or, in other words, any field that can be extracted by decoders. If you want to see another example of CDB lists, I suggest taking a look to a previous post titled: Monitoring USB devices in Windows using Wazuh.

If you have any questions about this, join our Slack #community channel! Our team and other contributors will help you.