Merge of botnet membership management communication attack and core ID2T.

Merge remote-tracking branch 'upstream/master' into merge_projects

.gitignore

@@ -34,3 +34,9 @@ resources/test/ID2T_results
 id2t
 run_tests
 test_efficiency
+
+# memory_profiler
+*.png
+*.dat
+
+.DS_Store

README.md

@@ -28,44 +28,82 @@ The following packages/libraries are required to compile the ID2T C++ modules
 * ``cmake`` (minimum version 2.8)
     - ubuntu: apt install build-essential cmake
     - arch: pacman -S cmake
+    - macos: brew install cmake
 * ``boost`` with the ``python`` component (minimum version 1.54)
     - ubuntu: apt install libboost-dev libboost-python-dev
     - arch: pacman -S boost boost-libs
+    - macos: brew install boost boost-python --with-python3
 * ``libtins`` (minimum version 3.4)
     - ubuntu: apt install libtins-dev (if you cannot find it in the official repository, install it manually from [here](https://github.com/mfontanini/libtins))
     - arch: (install from AUR, i.e. pacaur -S libtins, or manually from [here](https://github.com/mfontanini/libtins)).
+    - macos: brew install libtins
 * ``python`` development libraries
     - ubuntu: apt install python3-dev
-    - arch: pacman -S python
+    - arch: pacman -S python python-pip
+    - macos: brew install python
 * ``sqlite`` (minimum version 3.0)
     - ubuntu: apt install sqlite3
     - arch: pacman -S sqlite
+    - macos: brew install sqlite
+* ``tcpdump``
+    - ubuntu: apt install tcpdump
+    - arch: pacman -S tcpdump
+    - macos: brew install libdnet
+* ``coreutils`` (needed for greadlink)
+    - macos: brew install coreutils
 
 #### Required Python Packages
 The following python packages are required to run ID2T. Install the packages with your preferred package manager. For example, you can use pip3 (pip for python 3). Install pip3 in ubuntu with ``apt install python3-pip`` and install the packages with ``sudo pip3 install <packagename>``.
-* ``scapy`` (make sure its the python3 version)
+* ``pyxdg``
+* ``scapy-python3``
 * ``lea``
 * ``numpy``
 * ``matplotlib``
 * ``SciPy Stack`` (see [installation instructions](https://www.scipy.org/install.html))
+* ``coverage``
+* ``memory_profiler``
 
 #### Notes on the Minimum Package Versions
-The minimum version stated in the previous requirements are the versions we have used in the development of ID2T. Other (older) versions might also work; however, we cannot guarantee nor support them. Furthermore, some compilation scripts would need to be manually modified to accommodate these older versions.
+The minimum version stated in the previous requirements are the versions we used in the development of ID2T. Other (older) versions might also work; however, we can neither guarantee nor support them. Furthermore, some compilation scripts would need to be manually modified to accommodate these older versions.
 
+### Dependency installation script
+ID2T provides a dependency installation script, which is called during the execution of ``./build.sh``.
+
+#### Supported Systems
+* Linux Distributions
+    - Arch-based
+    - Debian-based
+* macOS
+
+##### Tested with
+* Arch Linux
+* Antergos
+* Kali
+* macOS (High) Sierra
+* Ubuntu (16.04, 17.10)
+* Zorin OS
 
 ### Compilation and Installation
-Once you satisfy all dependencies, clone the repository to get started with the installation:
+Clone the repository to get started with the installation:
 ``git clone https://git.tk.informatik.tu-darmstadt.de/SPIN/ID2T-toolkit``
 
-After cloning the repository, initialize its submodules with
+Install dependencies, initialize submodules, build the C++ modules and create the ID2T executables:
+``./build.sh``
+
+Or initialize its submodules manually:
     git submodule init
     git submodule update
 
-Build the C++ modules and create the ID2T executable:
-``./build.sh``
+To skip dependency installation use the ``--non-interactive`` argument:
+``./build.sh --non-interactive``
 
 Run ID2T with the command ``./id2t``.
 
+Run unit tests with the command ``./run_tests``.
+
+Run efficiency tests with the command ``./test_efficiency``.
+
+
 ## Usage examples
 In this section, we provide examples on how ID2T is used.
 
@@ -150,8 +188,17 @@ There are also parameterizable selectors which take conditions as input. Followi
 	-> returns the MAC address matching the given criteria
 	Supports the field: ipAddress
 
-Parameterizable selectors also allow for specifying another query in the comparison instead of a specific value, like the following example demonstrates:
-	macAddress(ipAddress=most_used(ipAddress))
+Parameterizable selectors also allow for specifying another query in the condition instead of a specific value, like the following example demonstrates:
+	macAddress(ipAddress in most_used(ipAddress))
+
+Conditions inside parameterizable selectors can contain all the usual comparison operators (<, <=, =, >=, >) when the right side of the condition is a single value. If the right side is a list, such as the return value of e.g. most_used(), the `` in ``-operator is to be used instead, unless the list is reduced to a single value by the use of an extractor.
+
+The following examples provide a demonstration of how lists can be used inside parameterizable selectors:
+```
+macAddress(ipAddress in ipAddress(pktssent > 1))         -> Returns the MAC addresses of all IP addresses that sent more than one packet
+macAddress(ipAddress = random(ipAddress(pktssent > 1)))  -> Returns the MAC address of a random IP address out of all IP addresses that sent more than one packet
+macAddress(ipAddress in [192.168.189.1,192.168.189.143]) -> Returns the MAC address of all IP addresses in the provided list
+```
 
 __Extractors__ are to be used on the result of a named query. If the result is a list, applying an extractor reduces the result set to a single element. If the result is already a single element, the extractor is ignored.
 ```
@@ -175,6 +222,8 @@ The [SemVer](http://semver.org/spec/v2.0.0.html) is used for versioning. For cur
 
 - __Carlos Garcia__ - _ID2T idea, guidance and suggestions during development_
 
+- __Aidmar Wainakh__ - _analysis and development of attacks as part of his Master Thesis_
+
 - __Leon Böck__ - _guidance and suggestions during development of Membership Management Communication Attack_
 
 - __Nikolay Milanov__ - _development of first prototype within his Master Thesis_
@@ -191,6 +240,16 @@ The [SemVer](http://semver.org/spec/v2.0.0.html) is used for versioning. For cur
 
 - __Denis Waßmann__ - _development of Membership Management Communication Attack_
 
+- __Stefano Acquaviti__ - _development of multiple attacks and general improvements_
+
+- __Jens Keim__ - _development of multiple attacks and general improvements_
+
+- __Roey Regev__ - _development of multiple attacks and general improvements_
+
+- __Stefan Schmidt__ - _development of multiple attacks and general improvements_
+
+- __Jonathan Speth__ - _development of multiple attacks and general improvements_
+
 ## License
 
 Distributed under the MIT license. See [LICENSE](LICENSE.md) for more information.

build.sh

@@ -14,13 +14,33 @@ else
     echo "Error: The 'build' directory was not found."
     exit
 fi
-cmake ..
 
-if [ -f Makefile ]; then
-    make
+which ninja &>/dev/null
+if [ $? != 0 ]; then
+    cmake ..
+
+    # Make sure we're able to get the number of cores
+    if [ $(uname) = 'Darwin' ]; then
+        NUMCORES=$(sysctl -n hw.logicalcpu)
+    else
+        NUMCORES=$(nproc)
+    fi
+
+    if [ -f Makefile ]; then
+        make -j$NUMCORES
+    else
+        echo "Error: 'cmake' did not finish successfully."
+        exit
+    fi
 else
-    echo "Error: 'cmake' did not finish successfully."
-    exit
+    cmake .. -G Ninja
+
+    if [ -f build.ninja ]; then
+        ninja
+    else
+        echo "Error: 'cmake' did not finish successfully."
+        exit
+    fi
 fi
 
 if [ $? -eq 0 ]; then
@@ -56,16 +76,23 @@ if [ $(uname) = 'Darwin' ]; then
 fi
 ID2T_DIR=\$(readlink -f \$0)
 SCRIPT_PATH=\${ID2T_DIR%/*}
-cd \$SCRIPT_PATH/code
+cd \$SCRIPT_PATH
+# Regenerate the statistics DB
+./id2t -i resources/test/reference_1998.pcap -r >/dev/null
+cd code
 # Execute tests
 set -e
+PRINT_COV=true
 testpath="discover -s Test/"
 if [ -e "Test/test_\$1.py" ]; then
     testpath="Test/test_\$1.py"
+    PRINT_COV=false
 fi
 PYTHONWARNINGS="ignore" coverage3 run --source=. -m unittest \$testpath >/dev/null
-coverage3 html
-coverage3 report -m
+if \$PRINT_COV ; then
+    coverage3 html
+    coverage3 report -m
+fi
 EOF
 
 # Create the test script
@@ -77,10 +104,37 @@ if [ $(uname) = 'Darwin' ]; then
 fi
 ID2T_DIR=\$(readlink -f \$0)
 SCRIPT_PATH=\${ID2T_DIR%/*}
-cd \$SCRIPT_PATH/code
+TEST_DIR=\${SCRIPT_PATH}/resources/test/
+TEST_PCAP=\${TEST_DIR}reference_1998.pcap
+PLOT_DIR=\${TEST_DIR}/plot/
+cd \${SCRIPT_PATH}/code
+error=0
 # Execute tests
-set -e
+set +e
 python3 -m unittest Test/efficiency_testing.py
+error=\$?
+cd \$SCRIPT_PATH
+mkdir \$PLOT_DIR
+smbloris="SMBLorisAttack attackers.count=4 packets.per-second=8.0"
+smbscan1="SMBScanAttack ip.src=192.168.178.1 ip.dst=192.168.178.10-192.168.179.253"
+smbscan2="SMBScanAttack ip.src=192.168.178.1 ip.dst=192.168.178.10-192.168.178.109 hosting.ip=192.168.178.10-192.168.178.109"
+ftp="FTPWinaXeExploit ip.src=192.168.178.1 ip.dst=192.168.178.10"
+porto="PortscanAttack ip.src=192.168.178.1 port.open=80"
+portc="PortscanAttack ip.src=192.168.178.1 port.open=20"
+sqli="SQLiAttack ip.dst=192.168.0.1"
+joomla="JoomlaRegPrivExploit ip.src=192.168.178.1"
+sality="SalityBotnet"
+ddos="DDoSAttack attackers.count=10 packets.per-second=95 attack.duration=10"
+ms17="MS17Scan ip.src=192.168.178.1"
+eb="EternalBlue"
+for i in "\$smbloris" "\$smbscan1" "\$smbscan2" "\$ftp" "\$porto" "\$portc" "\$sqli" "\$joomla" "\$sality" "\$ddos" "\$ms17" "\$eb"; do
+    mprof run ./id2t -i \${TEST_PCAP} -a \${i}
+    mprof plot -t "\${i}" -o "\${PLOT_DIR}\${i}.png"
+    mv mprofile_* "\${PLOT_DIR}\${i}.dat"
+done
+echo "\nPlotted images can be found in \"\${TEST_DIR}\"."
+echo "By executing \"mprof plot <file>.dat\" you can get a more detailed look."
+exit \$error
 EOF
 
 chmod +x ./code/CLI.py

code/Attack/AttackParameters.py

@@ -11,6 +11,7 @@ class Parameter(enum.Enum):
     IP_DESTINATION = 'ip.dst'  # destination IP address
     IP_DNS = 'ip.dns'  # IP address of DNS server
     HOSTING_IP = 'hosting.ip'
+    IP_VICTIM = 'ip.victim'
     # recommended type: MAC address ------------------------------
     MAC_SOURCE = 'mac.src'  # MAC address of source
     MAC_DESTINATION = 'mac.dst'  # MAC address of destination

code/Attack/BaseAttack.py

@@ -11,6 +11,7 @@ import socket
 import sys
 import tempfile
 import time
+import collections
 
 # TODO: double check this import
 # does it complain because libpcapreader is not a .py?
@@ -29,6 +30,8 @@ class BaseAttack(metaclass=abc.ABCMeta):
     Abstract base class for all attack classes. Provides basic functionalities, like parameter validation.
     """
 
+    ValuePair = collections.namedtuple('ValuePair', ['value', 'user_specified'])
+
     def __init__(self, name, description, attack_type):
         """
         To be called within the individual attack class to initialize the required parameters.
@@ -252,7 +255,7 @@ class BaseAttack(metaclass=abc.ABCMeta):
         try:
             import distutils.core
             import distutils.util
-            value = distutils.util.strtobool(value.lower())
+            value = bool(distutils.util.strtobool(value.lower()))
             is_bool = True
         except ValueError:
             is_bool = False
@@ -324,19 +327,16 @@ class BaseAttack(metaclass=abc.ABCMeta):
         """
         return self.finish_time - self.start_time
 
-    def add_param_value(self, param, value):
+    def add_param_value(self, param, value, user_specified: bool = True):
         """
         Adds the pair param : value to the dictionary of attack parameters. Prints and error message and skips the
         parameter if the validation fails.
 
         :param param: Name of the parameter that we wish to modify.
         :param value: The value we wish to assign to the specified parameter.
+        :param user_specified: Whether the value was specified by the user (or left default)
         :return: None.
         """
-        # This function call is valid only if there is a statistics object available.
-        if self.statistics is None:
-            print('Error: Attack parameter added without setting a statistics object first.')
-            exit(1)
 
         # by default no param is valid
         is_valid = False
@@ -394,6 +394,11 @@ class BaseAttack(metaclass=abc.ABCMeta):
         elif param_type == atkParam.ParameterTypes.TYPE_BOOLEAN:
             is_valid, value = self._is_boolean(value)
         elif param_type == atkParam.ParameterTypes.TYPE_PACKET_POSITION:
+            # This function call is valid only if there is a statistics object available.
+            if self.statistics is None:
+                print('Error: Statistics-dependent attack parameter added without setting a statistics object first.')
+                exit(1)
+
             ts = pr.pcap_processor(self.statistics.pcap_filepath, "False").get_timestamp_mu_sec(int(value))
             if 0 <= int(value) <= self.statistics.get_packet_count() and ts >= 0:
                 is_valid = True
@@ -421,7 +426,7 @@ class BaseAttack(metaclass=abc.ABCMeta):
 
         # add value iff validation was successful
         if is_valid:
-            self.params[param_name] = value
+            self.params[param_name] = self.ValuePair(value, user_specified)
         else:
             print("ERROR: Parameter " + str(param) + " or parameter value " + str(value) +
                   " not valid. Skipping parameter.")
@@ -433,7 +438,24 @@ class BaseAttack(metaclass=abc.ABCMeta):
         :param param: The parameter whose value is wanted.
         :return: The parameter's value.
         """
-        return self.params.get(param)
+        parameter = self.params.get(param)
+        if parameter is not None:
+            return parameter.value
+        else:
+            return None
+
+    def get_param_user_specified(self, param: atkParam.Parameter) -> bool:
+        """
+        Returns whether the parameter value was specified by the user for a given parameter.
+
+        :param param: The parameter whose user-specified flag is wanted.
+        :return: The parameter's user-specified flag.
+        """
+        parameter = self.params.get(param)
+        if parameter is not None:
+            return parameter.user_specified
+        else:
+            return False
 
     def check_parameters(self):
         """
@@ -518,7 +540,7 @@ class BaseAttack(metaclass=abc.ABCMeta):
            (IP,port) to host B (IP,port)
 
            :param exploit_raw_packets: A set of packets contains several conversations.
-           :return conversations: A set of arrays, each array contains the packet of specifc conversation
+           :return conversations: A set of arrays, each array contains the packet of specific conversation
            :return orderList_conversations: An array contains the conversations ids (IP_A,port_A, IP_b,port_B) in the
            order they appeared in the original packets.
            """
@@ -697,11 +719,21 @@ class BaseAttack(metaclass=abc.ABCMeta):
         """
 
         def is_invalid(ip_address_param: ipaddress.IPv4Address):
+            """
+            TODO FILL ME
+            :param ip_address_param:
+            :return:
+            """
             return ip_address_param.is_multicast or ip_address_param.is_unspecified or ip_address_param.is_loopback or \
                    ip_address_param.is_link_local or ip_address_param.is_reserved or ip_address_param.is_private
 
         # Generate a random IP from specific class
         def generate_address(ip_class_param):
+            """
+            TODO FILL ME
+            :param ip_class_param:
+            :return:
+            """
             if ip_class_param == "Unknown":
                 return ipaddress.IPv4Address(random.randint(0, 2 ** 32 - 1))
             else:
@@ -743,10 +775,19 @@ class BaseAttack(metaclass=abc.ABCMeta):
         """
 
         def is_invalid(ip_address: ipaddress.IPv6Address):
+            """
+            TODO FILL ME
+            :param ip_address:
+            :return:
+            """
             return ip_address.is_multicast or ip_address.is_unspecified or ip_address.is_loopback or \
                    ip_address.is_link_local or ip_address.is_private or ip_address.is_reserved
 
         def generate_address():
+            """
+            TODO FILL ME
+            :return:
+            """
             return ipaddress.IPv6Address(random.randint(0, 2 ** 128 - 1))
 
         ip_addresses = []
@@ -767,7 +808,7 @@ class BaseAttack(metaclass=abc.ABCMeta):
         Generates n random MAC addresses.
 
         :param n: The number of MAC addresses to be generated.
-        :return: A single MAC addres, or if n>1, a list of MAC addresses
+        :return: A single MAC address, or if n>1, a list of MAC addresses
         """
 
         def is_invalid(address_param: str):

code/Attack/DDoSAttack.py

@@ -55,7 +55,7 @@ class DDoSAttack(BaseAttack.BaseAttack):
         # attacker configuration
         num_attackers = rnd.randint(1, 16)
         # The most used IP class in background traffic
-        most_used_ip_class = Util.handle_most_used_outputs(self.statistics.process_db_query("most_used(ipClass)"))
+        most_used_ip_class = Util.handle_most_used_outputs(self.statistics.get_most_used_ip_class())
 
         self.add_param_value(atkParam.Parameter.IP_SOURCE,
                              self.generate_random_ipv4_address(most_used_ip_class, num_attackers))
@@ -75,6 +75,9 @@ class DDoSAttack(BaseAttack.BaseAttack):
         self.add_param_value(atkParam.Parameter.VICTIM_BUFFER, rnd.randint(1000, 10000))
 
     def generate_attack_packets(self):
+        """
+        Creates the attack packets.
+        """
         buffer_size = 1000
 
         # Determine source IP and MAC address
@@ -82,7 +85,7 @@ class DDoSAttack(BaseAttack.BaseAttack):
         if (num_attackers is not None) and (num_attackers is not 0):
             # user supplied atkParam.Parameter.NUMBER_ATTACKERS
             # The most used IP class in background traffic
-            most_used_ip_class = Util.handle_most_used_outputs(self.statistics.process_db_query("most_used(ipClass)"))
+            most_used_ip_class = Util.handle_most_used_outputs(self.statistics.get_most_used_ip_class())
             # Create random attackers based on user input atkParam.Parameter.NUMBER_ATTACKERS
             ip_source_list = self.generate_random_ipv4_address(most_used_ip_class, num_attackers)
             mac_source_list = self.generate_random_mac_address(num_attackers)
@@ -164,14 +167,14 @@ class DDoSAttack(BaseAttack.BaseAttack):
             destination_win_prob_dict = lea.Lea.fromValFreqsDict(destination_win_dist)
             destination_win_value = destination_win_prob_dict.random()
         else:
-            destination_win_value = self.statistics.process_db_query("most_used(winSize)")
+            destination_win_value = self.statistics.get_most_used_win_size()
 
         destination_win_value = Util.handle_most_used_outputs(destination_win_value)
 
         # MSS that was used by IP destination in background traffic
         mss_dst = self.statistics.get_most_used_mss(ip_destination)
         if mss_dst is None:
-            mss_dst = self.statistics.process_db_query("most_used(mssValue)")
+            mss_dst = self.statistics.get_most_used_mss_value()
 
         mss_dst = Util.handle_most_used_outputs(mss_dst)
 
@@ -284,6 +287,11 @@ class DDoSAttack(BaseAttack.BaseAttack):
                 self.packets = []
 
     def generate_attack_pcap(self):
+        """
+        Creates a pcap containing the attack packets.
+
+        :return: The location of the generated pcap file.
+        """
         if len(self.packets) > 0:
             self.packets = sorted(self.packets, key=lambda pkt: pkt.time)
             self.path_attack_pcap = self.write_attack_pcap(self.packets, True, self.path_attack_pcap)

code/Attack/EternalBlueExploit.py

@@ -79,7 +79,9 @@ class EternalBlueExploit(BaseAttack.BaseAttack):
         self.add_param_value(atkParam.Parameter.INJECT_AFTER_PACKET, rnd.randint(0, self.statistics.get_packet_count()))
 
     def generate_attack_packets(self):
-
+        """
+        Creates the attack packets.
+        """
         # Timestamp
         timestamp_next_pkt = self.get_param_value(atkParam.Parameter.INJECT_AT_TIMESTAMP)
         pps = self.get_param_value(atkParam.Parameter.PACKETS_PER_SECOND)
@@ -106,7 +108,7 @@ class EternalBlueExploit(BaseAttack.BaseAttack):
             source_ttl_prob_dict = lea.Lea.fromValFreqsDict(source_ttl_dist)
             source_ttl_value = source_ttl_prob_dict.random()
         else:
-            source_ttl_value = Util.handle_most_used_outputs(self.statistics.process_db_query("most_used(ttlValue)"))
+            source_ttl_value = Util.handle_most_used_outputs(self.statistics.get_most_used_ttl_value())
 
         destination_ttl_dist = self.statistics.get_ttl_distribution(ip_destination)
         if len(destination_ttl_dist) > 0:
@@ -114,7 +116,7 @@ class EternalBlueExploit(BaseAttack.BaseAttack):
             destination_ttl_value = destination_ttl_prob_dict.random()
         else:
             destination_ttl_value = Util.handle_most_used_outputs(
-                self.statistics.process_db_query("most_used(ttlValue)"))
+                self.statistics.get_most_used_ttl_value())
 
         # Set Window Size based on Window Size distribution of IP address
         source_win_dist = self.statistics.get_win_distribution(ip_source)
@@ -132,7 +134,7 @@ class EternalBlueExploit(BaseAttack.BaseAttack):
             destination_win_prob_dict = lea.Lea.fromValFreqsDict(destination_win_dist)
 
         # Set MSS (Maximum Segment Size) based on MSS distribution of IP address
-        mss_value = Util.handle_most_used_outputs(self.statistics.process_db_query("most_used(mssValue)"))
+        mss_value = Util.handle_most_used_outputs(self.statistics.get_most_used_mss_value())
         if not mss_value:
             mss_value = 1465
 
@@ -312,6 +314,11 @@ class EternalBlueExploit(BaseAttack.BaseAttack):
                     self.packets.append(new_pkt)
 
     def generate_attack_pcap(self):
+        """
+        Creates a pcap containing the attack packets.
+
+        :return: The location of the generated pcap file.
+        """
         # Store timestamp of first packet (for attack label)
         self.attack_start_utime = self.packets[0].time
         self.attack_end_utime = self.packets[-1].time

code/Attack/FTPWinaXeExploit.py

@@ -12,16 +12,18 @@ logging.getLogger("scapy.runtime").setLevel(logging.ERROR)
 
 # noinspection PyPep8
 
+#FTP Port used for Creation of packets
 ftp_port = 21
 
 
 class FTPWinaXeExploit(BaseAttack.BaseAttack):
     def __init__(self):
         """
-        Creates a new instance of the FTPWinaXeExploit.
+        Creates a new instance of the FTPExploit.
+        This attack injects a buffer overflow for the WinaXe FTP-client into the output pcap file.
         """
         # Initialize attack
-        super(FTPWinaXeExploit, self).__init__("FTPWinaXe Exploit", "Injects an WinaXe 7.7 FTP Exploit.",
+        super(FTPWinaXeExploit, self).__init__("FTPWinaXe Exploit", "Injects a WinaXe 7.7 FTP buffer overflow.",
                                                "Privilege elevation")
 
         # Define allowed parameters and their type
@@ -50,7 +52,7 @@ class FTPWinaXeExploit(BaseAttack.BaseAttack):
         most_used_ip_address = self.statistics.get_most_used_ip_address()
 
         # The most used IP class in background traffic
-        most_used_ip_class = Util.handle_most_used_outputs(self.statistics.process_db_query("most_used(ipClass)"))
+        most_used_ip_class = Util.handle_most_used_outputs(self.statistics.get_most_used_ip_class())
         attacker_ip = self.generate_random_ipv4_address(most_used_ip_class)
         self.add_param_value(atkParam.Parameter.IP_DESTINATION, attacker_ip)
         self.add_param_value(atkParam.Parameter.MAC_DESTINATION, self.generate_random_mac_address())
@@ -74,7 +76,9 @@ class FTPWinaXeExploit(BaseAttack.BaseAttack):
         self.add_param_value(atkParam.Parameter.CUSTOM_PAYLOAD_FILE, '')
 
     def generate_attack_packets(self):
-
+        """
+        Creates the attack packets.
+        """
         pps = self.get_param_value(atkParam.Parameter.PACKETS_PER_SECOND)
 
         # Timestamp
@@ -98,7 +102,7 @@ class FTPWinaXeExploit(BaseAttack.BaseAttack):
         # Create random victim if specified
         if self.get_param_value(atkParam.Parameter.IP_SOURCE_RANDOMIZE):
             # The most used IP class in background traffic
-            most_used_ip_class = Util.handle_most_used_outputs(self.statistics.process_db_query("most_used(ipClass)"))
+            most_used_ip_class = Util.handle_most_used_outputs(self.statistics.get_most_used_ip_class())
             ip_victim = self.generate_random_ipv4_address(most_used_ip_class, 1)
             mac_victim = self.generate_random_mac_address()
 
@@ -154,6 +158,7 @@ class FTPWinaXeExploit(BaseAttack.BaseAttack):
 
         custom_payload_file = self.get_param_value(atkParam.Parameter.CUSTOM_PAYLOAD_FILE)
 
+        # Generation of payload of the FTP exploit packet
         if custom_payload == '':
             if custom_payload_file == '':
                 payload = Util.get_rnd_bytes(custom_payload_limit, Util.forbidden_chars)
@@ -203,6 +208,11 @@ class FTPWinaXeExploit(BaseAttack.BaseAttack):
         self.packets.append(fin_ack_ack)
 
     def generate_attack_pcap(self):
+        """
+        Creates a pcap containing the attack packets.
+
+        :return: The location of the generated pcap file.
+        """
         # store end time of attack
         self.attack_end_utime = self.packets[-1].time
 

code/Attack/JoomlaRegPrivExploit.py

@@ -78,6 +78,9 @@ class JoomlaRegPrivExploit(BaseAttack.BaseAttack):
                               self.statistics.get_pps_received(most_used_ip_address)) / 2)
 
     def generate_attack_packets(self):
+        """
+        Creates the attack packets.
+        """
         # Timestamp
         timestamp_next_pkt = self.get_param_value(atkParam.Parameter.INJECT_AT_TIMESTAMP)
 
@@ -107,7 +110,7 @@ class JoomlaRegPrivExploit(BaseAttack.BaseAttack):
             source_ttl_prob_dict = lea.Lea.fromValFreqsDict(source_ttl_dist)
             source_ttl_value = source_ttl_prob_dict.random()
         else:
-            source_ttl_value = Util.handle_most_used_outputs(self.statistics.process_db_query("most_used(ttlValue)"))
+            source_ttl_value = Util.handle_most_used_outputs(self.statistics.get_most_used_ttl_value())
 
         destination_ttl_dist = self.statistics.get_ttl_distribution(ip_destination)
         if len(destination_ttl_dist) > 0:
@@ -115,7 +118,7 @@ class JoomlaRegPrivExploit(BaseAttack.BaseAttack):
             destination_ttl_value = destination_ttl_prob_dict.random()
         else:
             destination_ttl_value = Util.handle_most_used_outputs(
-                self.statistics.process_db_query("most_used(ttlValue)"))
+                self.statistics.get_most_used_ttl_value())
 
         # Inject Joomla_registration_privesc
         # Read joomla_registration_privesc pcap file
@@ -219,7 +222,11 @@ class JoomlaRegPrivExploit(BaseAttack.BaseAttack):
         exploit_raw_packets.close()
 
     def generate_attack_pcap(self):
+        """
+        Creates a pcap containing the attack packets.
 
+        :return: The location of the generated pcap file.
+        """
         # Store timestamp of first packet (for attack label)
         self.attack_start_utime = self.packets[0].time
         self.attack_end_utime = self.packets[-1].time

code/Attack/MS17ScanAttack.py

@@ -78,7 +78,9 @@ class MS17ScanAttack(BaseAttack.BaseAttack):
         self.add_param_value(atkParam.Parameter.INJECT_AFTER_PACKET, rnd.randint(0, self.statistics.get_packet_count()))
 
     def generate_attack_packets(self):
-
+        """
+        Creates the attack packets.
+        """
         # Timestamp
         timestamp_next_pkt = self.get_param_value(atkParam.Parameter.INJECT_AT_TIMESTAMP)
         pps = self.get_param_value(atkParam.Parameter.PACKETS_PER_SECOND)
@@ -104,7 +106,7 @@ class MS17ScanAttack(BaseAttack.BaseAttack):
             source_ttl_prob_dict = lea.Lea.fromValFreqsDict(source_ttl_dist)
             source_ttl_value = source_ttl_prob_dict.random()
         else:
-            source_ttl_value = Util.handle_most_used_outputs(self.statistics.process_db_query("most_used(ttlValue)"))
+            source_ttl_value = Util.handle_most_used_outputs(self.statistics.get_most_used_ttl_value())
 
         destination_ttl_dist = self.statistics.get_ttl_distribution(ip_destination)
         if len(destination_ttl_dist) > 0:
@@ -112,7 +114,7 @@ class MS17ScanAttack(BaseAttack.BaseAttack):
             destination_ttl_value = destination_ttl_prob_dict.random()
         else:
             destination_ttl_value = Util.handle_most_used_outputs(
-                self.statistics.process_db_query("most_used(ttlValue)"))
+                self.statistics.get_most_used_ttl_value())
 
         # Set Window Size based on Window Size distribution of IP address
         source_win_dist = self.statistics.get_win_distribution(ip_source)
@@ -130,7 +132,7 @@ class MS17ScanAttack(BaseAttack.BaseAttack):
             destination_win_prob_dict = lea.Lea.fromValFreqsDict(destination_win_dist)
 
         # Set MSS (Maximum Segment Size) based on MSS distribution of IP address
-        mss_value = Util.handle_most_used_outputs(self.statistics.process_db_query("most_used(mssValue)"))
+        mss_value = Util.handle_most_used_outputs(self.statistics.get_most_used_mss_value())
         if not mss_value:
             mss_value = 1465
 
@@ -219,6 +221,11 @@ class MS17ScanAttack(BaseAttack.BaseAttack):
         exploit_raw_packets.close()
 
     def generate_attack_pcap(self):
+        """
+        Creates a pcap containing the attack packets.
+
+        :return: The location of the generated pcap file.
+        """
         # Store timestamp of first packet (for attack label)
         self.attack_start_utime = self.packets[0].time
         self.attack_end_utime = self.packets[-1].time

code/Attack/MemcrashedSpooferAttack.py

@@ -0,0 +1,101 @@
+import logging
+import random as rnd
+import typing
+
+import scapy.layers.inet as inet
+
+import Attack.AttackParameters as atkParam
+import Attack.BaseAttack as BaseAttack
+import ID2TLib.Utility as Util
+import ID2TLib.Memcached as Memcd
+
+logging.getLogger("scapy.runtime").setLevel(logging.ERROR)
+
+
+class MemcrashedSpooferAttack(BaseAttack.BaseAttack):
+    def __init__(self):
+        """
+        Creates a new instance of the "Memcrashed" Memcached amplification attack.
+        """
+        # Initialize attack
+        super(MemcrashedSpooferAttack, self).__init__("Memcrashed Attack (Spoofer side)",
+                                               "Injects the spoofer-side of a Memcached amplification attack",
+                                               "Resource Exhaustion")
+
+        # Define allowed parameters and their type
+        self.supported_params.update({
+            atkParam.Parameter.IP_SOURCE: atkParam.ParameterTypes.TYPE_IP_ADDRESS,
+            atkParam.Parameter.MAC_SOURCE: atkParam.ParameterTypes.TYPE_MAC_ADDRESS,
+            atkParam.Parameter.IP_DESTINATION: atkParam.ParameterTypes.TYPE_IP_ADDRESS,
+            atkParam.Parameter.MAC_DESTINATION: atkParam.ParameterTypes.TYPE_MAC_ADDRESS,
+            atkParam.Parameter.IP_VICTIM: atkParam.ParameterTypes.TYPE_IP_ADDRESS,
+            atkParam.Parameter.INJECT_AT_TIMESTAMP: atkParam.ParameterTypes.TYPE_FLOAT,
+            atkParam.Parameter.INJECT_AFTER_PACKET: atkParam.ParameterTypes.TYPE_PACKET_POSITION,
+            atkParam.Parameter.PACKETS_PER_SECOND: atkParam.ParameterTypes.TYPE_FLOAT,
+            atkParam.Parameter.ATTACK_DURATION: atkParam.ParameterTypes.TYPE_INTEGER_POSITIVE
+        })
+
+    def init_params(self) -> None:
+        """
+        Initialize the parameters of this attack using the user supplied command line parameters.
+        Use the provided statistics to calculate default parameters and to process user
+        supplied queries.
+        """
+        # By default, the most used IP is the attacker
+        most_used_ip = self.statistics.get_most_used_ip_address()
+        self.add_param_value(atkParam.Parameter.IP_SOURCE, most_used_ip)
+        self.add_param_value(atkParam.Parameter.MAC_SOURCE, self.statistics.get_mac_address(most_used_ip))
+
+        # Target (i.e. amplifier) is a random public IP
+        self.add_param_value(atkParam.Parameter.IP_DESTINATION, self.generate_random_ipv4_address('A'))
+        self.add_param_value(atkParam.Parameter.MAC_DESTINATION, self.generate_random_mac_address())
+
+        # IP of the victim which is supposed to get hit by the amplified attack
+        self.add_param_value(atkParam.Parameter.IP_VICTIM, self.generate_random_ipv4_address('A'))
+
+        self.add_param_value(atkParam.Parameter.PACKETS_PER_SECOND, (self.statistics.get_pps_sent(most_used_ip) +
+                             self.statistics.get_pps_received(most_used_ip)) / 2)
+        self.add_param_value(atkParam.Parameter.ATTACK_DURATION, rnd.randint(5, 30))
+        self.add_param_value(atkParam.Parameter.INJECT_AFTER_PACKET, rnd.randint(0, self.statistics.get_packet_count()))
+
+    def generate_attack_packets(self) -> None:
+        ip_attacker = self.get_param_value(atkParam.Parameter.IP_SOURCE)
+        mac_attacker = self.get_param_value(atkParam.Parameter.MAC_SOURCE)
+        ip_amplifier = self.get_param_value(atkParam.Parameter.IP_DESTINATION)
+        mac_amplifier = self.get_param_value(atkParam.Parameter.MAC_DESTINATION)
+        ip_victim = self.get_param_value(atkParam.Parameter.IP_VICTIM)
+
+        pps = self.get_param_value(atkParam.Parameter.PACKETS_PER_SECOND)
+
+        timestamp_next_pkt = self.get_param_value(atkParam.Parameter.INJECT_AT_TIMESTAMP)
+        self.attack_start_utime = timestamp_next_pkt
+
+        attack_duration = self.get_param_value(atkParam.Parameter.ATTACK_DURATION)
+        attack_ends_time = timestamp_next_pkt + attack_duration
+
+        _, src_ttl, _ = self.get_ip_data(ip_attacker)
+        sport = Util.generate_source_port_from_platform('linux')
+
+        # Use MAC of the actual source, but the IP of the victim
+        attacker_ether = inet.Ether(src=mac_attacker, dst=mac_amplifier)
+        attacker_ip = inet.IP(src=ip_victim, dst=ip_amplifier, ttl=src_ttl, flags='DF')
+
+        while timestamp_next_pkt <= attack_ends_time:
+            request_udp = inet.UDP(sport=sport, dport=Memcd.memcached_port)
+            request_memcd = Memcd.Memcached_Request(Request=b'stats\r\n', RequestID=inet.RandShort())
+            request = (attacker_ether / attacker_ip / request_udp / request_memcd)
+            request.time = timestamp_next_pkt
+
+            self.packets.append(request)
+
+            timestamp_next_pkt = Util.update_timestamp(timestamp_next_pkt, pps)
+
+    def generate_attack_pcap(self) -> typing.Tuple[int, str]:
+        # store end time of attack
+        self.attack_end_utime = self.packets[-1].time
+
+        # write attack packets to pcap
+        pcap_path = self.write_attack_pcap(self.packets)
+
+        # return packet count and path
+        return len(self.packets), pcap_path

code/Attack/PortscanAttack.py

@@ -17,6 +17,7 @@ class PortscanAttack(BaseAttack.BaseAttack):
     def __init__(self):
         """
         Creates a new instance of the PortscanAttack.
+        This attack injects TCP Syn-requests and respective responses into the output pcap file.
         """
         # Initialize attack
         super(PortscanAttack, self).__init__("Portscan Attack", "Injects a nmap 'regular scan'",
@@ -55,7 +56,7 @@ class PortscanAttack(BaseAttack.BaseAttack):
         self.add_param_value(atkParam.Parameter.MAC_SOURCE, self.statistics.get_mac_address(most_used_ip_address))
 
         random_ip_address = self.statistics.get_random_ip_address()
-        # ip-dst should be valid and not equal to ip.src
+        # ip.dst should be valid and not equal to ip.src
         while not self.is_valid_ip_address(random_ip_address) or random_ip_address == most_used_ip_address:
             random_ip_address = self.statistics.get_random_ip_address()
 
@@ -76,6 +77,9 @@ class PortscanAttack(BaseAttack.BaseAttack):
         self.add_param_value(atkParam.Parameter.INJECT_AFTER_PACKET, rnd.randint(0, self.statistics.get_packet_count()))
 
     def generate_attack_packets(self):
+        """
+        Creates the attack packets.
+        """
         mac_source = self.get_param_value(atkParam.Parameter.MAC_SOURCE)
         mac_destination = self.get_param_value(atkParam.Parameter.MAC_DESTINATION)
         pps = self.get_param_value(atkParam.Parameter.PACKETS_PER_SECOND)
@@ -139,14 +143,13 @@ class PortscanAttack(BaseAttack.BaseAttack):
             source_mss_prob_dict = lea.Lea.fromValFreqsDict(source_mss_dist)
             source_mss_value = source_mss_prob_dict.random()
         else:
-            source_mss_value = Util.handle_most_used_outputs(self.statistics.process_db_query("most_used(mssValue)"))
+            source_mss_value = Util.handle_most_used_outputs(self.statistics.get_most_used_mss_value())
         destination_mss_dist = self.statistics.get_mss_distribution(ip_destination)
         if len(destination_mss_dist) > 0:
             destination_mss_prob_dict = lea.Lea.fromValFreqsDict(destination_mss_dist)
             destination_mss_value = destination_mss_prob_dict.random()
         else:
-            destination_mss_value = Util.handle_most_used_outputs(
-                self.statistics.process_db_query("most_used(mssValue)"))
+            destination_mss_value = Util.handle_most_used_outputs(self.statistics.get_most_used_mss_value())
 
         # Set TTL based on TTL distribution of IP address
         source_ttl_dist = self.statistics.get_ttl_distribution(ip_source)
@@ -154,14 +157,13 @@ class PortscanAttack(BaseAttack.BaseAttack):
             source_ttl_prob_dict = lea.Lea.fromValFreqsDict(source_ttl_dist)
             source_ttl_value = source_ttl_prob_dict.random()
         else:
-            source_ttl_value = Util.handle_most_used_outputs(self.statistics.process_db_query("most_used(ttlValue)"))
+            source_ttl_value = Util.handle_most_used_outputs(self.statistics.get_most_used_ttl_value())
         destination_ttl_dist = self.statistics.get_ttl_distribution(ip_destination)
         if len(destination_ttl_dist) > 0:
             destination_ttl_prob_dict = lea.Lea.fromValFreqsDict(destination_ttl_dist)
             destination_ttl_value = destination_ttl_prob_dict.random()
         else:
-            destination_ttl_value = Util.handle_most_used_outputs(
-                self.statistics.process_db_query("most_used(ttlValue)"))
+            destination_ttl_value = Util.handle_most_used_outputs(self.statistics.get_most_used_ttl_value())
 
         # Set Window Size based on Window Size distribution of IP address
         source_win_dist = self.statistics.get_win_distribution(ip_source)
@@ -169,14 +171,13 @@ class PortscanAttack(BaseAttack.BaseAttack):
             source_win_prob_dict = lea.Lea.fromValFreqsDict(source_win_dist)
             source_win_value = source_win_prob_dict.random()
         else:
-            source_win_value = Util.handle_most_used_outputs(self.statistics.process_db_query("most_used(winSize)"))
+            source_win_value = Util.handle_most_used_outputs(self.statistics.get_most_used_win_size())
         destination_win_dist = self.statistics.get_win_distribution(ip_destination)
         if len(destination_win_dist) > 0:
             destination_win_prob_dict = lea.Lea.fromValFreqsDict(destination_win_dist)
             destination_win_value = destination_win_prob_dict.random()
         else:
-            destination_win_value = Util.handle_most_used_outputs(
-                self.statistics.process_db_query("most_used(winSize)"))
+            destination_win_value = Util.handle_most_used_outputs(self.statistics.get_most_used_win_size())
 
         min_delay, max_delay = self.get_reply_delay(ip_destination)
 
@@ -232,6 +233,11 @@ class PortscanAttack(BaseAttack.BaseAttack):
             timestamp_next_pkt = Util.update_timestamp(timestamp_next_pkt, pps)
 
     def generate_attack_pcap(self):
+        """
+        Creates a pcap containing the attack packets.
+
+        :return: The location of the generated pcap file.
+        """
         # store end time of attack
         self.attack_end_utime = self.packets[-1].time
 

code/Attack/SMBLorisAttack.py

@@ -18,6 +18,8 @@ class SMBLorisAttack(BaseAttack.BaseAttack):
     def __init__(self):
         """
         Creates a new instance of the SMBLorisAttack.
+        This attack injects special SMB-packets, which exploit the SMBLoris DoS vulnerability, into the output pcap
+        file.
         """
         # Initialize attack
         super(SMBLorisAttack, self).__init__("SMBLoris Attack", "Injects an SMBLoris (D)DoS Attack",
@@ -48,7 +50,7 @@ class SMBLorisAttack(BaseAttack.BaseAttack):
         most_used_ip_address = self.statistics.get_most_used_ip_address()
 
         # The most used IP class in background traffic
-        most_used_ip_class = Util.handle_most_used_outputs(self.statistics.process_db_query("most_used(ipClass)"))
+        most_used_ip_class = Util.handle_most_used_outputs(self.statistics.get_most_used_ip_class())
         num_attackers = rnd.randint(1, 16)
         source_ip = self.generate_random_ipv4_address(most_used_ip_class, num_attackers)
 
@@ -72,6 +74,9 @@ class SMBLorisAttack(BaseAttack.BaseAttack):
         self.add_param_value(atkParam.Parameter.ATTACK_DURATION, 30)
 
     def generate_attack_packets(self):
+        """
+        Creates the attack packets.
+        """
         pps = self.get_param_value(atkParam.Parameter.PACKETS_PER_SECOND)
 
         # Timestamp
@@ -89,7 +94,7 @@ class SMBLorisAttack(BaseAttack.BaseAttack):
         # user supplied atkParam.Parameter.NUMBER_ATTACKERS
         if (num_attackers is not None) and (num_attackers is not 0):
             # The most used IP class in background traffic
-            most_used_ip_class = Util.handle_most_used_outputs(self.statistics.process_db_query("most_used(ipClass)"))
+            most_used_ip_class = Util.handle_most_used_outputs(self.statistics.get_most_used_ip_class())
             # Create random attackers based on user input atkParam.Parameter.NUMBER_ATTACKERS
             ip_source = self.generate_random_ipv4_address(most_used_ip_class, num_attackers)
             mac_source = self.generate_random_mac_address(num_attackers)
@@ -179,7 +184,7 @@ class SMBLorisAttack(BaseAttack.BaseAttack):
                 timestamp_next_pkt = Util.update_timestamp(timestamp_next_pkt, pps)
                 self.packets.append(ack)
 
-                # send NBT session header paket with maximum LENGTH-field
+                # send NBT session header packet with maximum LENGTH-field
                 req_tcp = inet.TCP(sport=sport, dport=SMBLib.smb_port, seq=attacker_seq, ack=victim_seq, flags='AP',
                                    window=source_win_value, options=[('MSS', source_mss_value)])
                 req_payload = NBTSession(TYPE=0x00, LENGTH=0x1FFFF)
@@ -201,6 +206,11 @@ class SMBLorisAttack(BaseAttack.BaseAttack):
                 sport += 1
 
     def generate_attack_pcap(self):
+        """
+        Creates a pcap containing the attack packets.
+
+        :return: The location of the generated pcap file.
+        """
         # store end time of attack
         self.attack_end_utime = self.packets[-1].time
 

code/Attack/SMBScanAttack.py

@@ -19,6 +19,9 @@ class SMBScanAttack(BaseAttack.BaseAttack):
     def __init__(self):
         """
         Creates a new instance of the SMBScanAttack.
+        This Attack injects TCP Syn Requests to the port 445 of several ips and related response into the output
+        pcap file.
+        If port 445 is open, it will simulate and inject the SMB Protocol Negotiation too.
         """
         # Initialize attack
         super(SMBScanAttack, self).__init__("SmbScan Attack", "Injects an SMB scan",
@@ -86,6 +89,9 @@ class SMBScanAttack(BaseAttack.BaseAttack):
         self.add_param_value(atkParam.Parameter.PROTOCOL_VERSION, "1")
 
     def generate_attack_packets(self):
+        """
+        Creates the attack packets.
+        """
 
         pps = self.get_param_value(atkParam.Parameter.PACKETS_PER_SECOND)
 
@@ -224,6 +230,7 @@ class SMBScanAttack(BaseAttack.BaseAttack):
                     confirm.time = timestamp_confirm
                     self.packets.append(confirm)
 
+                    # 3) Build SMB Negotiation packets
                     smb_mid = rnd.randint(1, 65535)
                     smb_pid = rnd.randint(1, 65535)
                     smb_req_tail_arr = []
@@ -242,6 +249,7 @@ class SMBScanAttack(BaseAttack.BaseAttack):
                             smb_req_tail_arr.append(SMBNegociate_Protocol_Request_Tail(BufferData=dia))
                             smb_req_tail_size += len(SMBNegociate_Protocol_Request_Tail(BufferData=dia))
 
+                    # Creation of SMB Negotiate Protocol Request packet
                     smb_req_head = SMBNegociate_Protocol_Request_Header(Flags2=0x2801, PID=smb_pid, MID=smb_mid,
                                                                         ByteCount=smb_req_tail_size)
                     smb_req_length = len(smb_req_head) + smb_req_tail_size
@@ -281,8 +289,9 @@ class SMBScanAttack(BaseAttack.BaseAttack):
                     end = Util.get_filetime_format(timestamp_smb_rsp - diff * 0.1)
                     system_time = rnd.randint(begin, end)
 
+                    # Creation of SMB Negotiate Protocol Response packets
                     if smb_version is not "1" and hosting_version is not "1":
-                        smb_rsp_paket = SMB2.SMB2_SYNC_Header(Flags=1)
+                        smb_rsp_packet = SMB2.SMB2_SYNC_Header(Flags=1)
                         smb_rsp_negotiate_body =\
                             SMB2.SMB2_Negotiate_Protocol_Response(DialectRevision=0x02ff, SecurityBufferOffset=124,
                                                                   SecurityBufferLength=len(security_blob),
@@ -291,22 +300,22 @@ class SMBScanAttack(BaseAttack.BaseAttack):
                                                                   MaxWriteSize=data_size, SystemTime=system_time,
                                                                   ServerStartTime=server_start_time,
                                                                   ServerGuid=server_guid)
-                        smb_rsp_length = len(smb_rsp_paket) + len(smb_rsp_negotiate_body)
+                        smb_rsp_length = len(smb_rsp_packet) + len(smb_rsp_negotiate_body)
                     else:
-                        smb_rsp_paket =\
+                        smb_rsp_packet =\
                             SMBNegociate_Protocol_Response_Advanced_Security(Start="\xffSMB", PID=smb_pid, MID=smb_mid,
                                                                              DialectIndex=5, SecurityBlob=security_blob)
-                        smb_rsp_length = len(smb_rsp_paket)
+                        smb_rsp_length = len(smb_rsp_packet)
                     smb_rsp_net_bio = NBTSession(TYPE=0x00, LENGTH=smb_rsp_length)
                     smb_rsp_tcp = inet.TCP(sport=SMBLib.smb_port, dport=sport, flags='PA', seq=victim_seq,
                                            ack=attacker_seq)
                     smb_rsp_ip = inet.IP(src=ip, dst=ip_source, ttl=destination_ttl_value)
                     smb_rsp_ether = inet.Ether(src=mac_destination, dst=mac_source)
-                    victim_seq += len(smb_rsp_net_bio) + len(smb_rsp_paket)
+                    victim_seq += len(smb_rsp_net_bio) + len(smb_rsp_packet)
                     if smb_version is not "1" and hosting_version is not "1":
                         victim_seq += len(smb_rsp_negotiate_body)
 
-                    smb_rsp_combined = (smb_rsp_ether / smb_rsp_ip / smb_rsp_tcp / smb_rsp_net_bio / smb_rsp_paket)
+                    smb_rsp_combined = (smb_rsp_ether / smb_rsp_ip / smb_rsp_tcp / smb_rsp_net_bio / smb_rsp_packet)
                     if smb_version is not "1" and hosting_version is not "1":
                         smb_rsp_combined = (smb_rsp_combined / smb_rsp_negotiate_body)
 
@@ -361,7 +370,11 @@ class SMBScanAttack(BaseAttack.BaseAttack):
             timestamp_next_pkt = Util.update_timestamp(timestamp_next_pkt, pps)
 
     def generate_attack_pcap(self):
+        """
+        Creates a pcap containing the attack packets.
 
+        :return: The location of the generated pcap file.
+        """
         # store end time of attack
         self.attack_end_utime = self.packets[-1].time
 

code/Attack/SQLiAttack.py

@@ -80,6 +80,9 @@ class SQLiAttack(BaseAttack.BaseAttack):
                               self.statistics.get_pps_received(most_used_ip_address)) / 2)
 
     def generate_attack_packets(self):
+        """
+        Creates the attack packets.
+        """
         # Timestamp
         timestamp_next_pkt = self.get_param_value(atkParam.Parameter.INJECT_AT_TIMESTAMP)
         pps = self.get_param_value(atkParam.Parameter.PACKETS_PER_SECOND)
@@ -111,7 +114,7 @@ class SQLiAttack(BaseAttack.BaseAttack):
             source_ttl_prob_dict = lea.Lea.fromValFreqsDict(source_ttl_dist)
             source_ttl_value = source_ttl_prob_dict.random()
         else:
-            source_ttl_value = Util.handle_most_used_outputs(self.statistics.process_db_query("most_used(ttlValue)"))
+            source_ttl_value = Util.handle_most_used_outputs(self.statistics.get_most_used_ttl_value())
 
         destination_ttl_dist = self.statistics.get_ttl_distribution(ip_destination)
         if len(destination_ttl_dist) > 0:
@@ -119,7 +122,7 @@ class SQLiAttack(BaseAttack.BaseAttack):
             destination_ttl_value = destination_ttl_prob_dict.random()
         else:
             destination_ttl_value = Util.handle_most_used_outputs(
-                self.statistics.process_db_query("most_used(ttlValue)"))
+                self.statistics.get_most_used_ttl_value())
 
         # Inject SQLi Attack
         # Read SQLi Attack pcap file
@@ -241,7 +244,11 @@ class SQLiAttack(BaseAttack.BaseAttack):
         exploit_raw_packets.close()
 
     def generate_attack_pcap(self):
+        """
+        Creates a pcap containing the attack packets.
 
+        :return: The location of the generated pcap file.
+        """
         # Store timestamp of first packet (for attack label)
         self.attack_start_utime = self.packets[0].time
         self.attack_end_utime = self.packets[-1].time

code/Attack/SalityBotnet.py

@@ -56,6 +56,9 @@ class SalityBotnet(BaseAttack.BaseAttack):
                               self.statistics.get_pps_received(most_used_ip_address)) / 2)
 
     def generate_attack_packets(self):
+        """
+        Creates the attack packets.
+        """
 
         # Timestamp
         timestamp_next_pkt = self.get_param_value(atkParam.Parameter.INJECT_AT_TIMESTAMP)
@@ -132,6 +135,11 @@ class SalityBotnet(BaseAttack.BaseAttack):
         exploit_raw_packets.close()
 
     def generate_attack_pcap(self):
+        """
+        Creates a pcap containing the attack packets.
+
+        :return: The location of the generated pcap file.
+        """
         # Store timestamp of first packet (for attack label)
         self.attack_start_utime = self.packets[0].time
         self.attack_end_utime = self.packets[-1].time

code/Core/AttackController.py

@@ -2,6 +2,7 @@ import importlib
 import sys
 import difflib
 import pkgutil
+import typing
 
 import Attack.AttackParameters as atkParam
 import Core.LabelManager as LabelManager
@@ -29,7 +30,7 @@ class AttackController:
         self.seed = None
         self.total_packets = 0
 
-    def set_seed(self, seed: int):
+    def set_seed(self, seed: int) -> None:
         """
         Sets rng seed.
 
@@ -37,6 +38,13 @@ class AttackController:
         """
         self.seed = seed
 
+    def get_seed(self) -> typing.Union[int, None]:
+        """
+        Gets rng seed.
+        :return: The current rng seed
+        """
+        return self.seed
+
     @staticmethod
     def choose_attack(input_name):
         """"
@@ -108,7 +116,13 @@ class AttackController:
         self.current_attack.set_statistics(self.statistics)
         if seed is not None:
             self.current_attack.set_seed(seed=seed)
+
         self.current_attack.init_params()
+
+        # Unset the user-specified-flag for all parameters set in init_params
+        for k, v in self.current_attack.params.items():
+            self.current_attack.params[k] = self.current_attack.ValuePair(v.value, False)
+
         # Record the attack
         self.added_attacks.append(self.current_attack)
 
@@ -117,7 +131,7 @@ class AttackController:
         Takes as input the name of an attack (classname) and the attack parameters as string. Parses the string of
         attack parameters, creates the attack by writing the attack packets and returns the path of the written pcap.
 
-        :param attack: The classname of the attack to injecect.
+        :param attack: The classname of the attack to inject.
         :param params: The parameters for attack customization, see attack class for supported params.
         :param time: Measure packet generation time or not.
         :return: The file path to the created pcap file.
@@ -134,7 +148,7 @@ class AttackController:
                 params_dict.append(entry.split('='))
             params_dict = dict(params_dict)
             # Check if Parameter.INJECT_AT_TIMESTAMP and Parameter.INJECT_AFTER_PACKET are provided at the same time
-            # if TRUE: delete Paramter.INJECT_AT_TIMESTAMP (lower priority) and use Parameter.INJECT_AFTER_PACKET
+            # if TRUE: delete Parameter.INJECT_AT_TIMESTAMP (lower priority) and use Parameter.INJECT_AFTER_PACKET
             if (atkParam.Parameter.INJECT_AFTER_PACKET.value in params_dict) and (
                         atkParam.Parameter.INJECT_AT_TIMESTAMP.value in params_dict):
                 print("CONFLICT: Parameters", atkParam.Parameter.INJECT_AT_TIMESTAMP.value, "and",
@@ -169,8 +183,8 @@ class AttackController:
         print(".)")
 
         # Store label into LabelManager
-        label = Label.Label(attack, self.get_attack_start_utime(),
-                            self.get_attack_end_utime(), attack_note)
+        label = Label.Label(attack, self.get_attack_start_utime(), self.get_attack_end_utime(),
+                            self.seed, self.current_attack.params, attack_note)
         self.label_mgr.add_labels(label)
 
         return temp_attack_pcap_path, duration

code/Core/Controller.py

@@ -3,6 +3,7 @@ import readline
 import sys
 import shutil
 import time
+import re
 
 import pyparsing as pp
 import Core.AttackController as atkCtrl
@@ -10,10 +11,11 @@ import Core.LabelManager as LabelManager
 import Core.Statistics as Statistics
 import ID2TLib.PcapFile as PcapFile
 import ID2TLib.Utility as Util
+import Core.StatsDatabase as StatsDB
 
 
 class Controller:
-    def __init__(self, pcap_file_path: str, do_extra_tests: bool, non_verbose: bool, pcap_out_path: str=None):
+    def __init__(self, pcap_file_path: str, do_extra_tests: bool, non_verbose: bool=True, pcap_out_path: str=None):
         """
         Creates a new Controller, acting as a central coordinator for the whole application.
 
@@ -167,7 +169,7 @@ class Controller:
             print("--> No packets were injected. Therefore no output files were created.")
 
         # print summary statistics
-        if not self.non_verbose:
+        if not self.non_verbose and len(attacks_config) is not 1:
             self.statistics.stats_summary_post_attack(self.added_packets)
 
     def process_db_queries(self, query, print_results=False):
@@ -187,6 +189,11 @@ class Controller:
 
     @staticmethod
     def process_help(params):
+        """
+        TODO: FILL ME
+        :param params:
+        :return:
+        """
         if not params:
             print("Query mode allows you to enter SQL-queries as well as named queries.")
             print()
@@ -221,7 +228,7 @@ class Controller:
             print()
         elif param == "least_used":
             print("least_used can be used as a selector for the following attributes:")
-            print("ipAddress | macAddress | portNumber | protocolName | ttlValue")
+            print("ipAddress | macAddress | portNumber | protocolName | ttlValue | mssValue | winSize | ipClass")
             print()
         elif param == "avg":
             print("avg can be used as a selector for the following attributes:")
@@ -229,7 +236,7 @@ class Controller:
             print()
         elif param == "all":
             print("all can be used as a selector for the following attributes:")
-            print("ipAddress | ttlValue | mss | macAddress | portNumber | protocolName")
+            print("ipAddress | ttlValue | mss | macAddress | portNumber | protocolName | winSize | ipClass")
             print()
         elif param in ["random", "first", "last"]:
             print("No additional info available for this keyword.")
@@ -242,6 +249,14 @@ class Controller:
                   "macAddress | ttlValue | ttlCount | portDirection | portNumber | portCount | protocolCount\n"
                   "protocolName")
             print()
+            print("The following operators can be used:")
+            print("<= | < | = | >= | > | in")
+            print()
+            print("A value can either be a simple values, a list of simple values separated by commas and enclosed "
+                  "in [] brackets, or another query.")
+            print()
+            print("When VALUE is a list (or a query returning a list), the usage of the 'in' operator is mandatory!")
+            print()
             print("See 'help examples;' for usage examples.")
             print()
         elif param == "macaddress":
@@ -250,6 +265,8 @@ class Controller:
             print("The following parameters can be specified:")
             print("ipAddress")
             print()
+            print("See 'help ipAddress' for information on valid operators and values.")
+            print()
             print("See 'help examples;' for usage examples.")
             print()
         elif param == "examples":
@@ -263,11 +280,60 @@ class Controller:
             print("\tSELECT avg(ttlValue) from ip_ttl;")
             print("Get a random IP address from all addresses that sent and received at least 10 packets:")
             print("\trandom(ipAddress(pktsSent > 10, pktsReceived > 10));")
+            print("Get the IP addresses used with one of the MAC addresses in a list:")
+            print("\tipAddress(macAddress in [08:00:27:a3:83:43, 52:54:00:12:35:02]);")
             print()
         else:
             print("Unknown keyword '" + param + "', try 'help;' to get a list of allowed keywords'")
             print()
 
+    def internal_command(self, query: str) -> bool:
+        # Strip off semicolon, split into command and parameters
+        query = query.strip(";").split(" ", 1)
+        cmd = query[0].strip().lower()
+        if len(query) > 1:
+            params = [p for p in re.split("(,|\\\".*?\\\"|'.*?')", query[1]) if p.strip(",").strip()]
+            params = list(map(lambda x: x.strip().strip("\"'"), params))
+        else:
+            params = []
+
+        if cmd == "help":
+            self.process_help(params)
+            return True
+        elif cmd == "labels":
+            if not self.label_manager.labels:
+                print("No labels found.")
+            else:
+                print("Attacks listed in the label file:")
+                print()
+                for i, label in enumerate(self.label_manager.labels):
+                    print("Attack number:   " + str(i))
+                    print("Attack name:     " + str(label.attack_name))
+                    print("Attack note:     " + str(label.attack_note))
+                    print("Attack seed:     " + str(label.seed))
+                    print("Start timestamp: " + str(label.timestamp_start))
+                    print("End timestamp:   " + str(label.timestamp_end))
+                    print()
+            print()
+            return True
+        elif cmd == "set":
+            if len(params) == 3:
+                if params[0].lower() == "attack_note":
+                    i = int(params[1])
+                    self.label_manager.labels[i].attack_note = params[2]
+                return True
+        elif cmd == "tables":
+            self.statisticsDB.process_db_query("SELECT name FROM sqlite_master WHERE type='table';", True)
+            return True
+        elif cmd == "columns":
+            self.statisticsDB.process_db_query("SELECT * FROM " + params[0].lower(), False)
+            columns = self.statisticsDB.get_field_types(params[0].lower())
+            for column in columns:
+                print(column + ": " + columns[column])
+            return True
+
+        return False
+
     def enter_query_mode(self):
         """
         Enters into the query mode. This is a read-eval-print-loop, where the user can input named queries or SQL
@@ -275,7 +341,18 @@ class Controller:
         """
 
         def make_completer(vocabulary):
+            """
+            TODO: FILL ME
+            :param vocabulary:
+            :return:
+            """
             def custom_template(text, state):
+                """
+                TODO: FILL ME
+                :param text:
+                :param state:
+                :return:
+                """
                 results = [x for x in vocabulary if x.startswith(text)] + [None]
                 return results[state]
 
@@ -301,30 +378,7 @@ class Controller:
             import sqlite3
             if sqlite3.complete_statement(buffer):
                 buffer = buffer.strip()
-                if buffer.lower().startswith('help'):
-                    buffer = buffer.strip(';')
-                    self.process_help(buffer.split(' ')[1:])
-                elif buffer.lower().strip() == 'labels;':
-                    if not self.label_manager.labels:
-                        print("No labels found.")
-                    else:
-                        print("Attacks listed in the label file:")
-                        print()
-                        for label in self.label_manager.labels:
-                            print("Attack name:     " + str(label.attack_name))
-                            print("Attack note:     " + str(label.attack_note))
-                            print("Start timestamp: " + str(label.timestamp_start))
-                            print("End timestamp:   " + str(label.timestamp_end))
-                            print()
-                    print()
-                elif buffer.lower().strip() == 'tables;':
-                    self.statisticsDB.process_db_query("SELECT name FROM sqlite_master WHERE type='table';", True)
-                elif buffer.lower().strip().startswith('columns '):
-                    self.statisticsDB.process_db_query("SELECT * FROM " + buffer.lower()[8:], False)
-                    columns = self.statisticsDB.get_field_types(buffer.lower()[8:].strip(";"))
-                    for column in columns:
-                        print(column + ": " + columns[column])
-                else:
+                if not self.internal_command(buffer):
                     try:
                         self.statisticsDB.process_db_query(buffer, True)
                     except sqlite3.Error as e:
@@ -336,15 +390,22 @@ class Controller:
                         for i in range(1, e.col):
                             sys.stderr.write(" ")
                         sys.stderr.write("^\n\n")
+                    except StatsDB.QueryExecutionException as e:
+                        sys.stderr.write("An error occured: ")
+                        sys.stderr.write(e.args[0] + "\n")
                 buffer = ""
 
         readline.set_history_length(1000)
         readline.write_history_file(history_file)
 
+        # Save the label file, in case content has changed
+        self.label_manager.write_label_file(self.pcap_src_path)
+
     def create_statistics_plot(self, params: str, entropy: bool):
         """
         Plots the statistics to a file by using the given customization parameters.
         """
+        print("Statistical plots are being generated", end="", flush=True)
         if params is not None and params[0] is not None:
             # FIXME: cleanup
             params_dict = dict([z.split("=") for z in params])

code/Core/LabelManager.py

@@ -1,23 +1,32 @@
+import importlib
 import datetime as dt
 import os.path
 import xml.dom.minidom as minidom
 
 import ID2TLib.Label as Label
+import ID2TLib.TestLibrary as Lib
 
 
 class LabelManager:
-    TAG_ROOT = 'LABELS'
+    TAG_ROOT = 'labels'
+    TAG_INPUT = 'input'
+    TAG_OUTPUT = 'output'
+    TAG_FILE_NAME = 'filename'
+    TAG_FILE_HASH = 'sha256'
     TAG_ATTACK = 'attack'
-    TAG_ATTACK_NAME = 'attack_name'
-    TAG_ATTACK_NOTE = 'attack_note'
+    TAG_ATTACK_NAME = 'name'
+    TAG_ATTACK_NOTE = 'note'
+    TAG_ATTACK_SEED = 'seed'
     TAG_TIMESTAMP_START = 'timestamp_start'
     TAG_TIMESTAMP_END = 'timestamp_end'
     TAG_TIMESTAMP = 'timestamp'
     TAG_TIMESTAMP_HR = 'timestamp_hr'
+    TAG_PARAMETERS = 'parameters'
     ATTR_VERSION = 'version_parser'
+    ATTR_PARAM_USERSPECIFIED = 'user_specified'
 
     # update this attribute if XML scheme was modified
-    ATTR_VERSION_VALUE = '0.2'
+    ATTR_VERSION_VALUE = '0.3'
 
     def __init__(self, filepath_pcap=None):
         """
@@ -26,6 +35,7 @@ class LabelManager:
         :param filepath_pcap: The path to the PCAP file associated to the labels.
         """
         self.labels = list()
+        self.filepath_input_pcap = filepath_pcap
 
         if filepath_pcap is not None:
             self.label_file_path = os.path.splitext(filepath_pcap)[0] + '_labels.xml'
@@ -58,6 +68,25 @@ class LabelManager:
         :param filepath: The path where the label file should be written to.
         """
 
+        def get_subtree_fileinfo(xml_tag_root, filename) -> minidom.Element:
+            """
+            Creates the subtree for pcap file information (filename and hash).
+
+            :return: The root node of the XML subtree
+            """
+
+            input_root = doc.createElement(xml_tag_root)
+
+            file = doc.createElement(self.TAG_FILE_NAME)
+            file.appendChild(doc.createTextNode(os.path.split(filename)[-1]))
+            input_root.appendChild(file)
+
+            hash_node = doc.createElement(self.TAG_FILE_HASH)
+            hash_node.appendChild(doc.createTextNode(Lib.get_sha256(filename)))
+            input_root.appendChild(hash_node)
+
+            return input_root
+
         def get_subtree_timestamp(xml_tag_root, timestamp_entry):
             """
             Creates the subtree for a given timestamp, consisting of the unix time format (seconds) and a human-readable
@@ -82,6 +111,23 @@ class LabelManager:
 
             return timestamp_root
 
+        def get_subtree_parameters(parameters):
+            """
+            Creates a subtree containing all parameters used to construct the attack
+
+            :param parameters: The list of parameters used to run the attack
+            :return: The root node of the XML subtree
+            """
+            parameters_root = doc.createElement(self.TAG_PARAMETERS)
+
+            for param_key, param_value in parameters.items():
+                param = doc.createElement(param_key.value)
+                param.appendChild(doc.createTextNode(str(param_value.value)))
+                param.setAttribute(self.ATTR_PARAM_USERSPECIFIED, str(param_value.user_specified))
+                parameters_root.appendChild(param)
+
+            return parameters_root
+
         if filepath is not None:
             self.label_file_path = os.path.splitext(filepath)[0] + '_labels.xml'
 
@@ -89,6 +135,9 @@ class LabelManager:
         doc = minidom.Document()
         node = doc.createElement(self.TAG_ROOT)
         node.setAttribute(self.ATTR_VERSION, self.ATTR_VERSION_VALUE)
+        node.appendChild(get_subtree_fileinfo(self.TAG_INPUT, self.filepath_input_pcap))
+        node.appendChild(get_subtree_fileinfo(self.TAG_OUTPUT, filepath))
+
         for label in self.labels:
             xml_tree = doc.createElement(self.TAG_ATTACK)
 
@@ -99,6 +148,9 @@ class LabelManager:
             attack_note = doc.createElement(self.TAG_ATTACK_NOTE)
             attack_note.appendChild(doc.createTextNode(str(label.attack_note)))
             xml_tree.appendChild(attack_note)
+            attack_seed = doc.createElement(self.TAG_ATTACK_SEED)
+            attack_seed.appendChild(doc.createTextNode(str(label.seed)))
+            xml_tree.appendChild(attack_seed)
 
             # add timestamp_start to XML tree
             xml_tree.appendChild(get_subtree_timestamp(self.TAG_TIMESTAMP_START, label.timestamp_start))
@@ -106,6 +158,9 @@ class LabelManager:
             # add timestamp_end to XML tree
             xml_tree.appendChild(get_subtree_timestamp(self.TAG_TIMESTAMP_END, label.timestamp_end))
 
+            # add parameters to XML tree
+            xml_tree.appendChild(get_subtree_parameters(label.parameters))
+
             node.appendChild(xml_tree)
 
         doc.appendChild(node)
@@ -155,6 +210,11 @@ class LabelManager:
                     "The file " + self.label_file_path + " was created by another version of ID2TLib.LabelManager. "
                                                          "Ignoring label file.")
 
+        self.input_filename = get_value_from_node(dom, self.TAG_INPUT, 1, 0)
+        self.input_hash = get_value_from_node(dom, self.TAG_INPUT, 3, 0)
+        self.output_filename = get_value_from_node(dom, self.TAG_OUTPUT, 1, 0)
+        self.output_hash = get_value_from_node(dom, self.TAG_OUTPUT, 3, 0)
+
         # Parse attacks from XML file
         attacks = dom.getElementsByTagName(self.TAG_ATTACK)
         count_labels = 0
@@ -163,7 +223,27 @@ class LabelManager:
             attack_note = get_value_from_node(a, self.TAG_ATTACK_NOTE, 0)
             timestamp_start = get_value_from_node(a, self.TAG_TIMESTAMP_START, 1, 0)
             timestamp_end = get_value_from_node(a, self.TAG_TIMESTAMP_END, 1, 0)
-            label = Label.Label(attack_name, float(timestamp_start), float(timestamp_end), attack_note)
+            attack_seed = get_value_from_node(a, self.TAG_ATTACK_SEED, 0)
+
+            # Instantiate this attack to create a parameter list with the correct types
+            attack_module = importlib.import_module("Attack." + attack_name)
+            attack_class = getattr(attack_module, attack_name)
+            attack = attack_class()
+
+            # Loop through all parameters listed in the XML file
+            param = a.getElementsByTagName(self.TAG_PARAMETERS)[0]
+            for param in param.childNodes:
+                # Skip empty text nodes returned by minidom
+                if not isinstance(param, minidom.Text):
+                    import distutils.util
+                    param_name = param.tagName
+                    param_value = param.childNodes[0].nodeValue
+                    param_userspecified = bool(distutils.util.strtobool(param.getAttribute(self.ATTR_PARAM_USERSPECIFIED)))
+                    attack.add_param_value(param_name, param_value, param_userspecified)
+
+            # Create the label from the data read
+            label = Label.Label(attack_name, float(timestamp_start), float(timestamp_end), attack_seed, attack.params,
+                                attack_note)
             self.labels.append(label)
             count_labels += 1
 

code/Core/QueryParser.py

@@ -3,24 +3,75 @@ import pyparsing as pp
 
 class QueryParser:
     def __init__(self):
+        """
+        Constructs a parser for all named queries using PyParsing.
+        """
         extractor = pp.Keyword("random") ^ pp.Keyword("first") ^ pp.Keyword("last")
-        selector = pp.Keyword("most_used") ^ pp.Keyword("least_used") ^ pp.Keyword("avg") ^ pp.Keyword("all")
-        attribute = pp.Keyword("ipaddress") ^ pp.Keyword("macaddress") ^ pp.Keyword("portnumber") ^ pp.Keyword("protocolname") ^ pp.Keyword("ttlvalue") ^ pp.Keyword("mssvalue") ^ pp.Keyword("winsize") ^ pp.Keyword("ipclass") ^ pp.Keyword("pktssent") ^ pp.Keyword("pktsreceived") ^ pp.Keyword("mss") ^ pp.Keyword("kbytesreceived") ^ pp.Keyword("kbytessent")
-        simple_selector_query = selector + pp.Suppress("(") + attribute + pp.Suppress(")")
 
-        param_selectors = pp.Keyword("ipaddress").setParseAction(pp.replaceWith("ipaddress_param")) ^ pp.Keyword("macaddress").setParseAction(pp.replaceWith("macaddress_param"))
-        operators = pp.Literal("=") ^ pp.Literal("<=") ^ pp.Literal("<") ^ pp.Literal(">=") ^ pp.Literal(">")
+        # Valid selectors - except "avg", because not all attributes can be combined with it
+        selector_no_avg = pp.Keyword("most_used") ^ pp.Keyword("least_used") ^ pp.Keyword("all")
+
+        # All attributes that cannot be combined with "avg"
+        attributes_no_avg = pp.Keyword("ipaddress") ^ pp.Keyword("macaddress") ^ pp.Keyword("portnumber") ^\
+                            pp.Keyword("protocolname") ^ pp.Keyword("winsize") ^ pp.Keyword("ipclass")
+
+        # All attributes that can be combined with "avg"
+        attributes_avg = pp.Keyword("ttlvalue") ^ pp.Keyword("mssvalue") ^\
+                         pp.Keyword("pktssent") ^ pp.Keyword("pktsreceived") ^ pp.Keyword("mss") ^\
+                         pp.Keyword("kbytesreceived") ^ pp.Keyword("kbytessent")
+
+        # Collection of all attributes for simpler specification
+        attributes_all = attributes_no_avg ^ attributes_avg
+
+        # Simple selector + attribute query, only allowing "avg" with compatible attributes
+        simple_selector_query = (selector_no_avg + pp.Suppress("(") + attributes_all + pp.Suppress(")")) ^\
+                                (pp.Keyword("avg") + pp.Suppress("(") + attributes_avg + pp.Suppress(")"))
+
+        # Selectors for parameterized queries - they are replaced in the result to avoid ambiguity
+        param_selectors = pp.Keyword("ipaddress").setParseAction(pp.replaceWith("ipaddress_param")) ^\
+                          pp.Keyword("macaddress").setParseAction(pp.replaceWith("macaddress_param"))
+
+        # All operators allowed in parameterized queries
+        operators = pp.Literal("<=") ^ pp.Literal("<") ^ pp.Literal("=") ^\
+                    pp.Literal(">=") ^ pp.Literal(">") ^ pp.CaselessLiteral("in")
+
+        # Placeholder for nesting in parameterized queries
         expr = pp.Forward()
-        comparison = pp.Group(attribute + operators + (pp.Word(pp.alphanums + ".:") ^ expr))
+
+        # Simple values for comparisons inside a parameterized query can be alphanumeric plus dot and colon
+        simple_value = pp.Word(pp.alphanums + ".:")
+
+        # Values in parameterized queries can either be simple values, or a list of them.
+        # If it's a list, we insert a "list"-token to be able to distinguish it
+        parameterized_value = simple_value ^\
+                              (pp.Suppress("[") + pp.Group(pp.Empty().addParseAction(pp.replaceWith('list')) +
+                               pp.delimitedList(simple_value)) + pp.Suppress("]"))
+
+        # One "attribute-operator-value" triplet for parameterized queries
+        comparison = pp.Group(attributes_all + operators + (parameterized_value ^ expr))
+
+        # A full parameterized query, consisting of a parameterized selector and a comma-separated list of comparisons
         parameterized_query = param_selectors + pp.Suppress("(") + pp.Group(pp.delimitedList(comparison)) + pp.Suppress(")")
-        # parameterized_query = param_selectors + pp.Suppress("(") + comparison + pp.Suppress(")")
 
+        # Combination of simple and parameterized queries
         all_selector_queries = (simple_selector_query ^ parameterized_query)
+
+        # All queries can be combined with an extractor
         extractor_selector_query = extractor + pp.Suppress("(") + all_selector_queries + pp.Suppress(")")
 
+        # Queries can be used with an extractor or without
         named_query = (extractor_selector_query ^ all_selector_queries)
+
+        # The placeholder can be replaced with any query
         expr << pp.Group(named_query)
+
+        # Make sure all queries end with a semicolon, and we're done
         self.full_query = named_query + pp.Suppress(";")
 
-    def parse_query(self, querystring):
+    def parse_query(self, querystring: str) -> pp.ParseResults:
+        """
+        Parses the passed query with a pre-constructed parser.
+        :param querystring: The named query to be executed
+        :return: A ParseResults-object, which essentially is a list of tokens
+        """
         return self.full_query.parseString(querystring)

code/Core/Statistics.py

@@ -48,7 +48,7 @@ class Statistics:
         statistics are calculated by the PCAP file processor and saved into the newly created database. Otherwise the
         statistics are gathered directly from the existing database.
 
-        :param flag_write_file: Indicates whether the statistics should be written addiotionally into a text file (True)
+        :param flag_write_file: Indicates whether the statistics should be written additionally into a text file (True)
         or not (False)
         :param flag_recalculate_stats: Indicates whether eventually existing statistics should be recalculated
         :param flag_print_statistics: Indicates whether the gathered basic statistics should be printed to the terminal
@@ -224,6 +224,11 @@ class Statistics:
         # self.stats_db.process_user_defined_query output is list of tuples, thus, we ned [0][0] to access data
 
         def count_frequncy(values_list):
+            """
+            TODO : FILL ME
+            :param values_list:
+            :return:
+            """
             values, freq_output = [], []
             for x in values_list:
                 if x in values:
@@ -515,38 +520,66 @@ class Statistics:
         return Util.handle_most_used_outputs(self.process_db_query("most_used(ipAddress)"))
 
     def get_ttl_distribution(self, ip_address: str):
+        """
+        TODO: FILL ME
+        :param ip_address:
+        :return:
+        """
         result = self.process_db_query('SELECT ttlValue, ttlCount from ip_ttl WHERE ipAddress="' + ip_address + '"')
         result_dict = {key: value for (key, value) in result}
         return result_dict
 
     def get_mss_distribution(self, ip_address: str):
+        """
+        TODO: FILL ME
+        :param ip_address:
+        :return:
+        """
         result = self.process_db_query('SELECT mssValue, mssCount from tcp_mss WHERE ipAddress="' + ip_address + '"')
         result_dict = {key: value for (key, value) in result}
         return result_dict
 
     def get_win_distribution(self, ip_address: str):
+        """
+        TODO: FILL ME
+        :param ip_address:
+        :return:
+        """
         result = self.process_db_query('SELECT winSize, winCount from tcp_win WHERE ipAddress="' + ip_address + '"')
         result_dict = {key: value for (key, value) in result}
         return result_dict
 
     def get_tos_distribution(self, ip_address: str):
+        """
+        TODO: FILL ME
+        :param ip_address:
+        :return:
+        """
         result = self.process_db_query('SELECT tosValue, tosCount from ip_tos WHERE ipAddress="' + ip_address + '"')
         result_dict = {key: value for (key, value) in result}
         return result_dict
 
     def get_ip_address_count(self):
+        """
+        TODO: FILL ME
+        :return:
+        """
         return self.process_db_query("SELECT COUNT(*) FROM ip_statistics")
 
     def get_ip_addresses(self):
+        """
+        TODO: FILL ME
+        :return:
+        """
         return self.process_db_query("SELECT ipAddress FROM ip_statistics")
 
     def get_random_ip_address(self, count: int = 1):
         """
-        :param count: The number of IP addreses to return
+        :param count: The number of IP addresses to return
         :return: A randomly chosen IP address from the dataset or iff param count is greater than one, a list of
         randomly chosen IP addresses
         """
-        ip_address_list = self.process_db_query("all(ipAddress)")
+        ip_address_list = self.process_db_query("SELECT ipAddress from ip_statistics ORDER BY ipAddress ASC")
         if count == 1:
             return random.choice(ip_address_list)
         else:
@@ -562,13 +595,45 @@ class Statistics:
         :param mac_address: the MAC address of which the IP shall be returned, if existing in DB
         :return: the IP address used in the dataset by a given MAC address
         """
-        return self.process_db_query('ipAddress(macAddress=' + mac_address + ")")
+        return self.process_db_query("SELECT DISTINCT ipAddress FROM ip_mac WHERE macAddress = '" + mac_address + "'")
 
     def get_mac_address(self, ip_address: str):
         """
         :return: The MAC address used in the dataset for the given IP address.
         """
-        return self.process_db_query('macAddress(ipAddress=' + ip_address + ")")
+        return self.process_db_query("SELECT DISTINCT macAddress from ip_mac WHERE ipAddress = '" + ip_address + "'")
+
+    def get_most_used_ttl_value(self):
+        """
+        :return: The most used TTL value.
+        """
+        return self.process_db_query("SELECT ttlValue FROM (SELECT ttlValue, SUM(ttlCount) as occ FROM ip_ttl GROUP BY "
+                                     "ttlValue) WHERE occ=(SELECT SUM(ttlCount) as occ FROM ip_ttl GROUP BY ttlValue "
+                                     "ORDER BY occ DESC LIMIT 1) ORDER BY ttlValue ASC")
+
+    def get_most_used_ip_class(self):
+        """
+        :return: The most used IP class.
+        """
+        return self.process_db_query("SELECT ipClass FROM (SELECT ipClass, COUNT(*) as occ from ip_statistics GROUP BY "
+                                     "ipClass ORDER BY occ DESC) WHERE occ=(SELECT COUNT(*) as occ from ip_statistics "
+                                     "GROUP BY ipClass ORDER BY occ DESC LIMIT 1) ORDER BY ipClass ASC")
+
+    def get_most_used_win_size(self):
+        """
+        :return: The most used window size.
+        """
+        return self.process_db_query("SELECT winSize FROM (SELECT winSize, SUM(winCount) as occ FROM tcp_win GROUP BY "
+                                     "winSize) WHERE occ=(SELECT SUM(winCount) as occ FROM tcp_win GROUP BY winSize "
+                                     "ORDER BY occ DESC LIMIT 1) ORDER BY winSize ASC")
+
+    def get_most_used_mss_value(self):
+        """
+        :return: The most used mss value.
+        """
+        return self.process_db_query("SELECT mssValue FROM (SELECT mssValue, SUM(mssCount) as occ FROM tcp_mss GROUP BY"
+                                     " mssValue) WHERE occ=(SELECT SUM(mssCount) as occ FROM tcp_mss GROUP BY mssValue "
+                                     "ORDER BY occ DESC LIMIT 1) ORDER BY mssValue ASC")
 
     def get_most_used_mss(self, ip_address: str):
         """
@@ -767,6 +832,15 @@ class Statistics:
         """
 
         def plot_distribution(query_output, title, x_label, y_label, file_ending: str):
+            """
+            TODO: FILL ME
+            :param query_output:
+            :param title:
+            :param x_label:
+            :param y_label:
+            :param file_ending:
+            :return:
+            """
             plt.gcf().clear()
             graphx, graphy = [], []
             for row in query_output:
@@ -785,6 +859,11 @@ class Statistics:
             return out
 
         def plot_ttl(file_ending: str):
+            """
+            TODO: FILL ME
+            :param file_ending:
+            :return:
+            """
             query_output = self.stats_db.process_user_defined_query(
                 "SELECT ttlValue, SUM(ttlCount) FROM ip_ttl GROUP BY ttlValue")
             title = "TTL Distribution"
@@ -794,6 +873,11 @@ class Statistics:
                 return plot_distribution(query_output, title, x_label, y_label, file_ending)
 
         def plot_mss(file_ending: str):
+            """
+            TODO: FILL ME
+            :param file_ending:
+            :return:
+            """
             query_output = self.stats_db.process_user_defined_query(
                 "SELECT mssValue, SUM(mssCount) FROM tcp_mss GROUP BY mssValue")
             title = "MSS Distribution"
@@ -803,6 +887,11 @@ class Statistics:
                 return plot_distribution(query_output, title, x_label, y_label, file_ending)
 
         def plot_win(file_ending: str):
+            """
+            TODO: FILL ME
+            :param file_ending:
+            :return:
+            """
             query_output = self.stats_db.process_user_defined_query(
                 "SELECT winSize, SUM(winCount) FROM tcp_win GROUP BY winSize")
             title = "Window Size Distribution"
@@ -812,6 +901,11 @@ class Statistics:
                 return plot_distribution(query_output, title, x_label, y_label, file_ending)
 
         def plot_protocol(file_ending: str):
+            """
+            TODO: FILL ME
+            :param file_ending:
+            :return:
+            """
             plt.gcf().clear()
             result = self.stats_db.process_user_defined_query(
                 "SELECT protocolName, SUM(protocolCount) FROM ip_protocols GROUP BY protocolName")
@@ -841,6 +935,11 @@ class Statistics:
                 print("Error plot protocol: No protocol values found!")
 
         def plot_port(file_ending: str):
+            """
+            TODO: FILL ME
+            :param file_ending:
+            :return:
+            """
             plt.gcf().clear()
             result = self.stats_db.process_user_defined_query(
                 "SELECT portNumber, SUM(portCount) FROM ip_ports GROUP BY portNumber")
@@ -862,6 +961,11 @@ class Statistics:
 
         # This distribution is not drawable for big datasets
         def plot_ip_src(file_ending: str):
+            """
+            TODO: FILL ME
+            :param file_ending:
+            :return:
+            """
             plt.gcf().clear()
             result = self.stats_db.process_user_defined_query(
                 "SELECT ipAddress, pktsSent FROM ip_statistics")
@@ -893,6 +997,11 @@ class Statistics:
 
         # This distribution is not drawable for big datasets
         def plot_ip_dst(file_ending: str):
+            """
+            TODO: FILL ME
+            :param file_ending:
+            :return:
+            """
             plt.gcf().clear()
             result = self.stats_db.process_user_defined_query(
                 "SELECT ipAddress, pktsReceived FROM ip_statistics")
@@ -923,6 +1032,15 @@ class Statistics:
             return out
 
         def plot_interval_statistics(query_output, title, x_label, y_label, file_ending: str):
+            """
+            TODO: FILL ME
+            :param query_output:
+            :param title:
+            :param x_label:
+            :param y_label:
+            :param file_ending:
+            :return:
+            """
             plt.gcf().clear()
             graphx, graphy = [], []
             for row in query_output:
@@ -948,6 +1066,11 @@ class Statistics:
             return out
 
         def plot_interval_pkt_count(file_ending: str):
+            """
+            TODO: FILL ME
+            :param file_ending:
+            :return:
+            """
             query_output = self.stats_db.process_user_defined_query(
                 "SELECT lastPktTimestamp, pktsCount FROM interval_statistics ORDER BY lastPktTimestamp")
             title = "Packet Rate"
@@ -957,6 +1080,11 @@ class Statistics:
                 return plot_interval_statistics(query_output, title, x_label, y_label, file_ending)
 
         def plot_interval_ip_src_ent(file_ending: str):
+            """
+            TODO: FILL ME
+            :param file_ending:
+            :return:
+            """
             query_output = self.stats_db.process_user_defined_query(
                 "SELECT lastPktTimestamp, ipSrcEntropy FROM interval_statistics ORDER BY lastPktTimestamp")
             title = "Source IP Entropy"
@@ -966,6 +1094,11 @@ class Statistics:
                 return plot_interval_statistics(query_output, title, x_label, y_label, file_ending)
 
         def plot_interval_ip_dst_ent(file_ending: str):
+            """
+            TODO: FILL ME
+            :param file_ending:
+            :return:
+            """
             query_output = self.stats_db.process_user_defined_query(
                 "SELECT lastPktTimestamp, ipDstEntropy FROM interval_statistics ORDER BY lastPktTimestamp")
             title = "Destination IP Entropy"
@@ -975,6 +1108,11 @@ class Statistics:
                 return plot_interval_statistics(query_output, title, x_label, y_label, file_ending)
 
         def plot_interval_new_ip(file_ending: str):
+            """
+            TODO: FILL ME
+            :param file_ending:
+            :return:
+            """
             query_output = self.stats_db.process_user_defined_query(
                 "SELECT lastPktTimestamp, newIPCount FROM interval_statistics ORDER BY lastPktTimestamp")
             title = "IP Novelty Distribution"
@@ -984,6 +1122,11 @@ class Statistics:
                 return plot_interval_statistics(query_output, title, x_label, y_label, file_ending)
 
         def plot_interval_new_port(file_ending: str):
+            """
+            TODO: FILL ME
+            :param file_ending:
+            :return:
+            """
             query_output = self.stats_db.process_user_defined_query(
                 "SELECT lastPktTimestamp, newPortCount FROM interval_statistics ORDER BY lastPktTimestamp")
             title = "Port Novelty Distribution"
@@ -993,6 +1136,11 @@ class Statistics:
                 return plot_interval_statistics(query_output, title, x_label, y_label, file_ending)
 
         def plot_interval_new_ttl(file_ending: str):
+            """
+            TODO: FILL ME
+            :param file_ending:
+            :return:
+            """
             query_output = self.stats_db.process_user_defined_query(
                 "SELECT lastPktTimestamp, newTTLCount FROM interval_statistics ORDER BY lastPktTimestamp")
             title = "TTL Novelty Distribution"
@@ -1002,6 +1150,11 @@ class Statistics:
                 return plot_interval_statistics(query_output, title, x_label, y_label, file_ending)
 
         def plot_interval_new_tos(file_ending: str):
+            """
+            TODO: FILL ME
+            :param file_ending:
+            :return:
+            """
             query_output = self.stats_db.process_user_defined_query(
                 "SELECT lastPktTimestamp, newToSCount FROM interval_statistics ORDER BY lastPktTimestamp")
             title = "ToS Novelty Distribution"
@@ -1011,6 +1164,11 @@ class Statistics:
                 return plot_interval_statistics(query_output, title, x_label, y_label, file_ending)
 
         def plot_interval_new_win_size(file_ending: str):
+            """
+            TODO: FILL ME
+            :param file_ending:
+            :return:
+            """
             query_output = self.stats_db.process_user_defined_query(
                 "SELECT lastPktTimestamp, newWinSizeCount FROM interval_statistics ORDER BY lastPktTimestamp")
             title = "Window Size Novelty Distribution"
@@ -1020,6 +1178,11 @@ class Statistics:
                 return plot_interval_statistics(query_output, title, x_label, y_label, file_ending)
 
         def plot_interval_new_mss(file_ending: str):
+            """
+            TODO: FILL ME
+            :param file_ending:
+            :return:
+            """
             query_output = self.stats_db.process_user_defined_query(
                 "SELECT lastPktTimestamp, newMSSCount FROM interval_statistics ORDER BY lastPktTimestamp")
             title = "MSS Novelty Distribution"
@@ -1029,6 +1192,11 @@ class Statistics:
                 return plot_interval_statistics(query_output, title, x_label, y_label, file_ending)
 
         def plot_interval_ip_dst_cum_ent(file_ending: str):
+            """
+            TODO: FILL ME
+            :param file_ending:
+            :return:
+            """
             plt.gcf().clear()
             result = self.stats_db.process_user_defined_query(
                 "SELECT lastPktTimestamp, ipDstCumEntropy FROM interval_statistics ORDER BY lastPktTimestamp")
@@ -1061,6 +1229,11 @@ class Statistics:
                 return out
 
         def plot_interval_ip_src_cum_ent(file_ending: str):
+            """
+            TODO: FILL ME
+            :param file_ending:
+            :return:
+            """
             plt.gcf().clear()
 
             result = self.stats_db.process_user_defined_query(
@@ -1486,45 +1659,76 @@ class Statistics:
             return out
 
         ttl_out_path = plot_ttl('.' + file_format)
+        print(".", end="", flush=True)
         mss_out_path = plot_mss('.' + file_format)
+        print(".", end="", flush=True)
         win_out_path = plot_win('.' + file_format)
+        print(".", end="", flush=True)
         protocol_out_path = plot_protocol('.' + file_format)
+        print(".", end="", flush=True)
         plot_interval_pktCount = plot_interval_pkt_count('.' + file_format)
+        print(".", end="", flush=True)
         if entropy:
             plot_interval_ip_src_ent = plot_interval_ip_src_ent('.' + file_format)
+            print(".", end="", flush=True)
             plot_interval_ip_dst_ent = plot_interval_ip_dst_ent('.' + file_format)
+            print(".", end="", flush=True)
             plot_interval_ip_src_cum_ent = plot_interval_ip_src_cum_ent('.' + file_format)
+            print(".", end="", flush=True)
             plot_interval_ip_dst_cum_ent = plot_interval_ip_dst_cum_ent('.' + file_format)
+            print(".", end="", flush=True)
         plot_interval_new_ip = plot_interval_new_ip('.' + file_format)
+        print(".", end="", flush=True)
         plot_interval_new_port = plot_interval_new_port('.' + file_format)
+        print(".", end="", flush=True)
         plot_interval_new_ttl = plot_interval_new_ttl('.' + file_format)
+        print(".", end="", flush=True)
         plot_interval_new_tos = plot_interval_new_tos('.' + file_format)
+        print(".", end="", flush=True)
         plot_interval_new_win_size = plot_interval_new_win_size('.' + file_format)
+        print(".", end="", flush=True)
         plot_interval_new_mss = plot_interval_new_mss('.' + file_format)
+        print(".", end="", flush=True)
         plot_hist_indegree_out = plot_histogram_degree("inDegree", "Histogram - Ingoing degree per IP Address",
             "Ingoing degree", "_plot-Histogram Ingoing Degree per IP" + file_format)
+        print(".", end="", flush=True)
         plot_hist_outdegree_out = plot_histogram_degree("outDegree", "Histogram - Outgoing degree per IP Address",
             "Outgoing degree", "_plot-Histogram Outgoing Degree per IP" + file_format)
+        print(".", end="", flush=True)
         plot_hist_overalldegree_out = plot_histogram_degree("overallDegree", "Histogram - Overall degree per IP Address",
             "Overall degree", "_plot-Histogram Overall Degree per IP" + file_format)
+        print(".", end="", flush=True)
         plot_hist_pkts_per_connection_out = plot_comm_histogram("pktsCount", "Histogram - Number of exchanged packets per connection",
             "Number of packets", "_plot-Histogram PktCount per Connection" + "." + file_format)
+        print(".", end="", flush=True)
         plot_hist_avgpkts_per_commint_out = plot_comm_histogram("avgIntervalPktCount", "Histogram - Average number of exchanged packets per communication interval",
             "Average number of packets", "_plot-Histogram Avg PktCount per Interval per Connection" + "." + file_format)
+        print(".", end="", flush=True)
         plot_hist_avgtime_betw_commints_out = plot_comm_histogram("avgTimeBetweenIntervals", "Histogram - Average time between communication intervals in seconds",
             "Average time between intervals", "_plot-Histogram Avg Time Between Intervals per Connection" + "." + file_format)
+        print(".", end="", flush=True)
         plot_hist_avg_int_time_per_connection_out = plot_comm_histogram("avgIntervalTime", "Histogram - Average duration of a communication interval in seconds",
             "Average interval time", "_plot-Histogram Avg Interval Time per Connection" + "." + file_format)
+        print(".", end="", flush=True)
         plot_hist_total_comm_duration_out = plot_comm_histogram("totalConversationDuration", "Histogram - Total communication duration in seconds",
             "Duration", "_plot-Histogram Communication Duration per Connection" + "." + file_format)
+        print(".", end="", flush=True)
         plot_out_degree = plot_out_degree('.' + file_format)
+        print(".", end="", flush=True)
         plot_in_degree = plot_in_degree('.' + file_format)
+        print(".", end="", flush=True)
         plot_overall_degree = plot_overall_degree('.' + file_format)
+        print(".", end="", flush=True)
         plot_packets_per_connection_out = plot_packets_per_connection('.' + file_format)
+        print(".", end="", flush=True)
         plot_avg_pkts_per_comm_interval_out = plot_avg_pkts_per_comm_interval('.' + file_format)
+        print(".", end="", flush=True)
         plot_avg_time_between_comm_interval_out = plot_avg_time_between_comm_interval('.' + file_format)
+        print(".", end="", flush=True)
         plot_avg_comm_interval_time_out = plot_avg_comm_interval_time("." + file_format)
+        print(".", end="", flush=True)
         plot_total_comm_duration_out = plot_total_comm_duration("." + file_format)
+        print(" done.")
 
         # Time consuming plot
         # port_out_path = plot_port('.' + format)
@@ -1566,6 +1770,8 @@ class Statistics:
         print("\nNew database has been generated, printing statistics summary... ")
         total_packet_count = self.get_packet_count()
         pdu_count = self.process_db_query("SELECT SUM(pktCount) FROM unrecognized_pdus")
+        if pdu_count is None:
+            pdu_count = 0
         pdu_share = pdu_count / total_packet_count * 100
         last_pdu_timestamp = self.process_db_query(
             "SELECT MAX(timestampLastOccurrence) FROM unrecognized_pdus")

code/Core/StatsDatabase.py

@@ -1,6 +1,6 @@
 import os.path
 import random as rnd
-import re
+import typing
 import sqlite3
 import sys
 
@@ -25,6 +25,10 @@ def dict_gen(curs: sqlite3.Cursor):
             yield dict(zip(field_names, row))
 
 
+class QueryExecutionException(Exception):
+    pass
+
+
 class StatsDatabase:
     def __init__(self, db_path: str):
         """
@@ -168,18 +172,48 @@ class StatsDatabase:
         field_types = self.get_field_types('ip_mac', 'ip_ttl', 'ip_ports', 'ip_protocols', 'ip_statistics', 'ip_mac')
         conditions = []
         for key, op, value in param_op_val:
+            # Check whether the value is not a simple value, but another query (or list)
             if isinstance(value, pp.ParseResults):
-                # If we have another query instead of a direct value, execute and replace it
-                value = self._execute_query_list(value)[0][0]
+                if value[0] == "list":
+                    # We have a list, cut the token off and use the remaining elements
+                    value = value[1:]
+
+                    # Lists can only be used with "in"
+                    if op is not "in":
+                        raise QueryExecutionException("List values require the usage of the 'in' operator!")
+                else:
+                    # If we have another query instead of a direct value, execute and replace it
+                    rvalue = self._execute_query_list(value)
+
+                    # Do we have a comparison operator with a multiple-result query?
+                    if op is not "in" and value[0] in ['most_used', 'least_used', 'all', 'ipaddress_param',
+                                                       'macaddress_param']:
+                        raise QueryExecutionException("The extractor '" + value[0] +
+                                                      "' may return more than one result!")
+
+                    # Make value contain a simple list with the results of the query
+                    value = map(lambda x: str(x[0]), rvalue)
+            else:
+                # Make sure value is a list now to simplify handling
+                value = [value]
+
             # this makes sure that TEXT fields are queried by strings,
             # e.g. ipAddress=192.168.178.1 --is-converted-to--> ipAddress='192.168.178.1'
             if field_types.get(key) == 'TEXT':
-                if not str(value).startswith("'") and not str(value).startswith('"'):
-                    value = "'" + value + "'"
+                def ensure_string(x):
+                    if not str(x).startswith("'") and not str(x).startswith('"'):
+                        return "'" + x + "'"
+                    else:
+                        return x
+                value = map(ensure_string, value)
+
+            # If we have more than one value, join them together, separated by commas
+            value = ",".join(map(str, value))
+
             # this replacement is required to remove ambiguity in SQL query
             if key == 'ipAddress':
                 key = 'ip_mac.ipAddress'
-            conditions.append(key + op + str(value))
+            conditions.append(key + " " + op + " (" + str(value) + ")")
 
         where_clause = " AND ".join(conditions)
         query += where_clause
@@ -230,6 +264,9 @@ class StatsDatabase:
         "least_used.winsize": "SELECT winSize FROM (SELECT winSize, SUM(winCount) as occ FROM tcp_win GROUP BY "
                               "winSize) WHERE occ=(SELECT SUM(winCount) as occ FROM tcp_win GROUP BY winSize "
                               "ORDER BY occ ASC LIMIT 1) ORDER BY winSize ASC",
+        "least_used.ipclass": "SELECT ipClass FROM (SELECT ipClass, COUNT(*) as occ from ip_statistics GROUP BY "
+                             "ipClass ORDER BY occ DESC) WHERE occ=(SELECT COUNT(*) as occ from ip_statistics "
+                             "GROUP BY ipClass ORDER BY occ ASC LIMIT 1) ORDER BY ipClass ASC",
         "avg.pktsreceived": "SELECT avg(pktsReceived) from ip_statistics",
         "avg.pktssent": "SELECT avg(pktsSent) from ip_statistics",
         "avg.kbytesreceived": "SELECT avg(kbytesReceived) from ip_statistics",
@@ -241,27 +278,32 @@ class StatsDatabase:
         "all.mss": "SELECT DISTINCT mssValue from tcp_mss ORDER BY mssValue ASC",
         "all.macaddress": "SELECT DISTINCT macAddress from ip_mac ORDER BY macAddress ASC",
         "all.portnumber": "SELECT DISTINCT portNumber from ip_ports ORDER BY portNumber ASC",
-        "all.protocolname": "SELECT DISTINCT protocolName from ip_protocols ORDER BY protocolName ASC"}
+        "all.protocolname": "SELECT DISTINCT protocolName from ip_protocols ORDER BY protocolName ASC",
+        "all.winsize": "SELECT DISTINCT winSize FROM tcp_win ORDER BY winSize ASC",
+        "all.ipclass": "SELECT DISTINCT ipClass FROM ip_statistics ORDER BY ipClass ASC"}
 
     def _execute_query_list(self, query_list):
         """
         Recursively executes a list of named queries. They are of the following form:
-        ['macaddress_param', [['ipaddress', '=', ['most_used', 'ipaddress']]]]
+        ['macaddress_param', [['ipaddress', 'in', ['most_used', 'ipaddress']]]]
         :param query_list: The query statement list obtained from the query parser
         :return: The result of the query (either a single result or a list).
         """
         if query_list[0] == "random":
-            return rnd.choice(self._execute_query_list(query_list[1:]))
+            return [rnd.choice(self._execute_query_list(query_list[1:]))]
         elif query_list[0] == "first":
-            return self._execute_query_list(query_list[1:])[0]
+            return [self._execute_query_list(query_list[1:])[0]]
         elif query_list[0] == "last":
-            return self._execute_query_list(query_list[1:])[-1]
+            return [self._execute_query_list(query_list[1:])[-1]]
         elif query_list[0] == "macaddress_param":
             return self.named_query_parameterized("macaddress", query_list[1])
         elif query_list[0] == "ipaddress_param":
             return self.named_query_parameterized("ipaddress", query_list[1])
         else:
             query = self.named_queries.get(query_list[0] + "." + query_list[1])
+            if query is None:
+                raise QueryExecutionException("The requested query '" + query_list[0] + "(" + query_list[1] +
+                                              ")' was not found in the internal query list!")
             self.cursor.execute(str(query))
             last_result = self.cursor.fetchall()
             return last_result
@@ -318,23 +360,23 @@ class StatsDatabase:
                 for i in range(0, len(result)):
                     print(str(self.cursor.description[i][0]) + ": " + str(result[i]))
             else:
-                self._print_query_results(query_string_in, result)
+                self._print_query_results(query_string_in, result if isinstance(result, list) else [result])
 
         return result
 
-    def _print_query_results(self, query_string_in: str, result):
+    def _print_query_results(self, query_string_in: str, result: typing.List[typing.Union[str, float, int]]) -> None:
         """
         Prints the results of a query.
         Based on http://stackoverflow.com/a/20383011/3017719.
 
         :param query_string_in: The query the results belong to
-        :param result: The results of the query
+        :param result: The list of query results
         """
         # Print number of results according to type of result
-        if isinstance(result, list):
-            print("Query returned " + str(len(result)) + " records:\n")
-        else:
+        if len(result) == 1:
             print("Query returned 1 record:\n")
+        else:
+            print("Query returned " + str(len(result)) + " records:\n")
 
         # Print query results
         if query_string_in.lstrip().upper().startswith(
@@ -343,8 +385,13 @@ class StatsDatabase:
             columns = []
             tavnit = '|'
             separator = '+'
-            for cd in self.cursor.description:
-                widths.append(len(cd) + 10)
+            for index, cd in enumerate(self.cursor.description):
+                max_col_length = 0
+                if len(result) > 0:
+                    max_col_length = max(list(map(lambda x:
+                                                  len(str(x[index] if len(self.cursor.description) > 1 else x)),
+                                                  result)))
+                widths.append(max(len(cd[0]), max_col_length))
                 columns.append(cd[0])
             for w in widths:
                 tavnit += " %-" + "%ss |" % (w,)
@@ -352,11 +399,8 @@ class StatsDatabase:
             print(separator)
             print(tavnit % tuple(columns))
             print(separator)
-            if isinstance(result, list):
-                for row in result:
-                    print(tavnit % row)
-            else:
-                print(tavnit % result)
+            for row in result:
+                print(tavnit % row)
             print(separator)
         else:
             print(result)

code/ID2TLib/Label.py

@@ -3,19 +3,22 @@ import functools
 
 @functools.total_ordering
 class Label:
-    def __init__(self, attack_name, timestamp_start, timestamp_end, attack_note=""):
+    def __init__(self, attack_name, timestamp_start, timestamp_end, seed, parameters, attack_note=""):
         """
         Creates a new attack label
 
         :param attack_name: The name of the associated attack
         :param timestamp_start: The timestamp as unix time of the first attack packet
         :param timestamp_end: The timestamp as unix time of the last attack packet
+        :param parameters: The list of parameters used to run the attack
         :param attack_note: A note associated to the attack (optional)
         """
         self.attack_name = attack_name
         self.timestamp_start = timestamp_start
         self.timestamp_end = timestamp_end
+        self.seed = seed
         self.attack_note = attack_note
+        self.parameters = parameters
 
     def __eq__(self, other):
         return self.timestamp == other.timestamp

code/ID2TLib/Memcached.py

@@ -0,0 +1,18 @@
+import scapy.packet as packet
+import scapy.fields as field
+import scapy.layers.inet as inet
+
+
+memcached_port = 11211
+
+
+class Memcached_Request(packet.Packet):
+    namez = "Memcached UDP request packet"
+    fields_desc = [field.ShortField("RequestID", 0),
+                   field.ShortField("SequenceNumber", 0),
+                   field.ShortField("DatagramCount", 1),
+                   field.ShortField("Reserved", 0),
+                   field.StrField("Request", "\r\n")]
+
+
+packet.bind_layers(inet.UDP, Memcached_Request, dport=11211)

code/ID2TLib/PcapFile.py

@@ -27,7 +27,7 @@ class PcapFile(object):
 
     def get_file_hash(self):
         """
-        Returns the hash for the loaded PCAP file. The hash is calculated bsaed on:
+        Returns the hash for the loaded PCAP file. The hash is calculated based on:
 
         - the file size in bytes
         - the first 224*40000 bytes of the file

code/ID2TLib/SMB2.py

@@ -2,7 +2,7 @@ import scapy.packet as packet
 import scapy.fields as field
 import scapy.layers.netbios as netbios
 
-
+# TODO: FILL ME
 class SMB2_SYNC_Header(packet.Packet):
     namez = "SMB2Negociate Protocol Response Header"
     fields_desc = [field.StrFixedLenField("Start", "\xfeSMB", 4),
@@ -21,7 +21,7 @@ class SMB2_SYNC_Header(packet.Packet):
                    field.LELongField("Signature2", 0)]
 
 
-# No Support of Security Buffer , Padding or Dialect Revision 0x0311
+# TODO: FILL ME Description was not correct anymore
 class SMB2_Negotiate_Protocol_Response(packet.Packet):
     namez = "SMB2Negociate Protocol Response"
     fields_desc = [field.LEShortField("StructureSize", 65),

code/ID2TLib/TestLibrary.py

@@ -3,10 +3,13 @@ import os
 import random as rnd
 
 import ID2TLib.Utility as Util
-
+# Directory of test resource files
 test_resource_dir = Util.TEST_DIR
+# Path to reference pcap
 test_pcap = Util.TEST_DIR + "reference_1998.pcap"
+# Several ips in the reference pcap
 test_pcap_ips = ["10.0.2.15", "52.85.173.182"]
+# Empty array for testing purposes
 test_pcap_empty = []
 
 """

code/ID2TLib/Utility.py

@@ -20,21 +20,28 @@ TEST_DIR = RESOURCE_DIR + "test/"
 OUT_DIR = None
 MISC_OUT_FILES = {}
 
+# List of common operation systems
 platforms = {"win7", "win10", "winxp", "win8.1", "macos", "linux", "win8", "winvista", "winnt", "win2000"}
+# Distribution of common operation systems
 platform_probability = {"win7": 48.43, "win10": 27.99, "winxp": 6.07, "win8.1": 6.07, "macos": 5.94, "linux": 3.38,
                         "win8": 1.35, "winvista": 0.46, "winnt": 0.31}
 
+# List of no-ops
 x86_nops = {b'\x90', b'\xfc', b'\xfd', b'\xf8', b'\xf9', b'\xf5', b'\x9b'}
+# List of pseudo no-ops (includes ops which won't change the state e.g. read access)
 x86_pseudo_nops = {b'\x97', b'\x96', b'\x95', b'\x93', b'\x92', b'\x91', b'\x99', b'\x4d', b'\x48', b'\x47', b'\x4f',
                    b'\x40', b'\x41', b'\x37', b'\x3f', b'\x27', b'\x2f', b'\x46', b'\x4e', b'\x98', b'\x9f', b'\x4a',
                    b'\x44', b'\x42', b'\x43', b'\x49', b'\x4b', b'\x45', b'\x4c', b'\x60', b'\x0e', b'\x1e', b'\x50',
                    b'\x55', b'\x53', b'\x51', b'\x57', b'\x52', b'\x06', b'\x56', b'\x54', b'\x16', b'\x58', b'\x5d',
                    b'\x5b', b'\x59', b'\x5f', b'\x5a', b'\x5e', b'\xd6'}
+# Characters which result in operational behaviour (e.g. FTPWinaXeExploit.py)
 forbidden_chars = [b'\x00', b'\x0a', b'\x0d']
 
+# Used in get_attacker_config
 attacker_port_mapping = {}
+# Used in get_attacker_config
 attacker_ttl_mapping = {}
-
+# Identifier for attacks
 generic_attack_names = {"attack", "exploit"}
 
 
@@ -67,7 +74,7 @@ def get_interval_pps(complement_interval_pps, timestamp):
     for row in complement_interval_pps:
         if timestamp <= row[0]:
             return row[1]
-    return complement_interval_pps[-1][1]  # in case the timstamp > capture max timestamp
+    return complement_interval_pps[-1][1]  # in case the timestamp > capture max timestamp
 
 
 def get_nth_random_element(*element_list):

code/Test/ID2TAttackTest.py

@@ -50,9 +50,8 @@ class ID2TAttackTest(unittest.TestCase):
             Lib.rename_test_result_files(controller, caller_function, attack_sub_dir, test_sub_dir)
 
     def temporal_efficiency_test(self, attack_args, time_limit=15, factor=1, seed=None, cleanup=True,
-                                 pcap=Lib.test_pcap,
-                                 flag_write_file=False, flag_recalculate_stats=False, flag_print_statistics=False,
-                                 attack_sub_dir=True, test_sub_dir=True):
+                                 pcap=Lib.test_pcap, flag_write_file=False, flag_recalculate_stats=False,
+                                 flag_print_statistics=False, attack_sub_dir=True, test_sub_dir=True):
         """
         Runs the attack with given aruments and monitors time efficiency.
 

code/Test/efficiency_testing.py

@@ -5,65 +5,50 @@ import Test.ID2TAttackTest as Test
 
 
 class EfficiencyTests(Test.ID2TAttackTest):
-    def test_SMBLoris_10_000(self):
-        self.temporal_efficiency_test([['SMBLorisAttack', 'attackers.count=30', 'packets.per-second=8.0']],
-                                      time_limit=15, factor=10000)
+    def test_SMBLoris(self):
+        self.temporal_efficiency_test([['SMBLorisAttack', 'attackers.count=4', 'packets.per-second=8.0']],
+                                      time_limit=1.5, factor=1000)
 
-    def test_SMBLoris_100_000(self):
-        self.temporal_efficiency_test([['SMBLorisAttack', 'attackers.count=30', 'packets.per-second=98']],
-                                      time_limit=150, factor=100000)
-
-    def test_SMBScan_10_000(self):
+    def test_SMBScan(self):
         self.temporal_efficiency_test([['SMBScanAttack', 'ip.src=192.168.178.1',
-                                        'ip.dst=192.168.178.10-192.168.197.145']], time_limit=15, factor=10000)
-
-    def test_SMBScan_100_000(self):
-        self.temporal_efficiency_test([['SMBScanAttack', 'ip.src=192.168.178.1', 'ip.dst=192.168.0.1-192.168.195.76']],
-                                      time_limit=150, factor=100000)
+                                        'ip.dst=192.168.178.10-192.168.179.253']], time_limit=1.5, factor=1000)
 
-    def test_SMBScan_hosting_10_000(self):
+    def test_SMBScan_hosting(self):
         self.temporal_efficiency_test([['SMBScanAttack', 'ip.src=192.168.178.1',
-                                        'ip.dst=192.168.178.10-192.168.181.241',
-                                        'hosting.ip=192.168.178.10-192.168.181.241']], time_limit=15, factor=10000)
-
-    def test_SMBScan_hosting_100_000(self):
-        self.temporal_efficiency_test([['SMBScanAttack', 'ip.src=192.168.178.1', 'ip.dst=192.168.178.10-192.168.217.25',
-                                        'hosting.ip=192.168.178.10-192.168.217.25']], time_limit=150, factor=100000)
+                                        'ip.dst=192.168.178.10-192.168.178.109',
+                                        'hosting.ip=192.168.178.10-192.168.178.109']], time_limit=1.5, factor=1000)
 
     @mock.patch('ID2TLib.Utility.get_rnd_bytes', side_effect=Lib.get_bytes)
     @mock.patch('ID2TLib.Utility.get_rnd_x86_nop', side_effect=Lib.get_x86_nop)
     def test_FTPExploit(self, mock_get_rnd_x86_nop, mock_get_rnd_bytes):
         self.temporal_efficiency_test([['FTPWinaXeExploit', 'ip.src=192.168.178.1', 'ip.dst=192.168.178.10']],
-                                      time_limit=15, factor=10000)
+                                      time_limit=1.5, factor=1000)
 
-    def test_PortscanAttack_open_10_000(self):
-        self.temporal_efficiency_test([['PortscanAttack', 'ip.src=192.168.178.1', 'port.open=80']], time_limit=15,
-                                      factor=10000)
+    def test_PortscanAttack_open(self):
+        self.temporal_efficiency_test([['PortscanAttack', 'ip.src=192.168.178.1', 'port.open=80']], time_limit=1.5,
+                                      factor=1000)
 
-    def test_PortscanAttack_close_10_000(self):
-        self.temporal_efficiency_test([['PortscanAttack', 'ip.src=192.168.178.1', 'port.open=20']], time_limit=15,
-                                      factor=10000)
+    def test_PortscanAttack_close(self):
+        self.temporal_efficiency_test([['PortscanAttack', 'ip.src=192.168.178.1', 'port.open=20']], time_limit=1.5,
+                                      factor=1000)
 
-    def test_SQLi_10_000(self):
-        # FIXME: sometimes it takes 15.34028493521018 instead of the normal 7.150923313737726 seconds
-        self.temporal_efficiency_test([['SQLiAttack', 'ip.dst=192.168.0.1']], time_limit=15, factor=10000)
+    def test_SQLi(self):
+        self.temporal_efficiency_test([['SQLiAttack', 'ip.dst=192.168.0.1']], time_limit=1.5, factor=1000)
 
-    def test_Joomla_10_000(self):
-        self.temporal_efficiency_test([['JoomlaRegPrivExploit', 'ip.src=192.168.178.1']], time_limit=15, factor=10000)
+    def test_Joomla(self):
+        self.temporal_efficiency_test([['JoomlaRegPrivExploit', 'ip.src=192.168.178.1']], time_limit=1.5, factor=1000)
 
-    def test_SalityBotnet_10_000(self):
-        self.temporal_efficiency_test([['SalityBotnet']], time_limit=15, factor=10000)
+    def test_SalityBotnet(self):
+        self.temporal_efficiency_test([['SalityBotnet']], time_limit=1.5, factor=1000)
 
     @mock.patch('Attack.BaseAttack.BaseAttack.write_attack_pcap', side_effect=Lib.write_attack_pcap)
-    def test_DDoS_10_000(self, mock_write_attack_pcap):
-        # TODO: update attack args, when DDoS gets refactored
-        self.temporal_efficiency_test([['DDoSAttack', 'attackers.count=100', 'packets.per-second=95',
-                                        'attack.duration=150']], time_limit=15, factor=10000)
+    def test_DDoS(self, mock_write_attack_pcap):
+        self.temporal_efficiency_test([['DDoSAttack', 'attackers.count=10', 'packets.per-second=95',
+                                        'attack.duration=10']], time_limit=1.5, factor=1000)
 
-    @mock.patch('Attack.BaseAttack.BaseAttack.write_attack_pcap', side_effect=Lib.write_attack_pcap)
-    def test_DDoS_100_000(self, mock_write_attack_pcap):
-        # TODO: update attack args, when DDoS gets refactored
-        self.temporal_efficiency_test([['DDoSAttack', 'attackers.count=1000', 'packets.per-second=950',
-                                        'attack.duration=300']], time_limit=150, factor=100000)
+    def test_MS17(self):
+        self.temporal_efficiency_test([['MS17Scan', 'ip.src=192.168.178.1']], time_limit=1.5, factor=1000)
 
-        # TODO: add temporal efficiency test(s) for EternalBlue and MS17
+    # FIXME: improve EternalBlue efficiency
+    #def test_EternalBlue(self):
+    #    self.temporal_efficiency_test([['EternalBlue']], time_limit=1.5, factor=1000)

code/Test/test_MemcrashedSpooferAttack.py

@@ -0,0 +1,15 @@
+import Test.ID2TAttackTest as Test
+
+sha_default = "065e7de040fb41bcaad81b705fb70f3e07807f0d3dc1efe0f437929ff33b49f8"
+sha_ips_not_in_pcap = "46c015fde4509227ee70fbe1557fe0efd3ac76abf58e00dcbbcf09d0b950fb5f"
+
+
+class UnitTestMemcrashedSpooferAttack(Test.ID2TAttackTest):
+    def test_memcrashed_spoofer_default(self):
+        self.checksum_test([['MemcrashedSpooferAttack']], sha_default)
+
+    def test_memcrashed_spoofer_ips_not_in_pcap(self):
+        self.checksum_test([['MemcrashedSpooferAttack', 'ip.src=1.1.1.1', 'ip.dst=2.2.2.2']], sha_ips_not_in_pcap)
+
+    def test_memcrashed_spoofer_order(self):
+        self.order_test([['MemcrashedSpooferAttack']])

code/Test/test_NamedQueries.py

@@ -0,0 +1,140 @@
+import unittest
+import pyparsing
+
+import ID2TLib.TestLibrary as Lib
+import Core.Controller as Ctrl
+
+controller = Ctrl.Controller(pcap_file_path=Lib.test_pcap, do_extra_tests=False, non_verbose=True)
+controller.load_pcap_statistics(flag_write_file=False, flag_recalculate_stats=True, flag_print_statistics=False)
+
+ip_addresses = ["10.0.2.15",      "104.83.103.45",  "13.107.21.200",  "131.253.61.100", "172.217.23.142",
+                "172.217.23.174", "192.168.33.254", "204.79.197.200", "23.51.123.27",   "35.161.3.50",
+                "52.11.17.245",   "52.34.37.177",   "52.39.210.199",  "52.41.250.141",  "52.85.173.182",
+                "54.149.74.139",  "54.187.98.195",  "54.192.44.108",  "54.192.44.177",  "72.247.178.113",
+                "72.247.178.67",  "93.184.220.29"]
+
+allWinSize = [0,     822,   1330,  5082,  8192,  9900,  27060, 35657, 39917, 47030, 50782, 51310, 52202, 52740, 55062,
+              56492, 58520, 59950, 59980, 61380, 62788, 62810, 62811, 62906, 63056, 63076, 63086, 63151, 63261, 63350,
+              63370, 63400, 63409, 63456, 63516, 63547, 63552, 63572, 63603, 63628, 63655, 63663, 63675, 63686, 63706,
+              63839, 63842, 63886, 63893, 63917, 63954, 63963, 63982, 63991, 64000, 64005, 64088, 64110, 64148, 64165,
+              64177, 64189, 64194, 64198, 64209, 64230, 64240, 65535]
+
+leastUsedWinASize = [822,   1330,  5082,  9900,  27060, 35657, 39917, 47030, 50782, 51310, 52202, 52740, 55062, 56492,
+                     58520, 59950, 59980, 61380, 63056, 63963, 63982, 64000, 64005, 64198, 64230]
+
+allPort = [53,    80,    443,   49157, 49160, 49163, 49164, 49165, 49166, 49167, 49168, 49169, 49170, 49171, 49172,
+           49173, 49174, 49175, 49176, 49177, 49178, 49179, 49180, 49181, 49182, 49183, 49184, 49185, 49186, 49187,
+           49188, 49189, 49190, 49191, 49192, 49193, 49194, 49195, 49196, 49197, 49247, 49323, 49470, 49636, 49695,
+           49798, 49927, 49935, 49945, 50262, 50836, 50968, 51143, 51166, 51350, 51451, 51669, 51713, 52033, 52135,
+           52399, 52520, 52644, 52697, 52743, 52786, 52964, 52981, 53059, 53234, 53461, 53691, 53708, 53745, 53836,
+           54049, 54446, 54593, 54598, 54652, 54663, 54717, 54853, 54930, 55004, 55018, 55119, 55125, 55299, 55310,
+           55463, 55650, 55667, 55752, 55843, 55851, 56146, 56325, 56567, 56589, 56750, 57049, 57179, 57275, 57520,
+           57653, 57840, 57957, 57991, 58401, 58440, 58645, 58797, 58814, 58905, 58913, 58943, 59380, 59408, 59461,
+           59467, 59652, 59660, 59718, 59746, 59844, 60006, 60209, 60414, 60422, 60659, 60696, 60708, 60756, 60827,
+           60840, 61181, 61300, 61592, 61718, 61738, 61769, 61807, 62412, 62428, 62447, 62490, 62625, 62626, 62664,
+           63425, 64096, 64121, 64137, 64252, 64334, 64337, 64479, 64509, 64637, 64807, 64811, 65448, 65487]
+
+
+class UnitTestNamedQueries(unittest.TestCase):
+    def test_most_used_ipaddress(self):
+        self.assertEqual(controller.statistics.process_db_query('most_used(ipaddress)'), '10.0.2.15')
+
+    def test_most_used_macaddress(self):
+        self.assertEqual(controller.statistics.process_db_query('most_used(macaddress)'), '52:54:00:12:35:02')
+
+    def test_most_used_portnumber(self):
+        self.assertEqual(controller.statistics.process_db_query('most_used(portnumber)'), 443)
+
+    def test_most_used_protocolname(self):
+        self.assertEqual(controller.statistics.process_db_query('most_used(protocolname)'), 'IPv4')
+
+    def test_most_used_ttlvalue(self):
+        self.assertEqual(controller.statistics.process_db_query('most_used(ttlvalue)'), 64)
+
+    def test_most_used_mssvalue(self):
+        self.assertEqual(controller.statistics.process_db_query('most_used(mssvalue)'), 1460)
+
+    def test_most_used_winsize(self):
+        self.assertEqual(controller.statistics.process_db_query('most_used(winsize)'), 65535)
+
+    def test_most_used_ipclass(self):
+        self.assertEqual(controller.statistics.process_db_query('most_used(ipclass)'), 'A')
+
+    def test_least_used_ipaddress(self):
+        self.assertEqual(controller.statistics.process_db_query('least_used(ipaddress)'), '72.247.178.113')
+
+    def test_least_used_macaddress(self):
+        self.assertEqual(controller.statistics.process_db_query('least_used(macaddress)'), '08:00:27:a3:83:43')
+
+    def test_least_used_portnumber(self):
+        self.assertEqual(controller.statistics.process_db_query('least_used(portnumber)'), [58645, 59844])
+
+    def test_least_used_protocolname(self):
+        self.assertEqual(controller.statistics.process_db_query('least_used(protocolname)'), 'UDP')
+
+    def test_least_used_ttlvalue(self):
+        self.assertEqual(controller.statistics.process_db_query('least_used(ttlvalue)'), 255)
+
+    def test_least_used_mssvalue(self):
+        self.assertEqual(controller.statistics.process_db_query('least_used(mssvalue)'), 1460)
+
+    def least_used_winsize(self):
+        self.assertEqual(controller.statistics.process_db_query('least_used(winsize)'), leastUsedWinASize)
+
+    def test_least_used_ipclass(self):
+        self.assertEqual(controller.statistics.process_db_query('least_used(ipclass)'), ['A-private', 'C', 'C-private'])
+
+    def test_avg_pktsreceived(self):
+        self.assertEqual(controller.statistics.process_db_query('avg(pktsreceived)'), 90.36363636363636)
+
+    def test_avg_pktssent(self):
+        self.assertEqual(controller.statistics.process_db_query('avg(pktssent)'), 90.36363636363636)
+
+    def test_avg_kbytesreceived(self):
+        self.assertEqual(controller.statistics.process_db_query('avg(kbytesreceived)'), 30.289683948863637)
+
+    def test_avg_kbytessent(self):
+        self.assertEqual(controller.statistics.process_db_query('avg(kbytessent)'), 30.289683948863637)
+
+    def test_avg_ttlvalue(self):
+        self.assertEqual(controller.statistics.process_db_query('avg(ttlvalue)'), 75.08695652173913)
+
+    def test_avg_mss(self):
+        self.assertEqual(controller.statistics.process_db_query('avg(mss)'), 1460.0)
+
+    def test_avg_ipaddress(self):
+        with self.assertRaises(pyparsing.ParseException):
+            controller.statistics.process_db_query('avg(ipAddress)')
+
+    def test_all_ipaddress(self):
+        self.assertEqual(controller.statistics.process_db_query('all(ipaddress)'), ip_addresses)
+
+    def test_all_ttlvalue(self):
+        self.assertEqual(controller.statistics.process_db_query('all(ttlvalue)'), [64, 128, 255])
+
+    def test_all_mss(self):
+        self.assertEqual(controller.statistics.process_db_query('all(mss)'), 1460)
+
+    def test_all_macaddress(self):
+        self.assertEqual(controller.statistics.process_db_query('all(macaddress)'), ['08:00:27:a3:83:43',
+                                                                                     '52:54:00:12:35:02'])
+
+    def test_all_portnumber(self):
+        self.assertEqual(controller.statistics.process_db_query('all(portnumber)'), allPort)
+
+    def test_all_protocolname(self):
+        self.assertEqual(controller.statistics.process_db_query('all(protocolname)'), ['IPv4', 'TCP', 'UDP'])
+
+    def test_all_winsize(self):
+        self.assertEqual(controller.statistics.process_db_query('all(winSize)'),
+                         allWinSize)
+
+    def test_all_ipclass(self):
+        self.assertEqual(controller.statistics.process_db_query('all(ipClass)'),
+                         ['A', 'A-private', 'B', 'C', 'C-private'])
+
+    def test_is_query_named_query(self):
+        self.assertTrue(controller.statistics.is_query('least_used(ipaddress)'))
+
+    def test_is_query_no_string(self):
+        self.assertFalse(controller.statistics.is_query(42))

code/Test/test_NestedNamedQueries.py

@@ -0,0 +1,38 @@
+import unittest
+import sqlite3
+import pyparsing
+
+import ID2TLib.TestLibrary as Lib
+import Core.Controller as Ctrl
+
+controller = Ctrl.Controller(pcap_file_path=Lib.test_pcap, do_extra_tests=False, non_verbose=True)
+controller.load_pcap_statistics(flag_write_file=False, flag_recalculate_stats=True, flag_print_statistics=False)
+
+
+class UnitTestNestedNamedQueries(unittest.TestCase):
+    def test_nested_query(self):
+        self.assertEqual(controller.statistics.process_db_query('macaddress(ipaddress in most_used(ipaddress))'),
+                         '08:00:27:a3:83:43')
+        self.assertEqual(controller.statistics.process_db_query('macaddress(ipaddress in least_used(ipaddress))'),
+                         '52:54:00:12:35:02')
+        self.assertEqual(controller.statistics.process_db_query('ipaddress(macaddress in least_used(macaddress))'),
+                         '10.0.2.15')
+        self.assertEqual(controller.statistics.process_db_query('ipaddress(macaddress in 08:00:27:a3:83:43)'),
+                         '10.0.2.15')
+        self.assertEqual(controller.statistics.process_db_query('ipaddress(macaddress in [08:00:27:a3:83:43])'),
+                         '10.0.2.15')
+        self.assertEqual(controller.statistics.process_db_query('ipaddress(macaddress in most_used(macaddress))'),
+                         ['104.83.103.45', '13.107.21.200', '131.253.61.100', '172.217.23.142', '172.217.23.174',
+                          '192.168.33.254', '204.79.197.200', '23.51.123.27', '35.161.3.50', '52.11.17.245',
+                          '52.34.37.177', '52.39.210.199', '52.41.250.141', '52.85.173.182', '54.149.74.139',
+                          '54.187.98.195', '54.192.44.108', '54.192.44.177', '72.247.178.113', '72.247.178.67',
+                          '93.184.220.29'])
+
+        # semantically incorrect query
+        with self.assertRaises(sqlite3.OperationalError):
+            controller.statistics.process_db_query('ipaddress(ipaddress in most_used(macaddress))')
+
+        # syntactically incorrect query
+        with self.assertRaises(pyparsing.ParseException):
+            controller.statistics.process_db_query('ipaddress(macaddress in '
+                                                   'most_used(macaddress(ipaddress in least_used(ipaddress))))')

code/Test/test_Queries.py

@@ -1,238 +0,0 @@
-import random
-import unittest
-
-import Core.Controller as Ctrl
-import ID2TLib.TestLibrary as Test
-
-# TODO: improve coverage
-
-controller = Ctrl.Controller(pcap_file_path=Test.test_pcap, do_extra_tests=False, non_verbose=True)
-controller.load_pcap_statistics(flag_write_file=False, flag_recalculate_stats=True, flag_print_statistics=False)
-
-file_information = [('Pcap file path', Test.test_pcap),
-                    ('Total packet count', 1998, 'packets'),
-                    ("Recognized packets", 1988, "packets"),
-                    ("Unrecognized packets", 10, "PDUs"), ("% Recognized packets", 99.49949949949949, "%"),
-                    ("% Unrecognized packets", 0.5005005005005005, "%"),
-                    ("Last unknown PDU", '1970-01-01 01:07:39.604899'),
-                    ('Capture duration', '25.4294414520264', 'seconds'),
-                    ('Capture start', '\t1970-01-01 01:01:45.647675'),
-                    ('Capture end', '\t1970-01-01 01:08:10.102034')]
-
-file_statistics = [('Avg. packet rate', 78.57034301757812, 'packets/sec'), ('Avg. packet size', 0.0, 'kbytes'),
-                   ('Avg. packets sent', 90.0, 'packets'), ('Avg. bandwidth in', 9.5290, 'kbit/s'),
-                   ('Avg. bandwidth out', 9.5290, 'kbit/s')]
-
-ip_addresses = ["10.0.2.15", "104.83.103.45", "13.107.21.200", "131.253.61.100", "172.217.23.142",
-                "172.217.23.174", "192.168.33.254", "204.79.197.200", "23.51.123.27", "35.161.3.50",
-                "52.11.17.245", "52.34.37.177", "52.39.210.199", "52.41.250.141", "52.85.173.182",
-                "54.149.74.139", "54.187.98.195", "54.192.44.108", "54.192.44.177", "72.247.178.113",
-                "72.247.178.67", "93.184.220.29"]
-ports = [53, 80, 443, 49157, 49160, 49163, 49164, 49165, 49166, 49167, 49168, 49169, 49170, 49171, 49172, 49173, 49174,
-         49175, 49176, 49177, 49178, 49179, 49180, 49181, 49182, 49183, 49184, 49185, 49186, 49187, 49188, 49189, 49190,
-         49191, 49192, 49193, 49194, 49195, 49196, 49197, 49247, 49323, 49470, 49636, 49695, 49798, 49927, 49935, 49945,
-         50262, 50836, 50968, 51143, 51166, 51350, 51451, 51669, 51713, 52033, 52135, 52399, 52520, 52644, 52697, 52743,
-         52786, 52964, 52981, 53059, 53234, 53461, 53691, 53708, 53745, 53836, 54049, 54446, 54593, 54598, 54652, 54663,
-         54717, 54853, 54930, 55004, 55018, 55119, 55125, 55299, 55310, 55463, 55650, 55667, 55752, 55843, 55851, 56146,
-         56325, 56567, 56589, 56750, 57049, 57179, 57275, 57520, 57653, 57840, 57957, 57991, 58401, 58440, 58645, 58797,
-         58814, 58905, 58913, 58943, 59380, 59408, 59461, 59467, 59652, 59660, 59718, 59746, 59844, 60006, 60209, 60414,
-         60422, 60659, 60696, 60708, 60756, 60827, 60840, 61181, 61300, 61592, 61718, 61738, 61769, 61807, 62412, 62428,
-         62447, 62490, 62625, 62626, 62664, 63425, 64096, 64121, 64137, 64252, 64334, 64337, 64479, 64509, 64637, 64807,
-         64811, 65448, 65487]
-
-
-class TestQueries(unittest.TestCase):
-    def test_get_file_information(self):
-        self.assertEqual(controller.statistics.get_file_information(), file_information)
-
-    def test_get_general_file_statistics(self):
-        file_stats = controller.statistics.get_general_file_statistics()
-        file_stats[3] = ('Avg. bandwidth in', round(file_stats[3][1], 4), 'kbit/s')
-        file_stats[4] = ('Avg. bandwidth out', round(file_stats[4][1], 4), 'kbit/s')
-        self.assertEqual(file_stats, file_statistics)
-
-    def test_get_capture_duration(self):
-        self.assertEqual(controller.statistics.get_capture_duration(), '25.4294414520264')
-
-    def test_get_pcap_timestamp_start(self):
-        self.assertEqual(controller.statistics.get_pcap_timestamp_start(), '1970-01-01 01:01:45.647675')
-
-    def test_get_pcap_timestamp_end(self):
-        self.assertEqual(controller.statistics.get_pcap_timestamp_end(), '1970-01-01 01:08:10.102034')
-
-    def test_get_pps_sent_1(self):
-        self.assertEqual(controller.statistics.get_pps_sent(ip_address='72.247.178.67'), 0)
-
-    def test_get_pps_sent_2(self):
-        self.assertEqual(controller.statistics.get_pps_sent(ip_address='10.0.2.15'), 32)
-
-    def test_get_pps_received_1(self):
-        self.assertEqual(controller.statistics.get_pps_received(ip_address='72.247.178.67'), 0)
-
-    def test_get_pps_received_2(self):
-        self.assertEqual(controller.statistics.get_pps_received(ip_address='10.0.2.15'), 46)
-
-    def test_get_packet_count(self):
-        self.assertEqual(controller.statistics.get_packet_count(), 1998)
-
-    def test_get_most_used_ip_address(self):
-        self.assertEqual(controller.statistics.get_most_used_ip_address(), '10.0.2.15')
-
-    def test_get_ttl_distribution_1(self):
-        self.assertEqual(controller.statistics.get_ttl_distribution(ip_address='72.247.178.67'), {64: 5})
-
-    def test_get_ttl_distribution_2(self):
-        self.assertEqual(controller.statistics.get_ttl_distribution(ip_address='10.0.2.15'), {128: 817})
-
-    def test_get_mss_distribution_1(self):
-        self.assertEqual(controller.statistics.get_mss_distribution(ip_address='72.247.178.67'), {1460: 1})
-
-    def test_get_mss_distribution_2(self):
-        self.assertEqual(controller.statistics.get_mss_distribution(ip_address='10.0.2.15'), {1460: 36})
-
-    def test_get_win_distribution_1(self):
-        self.assertEqual(controller.statistics.get_win_distribution(ip_address='72.247.178.67'), {65535: 5})
-
-    def test_get_tos_distribution_1(self):
-        self.assertEqual(controller.statistics.get_tos_distribution(ip_address='72.247.178.67'), {0: 5})
-
-    def test_get_tos_distribution_2(self):
-        self.assertEqual(controller.statistics.get_tos_distribution(ip_address='10.0.2.15'), {0: 817})
-
-    def test_get_ip_address_count(self):
-        self.assertEqual(controller.statistics.get_ip_address_count(), 22)
-
-    def test_get_ip_addresses(self):
-        self.assertEqual(controller.statistics.get_ip_addresses(), ip_addresses)
-
-    def test_get_random_ip_address(self):
-        random.seed(5)
-        self.assertEqual(controller.statistics.get_random_ip_address(), '72.247.178.113')
-
-    def test_get_random_ip_address_count_2(self):
-        random.seed(5)
-        self.assertEqual(controller.statistics.get_random_ip_address(2), ['72.247.178.113', '23.51.123.27'])
-
-    def test_get_mac_address_1(self):
-        self.assertEqual(controller.statistics.get_mac_address(ip_address='72.247.178.67'), '52:54:00:12:35:02')
-
-    def test_get_mac_address_2(self):
-        self.assertEqual(controller.statistics.get_mac_address(ip_address='10.0.2.15'), '08:00:27:a3:83:43')
-
-    def test_get_most_used_mss(self):
-        self.assertEqual(controller.statistics.get_most_used_mss(ip_address='10.0.2.15'), 1460)
-
-    def test_get_most_used_ttl(self):
-        self.assertEqual(controller.statistics.get_most_used_ttl(ip_address='10.0.2.15'), 128)
-
-    def test_is_query_no_string(self):
-        self.assertFalse(controller.statistics.is_query(42))
-
-    def test_is_query_named_query(self):
-        self.assertTrue(controller.statistics.is_query('least_used(ipaddress)'))
-
-    def test_is_query_standard_query(self):
-        self.assertTrue(controller.statistics.is_query('SELECT * from ip_statistics'))
-
-    def test_calculate_standard_deviation(self):
-        self.assertEqual(controller.statistics.calculate_standard_deviation([1, 1, 2, 3, 5, 8, 13, 21]),
-                         6.609652033201143)
-
-    def test_calculate_entropy(self):
-        self.assertEqual(controller.statistics.calculate_entropy([1, 1, 2, 3, 5, 8, 13, 21]), 2.371389165297016)
-
-    def test_calculate_entropy_normalized(self):
-        self.assertEqual(controller.statistics.calculate_entropy([1, 1, 2, 3, 5, 8, 13, 21], normalized=True),
-                         (2.371389165297016, 0.7904630550990053))
-
-    def test_calculate_complement_packet_rates_1(self):
-        cpr = controller.statistics.calculate_complement_packet_rates(0)[0:9]
-        self.assertEqual(cpr, [(186.418564, 0), (186.418824, 0), (186.419346, 0), (186.445361, 0),
-                               (186.46954399999998, 0), (186.476234, 0), (186.477304, 0), (186.48606999999998, 0),
-                               (186.486761, 0)])
-
-    def test_calculate_complement_packet_rates_2(self):
-        cpr = controller.statistics.calculate_complement_packet_rates(42)[0:9]
-        self.assertEqual(cpr, [(186.418564, 41), (186.418824, 42), (186.419346, 42), (186.445361, 42),
-                               (186.46954399999998, 42), (186.476234, 42), (186.477304, 42), (186.48606999999998, 42),
-                               (186.486761, 42)])
-
-    # NAMED QUERY TESTS
-    def test_most_used_ipaddress(self):
-        self.assertEqual(controller.statistics.process_db_query('most_used(ipaddress)'), '10.0.2.15')
-
-    def test_most_used_macaddress(self):
-        self.assertEqual(controller.statistics.process_db_query('most_used(macaddress)'), '52:54:00:12:35:02')
-
-    def test_most_used_portnumber(self):
-        self.assertEqual(controller.statistics.process_db_query('most_used(portnumber)'), 443)
-
-    def test_most_used_protocolname(self):
-        self.assertEqual(controller.statistics.process_db_query('most_used(protocolname)'), 'IPv4')
-
-    def test_most_used_ttlvalue(self):
-        self.assertEqual(controller.statistics.process_db_query('most_used(ttlvalue)'), 64)
-
-    def test_most_used_mssvalue(self):
-        self.assertEqual(controller.statistics.process_db_query('most_used(mssvalue)'), 1460)
-
-    def test_most_used_winsize(self):
-        self.assertEqual(controller.statistics.process_db_query('most_used(winsize)'), 65535)
-
-    def test_most_used_ipclass(self):
-        self.assertEqual(controller.statistics.process_db_query('most_used(ipclass)'), 'A')
-
-    def test_least_used_ipaddress(self):
-        self.assertEqual(controller.statistics.process_db_query('least_used(ipaddress)'), '72.247.178.113')
-
-    def test_least_used_macaddress(self):
-        self.assertEqual(controller.statistics.process_db_query('least_used(macaddress)'), '08:00:27:a3:83:43')
-
-    def test_least_used_portnumber(self):
-        self.assertEqual(controller.statistics.process_db_query('least_used(portnumber)'), [58645, 59844])
-
-    def test_least_used_protocolname(self):
-        self.assertEqual(controller.statistics.process_db_query('least_used(protocolname)'), 'UDP')
-
-    def test_least_used_ttlvalue(self):
-        self.assertEqual(controller.statistics.process_db_query('least_used(ttlvalue)'), 255)
-
-    def test_avg_pktsreceived(self):
-        self.assertEqual(controller.statistics.process_db_query('avg(pktsreceived)'), 90.36363636363636)
-
-    def test_avg_pktssent(self):
-        self.assertEqual(controller.statistics.process_db_query('avg(pktssent)'), 90.36363636363636)
-
-    def test_avg_kbytesreceived(self):
-        self.assertEqual(controller.statistics.process_db_query('avg(kbytesreceived)'), 30.289683948863637)
-
-    def test_avg_kbytessent(self):
-        self.assertEqual(controller.statistics.process_db_query('avg(kbytessent)'), 30.289683948863637)
-
-    def test_avg_ttlvalue(self):
-        self.assertEqual(controller.statistics.process_db_query('avg(ttlvalue)'), 75.08695652173913)
-
-    def test_avg_mss(self):
-        self.assertEqual(controller.statistics.process_db_query('avg(mss)'), 1460.0)
-
-    def test_all_ipaddress(self):
-        self.assertEqual(controller.statistics.process_db_query('all(ipaddress)'), ip_addresses)
-
-    def test_all_ttlvalue(self):
-        self.assertEqual(controller.statistics.process_db_query('all(ttlvalue)'), [64, 128, 255])
-
-    def test_all_mss(self):
-        self.assertEqual(controller.statistics.process_db_query('all(mss)'), 1460)
-
-    def test_all_macaddress(self):
-        self.assertEqual(controller.statistics.process_db_query('all(macaddress)'), ['08:00:27:a3:83:43',
-                                                                                     '52:54:00:12:35:02'])
-
-    def test_all_portnumber(self):
-        self.assertEqual(controller.statistics.process_db_query('all(portnumber)'), ports)
-
-    def test_all_protocolname(self):
-        self.assertEqual(controller.statistics.process_db_query('all(protocolname)'), ['IPv4', 'TCP', 'UDP'])
-
-    def test_nested_query(self):
-        self.assertEqual(controller.statistics.process_db_query('macaddress(ipaddress=most_used(ipaddress))'), '08:00:27:a3:83:43')

code/Test/test_SQLQueries.py

@@ -0,0 +1,45 @@
+import unittest
+import sqlite3
+
+import ID2TLib.TestLibrary as Lib
+import Core.Controller as Ctrl
+
+controller = Ctrl.Controller(pcap_file_path=Lib.test_pcap, do_extra_tests=False, non_verbose=True)
+controller.load_pcap_statistics(flag_write_file=False, flag_recalculate_stats=True, flag_print_statistics=False)
+
+
+class UnitTestSqlQueries(unittest.TestCase):
+    def test_apostrophe(self):
+        query = "Select ipAddress from ip_Statistics where pktsSent = '5'"
+        query2 = "Select ipAddress from ip_Statistics where pktsSent = 5"
+        self.assertEqual(controller.statistics.stats_db.process_db_query(query),
+                         controller.statistics.stats_db.process_db_query(query2))
+
+    def test_parenthesis(self):
+        query = "Select (ipAddress) from (ip_Statistics) where (pktsSent) = (2 + (3))"
+        self.assertEqual("72.247.178.67", controller.statistics.stats_db.process_db_query(query))
+
+    def test_noResult(self):
+        query = "Select ipAddress from ip_statistics where ipaddress = 'abc'"
+        self.assertEqual([], controller.statistics.stats_db.process_db_query(query))
+
+    def test_severalOperator(self):
+        query1 = "Select ipAddress from ip_Statistics where pktsSent = '5'"
+        query2 = "Select ipAddress from ip_Statistics where pktsSent < '5'"
+        query3 = "Select ipAddress from ip_Statistics where pktsSent <= '5'"
+        query4 = "Select ipAddress from ip_Statistics where pktsSent > '356'"
+        query5 = "Select ipAddress from ip_Statistics where pktsSent >= '356'"
+
+        self.assertEqual("72.247.178.67", controller.statistics.stats_db.process_db_query(query1))
+        self.assertEqual("72.247.178.113", controller.statistics.stats_db.process_db_query(query2))
+        self.assertEqual(["72.247.178.67", "72.247.178.113"], controller.statistics.stats_db.process_db_query(query3))
+        self.assertEqual("10.0.2.15", controller.statistics.stats_db.process_db_query(query4))
+        self.assertEqual(["10.0.2.15", "172.217.23.174"], controller.statistics.stats_db.process_db_query(query5))
+
+        # compare of tables with different dimension
+        with self.assertRaises(sqlite3.OperationalError):
+            controller.statistics.stats_db.process_db_query('Select ipAddress from ip_Statistics where pktsSent'
+                                                            '= (Select * from ip_Statistics)')
+
+    def test_is_query_standard_query(self):
+        self.assertTrue(controller.statistics.is_query('SELECT * from ip_statistics'))

code/Test/test_internalQueries.py

@@ -0,0 +1,148 @@
+import unittest
+import random
+
+import ID2TLib.TestLibrary as Lib
+import Core.Controller as Ctrl
+
+controller = Ctrl.Controller(pcap_file_path=Lib.test_pcap, do_extra_tests=False, non_verbose=True)
+controller.load_pcap_statistics(flag_write_file=False, flag_recalculate_stats=True, flag_print_statistics=False)
+
+ipAddresses = ['10.0.2.15', '104.83.103.45', '13.107.21.200', '131.253.61.100', '172.217.23.142', '172.217.23.174',
+               '192.168.33.254', '204.79.197.200', '23.51.123.27', '35.161.3.50', '52.11.17.245', '52.34.37.177',
+               '52.39.210.199', '52.41.250.141', '52.85.173.182', '54.149.74.139', '54.187.98.195', '54.192.44.108',
+               '54.192.44.177', '72.247.178.113', '72.247.178.67', '93.184.220.29']
+
+
+class UnitTestInternalQueries(unittest.TestCase):
+    # FILE METAINFORMATION TESTS
+    def test_get_file_information(self):
+        self.assertEqual(controller.statistics.get_file_information(),
+                         [('Pcap file path', Lib.test_pcap),
+                          ('Total packet count', 1998, 'packets'),
+                          ("Recognized packets", 1988, "packets"),
+                          ("Unrecognized packets", 10, "PDUs"), ("% Recognized packets", 99.49949949949949, "%"),
+                          ("% Unrecognized packets", 0.5005005005005005, "%"),
+                          ("Last unknown PDU", '1970-01-01 01:07:39.604899'),
+                          ('Capture duration', '25.4294414520264', 'seconds'),
+                          ('Capture start', '\t1970-01-01 01:01:45.647675'),
+                          ('Capture end', '\t1970-01-01 01:08:10.102034')])
+
+    def test_get_packet_count(self):
+        self.assertEqual(controller.statistics.get_packet_count(), 1998)
+
+    def test_get_capture_duration(self):
+        self.assertEqual(controller.statistics.get_capture_duration(), '25.4294414520264')
+
+    def test_get_pcap_timestamp_start(self):
+        self.assertEqual(controller.statistics.get_pcap_timestamp_start(), '1970-01-01 01:01:45.647675')
+
+    def test_get_pcap_timestamp_end(self):
+        self.assertEqual(controller.statistics.get_pcap_timestamp_end(), '1970-01-01 01:08:10.102034')
+
+    # FIXME: This seems to be the only testcase where float values differ slightly between macOS and Linux
+    def test_get_general_file_statistics(self):
+        file_stats = controller.statistics.get_general_file_statistics()
+        self.assertEqual(file_stats[0], ('Avg. packet rate', 78.57034301757812, 'packets/sec'))
+        self.assertEqual(file_stats[1], ('Avg. packet size', 0.0, 'kbytes'))
+        self.assertEqual(file_stats[2], ('Avg. packets sent', 90.0, 'packets'))
+        self.assertEqual(file_stats[3][0], 'Avg. bandwidth in')
+        self.assertAlmostEqual(file_stats[3][1], 9.529013633728027, places=5)
+        self.assertEqual(file_stats[3][2], 'kbit/s')
+        self.assertEqual(file_stats[4][0], 'Avg. bandwidth out')
+        self.assertAlmostEqual(file_stats[4][1], 9.529013633728027, places=5)
+
+    # INTERNAL QUERY TESTS
+    def test_get_ip_address_count(self):
+        self.assertEqual(controller.statistics.get_ip_address_count(), 22)
+
+    def test_get_ip_addresses(self):
+        self.assertEqual(controller.statistics.get_ip_addresses(), ipAddresses)
+
+    def test_get_most_used_ip_address(self):
+        self.assertEqual(controller.statistics.get_most_used_ip_address(), '10.0.2.15')
+
+    def test_get_random_ip_address(self):
+        random.seed(5)
+        self.assertEqual(controller.statistics.get_random_ip_address(), '72.247.178.113')
+
+    def test_get_random_ip_address_count_2(self):
+        random.seed(5)
+        self.assertEqual(controller.statistics.get_random_ip_address(2), ['72.247.178.113', '23.51.123.27'])
+
+    def test_get_ip_address_from_mac(self):
+        self.assertEqual(controller.statistics.get_ip_address_from_mac('08:00:27:a3:83:43'), '10.0.2.15')
+
+    def test_get_mac_address_1(self):
+        self.assertEqual(controller.statistics.get_mac_address(ip_address='72.247.178.67'), '52:54:00:12:35:02')
+
+    def test_get_mac_address_2(self):
+        self.assertEqual(controller.statistics.get_mac_address(ip_address='10.0.2.15'), '08:00:27:a3:83:43')
+
+    def test_get_most_used_mss(self):
+        self.assertEqual(controller.statistics.get_most_used_mss(ip_address='10.0.2.15'), 1460)
+
+    def test_get_most_used_ttl(self):
+        self.assertEqual(controller.statistics.get_most_used_ttl(ip_address='10.0.2.15'), 128)
+
+    def test_get_pps_sent_1(self):
+        self.assertEqual(controller.statistics.get_pps_sent(ip_address='72.247.178.67'), 0)
+
+    def test_get_pps_sent_2(self):
+        self.assertEqual(controller.statistics.get_pps_sent(ip_address='10.0.2.15'), 32)
+
+    def test_get_pps_sent_wrong_input(self):
+        # wrong input parameter
+        with self.assertRaises(TypeError):
+            self.assertEqual(controller.statistics.get_pps_sent('08:00:27:a3:83:43'), 32)
+
+    def test_get_pps_received_1(self):
+        self.assertEqual(controller.statistics.get_pps_received(ip_address='72.247.178.67'), 0)
+
+    def test_get_pps_received_2(self):
+        self.assertEqual(controller.statistics.get_pps_received(ip_address='10.0.2.15'), 46)
+
+    def test_get_ttl_distribution_1(self):
+        self.assertEqual(controller.statistics.get_ttl_distribution(ip_address='72.247.178.67'), {64: 5})
+
+    def test_get_ttl_distribution_2(self):
+        self.assertEqual(controller.statistics.get_ttl_distribution(ip_address='10.0.2.15'), {128: 817})
+
+    def test_get_mss_distribution_1(self):
+        self.assertEqual(controller.statistics.get_mss_distribution(ip_address='72.247.178.67'), {1460: 1})
+
+    def test_get_mss_distribution_2(self):
+        self.assertEqual(controller.statistics.get_mss_distribution(ip_address='10.0.2.15'), {1460: 36})
+
+    def test_get_win_distribution_1(self):
+        self.assertEqual(controller.statistics.get_win_distribution(ip_address='72.247.178.67'), {65535: 5})
+
+    def test_get_tos_distribution_1(self):
+        self.assertEqual(controller.statistics.get_tos_distribution(ip_address='72.247.178.67'), {0: 5})
+
+    def test_get_tos_distribution_2(self):
+        self.assertEqual(controller.statistics.get_tos_distribution(ip_address='10.0.2.15'), {0: 817})
+
+    # INTERNAL HELPER-FUNCTION TESTS
+    def test_calculate_standard_deviation(self):
+        self.assertEqual(controller.statistics.calculate_standard_deviation([1, 1, 2, 3, 5, 8, 13, 21]),
+                         6.609652033201143)
+
+    def test_calculate_entropy(self):
+        self.assertEqual(controller.statistics.calculate_entropy([1, 1, 2, 3, 5, 8, 13, 21]), 2.371389165297016)
+
+    def test_calculate_entropy_normalized(self):
+        self.assertEqual(controller.statistics.calculate_entropy([1, 1, 2, 3, 5, 8, 13, 21], normalized=True),
+                         (2.371389165297016, 0.7904630550990053))
+
+    def test_calculate_complement_packet_rates_1(self):
+        cpr = controller.statistics.calculate_complement_packet_rates(0)[0:9]
+        self.assertEqual(cpr, [(186.418564, 0), (186.418824, 0), (186.419346, 0), (186.445361, 0),
+                               (186.46954399999998, 0), (186.476234, 0), (186.477304, 0), (186.48606999999998, 0),
+                               (186.486761, 0)])
+
+    def test_calculate_complement_packet_rates_2(self):
+        cpr = controller.statistics.calculate_complement_packet_rates(42)[0:9]
+        self.assertEqual(cpr, [(186.418564, 41), (186.418824, 42), (186.419346, 42), (186.445361, 42),
+                               (186.46954399999998, 42), (186.476234, 42), (186.477304, 42),
+                               (186.48606999999998, 42),
+                               (186.486761, 42)])

code_boost/src/cxx/pcap_processor.cpp

@@ -97,8 +97,8 @@ std::string pcap_processor::merge_pcaps(const std::string pcap_path) {
             }
             iterator_base++;
         }
-    }    
-    
+    }
+
     // This may happen if the base PCAP is smaller than the attack PCAP
     // In this case append the remaining packets of the attack PCAP
     for (; iterator_attack != sniffer_attack.end(); iterator_attack++) {
@@ -121,8 +121,8 @@ void pcap_processor::collect_statistics() {
         std::cout << "Loading pcap..." << std::endl;
         FileSniffer sniffer(filePath);
         FileSniffer snifferOverview(filePath);
-        
-        SnifferIterator i = sniffer.begin();                
+
+        SnifferIterator i = sniffer.begin();
         std::chrono::microseconds currentPktTimestamp;
 
         // Save timestamp of first packet
@@ -135,20 +135,20 @@ void pcap_processor::collect_statistics() {
         std::chrono::microseconds firstTimestamp = stats.getTimestampFirstPacket();
 
         // An empty loop to know the capture duration, then choose a suitable time interval
-        SnifferIterator lastpkt; 
+        SnifferIterator lastpkt;
         for (SnifferIterator j = snifferOverview.begin(); j != snifferOverview.end(); ++j, ++totalPackets) {lastpkt = j;}
 
-        std::chrono::microseconds lastTimestamp = lastpkt->timestamp();                  
+        std::chrono::microseconds lastTimestamp = lastpkt->timestamp();
         std::chrono::microseconds captureDuration = lastTimestamp - firstTimestamp;
         if(captureDuration.count()<=0){
-            std::cout<<"ERROR: PCAP file is empty!"<<"\n";
+            std::cout << "ERROR: PCAP file is empty!" << std::endl;
             return;
         }
         long timeInterval_microsec = captureDuration.count() / timeIntervalsNum;
         std::chrono::duration<int, std::micro> timeInterval(timeInterval_microsec);
         std::chrono::microseconds barrier = timeInterval;
 
-        std::cout << "\n";
+        std::cout << std::endl;
         std::chrono::system_clock::time_point lastPrinted = std::chrono::system_clock::now();
 
         // Iterate over all packets and collect statistics
@@ -178,13 +178,18 @@ void pcap_processor::collect_statistics() {
             }
         }
 
-        std::cout << "\n";
-        
+        std::cout << "\rInspected packets: ";
+        std::cout << "100.0% (" << totalPackets << "/" << totalPackets << ")" << std::endl;
+
         // Save timestamp of last packet into statistics
         stats.setTimestampLastPacket(currentPktTimestamp);
 
         // Create the communication interval statistics from the gathered communication intervals within every extended conversation statistic
         stats.createCommIntervalStats();
+
+        if(hasUnrecognized) {
+            std::cout << "Unrecognized PDUs detected: Check 'unrecognized_pdus' table!" << std::endl;
+        }
     }
 }
 
@@ -212,7 +217,7 @@ void pcap_processor::process_packets(const Packet &pkt) {
     const PDU::PDUType pdu_l3_type = pdu_l3->pdu_type();
     std::string ipAddressSender;
     std::string ipAddressReceiver;
-    
+
     // PDU is IPv4
     if (pdu_l3_type == PDU::PDUType::IP) {
         const IP &ipLayer = (const IP &) *pdu_l3;
@@ -235,7 +240,7 @@ void pcap_processor::process_packets(const Packet &pkt) {
         // Assign IP Address to MAC Address
         stats.assignMacAddress(ipAddressSender, macAddressSender);
         stats.assignMacAddress(ipAddressReceiver, macAddressReceiver);
-        
+
     } // PDU is IPv6
     else if (pdu_l3_type == PDU::PDUType::IPv6) {
         const IPv6 &ipLayer = (const IPv6 &) *pdu_l3;
@@ -257,10 +262,7 @@ void pcap_processor::process_packets(const Packet &pkt) {
         stats.assignMacAddress(ipAddressReceiver, macAddressReceiver);
     } //PDU is unrecognized
     else {
-        if(!hasUnrecognized) {
-            std::cerr << "Unrecognized PDUs detected: Check 'unrecognized_pdus' table!" << std::endl;
-            hasUnrecognized = true;
-        }
+        hasUnrecognized = true;
 
         EthernetII eth = (const EthernetII &) *pdu_l2;
         Tins::Timestamp ts = pkt.timestamp();
@@ -273,8 +275,8 @@ void pcap_processor::process_packets(const Packet &pkt) {
     const PDU *pdu_l4 = pdu_l3->inner_pdu();
     if (pdu_l4 != 0) {
         // Protocol distribution - layer 4
-        PDU::PDUType p = pdu_l4->pdu_type();  
-        
+        PDU::PDUType p = pdu_l4->pdu_type();
+
         // Check for IPv4: payload
         if (pdu_l3_type == PDU::PDUType::IP) {
             stats.checkPayload(pdu_l4);
@@ -285,7 +287,7 @@ void pcap_processor::process_packets(const Packet &pkt) {
             
             // Check TCP checksum
             if (pdu_l3_type == PDU::PDUType::IP) {
-              stats.checkTCPChecksum(ipAddressSender, ipAddressReceiver, tcpPkt);
+                stats.checkTCPChecksum(ipAddressSender, ipAddressReceiver, tcpPkt);
             }
 
             stats.incrementProtocolCount(ipAddressSender, "TCP");
@@ -299,7 +301,7 @@ void pcap_processor::process_packets(const Packet &pkt) {
             int win = tcpPkt.window();
             stats.incrementWinCount(ipAddressSender, win);
 
-            try {                                                                
+            try {
                 int val = tcpPkt.mss();
 
                 // MSS distribution
@@ -309,7 +311,7 @@ void pcap_processor::process_packets(const Packet &pkt) {
             }
             stats.incrementPortCount(ipAddressSender, tcpPkt.sport(), ipAddressReceiver, tcpPkt.dport(), "TCP");
             stats.increasePortByteCount(ipAddressSender, tcpPkt.sport(), ipAddressReceiver, tcpPkt.dport(), sizeCurrentPacket, "TCP");
-            
+
           // UDP Packet
         } else if (p == PDU::PDUType::UDP) {
             const UDP udpPkt = (const UDP &) *pdu_l4;
@@ -318,7 +320,6 @@ void pcap_processor::process_packets(const Packet &pkt) {
             stats.incrementPortCount(ipAddressSender, udpPkt.sport(), ipAddressReceiver, udpPkt.dport(), "UDP");
             stats.increasePortByteCount(ipAddressSender, udpPkt.sport(), ipAddressReceiver, udpPkt.dport(), sizeCurrentPacket, "UDP");
             stats.addConvStatExt(ipAddressSender,udpPkt.sport(), ipAddressReceiver, udpPkt.dport(), "UDP", pkt.timestamp());
-          
         } else if (p == PDU::PDUType::ICMP) {
             stats.incrementProtocolCount(ipAddressSender, "ICMP");
             stats.increaseProtocolByteCount(ipAddressSender, "ICMP", sizeCurrentPacket);

resources/install_dependencies.sh

@@ -59,7 +59,7 @@ install_pkg_ubuntu()
 
 install_pkg_darwin()
 {
-    BREW_PKGS="cmake python coreutils libdnet libtins boost boost-python --with-python3"
+    BREW_PKGS="cmake python coreutils libdnet libtins sqlite boost boost-python --with-python3"
 
     # Check first to avoid unnecessary update
     echo -e "Packages: Checking..."
@@ -75,7 +75,7 @@ install_pkg_darwin()
 
 install_pip()
 {
-    PYTHON_MODULES="pyxdg lea numpy matplotlib scapy-python3 scipy coverage"
+    PYTHON_MODULES="pyxdg lea numpy matplotlib scapy-python3 scipy coverage memory_profiler"
     echo -e "Python modules: Checking..."
 
     # Check first to avoid unnecessary sudo