What's to be expected

The test for deterministic output is implemented as a python unit-test and can be run along with other tests. The test will call id2t from the shell and parse the resulting output to determine the files created by id2t. If id2t exits with error code 0 and the resulting pcap's are equal the testcase succeeds.

For every set of parameters created the testcase will run 3 times. The number of parameters sets and rounds per parameter set are found in static constants in the PcapComparison-class and can be changed there. The numbers were chosen at will, as long as they are greater than 1 they will be fine.

A static random seed and input file will be passed to ID2T, along with the random parameters. An example of the resulting command looks like this:
../id2t -i resources/telnet-raw.pcap --seed 42 -a MembersMgmtCommAttack hidden_mark=False ttl.from.caida=True

The first parameter here is the input-file, afterwards comes the seed and then the attack-name along with its parameters.

Testing random parameters

To test a greater coverage of the MembersMgmtCommAttack the testcase will create multiple sets of parameters. The test_pcap_comparison.py-file contains a dictionary ID2T_PARAMETER_GENERATORS that contains all the information to create random parameters. The dict contains items in the shape of parameter_name -> function that returns random parameter value. To add your parameter here simply add it's name and a function for its random values in that dict. As of now not all parameters used by ID2T are implemented and none of them are checked for compatibility, e.g. file.xml and file.csv might appear together even though they are mutually exclusive.

An example of random parameters looks like this:
packet.padding=22 hidden_mark=True packets.per-second=17.5 bots.count=4 nat.present=False

The output

To help the developer this testcase print several sections of output. Note that all logging is happened on stderr, this is to comply with pythons unit-test-framework that prints on stderr too. For every iteration the testcase will first print the actual shell command that will be executed, note that this is the command after every character has been escaped for shell use. Afterwards ID2T will be run and its output (stdout and stderr) will be printed too. If the generated pcap's are not equal an error will be thrown whose error message will give the exact occurence of the pcaps' difference.

After the test has been run, successfully or not, the testcase will list the files created by id2t during the various runs. This is followed by a list of files that will be kept for the developer to inspect because the have differences. If all runs were successful this list is empty and all files created will be deleted immedeately afterwards.

Files kept

In order to not mess up the filesystem the testcase deletes all temporary files created afterwards. This is unless there was a mismatch between two pcaps and one has to look over them. Then those files will be kept even after the testcase terminates with an error. Make sure to delete those yourself.

What can go wrong

There are a number of prerequisites for this testcase:

1. The ID2T-executable must exist in the parent-directory. Because ID2T is not in the path it must be called by its relative path. Keep in mind the output from id2t about the files created is relative to its home directory and not the test directory.
2. The magic string is not in ID2Ts output any more. To get the list of files created from id2t the testcase parses the output and looks for the line "Output files created:", everything after that will be considered as file created. If ID2T's output structure changes this parsing mechanism will fail, but it will throw an descriptive error message for some of the possible changes.
3. The attack's parameters are incoherent. As mentioned above the testcase might pass impossible combinations of parameters to the MembersMgmtCommAttack, it's the developers job to adjust that behaviour in the testcase and make ID2T print descriptive error messages in the attack-class.

This list should serve as a first step for troubleshooting because these are the weak points of the testcase as seen by the developer at the time of writing.

Environmental variables and where to find them

To get a certain amount of control over a testcase that is usually run as one of many this class accepts environment variables to change the random seed and/or the input file. These environmental variables are called ID2T_SRC_PCAP and ID2T_SEED and their values will be used instead of the default ones.