Appendix C of "Draft Report: Independent Technical Review of the Carnivore System": http://cryptome.org/carnivore-rev.htm
[Pages C-1 to C-32.]
C.1.1 SCENARIO
A court order authorizes collecting the noncontent header fields on e-mail messages sent to and from the target; it does not permit collecting the SUBJECT header or the body of the e-mail traffic.
C.1.2 PURPOSE
The purpose of this test is to verify that Carnivore does collect and preserve all of the information authorized by the court order and that no other system user's communication can be collected.
C.1.3 FILTER SETUP
To fulfill the collection criteria from the court order, the Carnivore filter used the following parameters for collection:
The filter menu screen filled in with the collection parameters is displayed in Figure C-1.
C.1.4 RESULT
Test not passed. Detailed testing steps for this test case are provided in Table C-1. After each step of the test was performed, Packeteer was used to put together all the data packets captured for a session and CoolMiner was used for result analysis. The result of each test step is recorded in the last column of the table. The test results clearly show for test case 1 that only the e-mail activities sent from and received by the target desktop computer was captured by Carnivore and all other types of traffic and other user's e-mail traffic were not captured.
However, Carnivore pen mode collection on an e-mail address did not collect
useful to and from information for POP3 users, but did collect correct
information for SMTP users. Carnivore did capture the sending traffic (SMTP
port 25) of the e-mail traffic that was sent from and to the target. For
e-mail sent from the target, CoolMiner shows the target's e-mail address
in the From: column. For e-mail sent to the target, instead of showing the
target's address in the To: column, CoolMiner shows the sender's (a nontarget)
address in the From: column. Note that this information can typically only
be collected if the sender of the e-mail is on the same subnetwork as the
target because the SMTP information is collected as the e-mail is being sent,
not received. If the sender is on a different subnetwork, Carnivore would
see the message when retrieved using POP3.
Figure C-1. Filter for Noncontent E-mail Collection
Table C-1. Test Steps and Results for Test Cases 1-4
The FBI provided a patch to fix the problem. After installing the patch, this test case was tested again. The Carnivore raw data for SMTP appeared to be correct; however, data is still missing on the POP3 e-mail receiver's address. The CoolMiner results show that for SMPT traffic, the from e-mail address (the target's in this test) is correctly displayed, but the to address (the nontarget's in this test) is not shown. Packeteer and CoolMiner appear to be looking for the other e-mail addresses in the to and from lines in the e-mail message, which Carnivore has purposely blanked out to avoid collecting information about communication between nontargeted entities. Carnivore should instead be looking for the rcpt-to lines, which is properly collected. Without this information, all an agent would know is that the target has either sent or received e-mail, but not to or from whom. Thus, pen mode collection for e-mail is not of much use. The test also shows that the time-stamp problem is fixed and is consistent with the system collection time.
Figure C-2 shows the result of pen mode e-mail collection that does not collect
any e-mail subject and contents.
Figure C-2. Result of Pen Mode E-mail Collection
[Original poor.]
The CoolMiner analysis result for pen mode e-mail collection provides information on how many bytes are transferred between the client and the server. Recording this information might be an issue of over-collecting because the court order only authorizes collecting e-mail from and to addresses, Also, in the Carnivore raw data the e-mail header is replaced by Xs, which can be counted to determine the amount of data.
To verify that Carnivore records the number of Xs according to the length of the data, IITRI ran two more e-mail pen mode tests; one with a length of 17 bytes of data in the subject field and the other with 29 bytes of data. Results show that an extra X was appended to the Carnivore raw data of the e-mail subject fields, i.e., 18 Xs in the subject field for the first e-mail and 30 Xs in the subject field for the second e-mail.
C.2.1 SCENARIO
A court order authorizes collecting source and destination information for HTTP activities by user John Doe. Specifically, the order authorizes collecting the IP address to which John Doe opens an HTTP connection. The order does not authorize collecting the complete URL portion of the browsing activity. Also, the target John Doe is configured to connect to the network through a DHCP server. John's laptop MAC address is 00104B60E229.
C.2.2 PURPOSE
To verify that Carnivore collects and preserves all of the target's HTTP connection information authorized by the court order, only that information, and not other users' web browsing source and destination information or content. C.2.3 FILTER SETUP To fulfill the collection criteria from the court order, the Carnivore filter used the following parameters for collection:
The filter screen filled in with the collection parameters is displayed in Figure C-3.
Figure C-3. Carnivore Filter for Noncontent Web Browsing
Collection
C.2.4 RESULT
Test passed. Detailed testing steps for this test case are provided in Table C-1. After each step of the test was performed, Packeteer and CoolMiner were used to perform results analysis. The result of each test step is recorded in the last column of the table. The test results show for test case 1 that only the activities of Web browsing performed from the target laptop computer was captured and all other types of traffic and other user's traffic (i.e., e-mail in this case) were not captured by Carnivore.
The CoolMiner analysis shows that only the HTTP (port 80) source and destination connections were captured by Carnivore from the target's laptop computer and no other types of traffic were captured from this collection.
Figures C-4 and C-5 present screen images from CoolMiner analysis. Figure C-4 shows the connection screen and Figure C-5 shows an example of the session screen that is displayed by clicking on the Start Time column of the first row on the connection screen. Ten HTTP connections were captured by Carnivore, and the first one was from the client of IP address 172.020.003.203 to the server of IP address 207.046.185.007. The target's laptop was assigned by the DHCP server to use the dynamic IP address 172.020.003.203.
Again, the results show that from the pen mode collection on HTTP port 80, none of the web browsing content or URL were collected; only the client and server HTTP connection information was collected. Collection does not start until after Carnivore determines the dynamic IP address, by viewing the DHCP protocol packets that request and assign the IP address.
The CoolMiner analysis results for pen mode web browsing activities collection provide information on how many bytes are transferred between the client and the server. This information is recorded in the To Server, and To Client columns of Figure C-4. Recording this information might be an issue of over-collecting because the court order only authorizes collecting the IP addresses of web activities, but none of the information on data sizes can be collected.
Figure C-4. Test Result for Noncontent Web Browsing
Collection
[Original poor.]
Figure C-5. Test Result for Noncontent Web Browsing Collection
[Original poor.]
C.3.1 SCENARIO
A court order authorizes collecting source and destination information for FTP activity by John Doe. Specifically, the order authorizes collecting the IP address to which John opens an FTP connection.
In addition, the target John Doe is configured to connect to the network through a DHCP and John's laptop MAC address is 00104B60E229. John is currently online and has been assigned an IP address of 172.20.3.201.
C.3.2 PURPOSE
To verify that Carnivore collects and preserves all of the target's inbound and outbound FTP traffic (i.e., connections to TCP ports 20 and 21) information authorized by the court order, only that information, and not other users' FTP source and destination information or contents.
C.3.3 FILTER SETUP
To fulfill the collection criteria from the court order, the Carnivore filter used the following parameters for collection:
The filter filled in with the collection parameters is displayed in Figure C-6.
Figure C-6. Filter Setup for Noncontent FTP Collection
C.3.4 RESULT
Test passed. Detailed testing steps for this test case are provided in Table C-1. After each step of the test was performed, Packeteer and CoolMiner were used to perform results analysis. The result of each test step is recorded in the last column of the table. The test results show only the connections of FTP activities from and to the target laptop computer were captured by Carnivore; all other types of traffic (i.e., web browsing in this case) and other user's traffic (i.e., e-mail from Mary Doe) were not captured.
The CoolMiner analysis shows that only the FTP (ports 20 and 21) inbound and outbound connections were captured from the target's laptop computer and no other types of traffic were captured from this collection. Figure C-7 provides the screen image from CoolMiner analysis. The heading of this screen shows that all of the sessions occurred on the connection between the client's IP address and the FTP server's IP address. There were eight FTP sessions in total, but no content information for any of these sessions was collected by Carnivore. The Startup entry is completely ignored by Carnivore software. Collection does not start until after Carnivore determines the dynamic IP address by viewing the DHCP protocol packets that request and assign the IP address.
Figure C-7. Test Result of Noncontent FTP Collection
The CoolMiner analysis results for pen mode collection of FTP activities provide information on how many bytes are transferred between the client and the server. This information is shown in the To Server, and To Client columns of Figure C-7. Recording this information might be an issue of over-collecting because the court order only authorizes collecting the IP addresses of the source and destination, but none of the information on message sizes can be collected.
C.4.1 SCENARIO
A court order authorizes intercepting the contents of communications to or from Mary Doe who has the fixed IP address 172.20.3.63. Specifically, the order authorizes intercepting all network communications to or from the target user's IP address.
C.4.2 PURPOSE
Verify that Carnivore collects and preserves all the authorized information from the target's communications and that no other users' (i.e., other IP addresses) communications can be collected.
C.4.3 FILTER SETUP
To fulfill the collection criteria from the court order, the Carnivore filter used the following parameters for collection:
The filter screen filled in with the collection parameters is displayed in Figure C-8.
Figure C-8. Filter Setup for Full Collection on a Fixed IP Address
C.4.4 RESULT
Test passed. Detailed testing steps for this test case are provided in Table C-1. After each step of the test was performed, Packeteer and CoolMiner were used to perform results analysis. The result of each test step is recorded in the last column of the table. The test results show for test case 4 that all communications in this test (i.e., e-mail, file transfer, and web browsing activities) to and from the target's fixed IP address (i.e., a desktop computer) were captured by Carnivore.
Figure C-9 illustrates the CoolMiner result of the communication collection.
Clicking on the FTP protocol on the screen shown in Figure C-9 displays the full content of the FTP session as shown in Figure C-10.
Figure C-9. CoolMiner Result of All Communication Collection
Figure C-10. Test Result of a Content FTP Collection
C.5.1 SCENARIO
A court order authorizes intercepting the contents of e-mail communications to or from Mary Doe who has the e-mail address mdoe@iitri.org.
C.5.2 PURPOSE
The purpose of this test is to verify that when configured to collect the authorized information from inbound and outbound (i.e., SMTP connections to TCP destination port 25 and POP3 connections to TCP destination port 110), Carnivore collects and preserves all of the authorized information and not other users' communications.
C.5.3 FILTER SETUP
To fulfill the collection criteria from the court order, the Carnivore filter used the following parameters for collection:
The filter screen filled in with the collection parameters is displayed in Figure C-11.
Figure C-11. Filter Setup for Content E-mail Collection
C.5.4 RESULT
Test passed. The e-mail of a target can be collected even when no IP address is input to the filter. The required inputs are SMTP (port 25), POP3 (port 110), and target's e-mail ID. This condition is true when the target is either at a fixed IP address or at a dynamic IP address. The filter does not allow any input to the user e-mail ID until the SMTP and POP3 ports are selected.
Figure C-12 illustrates the content of e-mail that was collected by Carnivore.
Figure C-12. Test Result of Content E-mail Collection
C.6 TEST 6 ALIAS E-MAIL COLLECTION
C.6.1 SCENARIO
A court order authorizes collecting the content of e-mail sent to and from Mary Doe. The ISP determined that Mary's Web e-mail address is marydoe@location.org. However, Mary made the alias "NOBODY" for her outgoing e-mail address. Carnivore will not collect Mary's e-mail by filtering on her original user ID marydoe.
C.6.2 PURPOSE
Verify that when configured to collect SMTP (port 25) and POP3 (port 110) e-mail messages and the target is using an alias for the origina l e-mail address, Carnivore cannot collect the target's mail by filtering on the target's original e-mail address.
C.6.3 FILTER SETUP To fulfill the collection criteria from the court order, the Carnivore filter used the following parameters for collection:
The filter screen filled in with the collection parameters is displayed in Figure C-13.
Figure C-13. Filter Setup for Alias E-mail Collection
C.6.4 RESULT
Test passed. A different address NOBODY@webmail6.location.org was seen at the receiver side; however, Carnivore did not capture this alias e-mail because the filter was set up for collection using the target's original e-mail ID.
Even though Mary made the alias "NOBODY" for her outgoing e-mail address, she still has to use "marydoe" as her login ID to get into her web mail account. Therefore, if the filter was set up using Text String data field with value of "&login=marydoe", which is Mary's original user ID, the result would be different. Carnivore would then collect web mail traffic via HTTP (port 80) on Mary's original user ID.
C.7.1 SCENARIO
Test if Carnivore will collect web browsing contents that contain a specific given text string. Both fixed and dynamically-allocated IP addresses will be used for the computers that generate the web traffic for this collection.
C.7.2 PURPOSE
When configured to collect HTTP (port 80) web browsing activities, verify that Carnivore collects only the web traffic containing the given text string, without over-collecting.
C.7.3 FILTER SETUP
To fulfill the collection criteria from the court order, the Carnivore filter used the following parameters for collection:
The filter screen filled in with the collection parameters is displayed in Figure C-14.
Figure C-14. Filter Setup for Text String Filtering on Web Activity Collection
C.7.4 RESULT
Test passed. Detailed testing steps for this test case are provided in Table C-2. Regardless of what web traffic came from the laptop or desktop computer, the results were consistent. Only those web pages containing the text string "delicious" were captured by Carnivore and those web pages that did not contain the specified text string were not captured.
Table C-2. Test Steps and Results for Filtering Text on Web Activities
CoolMiner showed many TCP sessions without finding the expected web pages,
though those pages might contain the word "delicious". Carnivore did not
find these pages because the browser was looking for automatic refresh web
pages from the servers, but Carnivore cannot resolve a domain name in this
situation. Figure C-15 shows the CoolMiner result of a web page containing
the text string "delicious".
Figure C-15. A Web Page Containing the Text String "delicious"
C.8.1 SCENARIO
When Carnivore was collecting Mary Doe's e-mail, an electrical power outage occurred, and Carnivore was terminated ungracefully. After the power is restored, Carnivore continues to collect Mary Doe's e-mail.
C.8.2 PURPOSE
Verify that after the power is restored, Carnivore automatically starts up and continues to collect what it was originally set up to collect. Also verify that Carnivore recovers all of the data that was collected before the outage occurred.
C.8.3 FILTER SETUP
To collect Mary Doe's e-mail, the Carnivore filter was set up using the following parameters:
The filter screen filled in with the collection parameters is displayed in Figure C-16.
Figure C-16. Filter Setup for Power Failure Test
C.8.4 RESULT
Test not passed. Carnivore did not recover consistently to a collecting state. The primary test system exhibited a TAPI error in connecting to the Ethernet card. It appears this error is caused by a race condition within Carnivore. The backup Carnivore system used in testing seemed not to exhibit this error condition. Others, including those at the FBI lab, exhibit this error condition intermittently. The FBI is going to investigate and fix this potential error.
The first e-mail sent out before the power outage occurred was not written
to disk by Carnivore. This condition was recorded as the actual result of
test step 3 in Table C-3. Repeated tests all showed the same failure. After
the power was restored and the system rebooted, the data file currently open
for writing always ended up being a zero-byte file. The FBI developers concluded
that this error is a problem with Carnivore in general and is the result
of a trade-off between processing speed, padding in the collected data to
a block size, or possibly losing some data. The system keeps the data in
the memory buffer until the specified block size of data is collected or
the collection is stopped, then the data is written to the disk. The block
size for the hard disk is 128 kbytes and for the removable disk, either Jazz
drive or Zip drive, is 64 kbytes.
Table C-3. Test Steps and Results for Power Failure Test
C.9.1 SCENARIO
Without providing a fixed IP address, DHCP, search text string, TCP ports, and e-mail users to the Carnivore filter, the system collects all of the TCP communications passing through the network segment to which Carnivore is connected.
C.9.2 PURPOSE
Verify that Carnivore has the capability to collect all of the communications passing through the tapping device.
C.9.3 FILTER SETUP
The simplest filter setup for Carnivore to collect all of the TCP communications was to check TCP collection on full mode without providing any other parameters.
The filter screen filled in with the collection parameters is displayed in Figure C-17.
Figure C-17. Filter Setup for TCP All Ports Full Collection
C.9.4 RESULT
Test passed. Detailed testing steps for this test case are provided in Table C-4. The results show that all TCP communications on the network segment being sniffed were captured by Carnivore. When turning on TCP full mode collection and not selecting any port, the default is to collect traffic from all TCP ports.
Table C-4. Test Steps and Results for Full TCP Ports Collection
Figure C-18 shows the CoolMiner results. FTP, web, POP3, SMTP, and Microsoft Exchange E- mail traffic are all captured by Carnivore and displayed by CoolMiner.
Figure C-18. Test Result of All Ports TCP Collection
C.10.1 SCENARIO
Without entering a fixed IP address and DHCP information to the filter, Carnivore collects all communication passing the tapping device. This test has been proved true from the test case 9 in paragraph C.9. The Carnivore filter screen provides three entry fields for DHCP setup, i.e., MAC address, Ports (67 and 68), and Startup IP. Also need to determine what data must be entered to the filter to collect communication from a specific DHCP-configured device. It is assumed that the Startup IP field can be used by Carnivore to begin collecting immediately the communication of a target who is already on line.
C.10.2 PURPOSE
There are two purposes of this test
1. Determine what data needs to be entered for DHCP.2. Verify that the Startup IP is useful for Carnivore to capture a target who is already on line before Carnivore starts collecting, and, therefore, there is no need to force a DHCP exchange when a correct Startup IP was set up in the filter.
C.10.3 FILTER SETUP
Three filters are used for this test
The screens for setting up these three filters are displayed in Figures C-19, C-20, and C-21, respectively.
Figure C-19. Filter Setup 1 for DHCP Data Entries Test
Figure C-20. Filter Setup 2 for DHCP Data Entries Test
Figure C-21. Filter Setup 3 for DHCP Data Entries Test
C.10.4 RESULT
Test passed for Purpose 1, but did not pass for Purpose 2. Detailed testing steps for this test case are provided in Table C-5. Steps 1 through 11 were used to test Purpose 1. Both MAC and DHCP ports are required data entries for the filter to collect communication from a specific dynamically-configured IP address.
Table C-5. Test Steps and Results for DHCP Filter
Steps 12 through 16 were used to test the Startup IP entry field. Without forcing a DHCP exchange on the laptop computer, even though a startup IP was given, Carnivore cannot capture the e-mail sent from the laptop computer. This test proves that the Startup IP field is not used by Carnivore as it was originally assumed. This condition was also verified by the FBI developers who stated that the Startup IP part of Carnivore 1.3.4 code was all commented out, but the GUI portion had not been removed.
C.11.1 SCENARIO
A court order authorizes collecting all SMTP or POP3 e-mail sent from and to a target that contains the key word "Planning". No target e-mail address is provided since the target uses a fixed IP address.
C.11.2 PURPOSE
Verify that when not providing the e-mail user ID to the filter, Carnivore has the capability to collect a target's e-mail that only contains the given text strings.
C.11.3 FILTER SETUP
For the first collection in this test, the filter parameters were set up using
C.11.4 RESULT
Test not passed. By examining the Carnivore raw data, IITRI noted that Carnivore collects SMTP (sending) e-mail that matches the key word correctly, but does not collect POP3 (receiving) e-mail correctly. However, by examining the CoolMiner analysis result, it is observed that if the text string is in the header (such as in the Subject), then CoolMiner displays the message as a valid SMTP message. If the text string is in the body of the message, CoolMiner does not display it as an SMTP message. This is because the SMTP header is not collected even though raw Carnivore data shows the packet with the text string is collected properly.
The results are consistent with the capabilities provided by the FBI developers.
The specified text strings have to be included in the packet and triggered
at the driver level to save processing time. This condition is a performance
trade off. However, Carnivore filters SMTP and POP3 e- mail users at the
application level; therefore, the e-mail traffic does not pass through the
text string filtering when e-mail user IDs are provided to the filter.
Table C-6. Test Steps and Results for Collecting E-mail of a Specific Text String
C.12.1 SCENARIO
A court order authorizes collecting the SMTP or POP3 e-mail messages sent from and to a target that contain a key word "Planning". The e-mail address of the target is mdoe@iitri.org and the target uses a fixed IP address.
C.12.2 PURPOSE
Verify that Carnivore has the capability to collect e-mail of a target that only contain the given text strings.
C.12.3 FILTER SETUP
For the first collection in this test, the filter parameters were set up using
The filter screen filled in with the collection parameters is displayed in Figure C-22.
Figure C-22. Filter Setup for Filtering on Text String and E-mail User
for E-mail Collection
C.12.4 RESULT
Test not passed. When given both a specific e-mail address and a text string, Carnivore collects all the target's e-mail whether or not the e-mail matches the given text string. The result is recorded in steps four through nine of Table C-7.
Table C-7. Test Steps and Results for Collecting E-mail of a Specific
Text String and an E-mail User
The result is consistent with the capabilities provided by the FBI developers. The specified text strings have to be included in the packet and triggered at the driver level to save processing time. This condition is a performance trade off. However, Carnivore filters SMTP and POP3 e-mail users at the application level; therefore, the e-mail traffic does not pass through the text string filtering when e-mail user IDs are provided to the filter.
C.13.1 SCENARIO
A court order authorizes collecting a target's file download FTP activities that contain the key word "Planning". The target uses a fixed IP address.
C.13.2 PURPOSE
Verify that Carnivore has the capability to collect the target's FTP (ports 20 and 21) communications that only contain the given text strings.
C.12.3 FILTER SETUP
For the first collection in this test, the filter parameters were setup using
The filter screen filled in with the collection parameters is displayed in Figure C-23.
Figure C-23. Filter Setup for Collecting FTP Activities Containing a Specific
Text String
C.13.4 RESULT
Test passed. Carnivore has the capability to collect FTP traffic that contained given text strings. However, it only collects the packets containing the text string or, if the Trigger on Full Session check box is checked, collects from the first packet containing the text string to the end of that session. In either case, Packeteer would fail to assemble all of the packets together for an entire FTP session and, in turn, CoolMiner would fail to analyze the result as shown in Figure C-24. The goal here is to test if Carnivore collects according to its filter setup, not to evaluate the post-processing tools, Packeteer or CoolMiner. The raw output from Carnivore contained the correctly collected data. The test results are shown in Table C-8.
Figure C-24. CoolMiner Analysis Screen for FTP Collection Triggered by Text String
Table C-8. Test Steps and Results for Filtering on Text String for FTP
Collection
Transcription and HTML by Cryptome.