explanation_on_how_to_packet_capture_for_only_certain_TCP_flags_v2.txtlinux
Packet capture for only certain TCP flags (for use with Fortigate 'sniffer' command): Information extracted from the Man pages of 'tcpdump' command: # Updated Jan 2007 for Fortinet KC by Stephen Dennis. There are 8 bits in the control bits section of the TCP header: Let's assume that we want to watch packets used in establishing a TCP connection. Recall that TCP uses a 3-way handshake protocol when it initial- izes a new connection; the connection sequence with regard to the TCP control bits is 1) Caller sends SYN 2) Recipient responds with SYN, ACK 3) Caller sends ACK Now we're interested in capturing packets that have only the SYN bit set (Step 1). Note that we don't want packets from step 2 (SYN-ACK), just a plain ini- tial SYN. What we need is a correct filter expression for tcpdump. Recall the structure of a TCP header without options: 0 15 31 ----------------------------------------------------------------- | source port | destination port | ----------------------------------------------------------------- | sequence number | ----------------------------------------------------------------- | acknowledgment number | ----------------------------------------------------------------- | HL | rsvd |C|E|U|A|P|R|S|F| window size | ----------------------------------------------------------------- | TCP checksum | urgent pointer | ----------------------------------------------------------------- A TCP header usually holds 20 octets of data, unless options are present. The first line of the graph contains octets 0 - 3, the second line shows octets 4 - 7 etc. Starting to count with 0, the relevant TCP control bits are contained in octet 13: 0 7| 15| 23| 31 ----------------|---------------|---------------|---------------- | HL | rsvd |C|E|U|A|P|R|S|F| window size | ----------------|---------------|---------------|---------------- | | 14th octet | | | Let's have a closer look at octet no. 13: | | |---------------| |C|E|U|A|P|R|S|F| <- TCP (and ECN) flags |---------------| |7 5 3 0| <- bit position These are the TCP control bits we are interested in. We have numbered the bits in this octet from 0 to 7, right to left, so the PSH bit is bit number 3 (4th bit from right), while the URG bit is number 5 (6th bit from right). Recall that we want to capture packets with only SYN set. Let's see what hap- pens to octet 13 if a TCP datagram arrives with the SYN bit set in its header: |C|E|U|A|P|R|S|F| <- TCP and ECN flags |---------------| |0 0 0 0 0 0 1 0| <- bit value |---------------| |7 6 5 4 3 2 1 0| <- bit position Looking at the control bits section we see that only bit number 1 (SYN) is set. Assuming that octet number 13 is an 8-bit unsigned integer in network byte order, the binary value of this octet is 00000010 and its decimal representation is: | bit | dec. | bit | | TCP | pos. | value | value | total | Flag ____|______|________|_______|_______|_____ | | | | | 2 ^ 7 = 128, x 0 = 0 | CWR (ECN Congestion Window Reduced) 2 ^ 6 = 64, x 0 = 0 | ECE (ECN Capable Echo) 2 ^ 5 = 32, x 0 = 0 | URG 2 ^ 4 = 16, x 0 = 0 | ACK 2 ^ 3 = 8, x 0 = 0 | PSH 2 ^ 2 = 4, x 0 = 0 | RST 2 ^ 1 = 2, x 1 = 2 | SYN* 2 ^ 0 = 1, x 0 = 0 | FIN ____|______|________|_______|_______|_____ sum: 2 ========================================== ('2 ^ 7 = 128' means '2 to the power of 7 equals 128') We're almost done, because now we know that if only SYN is set, the value of the 14th octet in the TCP header, when interpreted as an 8-bit unsigned integer in network byte order, must be exactly 2. This relationship can be expressed as 'tcp[13]==2' The expression says "match all TCP datagrams where octet 13 (i.e. the 14th octet) is equal to the decimal value 2", which is exactly what we want. But now, let's assume that we need to capture ALL packets with the SYN bit set, regardless of whether any other TCP control bits (such as ACK, URG, PSH etc.) are set. Let's see what happens to octet 13 when a TCP datagram arrives with both the SYN and ACK bits set: |C|E|U|A|P|R|S|F| <- TCP (and ECN) flags |---------------| |0 0 0 1 0 0 1 0| <- bit value |---------------| |7 6 5 4 3 2 1 0| <- bit position Now bits 4 and 1 are set in the 14th octet. The binary value of octet 13 is now: 00010010 which translates to a decimal value of: | bit | dec. | bit | | TCP | pos. | value | value | total | Flag ____|______|________|_______|____________ | | | | | 2 ^ 7 = 128, x 0 = 0 | CWR (ECN Congestion Window Reduced) 2 ^ 6 = 64, x 0 = 0 | ECE (ECN Capable Echo) 2 ^ 5 = 32, x 0 = 0 | URG 2 ^ 4 = 16, x 1 = 16 | ACK* 2 ^ 3 = 8, x 0 = 0 | PSH 2 ^ 2 = 4, x 0 = 0 | RST 2 ^ 1 = 2, x 1 = 2 | SYN* 2 ^ 0 = 1, x 0 = 0 | FIN ____|______|________|_______|_______|_____ sum: 18 ========================================== We can't just use 'tcp[13]==18' in the tcpdump filter expression, because that would match only those packets where both SYN and ACK are set, but not packets with only SYN set. Remember that we don't care if ACK or any other control bit is set, as long as SYN is set. In order to achieve our goal, we need to logically 'AND' the binary value of octet 13 with some other value to preserve the SYN bit. We know that we want SYN to be set in any case, so we'll logically AND the value in the 14th octet with the binary value of a SYN: v v 00010010 SYN-ACK 00000010 SYN AND 00000010 (we want SYN) AND 00000010 (we want SYN) -------- -------- = 00000010 = 00000010 ^ ^ We see that this AND operation delivers the same result regardless of whether any other TCP control bits are set or not. The decimal representation of the AND value as well as the result of this operation is 2 (binary 00000010), so we know that for packets with SYN set the following relation must hold true: ( ( value of octet 13 ) AND ( 2 ) ) == ( 2 ) This relationship can be expressed as: 'tcp[13]&2==2' The following list of expressions expand on this idea: 'tcp[13]==1' matches packets with only the FIN bit set to 1; 'tcp[13]&4==4' matches all packet with the RST bit set to 1; 'tcp[13]&8==8' matches all packet with the PSH bit set to 1; 'tcp[13]==16' matches packets with only the ACK bit set to 1; 'tcp[13]&32==32' matches all packets with the URG bit set to 1; 'tcp[13]&64==64' matches all packets with the ECE bit set to 1; 'tcp[13]&128==128' matches all packets with the CWR bit set to 1; 'tcp[13]==24' matches packets with only the PSH and ACK bits set to 1; Fortigate examples: -------------------- Example to sniff for packets with ONLY the TCP SYN set (no other flags set): # diag sniff packet wan1 'tcp[13]==2' Example to sniff for ANY packets with the TCP SYN set (regardless of other flags): # diag sniff packet wan1 'tcp[13]&2==2' Example to sniff for any packets on port 80 (HTTP) with the TCP SYN set: # diag sniff packet wan1 'port 80 and tcp[13]&2==2' Example to sniff for any packets on port 80 (HTTP) or 443 (HTTPS) with the TCP SYN set: # diag sniffer packet internal 'port 80 or 443 and tcp[13]&2==2'
fgt2eth.pl
此腳本依賴 wireshark.exe 和 text2pcap.exe
經過 getText2PcapCmd 子程序 獲取 text2pcap.exe 路徑,並拼接對應的命令行參數。
經過 getWiresharkCmd 子程序 獲取 wireshark.exe 路徑,並拼接對應的命令行參數。redis
#!/usr/bin/perl # This work (and included software, documentation or other related items) is # being provided by the copyright holders under the following license. By # obtaining, using and/or copying this work, you (the licensee) agree that you # have read, understood, and will comply with the following terms and conditions # # Permission to copy, modify, and distribute this software and its documentation # with or without modification, for any purpose and without fee or royalty is # hereby granted, provided that you include the following on ALL copies of the # software and documentation or portions thereof, including modifications: # # 1. The full text of this NOTICE in a location viewable to users of the # redistributed or derivative work. # 2. Notice of any changes or modifications to the files, including the date # changes were made. # # # THIS SOFTWARE AND DOCUMENTATION IS PROVIDED "AS IS," AND COPYRIGHT HOLDERS # MAKE NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT # LIMITED TO, WARRANTIES OF MERCHANTABILITY OR FITNESS FOR ANY PARTICULAR # PURPOSE OR THAT THE USE OF THE SOFTWARE OR DOCUMENTATION WILL NOT INFRINGE ANY # THIRD PARTY PATENTS, COPYRIGHTS, TRADEMARKS OR OTHER RIGHTS. # # COPYRIGHT HOLDERS WILL NOT BE LIABLE FOR ANY DIRECT, INDIRECT, SPECIAL OR # CONSEQUENTIAL DAMAGES ARISING OUT OF ANY USE OF THE SOFTWARE OR DOCUMENTATION. # # Title to copyright in this software and any associated documentation will at # all times remain with copyright holders. # # Copyright: Fortinet Inc - 2005 # # Fortinet EMEA Support # This script converts a Fortigate verbose 3 sniffer output file to a file that # can be opened by Ethereal. It uses the text2pcap binary included in the Ethereal package. # It is supplied as is and no technical support will be provided for it. my $version = "Dec 19 2014"; # ------------------ don't edit after this line ------------------------------- use strict; use Getopt::Long; use FileHandle; use Config; use vars qw ($debug $help $vers $in $out $lines $demux $childProcess); use Data::Dumper; # Autoflush $| = 1; # Global variables my $line_count = 0; my ($fh_in, $fh_out); my @fields = {}; my @subhexa = {}; my ($offset,$hexa,$garbage) = ""; my %outfileList; my %outfilenameList; # Get commandLine arguments getArgs(); # In order to support real-time display in wireshark/ethereal, we need to pipe # our stdout into wireshark stdin, which is not allowed by the OS... # The trick consists in creating a child process with the appropriate anonymous # pipes already in place and delegate the work to the child. spawnPipedProcess(); open(fh_in, '<', $in) or die "Cannot open file ".$in." for reading\n"; # Convert if( $debug ) { print STDERR "Conversion of file ".$in." phase 1 (FGT verbose 3 conversion)\n"; print STDERR "Output written to ".$out.".\n"; } my @packetArray = (); #Parse the entire source file my $DuplicateESP = 0; my $eth0 = 0; my $skipPacket = 0; followMode: while (<fh_in>) { s/^\d{2}(\d{4})/0x$1/; #and build an array from the current packet if( isTimeStamp() ) { $skipPacket = 0; if( not $demux and /eth0/ ) { $eth0++; $skipPacket = 1; } # Select the appropriate output file for the interface. $fh_out = getOutputFileHandler() if defined $demux; $skipPacket |= convertTimeStamp(); } elsif ( isHexData() and not $skipPacket ) { buildPacketArray(); adjustPacket(); _startConvert(); } } if( $out eq "-" ) { # no more incoming data. Wait 2 seconds and try again sleep 2; goto followMode; } print "** Skipped $eth0 packets captured on eth0\n" if $eth0; # Close files and start text2pcap if needed close (fh_in) or die "Cannot close file ".$in."\n"; my $text2pcap = getText2PcapCmd(); foreach my $intf( keys %outfileList ) { close $outfileList{ $intf }; my $filename_in = $outfilenameList{$intf}; my $filename_out = $filename_in; $filename_out =~ s/\.tmp$/\.pcap/; system("$text2pcap $filename_in $filename_out"); unlink($filename_in); } if( $debug ) { print STDERR "Output file to load in Ethereal is \'".$out."\'\n"; print STDERR "End of script\n"; } sub isHexData { /^(0x[0-9a-f]+[ \t\xa0]+)/ } sub isTimeStamp { /^[0-9]+[\.\-][0-9]+/ } # /Dummy comment to workaround Komodo parsing sub buildPacketArray { my $line = 0; @packetArray = (); do { # Format offset from 0x0000 to 000000 (text2pcap requirement) s/^0x([\da-f]{4})/00$1/; if ( s/^([\da-f]{6})\s+// ) { # Remove ASCII translation at the end of the line s/\s\s+.*//; my @bytes = /([\da-f]{2})\s?([\da-f]{2})?/g; $#bytes = 15; push @packetArray => @bytes; } $_ = <fh_in>; } until ( /^\s*$/ ); } sub convertTimeStamp { # Keep timestamps. return 1 if /truncated\-ip \- [0-9]+ bytes missing!/ ; if ( /^([0-9]+)\.([0-9]+) / ) { my $packet = 1; my $time = $1; # Extract days my $nbDays = int($time / 86400); my $day = sprintf("%0.2d", 1+$nbDays); $time = $time % 86400; # Extract hours my $hour = int($time / 3600 ); $time = $time % 3600; # Extract minutes my $minute = int( $time / 60); $time = $time % 60; # and remaining seconds my $sec = $time; _print("01/$day/2005 " . $hour . ":" . $minute . ":" . $sec . ".$2\n"); } elsif ( /^(\d+-\d+-\d+ \d+:\d+:\d+\.\d+) / ) { # absolute timestamp my $timestamp = $1; $timestamp =~ s/(\d+)-(\d+)-(\d+)/$3\/$2\/$1/; _print("$timestamp\n"); } # Check if line is a duplicate ESP packet (FGT display bug) return 0; } sub getOutputFileHandler { my ($currIntf) = $_ =~ / (\S+) (?:out|in) /; $currIntf = "[noIntf]" if $currIntf eq ""; if( not defined( $outfileList{$currIntf} )) { my $filename = $out ? $out : $in; $filename =~ s/\.zip$//g; my $suffix = ".$currIntf.tmp"; $suffix =~ s/\//-/g; # Escape '/' char in interface name $filename .= $suffix; open( $outfileList{$currIntf}, "> $filename"); $outfilenameList{$currIntf} = $filename; } return $outfileList{$currIntf}; } #---------- # name : adjustPacket # description: # Applies changes to the current packetArray to make it convertible into # ethereal format. # - Removes internal Fortigate tag when capture interface is any. # sub adjustPacket { next unless $packetArray[0][0] =~ /0x0000/; stripBytes( 7, 1 ) if $packetArray[0][8] =~ /0800/; stripBytes( 7, 1 ) if $packetArray[0][7] =~ /8893/; addHdrMAC() if $packetArray[0][1] =~ /45[0c]0/; $packetArray[0][7] =~ s/8890|8891/0800/ if $packetArray[0][0] =~ /0x0000/; } sub addHdrMAC { my $nbRows = scalar @packetArray; # And populate 0x0000 line unshift @packetArray => qw( 00 00 00 00 00 00 00 00 00 00 00 00 08 00 45 00 ); # left shift the IP ver+IHL (4500) stripBytes(14,2); } sub stripBytes { my $start = shift; my $nbBytes = shift; my @subArray = @packetArray[$start..$#packetArray-1]; shift @subArray for 1..$nbBytes; @packetArray = (@packetArray[0..$start-1],@subArray); } sub _startConvert { LINE: #Initialisation my $hexa = ""; my $garbage = ""; my $offset = 0; foreach my $byte (@packetArray) { _print( sprintf( "%0.6x ", $offset )) unless $offset % 16; _print( " ". $byte); $offset ++; _print("\n") unless $offset % 16; } _print( "\n"); $line_count++; if (defined($lines)) { if ($line_count >= $lines) { print STDERR "Reached max number of lines to write in output file\n"; last LINE; } } } sub spawnPipedProcess { # Spawn a new process to pipe ourselves with text2pcap my( $fgt2eth, $text2pcap, $wireshark ); # Are we already in the pipelined session ? return if $childProcess or $demux; $fgt2eth = $0 . " -in \"$in\" -out \"$out\" -childProcess"; $text2pcap = getText2PcapCmd(). "- "; $wireshark = getWiresharkCmd(); my $cmd = "$fgt2eth | $text2pcap"; # Prepare output filename if( $out eq "-" ) { $cmd .= "- | $wireshark"; } else { $out = $in . ".pcap" unless $out; $cmd .= " $out"; } print STDERR $cmd if $debug; open( CMD, "$cmd |" ); while( <CMD> ) { _print($_); } close CMD; # Make sure we don't go further in the parent process exit; } sub getText2PcapCmd { my $cmd; my @windowsPath = qw( c:\\Progra~1\\Ethereal c:\\Progra~1\\Wireshark ); if ($ENV{'OS'} =~ /windows/i) { if( $Config{osname} =~ /cygwin/i ) { #OS is Windows running Cygwin $cmd = "'/cygdrive/c/Program Files/Wireshark/text2pcap'"; } else { # OS is windows my $dir; for $dir( @windowsPath ) { $cmd = "$dir\\text2pcap.exe" if -e "$dir\\text2pcap.exe"; } } } else { # OS is linux :-) $cmd = "text2pcap"; } # Sanity die "Text2Pcap could not be found\n" unless $cmd; $cmd .= " -q -t \"%d/%m/%Y %H:%M:%S.\" "; return $cmd; } sub getWiresharkCmd { my $cmd; my @windowsPath = qw( c:\\Progra~1\\Wireshark ); if ($ENV{'OS'} =~ /windows/i) { if( $Config{osname} =~ /cygwin/i ) { # OS is Windows running Cygwin $cmd = "'/cygdrive/c/Program Files/Wireshark/wireshark'"; } else { # OS is windows my $dir; for $dir( @windowsPath ) { $cmd = "$dir\\wireshark.exe" if -e "$dir\\wireshark.exe"; } } } else { # OS is linux :-) $cmd = "wireshark"; } # Sanity die "wireshark could not be found\n" unless $cmd; $cmd .= " -k -i -"; return $cmd; } sub _print{ my $msg = shift; if( defined $fh_out ) { print $fh_out $msg; } else { print $msg; } } sub getArgs { # Control command line options GetOptions( "debug" => \$debug, # use -debug to turn on debug "version" => \$vers, # use -version to display version "help" => \$help, # use -help to display help page "in=s" => \$in, # use -in <filename> to specify an input file "out=s" => \$out, # use -out <filename> to specify an output file "lines=i" => \$lines, # use -lines <number> to stop after <number> lines written "demux" => \$demux, # use -demux to create one pcap per intf "childProcess" => \$childProcess, ); if ($help) { Print_help(); exit; } if ($vers) { Print_version(); exit; } # Sanity checks if (not(defined($in))) { Print_usage(); exit; } } #------------------------------------------------------------------------------ sub Print_usage { print <<EOT; Version : $version Usage : fgt2eth.pl -in <input_file_name> Mandatory argument are : -in <input_file> Specify the file to convert (FGT verbose 3 text file) Optional arguments are : -help Display help only -version Display script version and date -out <output_file> Specify the output file (Ethereal readable) By default <input_file>.pcap is used - will start wireshark for realtime follow-up -lines <lines> Only convert the first <lines> lines -demux Create one pcap file per interface (verbose 6 only) -debug Turns on debug mode EOT } #------------------------------------------------------------------------------ sub Print_help { print <<EOT; This script permits to sniff packets on the fortigate with built-in sniffer diag sniff interface <interface> verbose 3 filters '....' and to be able to open the captured packets with Ethereal free sniffer. * What do I need to know about this script ? - It can be sent to customers, but it is given as is. - No support is available for it as it is not an 'offical' fortinet product. - It should run on windows and linux as long as perl is installed. - To install perl on windows system, http://www.activestate.com/Products/ActivePerl/ - All lines from the source file that do not begin with '0x' are ignored. - Do not add garbage characters to the file during the capture - If possible do not hit the keyboard during capture.; Remarks concerning this script can be sent to eu_support\@fortinet.com Thanks to Claudio for this great idea and Ellery from Vancouver Team for the timestamps Cedric EOT } #------------------------------------------------------------------------------ sub Print_version { print "\nVersion : ".$version."\n\n"; } sub myDump { my $object = shift; my $dumper = new Data::Dumper([$object]); $dumper->Maxdepth(3); print STDERR $dumper->Dump; }
fgt2eth.02.2008.exeexpress
> fgt2eth.02.2008.exe /? Version : 1.22 (Feb 21 2008) Assuming OS: windows Usage : fgt2eth.pl -in <input_file_name> Mandatory argument are : -in <input_file> Specify the file to convert (FGT verbose 3 text file) Optional arguments are : -help Display help only -version Display script version and date -out <output_file> Specify the output file (Ethereal readable) By default '<input_file>.eth' is used -lines <lines> Only convert the first <lines> lines -system <system> Can be either linux or windows -debug Turns on debug mode
fgt2eth.12.2014.exewindows
> fgt2eth.02.2008.exe /? Version : 1.22 (Feb 21 2008) Assuming OS: windows Usage : fgt2eth.pl -in <input_file_name> Mandatory argument are : -in <input_file> Specify the file to convert (FGT verbose 3 text file) Optional arguments are : -help Display help only -version Display script version and date -out <output_file> Specify the output file (Ethereal readable) By default '<input_file>.eth' is used -lines <lines> Only convert the first <lines> lines -system <system> Can be either linux or windows -debug Turns on debug mode PS C:\Users\lsgxt\Downloads> .\fgt2eth.12.2014.exe /? Version : Dec 19 2014 Usage : fgt2eth.pl -in <input_file_name> Mandatory argument are : -in <input_file> Specify the file to convert (FGT verbose 3 text file) Optional arguments are : -help Display help only -version Display script version and date -out <output_file> Specify the output file (Ethereal readable) By default <input_file>.pcap is used - will start wireshark for realtime follow-up -lines <lines> Only convert the first <lines> lines -demux Create one pcap file per interface (verbose 6 only) -debug Turns on debug mode
Etherealsession
Mode LastWriteTime Length Name ---- ------------- ------ ---- -a---- 2013/11/30 9:08 84 convert.bat -a---- 2013/11/30 9:08 104 convertopen.bat -a---- 2010/11/17 4:19 1808230 fgt2eth.exe -a---- 2012/10/29 3:57 1149982 libglib-2.0-0.dll -a---- 2012/10/29 3:57 46371 libgmodule-2.0-0.dll -a---- 2012/9/4 4:30 130181 libintl-8.dll -a---- 2013/11/2 2:07 33200 libwsutil.dll -a---- 2011/2/20 14:03 211280 msvcp100.dll -a---- 2011/2/19 15:40 356176 msvcr100.dll -a---- 2013/12/3 2:14 1102 README.txt -a---- 2013/11/2 2:07 338864 text2pcap.exe -a---- 2018/11/30 10:28 48968 uninst.exe
=============== Endapp