Home > Cloud Cruiser 3 > Setting Up Collection > Universal collectors > Network Traffic Collector > Support information

Support information

The Network Traffic Collector is designed to run 24/7 without problems. However, there are a number of things that can cause interruption to the process. The following information should assist diagnosis and resolution of any problems.

Exported diagnostic information

The collector makes extensive use of logging and will record both problems and runtime events. Additionally, it contains a lot of information about the initialization process and the exporting of CCR files. Every entry in the log is timestamped.

In the event of a problem transferring a CCR file to Cloud Cruiser, the collector will write an entry into the logfile which refers you to a second log file called ccnetcollect_push.log. This second file will contain the last error encountered while pushing the CCR file to the Cloud Cruiser REST API.

During runtime, a small file called pc_metrics is updated (overwritten) every ten seconds. This file exposes a number of internal diagnostic counters. If the numbers in this file do not increase every ten seconds during runtime then no NetFlow V9 traffic is being received.

When running as an application, the ccnetcollect_push.log and pc_metrics files will be located in the same directory as the ccnetcollect binary. When running as a daemon, the files will be located in the directory specified by the daemon_dir configuration option.

 

The counters contained within the pc_metrics file are as follows (the additional notes in square brackets are not part of the file):

|------Packets--------|
NetflowV9     : 10102 packets                [NetFlow V9 packets seen]
Non-NetflowV9 : 0 packets                    [Non-NetFlow packets seen]
Parser rejects: 0 packets                    [Packets that could not be parsed]

|------Netflow--------|
Template      : 439 flowsets                 [NetFlow Template FlowSets seen]
Options       : 870 flowsets                 [NetFlow Options FlowSets seen]
Data          : 8793 flowsets                [NetFlow Data FlowSets seen]
________________
Records       : 100259 (total across all flowsets)

|------Internals------|
Collisions    : 3                    [Hash collisions while inserting sessions]
Overflows     : 0                    [“Packet Queue full” events]
T_cache misses: 0                    [DataFlow Records that did not have a cached
                                      template required to decode them]
PQ writes     : 10102                [Packets written to the Packet Queue]
IP filter     : 0                    [Packets filtered by the accept_from options]
Thread 00 load: 2526                 [The number of packets parsed by each of]
Thread 01 load: 2526                  the parser threads. Threads are numbered
Thread 02 load: 2525                  from 0 - 31. The configuration shown here
Thread 03 load: 2525                  has four parser threads]

 

The T_cache misses counter is worthy of special mention. When the collector is first started, especially if there is an existing ingress of NetFlow data, it may be that it receives NetFlow Data FlowSets before it receives the templates required to decode them. In such cases, the counter is incremented and the packet is discarded. Depending on the time between template updates sent by the NetFlow source this may represent a large number of packets. Once the required templates have been seen, parsed and cached, the counter should remain static so long as no further Data FlowSets without an associated template are received.

Debug mode

Debug mode can be enabled by setting debug=yes in the configuration file and restarting the collector in application mode (eg: with daemon_mode disabled). This will generate a huge amount of output to the screen (which can be redirected to disk) and should normally only be used in conjunction with developer support.

When debug mode is active, the performance of the collector will be slightly diminished, but the exported information may prove helpful in tracking down any issues with the traffic it is receiving.

It is recommended that when using debug mode, the packet_limit option in the configuration file is also set. The number specified in the packet_limit option defines the number of packets to receive before the collector automatically exits.

Note that the number of packets received is not necessarily the same as the number of packets processed. The packet_limit counter includes every single UDP packet received on the UDP port specified by listen_port and takes precedence over any filtering rules defined by the accept_from lines in the configuration file.

To illustrate this, if packet_limit = 50 and accept_from = 192.168.100.1 then if the first 40 packets are received from 192.168.100.100 and the next 10 packets are received from 192.168.100.1, the collector will exit after the 10th packet from 192.168.100.1, as the previous 40 filtered packets are included in the total packet count, thus the 10 from 192.168.100.1 make up the required 50 to satisfy the exit condition.

General connectivity

In order to work as intended, the collector must be receiving NetFlow v9 packets on the UDP port number specified in the configuration file. If no NetFlow V9 packets are being received, or if they are going to the wrong port, then the counters in the pc_metrics file will not increase.

Therefore if there is no output, or zero length CCR files output, verify that the NetFlow generator is sending packets to the collector on the required port.

The collector must be able to access the Cloud Cruiser server in order to push usage files via the REST API. If it is unable to connect successfully, information will be written to the log file.

NetFlow data

The Network Traffic Collector only works with NetFlow V9 data arriving over UDP. Earlier versions of NetFlow than V9 are ignored, but will be counted as "Parser rejects" in the pc_metrics file. NetFlow contains a lot of information that is not required by the collector. The key NetFlow fields that it looks for are IPV4_SRC_ADDR, IPV4_DST_ADDR and IN_BYTES, and these are required in order to generate CCR files. Support for additional NetFlow V9 fields can be added if required, but this is a developer task. Note that currently, the collector only supports field lengths of 32 bits (the NetFlow default) or 64 bits for the IN_BYTES field.  

CCR exports

Every ccr_export_interval minutes, a CCR file will be generated in ccr_export_dir. By default these two settings are 60, and ccr/ respectively, but they may have been overridden in the configuration file if, for example, you are getting a CCR file every minute instead of every half hour.

When the collector is run in application mode it will inform you of the export schedule as shown below.

Starting Network Traffic Collector ...
**************************************
Network Traffic Collector v2.1.1
(c) Copyright 2014 Cloud Cruiser Inc.
All Rights Reserved
Build No. : 960
Build Date: 20140625
INFO      : Current configuration will
            export a CCR every 15 mins
**************************************
The collector is now fully initialised
Listening : UDP on 192.168.126.130:2055

 

When the collector is run in daemon mode, equivalent information to the above is written to the logfile on startup.

The filename of an exported CCR file is <yyyymmdd_nnnn>.ccr, where nnnn is a number from 0001 - 9999. Whenever the collector performs an export, it locates the highest numbered file for today, adds 1 to it, and uses that as the export filename.

Records in the exported CCR file have the following format:

20140510,20140510,04:40:57,04:41:59,3,src_ip,10.20.30.40,dst_ip,10.34.58.78,direction,outbound,1,bytes,70

The direction identifier is derived by comparing the src_ip identifier with the internal_net items defined in the configuration file. Where the source IP is within one of those CIDR ranges, the value of the direction identifier will be set to outbound, else it is inbound.

The start and end times of each record indicate the times (to the nearest second) that traffic between the two IP addresses was first and last seen. Very short flows may have identical start and end times. This is not an error, as Cloud Cruiser treats end times as inclusive.

Pushing to the Cloud Cruiser REST API

Unless disabled using the ccr_push_flag setting in the configuration file, as soon as a CCR file has been exported an attempt will be made to transfer it to Cloud Cruiser via the REST API. If this attempt is successful, then by default the data in the CCR file will be appended to a file called <usage_dir>/ccnetcollect/yyyymmdd-POST.txt on the Cloud Cruiser server.

The cc_push_dir option in the configuration file can be used to overwrite the ccnetcollect portion of the path above.

CCR files are not deleted by the Network Traffic Collector, and as such can be pushed to Cloud Cruiser at a later time in the event that the automatic push fails for any reason. CCR pushes to Cloud Cruiser are split into two main processes, which execute sequentially:

  1. An HTTP connection is made to the Cloud Cruiser server using the cc_serveraddr and cc_port values specified in the configuration file.
  2. A REST call is made to Cloud Cruiser at the URL <ccServerURL>/rest/v1/usage/ccnetcollect (assuming the default cc_push_dir value is used. As noted above, this can be overridden in the configuration file, in which case the latter portion of the above URL will be representative of the configured value).

Any errors occurring in step 1 are logged. Any errors occurring in step 2 also generate an entry in the log file but this entry will refer you to a second log file called ccnetcollect_push.log, which contains the details returned by the Cloud Cruiser server if any error occurred.

If an error occurs, and the automatic push fails, then it is necessary to perform a manual push to Cloud Cruiser once the underlying problem has been resolved. The collector comes with a Perl script called push_to_cc.pl to accomplish this. Care should be taken to avoid pushing the same file more than once to Cloud Cruiser, as this can result in double-charging. Therefore, when using push_to_cc.pl please check the log file to identify the CCR files that were not automatically pushed.

Last modified

Tags

This page has no custom tags.

Classifications

This page has no classifications.
© Copyright 2018 Hewlett Packard Enterprise Development LP