The Multi-Vendor Networking Forum and Resources
Cisco ACE Troubleshooting

icon

Troubleshooting the Cisco ACE can be really easy after you learn a few commands. On this page I have given some good examples of how to troubleshoot your Cisco ACE. Please note that there are many more troubleshooting commands and methods for the ACE at cisco.com/go/ace



Troubleshooting ACE Hardware

How to view the number of contexts and context names?
Cisco-ACE/Admin# show context

Number of Contexts = 4

Name: Admin , Id: 0
Config count: 143
Description: 
Resource-class: default
FT Auto-sync running-cfg configured state: enabled
FT Auto-sync running-cfg actual state: enabled
FT Auto-sync startup-cfg configured state: enabled
FT Auto-sync startup-cfg actual state: enabled


Name: GN , Id: 1
Config count: 978
Description: GN-LB-Domain
Resource-class: GN
Vlans:  Vlan60, Vlan64, Vlan124, Vlan188
FT Auto-sync running-cfg configured state: enabled
FT Auto-sync running-cfg actual state: enabled
FT Auto-sync startup-cfg configured state: enabled
FT Auto-sync startup-cfg actual state: enabled


Name: GN2 , Id: 2
Config count: 163
Description: GN2-LB-Domain
Resource-class: GN2
Vlans:  Vlan260, Vlan266, Vlan224, Vlan288
FT Auto-sync running-cfg configured state: enabled
FT Auto-sync running-cfg actual state: enabled
FT Auto-sync startup-cfg configured state: enabled
FT Auto-sync startup-cfg actual state: enabled


How to view the licensed features on your Cisco ACE?
Cisco-ACE/Admin# show license usage
License                      Ins   Lic    Status   Expiry Date   Comments
                                  Count
--------------------------------------------------------------------------------
ACE-08G-LIC                   No     -    Unused                 -
ACE-16G-LIC                   No     -    Unused                 -
ACE-UPG1-LIC                  No     -    Unused                 -
ACE-UPG2-LIC                  No     -    Unused                 -
ACE-VIRT-020                  No     -    Unused                 -
ACE-VIRT-050                  No     -    Unused                 -
ACE-VIRT-100                  No     -    Unused                 -
ACE-VIRT-250                  No     -    Unused                 -
ACE-VIRT-UP1                  No     -    Unused                 -
ACE-VIRT-UP2                  No     -    Unused                 -
ACE-VIRT-UP3                  No     -    Unused                 -
ACE10-16G-LIC                 No     -    Unused                 -
ACE-SEC-LIC-K9                No     -    Unused                 -
ACE-SSL-05K-K9                No     -    Unused                 -
ACE-SSL-10K-K9                No     -    Unused                 -
ACE-SSL-15K-K9                No     -    Unused                 -
ACE-SSL-20K-K9                No     -    Unused                 -
ACE-SSL-UP1-K9                No     -    Unused                 -
ACE-SSL-UP2-K9                No     -    Unused                 -
ACE-SSL-UP3-K9                No     -    Unused                 -
--------------------------------------------------------------------------------



How to view the CPU utilization on a Cisco ACE?
Cisco-ACE/Context1# show processes cpu

CPU utilization for five seconds: 22%; one minute: 7%; five minutes: 7%
PID    Runtime(ms)  Invoked   uSecs  1Sec   5 Sec    1 Min   5 Min  Process
-----  -----------  --------  -----  -----   ----    ----    ----  -----------
    1        57974    722452     80    0.0   0.0 %   0.0 %   0.0 %  init
    2            7      1419      5    0.0   0.0 %   0.0 %   0.0 %  keventd
    3          308    114953      2    0.0   0.0 %   0.0 %   0.0 %  ksoftirqd_CP
U0
    4          481    144118      3    0.0   0.0 %   0.0 %   0.0 %  ksoftirqd_CP
U1
    5            0         1      6    0.0   0.0 %   0.0 %   0.0 %  kswapd
    6            0         1     12    0.0   0.0 %   0.0 %   0.0 %  bdflush
    7     13375564   1762848   7587    0.0   0.0 %   0.38%   0.41%  kupdated
   72         6858     60087    114    0.0   0.0 %   0.0 %   0.0 %  cron
  116          179      1732    103    0.0   0.0 %   0.0 %   0.0 %  loop0
  117          722      2143    337    0.0   0.0 %   0.0 %   0.0 %  kjournald
  123           43       119    368    0.0   0.0 %   0.0 %   0.0 %  loop1
  124         1547       118  13117    0.0   0.0 %   0.0 %   0.0 %  kjournald
  130        13043    393176     33    0.0   0.0 %   0.0 %   0.0 %  loop2
 
What check points does the Cisco ACE have?
Cisco-ACE/Context1# show checkpoint all

-------------------------------------------------------------------------
Checkpoint                    Size(in bytes)     Date(created on)
-------------------------------------------------------------------------
GN-09282010                  19429              Tue Sep 28 14:54:53 2010

How to view the buffer statistics on a Cisco ACE?
Cisco-ACE/Context1# show buffer stats

Control Plane Buffer Statistics
-------------------------------
Pool Name: DefaultCtrl , Priority: High
Total Buffers   : 75756        In Use       : 32894    
Total Allocated : 56125322     Hi Watermark : 75756    
Total Freed     : 56092428     Lo Watermark : 42859    
Alloc Failures  : 0         

Pool Name: DefaultData , Priority: Normal
Total Buffers   : 75756        In Use       : 32768    
Total Allocated : 163963755    Hi Watermark : 75756    
Total Freed     : 163930987    Lo Watermark : 41964    
Alloc Failures  : 0         

Totals
Buffers : 151512       Allocated : 220089077
In Use  : 65662        Freed     : 220023415

How to find the Model and Serial Number of a Cisco ACE? 
Cisco-ACE/Admin# show inventory

 NAME: "module 1", DESCR: "Application Control Engine Service Module"
 PID: ACE20-MOD-K9      , VID: V02, SN: <REMOVED>
 
 Cisco-ACE/Context1# show hardware

Hardware
  Product Number: ACE20-MOD-K9
  Serial Number:  <REMOVED>
  Card Index:     207
  Hardware Rev:   2.3
  Feature Bits:   0000 0002
  Slot No. :      1
  Type:           ACE
 
How to view the resource usage on a Cisco ACE?
  Cisco-ACE/Context1# show resource usage all
                                                     Allocation
        Resource         Current       Peak        Min        Max       Denied
-------------------------------------------------------------------------------
Context: GN
  conc-connections           2588       5140          0    8000000          0
  mgmt-connections             10         42          0     100000          0
  proxy-connections             0         29          0    1048574          0
  xlates                        0          0          0    1048574          0
  bandwidth                 53998  304609850          0  625000000          0
    throughput              53998  304562438          0  500000000          0
    mgmt-traffic rate           0      47412          0  125000000          0
  connections rate             13       3368          0    1000000          0
  ssl-connections rate          0          0          0       1000          0
  mac-miss rate                 0       2000          0       2000       2338
  inspect-conn rate             0          0          0       6000          0
  acl-memory               309056     309952          0   78610432          0
  sticky                        7         23    1384120          0          0
  regexp                        0          0          0    1048576          0
  syslog buffer           3930112    3930112          0    4194304          0
  syslog rate                   5       2023          0     100000          0
 
  Back to the top

 

 

 
Troubleshooting ACE Fault Tolerance (High Availability)


When a pair of Cisco ACE modules or appliances are in a Active/Standby configuration, you would use the following commands to view their peer status, peering IP address and state (active or standby)


Cisco-ACE/Context1# show ft peer summary

Peer Id                      : 1
State                        : FSM_PEER_STATE_COMPATIBLE
Maintenance mode             : MAINT_MODE_OFF
FT Vlan                      : 1000
FT Vlan IF State             : UP
My IP Addr                   : 10.1.45.17
Peer IP Addr                 : 10.1.45.18
Query Vlan                   : 464
Query Vlan IF State          : UP, Manual validation - please ping peer
Peer Query IP Addr           : 10.1.200.254
Heartbeat Interval           : 200
Heartbeat Count              : 20
SRG Compatibility            : COMPATIBLE
License Compatibility        : COMPATIBLE
FT Groups                    : 4

Cisco-ACE/Context1# show ft peer detail

Peer Id                      : 1
State                        : FSM_PEER_STATE_COMPATIBLE
Maintenance mode             : MAINT_MODE_OFF
FT Vlan                      : 1000
FT Vlan IF State             : UP
My IP Addr                   : 10.1.45.17
Peer IP Addr                 : 10.1.45.18
Query Vlan                   : 464
Query Vlan IF State          : UP, Manual validation - please ping peer
Peer Query IP Addr           : 10.1.200.254
Heartbeat Interval           : 200
Heartbeat Count              : 20
Tx Packets                   : 460380
Tx Bytes                     : 133554040
Rx Packets                   : 440133
Rx Bytes                     : 100451149
Rx Error Bytes               : 0
Tx Keepalive Packets         : 439601
Rx Keepalive Packets         : 439599
TL_CLOSE count               : 1
FT_VLAN_DOWN count           : 2
PEER_DOWN count              : 5
SRG Compatibility            : COMPATIBLE
License Compatibility        : COMPATIBLE
FT Groups                    : 4

Cisco-ACE/Context1# show ft group brief

FT Group ID: 2  My State:FSM_FT_STATE_ACTIVE    Peer State:FSM_FT_STATE_STANDBY_
HOT
                Context Name: GN       Context Id: 1
               
Cisco-ACE/Context1# show ft group status

FT Group                     : 2
Configured Status            : in-service
Maintenance mode             : MAINT_MODE_OFF
My State                     : FSM_FT_STATE_ACTIVE
Peer State                   : FSM_FT_STATE_STANDBY_HOT
Peer Id                      : 1
No. of Contexts              : 1
Cisco-ACE/Context1#

Cisco-ACE/Context1# show ft group summary

FT Group                     : 2
Configured Status            : in-service
Maintenance mode             : MAINT_MODE_OFF
My State                     : FSM_FT_STATE_ACTIVE
My Config Priority           : 100
My Net Priority              : 100
My Preempt                   : Disabled
Peer State                   : FSM_FT_STATE_STANDBY_HOT
Peer Config Priority         : 200
Peer Net Priority            : 200
Peer Preempt                 : Disabled
Peer Id                      : 1
No. of Contexts              : 1


Back to the top

If you found this helpful, help me by checking out the advertisement below. Thank you!


Troubleshooting ACE Traffic and Server Farms



How to view the status of a rserver, or physical server on a Cisco ACE? 
Cisco-ACE/Context1# show rserver <SERVER-NAME>

 rserver              : TEST-SERVER2, type: HOST
 state                : OPERATIONAL (verified by arp response)
 ---------------------------------
                                                ----------connections-----------
       real                  weight state        current    total              
   ---+---------------------+------+------------+----------+--------------------
   serverfarm: QA-TEST.443
       192.168.45.76:0        8      PROBE-FAILED 0          0                  
   serverfarm: QA-TEST.80
       192.168.45.76:0        8      OPERATIONAL  0          1015  
      
How to view the servers, status of the servers and connection count in a server farm on a Cisco ACE?
Cisco-ACE/Context1#
show serverfarm <SERVER-NAME>
 serverfarm     : QA-TEST.443, type: HOST
 total rservers : 2
 ---------------------------------
                                                ----------connections-----------
       real                  weight state        current    total      failures
   ---+---------------------+------+------------+----------+----------+---------
   rserver: TEST-SERVER1
       192.168.45.75:0        8      PROBE-FAILED 0          0          0
   rserver: TEST-SERVER2
       192.168.45.76:0        8      PROBE-FAILED 0          0          0
      
 How to view all of the serverfarms and the loadbalancing method on a Cisco ACE?
Cisco-ACE/Context1# show serverfarm

   serverfarm           type      rservers predictor          current conns  
  +--------------------+---------+--------+------------------+---------------
   QA-TEST.443
                        HOST      2        ROUNDROBIN         0        
 
 
How to view the probe status on a Cisco ACE?
Cisco-ACE/Context1# show probe
<PROBE-NAME>

 probe       : QA-TEST.443_PROBE
 type        : TCP
 state       : ACTIVE
----------------------------------------------
   port      : 443     address     : 0.0.0.0         addr type  : -          
   interval  : 120     pass intvl  : 300             pass count : 3   
   fail count: 3       recv timeout: 10  
                       --------------------- probe results --------------------
   probe association   probed-address  probes     failed     passed     health
   ------------------- ---------------+----------+----------+----------+-------
   serverfarm  : QA-TEST.443
     real      : TEST-SERVER1[0]
                       192.168.45.75    11792      11792      0          FAILED 
     real      : TEST-SERVER2[0]
                       192.168.45.76    11792      11792      0          FAILED 


Cisco-ACE/Context1# show probe <PROBE-NAME> detail

 probe       : QA-TEST.443_PROBE
 type        : TCP
 state       : ACTIVE
 description :
----------------------------------------------
   port      : 443     address     : 0.0.0.0         addr type  : -          
   interval  : 120     pass intvl  : 300             pass count : 3   
   fail count: 3       recv timeout: 10  
   conn termination : GRACEFUL 
   expect offset    : 0         , open timeout     : 10       
   expect regex     : -
   send data        : -
                       --------------------- probe results --------------------
   probe association   probed-address  probes     failed     passed     health
   ------------------- ---------------+----------+----------+----------+-------
   serverfarm  : QA-TEST.443
     real      : TEST-SERVER1[0]
                       192.168.45.75    11792      11792      0          FAILED 

   Socket state        : CLOSED
   No. Passed states   : 0         No. Failed states : 1
   No. Probes skipped  : 0         Last status code  : 0
   No. Out of Sockets  : 0         No. Internal error: 0
   Last disconnect err : Connection refused by server
   Last probe time     : Fri Aug 24 22:05:31 2012
   Last fail time      : Sat Jul 14 23:26:15 2012
   Last active time    : Never

     real      : TEST-SERVER2[0]
                       192.168.45.76    11792      11792      0          FAILED 

   Socket state        : CLOSED
   No. Passed states   : 0         No. Failed states : 1
   No. Probes skipped  : 0         Last status code  : 0
   No. Out of Sockets  : 0         No. Internal error: 0
   Last disconnect err : Connection refused by server
   Last probe time     : Fri Aug 24 22:05:49 2012
   Last fail time      : Sat Jul 14 23:26:33 2012
   Last active time    : Never
                      
Cisco-ACE/Context1# show probe <PROBE-NAME> detail

 probe       : QA-TEST.80_PROBE
 type        : TCP
 state       : ACTIVE
 description :
----------------------------------------------
   port      : 80      address     : 0.0.0.0         addr type  : -          
   interval  : 120     pass intvl  : 300             pass count : 3   
   fail count: 3       recv timeout: 10  
   conn termination : GRACEFUL 
   expect offset    : 0         , open timeout     : 10       
   expect regex     : -
   send data        : -
                       --------------------- probe results --------------------
   probe association   probed-address  probes     failed     passed     health
   ------------------- ---------------+----------+----------+----------+-------
   serverfarm  : QA-TEST.80
     real      : TEST-SERVER1[0]
                       192.168.45.75    29473      9          29464      SUCCESS

   Socket state        : CLOSED
   No. Passed states   : 2         No. Failed states : 1
   No. Probes skipped  : 0         Last status code  : 0
   No. Out of Sockets  : 0         No. Internal error: 0
   Last disconnect err :  -
   Last probe time     : Fri Aug 24 22:09:54 2012
   Last fail time      : Tue Jul 24 16:50:00 2012
   Last active time    : Tue Jul 24 17:07:00 2012

     real      : TEST-SERVER2[0]
                       192.168.45.76    29473      10         29463      SUCCESS

   Socket state        : CLOSED
   No. Passed states   : 2         No. Failed states : 1
   No. Probes skipped  : 0         Last status code  : 0
   No. Out of Sockets  : 0         No. Internal error: 0
   Last disconnect err :  -
   Last probe time     : Fri Aug 24 22:09:34 2012
   Last fail time      : Tue Jul 24 16:49:40 2012
   Last active time    : Tue Jul 24 17:06:40 2012

How to view sticky statistics on a Cisco ACE?
Cisco-ACE/Context1# show stats sticky

+------------------------------------------+
+----------- Sticky statistics ------------+
+------------------------------------------+
 Total sticky entries reused    : 0
 prior to expiry
 Total active sticky entries    : 7
 Total active reverse sticky    : 0
 entries
 Total active sticky conns      : 0
 Total static sticky entries    : 0
 
How to view connection statistics on a Cisco ACE?
 Cisco-ACE/Context1# show stats connection

+------------------------------------------+
+------- Connection statistics ------------+
+------------------------------------------+
 Total Connections Created  : 93791059
 Total Connections Current  : 1998
 Total Connections Destroyed: 36077222
 Total Connections Timed-out: 33830946
 Total Connections Failed   : 23880893
 
How to view loadbalancing statistics on a Cisco ACE?
Cisco-ACE/Context1# show stats loadbalance

+------------------------------------------+
+------- Loadbalance statistics -----------+
+------------------------------------------+
 Total version mismatch              : 0
 Total Layer4 decisions              : 37211
 Total Layer4 rejections             : 307684
 Total Layer7 decisions              : 0
 Total Layer7 rejections             : 0
 Total Layer4 LB policy misses       : 0
 Total Layer7 LB policy misses       : 0
 Total times rserver was unavailable : 0
 Total ACL denied                    : 0
 Total IDMap Lookup Failures         : 0
 Total Proxy misses                  : 0
 Total Misc Errors                   : 0
 Total L4 Close Before Process       : 0
 Total L7 Close Before Parse         : 0
 Total Close Msg for Valid Real      : 49258
 Total Close Msg for Invalid Real    : 0
 
How to view Interface and route information on a Cisco ACE? 
  Cisco-ACE/Context1# show ip interface brief
Interface       IP-Address      Status                  Protocol
vlan60         10.1.60.2      up                      up 
vlan64         10.1.200.7     up                      up 
vlan124        10.1.124.2     up                      up 
vlan188        192.168.45.2     up                      up 

Cisco-ACE/Context1# show ip route

Routing Table for Context GN (RouteId 1)

   Codes: H - host,   I - interface
          S - static,      N - nat
          A - need arp resolve,      E - ecmp

Destination         Gateway          Interface         Flags
------------------------------------------------------------------------
0.0.0.0             10.1.200.1      vlan464           S [0xc]
10.1.0.0/16        10.1.200.3      vlan464           S [0xc]
10.1.60.0/24       0.0.0.0          vlan160           IA [0x30]

Total route entries = 3

 


How to view traffic information on a Cisco ACE?

Cisco-ACE/Context1# show ip traffic
IP statistics:
        Rcvd :  5406501323 total, 4745995588255 bytes
                0 input errors, 0 no route
                0 unknown protocol
        Frags:  1614 reassembled, 0 couldn't reassemble
                1614 fragmented, 0 couldn't fragment
        Bcast:  30622 received, 0 sent
        Mcast:  16867464 received, 0 sent
        Sent :  5450608941 total, 4750784265385 bytes
                0 no route
        Drop :  0 no route, 0 out discarded
ICMP statistics:
        Rcvd : 0 redirects,  0 unreachable
                10208 echo, 0 echo reply, 0 mask requests, 0 mask replies, 0 que
nch
                0 parameter, 0  timestamp
        Sent : 0 redirects,  0 unreachable, 0 echo, 8950 echo reply
                0 mask requests, 0 mask replies, 0 quench, 0 timestamp
                0 parameter, 0  time exceeded
TCP statistics:
        Rcvd : 0 total,  0 errors
        Sent : 13045054 total
UDP statistics:
        Rcvd : 528746 total,  0 errors, 332 no port
        Sent : 53034939 total
ARP statistics:
        Rcvd : 4123300 packets 0 Errors 535997 requests 413791 responses
        Sent : 3355516 packets 23 Errors 2819244 requests 535997 responses
       
How to view connections going through a Cisco ACE?
 Cisco-ACE/Context1# show conn

total current connections : 2426

conn-id    np dir proto vlan source                destination           state
----------+--+---+-----+----+---------------------+---------------------+------+
553024     1  in  TCP   160  10.1.60.33:64904     10.1.64.41:1548      ESTAB
556303     1  out TCP   464  10.1.64.41:1548      10.1.60.33:64904     ESTAB
933752     1  in  UDP   464  10.1.200.7:1030      10.100.33.59:514     -- 

How to view the sticky database on a Cisco ACE?
  Cisco-ACE/Context1# show sticky database
sticky group : QATEST-GN.11111_STICKY
type         : IP   
timeout      : 1440          timeout-activeconns : FALSE
  sticky-entry          rserver-instance                 time-to-expire flags  
  ---------------------+--------------------------------+--------------+-------+
  1778462976            QATEST-GN03:0                  59358
 
How to view the service policy on a Cisco ACE?
Cisco-ACE/Admin# show service-policy POLICY summary

service-policy: POLICY
Class                            VIP             Prot  Port        VLAN          State    Curr Conns   Hit Count  Conns Drop
GOATSERVER-APP1.1159_CLASS      10.5.250.25    tcp   eq 1159     160,44       OUT-SRVC          0     2674263    2674263
GOATSERVER-APP1.4889_CLASS      10.5.250.25    tcp   eq 4889     160,44       OUT-SRVC          0           0          0
GOATSERVER-APP1.8080_CLASS      10.5.250.26    tcp   eq 8080     160,464       IN-SRVC           0       11629          0

How to view a specific service policy on a Cisco ACE?
Cisco-ACE/Admin# show service-policy GOAT-POLICY-DEV    GOAT-POLICY-DEV = YOUR POLICY'S NAME

Status     : ACTIVE
-----------------------------------------
Interface: vlan 44 124
  service-policy: GOAT-POLICY-DEV
    class: REAL_SERVERS_DEV
      nat:
        nat dynamic 1 vlan 124
        curr conns       : 0         , hit count        : 0        
        dropped conns    : 0        
        client pkt count : 0         , client byte count: 0                  
        server pkt count : 0         , server byte count: 0                  
        conn-rate-limit      : 0         , drop-count : 0        
        bandwidth-rate-limit : 0         , drop-count : 0        
    class: GOATSERVER-APP1.8080_CLASS
      loadbalance:
        L7 loadbalance policy: GOATSERVER-APP1.8080_POLICY
        VIP Route Metric     : 77
        VIP Route Advertise  : DISABLED
        VIP ICMP Reply       : ENABLED-WHEN-ACTIVE
        VIP State: INSERVICE
        curr conns       : 0         , hit count        : 0        
        dropped conns    : 0        
        client pkt count : 0         , client byte count: 0                  
        server pkt count : 0         , server byte count: 0                  
        conn-rate-limit      : 0         , drop-count : 0        
        bandwidth-rate-limit : 0         , drop-count : 0

Back to the top