Troubleshooting the Cisco ACE can be really easy after you learn a few commands. On this page I have given some good examples of how to troubleshoot your Cisco ACE. Please note that there are many more troubleshooting commands and methods for the ACE at cisco.com/go/ace
Troubleshooting ACE Hardware
How to view the number of contexts and context names?
Cisco-ACE/Admin# show context
Number of Contexts = 4
Name: Admin , Id: 0
Config count: 143
Description:
Resource-class: default
FT Auto-sync running-cfg configured state: enabled
FT Auto-sync running-cfg actual state: enabled
FT Auto-sync startup-cfg configured state: enabled
FT Auto-sync startup-cfg actual state: enabled
Name: GN , Id: 1
Config count: 978
Description: GN-LB-Domain
Resource-class: GN
Vlans: Vlan60, Vlan64, Vlan124, Vlan188
FT Auto-sync running-cfg configured state: enabled
FT Auto-sync running-cfg actual state: enabled
FT Auto-sync startup-cfg configured state: enabled
FT Auto-sync startup-cfg actual state: enabled
Name: GN2 , Id: 2
Config count: 163
Description: GN2-LB-Domain
Resource-class: GN2
Vlans: Vlan260, Vlan266, Vlan224, Vlan288
FT Auto-sync running-cfg configured state: enabled
FT Auto-sync running-cfg actual state: enabled
FT Auto-sync startup-cfg configured state: enabled
FT Auto-sync startup-cfg actual state: enabled
How to view the licensed features on your Cisco ACE?
Cisco-ACE/Admin# show license usage
License Ins Lic Status Expiry Date Comments
Count
--------------------------------------------------------------------------------
ACE-08G-LIC No - Unused -
ACE-16G-LIC No - Unused -
ACE-UPG1-LIC No - Unused -
ACE-UPG2-LIC No - Unused -
ACE-VIRT-020 No - Unused -
ACE-VIRT-050 No - Unused -
ACE-VIRT-100 No - Unused -
ACE-VIRT-250 No - Unused -
ACE-VIRT-UP1 No - Unused -
ACE-VIRT-UP2 No - Unused -
ACE-VIRT-UP3 No - Unused -
ACE10-16G-LIC No - Unused -
ACE-SEC-LIC-K9 No - Unused -
ACE-SSL-05K-K9 No - Unused -
ACE-SSL-10K-K9 No - Unused -
ACE-SSL-15K-K9 No - Unused -
ACE-SSL-20K-K9 No - Unused -
ACE-SSL-UP1-K9 No - Unused -
ACE-SSL-UP2-K9 No - Unused -
ACE-SSL-UP3-K9 No - Unused -
--------------------------------------------------------------------------------
How to view the CPU utilization on a Cisco ACE?
Cisco-ACE/Context1# show processes cpu
CPU utilization for five seconds: 22%; one minute: 7%; five minutes: 7%
PID Runtime(ms) Invoked uSecs 1Sec 5 Sec 1 Min 5 Min Process
----- ----------- -------- ----- ----- ---- ---- ---- -----------
1 57974 722452 80 0.0 0.0 % 0.0 % 0.0 % init
2 7 1419 5 0.0 0.0 % 0.0 % 0.0 % keventd
3 308 114953 2 0.0 0.0 % 0.0 % 0.0 % ksoftirqd_CP
U0
4 481 144118 3 0.0 0.0 % 0.0 % 0.0 % ksoftirqd_CP
U1
5 0 1 6 0.0 0.0 % 0.0 % 0.0 % kswapd
6 0 1 12 0.0 0.0 % 0.0 % 0.0 % bdflush
7 13375564 1762848 7587 0.0 0.0 % 0.38% 0.41% kupdated
72 6858 60087 114 0.0 0.0 % 0.0 % 0.0 % cron
116 179 1732 103 0.0 0.0 % 0.0 % 0.0 % loop0
117 722 2143 337 0.0 0.0 % 0.0 % 0.0 % kjournald
123 43 119 368 0.0 0.0 % 0.0 % 0.0 % loop1
124 1547 118 13117 0.0 0.0 % 0.0 % 0.0 % kjournald
130 13043 393176 33 0.0 0.0 % 0.0 % 0.0 % loop2
What check points does the Cisco ACE have?
Cisco-ACE/Context1# show checkpoint all
-------------------------------------------------------------------------
Checkpoint Size(in bytes) Date(created on)
-------------------------------------------------------------------------
GN-09282010 19429 Tue Sep 28 14:54:53 2010
How to view the buffer statistics on a Cisco ACE?
Cisco-ACE/Context1# show buffer stats
Control Plane Buffer Statistics
-------------------------------
Pool Name: DefaultCtrl , Priority: High
Total Buffers : 75756 In Use : 32894
Total Allocated : 56125322 Hi Watermark : 75756
Total Freed : 56092428 Lo Watermark : 42859
Alloc Failures : 0
Pool Name: DefaultData , Priority: Normal
Total Buffers : 75756 In Use : 32768
Total Allocated : 163963755 Hi Watermark : 75756
Total Freed : 163930987 Lo Watermark : 41964
Alloc Failures : 0
Totals
Buffers : 151512 Allocated : 220089077
In Use : 65662 Freed : 220023415
How to find the Model and Serial Number of a Cisco ACE?
Cisco-ACE/Admin# show inventory
NAME: "module 1", DESCR: "Application Control Engine Service Module"
PID: ACE20-MOD-K9 , VID: V02, SN: <REMOVED>
Cisco-ACE/Context1# show hardware
Hardware
Product Number: ACE20-MOD-K9
Serial Number: <REMOVED>
Card Index: 207
Hardware Rev: 2.3
Feature Bits: 0000 0002
Slot No. : 1
Type: ACE
How to view the resource usage on a Cisco ACE?
Cisco-ACE/Context1# show resource usage all
Allocation
Resource Current Peak Min Max Denied
-------------------------------------------------------------------------------
Context: GN
conc-connections 2588 5140 0 8000000 0
mgmt-connections 10 42 0 100000 0
proxy-connections 0 29 0 1048574 0
xlates 0 0 0 1048574 0
bandwidth 53998 304609850 0 625000000 0
throughput 53998 304562438 0 500000000 0
mgmt-traffic rate 0 47412 0 125000000 0
connections rate 13 3368 0 1000000 0
ssl-connections rate 0 0 0 1000 0
mac-miss rate 0 2000 0 2000 2338
inspect-conn rate 0 0 0 6000 0
acl-memory 309056 309952 0 78610432 0
sticky 7 23 1384120 0 0
regexp 0 0 0 1048576 0
syslog buffer 3930112 3930112 0 4194304 0
syslog rate 5 2023 0 100000 0
Back to the top
If you found this helpful, help me by checking out the advertisement below. Thank you!
Troubleshooting ACE Fault Tolerance (High Availability)
When a pair of Cisco ACE modules or appliances are in a Active/Standby configuration, you would use the following commands to view their peer status, peering IP address and state (active or standby)
Cisco-ACE/Context1# show ft peer summary
Peer Id : 1
State : FSM_PEER_STATE_COMPATIBLE
Maintenance mode : MAINT_MODE_OFF
FT Vlan : 1000
FT Vlan IF State : UP
My IP Addr : 10.1.45.17
Peer IP Addr : 10.1.45.18
Query Vlan : 464
Query Vlan IF State : UP, Manual validation - please ping peer
Peer Query IP Addr : 10.1.200.254
Heartbeat Interval : 200
Heartbeat Count : 20
SRG Compatibility : COMPATIBLE
License Compatibility : COMPATIBLE
FT Groups : 4
Cisco-ACE/Context1# show ft peer detail
Peer Id : 1
State : FSM_PEER_STATE_COMPATIBLE
Maintenance mode : MAINT_MODE_OFF
FT Vlan : 1000
FT Vlan IF State : UP
My IP Addr : 10.1.45.17
Peer IP Addr : 10.1.45.18
Query Vlan : 464
Query Vlan IF State : UP, Manual validation - please ping peer
Peer Query IP Addr : 10.1.200.254
Heartbeat Interval : 200
Heartbeat Count : 20
Tx Packets : 460380
Tx Bytes : 133554040
Rx Packets : 440133
Rx Bytes : 100451149
Rx Error Bytes : 0
Tx Keepalive Packets : 439601
Rx Keepalive Packets : 439599
TL_CLOSE count : 1
FT_VLAN_DOWN count : 2
PEER_DOWN count : 5
SRG Compatibility : COMPATIBLE
License Compatibility : COMPATIBLE
FT Groups : 4
Cisco-ACE/Context1# show ft group brief
FT Group ID: 2 My State:FSM_FT_STATE_ACTIVE Peer State:FSM_FT_STATE_STANDBY_
HOT
Context Name: GN Context Id: 1
Cisco-ACE/Context1# show ft group status
FT Group : 2
Configured Status : in-service
Maintenance mode : MAINT_MODE_OFF
My State : FSM_FT_STATE_ACTIVE
Peer State : FSM_FT_STATE_STANDBY_HOT
Peer Id : 1
No. of Contexts : 1
Cisco-ACE/Context1#
Cisco-ACE/Context1# show ft group summary
FT Group : 2
Configured Status : in-service
Maintenance mode : MAINT_MODE_OFF
My State : FSM_FT_STATE_ACTIVE
My Config Priority : 100
My Net Priority : 100
My Preempt : Disabled
Peer State : FSM_FT_STATE_STANDBY_HOT
Peer Config Priority : 200
Peer Net Priority : 200
Peer Preempt : Disabled
Peer Id : 1
No. of Contexts : 1
Back to the top
If you found this helpful, help me by checking out the advertisement below. Thank you!
Troubleshooting ACE Traffic and Server Farms
How to view the status of a rserver, or physical server on a Cisco ACE?
Cisco-ACE/Context1# show rserver <SERVER-NAME>
rserver : TEST-SERVER2, type: HOST
state : OPERATIONAL (verified by arp response)
---------------------------------
----------connections-----------
real weight state current total
---+---------------------+------+------------+----------+--------------------
serverfarm: QA-TEST.443
192.168.45.76:0 8 PROBE-FAILED 0 0
serverfarm: QA-TEST.80
192.168.45.76:0 8 OPERATIONAL 0 1015
How to view the servers, status of the servers and connection count in a server farm on a Cisco ACE?
Cisco-ACE/Context1# show serverfarm <SERVER-NAME>
serverfarm : QA-TEST.443, type: HOST
total rservers : 2
---------------------------------
----------connections-----------
real weight state current total failures
---+---------------------+------+------------+----------+----------+---------
rserver: TEST-SERVER1
192.168.45.75:0 8 PROBE-FAILED 0 0 0
rserver: TEST-SERVER2
192.168.45.76:0 8 PROBE-FAILED 0 0 0
How to view all of the serverfarms and the loadbalancing method on a Cisco ACE?
Cisco-ACE/Context1# show serverfarm
serverfarm type rservers predictor current conns
+--------------------+---------+--------+------------------+---------------
QA-TEST.443
HOST 2 ROUNDROBIN 0
How to view the probe status on a Cisco ACE?
Cisco-ACE/Context1# show probe <PROBE-NAME>
probe : QA-TEST.443_PROBE
type : TCP
state : ACTIVE
----------------------------------------------
port : 443 address : 0.0.0.0 addr type : -
interval : 120 pass intvl : 300 pass count : 3
fail count: 3 recv timeout: 10
--------------------- probe results --------------------
probe association probed-address probes failed passed health
------------------- ---------------+----------+----------+----------+-------
serverfarm : QA-TEST.443
real : TEST-SERVER1[0]
192.168.45.75 11792 11792 0 FAILED
real : TEST-SERVER2[0]
192.168.45.76 11792 11792 0 FAILED
Cisco-ACE/Context1# show probe <PROBE-NAME> detail
probe : QA-TEST.443_PROBE
type : TCP
state : ACTIVE
description :
----------------------------------------------
port : 443 address : 0.0.0.0 addr type : -
interval : 120 pass intvl : 300 pass count : 3
fail count: 3 recv timeout: 10
conn termination : GRACEFUL
expect offset : 0 , open timeout : 10
expect regex : -
send data : -
--------------------- probe results --------------------
probe association probed-address probes failed passed health
------------------- ---------------+----------+----------+----------+-------
serverfarm : QA-TEST.443
real : TEST-SERVER1[0]
192.168.45.75 11792 11792 0 FAILED
Socket state : CLOSED
No. Passed states : 0 No. Failed states : 1
No. Probes skipped : 0 Last status code : 0
No. Out of Sockets : 0 No. Internal error: 0
Last disconnect err : Connection refused by server
Last probe time : Fri Aug 24 22:05:31 2012
Last fail time : Sat Jul 14 23:26:15 2012
Last active time : Never
real : TEST-SERVER2[0]
192.168.45.76 11792 11792 0 FAILED
Socket state : CLOSED
No. Passed states : 0 No. Failed states : 1
No. Probes skipped : 0 Last status code : 0
No. Out of Sockets : 0 No. Internal error: 0
Last disconnect err : Connection refused by server
Last probe time : Fri Aug 24 22:05:49 2012
Last fail time : Sat Jul 14 23:26:33 2012
Last active time : Never
Cisco-ACE/Context1# show probe <PROBE-NAME> detail
probe : QA-TEST.80_PROBE
type : TCP
state : ACTIVE
description :
----------------------------------------------
port : 80 address : 0.0.0.0 addr type : -
interval : 120 pass intvl : 300 pass count : 3
fail count: 3 recv timeout: 10
conn termination : GRACEFUL
expect offset : 0 , open timeout : 10
expect regex : -
send data : -
--------------------- probe results --------------------
probe association probed-address probes failed passed health
------------------- ---------------+----------+----------+----------+-------
serverfarm : QA-TEST.80
real : TEST-SERVER1[0]
192.168.45.75 29473 9 29464 SUCCESS
Socket state : CLOSED
No. Passed states : 2 No. Failed states : 1
No. Probes skipped : 0 Last status code : 0
No. Out of Sockets : 0 No. Internal error: 0
Last disconnect err : -
Last probe time : Fri Aug 24 22:09:54 2012
Last fail time : Tue Jul 24 16:50:00 2012
Last active time : Tue Jul 24 17:07:00 2012
real : TEST-SERVER2[0]
192.168.45.76 29473 10 29463 SUCCESS
Socket state : CLOSED
No. Passed states : 2 No. Failed states : 1
No. Probes skipped : 0 Last status code : 0
No. Out of Sockets : 0 No. Internal error: 0
Last disconnect err : -
Last probe time : Fri Aug 24 22:09:34 2012
Last fail time : Tue Jul 24 16:49:40 2012
Last active time : Tue Jul 24 17:06:40 2012
How to view sticky statistics on a Cisco ACE?
Cisco-ACE/Context1# show stats sticky
+------------------------------------------+
+----------- Sticky statistics ------------+
+------------------------------------------+
Total sticky entries reused : 0
prior to expiry
Total active sticky entries : 7
Total active reverse sticky : 0
entries
Total active sticky conns : 0
Total static sticky entries : 0
How to view connection statistics on a Cisco ACE?
Cisco-ACE/Context1# show stats connection
+------------------------------------------+
+------- Connection statistics ------------+
+------------------------------------------+
Total Connections Created : 93791059
Total Connections Current : 1998
Total Connections Destroyed: 36077222
Total Connections Timed-out: 33830946
Total Connections Failed : 23880893
How to view loadbalancing statistics on a Cisco ACE?
Cisco-ACE/Context1# show stats loadbalance
+------------------------------------------+
+------- Loadbalance statistics -----------+
+------------------------------------------+
Total version mismatch : 0
Total Layer4 decisions : 37211
Total Layer4 rejections : 307684
Total Layer7 decisions : 0
Total Layer7 rejections : 0
Total Layer4 LB policy misses : 0
Total Layer7 LB policy misses : 0
Total times rserver was unavailable : 0
Total ACL denied : 0
Total IDMap Lookup Failures : 0
Total Proxy misses : 0
Total Misc Errors : 0
Total L4 Close Before Process : 0
Total L7 Close Before Parse : 0
Total Close Msg for Valid Real : 49258
Total Close Msg for Invalid Real : 0
How to view Interface and route information on a Cisco ACE?
Cisco-ACE/Context1# show ip interface brief
Interface IP-Address Status Protocol
vlan60 10.1.60.2 up up
vlan64 10.1.200.7 up up
vlan124 10.1.124.2 up up
vlan188 192.168.45.2 up up
Cisco-ACE/Context1# show ip route
Routing Table for Context GN (RouteId 1)
Codes: H - host, I - interface
S - static, N - nat
A - need arp resolve, E - ecmp
Destination Gateway Interface Flags
------------------------------------------------------------------------
0.0.0.0 10.1.200.1 vlan464 S [0xc]
10.1.0.0/16 10.1.200.3 vlan464 S [0xc]
10.1.60.0/24 0.0.0.0 vlan160 IA [0x30]
Total route entries = 3
How to view traffic information on a Cisco ACE?
Cisco-ACE/Context1# show ip traffic
IP statistics:
Rcvd : 5406501323 total, 4745995588255 bytes
0 input errors, 0 no route
0 unknown protocol
Frags: 1614 reassembled, 0 couldn't reassemble
1614 fragmented, 0 couldn't fragment
Bcast: 30622 received, 0 sent
Mcast: 16867464 received, 0 sent
Sent : 5450608941 total, 4750784265385 bytes
0 no route
Drop : 0 no route, 0 out discarded
ICMP statistics:
Rcvd : 0 redirects, 0 unreachable
10208 echo, 0 echo reply, 0 mask requests, 0 mask replies, 0 que
nch
0 parameter, 0 timestamp
Sent : 0 redirects, 0 unreachable, 0 echo, 8950 echo reply
0 mask requests, 0 mask replies, 0 quench, 0 timestamp
0 parameter, 0 time exceeded
TCP statistics:
Rcvd : 0 total, 0 errors
Sent : 13045054 total
UDP statistics:
Rcvd : 528746 total, 0 errors, 332 no port
Sent : 53034939 total
ARP statistics:
Rcvd : 4123300 packets 0 Errors 535997 requests 413791 responses
Sent : 3355516 packets 23 Errors 2819244 requests 535997 responses
How to view connections going through a Cisco ACE?
Cisco-ACE/Context1# show conn
total current connections : 2426
conn-id np dir proto vlan source destination state
----------+--+---+-----+----+---------------------+---------------------+------+
553024 1 in TCP 160 10.1.60.33:64904 10.1.64.41:1548 ESTAB
556303 1 out TCP 464 10.1.64.41:1548 10.1.60.33:64904 ESTAB
933752 1 in UDP 464 10.1.200.7:1030 10.100.33.59:514 --
Cisco-ACE/Context1# show sticky database
sticky group : QATEST-GN.11111_STICKY
type : IP
timeout : 1440 timeout-activeconns : FALSE
sticky-entry rserver-instance time-to-expire flags
---------------------+--------------------------------+--------------+-------+
1778462976 QATEST-GN03:0 59358
Back to the top
|
