Within this article we will show you how to deny traffic based on the HTTP User-Agent header.

This is achieved by configuring a header-field-group. Within this group we define a header string rule that matches any header that does not contain a defined string. This group is then associated to a content rule.

header-field-group deny-agent
  header-field ua1 user-agent not-contain "spider"

content VIP-88.88.88.88
  protocol tcp
  vip address 88.88.88.88
  port 80
  url "/*"
  header-field-rule deny-agent
  add service server1
  add service server2

Additional Reference

http://www.cisco.com/en/US/docs/app_ntwk_services/data_center_app_services/css11000series/v5.00/configuration/advanced/guide/HTTPHead.html