Chapter 7: Contrail Analytics :

The Analytics nodes collects the system state information such as usage statistics, debug information from all of the software modules across all of the nodes of the system(config,control,vrouter, etc).

Analytics stores the data in Cassandra DB, This data can be queries by REST APIs.

Contrail Analytics consists of the below services.

cloud@opencontrail:~/manage-scripts$ ./contrail-analytics.sh status
contrail-collector               RUNNING    pid 29716, uptime 10:45:11
contrail-query-engine            RUNNING    pid 29740, uptime 10:45:06
contrail-analytics-api           RUNNING    pid 29773, uptime 10:45:01
contrail-snmp-collector          RUNNING    pid 29788, uptime 10:44:55
contrail-topology                RUNNING    pid 29801, uptime 10:44:50
contrail-alarm-gen               RUNNING    pid 29836, uptime 10:44:44
cloud@opencontrail:~/manage-scripts$

Services:

Contrail-topology, Contrail-snmp-collector :

Contrail-topology, Contrail-snmp-collector used for Underlay Overlay Analytics.

Contrail-snmp-collector polls the underlay devices(physical router) and collects the information. Contrail-topology uses these information and prepares the topology with overlay and underlay information. More details are here

The prouter uve can be get it from below api.

http://10.0.1.4:8081/analytics/uves/prouters

Note: So we dont have to worry much about these two services . Mostly this will be useful for Production environment.

Contrail-Collector, Contrail-query-engine, Contrail-analytics-api :

contrail-collector receives status, logs, and flow information from all modules (for example, generators) and records them in cassandra.

contrail-analytics-api(contrail-opserver) provides an external REST API to report UVEs and to query logs and flows. The default port (8081). Some more details are here.

Contrail-query-engine interface with cassandra server, and retrieves the data based on the query (given by contrail-analytics-api).

The Flow is below,

  • Contrail-query-engine listens on 8091

  • contrail-analytics-api receives the REST API query from the User, and send this query to the Contrail-query-engine on port 8091.

  • Contrail-Query-engine interacts with cassandra and runs the query and retrieves the data. Place this data to the REDIS server.

  • Contrail-analytics-api reads the data from REDIS and respond back to the user.

Contrail-alarm-gen:

To be updated

API Services :

In my setup, contrail analytics runs in 10.0.1.6.

Understanding the Contrail Analytics API

Contrail Analytics exposes the REST Interface in port 8081.

GET  http://10.0.1.6:8081
{
  "href": "http://10.0.1.6:8081",
  "links": [
    {
      "link": {
        "href": "http://10.0.1.6:8081/documentation/index.html",
        "name": "documentation"
      }
    },
    {
      "link": {
        "href": "http://10.0.1.6:8081/documentation/messages/index.html",
        "name": "Message documentation"
      }
    },
    {
      "link": {
        "href": "http://10.0.1.6:8081/analytics",
        "name": "analytics"
      }
    }
  ]
}

Analytics API

/analytics endpoint is used for querying the analytics node.

GET http://10.0.1.6:8081/analytics

[ { "href": "http://10.0.1.6:8081/analytics/uves", "name": "uves" },
{ "href": "http://10.0.1.6:8081/analytics/uve-types", "name": "uve-types" },
 { "href": "http://10.0.1.6:8081/analytics/tables", "name": "tables" },
 { "href": "http://10.0.1.6:8081/analytics/queries", "name": "queries" },
  { "href": "http://10.0.1.6:8081/analytics/alarms", "name": "alarms" },
  { "href": "http://10.0.1.6:8081/analytics/uve-stream", "name": "uve-stream" },
   { "href": "http://10.0.1.6:8081/analytics/alarm-stream", "name": "alarm-stream" }
   ]

The Analytics API functions are broadly classified as "uves", "query the logs" and alarms. uve-stream, alarm-stream are stream APIs for uve and alarm functions.

UVEs

In simple terms, User Visible Entities are Contrail resources (virtual-network, virtual-machines, vrouter, routing-instances, etc). UVE APIs are used to query these resources.

GET http://10.0.1.6:8081/analytics/uves

[
{
    "href": "http://10.0.1.6:8081/analytics/uves/analytics-nodes",
    "name": "analytics-nodes"
  },
  {
    "href": "http://10.0.1.6:8081/analytics/uves/config-nodes",
    "name": "config-nodes"
  },
  {
    "href": "http://10.0.1.6:8081/analytics/uves/virtual-machines",
    "name": "virtual-machines"
  },
  {
    "href": "http://10.0.1.6:8081/analytics/uves/control-nodes",
    "name": "control-nodes"
  },
{
    "href": "http://10.0.1.6:8081/analytics/uves/virtual-networks",
    "name": "virtual-networks"
  },
 ................................(strippped )
]

Let us query the virtual-machines

GET http://10.0.1.6:8081/analytics/uves/virtual-machines

[
  {
    "href": "http://10.0.1.6:8081/analytics/uves/virtual-machine/f0240c92-12ca-4774-8b16-f42540dc9a1d?flat",
    "name": "f0240c92-12ca-4774-8b16-f42540dc9a1d"
  },
  {
    "href": "http://10.0.1.6:8081/analytics/uves/virtual-machine/2783564a-1d75-4302-b0fe-dd32b10ce2f5?flat",
    "name": "2783564a-1d75-4302-b0fe-dd32b10ce2f5"
  },
  {
    "href": "http://10.0.1.6:8081/analytics/uves/virtual-machine/2e6502c2-8e63-4081-a352-efcdaf56f905?flat",
    "name": "2e6502c2-8e63-4081-a352-efcdaf56f905"
  }
]

We have 3 Virtual-Machines running.Let us query 1 VM with ID, to get the full details of the VM.

GET http://10.0.1.6:8081/analytics/uves/virtual-machine/f0240c92-12ca-4774-8b16-f42540dc9a1d?flat

{
  "VirtualMachineStats": {
    "cpu_stats": [
      {
        "virt_memory": 1609064,
        "cpu_one_min_avg": 0.166667,
        "disk_used_bytes": 10620928,
        "vm_memory_quota": 131072,
        "peak_virt_memory": 2101852,
        "disk_allocated_bytes": 25165824,
        "rss": 225488
      }
    ]
  },
  "UveVirtualMachineAgent": {
    "udp_sport_bitmap": [
      "0",
      "0",
      "0",
      "0",
      "0",
      "2",
      "2097152",
      "0"
    ],
    "vm_name": "instance2",
    "tcp_sport_bitmap": [
      "1",
      "0",
      "0",
      "0",
      "0",
      "0",
      "32768",
      "0"
    ],
    "uuid": "f0240c92-12ca-4774-8b16-f42540dc9a1d",
    "udp_dport_bitmap": [
      "1",
      "0",
      "0",
      "0",
      "0",
      "0",
      "0",
      "0"
    ],
    "cpu_info": {
      "virt_memory": 1609064,
      "cpu_one_min_avg": 0.166667,
      "disk_used_bytes": 10620928,
      "vm_memory_quota": 131072,
      "peak_virt_memory": 2101852,
      "disk_allocated_bytes": 25165824,
      "rss": 225488
    },
    "vrouter": "devstack1",
    "tcp_dport_bitmap": [
      "1",
      "0",
      "0",
      "0",
      "0",
      "0",
      "32768",
      "0"
    ],
    "interface_list": [
      "default-domain:demo:2f877af3-b958-4444-84ad-f37c76d0338c"
    ]
  }
}

Query APIs (LOGs and Flows) :

Contrail Analytics stores all the logs/flows information of all nodes in to Cassandra DB. we can POST the proper query and retrive the information.

Reference : http://www.juniper.net/documentation/en_US/contrail3.2/topics/task/configuration/analytics-apis-log-flow-vnc.html

This below API response, the status of queries (errors, abanodend, processing...)

GET http://10.0.1.6:8081/analytics/queries

To know the list of tables contrail analytics have. We can perform the query for these tables.

GET http://10.0.1.6:8081/analytics/tables

[
  {
    "href": "http://10.0.1.6:8081/analytics/table/MessageTable",
    "type": "LOG",
    "name": "MessageTable"
  },
  {
    "href": "http://10.0.1.6:8081/analytics/table/FlowRecordTable",
    "type": "FLOW",
    "name": "FlowRecordTable"
  },
  {
    "href": "http://10.0.1.6:8081/analytics/table/FlowSeriesTable",
    "type": "FLOW",
    "name": "FlowSeriesTable"
  },
  {
    "href": "http://10.0.1.6:8081/analytics/table/OverlayToUnderlayFlowMap",
    "type": "FLOW",
    "name": "OverlayToUnderlayFlowMap"
  },
  {
    "display_name": "Service Chain",
    "href": "http://10.0.1.6:8081/analytics/table/ServiceChain",
    "type": "OBJECT",
    "name": "ServiceChain"
  },
  {
    "display_name": "Database Node",
    "href": "http://10.0.1.6:8081/analytics/table/ObjectDatabaseInfo",
    "type": "OBJECT",
    "name": "ObjectDatabaseInfo"
  }
  stripped...
  ]

To know the schema/other info of these tables, just GET API for the tables.

Query API

We have to construct the query(like SQL query) and POST it to the Analytics on.

Some reference.

Ex:

In table name "MessageTable",

  • We want to display fields of ["MessageTS", "Source", "ModuleId", "Category", "Messagetype", "SequenceNum", "Xmlmessage", "Type", "Level", "NodeType", "InstanceId"],
  • Where the condition is "name": "ModuleId", "value": "contrail-control",
  • Starting time
  • End Time

POST http://10.0.1.6:8081/analytics/query

Body:

{
"sort": 1, 
"start_time": "now-10m", 
"sort_fields": ["MessageTS"], 
"end_time": "now", 
"select_fields": ["MessageTS", "Source", "ModuleId", "Category", "Messagetype", "SequenceNum", "Xmlmessage", "Type", "Level", "NodeType", "InstanceId"], 
"table": "MessageTable", 
"where": [[{"suffix": null, "value2": null, "name": "ModuleId", "value": "contrail-control", "op": 1}]]

}

Response:

{
  "value": [
    {
      "Category": "IFMapPeer",
      "InstanceId": "0",
      "Level": 7,
      "MessageTS": 1494932372766850,
      "Messagetype": "IFMapDSResp",
      "ModuleId": "contrail-control",
      "NodeType": "Control",
      "SequenceNum": 14208,
      "Source": "opencontrail",
      "Type": 1,
      "Xmlmessage": "<IFMapDSResp type=\"sandesh\"><current_peer type=\"string\" identifier=\"1\">current_peer is 10.0.1.4:8443:in_use</current_peer><new_response type=\"string\" identifier=\"2\">DSResp is 10.0.1.4:8443, </new_response><file type=\"string\" identifier=\"-32768\">controller/src/ifmap/client/peer_server_finder.cc</file><line type=\"i32\" identifier=\"-32767\">132</line></IFMapDSResp>"
    },
    {
      "Category": null,
      "InstanceId": "0",
      "Level": 2147483647,
      "MessageTS": 1494932375338803,
      "Messagetype": "SandeshModuleClientTrace",
      "ModuleId": "contrail-control",
      "NodeType": "Control",
      "SequenceNum": 30327,
      "Source": "opencontrail",
      "Type": 6,
      "Xmlmessage": "<SandeshModuleClientTrace type=\"sandesh\"><data type=\"struct\" identifier=\"1\"><ModuleClientState><name type=\"string\" identifier=\"1\" key=\"ObjectGeneratorInfo\">opencontrail:Control:contrail-control:0</name><sm_queue_count type=\"u64\" identifier=\"5\">0</sm_queue_count><sm_stats type=\"struct\" identifier=\"6\"><SandeshStateMachineStats><ev_stats type=\"list\" identifier=\"1\"><"
      }
      ]
}
stripped

you can get all the logs, table details using the Query API.

Contrail-Logs Command :

Contrail-logs command is CLI for querying the analyzer and get the logs. Utilizing the query API , and made user friendly switches.

  1. contrail-logs --module contrail-control --start-time now-1m --end-time now
cloud@opencontrail:~$ contrail-logs --verbose --module contrail-control --start-time now-1m --end-time now

Performing query: {"sort": 1, "start_time": 1494933514431957, "sort_fields": ["MessageTS"], "end_time": 1494933574431957, "select_fields": ["MessageTS", "Source", "ModuleId", "Category", "Messagetype", "SequenceNum", "Xmlmessage", "Type", "Level", "NodeType", "InstanceId"], "table": "MessageTable", "where": [[{"suffix": null, "value2": null, "name": "ModuleId", "value": "contrail-control", "op": 1}]]}

2017 May 16 11:18:35.496732 opencontrail [Control:contrail-control:0:None][INVALID] : SandeshModuleClientTrace:30477 [ModuleClientState: name = opencontrail:Control:contrail-control:0, sm_queue_count = 0, [SandeshStateMachineStats: [ev_stats: [event = scm::EvDiscUpdate, enqueues = 14315, dequeues = 14315, enqueue_fails = 0, dequeue_fails = 0] [event = scm::EvIdleHoldTimerExpired, enqueues = 74, dequeues = 74, enqueue_fails = 0, dequeue_fails = 0] [event = scm::EvSandeshMessageRecv, enqueues = 20, dequeues = 20, enqueue_fails = 0, dequeue_fails = 0] [event = scm::EvSandeshSend, enqueues = 73442, dequeues = 73442, enqueue_fails = 0, dequeue_fails = 0] [event = scm::EvStart, enqueues = 1, dequeues = 1, enqueue_fails = 0, dequeue_fails = 0] [event = scm::EvTcpClose, enqueues = 13, dequeues = 13, enqueue_fails = 0, dequeue_fails = 0] [event = scm::EvTcpConnectFail, enqueues = 61, dequeues = 61, enqueue_fails = 0, dequeue_fails = 0] [event = scm::EvTcpConnected, enqueues = 14, dequeues = 14, enqueue_fails = 0, dequeue_fails = 0] [event = scm::EvTcpDeleteSession, enqueues = 74, dequeues = 74, enqueue_fails = 0, dequeue_fails = 0] [event = , enqueues = 88014, dequeues = 88014, enqueue_fails = 0, dequeue_fails = 0]], state = Established, state_since = 1494881745602577, last_state = Established, last_event = scm::EvSandeshSend, last_event_at = 1494933511345984], [SandeshSessionStats: num_recv_msg = 0, num_recv_msg_fail = 0, num_recv_fail = 0, num_send_msg = 15756, num_send_msg_fail = 0, num_send_buffer_fail = 0, num_wait_msgq_enqueue = 0, num_wait_msgq_dequeue = 0, num_write_ready_cb_error = 0], [SocketIOStats: bytes = 7748, calls = 5, average_bytes = 1549, blocked_duration = 00:00:00, blocked_count = 0, average_blocked_duration = , errors = 0], [SocketIOStats: bytes = 71025248, calls = 9249, average_bytes = 7679, blocked_duration = 00:00:00, blocked_count = 0, average_blocked_duration = , errors = 0], max_sm_queue_count = 69]

cloud@opencontrail:~$

You can find more details about contrail-logs here.

CONTRAIL ALARMS

References:

https://github.com/Juniper/contrail-controller/wiki/Contrail-Alerts-Overview

http://www.juniper.net/documentation/en_US/contrail3.2/topics/task/configuration/analytics-apis-vnc.html

https://www.juniper.net/documentation/en_US/contrail3.2/topics/task/configuration/underlay-overlay-mapping-vnc.html

http://www.juniper.net/documentation/en_US/contrail3.2/topics/concept/contrail-logs-vnc.html

https://www.juniper.net/documentation/en_US/contrail3.2/topics/task/operational/ha-analytics-vnc.html

https://github.com/Juniper/contrail-controller/wiki/OpenContrail-Internal-Services

http://www.opencontrail.org/opencontrail-analytics-query-api/

results matching ""

    No results matching ""