Chapter 7: Contrail Analytics :
The Analytics nodes collects the system state information such as usage statistics, debug information from all of the software modules across all of the nodes of the system(config,control,vrouter, etc).
Analytics stores the data in Cassandra DB, This data can be queries by REST APIs.
Contrail Analytics consists of the below services.
cloud@opencontrail:~/manage-scripts$ ./contrail-analytics.sh status
contrail-collector RUNNING pid 29716, uptime 10:45:11
contrail-query-engine RUNNING pid 29740, uptime 10:45:06
contrail-analytics-api RUNNING pid 29773, uptime 10:45:01
contrail-snmp-collector RUNNING pid 29788, uptime 10:44:55
contrail-topology RUNNING pid 29801, uptime 10:44:50
contrail-alarm-gen RUNNING pid 29836, uptime 10:44:44
cloud@opencontrail:~/manage-scripts$
Services:
Contrail-topology, Contrail-snmp-collector :
Contrail-topology, Contrail-snmp-collector used for Underlay Overlay Analytics.
Contrail-snmp-collector polls the underlay devices(physical router) and collects the information. Contrail-topology uses these information and prepares the topology with overlay and underlay information. More details are here
The prouter uve can be get it from below api.
http://10.0.1.4:8081/analytics/uves/prouters
Note: So we dont have to worry much about these two services . Mostly this will be useful for Production environment.
Contrail-Collector, Contrail-query-engine, Contrail-analytics-api :
contrail-collector receives status, logs, and flow information from all modules (for example, generators) and records them in cassandra.
contrail-analytics-api(contrail-opserver) provides an external REST API to report UVEs and to query logs and flows. The default port (8081). Some more details are here.
Contrail-query-engine interface with cassandra server, and retrieves the data based on the query (given by contrail-analytics-api).
The Flow is below,
Contrail-query-engine listens on 8091
contrail-analytics-api receives the REST API query from the User, and send this query to the Contrail-query-engine on port 8091.
Contrail-Query-engine interacts with cassandra and runs the query and retrieves the data. Place this data to the REDIS server.
Contrail-analytics-api reads the data from REDIS and respond back to the user.
Contrail-alarm-gen:
To be updated
API Services :
In my setup, contrail analytics runs in 10.0.1.6.
Understanding the Contrail Analytics API
Contrail Analytics exposes the REST Interface in port 8081.
GET http://10.0.1.6:8081
{
"href": "http://10.0.1.6:8081",
"links": [
{
"link": {
"href": "http://10.0.1.6:8081/documentation/index.html",
"name": "documentation"
}
},
{
"link": {
"href": "http://10.0.1.6:8081/documentation/messages/index.html",
"name": "Message documentation"
}
},
{
"link": {
"href": "http://10.0.1.6:8081/analytics",
"name": "analytics"
}
}
]
}
Analytics API
/analytics endpoint is used for querying the analytics node.
GET http://10.0.1.6:8081/analytics
[ { "href": "http://10.0.1.6:8081/analytics/uves", "name": "uves" },
{ "href": "http://10.0.1.6:8081/analytics/uve-types", "name": "uve-types" },
{ "href": "http://10.0.1.6:8081/analytics/tables", "name": "tables" },
{ "href": "http://10.0.1.6:8081/analytics/queries", "name": "queries" },
{ "href": "http://10.0.1.6:8081/analytics/alarms", "name": "alarms" },
{ "href": "http://10.0.1.6:8081/analytics/uve-stream", "name": "uve-stream" },
{ "href": "http://10.0.1.6:8081/analytics/alarm-stream", "name": "alarm-stream" }
]
The Analytics API functions are broadly classified as "uves", "query the logs" and alarms. uve-stream, alarm-stream are stream APIs for uve and alarm functions.
UVEs
In simple terms, User Visible Entities are Contrail resources (virtual-network, virtual-machines, vrouter, routing-instances, etc). UVE APIs are used to query these resources.
GET http://10.0.1.6:8081/analytics/uves
[
{
"href": "http://10.0.1.6:8081/analytics/uves/analytics-nodes",
"name": "analytics-nodes"
},
{
"href": "http://10.0.1.6:8081/analytics/uves/config-nodes",
"name": "config-nodes"
},
{
"href": "http://10.0.1.6:8081/analytics/uves/virtual-machines",
"name": "virtual-machines"
},
{
"href": "http://10.0.1.6:8081/analytics/uves/control-nodes",
"name": "control-nodes"
},
{
"href": "http://10.0.1.6:8081/analytics/uves/virtual-networks",
"name": "virtual-networks"
},
................................(strippped )
]
Let us query the virtual-machines
GET http://10.0.1.6:8081/analytics/uves/virtual-machines
[
{
"href": "http://10.0.1.6:8081/analytics/uves/virtual-machine/f0240c92-12ca-4774-8b16-f42540dc9a1d?flat",
"name": "f0240c92-12ca-4774-8b16-f42540dc9a1d"
},
{
"href": "http://10.0.1.6:8081/analytics/uves/virtual-machine/2783564a-1d75-4302-b0fe-dd32b10ce2f5?flat",
"name": "2783564a-1d75-4302-b0fe-dd32b10ce2f5"
},
{
"href": "http://10.0.1.6:8081/analytics/uves/virtual-machine/2e6502c2-8e63-4081-a352-efcdaf56f905?flat",
"name": "2e6502c2-8e63-4081-a352-efcdaf56f905"
}
]
We have 3 Virtual-Machines running.Let us query 1 VM with ID, to get the full details of the VM.
GET http://10.0.1.6:8081/analytics/uves/virtual-machine/f0240c92-12ca-4774-8b16-f42540dc9a1d?flat
{
"VirtualMachineStats": {
"cpu_stats": [
{
"virt_memory": 1609064,
"cpu_one_min_avg": 0.166667,
"disk_used_bytes": 10620928,
"vm_memory_quota": 131072,
"peak_virt_memory": 2101852,
"disk_allocated_bytes": 25165824,
"rss": 225488
}
]
},
"UveVirtualMachineAgent": {
"udp_sport_bitmap": [
"0",
"0",
"0",
"0",
"0",
"2",
"2097152",
"0"
],
"vm_name": "instance2",
"tcp_sport_bitmap": [
"1",
"0",
"0",
"0",
"0",
"0",
"32768",
"0"
],
"uuid": "f0240c92-12ca-4774-8b16-f42540dc9a1d",
"udp_dport_bitmap": [
"1",
"0",
"0",
"0",
"0",
"0",
"0",
"0"
],
"cpu_info": {
"virt_memory": 1609064,
"cpu_one_min_avg": 0.166667,
"disk_used_bytes": 10620928,
"vm_memory_quota": 131072,
"peak_virt_memory": 2101852,
"disk_allocated_bytes": 25165824,
"rss": 225488
},
"vrouter": "devstack1",
"tcp_dport_bitmap": [
"1",
"0",
"0",
"0",
"0",
"0",
"32768",
"0"
],
"interface_list": [
"default-domain:demo:2f877af3-b958-4444-84ad-f37c76d0338c"
]
}
}
Query APIs (LOGs and Flows) :
Contrail Analytics stores all the logs/flows information of all nodes in to Cassandra DB. we can POST the proper query and retrive the information.
This below API response, the status of queries (errors, abanodend, processing...)
GET http://10.0.1.6:8081/analytics/queries
To know the list of tables contrail analytics have. We can perform the query for these tables.
GET http://10.0.1.6:8081/analytics/tables
[
{
"href": "http://10.0.1.6:8081/analytics/table/MessageTable",
"type": "LOG",
"name": "MessageTable"
},
{
"href": "http://10.0.1.6:8081/analytics/table/FlowRecordTable",
"type": "FLOW",
"name": "FlowRecordTable"
},
{
"href": "http://10.0.1.6:8081/analytics/table/FlowSeriesTable",
"type": "FLOW",
"name": "FlowSeriesTable"
},
{
"href": "http://10.0.1.6:8081/analytics/table/OverlayToUnderlayFlowMap",
"type": "FLOW",
"name": "OverlayToUnderlayFlowMap"
},
{
"display_name": "Service Chain",
"href": "http://10.0.1.6:8081/analytics/table/ServiceChain",
"type": "OBJECT",
"name": "ServiceChain"
},
{
"display_name": "Database Node",
"href": "http://10.0.1.6:8081/analytics/table/ObjectDatabaseInfo",
"type": "OBJECT",
"name": "ObjectDatabaseInfo"
}
stripped...
]
To know the schema/other info of these tables, just GET API for the tables.
Query API
We have to construct the query(like SQL query) and POST it to the Analytics on.
Ex:
In table name "MessageTable",
- We want to display fields of ["MessageTS", "Source", "ModuleId", "Category", "Messagetype", "SequenceNum", "Xmlmessage", "Type", "Level", "NodeType", "InstanceId"],
- Where the condition is "name": "ModuleId", "value": "contrail-control",
- Starting time
- End Time
POST http://10.0.1.6:8081/analytics/query
Body:
{
"sort": 1,
"start_time": "now-10m",
"sort_fields": ["MessageTS"],
"end_time": "now",
"select_fields": ["MessageTS", "Source", "ModuleId", "Category", "Messagetype", "SequenceNum", "Xmlmessage", "Type", "Level", "NodeType", "InstanceId"],
"table": "MessageTable",
"where": [[{"suffix": null, "value2": null, "name": "ModuleId", "value": "contrail-control", "op": 1}]]
}
Response:
{
"value": [
{
"Category": "IFMapPeer",
"InstanceId": "0",
"Level": 7,
"MessageTS": 1494932372766850,
"Messagetype": "IFMapDSResp",
"ModuleId": "contrail-control",
"NodeType": "Control",
"SequenceNum": 14208,
"Source": "opencontrail",
"Type": 1,
"Xmlmessage": "<IFMapDSResp type=\"sandesh\"><current_peer type=\"string\" identifier=\"1\">current_peer is 10.0.1.4:8443:in_use</current_peer><new_response type=\"string\" identifier=\"2\">DSResp is 10.0.1.4:8443, </new_response><file type=\"string\" identifier=\"-32768\">controller/src/ifmap/client/peer_server_finder.cc</file><line type=\"i32\" identifier=\"-32767\">132</line></IFMapDSResp>"
},
{
"Category": null,
"InstanceId": "0",
"Level": 2147483647,
"MessageTS": 1494932375338803,
"Messagetype": "SandeshModuleClientTrace",
"ModuleId": "contrail-control",
"NodeType": "Control",
"SequenceNum": 30327,
"Source": "opencontrail",
"Type": 6,
"Xmlmessage": "<SandeshModuleClientTrace type=\"sandesh\"><data type=\"struct\" identifier=\"1\"><ModuleClientState><name type=\"string\" identifier=\"1\" key=\"ObjectGeneratorInfo\">opencontrail:Control:contrail-control:0</name><sm_queue_count type=\"u64\" identifier=\"5\">0</sm_queue_count><sm_stats type=\"struct\" identifier=\"6\"><SandeshStateMachineStats><ev_stats type=\"list\" identifier=\"1\"><"
}
]
}
stripped
you can get all the logs, table details using the Query API.
Contrail-Logs Command :
Contrail-logs command is CLI for querying the analyzer and get the logs. Utilizing the query API , and made user friendly switches.
- contrail-logs --module contrail-control --start-time now-1m --end-time now
cloud@opencontrail:~$ contrail-logs --verbose --module contrail-control --start-time now-1m --end-time now
Performing query: {"sort": 1, "start_time": 1494933514431957, "sort_fields": ["MessageTS"], "end_time": 1494933574431957, "select_fields": ["MessageTS", "Source", "ModuleId", "Category", "Messagetype", "SequenceNum", "Xmlmessage", "Type", "Level", "NodeType", "InstanceId"], "table": "MessageTable", "where": [[{"suffix": null, "value2": null, "name": "ModuleId", "value": "contrail-control", "op": 1}]]}
2017 May 16 11:18:35.496732 opencontrail [Control:contrail-control:0:None][INVALID] : SandeshModuleClientTrace:30477 [ModuleClientState: name = opencontrail:Control:contrail-control:0, sm_queue_count = 0, [SandeshStateMachineStats: [ev_stats: [event = scm::EvDiscUpdate, enqueues = 14315, dequeues = 14315, enqueue_fails = 0, dequeue_fails = 0] [event = scm::EvIdleHoldTimerExpired, enqueues = 74, dequeues = 74, enqueue_fails = 0, dequeue_fails = 0] [event = scm::EvSandeshMessageRecv, enqueues = 20, dequeues = 20, enqueue_fails = 0, dequeue_fails = 0] [event = scm::EvSandeshSend, enqueues = 73442, dequeues = 73442, enqueue_fails = 0, dequeue_fails = 0] [event = scm::EvStart, enqueues = 1, dequeues = 1, enqueue_fails = 0, dequeue_fails = 0] [event = scm::EvTcpClose, enqueues = 13, dequeues = 13, enqueue_fails = 0, dequeue_fails = 0] [event = scm::EvTcpConnectFail, enqueues = 61, dequeues = 61, enqueue_fails = 0, dequeue_fails = 0] [event = scm::EvTcpConnected, enqueues = 14, dequeues = 14, enqueue_fails = 0, dequeue_fails = 0] [event = scm::EvTcpDeleteSession, enqueues = 74, dequeues = 74, enqueue_fails = 0, dequeue_fails = 0] [event = , enqueues = 88014, dequeues = 88014, enqueue_fails = 0, dequeue_fails = 0]], state = Established, state_since = 1494881745602577, last_state = Established, last_event = scm::EvSandeshSend, last_event_at = 1494933511345984], [SandeshSessionStats: num_recv_msg = 0, num_recv_msg_fail = 0, num_recv_fail = 0, num_send_msg = 15756, num_send_msg_fail = 0, num_send_buffer_fail = 0, num_wait_msgq_enqueue = 0, num_wait_msgq_dequeue = 0, num_write_ready_cb_error = 0], [SocketIOStats: bytes = 7748, calls = 5, average_bytes = 1549, blocked_duration = 00:00:00, blocked_count = 0, average_blocked_duration = , errors = 0], [SocketIOStats: bytes = 71025248, calls = 9249, average_bytes = 7679, blocked_duration = 00:00:00, blocked_count = 0, average_blocked_duration = , errors = 0], max_sm_queue_count = 69]
cloud@opencontrail:~$
You can find more details about contrail-logs here.
CONTRAIL ALARMS
References:
https://github.com/Juniper/contrail-controller/wiki/Contrail-Alerts-Overview
http://www.juniper.net/documentation/en_US/contrail3.2/topics/concept/contrail-logs-vnc.html
https://github.com/Juniper/contrail-controller/wiki/OpenContrail-Internal-Services
http://www.opencontrail.org/opencontrail-analytics-query-api/