Using Remote Elasticsearch for Sensei Reporting

Announcement
Our new documentation site is alive!
You can view our latest documents here.

Sensei provides IT administrators with the option of storing reporting data using either Elasticsearch or a MongoDB database depending on the organization’s firewall hardware resources. Elasticsearch is the leading scalable open-source enterprise search engine designed to operate in real-time in distributed environments. MongoDB is a scalable document database with flexible querying and indexing.

If the firewall has enough memory, 8GB or more RAM, and a modern i3 CPU or later, Sensei will select and install an Elasticsearch instance for its database back end. When the amount of memory is 2 to 4GB and the CPU is somewhat weaker, Sensei will automatically install a local MongoDB database on the target system. Both databases are all installed locally during the Sensei’s initial configuration wizard. 

Starting with the Sensei 1.5 release, IT administrators can also completely offload the reporting database to a remote Elasticsearch instance, either in the cloud or as an on-premise custom Elasticsearch system. The following post will show how to configure Sensei with a remote Elasticsearch instance.

Remote Elasticsearch can be deployed in two ways:

1. Completely offload reporting data to a remote database. This option does not require a local database to be installed with the firewall. 
2. Use a local database and simultaneously stream a copy of the reporting data to a remote server.

The first option lets users install Sensei even on inexpensive hardware devices with RAM at 1GB or less. Follow the steps below to configure options one and two.

Completely Offload reporting to a Remote Elasticsearch Instance

Configuration Steps:

• After Sensei fresh installation or a factory reset, all Sensei menus redirect to wizard. In the first Welcome screen, open Upgrade to Premium link from upper right corner then activate your premium license.
• Next then Hardware Check
• Select Use a Remote Elasticsearch Database
• Enter the Database URI information: (URI example – http://elasticsearch_server_ip:9200 or https://elasticsearch_server_ip:9200).
• Enter the username and password.
• To configure Elasticsearch with a username and password see:
  • Configuring SSL, TLS, and HTTPS to secure Elasticsearch, Kibana, Beats, and Logstash
• To check connections and create indexes in the remote Elasticsearch instances click on “Install Database & Proceed.”
• The Wizard will advance if everything is correct.
• Sensei will store the Report Data in the remote Elasticsearch instance with the configuration.
• No Report Data will be stored locally, all data will be stored in the remote Elasticsearch database.
• Note: Database URI still could be used even if Elasticsearch was configured without the username and password

Stream Reporting Data to a Remote Elasticsearch Instance

Configuration Steps:

The following option requires SOHO or higher Sensei paid subscriptions.

• Go to: Configuration >  Reporting & Data > Stream Reporting Data to External Elasticsearch”
• Activate Enabled.
• The Database URI information: (URI example – http://elasticsearch_server_ip:9200 or https://elasticsearch_server_ip:9200).
• To check the connection and create indexes in the remote Elasticsearch instance click on “Check External Database & Create Indexes”
• The Wizard will advance if everything is correct.
• Sensei will store the reporting data for both Local and Remote Databases with this configuration.
• Note: Database URI still could be filled up even if Elasticsearch was configured without the username and password.

Configuring Kibana to Visualize Sensei Reporting Data

• Prefix for index names comes with the Sensei Premium licenses.
• To reach the Prefix > Sensei GUI > Configuration > About > Host Unique Identifier.

To use Sensei Reporting data in Kibana, Sensei’s prefix must be added to the Kibana index pattern: Open Kibana > Settings > Index Pattern

• Click on Create Index Pattern and 
• Paste Host Unique Identifier (To reach The prefix > Sensei GUI > Configuration > About > Host Unique Identifier).

• When you paste Host Unique Identifier, you will see an index list. Sensei creates 6 different indexes. They are:
1. “[Prefix]_conn_date”; For all TCP and UDP connections
2. “[Prefix]_sip_date”; For all SIP connections
3. “[Prefix]_dns_date”; For all DNS connections
4. “[Prefix]_http_date”; For all Http connections
5. “[Prefix]_tls_date”; For all https connections
6. “[Prefix]_alert_date”; For all blocked connectionsAdd _[conn,sip,dns,http,tls,alert]* one of them end of the prefix to report continuously. Then click Next step.

• Add _[conn,sip,dns,http,tls,alert]* one of them end of the prefix to report continuously. Then click Next step.

  * If you write or select a filename that you see in the list, it will report only related date.

• Select Start_Time > then click on Create Index Pattern.

• To create Report Graphics,  Open Kibana > Visualizations

• Click on Create New Visualization and select any chart.

• Select Index File.

• To complete configuration, Select Terms from Aggregation select box and appropriate Field Name from the field Select box in the right Bucket section.

• To update the chart, click on the Update button.

• If you add more than one machine Reports, you can save reports with related machine names. They will also be added to the Visualize Dashboard.