Splunk 101
Splunk Basics
Splunk is platform for data collection, data processing, evaluation and analysis
Splunk have great documentation online, here
Splunk have dedicated forum, here; If you dont know... some one else probably knows.
Splunk Orientation
On the top you can see Splunk Bar, there is access to you account, settings, help, etc.
On the left side there is Apps Panel, access to Apps like Search & Report, Enterprise Security (ES), etc.
In the middle of screen there is Splunk Panel with shortcuts to Tour, Add data, Splunk Apps and Docs.
Search & Report
There are six main parts as:
Apps bar (Different in each Splunk App)
Search bar (That is the place for you Query, see Splunk Quick Refference Guide)
Time range picker (Set time range, relative and absolute)
Timeline (See amount of Events in selected Time Range)
Fields sidebar (List of Fields in the Results)
Events viewer (Events that matched submitted Query)
Splunk Search
See Splunk Search Reference and Splunk Search Tutorial
Time
Time is recalculated according to your time zone, specified in user settings. E.g. one event has different time for user in Pacific zone and another time for user in Central Europe zone!
Fields & Values
Field Name: unique name, is case sensitive
Field Value: is NOT case sensitive
Basic Fields
Host: where was the message created or from which server has been the message send
Source: complete path to source of message, etc. /var/log/message
Sourcetype: is name for the same group of messages
Index: where the log messages are stored
Search Modes
Fast Mode
Fastest Search Mode
Shows only host, source and sourcetype Fields
Smart Mode
Default Search Mode
Does not Show All Fields, Splunk determines which field will be shown.
Verbose Mode
The Slowest Mode
Shows The Most Informations
Search Operators
AND
Default operator, it is recommended to use - for better reading
OR
NOT
|
like "pipe" in linux world
output of command is send to input command after |
Order of evaluation is 1. NOT, 2. OR and 3. AND; It is recommended to use round brackets.
Splunk Search Language (SPL)
SPL consists of Search Terms, Commands, Functions, Arguments and Clauses
For better review of query use control/command + \ ; it will move "pipe" to new line, it is more readable view
Top commands
fields, includes or excludes fields from search results
fields <argument1> ... <argumentX>
to exclude fields - <argument1>, it will affect only displayed results
table, retains searched data in a tabulated format
table <argument1> ... <argumentX>
to rearange column order just switch order of arguments
rename, renames fields
rename <field name> as "Desired field name"
dedup, remove events with duplicate values
dedup <field where duplicates will be removed> ... <another field to remove duplicates>
sort, display result in ascending or descending order
sort <field to be sorted> ... <field to be sorted>
by default ordering is ascending; sort + <field to be sorted in ascending order>, to descending order use sort - <field>
to sort some field in ascending and another in descending order; sort -<field in descending> +<field in ascending>
string data are sorted alphanumerically, numeric data are sorted numerically
we can also limit results, by adding limit=<number of events to be displayed>
Transforming commands
Are used to order result of search commands into a data table for statistical purposes
top, finds most common values of given field
top <field>; by default shows TOP 10
for more or less use limit argument; limit=<number of results>; limit=0 gives all results
there are more clauses: otherstr, showperc, showother, showcount, percentfield and countfield
split top values by another field is done with by clause; top <field> by <another field>
rare, shows the least common values of a field set
rare <field>
it has same clauses as top command
stats, to produce statistics
common functions:
count (number of events); count(field) will count only events where specified field is present
distinct count/dc (number of unique values); dc(field)
sum (sum of numerical values); only for fields with numerical value
average/avg (average of numerical values); only for fields with numerical value, missing or misformatted values are not added to calculation
list (list all values of field); list(field)
values (unique values of field); value(field)
it is helpfull to use by clause, to add appropriate fields
Search Performance
For better search performance is better specify index. Also data in data models speed up search process. Also more specific query can give you faster results. Try to use positive way of query instead of using NOT operator, it is faster.
Splunk Roles
Admin
Most powerfull role; Can install Apps, add data, create knowledge objects for all users,...
Power
Can create and share knowledge objects of an App and do realtime searches
User
See only their knowledge objects and those shared with them
Splunk Dashboards & Visualizations
See documentation for Dashboards a Visualizations
There exists Apps for Visualizations on splunkbase
Reports
It can be accessed by Reports Tab.
Save As → Report; Add name, description and specify if Time Picker is displayed
As Power User you can edit access rights of report. Be aware of Run As, if User do not have permission to view data related to report it could be misconfigured report. Reports can be accelerated, results are stored to smaller summary of data.
Vizualizations
Search that returns statistical values can be a chart. Charts can be based on numbers, time and location.
Vizualizations can be saved as Report or Dashboard panel.
Dashboards
It can be accessed by Dashboard Tab. Dashboard is collection of Reports.
Create search → Vizualizations tab, find the right vizualization for your data and customize it → Save As → Dashboard Panel → New/Existing → ( ... ) → Save
Once Time Range is there, it is neccessary to specify Time Range picker which is on our Dashboard. It will work only if it is inline search.
Pivot & Datasets
Knowledge objects that provide the data structure that drives Pivot. Created by Admin or Power roles. Data model is like framework, Pivot is the interface to the data. Each data model is made up of data sets. Datasets are smaller collections of you data defined for specific purpose, represented as tables. Datasets help user get answers faster.
Datasets
Datasets can be accessed from Datasets Tab.
Settings Menu → Settings → Data Models → Pivot on specified Data Model → Provides you interface for creating a Pivot.
If there is no data model, you can use Instant Pivot. Create a non-transforming Search → Statistics/Vizualization → Pivot → Specify from which data data model is created. Data model is created, we are able to create a Pivot.
Splunk has App for Datasets, Splunk Datasets Add-on, helping rapidly build dataset tables without SPL.
Lookups
Is a dataset. It is good for enhancing incomming data with another usefull data like GeoIP, Threat Intell, HTTP Status,...
To create lookup file you have to create lookup table and define lookup. Optionaly you can define run lookup automatically. Lookup field values are case sensitive.
Define Lookup Table
Settings → Lookups → Add new (Lookup Table Files), select App where lookup will be asociated, file to upload and add destination file name.
Define Lookup
Settings → Lookups → Add new (Lookup Definitions), select App where lookup will be asociated, select type and select file.
Define automatic lookups
Settings → Lookups → Add new (Automatic Lookups), select App where lookup will be asociated, choose name of auto lookup, select lookup table, where to apply a lookup (which events), what field is mapped to our lookup values
How to use lookup command
lookup <name of lookup> <field to match> as <field to match>
Splunk Apps
Apps can be installed via UI or manually. Each app can be found on Splunk Base. There are two different types: App and Add-on
Useful Apps (for Security)
If you have Enterprise Security, there is Splunk ES Content Update which extends ES functionality pretty well. Provides you analytical stories, which gives you hints and tips where to look, find,...
Splunk Security Essentials is also usefull App. ES it is not required, it also provides you analytical stories and gives you basic for Security in Splunk.
URL Toolbox gives you ability to work with URL, helps you with parsing,...
Enterprise Security (ES)
It is App which extends the security analysis functionality allowing to centralize security operations and investigation.
Documentation for ES can be found here
ES Content Update
It is Splunk App which extending Correlation rules and also contains some hints and tips, like cookbook. Its called Analytic Stories.
Corellation Search
It is recommended to know how to search, have data models and accelerated data models.
Can be managed in: ES → Content management → Specify type as Correlation Search
Source of Correlation
By default ES have over 60 correlation rules, it can be extended by Splunk App ES Content Update (There are more correlation rules, managed by Splunk and community)
Last updated