Splunk 101

Splunk Basics

Splunk is platform for data collection, data processing, evaluation and analysis

Splunk have great documentation online, here​

Splunk have dedicated forum, here; If you dont know... some one else probably knows.

Splunk Orientation

On the top you can see Splunk Bar, there is access to you account, settings, help, etc.

On the left side there is Apps Panel, access to Apps like Search & Report, Enterprise Security (ES), etc.

In the middle of screen there is Splunk Panel with shortcuts to Tour, Add data, Splunk Apps and Docs.

Search & Report

There are six main parts as:

  1. Apps bar (Different in each Splunk App)

  2. Search bar (That is the place for you Query, see Splunk Quick Refference Guide)

  3. Time range picker (Set time range, relative and absolute)

  4. Timeline (See amount of Events in selected Time Range)

  5. Fields sidebar (List of Fields in the Results)

  6. Events viewer (Events that matched submitted Query)

Splunk Search

See Splunk Search Reference and Splunk Search Tutorial​

Time

Time is recalculated according to your time zone, specified in user settings. E.g. one event has different time for user in Pacific zone and another time for user in Central Europe zone!

Fields & Values

Field Name: unique name, is case sensitive

Field Value: is NOT case sensitive

Basic Fields

  • Host: where was the message created or from which server has been the message send

  • Source: complete path to source of message, etc. /var/log/message

  • Sourcetype: is name for the same group of messages

  • Index: where the log messages are stored

Search Modes

  • Fast Mode

    • Fastest Search Mode

    • Shows only host, source and sourcetype Fields

  • Smart Mode

    • Default Search Mode

    • Does not Show All Fields, Splunk determines which field will be shown.

  • Verbose Mode

    • The Slowest Mode

    • Shows The Most Informations

Search Operators

  • AND

    • Default operator, it is recommended to use - for better reading

  • OR

  • NOT

  • |

    • like "pipe" in linux world

    • output of command is send to input command after |

Order of evaluation is 1. NOT, 2. OR and 3. AND; It is recommended to use round brackets.

Splunk Search Language (SPL)

SPL consists of Search Terms, Commands, Functions, Arguments and Clauses

For better review of query use control/command + \ ; it will move "pipe" to new line, it is more readable view

Top commands

  • fields, includes or excludes fields from search results

    • fields <argument1> ... <argumentX>

    • to exclude fields - <argument1>, it will affect only displayed results

  • table, retains searched data in a tabulated format

    • table <argument1> ... <argumentX>

    • to rearange column order just switch order of arguments

  • rename, renames fields

    • rename <field name> as "Desired field name"

  • dedup, remove events with duplicate values

    • dedup <field where duplicates will be removed> ... <another field to remove duplicates>

  • sort, display result in ascending or descending order

    • sort <field to be sorted> ... <field to be sorted>

    • by default ordering is ascending; sort + <field to be sorted in ascending order>, to descending order use sort - <field>

    • to sort some field in ascending and another in descending order; sort -<field in descending> +<field in ascending>

    • string data are sorted alphanumerically, numeric data are sorted numerically

    • we can also limit results, by adding limit=<number of events to be displayed>

Transforming commands

Are used to order result of search commands into a data table for statistical purposes

  • top, finds most common values of given field

    • top <field>; by default shows TOP 10

    • for more or less use limit argument; limit=<number of results>; limit=0 gives all results

    • there are more clauses: otherstr, showperc, showother, showcount, percentfield and countfield

    • split top values by another field is done with by clause; top <field> by <another field>

  • rare, shows the least common values of a field set

    • rare <field>

    • it has same clauses as top command

  • stats, to produce statistics

    • common functions:

      • count (number of events); count(field) will count only events where specified field is present

      • distinct count/dc (number of unique values); dc(field)

      • sum (sum of numerical values); only for fields with numerical value

      • average/avg (average of numerical values); only for fields with numerical value, missing or misformatted values are not added to calculation

      • list (list all values of field); list(field)

      • values (unique values of field); value(field)

    • it is helpfull to use by clause, to add appropriate fields

Search Performance

For better search performance is better specify index. Also data in data models speed up search process. Also more specific query can give you faster results. Try to use positive way of query instead of using NOT operator, it is faster.

Splunk Roles

  • Admin

    • Most powerfull role; Can install Apps, add data, create knowledge objects for all users,...

  • Power

    • Can create and share knowledge objects of an App and do realtime searches

  • User

    • See only their knowledge objects and those shared with them

Splunk Dashboards & Visualizations

See documentation for Dashboards a Visualizations

There exists Apps for Visualizations on splunkbase​

Reports

It can be accessed by Reports Tab.

Save As → Report; Add name, description and specify if Time Picker is displayed

As Power User you can edit access rights of report. Be aware of Run As, if User do not have permission to view data related to report it could be misconfigured report. Reports can be accelerated, results are stored to smaller summary of data.

Vizualizations

Search that returns statistical values can be a chart. Charts can be based on numbers, time and location.

Vizualizations can be saved as Report or Dashboard panel.

Dashboards

It can be accessed by Dashboard Tab. Dashboard is collection of Reports.

Create search → Vizualizations tab, find the right vizualization for your data and customize it → Save As → Dashboard Panel → New/Existing → ( ... ) → Save

Once Time Range is there, it is neccessary to specify Time Range picker which is on our Dashboard. It will work only if it is inline search.

Pivot & Datasets

Knowledge objects that provide the data structure that drives Pivot. Created by Admin or Power roles. Data model is like framework, Pivot is the interface to the data. Each data model is made up of data sets. Datasets are smaller collections of you data defined for specific purpose, represented as tables. Datasets help user get answers faster.

Datasets

Datasets can be accessed from Datasets Tab.

Settings Menu → Settings → Data Models → Pivot on specified Data Model → Provides you interface for creating a Pivot.

If there is no data model, you can use Instant Pivot. Create a non-transforming Search → Statistics/Vizualization → Pivot → Specify from which data data model is created. Data model is created, we are able to create a Pivot.

Splunk has App for Datasets, Splunk Datasets Add-on, helping rapidly build dataset tables without SPL.

Lookups

Is a dataset. It is good for enhancing incomming data with another usefull data like GeoIP, Threat Intell, HTTP Status,...

To create lookup file you have to create lookup table and define lookup. Optionaly you can define run lookup automatically. Lookup field values are case sensitive.

Define Lookup Table

Settings → Lookups → Add new (Lookup Table Files), select App where lookup will be asociated, file to upload and add destination file name.

Define Lookup

Settings → Lookups → Add new (Lookup Definitions), select App where lookup will be asociated, select type and select file.

Define automatic lookups

Settings → Lookups → Add new (Automatic Lookups), select App where lookup will be asociated, choose name of auto lookup, select lookup table, where to apply a lookup (which events), what field is mapped to our lookup values

How to use lookup command

lookup <name of lookup> <field to match> as <field to match>

Splunk Apps

Apps can be installed via UI or manually. Each app can be found on Splunk Base. There are two different types: App and Add-on

Useful Apps (for Security)

If you have Enterprise Security, there is Splunk ES Content Update which extends ES functionality pretty well. Provides you analytical stories, which gives you hints and tips where to look, find,...

​Splunk Security Essentials is also usefull App. ES it is not required, it also provides you analytical stories and gives you basic for Security in Splunk.

​URL Toolbox gives you ability to work with URL, helps you with parsing,...

Enterprise Security (ES)

It is App which extends the security analysis functionality allowing to centralize security operations and investigation.

Documentation for ES can be found here​

ES Content Update

It is Splunk App which extending Correlation rules and also contains some hints and tips, like cookbook. Its called Analytic Stories.

Corellation Search

It is recommended to know how to search, have data models and accelerated data models.

Can be managed in: ES → Content management → Specify type as Correlation Search

Source of Correlation

By default ES have over 60 correlation rules, it can be extended by Splunk App ES Content Update (There are more correlation rules, managed by Splunk and community)

PreviousIntroNextSplunk Power User Bootcamp

Last updated