Splunk SIEM & Graylog Integration

Project Overview

Goal: Build a comprehensive logging and monitoring solution that automatically collects, processes, and analyzes logs from all infrastructure components for security and operational insights.

┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Servers │───▶│ Splunk │───▶│ Dashboard │
│ Firewalls │ │ SIEM │ │ Alerts │
│ Network │ │ │ │ Reports │
│ Equipment │ └─────────────┘ └─────────────┘
└─────────────┘ │
│
┌─────────────┐ ▼
│ Raspberry │ ┌─────────────┐
│ Pi │───▶│ Graylog │
│ (Graylog) │ │ Processing │
└─────────────┘ └─────────────┘

Phase 1: Splunk SIEM Setup & Configuration

Step 1

Splunk Installation & Initial Setup

Download Splunk Enterprise: Get 60-day free trial from splunk.com
System Requirements: 8GB RAM, 4 cores, 100GB storage minimum
Installation: Deploy on dedicated server or VM
Initial configuration: Set admin password and configure basic settings

# Install Splunk on Linux wget -O splunk.tgz "https://download.splunk.com/..." tar -xzf splunk.tgz -C /opt/ cd /opt/splunk/bin ./splunk start --accept-license ./splunk enable boot-start

Step 2

Configure Universal Forwarders

Deploy lightweight forwarders on all systems to send logs to Splunk:

Install forwarders: Deploy on servers, firewalls, and network devices
Configure outputs: Point all forwarders to main Splunk instance
Set up inputs: Define which logs to collect from each system
Test connectivity: Verify data is flowing to Splunk indexer

# Configure forwarder outputs.conf [tcpout] defaultGroup = splunk_indexers [tcpout:splunk_indexers] server = splunk-server:9997 compressed = true

Step 3

Automated Syslog Ingestion Setup

Configure Syslog Receivers

Enable syslog input: Configure Splunk to receive syslog on port 514
Create index: Set up dedicated indexes for different log types
Source type configuration: Define parsing rules for different systems

# inputs.conf for syslog [udp://514] sourcetype = syslog index = network [tcp://514] sourcetype = syslog index = security

Automated Log Rotation & Retention

Index policies: Configure automatic archiving and deletion
Storage optimization: Set up hot/warm/cold data tiers
Compression: Enable compression for older logs

# indexes.conf [security] maxDataSize = auto_high_volume maxHotBuckets = 10 maxWarmDBCount = 300 frozenTimePeriodInSecs = 2592000

Step 4

Data Sources Integration

Network Infrastructure

Pfsense firewall logs
Switch and router syslogs
DHCP and DNS logs
VPN connection logs

System Logs

Linux system logs (auth, kern, mail)
Windows event logs
Application logs
Database audit logs

# Configure network device syslog forwarding # On Pfsense: Status → System Logs → Settings # Remote Syslog Servers: splunk-server:514 # Remote Syslog Contents: Everything # On Linux systems: echo "*.* @@splunk-server:514" >> /etc/rsyslog.conf systemctl restart rsyslog

Step 5

Security Analytics & Dashboards

Install Splunk ES: Deploy Enterprise Security app for advanced analytics
Create correlation rules: Set up rules to detect suspicious activities
Build dashboards: Create real-time monitoring dashboards
Configure alerts: Set up email/SMS notifications for critical events

# Example search for failed login attempts index=security sourcetype=syslog "authentication failure" | stats count by src_ip, user | where count > 5 | sort -count

Step 6

Automation Scripts & Scheduled Tasks

Automated reports: Schedule daily/weekly security reports
Data cleanup scripts: Automate old log archival and cleanup
Health monitoring: Monitor Splunk infrastructure health
Backup automation: Schedule configuration and index backups

# Python script for automated log analysis import splunklib.client as client service = client.connect( host="localhost", port=8089, username="admin", password="password" ) # Run daily security summary search job = service.jobs.create( 'search index=security | stats count by sourcetype' )

Side Project: Graylog Integration on Raspberry Pi

Why Graylog on Pi? Create a distributed logging solution for redundancy and specialized processing of specific log types.

Pi-1

Raspberry Pi Preparation

Hardware requirements: Pi 4 with 8GB RAM, 64GB+ SD card
OS installation: Install Ubuntu Server 64-bit
Initial setup: Configure SSH, update system, set static IP
Java installation: Install OpenJDK 17 for Graylog compatibility

# Prepare Raspberry Pi sudo apt update && sudo apt upgrade -y sudo apt install openjdk-17-jre-headless sudo systemctl disable bluetooth wifi # Set static IP in /etc/netplan/01-netcfg.yaml

Pi-2

Install Graylog Stack

MongoDB: Install and configure document database
Elasticsearch: Set up search and analytics engine
Graylog: Install main log management platform
Configuration: Configure all services to work together

# Install MongoDB curl -fsSL https://www.mongodb.org/static/pgp/server-5.0.asc | sudo gpg --dearmor -o /usr/share/keyrings/mongodb.gpg sudo apt install mongodb-org # Install Elasticsearch wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.9-amd64.deb sudo dpkg -i elasticsearch-7.17.9-amd64.deb # Install Graylog wget https://packages.graylog2.org/repo/packages/graylog-5.0-repository_latest.deb sudo dpkg -i graylog-5.0-repository_latest.deb sudo apt update && sudo apt install graylog-server

Pi-3

Configure Graylog for Distributed Logging

Input configuration: Set up syslog receivers on Pi
Processing rules: Create log parsing and enrichment rules
Output streams: Forward processed logs to main Splunk instance
Local dashboards: Create Pi-specific monitoring dashboards

# Graylog server.conf key settings is_master = true node_id_file = /etc/graylog/server/node-id password_secret = [generated-secret] root_password_sha2 = [sha2-hash] elasticsearch_hosts = http://localhost:9200 mongodb_uri = mongodb://localhost:27017/graylog

Pi-4

Integration with Main Splunk SIEM

Graylog-to-Splunk forwarding: Configure output to send enriched logs
Specialized processing: Use Pi for IoT/sensor log processing
Backup logging: Use as secondary log collection point
Edge processing: Filter and pre-process logs before main SIEM

# Configure Graylog output to Splunk # In Graylog Web Interface: # System → Outputs → Create Output # Type: Syslog TCP # Target: splunk-server:514 # Protocol: TCP # Format: Syslog

Pi Performance Optimization: Use SD card optimizations, disable swap, and configure log rotation to prevent SD card wear.

Monitoring & Alerting Configuration

Step 7

Real-Time Security Monitoring

Failed authentication alerts: Detect brute force attacks
Network anomaly detection: Unusual traffic patterns
System integrity monitoring: Unauthorized changes
Performance alerts: System resource exhaustion

# Splunk alert search for brute force detection index=security sourcetype=syslog "authentication failure" | bucket _time span=5m | stats count by _time, src_ip | where count > 10 | eval alert_level="HIGH"

Step 8

Automated Response & Reporting

Incident response automation: Automatic IP blocking for attackers
Compliance reporting: Automated PCI/HIPAA compliance reports
Executive dashboards: High-level security metrics
Forensic capabilities: Detailed investigation tools

Advanced Features & Optimization

Machine Learning Integration

Outlier detection: Use Splunk's ML toolkit for anomaly detection
Predictive analytics: Forecast security trends and capacity needs
User behavior analytics: Detect insider threats

Performance Tuning

Index optimization: Tune bucket sizes and retention policies
Search optimization: Create summary indexes for common queries
Hardware scaling: Add indexers and search heads as needed

Maintenance & Best Practices

Security Considerations: Ensure Splunk and Graylog are properly secured with SSL, authentication, and network segmentation.

Regular updates: Keep Splunk, Graylog, and all forwarders updated
Backup strategy: Regular backups of configurations and critical indexes
Capacity planning: Monitor storage usage and plan for growth
Documentation: Maintain detailed runbooks and procedures
Testing: Regular disaster recovery and failover testing

Project Outcomes: Complete visibility into infrastructure security posture, automated threat detection, 99.9% log collection reliability, and 30-second average detection time for security incidents.

End-to-End Observability with Splunk SIEM