A client called me after their Splunk deployment went live.
"Search performance is terrible. We bought the right hardware. What went wrong?"
Nothing was wrong with the hardware. Everything was wrong with where they put their forwarders.
The Decision Nobody Talks About
When teams plan a Splunk deployment, they spend weeks sizing indexers, designing search head clusters, and planning storage tiers. Forwarder placement gets fifteen minutes at the end of the meeting.
That fifteen-minute decision determines how well everything else performs.
Forwarders are not passive pipes. Where you place them relative to your data sources, your network boundaries, and your indexers determines latency, bandwidth consumption, data loss risk, and search performance across your entire environment.
The Three Placement Mistakes I See Constantly
After 20+ years of engineering experience, including 7 years designing and optimizing large-scale Splunk deployments in Federal and DoD environments, these three mistakes appear in nearly every troubled deployment:
1. Centralizing forwarders across a WAN link — Placing a heavy forwarder at headquarters to collect data from remote sites sounds efficient. Until the WAN link degrades and you start losing events. Data in transit across unreliable links needs local buffering. A forwarder at the remote site queues data locally and retransmits on reconnect. A centralized forwarder across a WAN has no such protection.
2. Ignoring bandwidth constraints — Splunk data is compressible but not free. A universal forwarder sending unfiltered Windows event logs across a 10 Mbps link will saturate it during peak hours. Heavy forwarders placed strategically can filter, aggregate, and compress before transmission — reducing bandwidth consumption by 60-80% in environments I've worked on.
3. Single forwarder as a collection point — One heavy forwarder collecting from 50 sources is a single point of failure. When it goes down, 50 sources go dark simultaneously. Distribute collection. If the environment can't support multiple heavy forwarders, at minimum configure universal forwarders at each source to send directly to indexers as a fallback.
The Network Topology Question
Before placing a single forwarder, answer three questions:
Where are the data sources physically located? A forwarder should always be as close to its data source as possible — ideally on the same network segment. Every network hop adds latency and failure risk.
What are the bandwidth constraints between collection points and indexers? Map your bandwidth. If you're sending 5 GB/day across a link with 1 GB/day of headroom, you have a problem waiting to happen.
What is the failure tolerance requirement? For high-availability environments, forwarders need redundant paths to indexers. Configure load-balanced indexer groups so a single indexer failure doesn't stop data collection.
The Topology That Works
In Federal environments where I've had to design for both performance and resilience, the pattern that consistently works is:
Universal forwarders at every endpoint sending to a heavy forwarder at each network boundary. Heavy forwarders filter, aggregate, and forward to indexers over dedicated collection network segments — isolated from user traffic. Indexers receive clean, pre-processed data with no WAN dependency in the collection path.
It sounds like more infrastructure. It's less complexity, fewer failure points, and significantly better performance than the alternatives.
Forwarder placement is an architecture decision, not a deployment detail.
Make it early. Make it deliberately. Your search performance, your data reliability, and your team's sanity at 2 AM depend on it.
If your forwarder deployment falls under STIG compliance requirements, the STIG Compliance App for Splunk tracks your hardening posture across the environment — including forwarder configurations. Upload your CKL files, get a dashboard, track compliance over time. It's free on Splunkbase: splunkbase.splunk.com/app/8486
What forwarder placement challenges have you run into in your environment? Drop them in the comments.
P.S. I'll be at .conf2026 this September. If you're going to be there, I'd love to connect in person — drop me a message.
═══════════════════════════════════════════════════════════════════════════════
───────────────────────────────────────────────────────────────────────────────
───────────────────────────────────────────────────────────────────────────────