GovCIO

Transforming Government IT

Senior Systems Architect – Engineer

Systems EngineerSystems EngineerFull TimeRemoteTeam 1,001-5,000Since 2010H1B No SponsorCompany SiteLinkedIn

Location

Virginia

Posted

42 days ago

Salary

$105K - $145K / year

Bachelor Degree10 yrs expEnglishCloudSplunk

Job Description

• Design, implement, and operate the Splunk environment. • Monitor overall Splunk health through the Monitoring Console (DMC) including indexer, search head, and cluster master status. • Track indexing rates, license usage, queue health, and search concurrency to identify performance or ingestion issues early. • Monitor CPU, memory, and disk utilization across all Splunk components to ensure optimal resource usage. • Respond promptly to health alerts, DMC warnings, or anomalies observed on monitoring dashboards. • Investigate and resolve common user-reported issues such as access problems, failed searches, or non-triggering alerts. • Troubleshoot data ingestion, parsing, and indexing issues across Universal Forwarders, Heavy Forwarders, and HEC endpoints. • Investigate missing or duplicate logs, timestamp errors, or sourcetype misassignments and escalate complex parsing issues to Engineering. • Validate new data source onboardings by confirming sourcetype assignment, timestamp accuracy, and field extraction integrity. • Support data source owners with forwarder deployment, syslog setup, and connectivity troubleshooting during initial onboarding. • Maintain data flow visibility from source → forwarder → indexer to confirm data completeness and performance. • Rotate and update credentials, API keys, or tokens used in data inputs, integrations, alerts, and scheduled searches. • Manage RBAC user and role mappings, handling access requests, entitlement reviews, and permission troubleshooting. • Provide end-user assistance with SPL searches, reports, alerts, and dashboards, including query optimization tips. • Maintain and update knowledge base articles, SOPs, and FAQs for repeatable issues and troubleshooting steps. • Log and escalate platform or parsing issues to the Engineering team with evidence such as logs, screenshots, and correlation IDs. • Open and manage Splunk Support cases for platform-level bugs, license problems, or critical system faults. • Monitor and manage ITSI service health, including KPIs, correlation searches, NEAP policies, and summary index latency. • Troubleshoot ITSI-related issues such as broken KPIs, delayed episodes, or missing notable events. • Perform capacity management by monitoring index growth, bucket rotation, and frozen data retention policies. • Conduct periodic system maintenance tasks, including orphaned object cleanup and knowledge object review. • Verify and maintain compliance with data governance and retention policies, ensuring secure and auditable configurations. • Participate in DR testing and validation to ensure Splunk data recovery and HA configurations are functioning as expected. • Document incidents, RCA findings, and preventive actions for future reference. • Collaborate closely with the Engineering team for escalations, root-cause investigations, and deployment verifications.

Job Requirements

  • Expert skills in Enterprise Security, ITSI, SOAR, and the Splunk product line.
  • Able to design, implement, and operate the Splunk Core, Enterprise Security, IT Service Intelligence (i.e., ITSI), Phantom (Security Orchestration, Automation, and Response (SOAR)), Splunk Cloud, Splunk On-Call, and Multi-Site Index Clustering environment.
  • Clearance Required: Must be able to obtain and maintain AOUSC Public Trust.

Benefits

  • Employee Assistance Program (EAP)
  • Corporate Discounts
  • Learning & Development platform, to include certification preparation content
  • Training, Education and Certification Assistance*
  • Referral Bonus Program
  • Internal Mobility Program
  • Pet Insurance
  • Flexible Work Environment

Related Categories

Related Job Pages

More Systems Engineer Jobs

Senior Pre-Sales Systems Engineer, Commercial

Pure Storage

Helping innovators uncomplicate data storage, forever.

Systems Engineer42 days ago
Full TimeRemoteTeam 1,001-5,000Since 2010H1B Sponsor

Senior Pre-Sales Systems Engineer at Pure Storage shaping innovative data storage solutions

AWSAzureCloudGoogle Cloud PlatformKubernetesLAMPLinuxMicroservicesNFSOracleSQLTCP/IPVMware
Maryland + 2 moreAll locations: Maryland, Virginia, Washington
$137.2K - $206.5K / year

Senior HR Systems Analyst – Dayforce

Great Minds

Creator of Eureka Math, Wit & Wisdom, and PhD Science curricula and Geodes books for emerging readers.

Systems Engineer42 days ago
Full TimeRemoteTeam 1,001-5,000Since 2007H1B No Sponsor

Senior HR Systems Analyst maintaining and enhancing HR systems for Great Minds

Azure
District of Columbia
$88K - $98K / year
Systems Engineer42 days ago
Full TimeRemoteTeam 1-10H1B No Sponsor

Staff Embedded Systems Engineer evolving modular firmware for connected devices at Inspiren

AWSIoTLinux
New York
$200K - $225K / year

Business Systems Analyst

Solera, Inc.

The global leader in vehicle lifecycle management.

Systems Engineer43 days ago
Full TimeRemoteTeam 5,001-10,000Since 2005H1B No Sponsor

Business Systems Analyst joining support team for data troubleshooting and customer support

SQL
Minnesota