GovCIO
Transforming Government IT
Senior Systems Architect – Engineer
Systems EngineerSystems EngineerFull TimeRemoteTeam 1,001-5,000Since 2010H1B No SponsorCompany SiteLinkedIn
Location
Virginia
Posted
42 days ago
Salary
$105K - $145K / year
Bachelor Degree10 yrs expEnglishCloudSplunk
Job Description
• Design, implement, and operate the Splunk environment.
• Monitor overall Splunk health through the Monitoring Console (DMC) including indexer, search head, and cluster master status.
• Track indexing rates, license usage, queue health, and search concurrency to identify performance or ingestion issues early.
• Monitor CPU, memory, and disk utilization across all Splunk components to ensure optimal resource usage.
• Respond promptly to health alerts, DMC warnings, or anomalies observed on monitoring dashboards.
• Investigate and resolve common user-reported issues such as access problems, failed searches, or non-triggering alerts.
• Troubleshoot data ingestion, parsing, and indexing issues across Universal Forwarders, Heavy Forwarders, and HEC endpoints.
• Investigate missing or duplicate logs, timestamp errors, or sourcetype misassignments and escalate complex parsing issues to Engineering.
• Validate new data source onboardings by confirming sourcetype assignment, timestamp accuracy, and field extraction integrity.
• Support data source owners with forwarder deployment, syslog setup, and connectivity troubleshooting during initial onboarding.
• Maintain data flow visibility from source → forwarder → indexer to confirm data completeness and performance.
• Rotate and update credentials, API keys, or tokens used in data inputs, integrations, alerts, and scheduled searches.
• Manage RBAC user and role mappings, handling access requests, entitlement reviews, and permission troubleshooting.
• Provide end-user assistance with SPL searches, reports, alerts, and dashboards, including query optimization tips.
• Maintain and update knowledge base articles, SOPs, and FAQs for repeatable issues and troubleshooting steps.
• Log and escalate platform or parsing issues to the Engineering team with evidence such as logs, screenshots, and correlation IDs.
• Open and manage Splunk Support cases for platform-level bugs, license problems, or critical system faults.
• Monitor and manage ITSI service health, including KPIs, correlation searches, NEAP policies, and summary index latency.
• Troubleshoot ITSI-related issues such as broken KPIs, delayed episodes, or missing notable events.
• Perform capacity management by monitoring index growth, bucket rotation, and frozen data retention policies.
• Conduct periodic system maintenance tasks, including orphaned object cleanup and knowledge object review.
• Verify and maintain compliance with data governance and retention policies, ensuring secure and auditable configurations.
• Participate in DR testing and validation to ensure Splunk data recovery and HA configurations are functioning as expected.
• Document incidents, RCA findings, and preventive actions for future reference.
• Collaborate closely with the Engineering team for escalations, root-cause investigations, and deployment verifications.
Job Requirements
- Expert skills in Enterprise Security, ITSI, SOAR, and the Splunk product line.
- Able to design, implement, and operate the Splunk Core, Enterprise Security, IT Service Intelligence (i.e., ITSI), Phantom (Security Orchestration, Automation, and Response (SOAR)), Splunk Cloud, Splunk On-Call, and Multi-Site Index Clustering environment.
- Clearance Required: Must be able to obtain and maintain AOUSC Public Trust.
Benefits
- Employee Assistance Program (EAP)
- Corporate Discounts
- Learning & Development platform, to include certification preparation content
- Training, Education and Certification Assistance*
- Referral Bonus Program
- Internal Mobility Program
- Pet Insurance
- Flexible Work Environment
Related Guides
Related Categories
Related Job Pages
More Systems Engineer Jobs
Senior Pre-Sales Systems Engineer, Commercial
Pure StorageHelping innovators uncomplicate data storage, forever.
Systems Engineer42 days ago
Full TimeRemoteTeam 1,001-5,000Since 2010H1B Sponsor
Senior Pre-Sales Systems Engineer at Pure Storage shaping innovative data storage solutions
AWSAzureCloudGoogle Cloud PlatformKubernetesLAMPLinuxMicroservicesNFSOracleSQLTCP/IPVMware
Senior HR Systems Analyst – Dayforce
Great MindsCreator of Eureka Math, Wit & Wisdom, and PhD Science curricula and Geodes books for emerging readers.
Systems Engineer42 days ago
Full TimeRemoteTeam 1,001-5,000Since 2007H1B No Sponsor
Senior HR Systems Analyst maintaining and enhancing HR systems for Great Minds
Azure
Systems Engineer42 days ago
Full TimeRemoteTeam 1-10H1B No Sponsor
Staff Embedded Systems Engineer evolving modular firmware for connected devices at Inspiren
AWSIoTLinux
Systems Engineer43 days ago
Full TimeRemoteTeam 5,001-10,000Since 2005H1B No Sponsor
Business Systems Analyst joining support team for data troubleshooting and customer support
SQL
Minnesota