Splunk for Enterprise Security
Splunk March - August 2015
The Summary:
After a successful initial product entry in the Security Information and Event Management (SIEM) product space, there was a strong company commitment to design innovations within this enterprise product area. Many current customers were adding Enterprise Security to their existing implementations, underscoring how well the core Splunk application technology had positioned them for SIEM market leadership.
The User Experience team was called upon to structure some research activities for the product team and produce insights to support their ongoing design and feature prioritization.
They also needed us to set up some creative activities to inspire new features and functionality not yet in their backlog.
This project brought together a multidisciplinary team to conduct field research with key customers, generate personas, complete an enterprise product innovation workshop, and bring new feature concepts out to local customers for validation.
The Issues:
In my experience, it is often the case that software product teams have little to no familiarity with research methods beyond usability testing. Teams at Splunk were no exception. Therefore we needed to convince product leadership and engineering of the benefits of the research activities planned, despite the time commitment. Many key customers were based in Europe and those in the US willing to host a site visit also required air travel. The ROI of formative research is always easier to communicate when there is a proof-of-concept (POC) project within the company to which you can point. But, this *was* the POC project to which we would point other product teams, so the additional drudgery securing commitments and endorsements would only burden us once.
The internal innovation workshop we organized, called a 'Splunk Dunk', was another significant time commitment for stakeholders. The tactical daily work of Agile sprints, meetings, and looming deadlines kept their schedules packed, which left little room to accommodate a full-week block. Product leadership engagement is critical for strong design innovation outcomes, therefore we challenged the importance of their existing commitments, on a case-by-case basis, in order to create the space we needed in the calendar.
Engaging local (Bay Area) customers for concept feedback was also an administrative time sink, requiring the support of Account Executives with strong customer relationships in some cases in order to establish contact.
Methods Applied:
Contextual Inquiry
Product Innovation Workshop
Concept Reviews
Primary Role:
Led all site visit planning, logistics, protocol development, and data collection.
Authored personas.
Collaborators:
-
VP of UX
-
Product Director
-
Design Manager
-
UI Designer
-
Engineer
Outcomes:
-
Personas for Enterprise Security team
-
Successful 'Splunk Dunk' innovation workshop -- field data and use cases as workshop content, workshop process supported issue prioritization and feature brainstorming, non-UX leadership exposed to design-led product planning
-
Confidence in product concept feedback from groups to guide design and development
-
Strong placement on the Gartner Quadrant for security applications
Research Details
Contextual Inquiry - Innovation Workshop - Concept Reviews
We maximized the value of all the customer site visits by using them as research training opportunities as well. Each was staffed with an interaction designer, a developer, and a project manager or other stakeholder. All of the administrative logistics were covered and each assistant was given ownership over an aspect of the research sessions. (e.g. equipment charging, setup, and session recording; note-taking; clarification of new technical terminology in the discussion; legal paperwork signatures and incentives; session moderation).
We met with Security teams across 6 customer sites, 3 of these dedicated to security topics with key Enterprise Security customers, complete with tours of active Security Operations Centers (SOCs). As a team, we prepared daily 'executive highlight' summaries so anyone not with us could get a sneak peek at what we were learning. We also collaborated on a deeper synthesis process after the site visits were completed, from which we generated persona artifacts and a set of summarized insights.
1 Contextual Inquiry
Customer site visit interviews conducted with several team members and a collaboration on data synthesis.
Interviewing a Security Analyst
[cannot expose photos of customer SOCs]
[Any detailed insights from site visits and the 'Splunk Dunk' workshop are protected IP, unless related to features currently in general release]
Here are a couple of site visit takeaways that inspired new security product features:
-
Investigations are messy process with many threads of thought happening concurrently. Evidence supporting just how unsupported the Analysts were included dozens and dozens of browser tabs open at the same time. They needed access to different parts of the application UI at the same time to make it easier to think through their research as interesting clues emerge.
-
Analysts are also responsible for assembling investigation reports so that others can audit their logic and resulting conclusions. An investigation can almost never summarized using a bulk capture of their investigation process. So, Analysts must capture screenshots, tracking and labeling them as they go, in order to make their reports. We need better support in the UI to reduce this burden on Analysts, who need to focus on the research
2 Product Innovation Workshop
Conduct a 'Splunk Dunk' with executive sponsors, product leadership, and UX leadership over 4½
days to create design-led product ideas that address user pain points.
A specific 'diverge/converge' pattern is followed for these workshops. Given the volume of content that is both presented and generated, along with the dynamics of group discussion, the activities are necessarily spread out across 4½ days.
Day 1: Present market intro, product walkthrough, research insights and personas, usage data, feature requests, current backlog.
Co-create 3-layer pyramid with bottom layer of user pain points, middle layer of related product areas, top layer of business strategy drivers.
Day 2: Prioritize areas of focus for the design work. Create user scenarios/user stories/use cases around them. Each team brainstorms a high volume of solution ideas from which they will choose the top 1 or 2. Bring together the relevant artifacts generated so far to pull together the story; Persona/s, Pain Point/s, user scenarios, and design idea/s.
Day 3: Hone in on the chosen design approach by using the supporting story artifacts to inform the details. Refine.
Day 4: Bring in representative target users from local customers to give user feedback to each team. Teams then iterate on their designs using what they learned. Teams prepare scenario-based presentations for executives.
Day 5: Deliver morning presentations, with group discussion.
Some newly implemented product feature ideas included in the subsequent concept reviews originated from this workshop.
[Workshop photos run the risk of exposing confidential IP]
3 Review Feature Concepts with Real Users
Engage local customers with Security Operations Centers (SOCs) for access to security agents to review new feature concepts.
At this point, designers and product managers had processed the newly generated workshop content and carried over to their backlogs any feature ideas they wanted to prioritize. Paper prototypes were mocked up by the designers and local customers were asked to host an on-site concept review with a few of their off-duty SOC Analysts.
Sessions consisted of walking through each mockup and discussing the background information that motivated it. Participants then responded with feedback and were also free to annotate the paper printouts given to them.
These were conducted in quick succession to make progress leaps with the user interface.
[no permission to show photos of concept review sessions]
Some of the new capabilities present in the design concepts:
-
Active incident volume overview chart based on severity
-
More filtering of the incident list from interactions on the header charts/fields
-
Improvements to the Incident detail panel, including Risk Scores for individual IPs and contextual actions on each info field to better support cognitive flow during incident investigations
-
‘Swim Lane’ visual charts to allow Analysts to scan across many categories of Incident types at the same time
-
Creation of investigation evidence containers (basically folders) for quickly gathering evidence/screenshots and easy access to toss in contents as an investigation progresses
Strong placement on the Gartner Quadrant
with indicator widgets and overview information
critical view for operations centers and analysts
compliance rules mean analysts must save screenshots of findings while investigating
with indicator widgets and overview information
Enterprise Security Product Screenshots
Gartner Magic Quadrant
Gartner has placed Splunk's Enterprise Security product in the leader/visionary quadrant in their Magic Quadrant report for 2016 Critical Capabilities for Security Information and Event Management (SIEM).
Gartner evaluated Splunk Enterprise and the Splunk Enterprise Security solution, which are used by organizations around the world.
"Splunk Enterprise Security meets the Critical Capabilities for Security Information and Event Management (SIEM) to improve the detection and response to advanced threats by providing broad security intelligence. The advanced security analytics capabilities of Splunk... provid[e] customers with the necessary features to implement advanced threat detection monitoring and insider threat use cases. Based on the need to protect against advanced threats, we have seen a growing number of organizations are using Splunk to augment or replace their existing SIEM deployment."
