My experience coleading purple team
I've been fortunate enough to manage a red team program for several years and since it's inception it has gone through many changes. What started out as adhoc engagements trying to see how far we could get/what problems we could find, turned into a mechanism to work more closely, and regularly with operations/it teams. More importantly, it's an approach to get operations teams to want to work with your security org more closely. This post will not discuss technical approaches to red teaming, it will discuss various strategies for making your program more impactful to the business. Below are my thoughts based on working with very talented red teamers, and operations teams.
Definitions
First some over simplified definitions for those unfamiliar.
Red: This is the group of hackers that penetrate everything, and provide guidance on how to fix the identified issues. They will also provide guidance on how to detect issues to the blue teams who may or may not have the maturity to write these detections on their own. They may, or may not also be in charge of running regular phishing campaigns.
Blue: This is typically two or more teams focusing on
- Incident Response: Responding to alerting/suspected incidents, building monitoring, rolling out endpoint security, breach readiness, and forensics
- System security & architecture: Network/host security posture (IT side, production side, dev environment, clouds, etc), alert definitions, hardening, patching, config management, vulnerability management, netsec/firewalls, security architecture, etc...
Purple: Purple teaming in my experience is the oversight of how red and blue operate, coordination to strengthen the effectiveness of both red/blue, and improved relationships with impacted stakeholders (dev/it/ops/etc). It likely isn't it's own team, it's the leaders of the blue/red teams coordinating with it's members and cross-org stakeholders to optimize how they operate.
Having come from a pentester background it's obvious why red teaming is sexy and desirable. On the business side I find purple teaming to be sexy for different reasons. One being that there's no one way to do it, and as I like to tell people, red teaming in 2018 is about where appsec was in 2005. A lot was starting to happen in appsec in the early 2000's and a lot is happening now on the red teaming side. I'd argue that purple teaming is more like where appsec was in 2000-2002. A handful of people know about it, some tools/services existed and people where basically just trial and erroring and reusing things that worked for them. Back then not many companies had appsec teams, and today very few companies have red team programs. Some people are giving talks but mostly focus on specific attack techniques, command and control tooling, phishing, and tooling/automation to speed up compromising/exfil/persistence/pivoting. Articles on purple teaming (including this one) seem to be trickling out slowly.
Specify goals, and outcomes for each red team engagement. Prioritize them with blue.
While open ended red team operations still yield good findings and should still be done with some cadence, more targeted engagements deep dive to answer specific questions. I've asked the red team to build a list of scenarios (a red teaming scenario database) of things we could test for. These should outline the areas of focus each engagement should focus on (not the same thing as trophies), and gives us an idea of things that can be tested for within a focal area. At the same time you should provide some flexibility for your testers so as to not tie their hands too much, as this may hinder potentially important findings from coming to light. It's important to ask blue team/s leaders what they'd like tested for and to add to this list, as well as to share some of the scenarios you plan on testing. Allowing blue to chime and prioritize particular red teaming scenarios is very important and helps them with their agenda, and the value the red team brings to the company. This isn't to say red should only be performing scenarios approved by blue, but allocating sufficient time for blue approved scenarios is a must. Especially if they are staffed right now, to handle the outcomes of things you planned on testing later.
Be able to answer these questions during each red team engagement
To increase the effectiveness of each engagement, the red/blue teams should be able to answer
- Was red team detected on each hop/escalation/pivot?
- How could red team have been detected/prevented from each hop/escalation/pivot?
- What are the root causes for each lapse that resulted?
- Could the red team have been detected faster?
Questions 1,2, and 4 will require close involvement with the blue teams. Each engagement should uplift not only outside teams but also the blue teams. The third question will also uplift blue teams, but will likely identify technical/procedural and technical issues within other teams.
Identify appropriate stakeholders to inform when engagements begin
Even when stealth engagements are occurring, a small handfull of people should be aware, namely incident response teams. You can choose to also inform the NOC/operations/IT lead (depending on what you're targeting), and possibly the oncall person from these teams. Let them know you're going to be doing something (it's up to you to decide how much to share), but that they shouldn't notify their teams since this provides a good test of how they respond. It's important that if for some reason things turn south during an engagement (such as an accidental outage occurring) that the teams likely to deal with it at least be informed on the leadership level. If you chose to inform them of engagement specifics, ask them if there are changes/additions you could make to help them answer questions about how they are doing (posture wise, procedure wise, etc). This touches on the next section.
Ask stakeholders if there are assumptions you could verify for them
It's important that if you're bringing findings to a team that will have to deal with it, that you show them you're willing to work for them however you can. Prior to starting an engagement against a group meet with one of the leaders, and a senior engineer, to explain the purpose of performing red teaming engagements. Ask them if there is anything they'd like you to check out for them. This could range to system posture, validating their processes (such as provisioning/decomming/patching/config management/etc). Often skeletons come out of the woodwork when discussions like this start, which can give you ideas for other engagements, or areas to verify elsewhere. They will appreciate being part of the process instead of just having work they didn't ask for being dumped on them.
Allocation sufficient time for red teaming
When you're in the early stages you may run a few full blown red team engagements per year with available staff. This is probably the right way to start off a red team program for most companies as it gives you an idea of the issues you'll find, and the level of work other orgs have to perform to deal with your findings. It's also very limiting and slows down building/strengthening the relationships with other teams such as IT, and the various operations groups. They may see you as identifying issues in snapshots of time, and some of the findings likely will get added to their backlog and linger. When planning/revising your red team roadmap, identify opportunities with the blue team/s for things you could be doing. This helps show the value the program brings and that there's a demand for it.
Be up front that security is important and that you understand this means more work for them. How can you help?
A bad approach to red teaming is dumping findings on another team and only providing technical support. You need to understand how well their team is staffed and how much work some of the things you're asking them to do truly is. Explain that you understand the outcome of your engagement may generate short term, and long term work. Often the people you're directly working with have little say on their goals for the quarter and beyond. You need to understand how they prioritize their work, and likely will need to speak with their program managers/project managers/pmo, or applicable management chain to see how work can be properly planned for, and captured/credited officially within their org. This may also include cheer-leading uplifts within their org to increase budgeting (hardware, headcount, software purchases, services, etc) to put them in a long term position of success.
Help them get credit for the fixes you're working with them on. Set goals.
As I've said earlier it's important to be a cheerleader for teams performing the security uplifts . This can be blue teams, or the it/dev/operations groups. It's important to highlight the benefit of the work they are doing with their leadership. Especially if that uplift saves them effort down the road and stops whack a mole situations. If you have a mechanism to bubble this up to the upper management, or executive level you should highlight the wins, and how red teaming helped them to uplift the security posture of the company. If possible work with their leadership to set official goals for the work they are doing.
Closing
Red/blue/purple is obviously a large problem space, and how each organization approaches it will need to vary. I have a few more articles in the pipeline, stay tuned!
Comments
You can follow this conversation by subscribing to the comment feed for this post.
All Comments are Moderated and will be delayed!
Post a comment