The Framework is well intended but not realistic

“The security is the security, the ATO process is a completely separate beast.”

NIST designed the Framework to be flexible and adaptable to different situations. Practitioners we spoke to felt that flexibility is difficult to manage in practice.

“A lot of things that we do in the Federal government are so focused on compliance. And what we’ve been trying to do for quite a while now is, instead of focusing on compliance, to focus on the intent of compliance. Obviously there’s an intention of the ATOs to make sure that people are building applications that are secure.”

Despite many attempts by our participants to implement the intent of the Framework, almost all practitioners ultimately used it as a compliance checklist. Reinforcing this notion, when asked about the Risk Management Framework, practitioners primarily talked about Special Publication 800-53, the list of security controls. Few interview participants mentioned the Risk Management Framework itself (SP 800-37).

Consistent with the flexibility allowed in applying the tasks in the RMF, organizations conduct initial control assessments during system development and implementation. Conducting such assessments in parallel with the development and implementation phases of the SDLC facilitates early identification of deficiencies and provides a cost-effective method for initiating corrective actions.

- Implement, NIST SP 800-37

Applying and assessing controls throughout the development process may be appropriate for iterative development processes…. This type of incremental assessment is appropriate if it is more efficient or cost-effective to do so.

- Assess, NIST SP 800-37

NIST provides guidance for the management of security controls as part of agile or iterative software development but this message is buried in the literature. Even those familiar with it found it impractical to implement. Developers universally focused on software development and technical security as separate processes from applying the Framework. The Framework was considered a compliance exercise to be addressed after software development was complete.

It is neither practical nor useful to employ a compliance approach to the selection of security controls.

- Beyond Compliance: Addressing the Political, Cultural and Technical Dimensions of Applying the Risk Management Framework, The MITRE Corporation, 2014

Some Federal agencies did allow for Limited ATOs with fewer controls for minimum viable products. Even in the case of a Limited ATO, however, software was developed first and the Framework compliance process was appended at the end.

“It always seems like no matter how much I try to get something built in from the beginning, either scope creep or the customer changes directions or something happens and we always end up slapping it on at the end and I hate that.”

“I think most agencies, like I said earlier, they buy, they build, then they bring in security. That’s a huge problem.”

“I know ATOs don’t solve anything, right? We’re just going through compliance mechanisms.”

“You would get through seven, eight months of ATO and there would be no change in the security posture of your system. It was really more like security theater than anything else.”

Not only does NIST not require developers or agencies to use the Framework, they cannot enforce the controls. NIST is not a regulatory agency. Still, while NIST lacks the authority to enforce standards, SP 800-53 is used by Inspectors General, the General Accountability Office, the Office of the Federal Chief Information Officer, and other oversight bodies to evaluate programs. Because of this, even though NIST is not a regulator, NIST guidance acts as de facto regulation in the eyes of Authorizing Officials and every practitioner treats it as such.

“NIST is not a regulatory agency, so NIST has no power to require anybody to do anything.”

“You can make an informed risk decision [but] no one does that because the [Inspector General] comes in and says ‘Why didn’t you do 800-53?’”

Ultimately, all interview participants felt that NIST was thoughtful and well-intentioned. However, there was an overwhelming sense that the Risk Management Framework was no longer an effective way to manage security.

“I think NIST is incredibly good at the academic, about the expertise around these things. They don’t know how to turn what they do into a process or into something that people understand what to do with it.”