Just In Time Vendor Evaluations for Security Practitioners

[Updated December 11, 2012]

Deleted the post accidentally so I was lucky enough to recover it.
[Original post]

Do you ever get put on the spot with Just in Time(JIT) vendor security evaluations?  Here is the scenario, your the security lead for your group or company.  The phone rings and it’s 4pm on Friday.  A stressed out project manager says, “We need a vendor security evaluation for Company A and Company B”.  Of course, you were not included in any product meetings and demonstrations.  You know next to nothing about the companies or products and your now on the critical path — welcome to security.  What will you do?

Security theories and best practices abound, and then there’s the practical application ugly stuff we really do in the field.  Consider for a moment a medical analogy, a critically traumatized patient arrives to your unit.  Do avoid them until you can wash your hands?  No, you jump in and save a life!  It’s really the same with security.  Often we are called to perform security triage under less than ideal circumstances.  If your engaged late with a Mission Impossible security review it’s best to set clear expectations about what you can reasonably accomplish.  Realigning expectations is not fun but it’s better to do this up front in project rather than surprise everyone at the end.

Many organizations have legal reviews and other qualification procedures but assuming all that is the concern of others, let’s concentrate on security.  We agreed the evaluations will not be comprehensive but still beneficial.  With that in mind, role up your sleeves, there are some things you can do.  One of key things I like to know about an organization is their value or emphasis on security.  How do you gauge an organizations interest in security?  It’s not easy, after all security is a sensitive subject for most organizations any many are not forthcoming about their security practices.

Often the best insight into an organizations dedication to security is the information they communicate without directly communicating.  Confused?  I’ll explain, if you know the key questions to ask and signs to watch, you can learn a lot about an organizations willingness to invest in security.  Almost every organization says security is important.  But we need to select the organizations that put what they say into practice and invest in security.  A companies willingness and dedication to security is oftentimes at least as valuable to understand as the security maturity of their products.  Companies serious about security learn from their mistakes and drive their own security program improvements.  Companies not dedicated to security are doomed to fail and driven to improve largely by angry customers.  Not a fun experience — especially if your the angry customer.

The point of this thought exercise is to assess the organizations interest or dedication to security not necessarily to assess their product offerings.  Of course, a product security assessment is absolutely essential for a comprehensive review.  For the review, I’m assuming you have some access to key personnel at each perspective organization to ask at least a few questions.  Now for the fun stuff.

Company Security Leadership
Most publicly traded companies publish a leadership web page describing corporate executive profiles.  Does the leadership page include a security executive?  Is there a security executive listed like Chief Security Officer, Chief Information Security Officer, VP or Director of Security?  If so, it’s a good sign the security function is well-leveled within the organization.  Why is leveling important?  Security is a tough job and it almost always comes directly into conflict with production schedules.  Leveling security properly within organizations helps drive plans and priorities favorable to the security mission.  If security is not represented on the leadership page then the security responsibility, if it exists, is included within the responsibilities of other executives.  If no security executive position is listed it’s not necessarily a cause for concern but a if a position is shown it’s definitely positive.

How Many Individuals are Dedicated Full-Time to Security?
Ask how many individuals are dedicated full-time to security.  An integer less than one is a bad response.  What is a good response?  It really depends on the type of organization, number of products, and many other factors.  One vendor I interviewed responded with zero.  Their reasoning was that security is everyone’s job and the function is divided among staff.  A reasonable answer but hardly a confidence builder given the size and scope of product offering under our consideration at the time.  Needless to say, the vender didn’t win my vote.  Don’t make any assumptions about the people securing the software — ask!  Don’t be surprised if the organization pushes back and does not share all the information you request.  If they don’t share specifics focus more about the related details.  For instance, if they will not share information about staffing focus questions on what they do.  If they will not discus location and composition of the security team ask instead about the engineering team (or see if it’s in their financial reports).

If you do receive some information around staffing levels consider the scope of products you plan to purchase along with the size of the organizations security program.  The level of security program investment commensurate with the software’s operational capabilities and risks.  If it’s an enterprise or cloud offering you should expect some significant investments in security.  If purchasing a non-critical application used by a few individuals behind the corporate firewalls then perhaps less investment in security is appropriate.

The Software Life Cycle
There are many ways to structure a security program depending upon the assets to defend.  In the development of software, integrating security into the software life cycle is important   The software life cycle is a multi-phase engineering model organizations follow for creating and delivering software.  The exact phases or steps of the model depend upon the life cycle model in use but most start with concept and requirements gathering.  Progressing into design and development then software testing.  Finally into delivery and deployment.  Consider asking which operational and security activities occur at each stage of the life cycle.  Your looking for a continuum of security activities throughout the life cycle.

During your review, you will receive all types of responses but consider the following.  In early product development phases, security must be involved to influence product architecture.  Security is the foundation and you can’t very well build a house first and then go back and pour the foundation — although oddly enough this happens all the time in security.  Moving into design and development, what measures are in place to ensure engineers write secure code?  Does the company provide training and tools to help automate security reviews?  Can the organization certify anyone updating code in source control has security training?  What types of security tests are executed and when?  Are all code changes or improvements tested completely before moved into production?  Who has authority to move changes into production?  Do coders have direct access to deploy their code into production?  What documentation artifacts are delivered in each development phase?  Are there any automated security tests, if so, what are they?  Answers to all or some of these questions will help you understand more about the organization and it’s security practices.

How is the Security Program Organized?
Not all organizations are structured the same way.  However, positions such as the following are typical across industry.

Security Officers/Directors/Managers
Program leadership.  Responsible for the overall security program.  Establishing policies and tracking program effectiveness.

Security Architects
Ensure projects are designed securely from the start.  Create security requirements.  Create software security policies.

Security Engineers
Program software security features.  Work with engineers to ensure software changes comply with security policies.  Perform security reviews.  Maintain specialized security tools.  Implement cryptography functions.

Security Testers
Help develop and execute non-functional software security tests.  Often times Architects will be involved in test to ensure test cases support requirements and policies.  Security tests are important because without them it’s impossible to understand product security posture.

Security Compliance
Security compliance teams ensure changes adhere to security policies.  For example, if a new web server is deployed in the production environment and not listed in an approved software specifications document compliance may follow-up with management or shutdown the non-compliant server.

Security Operations
These people watch network traffic going over the wire.  They are usually the first to see bad guys knocking on the door.

Security Forensics
Forensics teams preserve evidence.  Often this includes, imaging desktops and servers for trial.  Computer data used in a court of law must be handled properly to preserve evidence and ensure it’s free from tampering.

Details about the organizations security functions are important to ensure the organization has security visibility across the entire software life cycle.  For instance, simply testing for vulnerabilities after the software is complete is not generally acceptable since software vulnerabilities are costly to remediate after they have been fully developed.  Your looking for a continuum of security activity across the development and operational processes.

On-Shore vs. Off-shore Staff
Is the organizations security staff on or off shore or a mix of both?  Don’t assume staff are onshore.  This may or may not be a concern for you but some organizations are highly concerned about off shore security staff.  You may consider asking which responsibilities are charged to each group.

Security Patch Programs
Does the organization provide regular product security patches?  Do they issue emergency fixes as necessary?  How are patches communicated to customers?  You may consider checking the support area of the organizations web site.  You may find some patching information, security communications, or public policy information.

If your purchasing a cloud solution, are security vulnerabilities quietly fixed and pushed or are they communicated?  Are mitigating controls like Web Application Firewalls(WAF) available?  The industry loves to hate WAFs.  WAF’s are not a perfect technology but engineering can often take significant time to code, test, and deliver a solution.  In the interim, your servers are wide open with a “kick me” sign.  The sweet spot for the WAF is that they deliver temporary protection very quickly while your teams deploy a more fully baked solution.

Security and Exchange Commission Fillings
US companies that are traded publicly must file documents like the yearly statements with the commission such as the 10-K or quarterly statements like the10-Q.  Sometimes you can find some security bugaboos in these statements or worthwhile information about the company concerns, engineering practices, onshore vs. offshore development, locations/practices for data storage, etc.

Social Media
Social media is also another organization barometer.  Sites like Twitter, LinkedIn, and Glassdoor can provide some insight into what current and past employees have to say about the company.  I would not place too much emphasis on social media but it’s worth a look.  Keep in mind you may not be reviewing objective or fair commentary.  It’s not likely you will find information directly related to the organizations security program but you can definitely find interesting details about engineering or IT practices.  If the engineers are pushed hard and frustrated to grind out basic software features it’s not likely the organization will invest time in security polishing.  You might also search for leadership blogs but it’s unlikely these yield much since this crowd is well versed about what they should or should not say.

Other Non-Functional Requirements
Hunt for any information you can find on product performance.  Is there significant positive or negative chatter in the support or news groups on performance issues?  Product performance is an interesting indicator since it’s something difficult to evaluate until the solution is deployed deeply.  The point being, if an organization is not investing in performance it’s likely they are not making adequate security investments either.  Like performance, security is difficult to evaluate without tools and expertise.

Audit Results
Does the organization have any recent IT audit results they are willing to share with you? SAS70’s are common and typically provide some IT information.  Rather than focus on the result of the audit, an obvious thing to do, consider reviewing other information around IT organization and processes.  Often organizational clues can help piece together a more comprehensive operational picture along with other data you collect elsewhere.  Be careful about placing too much trust upon audit conclusions.  Some audits allow for those under review to craft their own control objectives which diminishes their effectiveness.  Independent audits are a good tool but form your own opinion based upon several sources of information.  Trust but verify is a good tenant of security.

My article is not a comprehensive set of techniques to profile organizations by any measure.  Instead my intent is to increase your confidence around an organizations appetite for security where perhaps you had no confidence previously.  Ultimately, if you don’t have what you need for a proper security review you will need to stop the review and escalate.  But if I can help you reach some comfort in your decision making process, then I will have saved you from stepping in front of project steam rollers and freight trains which is far easier on your nerves.  :o)

Paper with Pen. Digital image. Openclipart.org. http://openclipart.org/, 12 Feb. 2012. Web. 7 Nov. 2012. <http://openclipart.org/people/jhnri4/Exquisite_kwrite.svg>.

Author: milton

For bio see, https://www.securitycurmudgeon.com/about/