Carve COO Max Sobell presenting on “Shifting Security Left”​ at Giphy HQ.

Many a CTO and VP of Engineering has begrudgingly spent money on penetration tests in order to make their enterprise customers or auditors happy. You know how it goes: your team works hard to build a functional product, and despite making the customer happy in every way, they still come back with “show us your pen test report or we can’t buy your stuff.”

It’s even worse when you’ve already paid money for a pen test or security assessment on your own accord, and your customer says: “Not good enough. You need to use one of our approved vendors.”

In this post, I want the reader to learn how to get the most value out of a pen test engagement, whether you are performing it voluntarily, or you are being “voluntold” by a big customer or partner.

You get what you pay for

  • Not all pen tests are created equally, and you need to pick the vendor with the best methodology, not the cheapest price. With a diverse enough selection of vendors, you can get quotes ranging from hundreds of dollars to hundreds of thousands of dollars depending on your scope and the maturity of the vendor. Beware of cheaper options that rely solely on tooling – open source and commercial – as opposed to highly-skilled engineers to perform the “pen test.” Even worse, beware of expensive options that do the same thing for 100x the fee. For reference, Carve pen tests for start-ups are performed by highly-skilled engineers that leverage automation when appropriate, and range between $24,000 and $72,000 in fixed-price engagement fees.

Ask for customer references

  • Every pen test vendor says the same thing on their website, and they all have scary looking sample reports. But really good vendors have customers who are raving fans of their work and are happy to serve as references for prospective customers. Take advantage of these introductions to understand how and why these companies get value from their pen testing engagements. 

Do Threat Modeling

  • If you want a pen test that can improve the security of your products, and be an informative experience for your engineering team, you need a pen test that is guided by an accurate, tailored Threat Model. Phrases like SQL injection, cross-site scripting, SSRF, and other names of vulnerability classes are often thrown around by sales people and misrepresented as “threats.” These are generic vulnerabilities that can be present in any web application. You want a vendor who will consider threats unique to your software, for example: “attackers will attempt to make their own direct API calls to tamper with authorization parameters and access other users’ account data.”

Make it “grey box” (or “white box”)

  • Your pen test will usually need to happen quickly within a 2 to 3 week window. Giving pen testers access to your source code will allow them to quickly identify high-impact vulnerabilities, rule out false positives, and make better recommendations on how to remediate identified vulnerabilities. It’s true that attackers typically don’t have access to source, but they often have another significant advantage: unlimited time. Source code access will let the pen testers find high impact bugs within your short engagement window, and before the bad guys – or your big customer – find them. You don’t trust the pen test vendor with access to your source code? Find another vendor that you do trust.

Train your engineers based on your threat model & pen test results

  • At Carve, we use engagement results to tailor security training for engineering teams. After a grey box penetration test, a good pen tester aka security consultant will know more about your entire application than any individual engineer on your team. They will be able to speak about not just the vulnerabilities that exist now, but also what the big risks to your technology will be as you continue to build. The consultant will know the security strengths and weaknesses of both the software in question, and the SDLC processes that govern its creation and operation. When the consultant also happens to be an experienced software engineer, they will be able to connect with your engineering team based on mutual understanding and shared experience. All of these factors make Applied Threat Modeling Training far more impactful and valuable to your engineering team than generic, boiler-plate OWASP Top 10 training.

Bonus tip: avoid Q4 if you can

  • Many companies need pen testing performed annually, and they wait until the end of the year to have it done. If you are flexible and/or price constrained, it will benefit you to get your pen test done any other time of year. Firms generally have more availability during these times, and can offer more flexibly priced solutions.

There you go. Pen testing is one of the easiest things to do in the name of security because you just need to hire a 3rd party to do all of the heavy lifting. The hard part is picking the right vendor, and making sure your engineering team gets lasting value out of the engagement. If you do it right, pen testing can be a transformative experience for your engineering team, and can help you bolster a culture of security in your organization.If you’d like to learn what else you can do to more consistently reduce risk and improve the security of your ever evolving products earlier, sign up for our “Shift Security Left Webinar” and learn how the best teams implement “DevSecOps”.