No Starch Press's blog

This month, we continue our Author Spotlight series with an in-depth interview of Loren Kohnfelder—a true icon in the security realm, as well as the author of Designing Secure Software. In the following Q&A, we talk with him about the everlasting usefulness of threat modeling, why APIs are plagued by security issues, the unsolved mysteries of the SolarWinds hack, and what the recent Log4j exploit teaches us about the importance of prioritizing security design reviews.

Loren Kohnfelder is a highly influential veteran of the security industry, with over 20 years of experience working for companies such as Google and Microsoft—where he program-managed the .NET Framework. At Google, he was a founding member of the Privacy team, performing numerous security design reviews of large-scale commercial platforms and systems. Now retired and based in Hawaii, Loren expands upon his extraordinary contributions to security in his new book, detailing the concepts behind his personal perspective on technology (which can also be found on his blog), timeless insights on building software that's secure by design, and real-world guidance for non-security-experts.

No Starch Press: Aloha, Loren! We can’t talk about your new book without acknowledging the colossal impact your security work has had over the past five decades. For one, in your 1978 MIT thesis you invented Public Key Infrastructure (PKI), introducing the concepts of certificates, CRLs, and the model of trust underlying the entire freaking internet. You were also part of the Microsoft team that first applied threat modeling at scale, you co-developed the STRIDE threat-ID model (Spoofing identity, Tampering with data, Repudiation threats, Information disclosure, Denial of service and Elevation of privileges), and you helped bake security-design reviews into the development process at Google—all of which are key growth spurts in the evolution of software security.

Speaking of STRIDE, there are a lot of software professionals using the methodology who weren’t even born when you and Praerit Garg first invented it in the late ’90s. Pretty remarkable that something started in the era of desktop computing remains just as relevant in the age of cloud, mobile, and the web. Why do you think it continues to be so effective and seemingly immune from obsolescence?

Loren Kohnfelder: Aloha, Jen! STRIDE turned 23 this month, and the software landscape today is unrecognizable by comparison. Yet, the fundamentals of threat modeling are just as relevant as ever. I think that STRIDE's enduring value is due to its simplicity as an expression of very fundamental threats.

Since those early days, we now have pervasive internet connectivity, exponential growth in compute power and storage, cloud computing, and software itself is vastly more complex. All of these changes have grown the attack surface—plus our greater reliance on digital systems, as well as the massive amounts of data in the world, serve to increase motivation for attackers. For all of these reasons, threat modeling is more important than ever to gain a proactive view of the threat landscape for the best chance of designing, developing, and deploying a secure system.

It's important to note as well that STRIDE never purported to cover the only threats software needs to be concerned with, especially now that we have IoT, robots, and self-driving cars that directly act in the world, introducing new potential forms of harm. For applications beyond traditional information processing, it's critical to consider other possible threats for systems interacting with people and machines in powerful ways.

NSP: It’s normal for management to tap external security experts to ensure that tech products and systems are safe to deploy—often via a security review prior to release. An essential premise of your book rejects this standard in favor of moving security to the left. What’s wrong with the status quo, why is security by design better for the bottom line, and… are you ever worried that an angry mob of out-of-work security consultants might show up at your door?

LK: No worries at all that software security will be totally solved anytime soon, so there will continue to be strong demand for good minds defending our systems. This is the most challenging topic covered in the book, and my research included discussions with friends doing just that kind of work.

Rather than “reject,” I would say that I'm recommending moving left "in addition to." Here's what I wrote in the book (p. 235) on this: "Specialist consultants should supplement solid in-house security understanding and well-grounded practice, rather than being called in to carry the security burden alone." I don't think that any security consultant has ever concluded a review by saying, "I think I found the last vulnerability in the system!" So let's try to give them more secure software in the first place to review. The two approaches needn't be an either-or decision: the challenge is finding a good balance combining both approaches.

Honestly, I think the experts will appreciate reviewing well-designed systems without low-hanging fruit, so they can really demonstrate their chops by finding the more subtle flaws. In addition, solid design and review documents will provide a very useful map guiding their investigations compared to confronting a mass of code.

NSP: A point you make in the book is that “software security is at once a logical practice and an art form, one based on intuitive decision making.” This represents a paradigm shift for most developers, who tend to focus on “typical” use cases during the design phase—in other words, they presume the end product will be used as intended. You propose that they should actually be doing the opposite, that having a “security mindset” means looking at software and systems the way an attacker would. For those daunted by the prospect, can you explain what this means in practice?

LK: You have put your finger on the specific stretch that I'm inviting developers to make, and while the security mindset is a new perspective, I would say that it's more subtle than difficult. Having a security mindset involves seeing how unexpected actions might have surprising outcomes, such as realizing that a paperclip can be bent into a lockpick to open a tumbler lock. Another example from the book is a softball team deviously named "No Game Scheduled"—when the schedule was printed, other teams assumed the name meant that they had a bye, and therefore didn't show, forfeiting the game.

Again, this is a different viewpoint worth considering in addition to, not instead of, the usual. To help people new to the topic, the book is filled with all kinds of stories and basic examples that illustrate how attackers exploit obscure bugs. Malicious attacks on major systems regularly make the news, and we can decide to anticipate these eventualities throughout the development cycle, or not. It's worth adding that while security pros might be more facile at the security mindset, the software team members are the ones who know the code inside and out, so with a little practice they are better positioned for identifying these potential vulnerabilities.

NSP: Let’s talk about the bigger picture for a moment. We’re barely two years out from SolarWinds—one of the most effective cyber-espionage campaigns in history, where a routine software update launched an attack of epic proportions. If anything, it showed that threat actors know exactly how tech companies operate. Not to mention, the malicious code used in the attack was designed to be injected into the platform without arousing the suspicion of the development and build teams, which makes it all the more scary. If you could prescribe an industry-wide approach to preventing similar attacks in the future, what would it be?

LK: SolarWinds was a very sophisticated attack on a complex product, and the public information I've found doesn't provide a complete picture of exactly what actually happened. So my response here is based on a high-level take, not any specifics. First, I'd say that reliance on products like this, that are given broad administrative rights across large systems, puts a lot of high-value eggs in one basket.

I would love to see a detailed design document for the SolarWinds Orion product: did they anticipate potential threats (like what happened), and if so, what mitigations were built in? Publishing designs as a standard practice would give potential customers something substantial to evaluate, to see for themselves what risks products foresee and how they are mitigated. And when this kind of breach occurs, the design serves to guide analysis of events so we can learn how to do better in the future.

NSP: The massive scope of the SolarWinds incident—affecting dozens of companies and federal agencies—was made possible by the use of compromised X.509 certificates and PKI, in that attackers managed to distribute SUNBURST malware using software updates with valid digital signatures. Back in your time at MIT, you became known for defining PKI; today there’s an implication that code signed by software publishers is trustworthy, but in light of SolarWinds this no longer appears to be a safe assumption. Is there a solution to this on the horizon?

LK: I don't think it's possible to "fix" that, because it boils down to trust in the signing party and their competence. For example, if we are signing a contract and a lawyer uses sleight-of-hand to substitute a fraudulent version, I can be deceived into providing a valid legal signature. I don't know exactly what happened at SolarWinds, but they are the ones ultimately responsible for their code-signing key. In hindsight, I wonder if they fully realized how attractive their product could be to a sophisticated threat actor, and if they took the necessary precautions against that—which would be considerable. (For the record, I was not involved in the creation of the X.509 specification.)

Generally speaking, code signing is problematic because if vulnerabilities are found later, the signature remains valid—even though the code is known to be unsafe and no longer trustworthy. Administrators must go beyond checking for valid signatures, and also ensure that it's the latest version available, before trusting any code, and of course promptly install future updates that fix critical issues.

NSP: On that note, the concept of “good enough security” is predicated on the belief that the threat landscape is somehow static in nature. One thing you really stress in the book is the importance of understanding the full range of potential threats to information systems—and that means accepting that there are adversaries out there whose capabilities are higher than the current standards software developers abide by. How can dev and ops teams work together to implement security measures that not only address what is known, but also deal with threats as they evolve over time, so they can stay a step ahead of persistent adversaries?

LK: Just as you say, broad and deep threat-awareness is important as a starting point, and then the really hard part is choosing mitigations and ensuring that they do the job intended. This is a subjective process, and if you really want to stay ahead, as you say, that usually means aggressively mitigating just about every threat that you can identify, so as not to be blindsided.

Excellent point about the dangers of treating the threat landscape as static, and this is also a strong argument for moving left—because the more mitigation you work into the design, the better. (Plus, as the environment evolves, it's very hard to go back to the design for a redo!)

NSP: It’s common knowledge that “all software has bugs,” and that a subset of those bugs will be exploitable—ergo, the challenge of secure coding essentially amounts to not introducing flaws that become exploitable vulnerabilities. Seems easy enough! But programmers are only human, and while many of them make the effort to build protection mechanisms that improve safety in, say, the features of APIs, what are some everyday programming pitfalls you see as the root causes of most security failings?

LK: While “all software has bugs” is generally accepted as true, too often I think the connection to vulnerabilities is under-recognized. Instead, it's easy to rationalize lower quality standards since the end product will have bugs anyway. Part III of the book covers many of just these common pitfalls in a little over 100 pages, so I won't attempt to summarize all that here.

If your question about root causes goes deeper, asking why programming languages and API are so prone to security problems, then I would say it's often simply because the practice of software development goes back before there was much awareness of security. For example, the C language has been profoundly influential, and it's still widely used, but it also gave us arithmetic overflow and buffer overruns. The inventors surely knew about these potential flaws but had no way to imagine the 2022 digital ecosystem, and threats like ransomware. The same goes for API, which can fail to anticipate evolving threats that, once distributed, are very hard to fix later.

Another common cause in API design is failing to present a clean interface, in terms of trust relationship and security responsibility. API providers naturally want to offer lots of features and options, yet this makes the interface complicated to use. Since the implementation behind the API is typically opaque to the caller, it's easy for a mistake to arise. So it's imperative for the API documentation to provide clear security commitments, or detail exactly what precautions callers must take and why. Log4j is a perfect example of this problem: surely most applications reasonably assumed it was safe to log untrusted inputs, but the JNDI feature—that they may not even have been aware of—offered attackers an attractive point of entry.

NSP: Since you brought it up, and it sort of ties together everything we’ve been discussing, let’s talk about that Apache Log4j zero-day vulnerability (which continues to make headlines). Here we have a Java-based logging library used by millions of applications, that has a critical flaw described as basically “trivial” to exploit. Why is this bug considered so incredibly severe? And, even though your book was released before the issue was discovered, are there any nuggets of wisdom in it that address this type of issue—or that could help software developers solve the problems that led to it?

LK: Log4j could be the poster child for the importance of security design reviews. Much has been written already by folks who have examined this extensively, but clearly allowing LDAP access via JNDI was a design flaw. Whether the designer(s) recognized the threat or not, mitigated insufficiently, or simply failed to understand the consequences, is hard to say without a design document (much less a review). Skipping secure design and review means missing the best opportunities to catch exactly this sort of vulnerability before it ever gets released in the first place.

This vulnerability is nearly a Perfect Storm because of a combination of factors: it allows remote code execution (RCE) attacks, the vulnerable code is very widely used by Java applications, and as a logging utility it's often exposed to the attack surface. That last point deserves elaboration: attackers often poke at internet-facing interfaces using malformed inputs in hopes of triggering a bug that might be a vulnerability; and developers want to monitor the use of these interfaces, so they log the untrusted input, creating a direct connection to the vulnerable code in Log4j. It so happens that the book includes an example design document for a simple logging system (Appendix A), and that API explicitly uses structured data (as JSON) rather than strings with escape sequences that got Log4j into trouble.

Furthermore, threat modeling and secure software design should have informed all applications using Log4j of the risks involved in logging untrusted inputs. In the book's Afterword, I write about using software bills of material (that would have identified which applications use Log4j), and the importance of promptly updating dependencies (in this case, the slow response to Log4j is why it's still in the news), just to name a few additional mitigations that’d help. (I posted about Log4j at more length last year when it first became public.)

NSP: Wow, Loren—maybe you should come out of retirement! At the very least, Designing Secure Software should be required reading for everyone in the field, and it’s clearly becoming more urgent with every passing day.

LK: Thanks for your kind words and this opportunity to reflect on current events. The book is my way of stepping out of retirement to share what I've learned in hopes of nudging the industry in some good new directions. I certainly recognize that investing security effort from the design stage runs counter to a lot of prevailing practice, but I've seen it practiced to good effect, and now there's a manual available if anyone wants to try the methodology.

I think that our discussion nicely shows the value of moving beyond reactive security, moving left to be more proactive. The book offers lots of actionable ideas, and it's written for a broad software audience so we can get more developers as well as management, interface designers, and other stakeholders all involved. No doubt security lapses will continue to occur—but when they do, we need more transparency to fully understand exactly what happened and how best to respond, and then to take those learnings and institute the changes necessary to improve in the future.

For our first Author Spotlight interview of 2022, we have illustrious guest Michal Zalewski—world-class security researcher and author of the newly released Practical Doomsday: A User’s Guide to the End of the World. In the following Q&A, we talk with him about taking disaster preparedness back from the fringe, what he's learned from living through numerous calamities, the reason hackers have the edge over doomsday preppers in any real emergency, and why he’s got a solid backup plan “if this whole computer thing turns out to be a passing fad.”

Michal Zalewski (aka lcamtuf) has been the VP of Security & Privacy Engineering at Snap Inc. since 2018, following an 11-year stint at Google, where he built the product security program and helped set up a seminal bug bounty initiative. Originally hailing from Poland, he kick-started his career with frequent BugTraq posts in the ’90s, and went on to identify and publish research on hundreds of notable security flaws in the browsers and software powering our modern internet. In addition to his influence on the tech industry, Zalewski's known as the developer of the American Fuzzy Lop open-source fuzzer and other tools. He's also the author of two classic security books via No Starch Press, The Tangled Web (2011) and Silence on the Wire (2005), and is a recipient of the prestigious Lifetime Achievement Pwnie Award.

NSP: Gratulacje on your new book, Michal! Suffice it to say, a practical prep guide for doomsday scenarios could not be more timely (...all things considered). You even joke on Twitter that the past few years were an elaborate viral marketing campaign for the book’s release. But in fact, you’ve been writing on this subject since at least 2015. What first lured you into the disaster-preparedness genre?

Michal Zalewski: I keep asking myself the same question! For one, I simply grew up at an interesting time and in an interesting place: in a failed Soviet satellite state going through a period of profound political and economic strife. As a child, I didn’t think of it much, but as an adult, I look at my early years with a degree of terror and awe.

I also have this geeky curiosity about how complex systems work and how they might fail—and to be frank, I can’t quite grasp why we look at this problem so differently in the physical world versus the digital realm. After all, it’s normal to back up our files or use antivirus software; why is it wacky to buy fire extinguishers for one’s home or store several gallons of water and some canned food?

In my mind, risk modeling and common-sense preparations shouldn’t be a political issue and shouldn’t be the domain of people who are convinced that the end is nigh. If anything, having a backup plan is a wonderful way to dispel some of the worries and anxieties of everyday life.
 

NSP: Your personal bio illustrates one of the key points in the book—that disasters are not rare. In addition to growing up in Poland in the '80s, the book also brings up the experience of living through 9/11, the dot-com crash, and the housing crisis of 2008. Would you explain that larger theme within the context of your own trials and tribulations?

MZ: Oh, I don’t want to oversell my life story! My experiences are shared by tens of millions of people around the globe. Countless others have lived through much worse— famine, devastating natural disasters, wars.

I’m going to say one thing, though. Living through a sufficient number of calamities reveals a simple truth: that every generation gets to experience their own “winter of the century,” “recession of the century,” “pandemic of the century,” and so on. And every time, such events catch them off guard.

In most cases, it’s not a matter of life and death; most people make it through recessions, wildfires, and floods. But having a robust plan can make the situation much less stressful, and can make the recovery more certain and more swift.
 

NSP: Most people picture doomsday preppers as ex-military survivalist-types—not a self-described “city-raised computer nerd.” How has your hacker background informed the emergency-preparedness thought process you’re teaching readers in the book?

MZ: If there’s one obvious difference, it’s that in the physical realm, life-threatening incidents are fairly rare. In the world of computing, on the other hand, networks and applications are under constant attack. When you work in this domain, I think you start to appreciate the saying attributed to Mike Tyson: “everyone has a plan until they get punched in the mouth”— that is, theory seldom survives the clash with reality. By the end of the day, the surest way to get through an emergency is to be adaptable and resilient, not to have an impressive stockpile guns and bushcraft tools.

Another principle I picked up from the world of information security is that there is no limit to how much time, effort, and money you can spend in the pursuit of perfection—but perfection is not necessarily a useful goal. A good preparedness strategy needs to zero in on problems that are important, plausible, and can be addressed in a cost-effective way, without jeopardizing your quality of life should the apocalypse not come.
 

NSP: As a teenager, you became active in Europe’s fledgling infosec community, which led to consulting projects, pentesting gigs and, eventually, a remarkable career in the industry. Based on your own success, what do you think it takes to truly succeed in the infosec field and/or what’s your best career advice for aspiring security researchers?

MZ: I try to be careful with career advice—sometimes, people are successful despite their habits, not because of them. That said, I certainly found it helpful to always approach security in a bottom-up fashion. If you make the effort to understand how the underlying technologies really work, their failure modes become fairly self-evident too.

My best advice for aspiring professionals is different, though: perhaps the most underrated skill in tech are solid writing skills. That’s because technical prowess is not sufficient to succeed—you need to get others on board. I have a short Twitter thread with a handful of tips here.
 

NSP: In addition to your street cred in the security world, you’re credited with (inadvertently) helping hackerdom in another realm entirely—Hollywood. The Matrix Reloaded is lauded as the first major motion picture to accurately portray a hack. More specifically, your hack. For those who haven’t seen it, Trinity uses an Nmap port scan, followed by an SSH exploit to break into a power company and disable the city’s electric grid. In 2001, you discovered the SSH bug being depicted on screen. Can you tell us anything about your vulnerability report being in one of the movie’s most pivotal scenes?

MZ: I wish I had a cool story to tell! I was surprised (and flattered) to see my bug on the big screen. My other cinematic claim to fame is having my fuzzer—American Fuzzy Lop—surface in the TV series Mr. Robot.

Of course, my screen credits pale in comparison with the track record of the aforementioned NMap tool. The network scanner makes an appearance in at least a dozen films and TV series, reportedly including at least one porn flick.
 

NSP: In an example of life imitating art, the intelligence community has recently sounded the alarm over an “unprecedented” uptick in hackers targeting electric grids. Maybe if the fictional power company in The Matrix Reloaded had someone like you working for them, Trinity’s blackout-inducing exploit would have failed—which begs the question: do you think white-hat hackers could be the answer to the risk that APTs pose to critical infrastructure? Is it as simple as utility providers adopting bug-bounty bounty programs, such as the one your team launched at Google a decade ago?

MZ: Bug bounties are a cherry on top for a mature security program: they are a last-resort mechanism to catch a fraction of mistakes that slip past your internal defenses. But if you’re routinely letting vulnerabilities ship to production and then hope that talented strangers will catch them all, you’re playing a very dangerous game.

A comprehensive security program starts with minimizing the risk of such mistakes in the first place: building automation that makes it easy to do the right thing and difficult for humans to mess up. The second layer of defense are internal processes for vetting the design and implementation of your systems, and for penetration-testing or fuzzing the products before they go out the door.

Still, the problem faced by most utilities isn’t related to any of this: it’s that we have a fairly small pool of infosec talent and that companies are fiercely competing for that talent. The Wyoming Rural Electric Association doesn’t have it easy when even the most junior security engineer can land an interview with Amazon, Goldman Sachs, or SpaceX.
 

NSP: From your early years posting software vulnerabilities on BugTraq, to your research exposing the flawed security models of web browsers, to helping Google build its massive product security program, you've become known as one of the most influential people in infosec. Over the same decades, the internet has gone from a place of dial-up connections and friendly message-boards to a global network that governs nearly every aspect of digital society. Given your unique vantage point in this regard, what do you think is the most pressing challenge in the industry today?

MZ: I'm not an infosec malcontent—I think our industry has made impressive progress when it comes to reasoning about and reducing the risk of most types of security flaws. But as you note, the stakes are getting higher too: nowadays, almost everything is connected to the internet, and even the humble thermostat on your wall might be running more than ten millions lines of code. This makes absolute security a rather challenging goal.

In light of this, the two keywords that come to mind are "compartmentalization" and "containment." You have to plan for unavoidable mishaps and must have a way to prevent them from turning into disasters. For enterprises, this may involve dividing systems into smaller, well-understood blocks that can be cordoned off and monitored for anomalies with ease. The technologies and the architecture paradigms that make this possible are still in their infancy, but I think they hold a lot of promise.

Of course, we can practice compartmentalization and containment in everyday life, too. Perhaps only so much in your life should depend on the security of a single email provider or a single bank.
 

NSP: Last question! One of the prepper commandments in your book is, simply, “Learn new skills.” Why is this important for building a comprehensive disaster-preparedness plan, and what are some useful secondary skills that you have developed outside of infosec?

MZ: The point I make in the book is that the accelerating pace of technological change means that fewer and fewer jobs are for life. You know, in the 1990s, opening a VHS rental place or a music store was a sound business plan, journalism was a revered and well-paying gig, and the photographic film industry was a behemoth that consumed about a third of the global silver supply. We are probably going to see similar shifts in the coming decades. In particular, I’m not at all convinced that software engineers are still going to be an elite profession in 20-30 years.

It’s hard to predict the future, but it’s possible to hedge our bets—say, by pursuing potentially marketable hobbies on the side. Even if nothing happens, such pursuits are still rewarding on their own. I enjoy woodworking and tinkering with electronics. I could probably turn these hobbies into gainful employment if this whole computer thing turns out to be a passing fad.
 

Our always fascinating Author Spotlight series continues with Jon Gjengset – author of Rust for Rustaceans. In the following Q&A, we talk with him about what it means to be an intermediate programmer (and when, exactly, you become a Rustacean), how Rust “gives you the hangover first” for your code's own good, why getting over a language's learning curve sure beats reactive development, and how new users can help move the needle toward a better Rust.

A former PhD student in the Parallel and Distributed Operating Systems group at MIT CSAIL, Gjengset is a senior software engineer at Amazon Web Services (AWS), with a background in distributed systems research, web development, system security, and computer networks. At Amazon, his focus is on driving adoption of Rust internally, including building out internal infrastructure as well as interacting with the Rust ecosystem and community. Outside of the 9-to-5, he conducts live coding sessions on YouTube, is working on research related to a new database engine written in Rust, and shares his open-source projects on GitHub and Twitter.

No Starch Press: Congratulations on your new book! Everyone digs the title, Rust for Rustaceans – which is a tad more fitting than its original moniker, Intermediate Rust. I only bring this up because both names speak to who the book is for. Let’s talk about that. What does “intermediate'' mean to you in terms of using Rust? Specifically, what gap does your book fill for those who may have finished The Rust Programming Language, and are now revving to become *real* Rustaceans?

Jon Gjengset: Thank you! Yeah, I’m pretty happy with the title we went with, because as you’re getting at, the term “intermediate” is not exactly well-defined. In my mind, intermediate encapsulates all of the material that you wouldn’t need to know or feel comfortable digging into as a beginner to the language, but not so advanced that you’ll rarely run into it when you get to writing Rust code in the wild. Or, to phrase it differently, intermediate to me is the union of all the stuff that engineers working with Rust in real situations would pick up and find continuously useful after they’ve read The Rust Programming Language.

I also want to stress that the book is specifically not titled "The Path to Becoming a Rustacean," or anything along those lines. It’s not as though you’re not a real Rustacean until you’ve read this book, or that the knowledge the book contains is something every Rustacean knows. Quite the contrary – in my mind, you are a Rustacean from just before the first time you ask yourself whether you might be one, and it’s at that point you should consider picking up this book, whenever that may be. And for most people, I would imagine that point comes somewhere around two thirds through The Rust Programming Language, assuming you’re trying to actually use the language on the side.
 

NSP: Rust has been voted “the most loved language” on Stack Overflow for six years running. That said, it's also gained a reputation for being harder to learn than other popular languages. What do you tell developers who are competent in, say, Python but hesitant to try Rust because of the perceived learning curve?

JG: Rust is, without a doubt, a more difficult language to learn compared to its various siblings and cousins, especially if you’re coming from a different language that’s not as strict as Rust is. That said, I think it’s not so much Rust that’s hard to learn as it is the principles that Rust forces you to apply to your code. If you’re writing code in Python, to use your example, there are a whole host of problems the language lets you get away with not thinking about – that is, until they come back to bite you later. Whether that comes in the form of bugs due to dynamic typing, concurrency issues that only crop up during heavy load, or performance issues due to lack of careful memory management, you’re doing reactive development. You build something that kind of works first, and then go round and round fixing issues as you discover them.

Rust is different because it forces you to be more proactive. An apt quote from RustConf this year was that Rust “gives you the hangover first” – as a developer you’re forced to make explicit decisions about your program’s runtime behavior, and you’re forced to ensure that fairly large classes of bugs do not exist in your program, all before the compiler will accept your source code as valid. And that’s something developers need to learn, along with the associated skill of debugging at compile time as opposed to at runtime, as they do in other languages.

It’s that change to the development process that causes much of (though not all of) Rust’s steeper learning curve. And it’s a very real and non-trivial lesson to learn. I also suspect it’ll be a hugely valuable lesson going forward, with the industry’s increased focus on guaranteed correctness through things like formal verification, which only pushes the developer experience further in this direction. Not to mention that the lessons you pick up often translate back into other languages. When I now write code in Java, for instance, I am much more cognizant of the correctness and performance implications of that code because Rust has, in a sense, taught me how to reason better about those aspects of code.
 

NSP: In the initial 2015 release announcement, Rust creator Graydon Hoare called it “technology from the past come to save the future from itself.” More recently, Rust evangelist Carol Nichols described it as “trying to learn from the mistakes of C, and move the industry forward.” To give everyone some context for these sentiments, tell us what sets Rust apart safety-wise from “past” systems languages – in particular, C and C++ – when it comes to things like memory and ownership.

JG: I think Rust provides two main benefits over C and C++ in particular: ergonomics and safety. For ergonomics, Rust adopted a number of mechanisms traditionally associated with higher-level languages that make it easier to write concise, flexible, (mostly) easy-to-read, and hard-to-misuse code and interfaces – mechanisms like algebraic data types, pattern matching, fairly powerful generics, and first-class functions. These in turn make writing Rust feel less like what often comes to mind when we think about system programming – low-level code dealing just with raw pointers and bytes – and makes the language more approachable to more developers.

As for safety, Rust encodes more information about the semantics of code, access, and data in the type system, which allows it to be checked for correctness at compile-time. Properties like thread safety and exclusive mutability are enforced at the type-level in Rust, and the compiler simply won’t let you get them wrong. Rust’s strong type system also allows APIs to be designed to be misuse-resistent through typestate programming, which is very hard to pull off in less strict languages like C.

Rust’s choice to have an explicit break-the-glass mechanism in the form of the unsafe keyword also makes a big difference, because it allows the majority of the language to be guaranteed-safe while also allowing low-level bits to stay within the same language. This avoids the trap of, say, performance-sensitive Python programs where you have to drop to C for low-level bits, meaning you now need to be an expert in two programming languages! Not to mention that unsafe code serves as a natural audit trail for security reviews!
 

NSP: Along those same lines, Rust (like Go and Java) prevents programmers from introducing a variety of memory bugs into their code. This got the attention of the Internet Security Research Group, whose latest project, Prossimo, is endeavoring to replace basic internet programs written in C with memory-safe versions in Rust. Microsoft has also been very vocal about their adoption of Rust, and Google is backing a project bringing Rust to the Linux kernel underlying Android. As Rust is increasingly embraced and used for bigger and bigger projects, are there any niche or large-scale applications, or certain technology combos you’re most excited about?

JG: Putting aside the discussion about whether Rust prevents the same kinds of bugs in the same kinds of ways as languages like Go and Java, it’s definitely true that the move to these languages represent a significant boost to memory safety. And I think Rust in particular unlocked another segment of applications that would previously have been hard to port, such as those that would struggle to operate with a language runtime or automated garbage collection.

For me, some of the most exciting trajectories for Rust lie in its interoperability with other systems and languages, such as making Rust run on the web platform through WASM, providing a better performance-fallback for dynamic languages like Ruby or Python, and allowing component-by-component rewrites in established existing systems like cURL, Firefox, and Tor. The potential for adoption of Rust in the kernel is also very much up there if it might make kernel development more approachable than it currently is – kernel C programming can be very scary indeed, which means fewer contributors dare try.
 

NSP: In the book’s foreword, David Tolnay – a prolific contributor to the language, who served as your technical reviewer – says that he wants readers to “be free to think that we got something wrong in this book; that the best current guidance in here is missing something, and that you can accomplish something over the next couple years that is better than what anybody else has envisioned. That’s how Rust and its ecosystem have gotten to this point.” The community-driven development process he’s referencing is somewhat unique to Rust and its evolution. Could you briefly explain how that works?

JG: I’m very happy that David included that in his foreword, because it resonates strongly with me coming from a background in academia. The way we make progress is by constantly seeking to find new and better solutions, and questioning preconceived notions of what is and isn’t possible, or how things “should” be done. And I think that’s part of how Rust has managed to address as many pain points as it does. The well-known Rust adage of “fast, reliable, productive, pick three” is, in some sense, an embodiment of this sentiment – let’s not accept the traditional wisdom that this is a fundamental trade-off, and instead put in a lot of work and see if there’s a better way.

In terms of how it works in practice, my take is that you should always seek to understand why things are the way they are. Why is this API structured this way? Why doesn’t this type implement Send? Why is static required here? Why does the standard library not include random number generation? Often you’ll find that there is a solid and perhaps fundamental underlying reason, but other times you may just end up with more questions. You might find an argument that seems squishy and soft, and as you start poking at it you realize that maybe it isn’t true anymore. Maybe the technology has improved. Maybe new algorithms have been developed. Maybe it was based on a faulty assumption to begin with. Whatever it may be, the idea is to keep pulling at those threads in the hope that at the other end lies some insight that allows you to make something better.

The end result could be an objectively better replacement for some hallmark crate in the ecosystem, an easing of restrictions in the type system, or a change to the recommended way to write code – all of which move the needle along towards a better Rust. That sentiment's best summarized by David Tolnay’s self-quote from 2016: “This language seems neat but it's too bad all the worthwhile libraries are already being built by somebody else.”
 

NSP: Alumni of the Rust Core team have said that it’s a systems language designed for the next 40 years – quite an appealing hook for businesses and organizations that want their fundamental code base to be usable well into the future. What are some of the key design decisions that have made Rust, in effect, built to last?

JG: Rust takes backwards compatibility across versions of the compiler very seriously, and the intent is that (correct) code that compiled with an older version of the compiler should continue to compile indefinitely. To ensure this, larger changes to the language are tested by re-building all versions of all crates published to crates.io to check that there are no regressions. Of course, the flip side of backwards compatibility is that it can be difficult to make improvements to the language, especially around default behavior.

The Rust project’s idea to bridge this divide is the “edition” system. At its core, the idea is to periodically cut new Rust editions that crates can opt into to take advantage of the latest non-backwards-compatible improvements, but with the promise that crates using different editions can co-exist and interoperate, and that old editions will continue to be supported indefinitely. This necessarily limits what changes can be made through editions, but so far it has proven to be a good balance between “don’t break old stuff” and “enable development of new stuff” that is so vital to a language’s long-term health.

The Rust community’s commitment to semantic versioning also underpins some of Rust’s long-term stability promises – that is, by allowing crates to declare through their version number when they make breaking changes, Rust can ensure that even as dependencies change, their dependents will continue to build long into the future (though potentially losing out on improvements and bug fixes as old versions stop being maintained).
 

NSP: One of the goals listed on the Rust 2018 roadmap was to develop teaching resources for intermediate Rustaceans, which I believe is what spurred you to start streaming your live-coding sessions on YouTube. Developers have really embraced them as a way of learning how to use Rust “for real.” Why is it useful, in your view, for newcomers to see an experienced Rust programmer go through the whole development process and see real systems implemented in real time?

JG: Learning a language on your own is a daunting task that requires self-motivation and perseverance. You need to find a problem you’re interested in solving; you need to find the will to get through the initial learning curve where you’ll get stuck more often than you’ll make meaningful progress; and you have to accept the inevitable rabbit holes that you’ll go down when it turns out things don’t work the way you thought they did. That’s not an insurmountable challenge, and some people really enjoy the journey, but it is also time-consuming, humbling and, at times, quite frustrating. Especially because it can feel like you’re infinitely far from what you really wanted to build.

Watching experienced developers build something, especially if you’re watching live and can ask questions, provides a shortcut of sorts. You get to be directly exposed to good development and debugging processes; you get exposure to language mechanisms and tools that you may otherwise not have found for a while on your own; and you spend less time stuck searching for answers, since the experienced developer can probably explain why something doesn’t work shortly after discovering the problem. Of course, it’s not a complete replacement. You don’t get as much of a say in what problem is being worked on, which means you may not be as invested in it, and you won’t get the same exposure to teaching resources that you may later need as you’re trying to work things out on your own. Ultimately, I think of it as a worthwhile “experience booster” to supplement a healthy and steady diet of writing code yourself.
 

NSP: The popularity of your videos notwithstanding, you’ve said that part of what inspired you to write the book is that “they’re not for everyone,” and that some people – yourself included – have a different learning style. Given both mediums cover advanced topics (pinning, async, variance, and so on), would you say the book is an alternative to the live coding sessions, or is it designed to complement them? In other words, would a developer who’s watched your videos still benefit from the book (and vice versa)?

JG: It’s a bit of a mix. The "Crust of Rust" videos cover topics that are covered in the book, and the book covers topics in my videos, but often in fairly different ways. I think it’s likely that consuming both still leads to a deeper understanding than consuming either in isolation. But I also think that consuming either of them should be enough to at least give you the working knowledge you need to start playing with a given Rust feature yourself.

For readers of the book, I would actually recommend watching one of the longer live-coding streams on my channel (over the Crust videos), because they cover a lot of ground that’s hard to capture in a book. Topics like how to think about an error message, or how to navigate Rust documentation work best when demonstrated in practice. And who knows – you may even find the problem area interesting enough that you watch the whole thing to the end!

And with that… std::process::exit
 

Vickie Li is the resident developer evangelist at the application security firm ShiftLeft, and a self-described “professional investigator of nerdy stuff.” Her new book, Bug Bounty Bootcamp, leverages her expertise in offensive web security as well as her background in vulnerability research to introduce beginners to all aspects of web hacking, showing readers how to find, exploit, and report bugs through “bounty” programs. In her free time, when she’s not podcasting, speaking at conferences, or dropping infosec and cybersecurity knowledge on YouTube, she’s writing articles and blog posts about nipping security problems in the bug.

For the September edition of our ongoing Author Spotlight series, we talk with Vickie about her first bug bounty payout, how her success hacking apps made her a passionate advocate for secure development, and why she means it quite literally when she tells you that becoming a good web hacker is like learning to ride a unicycle.

No Starch Press: First of all, that’s a pretty impressive intro for someone in their mid-twenties! But let’s go back a few years. You graduated with a CS degree from Northwestern, then worked as a freelance web developer before getting into infosec, pentesting, and offensive-security content creation, which – correct me if I’m wrong – led to your current full-time gig as a developer evangelist. So where did your foray into bug hunting come into play, and how did you get started with bounty programs?

Vickie Li: I got interested in security through my university courses, and started bug bounties as a way to learn more about infosec. Hacking on bug bounty programs helped me learn a lot about web hacking and web application security in general. But sitting in front of my laptop all day, I started to lose motivation because I really wanted my work to connect me with other people, and doing bug bounties all alone was quite lonely. That’s why I started my technical blog, where I wrote about whatever I was learning at the moment. I really tried to make the blog posts easy to understand, because I hoped people who were studying the same thing would find it helpful.

My blog actually kickstarted my career in infosec. Because of it, I was able to get some freelance penetration testing and technical writing jobs, and eventually landed my current job at ShiftLeft. Knowing how to explain complex technical concepts also helped me with writing Bug Bounty Bootcamp and making it an approachable web-hacking book.

NSP: What was your first real catch, and what was it like earning your first paid bounty?

VL: I found my first paid bug – a CSRF – about a week into hunting for bugs. The bounty was just a hundred dollars, but it was amazing to be able to earn a bit of money as I learned about the field. The most memorable part about the experience was when [the company’s] security team triaged the bug I found, and fixed it on the website. It was very motivating to know that I can contribute to the security of a widely used site through my work!

NSP: Over the past year you’ve gone from working as a freelancer/bug hunter to a full-time gig as a “developer evangelist” – a job focused on bridging communications between external dev teams and your internal app-security colleagues. Can you elaborate on what exactly your day-to-day is like, and how it satisfies your infosec interests?

VL: My primary role at ShiftLeft involves making secure coding practices approachable for developers, and spreading the word about how static analysis can help in this process. Every day is different: I might be writing a blog post, preparing to speak at a conference, or helping my team understand the needs of developers during the security process. I really enjoy the work because it fits into my original motivation for getting into infosec: helping make the internet a safer place for everyone.

NSP: The name of your company refers to shifting security to the left – or, introducing security checks earlier in the development life cycle rather than at the end. You underscored this in a blog post, comparing app security to wearing a facemask during the pandemic (“Building a Security-First Culture”). At the same time, your book is about hunting for zero-day vulnerabilities and getting started in bounty programs. Do you ever worry that if you’re too good at your job there won’t be any more bugs to hunt?

VL: I am not worried about that. Shifting left and bug bounties are not an either-or situation. These practices work together to help organizations become more secure. Bug bounty hunters are creative and are constantly coming up with new ways to attack an application. Organizations can use bug bounty programs to tackle new and inventive attack vectors before malicious attackers discover them. But most bugs should still be discovered early in the development cycle, when they are the easiest to fix. Shifting left will help eliminate most security vulnerabilities in your applications, and bug bounties can help you catch the rest.

NSP: To take this question in the opposite direction, has your bug-hunting experience helped or informed your current work advocating for better security practices?

VL: During my time as a bug bounty hunter, I helped lots of developer teams fix security issues in their applications. That’s when I noticed that many serious security vulnerabilities stem from small programming mistakes that could be easily discovered with static analysis. It’s easier to find and fix vulnerabilities early in the development process because you do not risk an attacker exploiting it in production.

This experience made me a really passionate advocate for secure development and security education. Offensive security practices like penetration testing or bug bounties are a great way to secure your applications, but they should only be used as a fail-safe to catch novel bug classes and vulnerabilities that slip past security protocols during the development cycle.

NSP: The AppSec space, and the cybersecurity industry as a whole, lives in a constant state of change, with new types of exploits emerging every day. How do you keep up with the ever-evolving landscape?

VL: I’m known to be quiet on social media, but I actually use Twitter a lot – mostly to get informed on the latest security news and understand the security challenges people are currently facing. In other words, I am the classic Twitter lurker. I also read a lot of infosec books, and follow a few well-written security blogs and YouTube channels.

NSP: Are there any online resources (besides your own) that you can recommend to aspiring web hackers, bug hunters, or security researchers?

VL: I am a big fan of reading security books to gain in-depth knowledge about a topic, and then reading blog posts for the latest infosec techniques – Orange Tsai’s blog is one of my favorites. He is a really creative hacker and has been a big inspiration for me ever since I started. Also, Web Security Academy by PortSwigger is a great starting point for web hackers who want to get some hands-on experience.

NSP: Okay, I’ve saved the most pressing question for last. You recently posted on Twitter that you were having a hard time selling your unicycle. This implies that 1) you own a unicycle, and 2) that you know how to ride a unicycle. Do tell.

VL: Happy to announce that I have sold my unicycle to a new loving owner! I learned to unicycle in college because I’ve always thought it’s cool to have an uncommon skill like unicycling. Unicycling is really hard to learn! It took me countless falls and months of practice to finally learn to ride it in a straight line.

But, this experience really boosted my confidence in learning. Like web hacking, learning to unicycle is hard but possible if you put your mind to it and persist. Now when I am trying to learn something difficult, I know I can ‘cause hey – I learned to unicycle! 10/10 would recommend unicycling as a sport. There are few things in this world cooler than a unicycling hacker.