SEAL DRILLS: Attack Simulations to Improve Web3 Security
Kelsie Nabben and Primavera De Filippi
25 October, 2023
SEAL DRILL TIME
A message appeared in the Telegram chat application group chat: “Welcome to the SEAL Chaos Team's Simulation War Room.
You're now part of a real-time exercise set in the Yearn ecosystem. Your mission involves diagnosing and responding to a simulated incident as if it were real and the stakes were high. In other words, do everything you would normally do, including triaging, patching, and (simulating) communicating with the public. After the simulation is complete, you'll have a chance to go over how you did and where room for improvement might be found…
Good luck. The simulation is about to begin.”
This message marked the second attack simulation (or “SEAL Drills”) on a major Decentralised Finance (DeFi) protocol by the “SEAL Chaos Team”. More broadly, it marks an effort to shift the security landscape of the cryptocurrency ecosystem and presents insights into how formal and informal accountability mechanisms are developed in open-source, permissionless infrastructure.
Content Overview
· Introduction
· SEAL x Yearn: Attack Simulation
· Debrief
· Insights
· Conclusion
Introduction
To be clear, Web3 is not safe (yet?) for retail consumers and “normies” (normal people). Blockchain data platform ‘Chainanalysis’ reports that 2022 was the “biggest year ever for crypto hacking”, with $3.8 billion stolen. Decentralised Finance (DeFi) protocols, some of the most automated and transparent in terms of activity occurring on-chain, are also the most vulnerable at 82.1% of all cryptocurrency stolen. In what doesn’t look good for crypto’s reputation in the eyes of regulators, North Korea-linked hackers are breaking theft records. While it may be worth $10,000 – $100,000 for a professional security firm to audit the software code of a protocol's smart contracts, these prices can be prohibitive to experimental, un-funded start-up projects. A list of some of the largest hacks in crypto history, including whether or not they were audited (many were), can be found at rekt.news. Crypto regulatory enforcement is also increasing. The need for greater accountability within blockchain governance, in conjunction with appropriate regulatory measures, is clear.
Accountability refers to the mechanisms and processes through which individuals, organisations, or institutions are held responsible for their decisions and actions in relation to others. This is typically achieved through a combination of social norms and institutional rules for monitoring, enforcement, and potential sanctions. In legal and corporate governance, accountability is achieved between two actors when one is obliged to provide an account of its action, and the other has the power and authority to impose consequences as a result.
In technical modes of governance, “accountability protocols” refers to a design strategy to ensure that sufficient evidence of participation is provided in the rules of the protocol. Furthermore, scholars and practitioners argue that ‘ethical considerations need to be embedded into the software development process’. This post builds upon these conceptualisations to explore the accountability mechanisms occurring in blockchain governance. Specifically, we investigate “off-chain” (off the blockchain) accountability mechanisms that are formalised and verifiable to explore the extent to which blockchain communities can effectively self-regulate to achieve certain policy objectives.
Anthropologists have shown significant interest in the politics and practices of hacking as a social phenomenon, in relation to the production of open-source software and freedom. By employing digital ethnographic methods, we observe the case study of the SEAL Chaos Team. The Chaos Team is one work stream of SEAL, a grant funded not-for-profit organisation that provides a platform for the security community to be able to collaborate on and launch ambitious and impactful security initiatives that would otherwise be difficult to execute. ‘Whitehat’ or ‘ethical’ hacking is a process in which a hacker simulates an attack on a network or computer system to evaluate its security, with consent from that system’s owners. In other words, it is engaging in the tools, processes, and methods to find vulnerabilities and develop defences before the bad guys do. White rescues can also mean engaging in the exploit of a computer system with the intent to rescue funds ahead of an attacker. As a handful of volunteers, they engage in security enhancing initiatives, including attack simulations to help protocol teams prepare their processes and coordination for real events, the incident help desk (SEAL-911), the legal Safe Harbor Agreement for Whitehats, incident playbooks, and threat intel sharing. Given that the team works on the most sensitive security issues and also with many anons (anonymous people), we were particularly sensitive around our ‘ethnographic op-sec’ (operational security), including not interrupting security exercises and how data was shared and stored.
SEAL Chaos Team’s process includes engagement with a protocol team, background open-source research to understand the network services associated with the target protocol, a tabletop exercise to plan the simulation, and finally a window of time in which the simulated attack with occur (in a simulated operating environment) to test the protocol team’s response. In an interview with Isaac Patka, Founder of Shield3 security firm and leader of SEAL’s attack simulation (Chaos) team, he pointed out that in the early days of smart contract enabled public blockchains, there was little distinction between whitehats and ‘blackhats’ (bad guys). Many projects released protocols without prohibitively expensive professional security audits, and it was expected that hackers would identify and exploit bugs, and often be rewarded with bug bounties for this. It was like “let’s all try to wreck protocols as much as possible and see what survives.” The benefits of attack simulations can include improved security through identifying vulnerabilities, proactive defence, knowledge sharing, and training. With the Yearn attack simulation exercise coming up, the SEAL Chaos Team was prepared to “take this Yearn Plane ✈️ down!”.
🔥🔥🔥 SEAL x Yearn: Attack Simulation 🔥🔥🔥
It was 12:01PM EST on September 20, 2023 when Isaac Patka posted in the War Room that the simulation had begun.
The players on the ‘blue team’ (a.k.a. good guys) included a bunch of core contributors to the DeFi protocol ‘Yearn’, including Storm0x, FP, Poolpi Tako, wavey, dudesahn, Val, banteg, Spalen, and Rare Weasel.
My access to observe the simulation had been granted by the red team attackers; renowned Web3 whitehat Samczsun, and the coordinator of the simulation, Isaac Patka. 80 developer hours of work had gone into preparing the scenario, tools, notification channel, and ‘meme’ joke ads. Now, at the very beginning of the possible window of time for the exercise and half way through an early dinner in my time zone, the simulation had begun.
Dudesahn was the first to respond, saying “Lmao wait it’s starting at the earliest possible time?”. “I’m eating lunch so will be a bit slow for a few mins 😂”.
As the team spun up a voice channel to be able to talk and type at the same time, we were viewing a simulation of a number of Yearn vaults, and all had access to a second Telegram channel that was pinging us with incessant notifications with price updates from the open source block explorer ‘BlockScout’, Yearn vault liquidity reports, and an Arb Profit Report 😈 📈.
Within moments, the Yearn team started to diagnose the situation, stating “Okay so clearly ETH price is being manipulated”.
Dudesahn asks, “Where is this eth price being reported from?”.
Poolpi tako replies, “Compound oracle”.
Dudesahn says, “OK. So someone is attacking compound and it’s basically fucking us over”. The strategy of the attack was to manipulate the Compound oracle price, borrow USDC against that, and extract funds from the protocol.
By 12.03PM Storm0x had posted an urgent status report indicating an issue with Compound, mentioning the affected vaults and suggesting some strategies to mitigate the damage before halting the attack. “GOAL try to minimize damage impact, and assess if we need to take immediate action”.
Pulling out an emergency procedure card, Val shares a link, and Storm0x stresses the importance of setting the deposit limit to zero.
After some negotiation around the correct multi-signature wallet-based governance process they could follow with the participants present and without needing to wake anyone else in a different time zone up, the team agreed that they had quorum to sign a transaction.
Before signing, the Yearn team priority was to simulate the steps on a fork in a local operating environment to confirm the strategy would allow them to safely exit.
By 34 minutes into the simulation, ****the team was clear on the transactions affected in the vault, that it was an issue with Compound, and the need to minimize damage and assess immediate actions.
Looking at their manuals, “would emergency shut down do this?”. Poolpi stated “we have never used it in the past…”.
Val asked FP, “would you recommend collat target = 0 and tend, or emegencyShutdown?”
Storm0x emphasised, “can we get a simulation in local rpc to check that the steps are safe and we can delever?”, referring to the ‘unwidening’ of strategy so it withdraws from the underlying protocols and returns funds it ‘borrowed’ from the vault.
With that, Dudesahn commentated on the general sentiment, “we’re not fucked yet but we will be if we fail to withdraw”.
At 37 minutes into the exercise, not only was the vulnerability identified but a strategy was in place. After testing the command, the team decided they have quorum and were happy to run this command to initiate an emergency exit mode, a “set of actions that seek to unwind and divest funds back to the Vault as quickly and smoothly as possible, with as minimal losses as possible”.
The total time to safe harbour assets and declare it was 56 minutes, with Storm0x announcing and update in the chat “FUNDS ARE SAFU!! we are out from compound”.
Debrief
Upon debrief, the SEAL red team declared that Yearn’s effort was “impressive”. This was particularly regarding their pre-prepared emergency manuals, although the exercise itself warranted a review and update of these.
The Yearn team responded rapidly to identify the root cause of the attack was a dependency on a price oracle and not an issue with their code. They did acknowledge they could have done some things better, particularly the time to action the emergency exit mode to limit new deposits. In a normal scenario with all of the standard tooling available, the response would have been even faster.
Part of the immediate debrief procedure included consideration of if and how such a situation would be publicly communicated and reported on, including timestamps from the event. In the case of Yearn, a public disclosure policy is in place, which they have had to enact multiple times in the past.
Although very serious, the simulation also featured compulsory memes “for the lulz”. The fork (copy of software) of Blockscout that Isaac had included joke advertisements to replace the Paypal ones, created by SEAL Chaos team member David Desjardins, including “seals 4 sale”.
Insights
What this case study reveals is greater insight into how accountability mechanisms form in the digital, private governance settings of public blockchain infrastructure. SEAL is offering a voluntary, centralised service in a decentralised, permissionless ecosystem to help improve coordination among Whitehat hackers, as well as security outcomes for protocols. While attack simulations are one work stream of SEAL, others include the SEAL 911 emergency notification channel, and plans for a Request For Comment for a legal safe harbour agreement between protocol communities and Whitehat hackers, that is ratified on-chain by each protocol.
Of course, any coordination mechanism, whether on-chain or off-the-blockchain, addresses some vulnerabilities while potentially introducing others. For SEAL, this may include social engineering attempts to pose as a Whitehat or game the 911 channel, or lack of awareness of the state of a protocol community agreement with whitehat’s (whether ratified or revoked) in an emergency response scenario.
SEAL ran an ‘attack sim’ with Compound a couple of months ago, and already have a waitlist of high profile DeFi protocols to follow the Yearn exercise. According to Isaac, simulations push teams to up their detection game and relentlessly "mitigate, then resolve” security related matters. SEAL adds social resiliency to protocols. Attack simulations not only help a protocol to better prepare for unforeseen scenarios but also encourage teams to consider the issues that come with composable infrastructure, the cascade effects of how contagion in one protocol can spread, and how to communicate across protocol teams to coordinate responses (either in a simulation, or in a live situation via the (optional and non-exclusive) SEAL 911 disclosure service. After the Yearn simulation, Isaac offered in the chat, “If anyone wants to contribute to future attack scenarios and doomsday scenarios, feel free”.
Conclusion
This piece is part of a broader research project on ‘accountability in blockchain governance’ that explores the relationship between social accountability and technological guarantees. In particular, it explores the extent to which blockchain communities can ‘self-regulate’, and how they utilise technical and social regulatory modalities to do so. In the case of the SEAL Chaos Team, this piece finds that blockchain governance embraces a form of ‘co-regulation’ between public protocol communities and private ‘whitehat hackers’. Wherever possible, social norms of accountability are embedded into the transparent, technical rules of the protocol. What this case study demonstrates is the possibility for novel forms of co-regulation that are particularly suited to open-source, public, blockchain systems. Through public accountability mechanisms for private actors that align protocols and communities with regulatory requirements, and vice versa, Web3 may have a chance at improving security, whilst protecting some of its core values of permissionlessness and polycentricity.
References
Bella, G., and Paulson. L.C. (2006). Accountability protocols: Formalized and verified. ACM Trans. Inf. Syst. Secur. 9, 2 (May 2006), 138–161. https://doi.org/10.1145/1151414.1151416.
Bovens, M. (2007). Analysing and Assessing Accountability: A Conceptual Framework. European Law Journal, 13(4), 447-468. https://onlinelibrary.wiley.com/doi/full/10.1111/j.1468-0386.2007.00378.x.
Chainanalysis Team. (2023, February 1). 2022 Biggest Year Ever For Crypto Hacking. Chainalysis. https://www.chainalysis.com/blog/2022-biggest-year-ever-for-crypto-hacking/
Coleman, E. G. (2013). Coding freedom: The ethics and aesthetics of hacking. Princeton University Press.
Finck, Michèle, Digital Co-Regulation: Designing a Supranational Legal Framework for the Platform Economy (June 20, 2017). European Law Review (2018 Forthcoming), LSE Legal Studies Working Paper No. 15/2017, Available at SSRN: https://ssrn.com/abstract=2990043.
Kang, J. J. (2022, November 30). North Korea’s nuclear program is funded by stolen cryptocurrency. Could it collapse now that FTX has? The Conversation. http://theconversation.com/north-koreas-nuclear-program-is-funded-by-stolen-cryptocurrency-could-it-collapse-now-that-ftx-has-195559.
Malwa, S. (2020, September 29). Hackers Drain $15 Million From ‘Unreleased’ Yearn Finance Project. Decrypt. https://decrypt.co/43203/hackers-drain-15-million-from-unreleased-yearn-finance-project.
Nabben, K. (2023). Blockchain Governance: Accountability in Decentralised Technology Communities [Substack newsletter]. Kelsie - on the Cataclysmia of Digital Infrastructure. https://kelsienabben.substack.com/p/blockchain-governance-accountability.
Nabben, K., Burrrata, Michael Zargham, and Jessica Zartler. “DAO Vulnerabilities: A map of Lido governance risks and opportunities”. BlockScience Medium (blog). Available at: https://medium.com/block-science/dao-vulnerabilities-a-map-of-lido-governance-risks-opportunities-92bc6384ff68.
Rekt. (2023). Rekt.News, Leaderboard. Available online: https://rekt.news/leaderboard/. Accessed September, 2023.
Samczsun (@samczsun) / X. (2023, September 22). X (Formerly Twitter). https://twitter.com/samczsun.
Solidus Labs. 2023. US Regulators Broke Crypto Enforcement Records in 2022. Retrieved September 25, 2023, from https://www.soliduslabs.com/research/2023-crypto-enforcement-trends.
Yearn. (n.d.). Yearn-security/disclosures at master · yearn/yearn-security. GitHub. Retrieved September 26, 2023, from https://github.com/yearn/yearn-security/tree/master/disclosures.
Yearn Finance. (2023). Governance and Operations. yearn.fi. https://docs.yearn.fi/contributing/governance/governance-and-operations.
Zuber, N., Gogoll, J., Kacianka, S., Pretschner, A., & Nida-Rümelin, J. (2022). Empowered and embedded: Ethics and agile processes. Humanities and Social Sciences Communications, 9(1), Article 1. https://doi.org/10.1057/s41599-022-01206-4.