Lately, the Division of Homeland Safety (DHS) recognized the necessity to encourage hands-on studying via cybersecurity competitions to deal with a scarcity of expert cyber defenders. Likewise, in 2019, Govt Order 13870 addressed the necessity to establish, problem, and reward the USA authorities’s finest cybersecurity practitioners and groups throughout offensive and defensive cybersecurity disciplines. Nicely-developed cybersecurity competitions provide a manner for presidency organizations to meet that order.
The Software program Engineering Institute (SEI) has been working with the DHS Cybersecurity & Infrastructure Safety Company (CISA) to deliver distinctive cybersecurity challenges to the federal cyber workforce. This weblog submit highlights the SEI’s expertise creating cybersecurity challenges for the President’s Cup Cybersecurity Competitors and general-purpose pointers and finest practices for creating efficient challenges. It additionally discusses instruments the SEI has developed and made freely out there to assist the event of cybersecurity challenges. The SEI technical report Problem Growth Tips for Cybersecurity Competitions explores these concepts in higher element.
The Function and Worth of Cybersecurity Challenges
Cybersecurity challenges are the guts of cybersecurity competitions. They supply the hands-on duties rivals carry out as a part of the competitors. Cybersecurity challenges can take a number of types and might contain totally different responses, similar to performing actions on one or many digital machines (VM), analyzing varied kinds of information or information, or writing code. A single cybersecurity competitors may comprise a number of totally different challenges.
The objective of those cybersecurity challenges is to show or assess cybersecurity expertise via hands-on workout routines. Consequently, when constructing challenges, builders choose mission-critical work roles and duties from the Nationwide Initiative for Cybersecurity Training Workforce Framework for Cybersecurity (NICE Framework), a doc printed by the Nationwide Institute of Requirements in Expertise (NIST) and the Nationwide Initiative for Cybersecurity Careers and Research (NICCS). The NICE Framework defines 52 work roles and gives detailed details about the precise data, expertise, and talents (KSAs) required to carry out duties in every.
Utilizing the NICE Framework helps builders focus challenges on vital expertise that finest symbolize the cybersecurity workforce. Every problem clearly states which NICE work function and duties it targets. By figuring out the data and expertise every problem targets, rivals can simply concentrate on challenges that tackle their strengths in the course of the competitors and isolate studying alternatives when utilizing challenges for coaching.
Problem Planning
Creating profitable cybersecurity challenges begins with complete planning to find out the extent of problem for every problem, assessing the factors out there for every problem, and figuring out the instruments required to unravel the challenges. When it comes to problem, competitors organizers need members to really feel engaged and challenged. Challenges which can be too simple will make extra superior members lose curiosity, and challenges which can be too exhausting will frustrate rivals. Competitions usually ought to embody challenges which can be appropriate for all ranges—newbie, intermediate, and superior.
Scoring
Factors programs are used to reward rivals for the effort and time they spend fixing every problem. Furthermore, competitors organizers can use factors to find out competitor placement—rivals with greater scores can advance to future rounds, and organizers can acknowledge these with the very best factors as winners. Factors needs to be commensurate with the problem posed by the problem and energy required to unravel it. Level allocation could be a subjective course of, a matter we’ll return to within the part Problem Testing and Assessment part under.
Problem Tooling
Figuring out the instruments required to unravel a problem is a crucial step within the growth course of for 2 causes:
- It ensures that problem builders set up all required instruments within the problem surroundings.
- It’s good follow to supply rivals an inventory instruments out there within the problem surroundings, particularly for competitions by which organizers present rivals with the evaluation surroundings.
Builders needs to be cautious to construct challenges that don’t require using paid or licensed software program. Open supply or free instruments, purposes, and working programs are important as a result of some rivals won’t have entry to sure software program licenses, which might put them at an obstacle and even forestall them from finishing altogether.
Problem Growth
Builders should be well-versed in cybersecurity material to plan progressive approaches to check rivals. Not solely should builders establish the talents the problem will goal and the state of affairs it’ll simulate, they need to additionally develop the technical elements of the problem, implement an automatic and auditable grading system, incorporate variability, and write documentation for each the testers and the rivals.
Pre-Growth Concerns
Builders ought to start by figuring out the work roles and expertise their problem goals to evaluate. By so doing, they will construct extra exact challenges and keep away from together with duties that don’t assess relevant expertise or that check too vast an array of expertise. After they’ve outlined the work function related to a given problem, builders can type a problem thought.
The problem thought contains the technical duties rivals should full and the placement by which the problem state of affairs will happen. All problem duties ought to resemble the duties that professionals undertake as a part of their jobs. Builders are free to be as inventive as they need when constructing the state of affairs. Topical challenges based mostly on real-world cybersecurity occasions provide one other manner so as to add distinctive and inventive situations to challenges.
Technical Part Concerns
The technical elements of problem growth usually contain VM, community, and repair configuration. This configuration ensures the problem surroundings deploys appropriately when rivals try the problem. Growth of technical elements may embody:
- Configuring VMs or companies to include identified vulnerabilities
- Configuring routers, firewalls, companies, and many others., to the state builders need
- Staging assault artifacts or proof all through networks or logs
- Finishing different actions that put together the surroundings for the problem
Builders may also purposefully misconfigure elements of the surroundings if the problem targets figuring out and fixing misconfigurations.
Finest Practices for Creating Challenges
Every problem targets totally different expertise, so there is no such thing as a customary course of for creating a cybersecurity problem. Nonetheless, builders ought to apply the next finest practices:
- Make sure the technical expertise assessed by the problem are relevant in the true world.
- Make sure the instruments required to unravel the problem are free to make use of and out there to the rivals.
- Make an inventory of the instruments out there to rivals within the hosted surroundings.
- Guarantee challenges don’t drive rivals down a single resolution path. Rivals ought to have the ability to resolve challenges in any practical method.
- Take away pointless hints or shortcuts from the problem, together with command historical past, looking information, and different information that would enable rivals a shortcut to fixing the problem.
Problem Grading
Basically, builders ought to automate grading via an authoritative server that receives solutions from the rivals and determines what number of factors to award the submission. The submission system ought to usually ignore variations in capitalization, white area, particular characters, and different variations which can be finally irrelevant to correctness. Doing so ensures rivals aren’t unfairly penalized for immaterial errors.
Ignoring these errors may appear to contradict an evaluation of operational readiness in circumstances the place precise precision is required. Nonetheless, cybersecurity competitions have targets and concerns past evaluating operational proficiency, similar to guaranteeing a good competitors and inspiring broad participation.
Builders could apply totally different grading strategies, together with the next:
- Token discovery. In token-discovery grading, rivals should discover a string or token that follows an outlined format (these tokens may also be referred to as “flags”). Builders can place the token in any a part of the problem the place the competitor will discover it by finishing the problem duties.
- Query-and-answer issues. For question-and-answer issues, the competitor should discover the proper reply to a number of questions by performing problem duties. The solutions to the problem questions can take a number of types, similar to coming into file paths, IP addresses, hostnames, usernames, or different fields and codecs which can be clearly outlined.
- Surroundings verification. In surroundings verification grading, the system grades rivals based mostly on adjustments they make to the problem surroundings. Challenges can activity rivals with fixing a misconfiguration, mitigating a vulnerability, attacking a service, or another exercise the place success might be measured dynamically. When the grading system verifies adjustments to the surroundings state, it gives rivals with a hit token.
Problem Variation
Builders ought to embody some degree of variation between totally different deployments of a problem to permit for various appropriate solutions to the identical problem. Doing so is vital for 2 causes. First, it helps promote a good competitors by discouraging rivals from sharing solutions. Second, it permits competitors organizers to reuse challenges with out dropping instructional worth. Challenges that may be accomplished quite a few occasions with out leading to the identical reply allow rivals to be taught and hone their expertise via repeated follow of the identical problem.
Builders can introduce variation into challenges in a number of methods, relying on the kind of grading that they use:
- Token-based variation. Challenges utilizing token-discovery or environment-verification grading can randomly generate distinctive tokens for every competitor when the problem is deployed. Builders can insert dynamically generated submission tokens into the problem surroundings (e.g., inserting guestinfo variables into VMs), they usually can copy them to the areas the place they count on rivals to obtain the problem solutions.
- Query-and-answer variation. In question-and-answer challenges, builders can introduce variation by configuring totally different solutions to the identical questions or by asking totally different questions.
Problem Documentation
The 2 key paperwork builders should create in assist of their problem are the problem information and the answer information.
The problem information, which is seen to the rivals, gives a brief description of the problem, the talents and duties the problem assesses, the state of affairs and any background info that’s required to know the surroundings, machine credentials, and the submission space or areas.
The problem doc ought to describe the state of affairs in a manner that rivals can simply comply with and perceive. The problem state of affairs and background info ought to keep away from logical leaps and the problem degree shouldn’t hinge on info internationally ignored of the information.
The answer information gives a walk-through of 1 method to full the problem. Throughout testing, builders use the answer information to make sure the problem might be solved. Builders may launch the answer information to the general public after the conclusion of the competitors to function a group studying useful resource.
The supposed viewers for this information is the overall cybersecurity group. Consequently, builders ought to assume the reader is conversant in primary IT and cybersecurity expertise, however just isn’t an knowledgeable within the area. Screenshots and different pictures are useful additions to those guides.
Problem Testing and Assessment
After builders construct a problem, it ought to undergo a number of rounds of testing and assessment. Builders check challenges to make sure high quality, they usually assessment them to estimate the problem’s problem.
Builders ought to carry out an preliminary spherical of testing to catch any errors that come up in the course of the problem deployment and initialization course of. They need to additionally make sure that rivals can absolutely resolve the problem in no less than a technique. A second spherical of testing needs to be carried out by certified technical workers unfamiliar with the problem. Testers needs to be inspired to try fixing the problem on their very own however could also be offered the developer’s resolution information for assist.
The testers ought to guarantee every problem meets the next high quality assurance standards:
- The problem deploys as anticipated and with out errors.
- The problem VMs are accessible.
- The problem is solvable.
- There are not any unintentional shortcuts to fixing the problem.
- Problem directions and questions are correctly formatted and provides a transparent indication of what rivals should do.
Of their assessment of the problem, testers ought to take notes in regards to the content material, together with estimates of problem and size of time it will take rivals to unravel. After testers full their assessment, competitors organizers can study the problem assessments and evaluate every problem with others. This comparability ensures that simpler challenges stay in earlier rounds and are value fewer factors than challenges judged as tougher.
When deciding problem level allocations, organizers can use a base or customary rating allotment as a place to begin (e.g., all challenges are value 1,000 factors at the start of the method). Organizers can then enhance or lower level allocations based mostly on the out there problem information, maintaining in thoughts that the primary objective is for the variety of factors they allocate to a problem to instantly correspond with the trouble required for fixing it. Level allocations ought to take into account each the problem and the time it takes to unravel the problem.
SEI Open Supply Functions for Cybersecurity Problem Competitions
Builders can use a number of open supply purposes to develop challenges and to orchestrate cybersecurity competitions. The SEI has developed the next two purposes for operating cybersecurity competitions:
- TopoMojo is an open supply lab builder and participant software that builders can use to develop cybersecurity challenges. It gives digital workspaces by which problem growth can happen. The workspaces enable builders so as to add VMs, digital networks, and another sources which can be required for creating or fixing a single problem.
- Gameboard is an open supply software that organizers can use for orchestrating cybersecurity competitions. It allows organizers to create competitions that may both be group or particular person based mostly and that encompass both single or a number of rounds. Challenges are organized into rounds and rivals try to unravel as many challenges as they will to maximise their rating. Gameboard makes use of the TopoMojo API to deploy the rivals’ sport area for every problem.
Gameboard additionally serves because the authoritative location for rivals to submit solutions or tokens. Furthermore, as a part of dealing with reply and token submissions, Gameboard has logging, brute drive protections, and different options to make sure the integrity of the competitors.
Determine 1 exhibits how the TopoMojo and Gameboard purposes work together. Builders use TopoMojo workspaces to develop challenges. Rivals then use Gameboard to deploy and in- teract with challenges. When a participant deploys a problem, Gameboard will work together with the To- poMojo API to request a brand new sport area for the competitor. TopoMojo creates and returns the participant’s problem sport area.
Finest Practices Assist Higher Cybersecurity Competitions
The event practices we now have highlighted on this submit are the results of the SEI’s expertise creating cybersecurity challenges for the President’s Cup Cybersecurity Competitors. Cybersecurity competitions present a enjoyable and fascinating method to train technical expertise, establish and acknowledge cybersecurity expertise, and interact college students and professionals within the area. They’ll additionally function schooling and coaching alternatives. With the USA authorities, and the nation as a complete, dealing with a big scarcity within the cybersecurity workforce, cybersecurity competitions play an vital function in creating and increasing the workforce pipeline.
There isn’t any single method to run a contest, and there’s no one method to develop cybersecurity challenges. Nonetheless, these finest practices might help builders make sure the challenges they create are efficient and interesting. Problem growth is the one most vital and time-consuming facet of operating a cybersecurity competitors. It requires meticulous planning, technical growth, and a rigorous quality-assurance course of. In our expertise, these practices guarantee efficiently executed competitions and enduring, hands-on cybersecurity belongings that competitors organizers and others can reuse many occasions over.
If you need to be taught extra in regards to the work we do to strengthen the cybersecurity workforce and the instruments we now have developed to assist this mission, contact us at information@sei.cmu.edu.
