Projekt CKMT
Table of contents

Nuclei Logo

Introduction

In the realm of cybersecurity, efficient and scalable vulnerability scanning is paramount. Nuclei, developed by ProjectDiscovery, stands out as a powerful, fast, and customizable vulnerability scanner that uses simple YAML-based templates. It allows security professionals to perform targeted scans across various protocols like HTTP, DNS, TCP, and more.

However, when it comes to mass scanning—running Nuclei against hundreds or thousands of targets—there are challenges related to resource utilization, performance optimization, and accurate results. This post delves into strategies and configurations to optimize Nuclei for mass scanning, ensuring efficient use of resources and reliable outcomes.

Understanding Mass Scanning Challenges

Mass scanning involves running vulnerability scans on a large number of targets, which can introduce several challenges:

  • Resource Overutilization: High concurrency can lead to excessive CPU and memory usage, causing system instability or crashes.
  • Network Bottlenecks: Sending too many requests simultaneously can overwhelm network bandwidth or the target servers.
  • False Negatives/Positives: Improper configurations may result in missed vulnerabilities or incorrect detections.
  • Management Complexity: Handling large target lists requires careful planning to ensure efficient and organized scanning.

Resource Utilization in Nuclei

Nuclei is designed for high concurrency and speed, making extensive use of network I/O operations. Understanding how Nuclei consumes resources is crucial for optimization:

  • Concurrency and Memory Usage: There’s a direct correlation between the concurrency level and memory consumption. Higher concurrency increases resource usage.
  • Template Complexity: Templates with multiple payloads or complex logic require more processing power and memory.
  • Response Size: Large responses from targets consume more memory, especially if many are processed in parallel.

Key Flags and Options for Optimization

Properly configuring Nuclei’s flags and options can significantly improve performance and resource utilization during mass scanning.

Concurrency Settings

-c or -concurrency

  • Description: Controls the number of templates to run in parallel.

  • Default: 25

  • Usage:

    nuclei -l targets.txt -t templates/ -c 50

  • Recommendation: Adjust based on your system’s CPU and memory capacity. Increasing concurrency speeds up scanning but uses more resources.

-bs or -bulk-size

  • Description: Controls the number of targets to scan in parallel per template.

  • Default: 25

  • Usage:

    nuclei -l targets.txt -t templates/ -bs 50

  • Recommendation: Lower the value if experiencing high memory usage or network saturation.

Scan Strategy

-ss or -scan-strategy

  • Description: Determines the order in which templates and targets are processed.

  • Options:

    • auto or template-spray (default): Scans all targets with one template before moving to the next template.
    • host-spray: Scans all templates against one target before moving to the next target.

  • Usage:

    nuclei -l targets.txt -t templates/ -ss host-spray

  • Recommendation: Use host-spray to reduce memory consumption, as it keeps fewer templates in memory at a time.

Rate Limiting

-rl or -rate-limit

  • Description: Sets a global rate limit for the number of requests per second.

  • Default: 150

  • Usage:

    nuclei -l targets.txt -t templates/ -rl 100

  • Recommendation: Adjust to prevent overwhelming your network or the target servers.

-rlm or -rate-limit-minute

  • Description: Sets rate limit in requests per minute.

  • Usage:

    nuclei -l targets.txt -t templates/ -rlm 6000

  • Note: Mutually exclusive with -rl.

Response Handling

-rsr or -response-size-read

  • Description: Limits the maximum size of HTTP responses read.

  • Default: 4MB

  • Usage:

    nuclei -l targets.txt -t templates/ -rsr 1048576

  • Recommendation: Keep default unless you expect larger responses. Reducing can save memory.

Timeouts and Retries

-timeout

  • Description: Sets the maximum time to wait for a response (in seconds).

  • Default: 10

  • Usage:

    nuclei -l targets.txt -t templates/ -timeout 20

  • Recommendation: Increase if scanning targets with high latency.

-retries

  • Description: Number of times to retry a failed request.

  • Default: 1

  • Usage:

    nuclei -l targets.txt -t templates/ -retries 2

  • Recommendation: Increase for unstable networks or unreliable targets.

Advanced Concurrency Controls

Headless Templates Concurrency

  • -headc and -hbs

  • Description: Control concurrency for headless browser operations.

  • Usage:

    nuclei -l targets.txt -t templates/ -headless -headc 5 -hbs 5

  • Recommendation: Keep these values low due to the high resource consumption of headless browsers.

JavaScript Execution Concurrency

-jsc or -js-concurrency

  • Description: Controls concurrency for JavaScript runtime executions.

  • Default: 120

  • Usage:

    nuclei -l targets.txt -t templates/ -jsc 60

  • Recommendation: Adjust based on resource availability.

Payload Concurrency

-pc or -payload-concurrency

  • Description: Manages concurrency for templates using payloads.

  • Default: 25

  • Usage:

    nuclei -l targets.txt -t templates/ -pc 10

  • Recommendation: Lower if experiencing high memory usage.

Practical Examples

Example 1: Basic Mass Scan Optimization

Scenario: Scanning 1,000 targets using HTTP templates without overwhelming the system.

Command:

nuclei -l targets.txt -t http/ -ss host-spray -c 50 -bs 50 -rl 100 -timeout 15

Explanation:

  • -l targets.txt: List of 1,000 targets.
  • -t http/: Using all HTTP templates.
  • -ss host-spray: Scans all templates against one target before moving on.
  • -c 50: Runs 50 templates in parallel.
  • -bs 50: Processes 50 targets in parallel per template.
  • -rl 100: Limits to 100 requests per second.
  • -timeout 15: Increases timeout to 15 seconds.

Example 2: Handling Resource Constraints

Scenario: Scanning on a machine with limited memory (4 GB RAM).

Command:

nuclei -l targets.txt -t templates/ -ss host-spray -c 20 -bs 20 -rl 50 -rsr 1048576

Explanation:

  • -c 20: Lower concurrency to reduce CPU and memory usage.
  • -bs 20: Lower bulk size to limit the number of targets processed in parallel.
  • -rl 50: Reduces the rate limit to prevent network saturation.
  • -rsr 1048576: Limits response size read to 1 MB.

Example 3: Optimizing Headless Scans

Scenario: Running headless templates that require browser automation.

Command:

nuclei -l targets.txt -t headless/ -headless -headc 5 -hbs 5 -timeout 30

Explanation:

  • -headless: Enables headless browser support.
  • -headc 5: Limits headless template concurrency to 5.
  • -hbs 5: Limits headless bulk size to 5 targets at a time.
  • -timeout 30: Increases timeout to account for longer page loads.

Example 4: Scanning with Limited RAM (1 GB)

Scenario: Scanning 500 targets on a machine with 1 GB RAM.

Challenges:

  • Limited memory requires strict control over concurrency and resource usage.
  • Need to prevent out-of-memory (OOM) errors.

Command:

nuclei -l targets.txt -t templates/ -ss host-spray -c 5 -bs 5 -rl 20 -rsr 524288 -timeout 20

Explanation:

  • -c 5: Reduces the number of templates run in parallel to 5.
  • -bs 5: Processes 5 targets in parallel per template.
  • -rl 20: Limits requests to 20 per second.
  • -rsr 524288: Limits response size read to 512 KB.
  • -timeout 20: Sets a reasonable timeout to accommodate slower targets.

Additional Tips:

  • Batch Targets: Split targets.txt into smaller files (e.g., 100 targets each) and run sequentially.
  • Disable Unnecessary Features: Avoid using headless templates or complex payloads unless necessary.
  • Monitor Memory Usage: Use tools like top or htop to keep an eye on RAM usage during the scan.

Example 5: Scanning with Moderate RAM (2 GB)

Scenario: Scanning 1,000 targets on a machine with 2 GB RAM.

Command:

nuclei -l targets.txt -t templates/ -ss host-spray -c 10 -bs 10 -rl 40 -rsr 1048576 -timeout 20

Explanation:

  • -c 10: Runs 10 templates in parallel.
  • -bs 10: Processes 10 targets in parallel per template.
  • -rl 40: Limits requests to 40 per second.
  • -rsr 1048576: Limits response size read to 1 MB.
  • -timeout 20: Sets timeout to handle variable response times.

Optimization Strategies:

  • Increase Concurrency Gradually: Test with -c 10 and -bs 10; if resources allow, incrementally increase.
  • Adjust Rate Limiting: Ensure the network bandwidth can handle 40 RPS without packet loss.
  • Batch Processing: Consider processing targets in batches if resource constraints persist.

Example 6: Scanning on High-Latency Networks

Scenario: Scanning targets where network latency is high, possibly leading to timeouts.

Command:

nuclei -l targets.txt -t templates/ -timeout 30 -retries 3 -c 15 -bs 15 -rl 50

Explanation:

  • -timeout 30: Increases timeout to 30 seconds to accommodate latency.
  • -retries 3: Retries failed requests up to 3 times.
  • -c 15, -bs 15: Moderate concurrency to balance resource usage.
  • -rl 50: Limits requests to 50 per second to prevent network saturation.

Considerations:

  • Monitor for Timeouts: Adjust -timeout and -retries based on observed failures.
  • Adjust Concurrency: High latency can tie up connections; reducing concurrency can prevent resource exhaustion.

Advanced Optimization Techniques

Example 7: Scanning with Very Limited RAM (1 GB)

Scenario: Scanning 200 targets on a machine with 1 GB RAM and limited CPU resources.

Challenges:

  • Extreme memory constraints require minimal concurrency.
  • Risk of OOM errors if resource limits are exceeded.

Command:

nuclei -l targets.txt -t templates/ -ss host-spray -c 2 -bs 2 -rl 10 -rsr 262144 -timeout 30 -mhe 5

Explanation:

  • -c 2: Runs only 2 templates in parallel.
  • -bs 2: Processes 2 targets in parallel per template.
  • -rl 10: Limits requests to 10 per second.
  • -rsr 262144: Limits response size read to 256 KB.
  • -timeout 30: Increases timeout to handle slow responses.
  • -mhe 5: Skips hosts after 5 errors to conserve resources.

Additional Recommendations:

  • Disable DNS Resolution: If not necessary, avoid DNS lookups which consume resources.
  • Use Minimal Templates: Only run essential templates to reduce processing load.
  • Monitor Swap Usage: Ensure the system is not swapping excessively, which can degrade performance.

Example 8: Scanning with Limited RAM but Higher CPU (2 GB RAM)

Scenario: Scanning 500 targets on a machine with 2 GB RAM and a quad-core CPU.

Command:

nuclei -l targets.txt -t templates/ -ss host-spray -c 8 -bs 8 -rl 30 -rsr 524288 -timeout 25 -retries 2

Explanation:

  • -c 8: Utilizes the quad-core CPU by running 8 templates in parallel.
  • -bs 8: Processes 8 targets in parallel per template.
  • -rl 30: Limits requests to 30 per second to balance CPU and network usage.
  • -rsr 524288: Limits response size read to 512 KB.
  • -timeout 25: Adequate timeout for moderately slow targets.
  • -retries 2: Retries failed requests to improve coverage.

Optimization Tips:

  • Leverage CPU Cores: Adjust -c and -bs to utilize available CPU cores effectively.
  • Adjust Rate Limits: Ensure the rate limit does not exceed network capabilities.
  • Selective Templates: Prioritize templates that are most relevant to your targets to save resources.

Example 9: Scanning with Low Bandwidth

Scenario: Scanning 300 targets on a machine with sufficient RAM but limited network bandwidth.

Command:

nuclei -l targets.txt -t templates/ -ss host-spray -c 15 -bs 5 -rl 10 -timeout 20

Explanation:

  • -c 15: Runs 15 templates in parallel to utilize CPU.
  • -bs 5: Processes only 5 targets in parallel to limit bandwidth usage.
  • -rl 10: Limits requests to 10 per second to prevent saturating the network.
  • -timeout 20: Allows for slower network responses.

Strategies:

  • Decrease Bulk Size: Reducing -bs helps manage network throughput.
  • Adjust Rate Limit: Lower -rl to prevent network congestion.
  • Monitor Network Usage: Use tools like iftop or nload to observe real-time bandwidth usage.

Example 10: Scanning with Unstable Network Connection

Scenario: Scanning 100 targets over a flaky network connection.

Command:

nuclei -l targets.txt -t templates/ -timeout 40 -retries 5 -c 5 -bs 5 -rl 10 -mhe 3

Explanation:

  • -timeout 40: Increases timeout to accommodate network instability.
  • -retries 5: Retries failed requests multiple times.
  • -c 5, -bs 5: Keeps concurrency low to reduce the impact of connection drops.
  • -rl 10: Limits requests per second to avoid overwhelming the network.
  • -mhe 3: Skips hosts after 3 errors to prevent wasting resources on unreachable targets.

Considerations:

  • Increase Retries and Timeouts: Helps to mitigate temporary network issues.
  • Limit Concurrency: Reduces the number of simultaneous connections.
  • Use System Resolvers: If DNS resolution is problematic, consider using -sr to fallback on system DNS.

Advanced Optimization Techniques for Nuclei Scanning

To maximize the speed of your Nuclei scans while effectively managing system resource usage, consider implementing advanced optimization techniques. This section provides additional examples and a comprehensive table to help you configure Nuclei for the best performance based on your system’s resources.

Understanding Speed vs. Resource Usage

Optimizing for speed often means increasing concurrency, bulk size, and rate limits. However, these adjustments can lead to higher CPU, memory, and network resource consumption. The key is to find a balance that maximizes speed without overwhelming your system.

Additional Examples for Speed Optimization

Example 11: Maximizing Speed on High-Performance Systems

Scenario: Scanning 10,000 targets on a machine with 16 GB RAM and 8-core CPU.

Command:

nuclei -l targets.txt -t templates/ -c 100 -bs 100 -rl 1000 -timeout 10 -retries 1 -ss auto

Explanation:

  • -c 100: Runs 100 templates in parallel to utilize multiple CPU cores.
  • -bs 100: Processes 100 targets in parallel per template.
  • -rl 1000: Allows up to 1000 requests per second.
  • -timeout 10: Sets a lower timeout since high-performance systems and networks can handle quick responses.
  • -retries 1: Minimizes retries to save time.
  • -ss auto: Uses the default scan strategy optimized for speed.

Tips:

  • Monitor System Resources: Ensure the system can handle the increased load without throttling or crashing.
  • Optimize Network Bandwidth: Make sure the network bandwidth supports the high rate of requests.

Example 12: Speed Optimization with Multiple Templates

Scenario: Running a targeted scan using multiple templates against a single target.

Command:

nuclei -u example.com -t cves/ -t exposures/ -t misconfigurations/ -c 200 -rl 500

Explanation:

  • -u example.com: Targeting a single host.
  • -t cves/, -t exposures/, -t misconfigurations/: Specifying multiple template directories.
  • -c 200: High concurrency since it’s only one target.
  • -rl 500: High rate limit to speed up scanning.

Considerations:

  • Single Target Scans: When scanning a single target, you can increase concurrency (-c) significantly as there will be fewer resource constraints.
  • Template Selection: Running only relevant templates can reduce scanning time.

Example 13: Optimizing for Best Speed with Balanced Resource Usage

Scenario: Scanning 5,000 targets with moderate system resources (8 GB RAM, Quad-core CPU).

Command:

nuclei -l targets.txt -t critical/ -t high/ -ss auto -c 50 -bs 50 -rl 500 -timeout 15 -retries 1

Explanation:

  • -t critical/, -t high/: Focus on high-severity templates to quickly identify significant vulnerabilities.
  • -ss auto: Allows Nuclei to choose the best scan strategy dynamically.
  • -c 50: Moderate concurrency to balance CPU usage.
  • -bs 50: Reasonable bulk size to manage memory consumption.
  • -rl 500: High rate limit to increase scanning speed.
  • -timeout 15, -retries 1: Optimized for faster completion.

The following table provides guidelines on how to configure Nuclei parameters based on your system’s RAM and CPU cores.

Resource Level RAM CPU Cores Concurrency (-c) Bulk Size (-bs) Rate Limit (-rl) Timeout (-timeout) Retries (-retries)
Very Low Resources 1 GB 1 2 2 10 30 3
Low Resources 2 GB 2 5 5 20 25 2
Moderate Resources 4 GB 4 10 10 50 20 2
High Resources 8 GB 4-8 25 25 200 15 1
Very High Resources 16+ GB 8+ 50+ 50+ 500+ 10 1
Maximum Performance 32+ GB 16+ 100+ 100+ 1000+ 10 1

Notes:

  • Adjust Incrementally: Start with conservative values and increase them gradually while monitoring system performance.
  • Monitor Resource Usage: Use tools like htop, top, nload, or other monitoring solutions.
  • Consider Network Bandwidth: Ensure your network can handle the increased number of requests without packet loss.
  • Use Scan Strategies: For large target lists, strategies like -ss host-spray can help reduce memory usage.

Additional Tips for Speed Optimization

Template Selection

  • Prioritize High-Severity Templates: Focus on templates with critical or high severity to quickly identify significant vulnerabilities.
  • Exclude Resource-Intensive Templates: Temporarily exclude templates that consume more resources or take longer to execute.

Disable Unnecessary Features

  • Disable Interactsh if Not Needed:

    nuclei -ni ...

  • Avoid Headless Scanning Unless Necessary: Headless scans consume significant resources and can slow down the scanning process.

Utilize Multiple Machines

  • Distribute the Workload: Split the target list across multiple machines to parallelize the scanning process and reduce total scan time.

Use Pre-Processing

  • HTTP Probing: Use tools like httpx to filter live targets before scanning.

    httpx -l targets.txt -silent -o live_targets.txt
nuclei -l live_targets.txt ...

    This ensures that Nuclei only scans active hosts, saving time and resources.

Leverage Caching

  • Enable Project Caching:

    nuclei -l targets.txt -project -project-path nuclei_cache ...

    This prevents re-scanning the same requests, saving time on subsequent runs.

Monitor and Tune Regularly

  • Adjust Based on Live Feedback: Use Nuclei’s statistics and logging features to monitor performance and adjust parameters in real-time.

    nuclei -l targets.txt -stats -si 5

    This displays statistics every 5 seconds.

Monitoring and Troubleshooting

  • Use Verbose Output:

    nuclei -l targets.txt -t templates/ -v

  • Enable Debugging:

    nuclei -l targets.txt -t templates/ -debug -debug-req -debug-resp

  • Monitor Statistics:

    nuclei -l targets.txt -t templates/ -stats -si 10

  • Handle Errors:

    • Max Host Errors:

      nuclei -l targets.txt -t templates/ -mhe 10

    • Disable Max Host Error Removal:

      nuclei -l targets.txt -t templates/ -nmhe

  • Profiling:

    • Enable Profiling:

      export PPROF=1
nuclei -l targets.txt -t templates/

    • Set Profiling Interval:

      export PPROF_TIME=30s

Conclusion

Optimizing Nuclei for mass scanning requires a balance between speed and resource management. By understanding and adjusting key flags and options, you can tailor Nuclei’s performance to your specific environment and needs. Always start with conservative settings and incrementally adjust to find the optimal configuration.

Remember to monitor system resources during scans and adjust parameters accordingly. With proper optimization, Nuclei becomes an even more powerful tool in your cybersecurity arsenal, capable of efficiently scanning vast numbers of targets.

Additional Resources

By following the strategies outlined in this guide, you can optimize Nuclei for mass scanning, ensuring efficient, reliable, and accurate vulnerability assessments across extensive target lists. Happy scanning!