Dark web threat actor de-anonymization

1. Executive Snapshot

The National Technical Research Organisation (NTRO) requires an autonomous threat intelligence and de-anonymization platform to identify, correlate, and track malicious actors operating across Tor hidden services, underground marketplaces, and darknet forums [PS]. Dark web threat actors routinely exploit anonymity networks to trade illicit goods, stolen data, hacking tools, and conduct cyber extortion while concealing their real-world identities [PS]. Existing manual investigation methods struggle against frequent actor rebranding, infrastructure rotation, and multi-alias identities [INFERENCE]. To address this bottleneck, the proposed platform—code-named AegisDark—combines autonomous Tor infrastructure misconfiguration scanning (detecting exposed /server-status pages, SSL certificate leaks, service banners, and descriptor anomalies), multi-source identity graph correlation (mapping handles, PGP fingerprints, crypto wallets, and trust networks), and AI-driven stylometric and behavioral profiling [INFERENCE]. The system surfaces actor attribution confidence, timeline queries, and actionable report exports (CSV, JSON, PDF/Executive summary) for cyber intelligence analysts [PS].

Problem Statement Metadata

Fundamental System Objective: To autonomously discover, correlate, and attribute dark web threat actors to real-world entities by combining infrastructure misconfiguration analysis, multi-platform identity graph mapping, and AI stylometric profiling into an analytical intelligence dashboard [INFERENCE].

2. Problem Deconstruction

The problem statement provided by NTRO decomposes into atomic functional, technical, and operational requirements [PS]:

• Requirement A (Autonomous Multi-Source Gathering):

  • Source Text: "continuously gathering their footprints from a range of sources (marketplaces, forums, deep web etc.) and linking them to the identifying information available on those sources." [PS]
  • Interpretation: The platform must ingest data from heterogeneous onion services, forums, and leak repositories without manual intervention [INFERENCE].
  • Implementation Implication: Distributed Tor crawlers with socks5 proxy rotation, rate-limiting bypass, HTML parsing, and structured data extraction [INFERENCE].
  • Confidence: [PS]

• Requirement B (Tor Infrastructure Misconfiguration & Origin Detection):

  • Source Text: "finding misconfigurations in Tor hidden services-such as exposed server-status pages, SSL certificates tied to clearnet domains, default service banners, descriptor inconsistencies, etc and matching them with clearnet infrastructure to point to the likely origin servers." [PS]
  • Interpretation: Passive and active network probing of .onion endpoints to extract infrastructure fingerprints that match clearnet IP addresses or domains [INFERENCE].
  • Implementation Implication: Automated HTTP scanner checking /server-status, /phpinfo, TLS certificate Subject Alternative Names (SANs), SSH host keys, Favicon hashes (MMH3), and shodan/censys cross-referencing [INFERENCE].
  • Confidence: [PS]

• Requirement C (Heterogeneous Relationship Graph Mapping):

  • Source Text: "mapping threat actors across multiple marketplaces into a single relationship graph of handles, PGP keys, wallets and trust links." [PS]
  • Interpretation: Link aliases across disparate sites by unifying PGP public key fingerprints, cryptocurrency deposit/payout addresses, handle overlaps, and forum vouching networks [INFERENCE].
  • Implementation Implication: Property Graph Database (Neo4j) with entity resolution logic and deterministic/probabilistic linkage rules [INFERENCE].
  • Confidence: [PS]

• Requirement D (AI Stylometric Persona Identification & Behavioral Profiling):

  • Source Text: "using AI-based analysis, including stylometric persona identification and behavioural profiling, to link rebranded or migrated personas to known threat actors." [PS]
  • Interpretation: When actors abandon old handles or PGP keys (rebranding), the system must match writing style and activity timing to link the new persona to the legacy profile [INFERENCE].
  • Implementation Implication: NLP pipeline combining character/word n-gram TF-IDF vectors, functional word frequencies, syntactic parse trees, Transformer style embeddings (Sentence-BERT), and posting schedule distributions [INFERENCE].
  • Confidence: [PS]

• Requirement E (Analytical Front-End & Timeline Queries):

  • Source Text: "provide an analytical front end to query the database across a chosen timeline and shall work in an autonomous mode, drawing on available sources of good quality and reliability." [PS]
  • Interpretation: Interactive GUI for cyber analysts to search actors, visualize graph connections, track activity timelines, and inspect source evidence [INFERENCE].
  • Implementation Implication: React/Next.js dashboard with graph visualization (Cytoscape.js/D3.js), filterable timelines, score indicators, and source reliability filtering [INFERENCE].
  • Confidence: [PS]

• Requirement F (Data Storage, Attribution & Export Capabilities):

  • Source Text: "collection, storage, contextualization and querying (through GUI/dashboards) of dark web threat actor intelligence-covering actor profiles, identifiers (handles, PGP keys, wallets etc.), hidden service infrastructure indicators, persona linkages, attribution confidence, category, last scan date and source. The system shall also provide the facility to export the result set in CSV, JSON and report formats." [PS]
  • Interpretation: Robust database schema storing comprehensive metadata and an export module generating machine-readable (JSON/CSV) and human-readable intelligence dossiers [INFERENCE].
  • Implementation Implication: PostgreSQL for relational/structured intel, Neo4j for relationship graphs, and automated PDF/JSON/CSV report generation engine [INFERENCE].
  • Confidence: [PS]

3. Stakeholders & User Needs

4. Requirements Traceability

5. Constraints & Environment Realities

• Tor Anonymity & Architectural Isolation: Onion routing explicitly encrypts traffic in 3 hops (Guard, Middle, Exit/Rendezvous) [KNOWN]. De-anonymization cannot rely on breaking Tor's cryptography; it must rely on operational security (OPSEC) failures by server operators or threat actors [RESEARCHED].
• Tor Network Latency & Rate Limits: Tor connections experience high latency (500ms – 3000ms) and low throughput compared to clearnet [KNOWN]. Crawlers must handle timeouts gracefully and use concurrency without overloading Tor circuits [INFERENCE].
• Anti-Scraping Mechanisms: Underground forums heavily employ custom CAPTCHAs, Cloudflare/DDoS-Guard protection, invite-only login walls, and session timeouts [KNOWN].
• Data Ephemerality & Dynamic Infrastructure: Dark web marketplaces and forums frequently exit-scam, get seized, or shift .onion V3 domains [KNOWN]. Data ingestion must capture full DOM snapshots and raw headers before endpoints disappear [INFERENCE].
• Computational Hardware Constraints: Large Language Models (LLMs) and massive graph analytics require GPU acceleration and substantial RAM [KNOWN]. The hackathon prototype must run efficiently on accessible hardware (e.g., single NVIDIA RTX GPU or CPU-optimized lightweight models like FastText/DistilBERT) [ASSUMPTION].

6. Research Questions

1. Infrastructure: What proportion of active Tor hidden services expose /server-status, /phpinfo, or TLS certificates containing clearnet hostnames or IPs? [NEEDS VALIDATION]
2. Stylometry: What is the baseline accuracy of stylometric author verification across short forum posts (10–50 words) versus long pastes/announcements? [RESEARCHED]
3. Blockchain Analytics: How effectively can public Bitcoin/Ethereum transaction graphs link darknet market deposit wallets to centralized exchange deposit addresses without proprietary commercial feeds? [RESEARCHED]
4. OPSEC Failures: What are the most common unique identifiers leaked by threat actors across multiple forums (e.g., re-used PGP keys, Jabber/TOX IDs, custom signature graphics)? [KNOWN]

7. Data Requirements & Sources

8. Existing Solutions & Gap Analysis

1. OnionScan (Open-Source Tool)

• Description: Open-source Go utility that inspects Tor hidden services for operational security vulnerabilities, open directories, /server-status leaks, and EXIF metadata [RESEARCHED].
• Strengths: Lightweight, directly addresses infrastructure misconfigurations [RESEARCHED].
• Weaknesses: Unmaintained since 2017; limited V3 onion support; lacks AI stylometry, actor graph mapping, and timeline analytics [RESEARCHED].
• Relevance: Direct reference for Requirement B [INFERENCE].
• Gap: Does not perform persona cross-linking, stylometry, or transaction graph analysis [INFERENCE].
• Source: OnionScan Documentation / GitHub [RESEARCHED]. Confidence: [RESEARCHED]

2. Chainalysis / Elliptic (Commercial Crypto Analytics)

• Description: Enterprise blockchain intelligence platforms that cluster wallet addresses and trace illicit fund flows across darknet markets and exchanges [RESEARCHED].
• Strengths: Massive proprietary wallet attribution database and heuristic clustering algorithms [RESEARCHED].
• Weaknesses: Closed-source, extremely costly, focused solely on cryptocurrency transactions rather than stylometry or infrastructure leaks [INFERENCE].
• Relevance: Models the crypto wallet graph requirement [INFERENCE].
• Gap: Completely detached from NLP stylometry and Tor infrastructure misconfiguration scanning [INFERENCE].
• Source: Elliptic Law Enforcement Overview [RESEARCHED]. Confidence: [RESEARCHED]

3. SpiderFoot / Maltego (OSINT Graph Platforms)

• Description: Link-analysis and OSINT automation platforms that visualize relationships between domains, IP addresses, email handles, and social profiles [RESEARCHED].
• Strengths: Flexible graph visualization and extensive transform integrations [RESEARCHED].
• Weaknesses: Requires manual trigger of transforms; lacks custom NLP stylometric engines for dark web text profiling [INFERENCE].
• Relevance: Informs GUI graph layout and multi-entity correlation [INFERENCE].
• Gap: Does not provide autonomous stylometric author matching or automated onion descriptor analysis out of the box [INFERENCE].
• Source: SpiderFoot OSINT Tool Documentation [RESEARCHED]. Confidence: [RESEARCHED]

9. Candidate Solution Approaches

Approach 1: Rules-Based Heuristic Aggregator

• Mechanism: Regular expressions and exact-match deterministic rules to parse handles, PGP keys, BTC wallets, and IP addresses. Matches /server-status IP strings directly against clearnet databases [INFERENCE].
• Problem Fit: Solves direct OPSEC leaks well [INFERENCE].
• Accuracy Potential: 100% precision for exact matches (PGP, IP), 0% recall for rebranded personas or modified handles [INFERENCE].
• Data Requirements: Raw text and HTTP response headers [INFERENCE].
• Complexity: Low [INFERENCE].
• Explainability: Perfect (100% rule-based) [INFERENCE].
• Cost: Very Low [INFERENCE].
• Scalability: High [INFERENCE].
• Hackathon Feasibility: Excellent [INFERENCE].
• Production Feasibility: Poor (Easily bypassed when threat actors change handles or use basic OPSEC) [INFERENCE].

Approach 2: Deep Learning & Heavy LLM Agent Pipeline

• Mechanism: Fine-tuned Llama-3 / Mistral LLMs for content extraction, agentic web crawling, and deep transformer-based stylometric embeddings for every post [INFERENCE].
• Problem Fit: High semantic understanding of forum posts [INFERENCE].
• Accuracy Potential: High stylometric nuance [INFERENCE].
• Data Requirements: Massive labeled text datasets for LLM fine-tuning [INFERENCE].
• Complexity: Very High [INFERENCE].
• Explainability: Low (Black-box neural network embeddings) [INFERENCE].
• Cost: High (Requires expensive GPU cluster for real-time inference) [INFERENCE].
• Scalability: Poor without heavy infrastructure funding [INFERENCE].
• Hackathon Feasibility: Very Low (Prone to latency, API timeouts, and hardware limits) [INFERENCE].
• Production Feasibility: Moderate [INFERENCE].

Approach 3: Hybrid Architecture (Multi-Layered Heuristic Graph + Feature-Based AI Stylometry)

• Mechanism:
  1. Layer 1 (Deterministic Graph): Exact matching on PGP fingerprints, wallet clusters (Common Input Heuristic), SSH host keys, and SSL cert serials in Neo4j [INFERENCE].
  2. Layer 2 (Infrastructure Probing): Automated Tor HTTP engine extracting headers, /server-status IPs, Favicon MMH3 hashes, and cross-matching with Censys/Shodan APIs [INFERENCE].
  3. Layer 3 (AI Stylometry): Dual-headed NLP pipeline combining fast character n-gram TF-IDF + Support Vector Machines (SVM) with Sentence-BERT embeddings to produce an Attribution Confidence Score (0–100%) [RESEARCHED].
  4. Layer 4 (Analytics UI): Timeline-driven dashboard with Cytoscape/D3 graph visualization [INFERENCE].
• Problem Fit: Addresses all requirements in the NTRO problem statement [INFERENCE].
• Accuracy Potential: High (Combines 100% accurate deterministic rules with probabilistic ML) [INFERENCE].
• Complexity: Moderate [INFERENCE].
• Explainability: High (Provides specific evidence trails: e.g., "PGP Match: 100%, Stylometric Score: 84%") [INFERENCE].
• Cost: Low to Moderate (Runs on standard cloud/hackathon hardware) [INFERENCE].
• Scalability: Very High [INFERENCE].
• Hackathon Feasibility: High [INFERENCE].
• Production Feasibility: High [INFERENCE].

10. Recommended Approach & Why

Selected Approach: Approach 3 (Hybrid Architecture: Multi-Layered Heuristic Graph + Feature-Based AI Stylometry) [INFERENCE].

Justification & Rejection of Alternatives

• Why Approach 1 was rejected: Pure heuristic rules completely fail when a threat actor rebrands (e.g., switching from Handle AlphaDemon to ShadowZero without re-using PGP keys) [INFERENCE]. It fails Requirement D [PS].
• Why Approach 2 was rejected: Relying purely on end-to-end heavy LLMs is cost-prohibitive, introduces hallucination risks into criminal investigations, lacks explainability, and cannot run efficiently within a 36-hour hackathon environment [INFERENCE].
• Why Approach 3 Wins: The hybrid approach cleanly separates deterministic evidence (PGP keys, wallet addresses, SSL certificates) from probabilistic evidence (stylometry, posting schedules) [INFERENCE]. An analyst can see why two personas are linked (e.g., "Shared BTC Wallet + 88% Stylometric Match") [INFERENCE]. It fully satisfies all explicit requirements of SIH26151 while remaining implementable by a 4-6 student team [INFERENCE].

11. Technical Architecture

====================================================================================================
                                      AEGISDARK SYSTEM ARCHITECTURE
====================================================================================================

[ INGESTION & SCANNING LAYER ]
  +--------------------------+    +--------------------------+    +--------------------------+
  |  Tor Scraping Engine     |    |  InfraScanner Module     |    |  Clearnet OSINT API      |
  |  (Python/Playwright/Tor) |    |  (Header/Cert/Status)    |    |  (Censys / Shodan / BTC) |
  +-------------+------------+    +------------+-------------+    +------------+-------------+
                |                              |                               |
                +----------------------+-------+-------------------------------+
                                       |
                                       v
[ PROCESSING & FEATURE EXTRACTION LAYER ]
  +------------------------------------------------------------------------------------------+
  | Task Queue (Celery + Redis)                                                               |
  |  ├─ Entity Extractor (Regex & SpaCy NER: Handles, PGP, BTC/ETH Wallets, Email, IPs)     |
  |  ├─ Infrastructure Matcher (Favicon MMH3, Cert Serial, Server Banner, /server-status)   |
  |  └─ NLP Stylometry Pipeline (TF-IDF Character N-Grams + Sentence-BERT Embeddings)        |
  +------------------------------------+-----------------------------------------------------+
                                       |
                                       v
[ DUAL PERSISTENCE LAYER ]
  +------------------------------------+-----------------------------------------------------+
  | Relational Store (PostgreSQL)      | Property Graph Store (Neo4j)                        |
  |  ├─ Actor Profiles                 |  ├─ Nodes: (:Actor), (:Handle), (:Wallet), (:IP)   |
  |  ├─ Raw Scrapes & Snapshots        |  └─ Edges: [:USES_HANDLE], [:TRANSFERRED_TO],      |
  |  └─ Audit Logs & Scan History      |             [:HOSTED_ON], [:STYLOGRAPHIC_MATCH]     |
  +------------------------------------+-----------------------------------------------------+
                                       |
                                       v
[ ANALYTICAL GUI & EXPORT LAYER ]
  +------------------------------------------------------------------------------------------+
  | Next.js Analytical Workbench Dashboard                                                    |
  |  ├─ Timeline Query Engine (Filter by date, category, source reliability)                  |
  |  ├─ Interactive Threat Actor Relationship Graph (Cytoscape.js / D3.js)                    |
  |  ├─ Attribution Confidence Breakdown (Deterministic vs Probabilistic Scores)             |
  |  └─ Export Engine (JSON / CSV / Forensic PDF Intelligence Dossier)                         |
  +------------------------------------------------------------------------------------------+
====================================================================================================

Data-Flow Narrative

1. Ingestion: The Tor Scraping Engine fetches target onion marketplace and forum pages via SOCKS5 Tor proxies [INFERENCE].
2. Infrastructure Probing: Concurrently, the InfraScanner issues HTTP HEAD/GET requests to target .onion domains checking for exposed /server-status, /phpinfo, TLS certificate Subject Alternative Names (SANs), and Favicon hashes [INFERENCE].
3. Entity & Feature Extraction: Raw HTML and text are pushed to Celery workers. RegEx and SpaCy NER identify PGP public keys, cryptocurrency wallets (BTC, ETH, XMR), handles, and contact aliases [INFERENCE].
4. Clearnet Cross-Matching: Leaked IP addresses, TLS fingerprints, or Favicon hashes are cross-referenced via Shodan/Censys APIs to identify clearnet origin servers [INFERENCE].
5. Stylometric Comparison: New textual posts are processed by the NLP Stylometry Pipeline, comparing TF-IDF character n-grams and Sentence-BERT vector cosine similarity against existing actor profiles in PostgreSQL [RESEARCHED].
6. Graph Linkage & Scoring: Neo4j inserts/updates nodes and directed edges [INFERENCE]. An overall Attribution Confidence Score ($S$) is calculated: $$S = w_1 \cdot \text{DeterministicMatch} + w_2 \cdot \text{StylometricScore} + w_3 \cdot \text{InfraMatch}$$
7. Visualization & Export: Cyber analysts query the Next.js GUI across time windows, inspect the graph topology, examine origin IP evidence, and export comprehensive CSV, JSON, or PDF reports [PS].

12. Technology Stack

13. AI/ML Justification

AI Component 1: Stylometric Persona Identification Engine

• Purpose: Link rebranded or alt-account personas across darknet forums to known threat actors when no shared PGP keys or handles exist [PS].
• Model Type: Hybrid Ensemble (Character N-Gram TF-IDF + Support Vector Classifier + Sentence-BERT Vector Cosine Similarity) [RESEARCHED].
• Input: Raw text corpus (forum posts, pastebins, marketplace listings) min 100 words [INFERENCE].
• Output: Cosine similarity score (0.00 to 1.00) and candidate actor rank list with confidence percentage [INFERENCE].
• Data Required: Labeled forum post datasets (e.g., synthetic dark web forum corpus) [INFERENCE].
• Why AI is Preferable: Human writing style contains subconscious idiosyncratic patterns (punctuation usage, vocabulary richness, structural n-grams) that cannot be captured by simple keyword matching [RESEARCHED].
• Evaluation: Precision@K, Recall, F1-score on held-out author attribution verification sets [INFERENCE].
• Failure Mode: False positive attribution if actors intentionally mask writing style (adversarial stylometry) or use generative AI [RESEARCHED].
• Human Oversight: High. AI output provides a probabilistic score; analyst must manually verify before taking operational/legal action [INFERENCE].

AI Component 2: Named Entity Recognition (NER) for Threat Intel

• Purpose: Automatically parse handles, crypto wallets, Jabber IDs, and PGP fingerprints from unformatted HTML/text [INFERENCE].
• Model Type: SpaCy Custom Transformer NER pipeline combined with Regex fallback rules [INFERENCE].
• Input: Raw scraped Web page text [INFERENCE].
• Output: Structured JSON entities (HANDLE, BITCOIN_ADDRESS, PGP_FINGERPRINT, ONION_URL, IP_ADDRESS) [INFERENCE].
• Data Required: Annotated cybersecurity and darkweb forum text samples [INFERENCE].
• Why AI is Preferable: RegEx handles known formats (e.g. BTC addresses), but custom NER is required to distinguish actor handles from generic forum vocabulary in unstructured text [INFERENCE].
• Evaluation: Entity extraction Precision and Recall [INFERENCE].
• Failure Mode: Missed entities (false negatives) or misclassifying generic words as handles [INFERENCE].
• Human Oversight: Medium. Extracted entities can be manually edited in the analytical dashboard [INFERENCE].

Where AI Should NOT Be Used

• Infrastructure Matching: Origin IP extraction from /server-status or SSL certificate Subject Alternative Names (SANs) must rely 100% on deterministic string and cryptographic hash matching [INFERENCE]. AI must never hallucinate IP addresses or TLS fingerprints [INFERENCE].

14. Data Architecture & Offline/Low-Network Design

Database Schema Design

Relational Schema (PostgreSQL)

CREATE TABLE threat_actors (
    actor_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    canonical_name VARCHAR(100) NOT NULL,
    category VARCHAR(50) NOT NULL, -- e.g., Ransomware, Financial Fraud, Initial Access
    attribution_confidence NUMERIC(5,2) NOT NULL, -- 0.00 to 100.00%
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE actor_identifiers (
    identifier_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    actor_id UUID REFERENCES threat_actors(actor_id),
    type VARCHAR(30) NOT NULL, -- HANDLE, PGP_KEY, BTC_WALLET, ETH_WALLET, EMAIL
    value TEXT NOT NULL,
    source_url TEXT NOT NULL,
    first_seen TIMESTAMP WITH TIME ZONE,
    last_seen TIMESTAMP WITH TIME ZONE
);

CREATE TABLE onion_infrastructure (
    infra_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    onion_address VARCHAR(255) NOT NULL,
    server_banner TEXT,
    ssl_cert_fingerprint VARCHAR(128),
    leaked_origin_ip VARCHAR(45),
    mod_status_exposed BOOLEAN DEFAULT FALSE,
    favicon_mmh3_hash VARCHAR(64),
    last_scan_date TIMESTAMP WITH TIME ZONE,
    source_reliability_score NUMERIC(3,2)
);

Graph Schema (Neo4j Cypher Data Model)

• Nodes:
  • (:Actor {id, canonical_name, confidence})
  • (:Handle {name, platform})
  • (:PGPKey {fingerprint, key_id})
  • (:Wallet {address, coin_type})
  • (:OnionService {url, status})
  • (:ClearnetIP {ip_address, country, isp})
• Relationships:
  • (:Actor)-[:USES_HANDLE]->(:Handle)
  • (:Handle)-[:SIGNED_WITH]->(:PGPKey)
  • (:Handle)-[:RECEIVES_FUNDS]->(:Wallet)
  • (:OnionService)-[:EXPOSES_ORIGIN]->(:ClearnetIP)
  • (:Actor)-[:STYLOGRAPHIC_MATCH {score: 0.88}]->(:Actor)

Offline / Low-Network Optimization

• Local Caching: Frontend leverages React Query and IndexedDB to cache active graph structures locally, enabling fluid UI exploration without re-querying the graph database [INFERENCE].
• Batch Scanning Queue: Crawling tasks are queued locally in Redis and executed in adaptive batches to prevent clogging Tor circuits or triggering target IP bans [INFERENCE].

15. Security, Privacy & Governance

• Handling Sensitive Data: Threat actor profiles contain illicit forum communications, stolen credentials, and PGP signatures [KNOWN]. Storage disks must use AES-256 encryption at rest [INFERENCE].
• Operational Security (OPSEC) for Scrapers: Scrapers operating from NTRO infrastructure must route 100% of outbound traffic through isolated Tor proxies with strict circuit rotation to avoid revealing agency IP addresses to threat actors [INFERENCE].
• Role-Based Access Control (RBAC):
  • Tier 1 (Analyst): Query, view graph, filter timeline, export reports [INFERENCE].
  • Tier 2 (Senior Investigator): Merge profiles, override AI attribution scores, add manual intelligence nodes [INFERENCE].
  • Tier 3 (Admin): Manage crawler configurations, user provisioning, API keys [INFERENCE].
• Regulatory & Legal Disclaimer: De-anonymization intelligence generated by AegisDark provides investigative leads [INFERENCE]. Legal attribution and law enforcement action require independent evidence verification and judicial authorization [INFERENCE].

16. MVP Scope (Hackathon Build)

+-----------------------------------------------------------------------------------+
|                                  HACKATHON MVP SCOPE                              |
+-----------------------------------------------------------------------------------+
|  MUST BUILD (Included in Prototype)                                               |
|  ├─ Local Tor Scraping & Misconfiguration Probing Engine                          |
|  ├─ Ingestion of 3-5 Simulated Darknet Forum & Marketplace Datasets               |
|  ├─ Entity Extractor (Handles, PGP Fingerprints, Crypto Wallets, IPs)             |
|  ├─ Neo4j Graph Visualizer with Cytoscape.js                                       |
|  ├─ AI Stylometry Module (TF-IDF + SVM + Cosine Vector Match)                     |
|  ├─ Timeline Filterable Analytical Dashboard                                      |
|  └─ Export Engine (CSV, JSON, Formatted Intelligence Dossier PDF)                 |
+-----------------------------------------------------------------------------------+
|  DESCRBED ONLY / OUT OF SCOPE (Production Deployment Requirement)                  |
|  ├─ Live Active Exploitation or Zero-Day Payload Injection                        |
|  ├─ Mass Unbounded Scraping of Thousands of Live Onion Services                   |
|  ├─ Monero (XMR) RingCT Tracing Integration                                       |
|  └─ Real-Time Direct Integration with Classified NTRO Internal Feeds              |
+-----------------------------------------------------------------------------------+

17. Live Demo Simulation Plan

To ensure a seamless, reliable demonstration during SIH judging without depending on unpredictable Tor network latency or offline .onion sites, the team will utilize a Deterministic Simulation Rig:

1. Simulated Tor Environment:
   • Run a local Docker container serving mock .onion websites (e.g., a mock darknet market "SilkPath" and a forum "DreadSec") over local Nginx endpoints [ASSUMPTION].
   • Inject realistic misconfigurations into mock site A: An exposed /server-status page leaking a simulated clearnet IP 198.51.100.42 [ASSUMPTION].
   • Inject an SSL certificate with a Subject Alternative Name (SAN) matching api.silkpath-clearnet.com into mock site B [ASSUMPTION].
2. Simulated Rebranding Case Study:
   • Actor Alpha: Threat actor GhostRider operates on Forum A with known PGP Key 0x9B1A... and Bitcoin Wallet 1A1zP1... [ASSUMPTION].
   • Actor Beta: A "new" actor ViperX appears on Forum B with no PGP key, selling similar ransomware tools [ASSUMPTION].
   • The AI Demo: The judge triggers the Stylometry Analysis Button. AegisDark processes ViperX's text posts, compares them against GhostRider, extracts syntactic n-grams, and returns an 89.4% Stylometric Similarity Match [ASSUMPTION].
3. Graph & Export Demo:
   • The Neo4j Cytoscape graph dynamically merges GhostRider and ViperX under a single canonical profile [ASSUMPTION].
   • The student clicks Export Forensic Brief, generating a structured PDF/JSON intelligence report containing the full evidence chain [PS].

18. Implementation Roadmap

====================================================================================================
                        DEVELOPMENT ROADMAP (TARGET: 30 SEPTEMBER 2026)
====================================================================================================

Phase 1: Environment & Core Data Pipeline (Weeks 1 - 2)
  ├── Set up PostgreSQL schema, Neo4j graph structure, Docker environment
  ├── Build Stem/Tor SOCKS5 scraper module & HTML parser
  └── Implement regex/SpaCy NER entity extraction pipeline

Phase 2: Misconfiguration Scanner & Clearnet Linker (Weeks 3 - 4)
  ├── Build InfraScanner HTTP module (/server-status, SSL SAN, Favicon MMH3)
  ├── Integrate Shodan / Censys API cross-referencing logic
  └── Build deterministic graph linkage rules in Cypher

Phase 3: AI Stylometry Engine & Behavioral Profiler (Weeks 5 - 6)
  ├── Train Character N-Gram + SVM author identification model
  ├── Implement Sentence-BERT embedding vector similarity pipeline
  └── Implement temporal posting activity profiler

Phase 4: Analytical GUI Workbench & Timeline (Weeks 7 - 8)
  ├── Develop Next.js dashboard UI with Tailwind CSS
  ├── Implement Cytoscape.js interactive node-graph visualizer
  └── Build timeline query sliders, filters, and export engine (CSV/JSON/PDF)

Phase 5: End-to-End Testing & SIH Demo Preparation (Weeks 9 - 10)
  ├── Perform accuracy testing on synthetic dark web dataset
  ├── Build local Docker simulation environment for judging
  └── Finalize presentation, dossier, and video demonstration
====================================================================================================

19. Testing & Validation Plan

• Functional Testing:
  • Verify that regex/NER pipelines correctly extract 100% of PGP public keys, BTC addresses, and IPv4 addresses from test HTML samples [INFERENCE].
  • Validate CSV, JSON, and PDF export completeness against schema definitions [PS].
• Misconfiguration Detection Accuracy:
  • Test HTTP probe against 20 test web servers with deliberate OPSEC flaws (/server-status enabled, exposed .git, custom headers) [INFERENCE]. Success criteria: 100% detection rate [INFERENCE].
• AI Stylometry Verification:
  • Evaluate model accuracy on a benchmark author attribution corpus (e.g., 50 authors, 20 posts each) [RESEARCHED].
  • Measure Precision, Recall, and F1-Score [INFERENCE]. Target F1-Score: $> 0.82$ [ASSUMPTION].
• False Positive / False Negative Stress Testing:
  • Inject decoy forum posts written by different authors discussing the same topic [INFERENCE]. Ensure stylometry score remains below threshold ($< 0.50$) to avoid false attribution [INFERENCE].
• Performance & Offline Testing:
  • Verify graph rendering speed with 1,000+ nodes in Cytoscape.js ($< 1.5$ second render time) [INFERENCE].

20. Risk Register

21. Cost & Scalability

Hackathon Prototype

• Infrastructure Cost: $0 (Runs locally on student laptops using Docker, PostgreSQL, Neo4j Desktop, and Scikit-Learn/PyTorch) [ASSUMPTION].
• API Costs: $0 (Utilizes free community tiers for Censys, Shodan, and public Bitcoin APIs) [ASSUMPTION].

Production System (NTRO Deployment Scale)

• Cloud Compute (Kubernetes / On-Premise GPU Cluster): Dedicated server nodes for continuous Tor crawling, PostgreSQL storage, and Neo4j enterprise cluster [INFERENCE].
• Commercial Threat Feeds & Blockchain API Subscriptions: Enterprise licenses for Chainalysis/Elliptic APIs and commercial OSINT indices [INFERENCE].
• Scalability Design: Horizontal worker scaling using Kubernetes (K8s) for Celery crawler pods, with Neo4j read-replicas for multi-analyst query workloads [INFERENCE].

22. Differentiation

23. Future Scope

1. Cross-Chain Cryptocurrency Analytics: Expand financial graph mapping to include Ethereum smart contracts, privacy coin heuristics (Monero ring signature analysis), and decentralized exchange (DEX) bridges [RESEARCHED].
2. Multilingual Stylometric Embeddings: Train language-agnostic stylometric models (e.g., XLM-RoBERTa) to detect threat actors operating across Russian, Mandarin, and English darknet forums [RESEARCHED].
3. Autonomous Darknet Honeypots: Deploy specialized low-interaction honeypots to capture active scanning and exploit attempts originating from target threat groups [INFERENCE].

24. Assumptions & Open Items Log

Consolidated Assumptions

1. [ASSUMPTION] The hackathon live demo will utilize Dockerized local mock onion services to avoid Tor network instability during judging.
2. [ASSUMPTION] Scikit-Learn SVM combined with Sentence-BERT embeddings provides sufficient accuracy for stylometric profiling within hackathon hardware limits.
3. [ASSUMPTION] Public API free tiers (Censys, Shodan, Blockstream) will suffice for cross-referencing mock clearnet infrastructure during evaluation.

Open Items & Validation Requirements

1. [NEEDS VALIDATION] Exact NTRO institutional taxonomy for categorizing threat actors (e.g., specific APT naming conventions vs generic threat categories).
2. [NEEDS VALIDATION] Legal frameworks and chain-of-custody protocols required for exporting admissible evidence reports in Indian jurisdiction.

Final Recommendation

Build AegisDark: A hybrid dark web threat actor de-anonymization platform integrating automated Tor infrastructure misconfiguration scanning, Neo4j multi-entity relationship mapping, and AI-driven stylometric author identification [INFERENCE].

Recommended Stack

• Frontend: Next.js (React), Tailwind CSS, Cytoscape.js [INFERENCE].
• Backend: FastAPI (Python), Celery, Redis [INFERENCE].
• Databases: PostgreSQL (Relational Intel), Neo4j (Property Graph) [INFERENCE].
• AI/ML: Scikit-Learn (TF-IDF + SVM), Sentence-Transformers (Sentence-BERT), SpaCy [INFERENCE].

Biggest Dependency

Availability of representative forum text samples to calibrate stylometric NLP threshold baselines [INFERENCE].

Biggest Risk

Tor circuit blocking, high latency, and CAPTCHA roadblocks during active scraping [INFERENCE].

Biggest Technical Challenge

Engineered balance between precision and recall in stylometric author identification to prevent false positive attributions [INFERENCE].

First 5 Actions

1. Initialize GitHub repository and Docker environment containing PostgreSQL, Neo4j, and Redis [INFERENCE].
2. Implement regex parsing module for PGP keys, Bitcoin wallets, and IPv4 strings [INFERENCE].
3. Build the InfraScanner module to parse /server-status HTML and extract SSL certificate SANs [INFERENCE].
4. Train the Scikit-Learn TF-IDF + SVM stylometry pipeline on sample multi-author text corpora [INFERENCE].
5. Develop the Next.js Cytoscape.js interactive graph component and timeline query interface [INFERENCE].

Validation Before Coding

Verify that Neo4j Cypher graph queries efficiently resolve 3-hop relationships ((:Actor)-[:USES_HANDLE]->(:Handle)-[:RECEIVES_FUNDS]->(:Wallet)) in under 100ms on local developer hardware [INFERENCE].

Buildability Score /100

92 / 100 (Highly feasible for a 4-6 student team using Python, Neo4j, Next.js, and standard ML libraries) [INFERENCE].

Confidence

High (All explicit requirements from NTRO SIH26151 are accounted for and mapped to proven open-source technologies and techniques) [INFERENCE].

Research Sources

1. Title: Hiding in text/plain sight: Security defences of Tor Onion Services
   • URL: https://arxiv.org/html/2312.00545v1
   • Source Type: Academic Research Publication
   • Supported Claim: Tor onion service security misconfigurations, TLS certificate exposure, and HTTP header leaks.
   • Reliability: High
2. Title: Full-Spectrum Dark Web Hunting, Monitoring, and Adversarial Analysis
   • URL: https://medium.com/@0xSilent/full-spectrum-dark-web-hunting-monitoring-and-adversarial-analysis-510f64ab6cbc
   • Source Type: Technical Security Article
   • Supported Claim: Dark web crawling techniques, OnionScan usage, /server-status probing, and stylometric author attribution pipelines.
   • Reliability: High
3. Title: Identifying Similar Users Between Dark Web and Surface Web Using BERTopic and Authorship Attribution
   • URL: https://www.mdpi.com/2079-9292/14/1/148
   • Source Type: Peer-Reviewed Journal Article (MDPI Electronics)
   • Supported Claim: Stylometric authorship attribution and NLP embeddings for linking dark web personas.
   • Reliability: High
4. Title: National Technical Research Organisation (NTRO) Overview
   • URL: https://modeldiplomat.com/learn/glossary/national-technical-research-organisation
   • Source Type: Organizational Reference
   • Supported Claim: NTRO mandate in signals intelligence, cyber intelligence, and national security infrastructure.
   • Reliability: High
5. Title: Dark Web Investigation Setup & OnionScan Overview
   • URL: https://hunch.ly/resources/Hunchly-Dark-Web-Setup.pdf
   • Source Type: Forensic Investigation Guide (Hunchly)
   • Supported Claim: Technical indicators for de-anonymizing hidden services using SSL certificates, Censys cross-queries, and operational security flaws.
   • Reliability: High