SatQuery AI - An Interactive Vision-Language Assistant for Multimodal Remote Sensing Image Analysis through Text Queries

SATQUERY AI - SIH26167 RESEARCH & ARCHITECTURE DOSSIER

1. Executive Snapshot

SatQuery AI is an agentic, multimodal vision-language ecosystem engineered for the Indian Space Research Organisation (ISRO) to democratize high-resolution Earth Observation (EO) analysis [PS]. Traditional satellite interpretation requires deep GIS expertise, manual band composition, and complex model pipelines; SatQuery AI replaces this with a natural-language query interface backed by an intelligent routing agent and domain-adapted AI models [PS, INFERENCE]. The system ingests single-modality optical/SAR imagery, co-registered optical–SAR pairs, and bi-temporal image series in native GeoTIFF/TIFF formats (as well as benchmark PNG/JPEG formats) [PS]. By dynamically validating input metadata, sequencing task-specific specialist models (VQA, captioning/grounding, change detection, and SAR–optical cross-modal fusion), and aggregating textual and visual feedback (bounding boxes, change masks, confidence scores), SatQuery AI produces evidence-grounded, auditable intelligence reports [PS]. Adapted using the multi-sensor BigEarthNet.txt dataset and evaluated on public benchmarks (VRSBench, RSVQA, CDVQA) as well as ISRO/SAC's proprietary Cartosat-2S optical and RISAT C-band SAR evaluation dataset, SatQuery AI bridges the gap between raw satellite telemetry and operational decision-making [PS, RESEARCHED].

Metadata Summary

2. Problem Deconstruction

The problem description provided by ISRO was deconstructed into atomic functional and non-functional requirements. Every explicit requirement is preserved with its exact lettered prefix where applicable.

• [REQ-A] Remote Sensing Domain Adaptation: The system must include a vision or vision-language component that is fine-tuned or domain-adapted using BigEarthNet.txt or open-source remote sensing datasets to handle satellite spectral bands, polarizations, and spatial structures [PS].
• [REQ-B] Single-Image Visual Question Answering (Mandatory Baseline): The system must answer natural language questions for a single optical, multispectral, or Synthetic Aperture Radar (SAR) image [PS].
• [REQ-C] Secondary Single-Image Task: The system must implement at least one additional single-image capability: either detailed natural language scene captioning OR text-guided region grounding (predicting spatial bounding boxes/polygons for queried land features) [PS].
• [REQ-D] Multi-Image Bi-Temporal Change Analysis: The system must perform change description or change-based visual question answering (CDVQA) on a bi-temporal pair of images acquired at different times over the same geographic area [PS]. Where reference masks or alignment permit, it should additionally generate a spatial change map [PS].
• [REQ-E] Cross-Modal Pair Analysis (Optical + SAR): The system must perform joint information extraction and cross-modal analysis on co-registered optical/multispectral and SAR image pairs (e.g., leveraging SAR cloud penetration and optical spectral signatures) [PS].
• [REQ-F] Agentic Model and Tool Orchestration: The backend must feature an automated controller that:
  1. Interprets user intent and classifies the requested remote-sensing task [PS].
  2. Validates input image count, modality (optical/SAR), format, spatial resolution, projection system, and compatibility [PS].
  3. Selects and sequences appropriate specialist tools/models from a predefined tool registry [PS].
  4. Configures only permitted task parameters and executes the workflow [PS].
  5. Combines textual answers and visual grounding maps, calculates confidence estimates, and returns visual evidence [PS].
  6. Generates an auditable execution summary containing selected tasks, model/tool names, and execution parameters [PS].
• [REQ-G] Supported File Formats: The system must support native geospatial formats (GeoTIFF / TIFF) containing multi-band geospatial metadata [PS]. Benchmark formats (PNG / JPEG) are accepted strictly for public evaluation datasets (VRSBench, RSVQA, CDVQA) [PS].
• [REQ-H] Interactive User Interface & Deliverables: An interactive web GUI or web application must be delivered, capable of visualization, confidence displaying, execution auditing, and downloading PDF/JSON reports [PS].
• [REQ-I] Evaluation Compliance: System performance will be evaluated on public test splits (VRSBench, RSVQA, CDVQA) and an undisclosed ISRO Space Applications Centre (SAC) evaluation set featuring co-registered Cartosat-2S (optical, 0.6m-1.6m) and RISAT (C-band SAR) image pairs [PS, RESEARCHED].

3. Stakeholders & User Needs

4. Requirements Traceability

5. Constraints & Environment Realities

• Geospatial Data Formats & Coordinate Reference Systems (CRS): Satellite imagery comes in GeoTIFF format with embedded spatial metadata (bounding coordinates, affine transform, EPSG projections) [PS, KNOWN]. Modern deep learning models expect standard 3-channel RGB uint8 arrays [KNOWN]. The system must dynamically convert multi-band 11-bit/12-bit floating-point rasters into normalized displayable and model-digestible arrays without dropping geographic alignment [PS, INFERENCE].
• Sensor Modality Discrepancies: Optical sensors (Cartosat-2S: 0.65m PAN, 1.6m MS; Sentinel-2: 10m RGB-NIR) capture reflected solar light, which is obstructed by clouds and night [KNOWN, RESEARCHED]. SAR sensors (RISAT-1 C-band, Sentinel-1) emit active microwave signals and measure backscatter in polarizations (VV, VH, HH), functioning through cloud cover and at night [KNOWN, RESEARCHED]. Combining these requires handling distinct physics, dynamic ranges, and speckle noise [INFERENCE].
• Compute & VRAM Limitations: Fine-tuning and running large Vision-Language Models (e.g., 7B-14B parameter MLLMs) on high-resolution satellite tiles requires substantial GPU memory [KNOWN]. During a hackathon demo, real-time inference must be kept under 3 seconds per query, necessitating parameter-efficient fine-tuning (LoRA/QLoRA) and lightweight specialized vision heads [INFERENCE, ASSUMPTION].
• Bandwidth & Deployment Environment: Field workers and regional authorities often operate in low-bandwidth or disconnected environments [KNOWN]. High-resolution satellite tiles (which can exceed 100 MB each) cannot be transferred repeatedly over weak networks; tiled Web Map Service (WMS/WMTS) streaming or client-side caching is required [INFERENCE, ASSUMPTION].
• Strict No-Hallucination & Auditability Rules: ISRO applications demand verifiable visual evidence [PS]. Models cannot simply "claim" an area changed or a building exists without highlighting the exact pixel region or providing an execution trace detailing which tool generated the inference [PS, INFERENCE].

6. Research Questions

1. RQ-1: How can a single vision encoder digest multi-spectral optical data (4–12 bands) and dual-polarization SAR data (VV/VH or HH/HV) without discarding non-RGB information? [RESEARCHED]
2. RQ-2: What specific instruction-tuning format is utilized in BigEarthNet.txt (arXiv:2603.29630) to bridge Sentinel-1/2 pairs with multi-task text prompts? [RESEARCHED]
3. RQ-3: How can the agentic orchestrator reliably parse complex user intent into valid tool parameters without relying on unpredictable internal LLM thinking text? [PS, INFERENCE]
4. RQ-4: What feature-level or decision-level fusion architecture yields the highest accuracy when resolving spatial queries on co-registered Cartosat-2S optical and RISAT C-band SAR pairs under cloud cover? [RESEARCHED, INFERENCE]
5. RQ-5: How do benchmark coordinates in VRSBench (normalized 0–100 or 0–1000) translate back to geospatial coordinates (Latitude/Longitude or EPSG:3857) on high-resolution GeoTIFF rasters? [RESEARCHED, INFERENCE]

7. Data Requirements & Sources

8. Existing Solutions & Gap Analysis

9. Candidate Solution Approaches

Approach 1: Monolithic Multi-Task RS-VLM

• Mechanism: Single large Vision-Language Model (e.g., Qwen2-VL 7B fine-tuned across all tasks simultaneously) taking concat images directly into visual tokens.
• Problem Fit: Moderate [INFERENCE]. Concatenating multi-temporal and multi-modal images into a single vision encoder rapidly exhausts token windows and reduces spatial resolution [KNOWN].
• Accuracy Potential: 68–74% on VQA; poor on fine-grained change mask segmentation [INFERENCE].
• Complexity & Effort: Low system complexity, high fine-tuning compute requirements [INFERENCE].
• Hackathon Feasibility: Medium; heavy VRAM requirements (24GB+) [INFERENCE].
• Production Feasibility: Low due to monolithic inflexibility when adding new sensor types [INFERENCE].

Approach 2: Pipeline Rule-Based Router + Specialized ML Modules

• Mechanism: Hardcoded Python router that parses query keywords (e.g., "changed" -> Change Model; "where is" -> Grounding Model) and calls specific PyTorch models.
• Problem Fit: Sub-optimal [INFERENCE]. Fails on complex, ambiguous, or multi-intent user queries [INFERENCE].
• Accuracy Potential: High on narrow tasks, zero robustness on un-modeled phrasing [INFERENCE].
• Complexity & Effort: Low AI orchestration complexity, high manual rule maintenance [INFERENCE].
• Hackathon Feasibility: High; easily runnable [INFERENCE].
• Production Feasibility: Poor; brittle interface unsuitable for broad natural language [INFERENCE].

Approach 3: Agentic Orchestrator + Fine-Tuned RS Foundation Backbone + Tool Registry (RECOMMENDED)

• Mechanism: An agentic controller (LangGraph / Function Calling agent) acts as the central coordinator. It validates image formats/metadata, classifies query intent, dynamically selects specialized micro-models (RemoteCLIP VQA, VisTA Change Grounder, SAR-Optical Fusion Net) from a tool registry, executes the parameter pipeline, and aggregates results [PS, INFERENCE].
• Problem Fit: Perfect [PS, INFERENCE]. Directly satisfies explicit ISRO requirement [REQ-F] for agentic orchestration with observable execution traces [PS].
• Accuracy Potential: Superior (82–89%+ across benchmarks); specialized models handle spatial masks and VQA with high precision [INFERENCE].
• Complexity & Effort: High architectural design effort; modular implementation [INFERENCE].
• Hackathon Feasibility: Excellent for 4–6 student team; division of labor across UI, Agent Router, VQA Tool, and Change Tool [INFERENCE].
• Production Feasibility: High; new specialized tools (e.g., Hyperspectral PRISMA tool) can be registered without retraining the core agent [INFERENCE].

10. Recommended Approach & Why

Selected Solution: Agentic Orchestrator with RS-Adapted Multi-Tool Registry (Approach 3)

                       +-------------------------------------------------+
                       |            User Natural Language Query           |
                       |       & Input Rasters (Single / Pair GeoTIFF)   |
                       +-------------------------------------------------+
                                                |
                                                v
                       +-------------------------------------------------+
                       |          Input Compatibility & Metadata         |
                       |          Validator (GDAL / Rasterio / CRS)      |
                       +-------------------------------------------------+
                                                |
                                                v
                       +-------------------------------------------------+
                       |      Agentic Router (LangGraph Controller)       |
                       |  Intent Classification & Parameter Validation   |
                       +-------------------------------------------------+
                                                |
            +-----------------------------------+-----------------------------------+
            |                                   |                                   |
            v                                   v                                   v
+-----------------------+           +-----------------------+           +-----------------------+
|  Tool 1: RS-VQA       |           |  Tool 2: Grounding    |           |  Tool 3: Bi-Temporal  |
|  (Qwen2-VL /          |           |  & Captioning         |           |  Change & Mask Net    |
|  BigEarthNet Fine-    |           |  (VRSBench-Ref /      |           |  (VisTA / CDVQA /     |
|  Tuned Adapter)       |           |  Bounding Box Head)   |           |  Change Map Generator)|
+-----------------------+           +-----------------------+           +-----------------------+
            |                                   |                                   |
            +-----------------------------------+-----------------------------------+
                                                |
                                                v
                       +-------------------------------------------------+
                       |       Tool Output Aggregator & Confidence       |
                       |       Estimator + Auditable Execution Trace     |
                       +-------------------------------------------------+
                                                |
                                                v
                       +-------------------------------------------------+
                       |  Interactive Dashboard (Text Answer + Visual    |
                       |   Bounding Box / Change Mask + Execution JSON)  |
                       +-------------------------------------------------+

Why Alternatives Were Rejected:

1. Monolithic Models (Approach 1) fail to produce high-resolution pixel-level binary change maps alongside open-ended text answers simultaneously without extreme model bloat and latency [INFERENCE].
2. Static Keyword Rules (Approach 2) fail ISRO's requirement for intelligent natural language query comprehension and lack adaptability across diverse benchmark phrasing [PS, INFERENCE].
3. Approach 3 strictly aligns with ISRO's explicitly stated vision: "The novelty of SatQuery AI lies in its agentic, query-driven framework. Instead of applying a single generic VLM, the system selects and executes suitable remote-sensing specialist models..." [PS].

11. Technical Architecture

Layered Architecture Diagram

====================================================================================================
1. PRESENTATION LAYER (Web GUI / UI)
   +----------------------------------------------------------------------------------------------+
   | React 18 + TailwindCSS + Leaflet / OpenLayers Geospatial Map Component                       |
   | - Multi-Layer Canvas (Base Map, Bounding Box Overlays, GeoJSON Change Vector Overlays)      |
   | - Interactive Conversational Chat Drawer & Audit Log Inspector                               |
   | - GeoTIFF File Upload Dropzone (Metadata preview, EPSG verification, thumbnail renderer)      |
   +----------------------------------------------------------------------------------------------+
                                                | HTTP / REST / WebSockets
====================================================================================================
2. AGENTIC ORCHESTRATION & API LAYER (FastAPI + LangGraph)
   +----------------------------------------------------------------------------------------------+
   | LangGraph Execution Controller                                                                |
   |  ├── Query Intent Analyzer (Task Classification: VQA | Grounding | Change | Cross-Modal)     |
   |  ├── Input Compatibility Checker (Rasterio: shape, channels, EPSG, nodata values)           |
   |  ├── Model Registry Router (Matches task + input modalities to optimal specialist tool)      |
   |  └── Execution Audit Logger (Records Tool_ID, Parameters, Execution Time, Confidence)        |
   +----------------------------------------------------------------------------------------------+
                                                | In-Memory / IPC
====================================================================================================
3. SPECIALIST AI / ML TOOL REGISTRY LAYER
   +------------------------------+------------------------------+--------------------------------+
   | Tool 1: Single-Image VQA     | Tool 2: Visual Grounding     | Tool 3: Bi-Temporal Change     |
   | - Fine-tuned Qwen2-VL-2B /   | - RemoteCLIP + Grounding Head| - Siamese Feature Extractor +  |
   |   RSVQA Adapter              | - Predicts [x1, y1, x2, y2]   |   Change Map U-Net Decoder     |
   +------------------------------+------------------------------+--------------------------------+
   | Tool 4: SAR-Optical Fusion   | Tool 5: Image Captioner      | Tool 6: Geospatial Converter   |
   | - Cross-Attention Alignment  | - Detailed LULC Scene        | - Pixel BBox -> Lat/Lon GeoJSON|
   |   (Sentinel-1/2, Cartosat/RISAT)|   Describer (VRSBench)       | - Affine Matrix Reprojection   |
   +------------------------------+------------------------------+--------------------------------+
                                                | Tensor Streams
====================================================================================================
4. GEOSPATIAL DATA PROCESSING & INGESTION LAYER
   +----------------------------------------------------------------------------------------------+
   | Python Geospatial Engine (Rasterio, GDAL, NumPy, PyTorch DataLoaders)                        |
   | - Multi-Band Extraction (Cartosat MS 4-Band, Sentinel-2 12-Band, RISAT/Sentinel-1 VV/VH SAR)  |
   | - Histogram Normalization & Speckle Filtering (Lee Filter for SAR backscatter)              |
   | - Co-Registration Verification & Affine Resampling (Bilinear / Nearest Neighbor)            |
   +----------------------------------------------------------------------------------------------+
                                                | Disk I/O / S3 Storage
====================================================================================================
5. STORAGE & DATABASE LAYER
   +----------------------------------------------------------------------------------------------+
   | PostgreSQL + PostGIS (Spatial Data, Metadata, User History)                                  |
   | Redis Cache (Session States, Pre-calculated Raster Embeddings, Tool Execution Traces)       |
   | Local GeoTIFF Storage / S3 Bucket (Raster Store)                                             |
   +----------------------------------------------------------------------------------------------+
====================================================================================================

Detailed Data-Flow Narrative

1. Ingestion & Validation: User uploads satellite imagery (e.g., t1_optical.tif and t2_optical.tif for change detection, or optical.tif + sar.tif for cross-modal analysis) via the React dashboard [PS, ASSUMPTION]. The Rasterio validation middleware extracts raster metadata: width, height, band count, data type, coordinate reference system (CRS), and affine transform matrix [PS, KNOWN].
2. Query & Routing: User submits a textual query (e.g., "What built-up structures changed between these two dates and where?") [PS]. The LangGraph Agentic Controller parses the text alongside the validated image metadata [PS, INFERENCE].
3. Tool Selection & Execution: The orchestrator recognizes a bi-temporal input pair + change query [PS]. It selects Tool_3_BiTemporal_Change_Engine and configures permitted parameters (threshold = 0.5, resolution = native) [PS, ASSUMPTION].
4. Inference & Grounding: Tool_3 executes a forward pass through the Siamese vision backbone, generating a textual change summary and a binary 2D change probability matrix [RESEARCHED, INFERENCE].
5. Geospatial Vectorization: The visual output is passed to Tool_6_Geospatial_Converter, which uses the image's affine matrix to map pixel change masks into GeoJSON polygons or bounding box coordinates [INFERENCE, KNOWN].
6. Audit Trace & Response Construction: The orchestrator logs an auditable trace (task_type, model_name, parameters_used, execution_time_ms, confidence_score) and packages the final payload containing textual answers, GeoJSON vector layers, and confidence metrics to the GUI [PS, INFERENCE].

12. Technology Stack

13. AI/ML Justification

1. Single-Image VQA & Captioning Engine

• Purpose: Answers natural language questions and provides detailed LULC descriptions for optical or SAR rasters [PS].
• Model Type: Adapted Vision-Language Transformer (Qwen2-VL / RemoteCLIP backbone) [RESEARCHED].
• Input: Normalized 3-band RGB/False-Color array + Text Prompt [PS, INFERENCE].
• Output: Textual answer string + Confidence score [PS].
• Data Required: RSVQA and VRSBench-VQA [PS, RESEARCHED].
• Why AI is Necessary: Rule-based keyword matching cannot interpret unstructured natural language questions over complex visual satellite patterns [INFERENCE].
• Evaluation Metric: Accuracy, BLEU-4, METEOR, CIDEr [RESEARCHED].
• Failure Mode & Fallback: Low confidence prompt triggers request for query clarification or defaults to general LULC scene description [INFERENCE].
• Human Oversight: High for operational mission decisions; automated trace shown to user [PS].

2. Visual Grounding / Bounding Box Predictor

• Purpose: Identifies and localizes specific queried features (e.g., "Highlight the water body") [PS].
• Model Type: Object Detection / Referring Expression Head (GeoChat / Grounding-DINO style adapter) [RESEARCHED].
• Input: Image array + Target query term [PS].
• Output: Bounding box coordinates [x1, y1, x2, y2] normalized to [0-100] or GeoJSON BBox [PS, RESEARCHED].
• Data Required: VRSBench-Ref dataset [PS, RESEARCHED].
• Why AI is Necessary: Traditional spectral thresholding (e.g., NDVI) cannot isolate specific semantic objects like individual ships or buildings [KNOWN].
• Evaluation Metric: mean IoU (Intersection over Union) @ threshold 0.5 [RESEARCHED].
• Failure Mode: False positive bounding boxes over clutter [INFERENCE].

3. Bi-Temporal Change Detection & CDVQA Engine

• Purpose: Identifies "what changed and where" between time $T_1$ and time $T_2$ [PS].
• Model Type: Siamese Feature Extractor + Vision-Language Transformer Decoder (VisTA architecture) [RESEARCHED].
• Input: Pair of co-registered bi-temporal rasters ($T_1, T_2$) + Text Query [PS].
• Output: Textual change explanation + 2D spatial change probability mask [PS, RESEARCHED].
• Data Required: CDVQA and QAG-360K datasets [PS, RESEARCHED].
• Why AI is Necessary: Simple image subtraction flags seasonal vegetation changes or shadow differences as false changes [KNOWN].
• Evaluation Metric: F1-Score for change mask, BLEU-4 for textual change VQA [RESEARCHED].

4. Cross-Modal Optical–SAR Fusion Module

• Purpose: Combines optical spectral detail with SAR structural backscatter for day/night and cloud-resilient analysis [PS].
• Model Type: Cross-Attention Multi-Modal Encoder [RESEARCHED].
• Input: Co-registered Optical array (4-band) + SAR array (VV, VH polarizations) [PS].
• Output: Fused latent representation fed into VQA/Grounding heads [INFERENCE].
• Data Required: BigEarthNet.txt (co-registered Sentinel-1 SAR & Sentinel-2 Optical) [PS, RESEARCHED].
• Why AI is Necessary: Optical and SAR measure entirely different physical phenomena (reflectance vs. microwave backscatter); physics-based fusion requires complex manual calibration [KNOWN, INFERENCE].

Explicit Non-AI Component Identification:

• Raster Compatibility & CRS Verification: Solved strictly via deterministic Rasterio/GDAL checks (affine transform matrix, EPSG code match, band count). AI MUST NOT BE USED HERE [INFERENCE].
• Coordinate Transformation (Pixel to Lat/Lon): Matrix multiplication using affine matrix. AI MUST NOT BE USED HERE [INFERENCE].

14. Data Architecture & Offline/Low-Network Design

Spatial Data Schema (PostgreSQL / PostGIS)

-- Core Imagery Table
CREATE TABLE satellite_rasters (
    raster_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    filename VARCHAR(255) NOT NULL,
    modality VARCHAR(50) CHECK (modality IN ('OPTICAL', 'SAR', 'CROSS_MODAL_PAIR', 'BITEMPORAL_PAIR')),
    acquisition_date TIMESTAMP,
    sensor_name VARCHAR(100), -- Cartosat-2S, RISAT-1, Sentinel-1, Sentinel-2
    spatial_resolution_m NUMERIC(5,2),
    crs_epsg INT DEFAULT 4326,
    bounds GEOMETRY(Polygon, 4326),
    file_path VARCHAR(512) NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Agentic Query Execution Audit Table
CREATE TABLE query_audit_logs (
    log_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    raster_id UUID REFERENCES satellite_rasters(raster_id),
    user_query TEXT NOT NULL,
    detected_task VARCHAR(100) NOT NULL,
    selected_tool VARCHAR(100) NOT NULL,
    execution_parameters JSONB NOT NULL,
    execution_time_ms INT NOT NULL,
    estimated_confidence NUMERIC(4,3),
    output_text TEXT,
    output_vector_geojson JSONB,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Offline & Low-Network Synchronization Strategy

1. Client-Side Vector Rendering: Rather than streaming full high-resolution altered rasters back over the network, the backend converts change masks and bounding boxes into compressed GeoJSON vectors (typically < 20 KB) [INFERENCE, ASSUMPTION].
2. Local Tile Caching (IndexedDB): The web GUI caches base map vector tiles locally in the browser's IndexedDB, allowing smooth pan/zoom without re-fetching underlying imagery [INFERENCE, ASSUMPTION].
3. Progressive Web Application (PWA) Offline Mode: Operational field users can queue natural language queries locally when connectivity drops; queries execute automatically once network heartbeats resume [INFERENCE, ASSUMPTION].

15. Security, Privacy & Governance

• Sensitive Data & Defense Implications: High-resolution satellite data (e.g., Cartosat-2S at 0.6m) can reveal sensitive infrastructure, defense installations, or border assets [KNOWN]. SatQuery AI implements Role-Based Access Control (RBAC) preventing unauthorized downloading of raw GeoTIFF tiles while restricting visual responses according to user clearance levels [INFERENCE, ASSUMPTION].
• Data Integrity & No-Cloud Leakage: Evaluation data supplied by ISRO/SAC (Cartosat-2S and RISAT evaluation sets) is completely isolated [PS]. Zero telemetry or image data is transmitted to third-party public cloud APIs (e.g., OpenAI, Anthropic). All inferences execute on local isolated hardware [PS, INFERENCE].
• Auditability & Liability: Operational decisions made in disaster response or urban planning must be traceable [KNOWN]. The agent produces an immutable, append-only audit record in JSON format detailing tool selection, model weights, parameter constraints, and confidence metrics for every generated answer [PS, INFERENCE].
• Handling Model Hallucinations: When confidence scores fall below a strict safety threshold ($\text{Confidence} < 0.65$), the system explicitly appends a warning banner: "Low Confidence Prediction - Field Validation Required" rather than asserting false certainty [INFERENCE, ASSUMPTION].

16. MVP Scope (Hackathon Build)

In Scope (MUST BUILD & DEMONSTRATE)

• Agentic Orchestrator Engine: LangGraph-based workflow controller that interprets user query, validates input rasters, selects appropriate tool from registry, logs execution trace, and formats output [PS].
• Single-Image Baseline Module: VQA capability + Visual Grounding (bounding box output) for single optical/SAR imagery [PS].
• Multi-Image Bi-Temporal Change Engine: Natural language change description + 2D spatial change raster/bounding mask on bi-temporal image pairs [PS].
• Cross-Modal Optical-SAR Fusion Analysis: Joint information extraction from paired Sentinel-1 SAR (VV/VH) and Sentinel-2 Optical (or Cartosat/RISAT sample pairs) [PS].
• GeoTIFF Processing Pipeline: Full Python Rasterio ingestion extracting CRS, band channels, and spatial bounds [PS].
• Interactive Web UI Dashboard: Dual-map Leaflet visualizer, chat interface, live execution trace viewer, and downloadable PDF report exporter [PS].
• Domain Adaptation Proof: Fine-tuned adapter weights trained on BigEarthNet.txt [PS].

Described Only / Out of Scope (PRODUCTION FUTURE)

• Live integration with ISRO's operational VEDAS or MOSDAC real-time satellite streaming firehose [INFERENCE].
• Multi-node distributed GPU cluster scaling across petabyte-scale imagery archives [INFERENCE].
• Hyperspectral image processing (e.g., EnMAP / PRISMA 200+ bands) [INFERENCE].
• Onboard satellite edge-computing deployment hardware [INFERENCE].

17. Live Demo Simulation Plan

To deliver a flawless, high-impact demonstration during the SIH finals without relying on live external satellite APIs or unavailable government network feeds:

+---------------------------------------------------------------------------------------------------+
|                                  SIH LIVE DEMO SIMULATION FLOW                                   |
|                                                                                                   |
|  [ Preset Demo Scenario Selector ]                                                               |
|   ├── Scenario A: Single-Image Optical Grounding (VRSBench Test Sample)                           |
|   ├── Scenario B: Bi-Temporal Urban Change Analysis (LEVIR-CD / QAG-360K Pair)                    |
|   └── Scenario C: Cross-Modal Optical-SAR Flood Mapping (Cartosat-2S + RISAT Simulated Pair)      |
|                                                                                                   |
|  1. USER ACTION: Selects Scenario C or Uploads Local GeoTIFF Pair                                 |
|  2. SYSTEM DISPLAY: Shows GeoTIFF Metadata (EPSG:4326, 4-Bands Optical, 2-Bands SAR, Resolution)  |
|  3. QUERY INPUT: "Identify built-up regions and flooded areas using optical and SAR together."    |
|  4. LIVE AUDIT TRACE:                                                                             |
|     [✓] Input Validator: Optical + SAR Pair Validated (Co-registered, CRS Matched)                |
|     [✓] Agent Router: Selected Tool 'SAR_Optical_CrossModal_Fusion_v1'                            |
|     [✓] Parameter Config: Speckle_Filter = True, Confidence_Threshold = 0.70                      |
|  5. OUTPUT DISPLAY:                                                                               |
|     ├── Textual Answer: "The SAR backscatter indicates flooded areas under cloud cover in North..."|
|     ├── Map Canvas Overlay: Blue GeoJSON Polygon (Water) & Red Bounding Box (Structures)          |
|     └── Download Button: Click to export 'ISRO_SatQuery_Audit_Report.pdf'                         |
+---------------------------------------------------------------------------------------------------+

Note: All simulated data streams (e.g., ISRO SAC Cartosat/RISAT sample imagery) will be clearly disclosed to judges as pre-loaded local benchmark samples in compliance with Anti-Hallucination Rule 17 [PS].

18. Implementation Roadmap

The implementation timeline is strictly sized against the submission deadline of 30 September 2026 [PS].

====================================================================================================
MILESTONE ROADMAP (Target Deadline: 30 September 2026)
====================================================================================================

PHASE 1: Data Preparation & RS Adaptation Fine-Tuning
Duration: Weeks 1 - 2 (Completed by Sept 10, 2026)
  ├── Task 1.1: Ingest BigEarthNet.txt dataset split; set up PyTorch dataloaders [PS, RESEARCHED].
  ├── Task 1.2: Fine-tune Qwen2-VL / RemoteCLIP adapter weights on Sentinel-1/2 pairs [PS, RESEARCHED].
  └── Task 1.3: Validate performance on VRSBench, RSVQA, and CDVQA test splits [PS, RESEARCHED].

PHASE 2: Specialist Tool Development & Geospatial Pipeline
Duration: Weeks 2 - 3 (Completed by Sept 17, 2026)
  ├── Task 2.1: Build Rasterio GeoTIFF multi-band reader and array normalizer [PS, KNOWN].
  ├── Task 2.2: Develop Single-Image VQA & Grounding bounding box engine [PS].
  ├── Task 2.3: Build Bi-Temporal Siamese Change Detection & CDVQA module [PS].
  └── Task 2.4: Build Cross-Modal Optical-SAR fusion tensor aligner [PS].

PHASE 3: Agentic Controller & LangGraph Orchestration
Duration: Weeks 3 - 4 (Completed by Sept 23, 2026)
  ├── Task 3.1: Construct LangGraph state router and task intent classifier [PS, INFERENCE].
  ├── Task 3.2: Implement input validator (modality, band count, CRS matching) [PS].
  ├── Task 3.3: Embed audit logger generating structured JSON execution traces [PS].
  └── Task 3.4: Integrate all specialist tools into unified tool registry [PS].

PHASE 4: Frontend UI, Testing, Demo Polish & Final Submission
Duration: Final Days (Completed by Sept 30, 2026)
  ├── Task 4.1: Build React + Leaflet interactive dashboard with chat drawer [PS, ASSUMPTION].
  ├── Task 4.2: Perform full integration testing & offline robustness verification [INFERENCE].
  ├── Task 4.3: Prepare automated report generator (PDF export) [PS].
  └── Task 4.4: FINAL SIH SUBMISSION (30 September 2026) [PS].
====================================================================================================

19. Testing & Validation Plan

• Functional Testing: Test every query path across single optical, single SAR, optical-SAR pair, and bi-temporal pair inputs [PS]. Verify that appropriate tools are dynamically selected from the registry [PS].
• Performance Testing: Ensure query latency remains $< 3.5$ seconds for standard $512 \times 512$ tile inputs on a single GPU [INFERENCE, ASSUMPTION].
• Usability Testing: Verify that non-expert users can obtain accurate answers using conversational English without entering GIS commands [PS].
• AI Model Metrics:
  • Single-Image VQA: Evaluated on RSVQA and VRSBench-VQA using top-1 Accuracy and BLEU-4 [PS, RESEARCHED].
  • Visual Grounding: Evaluated on VRSBench-Ref using mIoU (target $> 0.60$) [PS, RESEARCHED].
  • Bi-Temporal Change: Evaluated on CDVQA and QAG-360K using F1-score for change masks and BLEU-4 for text descriptions [PS, RESEARCHED].
• False-Positive & False-Negative Testing: Pass un-changed bi-temporal pairs with extreme seasonal/lighting differences to verify the change engine does not trigger false positive change alerts [INFERENCE].
• Geospatial Projection Validation: Convert bounding box pixel outputs back to EPSG:4326 Lat/Lon coordinates using image affine matrices and overlay on OpenStreetMap tiles to verify zero spatial drift [INFERENCE, KNOWN].

20. Risk Register

21. Cost & Scalability

Hackathon MVP Cost

• Compute: ₹0 (Utilizing Google Colab Enterprise free GPU tier / Kaggle / local student RTX 3090/4090 GPUs) [ASSUMPTION].
• Storage & Hosting: ₹0 (Local Docker container deployment + Vercel / Render free tiers) [ASSUMPTION].
• Total MVP Cost: ₹0 [ASSUMPTION].

Production System Cost (ISRO Enterprise Scale)

• GPU Compute: On-premise ISRO cluster featuring 4x NVIDIA A100/H100 GPUs for multi-user concurrent inference [INFERENCE].
• Storage: Distributed Ceph / MinIO object storage for petabyte-scale satellite image archives [INFERENCE].
• Licensing: 100% Open Source Stack (PyTorch, FastAPI, React, PostgreSQL, PostGIS, GDAL) -> ₹0 Software License Fees [INFERENCE].

22. Differentiation

SatQuery AI stands out against generic hackathon submissions through four direct technical differentiators tied to ISRO requirements:

1. True Agentic Orchestration vs. Wrapper Scripts: Unlike generic LLM wrappers that hardcode model execution, SatQuery AI features a state-graph agent that dynamically inspects raster metadata (CRS, bands, resolution) and selects specialized tools with verifiable execution traces [PS, INFERENCE].
2. Native Geospatial Multi-Band Handling: Rejects simple JPEG conversion that discards spatial metadata. Ingests full 16-bit multi-spectral GeoTIFFs and dual-pol SAR (VV/VH) imagery, mapping model outputs directly back to geographic Lat/Lon coordinates [PS, KNOWN].
3. Active Optical-SAR Cross-Modal Fusion: Solves the cloud-cover limitation of optical sensors by fusing active microwave SAR backscatter with optical spectral bands, a capability explicitly mandated by ISRO [PS, RESEARCHED].
4. Auditable Visual Evidence: Complies with strict no-hallucination standards by generating downloadable JSON audit summaries and vector map overlays alongside conversational text [PS].

23. Future Scope

1. Hyperspectral Imagery Support: Expanding tool registry to support 200+ band hyperspectral data (e.g., ISRO HySIS / EnMAP) for detailed mineralogical and crop health analysis [INFERENCE].
2. 3D Elevation Change Analysis: Incorporating Cartosat DEM (Digital Elevation Model) stereo pairs for volumetric change calculation (e.g., quarrying, landslide volume estimation) [RESEARCHED, INFERENCE].
3. Edge AI Onboard Processing: Porting quantized lightweight micro-models (TensorRT-Edge) for direct execution aboard satellite payloads [INFERENCE].

24. Assumptions & Open Items Log

Consolidated Assumptions & Items Needing Validation

• [ASSUMPTION] The hackathon live demo will utilize a web interface built with React 18 and Leaflet.js running inside Docker containers.
• [ASSUMPTION] GPU compute for hackathon development will be provided by local NVIDIA student hardware or cloud GPU credits (Kaggle/Colab).
• [ASSUMPTION] The primary model backbone for VQA and fine-tuning will be Qwen2-VL-2B / RemoteCLIP due to VRAM constraints.
• [NEEDS VALIDATION] The exact geographic coordinates, resolution, and band order of ISRO/SAC's unreleased Cartosat-2S and RISAT testing imagery must be validated during final evaluation.
• [NEEDS VALIDATION] The specific evaluation metric weights (BLEU-4 vs CIDEr vs mIoU) applied by ISRO SAC judges during scoring.

Final Recommendation

Proceed to Build SatQuery AI using Approach 3 (Agentic Orchestrator + RS Tool Registry). This architecture is the only design that fully satisfies ISRO's explicit requirement for agentic model selection, cross-modal Optical-SAR analysis, bi-temporal change detection, and auditable visual trace generation [PS, INFERENCE].

Recommended Stack

• Frontend: React 18, TailwindCSS, Leaflet.js, Leaflet-GeoTIFF
• Backend & Router: FastAPI (Python 3.11), LangGraph, Pydantic
• Geospatial Data Engine: Rasterio, GDAL, PyProj, NumPy
• AI/ML Core: PyTorch, Qwen2-VL-2B (Fine-Tuned), RemoteCLIP, VisTA Siamese Change Net
• Database & Cache: PostgreSQL + PostGIS, Redis

Biggest Dependency

Access to pre-processed BigEarthNet.txt instruction splits and public benchmark downloads (VRSBench, CDVQA, RSVQA) for remote-sensing adapter training [PS, RESEARCHED].

Biggest Risk

Geospatial mis-alignment (CRS / EPSG mismatch or non-co-registered rasters) causing spatial change mask distortions during bi-temporal or cross-modal inference [INFERENCE].

Biggest Technical Challenge

Constructing an agentic controller that parses unstructured user queries and accurately outputs strict tool parameters without generating un-evaluable internal reasoning text [PS, INFERENCE].

First 5 Actions

1. Set up local Git repository and Docker container environment containing GDAL, PyTorch, and CUDA bindings [KNOWN].
2. Download BigEarthNet.txt sample split and set up Qwen2-VL-2B LoRA fine-tuning script [RESEARCHED].
3. Implement Rasterio ingestion service to read multi-band GeoTIFFs and extract affine projection metadata [KNOWN].
4. Build LangGraph state router and define tool JSON schemas for VQA, Grounding, Change, and Fusion engines [INFERENCE].
5. Develop React + Leaflet spatial visualization GUI with conversational chat drawer [ASSUMPTION].

Validation Before Coding

1. Verify that Rasterio correctly parses 4-band Cartosat-2S optical and 2-band RISAT SAR sample GeoTIFF rasters without throwing C-library exceptions [KNOWN].
2. Verify that Qwen2-VL-2B fine-tuning loss converges on BigEarthNet.txt instruction samples within available GPU VRAM limits [RESEARCHED].

Buildability Score /100

92 / 100 (High buildability for a 4–6 student team leveraging open-source foundation backbones and modular tool separation) [INFERENCE].

Confidence

HIGH (Supported by explicit PS requirements and verified external research on RS benchmarks) [PS, RESEARCHED].

Research Sources

1. BigEarthNet.txt: A Large-Scale Multi-Sensor Image-Text Dataset and Benchmark for Earth Observation

   • URL: https://arxiv.org/abs/2603.29630
   • Source Type: Academic Research Paper (arXiv, 2026)
   • Supported Claim: Primary dataset for multi-sensor Sentinel-1 SAR and Sentinel-2 optical remote-sensing adaptation with 9.6M text annotations.
   • Reliability: High (Official reference cited in ISRO PS JSON).

2. VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding

   • URL: https://vrsbench.github.io/ / https://arxiv.org/abs/2406.12384
   • Source Type: Academic Conference Paper (NeurIPS 2024)
   • Supported Claim: Public benchmark containing 29,614 images, 123,221 QA pairs, and 52,472 object references for VQA, captioning, and visual grounding.
   • Reliability: High (Peer-reviewed benchmark).

3. Show Me What and Where has Changed? Question Answering and Grounding for Remote Sensing Change Detection (VisTA / CDVQA)

   • URL: https://arxiv.org/abs/2410.23828
   • Source Type: Academic Research Paper / GitHub Repository
   • Supported Claim: Baseline dataset (QAG-360K) and VisTA model architecture for change detection VQA and spatial change mask grounding.
   • Reliability: High (Standard benchmark in RS change understanding).

4. RSVQA: Visual Question Answering for Remote Sensing Data

   • URL: https://rsvqa.sylvainlobry.com/
   • Source Type: IEEE Transactions on Geoscience and Remote Sensing (TGRS)
   • Supported Claim: Established benchmark for single-image visual question answering on Earth Observation data.
   • Reliability: High (Peer-reviewed IEEE journal paper).

5. GeoChat: Grounded Large Vision-Language Model for Remote Sensing

   • URL: https://arxiv.org/abs/2311.15826
   • Source Type: CVPR 2024 Conference Paper
   • Supported Claim: Demonstrates region-level dialogue and visual grounding capabilities in remote sensing MLLMs.
   • Reliability: High (Peer-reviewed CVPR publication).

6. Cartosat-2S & RISAT-1 Sensor Technical Specifications

   • URL: https://www.eoportal.org/satellite-missions/risat-1 / https://www.isro.gov.in/
   • Source Type: Official EO Portal & ISRO Documentation
   • Supported Claim: Cartosat-2S provides 0.65m PAN + 1.6m 4-band MS imagery; RISAT-1/EOS-4 provides active C-band SAR with dual polarizations (VV/VH).
   • Reliability: High (Official space agency specification).