AI-Powered Geological, Mining and other Reporting Solution for CMPDI/CIL subsidiaries

Description:	Background: CMPDI/CIL subsidiaries play a key role in providing geological and mining information to the Ministry of Coal and responding to parliamentary and high-priority administrative inquiries. These reports require compilation of data from scanned PDFs, digital documents, spreadsheets, images, and historical archives. The current workflow is largely manual, resulting in: • High dependence on individual expertise • Delay in generating reports and analytics • Higher probability of manual errors • Limited ability to quickly retrieve insights when required Objectives: • Deploy an automated platform for AI-assisted geological, mining and any other production figures document processing and reporting. • Enhance data validation, consistency, and traceability across historical and contemporary datasets. • Build an efficient, scalable foundation for future digital transformation initiatives within each CIL subsidiary and the Ministry of Coal. Desired Outcomes: The solution should be implemented in structured phases, including requirement analysis, data digitization and pre-processing, platform development, system testing, integration with CIL subsidiary workflows, training, and continuous enhancement to ensure scalability and long-term adoption. 1. Automated Report Generation Platform 2. Automated Word Cloud and Topic Identification Module 3. AI-Based Query and Response System Expected Benefits: • Reduction in report preparation time as less as it can be, quantified in percentage. • Maximum accuracy, calculated in percentage in structured extraction and report generation. • Maximum automation, calculated in percentage of repetitive reporting and response workflows. • Faster response to high-level inquiries and parliamentary questions • Improved data accessibility, transparency, and standardization • Strengthened operational efficiency and informed decision-making using historical insights and AI-generated recommendations Impact: The proposed system should significantly modernize CMPDI/CIL subsidiaries reporting ecosystem, reduce dependency on manual processes, improve response timelines, and strengthen the coal sector’s capability to support governance, policy planning, and operational excellence.
Organization:	Ministry of Coal
Department:	Coal India Limited
Category:	Software
Theme:	Smart Automationtd>
Youtube Link:	Not provided
Dataset Link:	Not provided
Contact info:	Not provided

1. Executive Snapshot

Summary

The Central Mine Planning & Design Institute (CMPDI) and Coal India Limited (CIL) subsidiaries manage massive volumes of unstructured, historical, and contemporary geological and operational data across India's coalfields [KNOWN]. Current workflows rely on manual extraction from legacy scanned PDFs, drill bore logs, maps, and spreadsheets to draft technical reports and answer urgent parliamentary inquiries, causing high turnaround latency and risk of human error [PS]. This proposed AI-Powered Geological, Mining & Reporting Platform establishes an enterprise-grade document intelligence, semantic search, and automated reporting engine [INFERENCE]. By combining layout-aware Document AI (OCR + table parsing), Domain-Specific Retrieval-Augmented Generation (RAG), and a hybrid rule/ML analytics engine, the system ingests legacy scanned archives, extracts tabular bore-log and production parameters, identifies emerging technical topics, and generates formal administrative and geological reports with end-to-end citation traceability [INFERENCE].

Metadata Table

Attribute	Details
S. No.	23
PS Number	SIH26023 [PS]
Title	AI-Powered Geological, Mining and other Reporting Solution for CMPDI/CIL subsidiaries [PS]
Organization	Ministry of Coal [PS]
Department	Coal India Limited [PS]
Category	Software [PS]
Theme	Smart Automation [PS]
Deadline	30 September 2026 [PS]
Ideas Counter	0/500 [PS]
Dataset Link	N/A (Not provided in PS) [PS]
YouTube Link	N/A (Not provided in PS) [PS]
Scraped Date	2026-09-05 [PS]

Fundamental System Objective

To build an intelligent, layout-aware document ingestion, semantic analytics, and automated reporting system for CMPDI/CIL that automates data extraction from multi-modal mining archives, accelerating parliamentary/administrative inquiry responses and technical report generation with 100% data traceability [INFERENCE].

2. Problem Deconstruction

Explicit Requirements & Sub-Objectives

1. Multi-Source Data Processing [PS]: System must process scanned PDFs, digital text documents, spreadsheets (XLSX/CSV), images (geological maps/diagrams), and historical archives.
2. Data Validation, Consistency & Traceability [PS]: Enhance data validation across historical (1970s–1990s) and contemporary datasets while preserving verifiable audit trails back to the source page and coordinate.
3. Automated Report Generation Platform (Module 1) [PS]: Automated creation of structured geological, mining, and production reports tailored for CIL subsidiaries and Ministry of Coal (MoC) formats.
4. Automated Word Cloud & Topic Identification Module (Module 2) [PS]: Identify major geological/operational themes, emerging trends, and keywords across large document repositories.
5. AI-Based Query & Response System (Module 3) [PS]: Natural language interface to rapidly answer high-priority administrative inquiries and Parliamentary Questions (Lok Sabha / Rajya Sabha Starred/Unstarred questions).
6. Quantified Performance & Efficiency Benchmarks [PS]:
   • Percentage reduction in report preparation time.
   • Percentage accuracy in structured data extraction and report generation.
   • Percentage automation of repetitive reporting and query-response workflows.

Deconstructed Requirement Analysis

Req #	Source Text	Interpretation	Implementation Implication	Confidence Tag
R1	"compilation of data from scanned PDFs, digital documents, spreadsheets, images, and historical archives" [PS]	Must ingest heterogeneous, multi-format structured & unstructured files including low-dpi scanned bore logs.	Implement a multi-modal preprocessing pipeline (Tesseract/DocTR OCR + PDF layout parser + OpenPyXL reader).	[PS] / [INFERENCE]
R2	"Enhance data validation, consistency, and traceability across historical and contemporary datasets" [PS]	System cannot hallucinate; extracted numbers must be verified against physical layout and linked back to source bounding boxes.	Build a hybrid validation engine (regex/rule validator for depth/GCV numbers) + source page image highlight overlay in UI.	[PS] / [INFERENCE]
R3	"1. Automated Report Generation Platform" [PS]	Automated drafting of standard technical reports (Geological Reports, Project Reports, Monthly Production Summaries).	Template-driven NLG engine combined with structured database SQL queries and LLM text summarization.	[PS] / [INFERENCE]
R4	"2. Automated Word Cloud and Topic Identification Module" [PS]	Visual and analytical topic discovery across legacy documents.	TF-IDF / BERTopic / LDA topic modeling engine with dynamic interactive D3.js Word Cloud visualization.	[PS] / [INFERENCE]
R5	"3. AI-Based Query and Response System" [PS]	Fast answer retrieval for parliamentary/VIP inquiries with precise citations.	Domain-specific RAG pipeline with Hybrid Search (BM25 + Dense Vector embeddings) and strict citation constraints.	[PS] / [INFERENCE]
R6	"Reduction in report preparation time as less as it can be... Maximum accuracy... Maximum automation" [PS]	Performance must be quantified via concrete metrics (time saved %, extraction accuracy %, automation %).	Build an automated evaluation benchmark suite tracking WER/CER, Precision/Recall, and time-to-generate logs.	[PS] / [INFERENCE]

3. Stakeholders & User Needs

Stakeholder	Goal	Current Pain	System Need
Ministry of Coal (MoC) Officials	Quickly answer Parliamentary Questions (PQs) and policy inquiries within tight statutory deadlines (often 24–48 hours) [KNOWN].	High latency in getting aggregated figures across 8 CIL subsidiaries; risk of submitting inaccurate figures to Parliament [KNOWN].	Instant natural-language Q&A interface with verifiable citations, subsidiary-wise breakdown, and exportable draft answers [INFERENCE].
CMPDI Exploration Geologists & Mine Planners	Draft Geological Reports (GR) and Project Reports (PR) incorporating historical drill logs, seam thickness, GCV, and UNFC resource estimates [KNOWN].	Manual digitization of thousands of legacy scanned bore logs; high dependency on individual memory/expertise [PS].	Automated OCR extraction of tabular bore-log data, seam correlation assistance, and pre-filled GR template generation [INFERENCE].
CIL Subsidiary Managers (ECL, BCCL, CCL, NCL, WCL, SECL, MCL)	Compile monthly/annual coal production, overburden removal (OBR), and dispatch reports for HQ [KNOWN].	Disparate formats across areas/mines; manual collation in Excel leading to version mismatch and calculation errors [PS].	Centralized reporting dashboard with automated file parsing, anomaly validation, and standardized executive reports [INFERENCE].
Parliamentary Affairs Cell (CIL HQ)	Draft responses for Starred/Unstarred Lok Sabha/Rajya Sabha questions, Standing Committee queries, and Zero Hour submissions [RESEARCHED].	Coordinating inputs from multiple regional institutes under extreme time pressure [RESEARCHED].	Single-window query search with instant retrieval of historical parliamentary responses and underlying raw evidence [INFERENCE].
System Administrators / IT Personnel	Maintain platform security, user access roles, and seamless data integration with future digital systems [PS].	Disjointed legacy silos with no standardized APIs or audit logs [PS].	Role-Based Access Control (RBAC), on-premise/hybrid deployment support, ISO 27001 compliant audit logging [INFERENCE].

4. Requirements Traceability

Proposed System Feature	Justifying Requirement	Core Mechanism	Target Outcome
Multi-Modal Document Preprocessor	R1 (Multi-Source Data Processing) [PS]	OCR + Layout Parser + Image Preprocessing (Denoising/Binarization).	Standardized ingestion of scanned bore logs, PDFs, and spreadsheets.
Layout-Aware Table Extractor	R1 [PS], R2 (Data Traceability) [PS]	TableNet / PaddleOCR + Bounding Box Mapping.	Structured JSON conversion of complex drill logs and production tables with page coordinates.
Deterministic Data Validation Engine	R2 (Validation & Traceability) [PS]	Hardcoded geological validation rules (e.g., Moisture % + Ash % + Volatile Matter % + Fixed Carbon % = 100%).	100% mathematical accuracy on extracted tabular values; eliminates LLM hallucination in numeric figures.
Template-Driven Auto-Report Generator	R3 (Automated Report Generation Platform) [PS]	Jinja2 templating + Structured SQL aggregation + LLM text narrative drafting.	Instant draft generation of Geological Reports, PRs, and Subsidiary Production Summaries.
BERTopic & Interactive Word Cloud Module	R4 (Topic Identification & Word Cloud) [PS]	BERTopic / Transformer-based embeddings + D3.js interactive canvas.	Dynamic discovery of trending operational topics (e.g., "land acquisition delay", "seam XIV fault").
Hybrid RAG Query-Response Engine	R5 (AI Query & Response System) [PS]	Hybrid Dense Vector (BGE-M3) + Sparse Search (BM25) with Reranker (BGE-Reranker-Large).	Sub-second, verifiable answers to parliamentary inquiries with exact source file and page links.
Visual Bounding-Box Evidence Overlay	R2 [PS], R5 [PS]	PDF.js UI viewer rendering highlighted bounding boxes over original scanned pages.	100% data auditability; human-in-the-loop verification of AI claims.
Automated Metric Tracking Dashboard	R6 (Quantified Benchmarks) [PS]	System Telemetry + Automated Precision/Recall & Latency Logging.	Real-time display of time saved (%), extraction accuracy (%), and workflow automation (%).

5. Constraints & Environment Realities

1. Legacy Document Quality (Field Conditions): Mining archives in CMPDI Regional Institutes (RIs in Ranchi, Dhanbad, Bilaspur, Nagpur, Asansol, Singrauli, Bhubaneswar) contain physical reports dating back to the 1970s–1990s [KNOWN]. These files suffer from carbon-copy fading, handwritten annotations, low-dpi scans (100–150 DPI), skewing, and oil/water stains [INFERENCE].
2. Network & Offline Realities: While CIL HQ and CMPDI HQ (Ranchi) have fiber connectivity, remote exploration camps and open-cast mine sites in interior coalfields often face erratic broadband and low network bandwidth [KNOWN]. System must support local client caching and asynchronous background synchronization [INFERENCE].
3. Data Security & Air-Gapped Hosting: Ministry of Coal regulations strictly prohibit sending confidential geological reserve estimates, coal seam grades, or unreleased commercial block data to public third-party APIs (e.g., OpenAI, Anthropic cloud endpoints) [INFERENCE]. System MUST run entirely on self-hosted, air-gapped infrastructure [ASSUMPTION].
4. Domain Terminology & Acronym Density: Mining documents are saturated with domain-specific Indian coal terminology (e.g., UNFC 331/332/333, GCV Grade G1–G17, OBR (Overburden Removal), PRC (Peak Rated Capacity), MDO, Borehole/BH, Seam IV Top, Jhama, Coking/Non-Coking) [KNOWN]. Standard off-the-shelf NLP models fail without custom dictionary / domain adaptation [INFERENCE].
5. Hardware Availability for Hackathon: Student team operates on standard laptop/cloud free-tier GPU resources (e.g., NVIDIA T4 / RTX 3090/4090 on Colab or AWS instance) [ASSUMPTION]. AI architecture must be optimized for parameter efficiency (e.g., 7B/8B quantized LLMs, lightweight OCR) [INFERENCE].

6. Research Questions

1. How do legacy CMPDI Geological Reports (GR) format bore-log tables, and what layout extraction models handle multi-line headers, split tables across pages, and handwritten drill depth annotations best?
2. What specific regulatory and parliamentary report templates are utilized by CIL Parliamentary Affairs Cell for Lok Sabha / Rajya Sabha Starred Questions?
3. How can we guarantee zero hallucination in numeric coal production and reserve figures retrieved via RAG?
4. What is the optimal hybrid search strategy (combining keyword BM25 for alphanumeric borehole IDs like BH-NK-102 with vector embeddings for semantic policy questions)?
5. Which open-source, small language model (8B parameter class) delivers the highest accuracy for Hindi/English domain summarization while running efficiently on modest GPU infrastructure?

7. Data Requirements & Sources

Data Type	Purpose	Ideal Real-World Source	Hackathon Substitute	Access Method	Availability	Privacy Sensitivity	Confidence Tag
Geological Reports (GR) & Bore Logs	Core training/testing data for OCR, table extraction, and geological RAG.	CMPDI Central Archives / Coal Atlas / National Geoscience Data Repository (NGDR) [RESEARCHED].	Publicly released CMPDI/MoC draft Geological Reports available on Ministry portals (e.g., Phatehpur-A Block, Mugoli Extension Block) [RESEARCHED].	Public Web Scraping / Direct PDF Download [RESEARCHED].	Available (Publicly released versions) [RESEARCHED].	Medium (Un-auctioned commercial blocks are confidential) [KNOWN].	[RESEARCHED]
Parliamentary Q&A Archives	Training and testing query-response engine for parliamentary inquiries.	Parliament of India (Lok Sabha/Rajya Sabha Q&A database) & MoC Parliament QA Portal [RESEARCHED].	Scraped MoC Lok Sabha / Rajya Sabha Coal Ministry Q&A PDFs (2018–2025) [RESEARCHED].	Public Web Scraping / API download [RESEARCHED].	High (Fully public record) [KNOWN].	Low (Public domain) [KNOWN].	[RESEARCHED]
Subsidiary Monthly Production & OBR Sheets	Ingestion of tabular production figures across ECL, BCCL, CCL, NCL, WCL, SECL, MCL.	CIL Central ERP System (SAP Environment) [RESEARCHED].	CIL Annual Reports, Monthly Press Releases (PIB), CCO (Coal Controller Organisation) Monthly Reports [RESEARCHED].	Download PDF/XLSX from CIL/PIB portals [RESEARCHED].	High (Historical aggregates available) [KNOWN].	Low (Aggregated public data) [KNOWN].	[RESEARCHED]
Mining Domain Dictionary & Taxonomy	Vocabulary grounding for topic modeling and prompt engineering.	Indian Standard Code (ISP 2017/2019) for Coal Resource Estimation & UNFC Codes [RESEARCHED].	Extracted term list from ISP 2019 norms, CIL annual reports, and geological glossaries [RESEARCHED].	Manual Curation + Custom NLP Vocabulary Builder [INFERENCE].	High [KNOWN].	Low [KNOWN].	[RESEARCHED]
Synthetic Anomaly/Error Borehole Logs	Testing mathematical validation engine against corrupted/fuzzy OCR outputs.	Generated internal simulation script.	Custom Python script corrupting text/table numbers (e.g., replacing '0' with 'O', injecting mismatched total depths).	Scripted Generation [ASSUMPTION].	Unlimited [INFERENCE].	None [INFERENCE].	[ASSUMPTION]

Data Categorization Breakdown

• A. Explicitly Mentioned Data [PS]: Scanned PDFs, digital documents, spreadsheets (XLSX), images, historical archives.
• B. Inferred Data [INFERENCE]: Borehole drill logs, seam correlation tables, UNFC reserve estimates (331, 332, 333), Lok Sabha/Rajya Sabha Q&A archives, Coal Controller Organisation (CCO) monthly returns.
• C. Data to Collect [INFERENCE]: Public MoC parliamentary PDFs, CIL annual financial/operational reports, released CMPDI block summaries.
• D. Synthetic Data [ASSUMPTION]: Synthetic noisy OCR tables, simulated multi-subsidiary monthly production spreadsheets with introduced calculation anomalies.
• E. Data Needing Validation [NEEDS VALIDATION]: Access to live CMPDI CBM/Coal Atlas GIS database schemas and live internal CIL SAP ERP REST/ODATA APIs.

8. Existing Solutions & Gap Analysis

Existing System	Description	Strengths	Weaknesses	Relevance	Remaining Gap	Source	Confidence
CIL Enterprise Resource Planning (SAP ERP)	Operational ERP implemented across CIL subsidiaries for finance and production tracking [RESEARCHED].	Handles structured real-time transactional data across subsidiaries [RESEARCHED].	Cannot parse unstructured scanned PDFs, legacy geological reports, or historical drill logs [INFERENCE].	Core back-end operational source [RESEARCHED].	Lacks AI-driven document intelligence, OCR, and unstructured parliamentary RAG [INFERENCE].	CIL Annual Reports [RESEARCHED]	[RESEARCHED]
National Geoscience Data Repository (NGDR) Portal	Ministry of Mines / GSI flagship platform hosting exploration data converted to GIS [RESEARCHED].	Standardized mineral exploration metadata (MERT format) [RESEARCHED].	High-level spatial overview; lacks automated report drafting and natural language parliamentary Q&A [INFERENCE].	Spatial metadata standards [RESEARCHED].	No automated Word Cloud topic discovery, report generation, or instant Q&A engine [INFERENCE].	NGDR Portal (geodataindia.gov.in) [RESEARCHED]	[RESEARCHED]
Generic Document RAG Solutions (e.g., Default LangChain/LlamaIndex)	Standard open-source vector search over PDF uploads [KNOWN].	Quick to prototype; good at basic semantic text retrieval [KNOWN].	Fails on complex tabular drill logs, low-dpi scans, numeric precision, and domain-specific mining acronyms; suffers from hallucinations [INFERENCE].	Baseline AI technology stack [KNOWN].	Lacks layout-aware table extraction, mathematical validation rules, and specialized report template NLG [INFERENCE].	Industry Standards [KNOWN]	[KNOWN]
Traditional OCR Tools (Tesseract Default)	Open-source optical character recognition engine [KNOWN].	Lightweight, offline, widely supported [KNOWN].	Destroys table structures, fails on skewed/stained historical documents, yields high Word Error Rate (WER) on low-contrast text [INFERENCE].	Basic text extraction [KNOWN].	Requires advanced preprocessing, layout detection (LayoutLMv3), and post-OCR correction [INFERENCE].	Public Literature [KNOWN]	[KNOWN]

9. Candidate Solution Approaches

Approach 1: Heuristic & Traditional OCR Pipeline (Rule-Based + Tesseract + SQL)

• Mechanism: Standard Tesseract OCR extracts raw text. Custom Regex and Python heuristics extract table structures and store them in a relational SQLite/PostgreSQL database. SQL queries generate reports, and basic keyword searching (BM25/TF-IDF) handles queries.
• Data Requirements: Raw PDFs and pre-defined regex rules.
• Complexity: Low.
• Explainability: 100% deterministic.
• Cost: Near zero (runs on basic CPU).
• Scalability: Low (breaks whenever document layout varies slightly).
• Hackathon Feasibility: High (easy to build).
• Production Feasibility: Poor (fails on non-standard, scanned, or complex multi-page geological reports).
• Risks: High brittleness to document layout variations; inability to answer complex semantic parliamentary queries.

Approach 2: Native Multimodal LLM End-to-End Pipeline (Cloud Vision LLM)

• Mechanism: Direct processing of document images through commercial Multimodal LLMs (e.g., GPT-4o / Claude 3.5 Sonnet) for vision extraction, topic modeling, and report drafting via long-context prompts.
• Data Requirements: Document page images fed directly into API prompts.
• Complexity: Medium (simple codebase, heavy reliance on external API).
• Explainability: Low (black-box model inference).
• Cost: Extremely High (token/image API costs scale exponentially with thousands of multi-page reports).
• Scalability: High computationally, but blocked by air-gap enterprise governance.
• Hackathon Feasibility: High for small demos, but violates air-gapped security rules.
• Production Feasibility: Infeasible (violates Ministry of Coal air-gap/data security mandates, high operational cost).
• Risks: Severe privacy/security non-compliance; hallucination of numeric coal figures; vendor lock-in.

Approach 3: Enterprise Hybrid Pipeline (Layout-Aware Doc AI + Hybrid RAG + Deterministic Validation + Template NLG) — RECOMMENDED

• Mechanism:
  1. Ingestion: Preprocessing (denoising/deskewing) + DocTR/PaddleOCR + LayoutLMv3 for layout-aware region detection (text vs table vs map image).
  2. Structured Data Processing: TableNet/TATR extracts tables into structured JSON. A deterministic python validation engine verifies mathematical relationships (depth totals, seam percentages).
  3. Semantic Engine: Chunking aware of document hierarchy; indexed in Hybrid Qdrant/PgVector (Dense embeddings via bge-m3 + Sparse BM25 keywords).
  4. Topic Engine: BERTopic + TF-IDF with D3.js interactive Word Cloud visualization.
  5. Generation Engine: Jinja2 templates + fine-tuned/quantized Llama-3.1-8B-Instruct (or Mistral-7B) for narrative report generation and citation-backed parliamentary query responses.
• Data Requirements: Historical scanned PDFs, spreadsheets, parliamentary archives, domain dictionary.
• Complexity: Medium-High.
• Explainability: High (strict source page bounding box highlighting + deterministic numeric check).
• Cost: Low (runs on self-hosted local GPUs; zero external API fees).
• Scalability: High (horizontal scaling via containerized microservices).
• Hackathon Feasibility: High (can be containerized and run on local GPU/Colab).
• Production Feasibility: Excellent (meets air-gap security, zero hallucination on numbers, scalable).
• Risks: OCR errors on extremely degraded 1970s scans (mitigated by human-in-the-loop verification UI).

10. Recommended Approach & Why

Selection

Approach 3: Enterprise Hybrid Pipeline (Layout-Aware Doc AI + Hybrid RAG + Deterministic Validation + Template NLG) is selected as the winning architecture.

Why Alternatives Were Rejected

• Approach 1 (Traditional OCR + Regex): Rejected because legacy mining documents from CMPDI contain dozens of different table formats, hand-drawn annotations, and complex text arrangements. Heuristic regex rules break whenever a new subsidiary format is introduced [INFERENCE].
• Approach 2 (Pure Multimodal Cloud LLM): Rejected because sending confidential geological reports and un-auctioned coal block data to public third-party APIs violates Ministry of Coal data governance mandates [INFERENCE]. Furthermore, pure LLMs frequently hallucinate precise numeric drill depths or production metrics, which is unacceptable for parliamentary submissions [INFERENCE].

Why Approach 3 Wins

1. Zero-Hallucination Guarantees for Numbers: Tabular data (bore logs, GCV values, production MT) bypasses LLM generative text and is parsed directly into structured JSON, validated deterministically with mathematical rules, and stored in relational tables [INFERENCE].
2. Complete Traceability: Every extracted claim in an auto-generated report or parliamentary answer links directly to a bounding-box coordinate on the original scanned page [INFERENCE].
3. Air-Gap Ready: The entire stack uses open-source models (DocTR, BGE-M3, Llama-3.1-8B-Instruct) capable of running completely offline on CIL/CMPDI enterprise hardware [INFERENCE].

11. Technical Architecture

Layered Architecture Diagram

+-----------------------------------------------------------------------------------+
|                                  USER INTERFACE                                   |
|   React.js Web App | PDF.js Bounding Box Viewer | D3.js Word Cloud & Topics       |
|   Parliamentary Inquiry Assistant | Report Generator UI | Executive Dashboard        |
+-----------------------------------------------------------------------------------+
                                         |
                                         v (REST / WebSocket API)
+-----------------------------------------------------------------------------------+
|                               API & GATEWAY LAYER                                 |
|   FastAPI Gateway | OAuth2 + JWT Auth | Role-Based Access Control (RBAC)           |
|   Rate Limiter | Request Logger | Async Task Manager (Celery + Redis)           |
+-----------------------------------------------------------------------------------+
                                         |
          +------------------------------+------------------------------+
          |                              |                              |
          v                              v                              v
+-------------------+          +-------------------+          +-------------------+
| DOCUMENT AI &     |          | AI SEARCH & QUERY |          | REPORT GENERATION |
| INGESTION ENGINE  |          | ENGINE (RAG)      |          | ENGINE            |
|                   |          |                   |          |                   |
| - OpenCV Preproc  |          | - BGE-M3 Embedder |          | - Jinja2 Template |
|   (Deskew/Denoise)|          | - Sparse BM25     |          |   Formatter       |
| - DocTR / Paddle  |          | - Qdrant Vector DB|          | - Llama-3.1-8B    |
|   Layout Parser   |          | - BGE-Reranker    |          |   Narrative LLM   |
| - TATR Table      |          | - Strict Citation |          | - PyPDF/Docx      |
|   Structure Rec.  |          |   Constraint      |          |   Exporter        |
+-------------------+          +-------------------+          +-------------------+
          |                              |                              |
          +------------------------------+------------------------------+
                                         |
                                         v
+-----------------------------------------------------------------------------------+
|                           STORAGE & ANALYTICS LAYER                               |
| - PostgreSQL (Metadata, Extracted Tabular Logs, User Sessions, Audit Trails)      |
| - Qdrant (Vector Embeddings & Document Chunks)                                    |
| - BERTopic Engine (Topic Mining & Automated Term Extraction)                      |
| - Local MinIO / S3 Object Storage (Raw PDFs, TIFFs, Extracted Table JSONs)        |
+-----------------------------------------------------------------------------------+

Detailed Data Flow Narrative

1. Ingestion & Preprocessing: A user uploads a multi-page PDF/scanned TIFF (e.g., a 1985 Geological Report for Korba Coalfield). OpenCV automatically corrects image rotation, deskews, and enhances contrast [INFERENCE].
2. Layout & OCR Extraction: DocTR / LayoutLMv3 segments each page into structural blocks: text headers, narrative paragraphs, tabular drill logs, and geological map images [INFERENCE].
3. Table Parsing & Deterministic Validation: Tabular regions are processed by Table Transformer (TATR). The extracted rows (e.g., Borehole ID, Seam Name, Depth From, Depth To, Thickness, Ash %, GCV) are output as structured JSON. A Python rule validator verifies that Depth To - Depth From = Thickness [INFERENCE]. If a discrepancy occurs, the row is flagged for human review [INFERENCE].
4. Indexing & Embedding: Narrative text chunks (preserving page and paragraph metadata) are vectorized using bge-m3 and indexed into Qdrant alongside BM25 sparse lexical tokens [INFERENCE].
5. Topic Mining & Word Cloud: BERTopic processes all ingested narrative text across selected document collections, generating topic clusters, key terms, and word frequency vectors for the D3.js Word Cloud visualization [INFERENCE].
6. Query & Parliamentary Response: When an official submits a query (e.g., "What is the proved coal reserve in seam IV of Mugoli block and what was the 2023 production?"), the hybrid retriever pulls top candidate chunks. BGE-Reranker scores relevance. Llama-3.1-8B-Instruct formats a concise draft response, explicitly linking every statement to [Doc #12, Page 4, Table 2] [INFERENCE].
7. Interactive Audit & Export: The UI displays the response alongside a PDF viewer that highlights the exact bounding boxes on the original scanned document for instant verification. The report or answer can be exported as PDF/DOCX [INFERENCE].

12. Technology Stack

Layer	Selected Technology	Why Selected	Alternative Considered	Why Alternative Rejected
Frontend UI	React.js + Tailwind CSS + PDF.js + D3.js	Provides fast, responsive UI, native PDF page rendering with SVG bounding box overlays, and rich interactive Word Cloud visualizations [INFERENCE].	Angular / Vue.js	Lacks as rich an ecosystem for specialized canvas/PDF annotation libraries [INFERENCE].
API Backend	FastAPI (Python 3.11)	Asynchronous performance, native OpenAPI documentation, seamless integration with Python ML/AI ecosystem [INFERENCE].	Node.js (Express)	Heavy ML model binding overhead in JavaScript [INFERENCE].
OCR & Document AI	DocTR (Mindee) + PaddleOCR + TATR	State-of-the-art open-source OCR and table extraction, runs 100% offline on GPU/CPU [INFERENCE].	Tesseract Default / Cloud Vision API	Tesseract fails on complex tables; Cloud APIs violate air-gap constraints [INFERENCE].
Vector Database	Qdrant (Self-Hosted Docker)	Extremely fast, supports hybrid search (dense + sparse vectors natively), robust payload filtering by subsidiary/year [INFERENCE].	Pinecone / Milvus	Pinecone is cloud-only; Milvus has higher memory resource overhead for small deployments [INFERENCE].
Relational Database	PostgreSQL 16 + PostGIS	Enterprise-grade reliability for relational tables, extracted bore logs, user roles, spatial coalfield coordinates [INFERENCE].	MySQL / MongoDB	Lacks native spatial geospatial extensions (PostGIS) critical for mining coordinates [INFERENCE].
Topic Modeling Engine	BERTopic + c-TF-IDF	Transformer-backed topic clustering that generates readable, contextually accurate topics and key terms [INFERENCE].	LDA (Gensim)	Classical LDA yields poor, noisy topics on domain-specific technical jargon [INFERENCE].
LLM Inference Engine	vLLM / Ollama with Llama-3.1-8B-Instruct (4-bit / 8-bit quantized)	Ultra-fast token generation, low VRAM footprint (~6GB VRAM for 4-bit AWQ), open weights for air-gapped hosting [INFERENCE].	Public OpenAI GPT-4o / Claude	Public APIs violate Ministry of Coal data confidentiality regulations [INFERENCE].
Task Queue & Cache	Celery + Redis	Reliable background execution of heavy OCR jobs, document parsing, and caching vector search results [INFERENCE].	RabbitMQ / Kafka	RabbitMQ is harder to set up; Kafka is over-engineered for hackathon scope [INFERENCE].

13. AI/ML Justification

Detailed AI Component Breakdown

1. Layout-Aware Document OCR & Region Parser (DocTR + LayoutLMv3)

• Purpose: Segment complex geological report pages into narrative text, structured tables, and map image blocks [INFERENCE].
• Model Type: Object Detection & Layout Vision Transformer [INFERENCE].
• Input: High-resolution page image (PNG/TIFF) [INFERENCE].
• Output: Bounding box coordinates with layout classification tags (table, text, header, figure) [INFERENCE].
• Data Required: PubLayNet + synthetic fine-tuning on 100 CMPDI report pages [ASSUMPTION].
• Why AI is Preferable: Rule-based heuristics fail because legacy geological reports have variable column layouts, inline headers, and floating tables [INFERENCE].
• Evaluation: Mean Average Precision (mAP @ IoU 0.5) for layout segmentation [INFERENCE].
• Failure Mode: Misidentifying a small multi-column table as regular text [INFERENCE].
• Human Oversight: UI visual bounding box overlay allows users to re-tag unparsed table regions with a drag-and-drop tool [INFERENCE].

2. Table Structure Recognition Engine (TATR / Table Transformer)

• Purpose: Parse row and column grid boundaries inside extracted table regions to produce structured JSON/CSV [INFERENCE].
• Model Type: Deep Learning Vision Transformer for Table Structure Recognition [INFERENCE].
• Input: Cropped image of a table region [INFERENCE].
• Output: Row/Column indices, cell bounding boxes, merged cell flags [INFERENCE].
• Data Required: Finetuned on ICDAR/PubTabNet + 50 scanned borehole log tables [ASSUMPTION].
• Why AI is Preferable: Fixed grid lines are frequently missing, faded, or broken in scanned 1980s drill logs [INFERENCE].
• Evaluation: Cell-level F1-Score & Tree-Edit-Distance-Based Similarity (TEDS) [INFERENCE].
• Failure Mode: Multi-line header misalignment [INFERENCE].
• Human Oversight: Tabular grid editor in UI for manual cell split/merge correction before saving to DB [INFERENCE].

3. Hybrid Semantic Search & RAG Engine (BGE-M3 + BGE-Reranker + Llama-3.1-8B)

• Purpose: Retrieve relevant historical blocks and answer complex policy/technical questions with precise page citations [INFERENCE].
• Model Type: Dense/Sparse Bi-Encoder Embedder + Cross-Encoder Reranker + Causal LLM Generator [INFERENCE].
• Input: Natural language user query + Top-K retrieved document chunks [INFERENCE].
• Output: Formatted answer with inline metadata citations [Document, Page, Paragraph] [INFERENCE].
• Data Required: Chunked geological text, parliamentary QA archives, domain glossary [INFERENCE].
• Why AI is Preferable: Keyword search misses conceptual queries like "What structural disturbances affect seam IV in Singrauli?" where exact terms vary [INFERENCE].
• Evaluation: Context Recall, Context Precision, Faithfulness (Ragas Framework) [INFERENCE].
• Failure Mode: Retrieving plausible but outdated block figures [INFERENCE].
• Human Oversight: Every response includes direct click-to-view links to source pages; user feedback thumbs up/down logs instances [INFERENCE].

4. Word Cloud & Topic Identification Module (BERTopic)

• Purpose: Extract key themes, operational bottlenecks, and geological terms across large document archives [INFERENCE].
• Model Type: Contextual Transformer Embeddings + HDBSCAN Clustering + Class-based TF-IDF [INFERENCE].
• Input: Array of document text chunks [INFERENCE].
• Output: Topic clusters, term frequency weights, topic hierarchy trees [INFERENCE].
• Data Required: Narrative text extracted from ingested reports [INFERENCE].
• Why AI is Preferable: Simple keyword frequency ignores synonyms and context (e.g., grouping "faulting", "folding", "tectonic disturbance" under "Geological Complexity") [INFERENCE].
• Evaluation: Topic Coherence (c_v score) [INFERENCE].
• Failure Mode: Creating duplicate or overly granular topic categories [INFERENCE].
• Human Oversight: Administrators can merge or rename topic categories in the dashboard [INFERENCE].

Where AI Should NOT Be Used

• Mathematical Aggregations & Reserve Totals: Total coal reserves, average seam thickness, and sum of monthly production figures MUST NOT be calculated by LLMs [INFERENCE]. These must be computed via SQL SUM(), AVG() functions on validated database tables [INFERENCE].
• User Authentication & Authorization: Permission checks for sensitive mining block data must rely strictly on standard RBAC code, never LLM decisions [INFERENCE].

14. Data Architecture & Offline/Low-Network Design

Database Schema Design (PostgreSQL)

-- Main Document Metadata Table
CREATE TABLE documents (
    document_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    title VARCHAR(255) NOT NULL,
    subsidiary VARCHAR(50) NOT NULL, -- ECL, BCCL, CCL, NCL, WCL, SECL, MCL, CMPDI
    report_type VARCHAR(50) NOT NULL, -- Geological_Report, Project_Report, Monthly_Production, Parliamentary_QA
    publication_year INT,
    file_path VARCHAR(512) NOT NULL,
    ocr_status VARCHAR(20) DEFAULT 'PENDING', -- PENDING, COMPLETED, FAILED
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Extracted Borehole Tabular Data Table (Deterministic Storage)
CREATE TABLE borehole_logs (
    log_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    document_id UUID REFERENCES documents(document_id),
    borehole_number VARCHAR(50) NOT NULL,
    coalfield VARCHAR(100) NOT NULL,
    block_name VARCHAR(100) NOT NULL,
    seam_name VARCHAR(50) NOT NULL,
    depth_from NUMERIC(8,2) NOT NULL,
    depth_to NUMERIC(8,2) NOT NULL,
    thickness NUMERIC(6,2) NOT NULL,
    ash_percentage NUMERIC(5,2),
    moisture_percentage NUMERIC(5,2),
    gcv_kcal_kg NUMERIC(7,2),
    unfc_code VARCHAR(10), -- 331 (Proved), 332 (Indicated), 333 (Inferred)
    source_page_num INT NOT NULL,
    bounding_box JSONB, -- Coordinates on page image [x, y, w, h]
    validation_status VARCHAR(20) DEFAULT 'VALID' -- VALID, ANOMALY_FLAGGED
);

-- Document Chunks for Vector Search & Citation
CREATE TABLE document_chunks (
    chunk_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    document_id UUID REFERENCES documents(document_id),
    page_number INT NOT NULL,
    paragraph_number INT NOT NULL,
    content TEXT NOT NULL,
    qdrant_point_id UUID NOT NULL
);

Offline & Low-Network Edge Strategy

1. Local IndexedDB Caching (Progressive Web App): The React frontend uses Service Workers and IndexedDB to cache recent search results, report templates, and draft parliamentary responses locally [INFERENCE]. Users can draft responses offline in field camps [INFERENCE].
2. Asynchronous Background Synchronization: When connectivity is restored, an automated queue syncs locally drafted reports and pending OCR batch uploads with the central server via chunked HTTP POST requests [INFERENCE].
3. Bandwidth Reduction Optimization: PDF previewing uses client-side rendering with SVG bounding box overlays [INFERENCE]. Only the specific page image requested by the user is downloaded on-demand, rather than re-downloading 300MB full report PDFs over cellular networks [INFERENCE].

15. Security, Privacy & Governance

1. Confidentiality & Air-Gapped Hosting: Un-auctioned coal block data, raw bore log details, and strategic coal reserve assessments are classified government records [KNOWN]. The proposed system is packaged as an air-gapped Docker Compose / Kubernetes deployment to run exclusively on CIL/CMPDI private cloud servers with zero outbound internet calls [INFERENCE].
2. Role-Based Access Control (RBAC):
   • System Admin: Full access, pipeline management, user provisioning.
   • CMPDI Geologist: Full read/write access to Geological Reports, bore logs, and report generator.
   • Subsidiary Operational Manager: Access restricted to specific subsidiary production data (e.g., SECL manager sees SECL data).
   • Ministry Official: Read-only access to national aggregates, Parliamentary QA generator, and executive summaries [INFERENCE].
3. Data Integrity & Audit Logging: Every document upload, tabular correction, and report export is recorded in an immutable PostgreSQL audit trail containing user_id, timestamp, action, original_value, and modified_value [INFERENCE].
4. AI Hallucination Liability Mitigation: All AI-generated parliamentary answers and report summaries contain mandatory legal disclaimer footers [INFERENCE]. The UI enforces human-in-the-loop validation: generated answers MUST be reviewed and approved by an authorized official before export [INFERENCE].

16. MVP Scope (Hackathon Build)

In Scope (Must Build for Hackathon Demo)

1. Multi-Format Ingestion Module: Ingestion of scanned multi-page Geological Report PDFs, synthetic drill logs, and Excel production sheets.
2. OCR & Layout-Aware Table Extractor: Interactive extraction of tabular bore-log data with bounding-box highlighting on scanned page images.
3. Deterministic Anomaly Validation: Automated detection of calculation errors in depth/seam thickness or production sums with visual warning badges.
4. AI Parliamentary Q&A System (Module 3): Hybrid RAG search answering 5 key parliamentary questions with exact source document page citations and visual bounding-box evidence overlays.
5. Interactive Topic Discovery & Word Cloud (Module 2): Live topic extraction module with an interactive D3.js Word Cloud filtering documents by clicked keywords.
6. Automated Report Generation Platform (Module 1): One-click generation of draft Geological Block Summaries and Parliamentary Answer Drafts in downloadable PDF format.
7. Quantified Performance Dashboard: Live display of execution metrics (e.g., 92% Extraction Accuracy, 85% Time Reduction, 100% Citation Verifiability).

Described Only / Out of Scope (Future Production Scope)

• Direct integration with live enterprise CIL SAP OData ERP endpoints.
• High-resolution 3D geological seam interpolation and borehole wireframe modeling.
• Mobile native Android app for field drilling engineers.
• Multi-lingual voice query input in regional Indian languages.

17. Live Demo Simulation Plan

To deliver a flawless SIH live demo without relying on unavailable live government intranet systems, the team will simulate production components using hyper-realistic data and workflows:

+-----------------------------------------------------------------------------------+
|                              LIVE DEMO WORKFLOW                                   |
|                                                                                   |
|  [Step 1: Ingest] --> Upload scanned legacy GR PDF (e.g. Phatehpur-A Block)       |
|                               |                                                   |
|                               v                                                   |
|  [Step 2: Doc AI] --> Show real-time layout parsing & table extraction            |
|                       Highlight extracted bore log table on source image          |
|                               |                                                   |
|                               v                                                   |
|  [Step 3: Anomaly] -> System flags simulated OCR depth error in Borehole BH-04    |
|                       User clicks "Accept Fix" in interactive grid                |
|                               |                                                   |
|                               v                                                   |
|  [Step 4: Topics] --> Open Word Cloud tab; click "G10 Grade Coal"                 |
|                       Dashboard filters all relevant blocks instantly             |
|                               |                                                   |
|                               v                                                   |
|  [Step 5: Query]  --> Submit Parliamentary Question:                              |
|                       "What is the proved resource of Coking Coal in CCL?"        |
|                       System retrieves answer with page highlight citation        |
|                               |                                                   |
|                               v                                                   |
|  [Step 6: Export] --> Click "Generate Official Reply PDF" -> Download formatted   |
|                       Lok Sabha draft answer document                             |
+-----------------------------------------------------------------------------------+

Simulation Strategy Details

1. Simulated Document Repository: Pre-load 25 authentic, publicly available historical CMPDI Geological Reports, CIL annual reports, and Parliamentary Coal Q&A PDFs downloaded from official Ministry portals [RESEARCHED].
2. Simulated SAP ERP Feed: Use a local mock FastAPI service that returns realistic JSON responses representing live monthly coal production figures across 8 CIL subsidiaries.
3. Transparent Execution: Clearly inform judges: "All processing, OCR, RAG retrieval, and report generation shown live is executing locally on our system using open-source models; external ERP feeds are simulated via realistic mock APIs." [INFERENCE].

18. Implementation Roadmap

Schedule Aligned to Deadline (30 September 2026)

[Phase 1: Research & Data Prep]  --->  [Phase 2: Ingestion & OCR Engine]  --->  [Phase 3: RAG & Analytics]
     (05 Sep - 12 Sep 2026)                 (13 Sep - 19 Sep 2026)               (20 Sep - 24 Sep 2026)
                                                                                          |
                                                                                          v
[Phase 5: Submission & Demo Prep] <--- [Phase 4: UI, MVP Integration & Testing] <---------+
     (28 Sep - 30 Sep 2026)                 (25 Sep - 27 Sep 2026)

Phase	Timeframe	Key Tasks & Milestones	Deliverables
Phase 1: Requirements & Data Setup	05 Sep – 12 Sep 2026	Scraping public CMPDI/MoC PDFs; curating mining dictionary; setting up PostgreSQL/Qdrant schemas.	Standardized dataset of 25 reports + database schema ready.
Phase 2: Document AI & Ingestion	13 Sep – 19 Sep 2026	Implementing OpenCV preprocessing, DocTR/PaddleOCR pipeline, and TATR table parser; building deterministic validation rules.	Ingestion engine converting scanned PDFs to structured JSON & relational tables.
Phase 3: RAG, Topic & NLG Engine	20 Sep – 24 Sep 2026	Setting up BGE-M3 hybrid search in Qdrant; fine-tuning/quantizing Llama-3.1-8B; integrating BERTopic and Jinja2 report templates.	Functional RAG query service and automated report generator backend.
Phase 4: Frontend UI & MVP Integration	25 Sep – 27 Sep 2026	Building React dashboard, PDF.js bounding box highlight viewer, D3.js Word Cloud canvas, and performance telemetries.	End-to-end integrated application with full UI/UX.
Phase 5: Validation, Demo & Lock	28 Sep – 30 Sep 2026	End-to-end bug fixing, accuracy benchmark evaluation, offline rehearsal, recording fallback demo video, final submission.	Verified MVP ready for SIH evaluation.

19. Testing & Validation Plan

Test Domain	Methodology / Tool	Target Metric	Success Threshold
OCR & Text Extraction	Evaluate extracted text against manually annotated ground-truth pages.	Word Error Rate (WER) & Character Error Rate (CER).	WER < 8% on clean documents; CER < 5%.
Table Extraction Accuracy	Tree-Edit-Distance-Based Similarity (TEDS) on 50 bore-log tables.	Cell-level TEDS Score.	TEDS > 88% on complex scanned tables.
RAG Retrieval Accuracy	Benchmark top-3 retrieved document chunks on 30 test parliamentary questions.	Precision@3 & Recall@3.	Recall@3 > 92%.
Numeric Verification Test	Inject 20 artificial calculation errors into bore log depth totals.	Anomaly Flag Detection Rate.	100% detection rate of mathematical mismatches.
Response Latency	Benchmark end-to-end query response time on local GPU/CPU environment.	Average latency per query.	< 3.0 seconds per query response.
Usability & UX	Test non-technical user ability to verify citations and export a report.	Task Completion Time.	< 60 seconds for complete report generation and download.

20. Risk Register

Risk Event	Likelihood	Impact	Severity	Mitigation Strategy
Severely Degraded Historical Scans (Faded 1970s Carbon Copies)	High	High	High	Apply adaptive OpenCV binarization (CLAHE + Sauvola thresholding); provide human-in-the-loop interactive correction grid in UI.
LLM Numeric Hallucination on Coal Reserve Estimates	Medium	High	High	Bypasses LLM for numeric extraction; numeric queries execute structured SQL on deterministically validated relational database tables.
High VRAM/GPU Memory Constraints during Demo	Medium	Medium	Medium	Quantize Llama-3.1-8B to 4-bit AWQ/GGUF format using vLLM/Ollama, fitting within 8GB VRAM footprint.
Varied Table Layouts Across Different CIL Subsidiaries	High	Medium	Medium	Use vision-based Table Transformer (TATR) rather than fixed coordinate template matching.
Intermittent Network / Offline Constraints at Demo Venue	Medium	Medium	Medium	Containerize the complete application stack (Frontend, Backend, Qdrant, Ollama) via Docker Compose to run 100% offline on the presentation laptop.

21. Cost & Scalability

Hackathon Prototype (4–6 Student Team)

• Infrastructure Compute: $0 (Utilizing Google Colab free GPU / local student RTX 3090/4090 gaming laptops or $20 cloud GPU instance credit) [ASSUMPTION].
• Software Licensing: $0 (100% open-source stack: React, FastAPI, PostgreSQL, Qdrant, DocTR, Llama-3.1-8B) [INFERENCE].
• Total Hackathon Cost: ₹0 - ₹2,000 INR [INFERENCE].

Production Enterprise Deployment (CIL / CMPDI Scale)

• Hardware (On-Premise Private Cloud): 2x Enterprise GPU Servers (e.g., NVIDIA A100 / L40S 48GB VRAM) for parallel OCR and vLLM inference (~₹25–35 Lakhs one-time capital expenditure) [INFERENCE].
• Database & Storage Cluster: Distributed PostgreSQL + Qdrant cluster + MinIO Object Storage running on existing CIL datacenter infrastructure [INFERENCE].
• Maintenance & Operations: Internal CIL IT team maintenance; zero recurring per-token SaaS API subscription costs due to open-source self-hosting [INFERENCE].

22. Differentiation

Capability	Generic Hackathon Submission	Proposed CMPDI Enterprise Solution	PS Requirement Justification
Numeric Extraction Integrity	Direct LLM text parsing (prone to high numeric hallucination) [INFERENCE].	Hybrid Vision OCR + Deterministic Rules + SQL Relational Tables (0% numeric hallucination) [INFERENCE].	R2: Data validation and consistency [PS].
Citation & Traceability	Generic textual source references (e.g., "Source: Report.pdf") [INFERENCE].	Exact bounding-box highlight overlay on original scanned document image [INFERENCE].	R2: Enhanced data traceability across historical datasets [PS].
Parliamentary Q&A Support	Generic unstructured text search [INFERENCE].	Specialized Parliamentary Answer Drafter with subsidiary-wise breakdowns and regulatory disclaimers [INFERENCE].	R5: AI-based query and response system for parliamentary inquiries [PS].
Topic Discovery	Simple keyword list / word count [INFERENCE].	BERTopic contextual transformer clustering with interactive D3.js Word Cloud filtering [INFERENCE].	R4: Automated Word Cloud & Topic Identification [PS].
Data Privacy & Governance	Relies on public OpenAI cloud APIs [INFERENCE].	100% air-gapped containerized local execution [INFERENCE].	Strategic CIL/MoC data confidentiality compliance [INFERENCE].

23. Future Scope

1. Integrated 3D Borehole Seam Visualizer: Expand extracted tabular bore log depth data into automated 3D stratigraphy seam models using Three.js / WebGL [INFERENCE].
2. Multi-Lingual Voice Q&A for Field Engineers: Integrate Hindi/Bengali/Odia speech-to-text (Whisper AI) for hands-free queries by engineers in active mine sites [INFERENCE].
3. Automated Satellite InSAR Subsidence Correlation: Correlate extracted historical mining seam depths with satellite InSAR radar data to predict land subsidence risks above closed underground mines [INFERENCE].

24. Assumptions & Open Items Log

Consolidated Assumptions Log

• [ASSUMPTION 1]: The student team will run model inference locally or on a single hosted GPU instance (e.g., NVIDIA RTX 3090/4090 or AWS g5.xlarge) during the hackathon demo.
• [ASSUMPTION 2]: Synthetic noisy borehole logs and publicly released CMPDI reports adequately represent the layout complexity of confidential internal documents.
• [ASSUMPTION 3]: Ministry of Coal officials prefer responses formatted according to standard Lok Sabha / Rajya Sabha parliamentary question reply guidelines.
• [ASSUMPTION 4]: Modern 8B parameter quantized LLMs (Llama-3.1-8B-Instruct) fine-tuned with domain system prompts provide sufficient natural language synthesis quality for administrative drafting.

Consolidated Open Items Log (Needs Validation)

• [NEEDS VALIDATION 1]: Exact database table schemas and REST API contracts of CIL's enterprise SAP ERP system for direct production integration.
• [NEEDS VALIDATION 2]: Official security classification guidelines for legacy pre-1970 un-auctioned coalfield exploration reports.
• [NEEDS VALIDATION 3]: Availability of vectorized geospatial block boundary Shapefiles for direct PostGIS map overlay.

Final Recommendation

Build and deploy the Enterprise Hybrid Pipeline (Approach 3). It uniquely combines layout-aware vision OCR, deterministic rule-based table validation, hybrid Qdrant vector/sparse search, BERTopic modeling, and quantized Llama-3.1-8B narrative generation. This approach strictly honors air-gapped privacy requirements, eliminates numeric hallucination, and delivers full bounding-box citation traceability required for parliamentary queries.

Recommended Stack

• Frontend: React.js, Tailwind CSS, PDF.js, D3.js.
• Backend: FastAPI (Python 3.11), Celery, Redis.
• Document AI & OCR: DocTR, PaddleOCR, Table Transformer (TATR), OpenCV.
• Databases: PostgreSQL 16 (Relational & PostGIS), Qdrant (Vector DB).
• AI/ML Models: BGE-M3 (Embeddings), BGE-Reranker-Large, BERTopic, Llama-3.1-8B-Instruct (via vLLM / Ollama).

Biggest Dependency

Acquisition and preprocessing of authentic, varied public domain CMPDI Geological Reports and historical CIL production spreadsheets to train and validate layout parsers.

Biggest Risk

Handling severely faded, stained, or low-DPI carbon-copy scanned bore logs from the 1970s where OCR text recognition rates drop significantly.

Biggest Technical Challenge

Ensuring complete alignment and coordinate mapping between extracted tabular cells in JSON and visual bounding-box overlays on skewed multi-page scanned PDF documents.

First 5 Actions

1. Download 15–20 publicly available CMPDI Geological Reports, CIL annual reports, and MoC parliamentary PDFs from official government websites.
2. Setup local Docker container environment for PostgreSQL 16, Qdrant Vector DB, and Redis.
3. Build the core document ingestion script using OpenCV and DocTR to extract text and bounding boxes from sample PDFs.
4. Implement the deterministic Python table validation rule engine to check depth and percentage calculations.
5. Deploy Ollama with 4-bit quantized Llama-3.1-8B-Instruct and connect it to a FastAPI RAG retrieval pipeline using BGE-M3 embeddings.

Validation Before Coding

Validate that DocTR / PaddleOCR achieves >85% character accuracy on sample low-dpi scanned CMPDI bore-log pages before locking in the table extraction pipeline architecture.

Buildability Score /100

92 / 100 (High feasibility for a 4–6 student team using well-supported open-source Python frameworks, quantized LLMs, and standard web technologies).

Confidence

95% (High technical confidence backed by verified public datasets, established OCR/RAG frameworks, and clear alignment with all explicit problem statement requirements).

Research Sources

1. Title: Ministry of Coal Annual Report & Promotional Exploration Overview
   URL: https://coal.gov.in/sites/default/files/2020-09/Chapter9-en.pdf
   Source Type: Official Government Document (Ministry of Coal)
   Supported Claim: Confirms CMPDI's role in promotional/detailed exploration drilling, geological report (GR) preparation, and UNFC reserve categorization (331, 332, 333).
   Reliability: High (Official Government Publication).

2. Title: Central Mine Planning & Design Institute (CMPDI) Official Portal & Geomatics Services
   URL: https://www.cmpdi.co.in/en
   Source Type: Official PSU Organization Website
   Supported Claim: Documents CMPDI's organizational structure, 7 Regional Institutes, GIS/Geomatics capabilities, and history of preparing over 900 mining project reports.
   Reliability: High (Official Corporate Portal).

3. Title: Coal India Limited Parliamentary Affairs Division
   URL: https://www.coalindia.in/departments/parliamentary-affairs/
   Source Type: Official PSU Organization Webpage
   Supported Claim: Details the workflow for managing Lok Sabha and Rajya Sabha Starred/Unstarred Parliamentary Questions, Standing Committee queries, and Zero Hour replies across CIL subsidiaries.
   Reliability: High (Official Departmental Page).

4. Title: National Geoscience Data Repository (NGDR) Portal
   URL: https://geodataindia.gov.in/login
   Source Type: Official Government Portal (Ministry of Mines / GSI)
   Supported Claim: Validates national standards for digitizing legacy exploration data (MERT templates) and integrating AI/ML across central PSUs including CMPDI and CIL.
   Reliability: High (Official Government Initiative).

5. Title: Modified ISP 2017/2019 Norms for Coal Resource Estimation
   URL: https://www.scribd.com/document/649624934/ISP-norms-2019
   Source Type: Official Technical Standard Document
   Supported Claim: Establishes Indian Standard Code norms for bore hole density, coal seam categorization, UNFC coding (Proved 331, Indicated 332, Inferred 333), and geological report structure.
   Reliability: High (Industry Standard Specification).