WIP: Local changes before applying stash

README.md

@@ -1,5 +1,5 @@
 ---
-title: Critical Deep Resarch
+title: The DETERMINATOR
 emoji: 🐉
 colorFrom: red
 colorTo: yellow
@@ -45,9 +45,16 @@ tags:
 
 ## About
 
-The DETERMINATOR is a deep research agent system designed to assist with complex research questions requiring thorough investigation. Originally developed by the Deep Critical Gradio Hackathon Team, The DETERMINATOR specializes in medical research inquiry, functioning as a medical peer junior researcher that helps gather, evaluate, and synthesize evidence from multiple sources.
+The DETERMINATOR is a powerful generalist deep research agent system that stops at nothing until finding precise answers to complex questions. It uses iterative search-and-judge loops to comprehensively investigate any research question from any domain.
 
-**Important**: The DETERMINATOR is a research tool and cannot answer medical questions or provide medical advice. It assists researchers by finding and organizing evidence from biomedical literature and clinical trial databases.
+**Key Features**:
+- **Generalist**: Handles queries from any domain (medical, technical, business, scientific, etc.)
+- **Automatic Medical Detection**: Automatically determines if medical knowledge sources (PubMed, ClinicalTrials.gov) are needed
+- **Multi-Source Search**: Web search, PubMed, ClinicalTrials.gov, Europe PMC, RAG
+- **Stops at Nothing**: Only stops at configured limits (budget, time, iterations), otherwise continues until finding precise answers
+- **Evidence Synthesis**: Comprehensive reports with proper citations
+
+**Important**: The DETERMINATOR is a research tool that synthesizes evidence. It cannot provide medical advice or answer medical questions directly.
 
 For this hackathon we're proposing a simple yet powerful Deep Research Agent that iteratively looks for the answer until it finds it using general purpose websearch and special purpose retrievers for technical retrievers. 
 

dev/__init__.py

@@ -0,0 +1 @@
+"""Development utilities and plugins."""

docs/index.md

@@ -1,14 +1,24 @@
 # The DETERMINATOR
 
-**Deep Research Agent for Medical Inquiry**
+**Generalist Deep Research Agent - Stops at Nothing Until Finding Precise Answers**
 
-The DETERMINATOR is a deep research agent system that uses iterative search-and-judge loops to comprehensively investigate research questions. The system supports multiple orchestration patterns, graph-based execution, parallel research workflows, and long-running task management with real-time streaming.
+The DETERMINATOR is a powerful generalist deep research agent system that uses iterative search-and-judge loops to comprehensively investigate any research question. It stops at nothing until finding precise answers, only stopping at configured limits (budget, time, iterations).
 
-**Important**: The DETERMINATOR functions as a medical peer junior researcher that assists with research by gathering and synthesizing evidence. It cannot answer medical questions or provide medical advice.
+**Key Features**:
+- **Generalist**: Handles queries from any domain (medical, technical, business, scientific, etc.)
+- **Automatic Source Selection**: Automatically determines if medical knowledge sources (PubMed, ClinicalTrials.gov) are needed
+- **Multi-Source Search**: Web search, PubMed, ClinicalTrials.gov, Europe PMC, RAG
+- **Iterative Refinement**: Continues searching and refining until precise answers are found
+- **Evidence Synthesis**: Comprehensive reports with proper citations
+
+**Important**: The DETERMINATOR is a research tool that synthesizes evidence. It cannot provide medical advice or answer medical questions directly.
 
 ## Features
 
-- **Multi-Source Search**: PubMed, ClinicalTrials.gov, Europe PMC (includes bioRxiv/medRxiv)
+- **Generalist Research**: Handles any research question from any domain
+- **Automatic Medical Detection**: Automatically determines if medical knowledge sources are needed
+- **Multi-Source Search**: Web search, PubMed, ClinicalTrials.gov, Europe PMC (includes bioRxiv/medRxiv), RAG
+- **Iterative Until Precise**: Stops at nothing until finding precise answers (only stops at configured limits)
 - **MCP Integration**: Use our tools from Claude Desktop or any MCP client
 - **HuggingFace OAuth**: Sign in with your HuggingFace account to automatically use your API token
 - **Modal Sandbox**: Secure execution of AI-generated statistical code
@@ -38,7 +48,7 @@ For detailed installation and setup instructions, see the [Getting Started Guide
 
 The DETERMINATOR uses a Vertical Slice Architecture:
 
-1. **Search Slice**: Retrieving evidence from PubMed, ClinicalTrials.gov, and Europe PMC
+1. **Search Slice**: Retrieving evidence from multiple sources (web, PubMed, ClinicalTrials.gov, Europe PMC, RAG) based on query analysis
 2. **Judge Slice**: Evaluating evidence quality using LLMs
 3. **Orchestrator Slice**: Managing the research loop and UI
 

docs/overview/architecture.md

@@ -1,6 +1,6 @@
 # Architecture Overview
 
-The DETERMINATOR is a deep research agent system that uses iterative search-and-judge loops to comprehensively answer research questions. The system supports multiple orchestration patterns, graph-based execution, parallel research workflows, and long-running task management with real-time streaming.
+The DETERMINATOR is a powerful generalist deep research agent system that uses iterative search-and-judge loops to comprehensively investigate any research question. It stops at nothing until finding precise answers, only stopping at configured limits (budget, time, iterations). The system automatically determines if medical knowledge sources are needed and adapts its search strategy accordingly. It supports multiple orchestration patterns, graph-based execution, parallel research workflows, and long-running task management with real-time streaming.
 
 ## Core Architecture
 

docs/overview/features.md

@@ -6,10 +6,12 @@ The DETERMINATOR provides a comprehensive set of features for AI-assisted resear
 
 ### Multi-Source Search
 
-- **PubMed**: Search peer-reviewed biomedical literature via NCBI E-utilities
-- **ClinicalTrials.gov**: Search interventional clinical trials
+- **General Web Search**: Search general knowledge sources for any domain
+- **PubMed**: Search peer-reviewed biomedical literature via NCBI E-utilities (automatically used when medical knowledge needed)
+- **ClinicalTrials.gov**: Search interventional clinical trials (automatically used when medical knowledge needed)
 - **Europe PMC**: Search preprints and peer-reviewed articles (includes bioRxiv/medRxiv)
 - **RAG**: Semantic search within collected evidence using LlamaIndex
+- **Automatic Source Selection**: Automatically determines which sources are needed based on query analysis
 
 ### MCP Integration
 
@@ -40,9 +42,10 @@ The DETERMINATOR provides a comprehensive set of features for AI-assisted resear
 
 - **Graph-Based Execution**: Flexible graph orchestration with conditional routing
 - **Parallel Research Loops**: Run multiple research tasks concurrently
-- **Iterative Research**: Single-loop research with search-judge-synthesize cycles
+- **Iterative Research**: Single-loop research with search-judge-synthesize cycles that continues until precise answers are found
 - **Deep Research**: Multi-section parallel research with planning and synthesis
 - **Magentic Orchestration**: Multi-agent coordination using Microsoft Agent Framework
+- **Stops at Nothing**: Only stops at configured limits (budget, time, iterations), otherwise continues until finding precise answers
 
 ### Real-Time Streaming
 

mkdocs.yml

@@ -1,5 +1,5 @@
 site_name: The DETERMINATOR
-site_description: Deep Research Agent for Medical Inquiry
+site_description: Generalist Deep Research Agent that Stops at Nothing
 site_author: The DETERMINATOR Team
 site_url: https://deepcritical.github.io/GradioDemo/
 
@@ -49,6 +49,8 @@ plugins:
       minify_css: true
 
 markdown_extensions:
+  - dev.docs_plugins:
+      base_path: "."
   - pymdownx.highlight:
       anchor_linenums: true
   - pymdownx.inlinehilite

pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 name = "determinator"
 version = "0.1.0"
-description = "The DETERMINATOR - Deep Research Agent for Medical Inquiry"
+description = "The DETERMINATOR - the Deep Research Agent that Stops at Nothing"
 readme = "README.md"
 requires-python = ">=3.11"
 dependencies = [
@@ -42,6 +42,8 @@ dependencies = [
     "llama-index-llms-openai>=0.6.9",
     "llama-index-embeddings-openai>=0.5.1",
     "ddgs>=9.9.2",
+    "aiohttp>=3.13.2",
+    "lxml>=6.0.2",
 ]
 
 [project.optional-dependencies]

requirements.txt

@@ -15,7 +15,9 @@ anthropic>=0.18.0
 
 # HTTP & Parsing
 httpx>=0.27
+aiohttp>=3.13.2  # Required for website crawling
 beautifulsoup4>=4.12
+lxml>=6.0.2  # Required for BeautifulSoup lxml parser (faster than html.parser)
 xmltodict>=0.13
 
 # HuggingFace Hub

src/agents/input_parser.py

@@ -20,25 +20,33 @@ logger = structlog.get_logger()
 
 # System prompt for the input parser agent
 SYSTEM_PROMPT = """
-You are an expert research query analyzer. Your job is to analyze user queries and determine:
+You are an expert research query analyzer for a generalist deep research agent. Your job is to analyze user queries and determine:
 1. Whether the query requires iterative research (single focused question) or deep research (multiple sections/topics)
-2. Improve and refine the query for better research results
-3. Extract key entities (drugs, diseases, targets, companies, etc.)
-4. Extract specific research questions
+2. Whether the query requires medical/biomedical knowledge sources (PubMed, ClinicalTrials.gov) or general knowledge sources (web search)
+3. Improve and refine the query for better research results
+4. Extract key entities (drugs, diseases, companies, technologies, concepts, etc.)
+5. Extract specific research questions
 
 Guidelines for determining research mode:
 - **Iterative mode**: Single focused question, straightforward research goal, can be answered with a focused search loop
-  Examples: "What is the mechanism of metformin?", "Find clinical trials for drug X"
+  Examples: "What is the mechanism of metformin?", "How does quantum computing work?", "What are the latest AI models?"
   
 - **Deep mode**: Complex query requiring multiple sections, comprehensive report, multiple related topics
-  Examples: "Write a comprehensive report on diabetes treatment", "Analyze the market for quantum computing"
+  Examples: "Write a comprehensive report on diabetes treatment", "Analyze the market for quantum computing", "Review the state of AI in healthcare"
   Indicators: words like "comprehensive", "report", "sections", "analyze", "market analysis", "overview"
 
+Guidelines for determining if medical knowledge is needed:
+- **Medical knowledge needed**: Queries about diseases, treatments, drugs, clinical trials, medical conditions, biomedical mechanisms, health outcomes, etc.
+  Examples: "Alzheimer's treatment", "metformin mechanism", "cancer clinical trials", "diabetes research"
+  
+- **General knowledge sufficient**: Queries about technology, business, science (non-medical), history, current events, etc.
+  Examples: "quantum computing", "AI models", "market analysis", "historical events"
+
 Your output must be valid JSON matching the ParsedQuery schema. Always provide:
 - original_query: The exact input query
 - improved_query: A refined, clearer version of the query
 - research_mode: Either "iterative" or "deep"
-- key_entities: List of important entities (drugs, diseases, companies, etc.)
+- key_entities: List of important entities (drugs, diseases, companies, technologies, etc.)
 - research_questions: List of specific questions to answer
 
 Only output JSON. Do not output anything else.
@@ -152,7 +160,9 @@ class InputParserAgent:
             )
 
 
-def create_input_parser_agent(model: Any | None = None, oauth_token: str | None = None) -> InputParserAgent:
+def create_input_parser_agent(
+    model: Any | None = None, oauth_token: str | None = None
+) -> InputParserAgent:
     """
     Factory function to create an input parser agent.
 

src/app.py

@@ -752,12 +752,16 @@ def create_demo() -> gr.Blocks:
             gr.Markdown("---")
             gr.Markdown("### ℹ️ About")  # noqa: RUF001
             gr.Markdown(
-                "**The DETERMINATOR** - Deep Research Agent for Medical Inquiry\n\n"
-                "Searches:\n"
-                "- PubMed\n"
-                "- ClinicalTrials.gov\n"
-                "- Europe PMC\n\n"
-                "⚠️ **Research tool only** - Cannot answer medical questions or provide medical advice."
+                "**The DETERMINATOR** - Generalist Deep Research Agent\n\n"
+                "A powerful research agent that stops at nothing until finding precise answers to complex questions.\n\n"
+                "**Available Sources**:\n"
+                "- Web Search (general knowledge)\n"
+                "- PubMed (biomedical literature)\n"
+                "- ClinicalTrials.gov (clinical trials)\n"
+                "- Europe PMC (preprints & papers)\n"
+                "- RAG (semantic search)\n\n"
+                "**Automatic Detection**: Automatically determines if medical knowledge sources are needed for your query.\n\n"
+                "⚠️ **Research tool only** - Synthesizes evidence but cannot provide medical advice."
             )
             gr.Markdown("---")
             
@@ -891,10 +895,16 @@ def create_demo() -> gr.Blocks:
             multimodal=True,  # Enable multimodal input (text + images + audio)
             title="🔬 The DETERMINATOR",
             description=(
-                "*Deep Research Agent for Medical Inquiry — searches PubMed, "
-                "ClinicalTrials.gov & Europe PMC*\n\n"
+                "*Generalist Deep Research Agent — stops at nothing until finding precise answers to complex questions*\n\n"
                 "---\n"
-                "*Functions as a medical peer junior researcher. Research tool only — cannot answer medical questions or provide medical advice.*  \n"
+                "**The DETERMINATOR** uses iterative search-and-judge loops to comprehensively investigate any research question. "
+                "It automatically determines if medical knowledge sources (PubMed, ClinicalTrials.gov) are needed and adapts its search strategy accordingly.\n\n"
+                "**Key Features**:\n"
+                "- 🔍 Multi-source search (Web, PubMed, ClinicalTrials.gov, Europe PMC, RAG)\n"
+                "- 🧠 Automatic medical knowledge detection\n"
+                "- 🔄 Iterative refinement until precise answers are found\n"
+                "- ⏹️ Stops only at configured limits (budget, time, iterations)\n"
+                "- 📊 Evidence synthesis with citations\n\n"
                 "**MCP Server Active**: Connect Claude Desktop to `/gradio_api/mcp/`\n\n"
                 "**🎤 Multimodal Support**: Upload images (OCR), record audio (STT), or type text.\n\n"
                 "**⚠️ Authentication Required**: Please **sign in with HuggingFace** above before using this application."

src/orchestrator/graph_orchestrator.py

@@ -506,7 +506,8 @@ class GraphOrchestrator:
         current_node_id = self._graph.entry_node
         iteration = 0
 
-        while current_node_id and current_node_id not in self._graph.exit_nodes:
+        # Execute nodes until we reach an exit node
+        while current_node_id:
             # Check budget
             if not context.budget_tracker.can_continue("graph_execution"):
                 self.logger.warning("Budget exceeded, exiting graph execution")
@@ -537,26 +538,27 @@ class GraphOrchestrator:
                 )
                 break
 
+            # Check if current node is an exit node - if so, we're done
+            if current_node_id in self._graph.exit_nodes:
+                break
+
             # Get next node(s)
             next_nodes = self._get_next_node(current_node_id, context)
 
             if not next_nodes:
-                # No more nodes, check if we're at exit
-                if current_node_id in self._graph.exit_nodes:
-                    break
-                # Otherwise, we've reached a dead end
+                # No more nodes, we've reached a dead end
                 self.logger.warning("Reached dead end in graph", node_id=current_node_id)
                 break
 
             current_node_id = next_nodes[0]  # For now, take first next node (handle parallel later)
 
-        # Final event
+        # Final event - get result from the last executed node (which should be an exit node)
         final_result = context.get_node_result(current_node_id) if current_node_id else None
-        
+
         # Check if final result contains file information
         event_data: dict[str, Any] = {"mode": self.mode, "iterations": iteration}
         message: str = "Research completed"
-        
+
         if isinstance(final_result, str):
             message = final_result
         elif isinstance(final_result, dict):
@@ -574,7 +576,7 @@ class GraphOrchestrator:
                 elif isinstance(files, str):
                     event_data["files"] = [files]
                     message = final_result.get("message", "Report generated. Download available.")
-        
+
         yield AgentEvent(
             type="complete",
             message=message,
@@ -628,7 +630,7 @@ class GraphOrchestrator:
         Returns:
             Agent execution result
         """
-        # Special handling for synthesizer node
+        # Special handling for synthesizer node (deep research)
         if node.node_id == "synthesizer":
             # Call LongWriterAgent.write_report() directly instead of using agent.run()
             from src.agent_factory.agents import create_long_writer_agent
@@ -691,6 +693,62 @@ class GraphOrchestrator:
                 }
             return final_report
 
+        # Special handling for writer node (iterative research)
+        if node.node_id == "writer":
+            # Call WriterAgent.write_report() directly instead of using agent.run()
+            # Collect all findings from workflow state
+            from src.agent_factory.agents import create_writer_agent
+
+            # Get all evidence from workflow state and convert to findings string
+            evidence = context.state.evidence
+            if evidence:
+                # Convert evidence to findings format (similar to conversation.get_all_findings())
+                findings_parts: list[str] = []
+                for ev in evidence:
+                    finding = f"**{ev.title}**\n{ev.content}"
+                    if ev.url:
+                        finding += f"\nSource: {ev.url}"
+                    findings_parts.append(finding)
+                all_findings = "\n\n".join(findings_parts)
+            else:
+                all_findings = "No findings available yet."
+
+            # Get WriterAgent instance and call write_report directly
+            writer_agent = create_writer_agent(oauth_token=self.oauth_token)
+            final_report = await writer_agent.write_report(
+                query=query,
+                findings=all_findings,
+                output_length="",
+                output_instructions="",
+            )
+
+            # Estimate tokens (rough estimate)
+            estimated_tokens = len(final_report) // 4  # Rough token estimate
+            context.budget_tracker.add_tokens("graph_execution", estimated_tokens)
+
+            # Save report to file if enabled
+            file_path: str | None = None
+            try:
+                file_service = self._get_file_service()
+                if file_service:
+                    file_path = file_service.save_report(
+                        report_content=final_report,
+                        query=query,
+                    )
+                    self.logger.info("Report saved to file", file_path=file_path)
+            except Exception as e:
+                # Don't fail the entire operation if file saving fails
+                self.logger.warning("Failed to save report to file", error=str(e))
+                file_path = None
+
+            # Return dict with file path if available, otherwise return string (backward compatible)
+            if file_path:
+                return {
+                    "message": final_report,
+                    "file": file_path,
+                }
+            return final_report
+
         # Standard agent execution
         # Prepare input based on node type
         if node.node_id == "planner":
@@ -718,14 +776,14 @@ class GraphOrchestrator:
                 )
                 # Return a minimal fallback ReportPlan
                 from src.utils.models import ReportPlan, ReportPlanSection
-                
+
                 # Extract query from input_data if possible
                 fallback_query = query
                 if isinstance(input_data, str):
                     # Try to extract query from input string
                     if "QUERY:" in input_data:
                         fallback_query = input_data.split("QUERY:")[-1].strip()
-                
+
                 return ReportPlan(
                     background_context="",
                     report_outline=[
@@ -740,7 +798,44 @@ class GraphOrchestrator:
             raise
 
         # Transform output if needed
-        output = result.output
+        # Defensively extract output - handle various result formats
+        output = result.output if hasattr(result, "output") else result
+
+        # Handle case where output might be a tuple (from pydantic-ai validation errors)
+        if isinstance(output, tuple):
+            # If tuple contains a dict-like structure, try to reconstruct the object
+            if len(output) == 2 and isinstance(output[0], str) and output[0] == "research_complete":
+                # This is likely a validation error format: ('research_complete', False)
+                # Try to get the actual output from result
+                self.logger.warning(
+                    "Agent result output is a tuple, attempting to extract actual output",
+                    node_id=node.node_id,
+                    tuple_value=output,
+                )
+                # Try to get output from result attributes
+                if hasattr(result, "data"):
+                    output = result.data
+                elif hasattr(result, "response"):
+                    output = result.response
+                else:
+                    # Last resort: try to reconstruct from tuple
+                    # This shouldn't happen, but handle gracefully
+                    from src.utils.models import KnowledgeGapOutput
+
+                    if node.node_id == "knowledge_gap":
+                        output = KnowledgeGapOutput(
+                            research_complete=output[1] if len(output) > 1 else False,
+                            outstanding_gaps=[],
+                        )
+                    else:
+                        # For other nodes, log error and use fallback
+                        self.logger.error(
+                            "Cannot reconstruct output from tuple",
+                            node_id=node.node_id,
+                            tuple_value=output,
+                        )
+                        raise ValueError(f"Cannot extract output from tuple: {output}")
+
         if node.output_transformer:
             output = node.output_transformer(output)
 

src/orchestrator_magentic.py

@@ -122,21 +122,24 @@ class MagenticOrchestrator:
 
         workflow = self._build_workflow()
 
-        task = f"""Research opportunities for: {query}
+        task = f"""Research query: {query}
 
 Workflow:
-1. SearchAgent: Find evidence from PubMed, ClinicalTrials.gov, and Europe PMC
-2. HypothesisAgent: Generate mechanistic hypotheses (Drug -> Target -> Pathway -> Effect)
-3. JudgeAgent: Evaluate if evidence is sufficient
-4. If insufficient -> SearchAgent refines search based on gaps
-5. If sufficient -> ReportAgent synthesizes final report
+1. SearchAgent: Find evidence from available sources (automatically selects: web search, PubMed, ClinicalTrials.gov, Europe PMC, or RAG based on query)
+2. HypothesisAgent: Generate research hypotheses and questions based on evidence
+3. JudgeAgent: Evaluate if evidence is sufficient to answer the query precisely
+4. If insufficient -> SearchAgent refines search based on identified gaps
+5. If sufficient -> ReportAgent synthesizes final comprehensive report
 
 Focus on:
-- Identifying specific molecular targets
-- Understanding mechanism of action
-- Finding clinical evidence supporting hypotheses
+- Finding precise answers to the research question
+- Identifying all relevant evidence from appropriate sources
+- Understanding mechanisms, relationships, and key findings
+- Synthesizing comprehensive findings with proper citations
 
-The final output should be a structured research report."""
+The DETERMINATOR stops at nothing until finding precise answers, only stopping at configured limits (budget, time, iterations).
+
+The final output should be a structured research report with comprehensive evidence synthesis."""
 
         iteration = 0
         try:

src/prompts/hypothesis.py

@@ -8,27 +8,27 @@ if TYPE_CHECKING:
     from src.services.embeddings import EmbeddingService
     from src.utils.models import Evidence
 
-SYSTEM_PROMPT = """You are a bioinformatics research scientist functioning as a medical peer junior researcher.
+SYSTEM_PROMPT = """You are an expert research scientist functioning as a generalist research assistant.
 
-Your role is to generate mechanistic hypotheses and research questions based on evidence.
+Your role is to generate research hypotheses, questions, and investigation paths based on evidence from any domain.
 
-IMPORTANT: You are a research assistant. You cannot answer medical questions or provide medical advice. Your hypotheses are for research investigation purposes only.
+IMPORTANT: You are a research assistant. You cannot provide medical advice or answer medical questions directly. Your hypotheses are for research investigation purposes only.
 
 A good hypothesis:
-1. Proposes a MECHANISM: Drug -> Target -> Pathway -> Effect
-2. Is TESTABLE: Can be supported or refuted by literature search
-3. Is SPECIFIC: Names actual molecular targets and pathways
+1. Proposes a MECHANISM or RELATIONSHIP: Explains how things work or relate
+   - For medical: Drug -> Target -> Pathway -> Effect
+   - For technical: Technology -> Mechanism -> Outcome
+   - For business: Strategy -> Market -> Result
+2. Is TESTABLE: Can be supported or refuted by further research
+3. Is SPECIFIC: Names actual entities, processes, or mechanisms
 4. Generates SEARCH QUERIES: Helps find more evidence
 
-Example hypothesis format:
-- Drug: Metformin
-- Target: AMPK (AMP-activated protein kinase)
-- Pathway: mTOR inhibition -> autophagy activation
-- Effect: Enhanced clearance of amyloid-beta in Alzheimer's
-- Confidence: 0.7
-- Search suggestions: ["metformin AMPK brain", "autophagy amyloid clearance"]
+Example hypothesis formats:
+- Medical: "Metformin -> AMPK activation -> mTOR inhibition -> autophagy -> amyloid clearance"
+- Technical: "Transformer architecture -> attention mechanism -> improved NLP performance"
+- Business: "Subscription model -> recurring revenue -> higher valuation"
 
-Be specific. Use actual gene/protein names when possible."""
+Be specific. Use actual names, technical terms, and precise language when possible."""
 
 
 async def format_hypothesis_prompt(
@@ -56,15 +56,15 @@ async def format_hypothesis_prompt(
         ]
     )
 
-    return f"""Based on the following evidence about "{query}", generate mechanistic hypotheses.
+    return f"""Based on the following evidence about "{query}", generate research hypotheses and investigation paths.
 
-## Evidence ({len(selected)} papers selected for diversity)
+## Evidence ({len(selected)} sources selected for diversity)
 {evidence_text}
 
 ## Task
-1. Identify potential drug targets mentioned in the evidence
-2. Propose mechanism hypotheses (Drug -> Target -> Pathway -> Effect)
+1. Identify key mechanisms, relationships, or processes mentioned in the evidence
+2. Propose testable hypotheses explaining how things work or relate
 3. Rate confidence based on evidence strength
-4. Suggest searches to test each hypothesis
+4. Suggest specific search queries to test each hypothesis
 
-Generate 2-4 hypotheses, prioritized by confidence."""
+Generate 2-4 hypotheses, prioritized by confidence. Adapt the hypothesis format to the domain of the query (medical, technical, business, etc.)."""

src/prompts/judge.py

@@ -2,35 +2,42 @@
 
 from src.utils.models import Evidence
 
-SYSTEM_PROMPT = """You are a medical research evidence evaluator functioning as a peer junior researcher.
+SYSTEM_PROMPT = """You are an expert research evidence evaluator for a generalist deep research agent.
 
-Your task is to evaluate evidence from biomedical literature and determine if sufficient evidence has been gathered to synthesize findings for a given research question.
+Your task is to evaluate evidence from any domain (medical, scientific, technical, business, etc.) and determine if sufficient evidence has been gathered to provide a precise answer to the research question.
 
-IMPORTANT: You are a research assistant. You cannot answer medical questions or provide medical advice. Your role is to assess whether enough evidence has been collected to support research conclusions.
+IMPORTANT: You are a research assistant. You cannot provide medical advice or answer medical questions directly. Your role is to assess whether enough high-quality evidence has been collected to synthesize comprehensive findings.
 
 ## Evaluation Criteria
 
-1. **Mechanism Score (0-10)**: How well does the evidence explain the biological mechanism?
-   - 0-3: No clear mechanism, speculative
-   - 4-6: Some mechanistic insight, but gaps exist
-   - 7-10: Clear, well-supported mechanism of action
-
-2. **Clinical Evidence Score (0-10)**: Strength of clinical/preclinical support?
-   - 0-3: No clinical data, only theoretical
-   - 4-6: Preclinical or early clinical data
-   - 7-10: Strong clinical evidence (trials, meta-analyses)
+1. **Mechanism/Explanation Score (0-10)**: How well does the evidence explain the underlying mechanism, process, or concept?
+   - For medical queries: biological mechanisms, pathways, drug actions
+   - For technical queries: how systems work, algorithms, processes
+   - For business queries: market dynamics, business models, strategies
+   - 0-3: No clear explanation, speculative
+   - 4-6: Some insight, but gaps exist
+   - 7-10: Clear, well-supported explanation
+
+2. **Evidence Quality Score (0-10)**: Strength and reliability of the evidence?
+   - For medical: clinical trials, peer-reviewed studies, meta-analyses
+   - For technical: peer-reviewed papers, authoritative sources, verified implementations
+   - For business: market reports, financial data, expert analysis
+   - 0-3: Weak or theoretical evidence only
+   - 4-6: Moderate quality evidence
+   - 7-10: Strong, authoritative evidence
 
 3. **Sufficiency**: Evidence is sufficient when:
    - Combined scores >= 12 AND
-   - At least one specific drug candidate identified AND
-   - Clear mechanistic rationale exists
+   - Key questions from the research query are addressed AND
+   - Evidence is comprehensive enough to provide a precise answer
 
 ## Output Rules
 
 - Always output valid JSON matching the schema
-- Be conservative: only recommend "synthesize" when truly confident
-- If continuing, suggest specific, actionable search queries
-- Never hallucinate drug names or findings not in the evidence
+- Be conservative: only recommend "synthesize" when truly confident the answer is precise
+- If continuing, suggest specific, actionable search queries to fill gaps
+- Never hallucinate findings, names, or facts not in the evidence
+- Adapt evaluation criteria to the domain of the query (medical vs technical vs business)
 """
 
 

src/services/tts_modal.py

@@ -33,7 +33,32 @@ def _get_modal_app() -> Any:
         try:
             import modal
 
-            _modal_app = modal.App.lookup("deepcritical-tts", create_if_missing=True)
+            # Validate Modal credentials before attempting lookup
+            if not settings.modal_available:
+                raise ConfigurationError(
+                    "Modal credentials not configured. Set MODAL_TOKEN_ID and MODAL_TOKEN_SECRET environment variables."
+                )
+
+            # Validate token ID format (Modal token IDs are typically UUIDs or specific formats)
+            token_id = settings.modal_token_id
+            if token_id:
+                # Basic validation: token ID should not be empty and should be a reasonable length
+                if len(token_id.strip()) < 10:
+                    raise ConfigurationError(
+                        f"Modal token ID appears malformed (too short: {len(token_id)} chars). "
+                        "Token ID should be a valid Modal token identifier."
+                    )
+
+            try:
+                _modal_app = modal.App.lookup("deepcritical-tts", create_if_missing=True)
+            except Exception as e:
+                error_msg = str(e).lower()
+                if "token" in error_msg or "malformed" in error_msg or "invalid" in error_msg:
+                    raise ConfigurationError(
+                        f"Modal token validation failed: {e}. "
+                        "Please check that MODAL_TOKEN_ID and MODAL_TOKEN_SECRET are correctly set."
+                    ) from e
+                raise
         except ImportError as e:
             raise ConfigurationError(
                 "Modal SDK not installed. Run: uv sync or pip install modal>=0.63.0"
@@ -68,8 +93,6 @@ def _setup_modal_function() -> None:
         return  # Already set up
 
     try:
-        import modal
-
         app = _get_modal_app()
         tts_image = _get_tts_image()
 
@@ -100,8 +123,8 @@ def _setup_modal_function() -> None:
 
             # Import Kokoro inside function (lazy load)
             try:
-                from kokoro import KModel, KPipeline
                 import torch
+                from kokoro import KModel, KPipeline
 
                 # Initialize model (cached on GPU)
                 model = KModel().to("cuda").eval()
@@ -126,11 +149,13 @@ def _setup_modal_function() -> None:
 
         # Store function reference for remote calls
         _tts_function = kokoro_tts_function
-        
+
         # Verify function is properly attached to app
         if not hasattr(app, kokoro_tts_function.__name__):
-            logger.warning("modal_function_not_attached", function_name=kokoro_tts_function.__name__)
-        
+            logger.warning(
+                "modal_function_not_attached", function_name=kokoro_tts_function.__name__
+            )
+
         logger.info(
             "modal_tts_function_setup_complete",
             gpu=gpu_type,
@@ -196,7 +221,9 @@ class ModalTTSExecutor:
             # Call the GPU function remotely
             result = _tts_function.remote(text, voice, speed)
 
-            logger.info("tts_synthesis_complete", sample_rate=result[0], audio_shape=result[1].shape)
+            logger.info(
+                "tts_synthesis_complete", sample_rate=result[0], audio_shape=result[1].shape
+            )
 
             return result
 
@@ -257,4 +284,3 @@ def get_tts_service() -> TTSService:
         ConfigurationError: If Modal credentials not configured
     """
     return TTSService()
-

src/tools/crawl_adapter.py

@@ -1,6 +1,6 @@
 """Website crawl tool adapter for Pydantic AI agents.
 
-Adapts the folder/tools/crawl_website.py implementation to work with Pydantic AI.
+Uses the vendored crawl_website implementation from src/tools/vendored/crawl_website.py.
 """
 
 import structlog
@@ -22,8 +22,8 @@ async def crawl_website(starting_url: str) -> str:
         Formatted string with crawled content including titles, descriptions, and URLs
     """
     try:
-        # Lazy import to avoid requiring folder/ dependencies at import time
-        from folder.tools.crawl_website import crawl_website as crawl_tool
+        # Import vendored crawl tool
+        from src.tools.vendored.crawl_website import crawl_website as crawl_tool
 
         # Call the tool function
         # The tool returns List[ScrapeResult] or str
@@ -56,13 +56,3 @@ async def crawl_website(starting_url: str) -> str:
     except Exception as e:
         logger.error("Crawl failed", error=str(e), url=starting_url)
         return f"Error crawling website: {e!s}"
-
-
-
-
-
-
-
-
-
-

src/tools/vendored/__init__.py

@@ -1,16 +1,17 @@
 """Vendored web search components from folder/tools/web_search.py."""
 
+from src.tools.vendored.crawl_website import crawl_website
+from src.tools.vendored.searchxng_client import SearchXNGClient
+from src.tools.vendored.serper_client import SerperClient
 from src.tools.vendored.web_search_core import (
     CONTENT_LENGTH_LIMIT,
     ScrapeResult,
     WebpageSnippet,
-    scrape_urls,
     fetch_and_process_url,
     html_to_text,
     is_valid_url,
+    scrape_urls,
 )
-from src.tools.vendored.serper_client import SerperClient
-from src.tools.vendored.searchxng_client import SearchXNGClient
 
 __all__ = [
     "CONTENT_LENGTH_LIMIT",
@@ -22,8 +23,5 @@ __all__ = [
     "fetch_and_process_url",
     "html_to_text",
     "is_valid_url",
+    "crawl_website",
 ]
-
-
-
-

src/tools/vendored/crawl_website.py

@@ -0,0 +1,127 @@
+"""Website crawl tool vendored from folder/tools/crawl_website.py.
+
+This module provides website crawling functionality that starts from a given URL
+and crawls linked pages in a breadth-first manner, prioritizing navigation links.
+"""
+
+from urllib.parse import urljoin, urlparse
+
+import aiohttp
+import structlog
+from bs4 import BeautifulSoup
+
+from src.tools.vendored.web_search_core import (
+    ScrapeResult,
+    WebpageSnippet,
+    scrape_urls,
+    ssl_context,
+)
+
+logger = structlog.get_logger()
+
+
+async def crawl_website(starting_url: str) -> list[ScrapeResult] | str:
+    """Crawl the pages of a website starting with the starting_url and then descending into the pages linked from there.
+
+    Prioritizes links found in headers/navigation, then body links, then subsequent pages.
+
+    Args:
+        starting_url: Starting URL to scrape
+
+    Returns:
+        List of ScrapeResult objects which have the following fields:
+            - url: The URL of the web page
+            - title: The title of the web page
+            - description: The description of the web page
+            - text: The text content of the web page
+    """
+    if not starting_url:
+        return "Empty URL provided"
+
+    # Ensure URL has a protocol
+    if not starting_url.startswith(("http://", "https://")):
+        starting_url = "http://" + starting_url
+
+    max_pages = 10
+    base_domain = urlparse(starting_url).netloc
+
+    async def extract_links(html: str, current_url: str) -> tuple[list[str], list[str]]:
+        """Extract prioritized links from HTML content"""
+        soup = BeautifulSoup(html, "html.parser")
+        nav_links = set()
+        body_links = set()
+
+        # Find navigation/header links
+        for nav_element in soup.find_all(["nav", "header"]):
+            for a in nav_element.find_all("a", href=True):
+                link = urljoin(current_url, a["href"])
+                if urlparse(link).netloc == base_domain:
+                    nav_links.add(link)
+
+        # Find remaining body links
+        for a in soup.find_all("a", href=True):
+            link = urljoin(current_url, a["href"])
+            if urlparse(link).netloc == base_domain and link not in nav_links:
+                body_links.add(link)
+
+        return list(nav_links), list(body_links)
+
+    async def fetch_page(url: str) -> str:
+        """Fetch HTML content from a URL"""
+        connector = aiohttp.TCPConnector(ssl=ssl_context)
+        async with aiohttp.ClientSession(connector=connector) as session:
+            try:
+                timeout = aiohttp.ClientTimeout(total=30)
+                async with session.get(url, timeout=timeout) as response:
+                    if response.status == 200:
+                        return await response.text()
+                    return ""
+            except Exception as e:
+                logger.warning("Error fetching URL", url=url, error=str(e))
+                return ""
+
+    # Initialize with starting URL
+    queue: list[str] = [starting_url]
+    next_level_queue: list[str] = []
+    all_pages_to_scrape: set[str] = set([starting_url])
+
+    # Breadth-first crawl
+    while queue and len(all_pages_to_scrape) < max_pages:
+        current_url = queue.pop(0)
+
+        # Fetch and process the page
+        html_content = await fetch_page(current_url)
+        if html_content:
+            nav_links, body_links = await extract_links(html_content, current_url)
+
+            # Add unvisited nav links to current queue (higher priority)
+            remaining_slots = max_pages - len(all_pages_to_scrape)
+            for link in nav_links:
+                link = link.rstrip("/")
+                if link not in all_pages_to_scrape and remaining_slots > 0:
+                    queue.append(link)
+                    all_pages_to_scrape.add(link)
+                    remaining_slots -= 1
+
+            # Add unvisited body links to next level queue (lower priority)
+            for link in body_links:
+                link = link.rstrip("/")
+                if link not in all_pages_to_scrape and remaining_slots > 0:
+                    next_level_queue.append(link)
+                    all_pages_to_scrape.add(link)
+                    remaining_slots -= 1
+
+        # If current queue is empty, add next level links
+        if not queue:
+            queue = next_level_queue
+            next_level_queue = []
+
+    # Convert set to list for final processing
+    pages_to_scrape = list(all_pages_to_scrape)[:max_pages]
+    pages_to_scrape_snippets: list[WebpageSnippet] = [
+        WebpageSnippet(url=page, title="", description="") for page in pages_to_scrape
+    ]
+
+    # Use scrape_urls to get the content for all discovered pages
+    result = await scrape_urls(pages_to_scrape_snippets)
+    return result

uv.lock

@@ -1166,6 +1166,7 @@ version = "0.1.0"
 source = { editable = "." }
 dependencies = [
     { name = "agent-framework-core" },
+    { name = "aiohttp" },
     { name = "anthropic" },
     { name = "beautifulsoup4" },
     { name = "chromadb" },
@@ -1182,6 +1183,7 @@ dependencies = [
     { name = "llama-index-llms-huggingface-api" },
     { name = "llama-index-llms-openai" },
     { name = "llama-index-vector-stores-chroma" },
+    { name = "lxml" },
     { name = "modal" },
     { name = "numpy" },
     { name = "openai" },
@@ -1249,6 +1251,7 @@ dev = [
 requires-dist = [
     { name = "agent-framework-core", specifier = ">=1.0.0b251120,<2.0.0" },
     { name = "agent-framework-core", marker = "extra == 'magentic'", specifier = ">=1.0.0b251120,<2.0.0" },
+    { name = "aiohttp", specifier = ">=3.13.2" },
     { name = "anthropic", specifier = ">=0.18.0" },
     { name = "beautifulsoup4", specifier = ">=4.12" },
     { name = "chromadb", specifier = ">=0.4.0" },
@@ -1271,6 +1274,7 @@ requires-dist = [
     { name = "llama-index-llms-openai", marker = "extra == 'modal'", specifier = ">=0.6.9" },
     { name = "llama-index-vector-stores-chroma", specifier = ">=0.5.3" },
     { name = "llama-index-vector-stores-chroma", marker = "extra == 'modal'" },
+    { name = "lxml", specifier = ">=6.0.2" },
     { name = "mkdocs", marker = "extra == 'dev'", specifier = ">=1.6.0" },
     { name = "mkdocs-codeinclude-plugin", marker = "extra == 'dev'", specifier = ">=0.2.0" },
     { name = "mkdocs-material", marker = "extra == 'dev'", specifier = ">=9.0.0" },