# Technical Document Extraction: Multimodal Model Architecture This image illustrates a technical pipeline for a multimodal machine learning model designed to process visual screen data and text queries to generate predictions. ## 1. Input Components ### Screen (Visual Input) The leftmost component is a mobile application screenshot for "NICHE". * **Header:** Contains a hamburger menu, the "NICHE" logo, and a "Log In" button. * **Search Bar:** Contains the text "K12 Schools Tulsa Area". * **Content Cards:** * "Best School Districts" with a 2021 Best Schools badge. * "Invest in Your Child's Future" with a piggy bank illustration. * List items: "Best Places to Buy a House" and "Best Places to Raise a Family". ### Text Input Located at the bottom center, providing context for the model. * **Content:** `'Question: What is the text in the search bar?'` --- ## 2. Processing Pipeline (Flow) ### Step 1: pix2struct patching The screen image is passed into a patching module. * **Mechanism:** The image is divided into an "Aspect ratio preserving grid with max e.g 25 patches". * **Sub-visuals:** * A **5x5** grid example showing a flight booking interface. * A **4x6** grid example showing the NICHE mobile screen divided into green-tinted rectangular patches. ### Step 2: Vision Encoder (ViT) The patched image data flows into a **Vision Encoder (ViT)**, represented by a light green block. ### Step 3: embed + concat The output from the Vision Encoder and the **Text input** are merged in this stage. * The text query is embedded and concatenated with the visual embeddings. ### Step 4: T5 Multimodal Encoder The concatenated data enters a grey block labeled **T5 Multimodal Encoder**. * **Internal Component:** **Cross-attn + FFW** (Cross-attention and Feed-Forward Network). * **Repetition:** This block is repeated **x N** times. * **Output:** Key (**K**) and Value (**V**) vectors are passed to the next stage. ### Step 5: T5 Decoder A large grey block representing the decoding phase. * **Internal Components:** 1. **Self-attn** (Self-attention) 2. **Cross-attn + FFW** (Cross-attention and Feed-Forward Network) * **Repetition:** This sequence is repeated **x N** times. --- ## 3. Output ### Model predictions The final output generated by the T5 Decoder. * **Result:** `'K12 Schools Tulsa Area'` * **Logic Check:** This correctly answers the input question by extracting the specific text found in the search bar of the original screen image.

# Technical Document Extraction: System Architecture Diagram ## 1. Screen Interface (Left Panel) ### Header Section - **Timestamp**: `12:45` - **Status Icons**: Settings, Share, Location - **Network**: `4G` with full signal strength - **Battery**: Full charge indicator ### Main Content - **App Header**: - **Logo**: `NICHE` with stylized `N` icon - **Navigation**: Hamburger menu icon (☰) - **Search Bar**: - Query: `K12 Schools Tulsa Area` - Search icon (🔍) - **Content Sections**: 1. **Best School Districts** - Visual: School buildings with American flag - Label: `2021 BEST SCHOOLS` - Subtext: `NICHE` 2. **Invest in Your Child's Future** - Visual: Piggy bank with graduation cap - Text: `Start saving for college today.` 3. **Considering a Move to Tulsa Area?** - Subsections: - `Best Places to Buy a House` (House icon) - `Best Places to Raise a Family` (Ice cream icon) ## 2. System Architecture Diagram (Right Panel) ### Component Flow 1. **Input Processing** - **Text Input**: - Query: `What is the text in the search bar?` - **Image Input**: - **Aspect Ratio**: 5x5 and 4x6 grids - **Patching**: `pix2struct patching` with max 25 patches 2. **Vision Encoder (ViT)** - Processes image patches - Output: Embeddings 3. **Multimodal Fusion** - **Embed + Concat**: Combines text and image embeddings 4. **T5 Multimodal Encoder** - **Cross-Attention + Feed-Forward (FFW)** layers - Processes fused embeddings 5. **T5 Decoder** - **Self-Attention** layers - **Cross-Attention + FFW** layers - Output: Model predictions ### Model Predictions - Final Output: `K12 Schools Tulsa Area` ## 3. Key Technical Elements - **Vision Encoder**: Vision Transformer (ViT) - **Attention Mechanisms**: - Cross-Attention (K, V inputs) - Self-Attention (Q, K, V inputs) - **Feed-Forward Networks (FFW)**: Position-wise transformations ## 4. Spatial Grounding - **Legend Placement**: Not explicitly shown (diagram uses direct labeling) - **Color Coding**: - Green: Vision Encoder components - Light Green: Attention/FFW layers - Gray: Structural elements (embeddings, concatenation) ## 5. Textual Elements - **Embedded Text in Diagram**: - `Aspect ratio preserving grid with max e.g 25 patches` - `Cross-attn + FFW` (repeated in encoder/decoder) ## 6. Data Flow Summary