# Vision-language extraction of pedon data (Module 2)

Module 2 (`vlm-*`) lets `soilKey` build a `PedonRecord` from a field-description PDF, a profile photo, or a fieldsheet image – using a vision-language model (VLM) for the extraction and JSON-Schema validation as a hard gate. The taxonomic key itself is never delegated to the LLM: the VLM is restricted to extraction, and every extracted value is recorded in the provenance log so the evidence grade reflects the lower confidence of VLM-sourced data.

This vignette walks the extraction loop end to end, using a `MockVLMProvider` so the example runs offline and without API keys. Swapping the mock for a real `ellmer` chat (Anthropic, OpenAI, etc.) is a one-line change.

# 1. The mock provider

`MockVLMProvider` exposes the same `$chat()` method as an `ellmer` chat object, but pops responses from a pre-loaded queue. Because all of `soilKey`’s VLM logic talks to the provider through `$chat()`, swapping it for a real chat (or for any custom backend) is transparent.

    mock <- MockVLMProvider$new(responses = list())
    class(mock)
    #> [1] "MockVLMProvider" "R6"

# 2. Schemas and prompts

VLM responses are constrained by a JSON Schema (Draft-07). The schema for horizon extraction is shipped at `inst/schemas/horizon.json`; the rendered prompt template lives at `inst/prompts/extract_horizons.md`.

    sch <- jsonlite::fromJSON(soilKey:::load_schema("horizon"),
                                simplifyVector = FALSE)
    length(sch$properties$horizons$items$properties)
    #> [1] 35
    head(names(sch$properties$horizons$items$properties), 12)
    #>  [1] "top_cm"                "bottom_cm"             "designation"
    #>  [4] "boundary_distinctness" "boundary_topography"   "munsell_moist"
    #>  [7] "munsell_dry"           "structure_grade"       "structure_size"
    #> [10] "structure_type"        "consistence_moist"     "clay_films_amount"

The schema enforces strict types – `top_cm` must be numeric, `designation` must be a string, Munsell colours come as `{hue, value, chroma, confidence, source_quote}` triples, etc. Any response that fails validation triggers the retry loop with the validation error included in the next prompt.

# 3. Walking the extraction loop with the mock

Suppose the model returns the following horizon-level JSON for a synthetic Latossolo description:

    horizon_json <- '{
      "horizons": [
        {
          "top_cm": 0,
          "bottom_cm": 15,
          "designation": "A",
          "munsell_moist": {"hue": "2.5YR", "value": 3, "chroma": 4,
                              "confidence": 0.85, "source_quote": "vermelho-escuro"},
          "clay_pct": {"value": 50, "confidence": 0.9, "source_quote": "muito argilosa (50%)"},
          "oc_pct"  : {"value": 2.0, "confidence": 0.85, "source_quote": "C org. 2.0%"}
        },
        {
          "top_cm": 15,
          "bottom_cm": 65,
          "designation": "Bw1",
          "munsell_moist": {"hue": "2.5YR", "value": 3, "chroma": 6,
                              "confidence": 0.85, "source_quote": "vermelho"},
          "clay_pct": {"value": 60, "confidence": 0.9, "source_quote": "muito argilosa"},
          "oc_pct"  : {"value": 1.2, "confidence": 0.85, "source_quote": "C org. 1.2%"}
        }
      ]
    }'

Wire the mock provider to return this JSON, send a prompt through `validate_or_retry`, and watch the loop:

    mock <- MockVLMProvider$new(responses = list(horizon_json))

    res <- soilKey:::validate_or_retry(
      provider    = mock,
      prompt      = "extract horizons from <fake document>",
      schema      = "horizon",
      max_retries = 0L
    )

    str(res, max.level = 2)
    #> List of 3
    #>  $ data    :List of 1
    #>   ..$ horizons:List of 2
    #>  $ raw     : chr "{\n  \"horizons\": [\n    {\n      \"top_cm\": 0,\n      \"bottom_cm\": 15,\n      \"designation\": \"A\",\n   "| __truncated__
    #>  $ attempts: int 1

The returned `res$data` is the parsed (validated) R object. `res$attempts == 1` because the canned response passed validation on the first try. The `validation_error_at` argument of `MockVLMProvider$new()` forces an attempt to return malformed JSON, exercising the retry path – see `tests/testthat/test-vlm-extract.R` for a worked retry-loop test.

# 4. Merging extraction into a `PedonRecord`

`apply_horizons_extraction()` consumes the parsed JSON and merges it into a `PedonRecord`, recording each value in the provenance log with `source = "extracted_vlm"`.

    pr <- PedonRecord$new(
      site = list(id = "VLM-demo", lat = -22.5, lon = -43.7,
                    country = "BR", parent_material = "gneiss"),
      horizons = data.table::data.table(top_cm = numeric(0), bottom_cm = numeric(0))
    )

    added <- soilKey:::apply_horizons_extraction(pr, res$data, overwrite = TRUE)
    cat("Provenance entries added:", added, "\n")
    #> Provenance entries added: 12

    # Inspect what landed in the horizons table.
    pr$horizons[, .(top_cm, bottom_cm, designation,
                    munsell_hue_moist, munsell_value_moist, munsell_chroma_moist,
                    clay_pct, oc_pct)]
    #>    top_cm bottom_cm designation munsell_hue_moist munsell_value_moist
    #>     <num>     <num>      <char>            <char>               <num>
    #> 1:      0        15           A             2.5YR                   3
    #> 2:     15        65         Bw1             2.5YR                   3
    #>    munsell_chroma_moist clay_pct oc_pct
    #>                   <num>    <num>  <num>
    #> 1:                    4       50    2.0
    #> 2:                    6       60    1.2

# 5. Provenance and evidence grade

Every value that came from the VLM carries `source = "extracted_vlm"` in the provenance log. The classification’s evidence grade reacts:

    prov <- pr$provenance
    head(prov[, .(horizon_idx, attribute, source, confidence)])
    #>    horizon_idx            attribute        source confidence
    #>          <int>               <char>        <char>      <num>
    #> 1:           1          designation extracted_vlm         NA
    #> 2:           1    munsell_hue_moist extracted_vlm       0.85
    #> 3:           1  munsell_value_moist extracted_vlm       0.85
    #> 4:           1 munsell_chroma_moist extracted_vlm       0.85
    #> 5:           1             clay_pct extracted_vlm       0.90
    #> 6:           1               oc_pct extracted_vlm       0.85

Even the simplest call to `classify_wrb2022()` will reflect the lower-confidence VLM source via a lower evidence grade than the same profile assembled from lab data. (See vignette 02 for the grade ladder.)

# 6. Production: swap the mock for an `ellmer` chat

In production, replace `MockVLMProvider` with a chat object from `ellmer`. The rest of the pipeline is unchanged.

## 6a. Local-first with Gemma 4 (Ollama)

For sensitive field descriptions, governmental surveys or just for reproducibility without an API key, the recommended path is a local Gemma 4 model via Ollama. The data never leaves your machine.

    # Install once (any platform)
    ollama pull gemma4:e4b           # ~3 GB, multimodal, fits a laptop
    # ollama pull gemma4:31b         # frontier dense, best quality
    # ollama serve                   # background server

    # Local Gemma 4 edge -- multimodal text + image (and audio).
    provider <- vlm_provider("ollama")             # default: gemma4:e4b
    # provider <- vlm_provider("ollama", model = "gemma4:31b")  # frontier

    extract_horizons_from_pdf(
      pedon       = pr,
      pdf_path    = "field-reports/perfil-LV-001.pdf",
      provider    = provider,
      max_retries = 3L
    )

## 6b. Cloud providers (Anthropic, OpenAI, Google)

    # install.packages("ellmer")

    # Anthropic Claude (needs ANTHROPIC_API_KEY in the environment).
    provider <- vlm_provider("anthropic")          # default: claude-sonnet-4-7

    # Or OpenAI / Google in the same one-liner shape:
    # provider <- vlm_provider("openai")           # default: gpt-4o
    # provider <- vlm_provider("google")           # default: gemini-2.0-pro

    extract_horizons_from_pdf(
      pedon       = pr,
      pdf_path    = "field-reports/perfil-LV-001.pdf",
      provider    = provider,
      max_retries = 3L
    )

## 6c. The one-liner: `classify_from_documents()`

For the canonical case – “I have a PDF and (optionally) a profile photo, give me the three classifications and a one-pager report” – chain everything in a single call. The default provider is local Gemma 4 edge:

    res <- classify_from_documents(
      pdf      = "perfil_042_descricao.pdf",
      image    = "perfil_042_parede.jpg",
      report   = "perfil_042.html"            # optional output report
    )

    res$classifications$wrb$name
    #> "Geric Ferric Rhodic Chromic Ferralsol (Clayic, Humic, Dystric, Ochric, Rubic)"

    res$classifications$sibcs$name
    #> "Latossolos Vermelhos Distroficos tipicos, argilosa, moderado"

    res$classifications$usda$name
    #> "Rhodic Hapludox"

Photos go through `extract_munsell_from_photo()`; site metadata through `extract_site_from_fieldsheet()`. All three follow the same validate-or-retry contract so they accept any provider that exposes `$chat(prompt, ...)`.

The next vignette (`v05_spatial_spectra_pipeline`) shows how the SoilGrids spatial prior and OSSL Vis-NIR predictions can fill remaining gaps in the pedon before classification.