---
title: "Non-dendritic networks"
author: "dblodgett@usgs.gov"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Non-dendritic networks}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
library(hydroloom)
library(dplyr)

local <- (Sys.getenv("BUILD_VIGNETTES") == "TRUE")

knitr::opts_chunk$set(
  collapse = TRUE,
  warning = FALSE,
  comment = "#>",
  fig.width = 6,
  fig.height = 6,
  fig.align = "center",
  eval = local
)

oldoption <- options(scipen = 9999)
```

# Introduction

`vignette("hydroloom")` and `vignette("advanced_network")` cover the basics of network topology representation and the attributes that build on a strictly dendritic network. This vignette extends those topics to the `hydroloom` functionality that supports non-dendritic networks.

The term "non-dendritic" refers to any network that does not follow a dendritic flow pattern. Typically that means one or more flowlines diverting from a primary flow path. Non-dendritic can also refer to endorheic basins nested within an otherwise dendritic basin — but this article addresses the diverted-flowline case.

The terms "diversion" and "divergence" are used in similar contexts but carry distinct meanings. A **diversion is a diverted flowline** (waterbody), whereas a **divergence is where the network diverges**. "Diversion" is often associated with anthropogenic features, but a diversion can also be a naturally occurring diverted flow.

# Non-dendritic topology attributes

In a non-dendritic network, one downstream path at each divergence is treated as primary and the others as secondary. `hydroloom` supports several attributes for tracking that primary/secondary categorization.

## fromnode and tonode

`fromnode` and `tonode` store a flow network as an edge-node topology where every feature has exactly one upstream node and exactly one downstream node. Nodes give every confluence and divergence a single identifier, which makes converting a flow network to a graph and many downstream analyses cleaner.

## divergence

The `divergence` attribute indicates if a downstream connection is primary (1) or secondary (2). If 0, a connection is not downstream of a divergence. This attribute is useful as it facilitates following a flow network in the "downstream mainstem" direction at every divergence.

## return divergence

The `return divergence` attribute indicates that one or more of the features upstream of a given feature originates from a divergence.  If 0, the upstream features are not part of a diversion. If 1, one or more of the upstream features is part of a diversion.

## stream calculator

The `stream calculator` attribute is part of the modified Strahler stream order as implemented in the NHDPlus data model. It indicates if a given feature is part of the downstream mainstem dendritic network or is part of a diverted path. If 0, the path is part of a diversion. Otherwise `stream calculator` will be equal to stream order. When generating Strahler stream order, if stream calculator is 0 for a given feature, that feature is not considered for incrementing downstream stream order.

## summary

As a system, `stream calculator`, `divergence` and `return divergence` support network navigation and processing in the context of diverted paths.

1. A feature at the top of a diversion will have `divergence` set to 1.
1. All features that are part of a diversion that has not yet recombined with a main path, will have `stream calculator` set to 0.
1. A feature that is just downstream of where a diversion recombines with a main path will have `return divergence` set to 1.

## Divergence case study: dropped vs preserved secondary paths

Before applying the full pipeline to real data, look at what happens
to a divergence when a non-dendritic network is forced into the
dendritic `hy_topo` form. The five-edge example from
`vignette("hydroloom")` has one divergence at feature 1 and one
confluence at feature 5.

We start in `hy_node` form with `fromnode`/`tonode` and a `divergence`
column marking the secondary path. The network looks like this, with
feature 1 flowing into node N2, where it splits into feature 2 (main
path) and feature 4 (diverted secondary path), both of which rejoin at
node N4 and flow out through feature 5:

```
       N1
        |
        1
        | 
        v
       N2
      /  \
     2    4
    /      \ (4 is diverted)
   v        v
   N3       |
    \       |
     3      | 
      \    /
       v  v
        N4
         |
         5
         |
         v
         N5
```

```{r synth_node, eval=TRUE}
library(hydroloom)

node_df <- data.frame(
  id         = c(1, 2, 3, 4, 5),
  fromnode   = c("N1", "N2", "N3", "N2", "N4"),
  tonode     = c("N2", "N3", "N4", "N4", "N5"),
  divergence = c(0,  2,  0,  1,  0)
)

x_node <- hy(node_df)
class(x_node)
nrow(x_node)
```

Converting to a dendritic edge list with
`add_toids(return_dendritic = TRUE)` keeps one row per feature and
drops the secondary downstream connection from feature 1:

```{r synth_topo, eval=TRUE}
x_topo <- add_toids(x_node, return_dendritic = TRUE)
class(x_topo)
nrow(x_topo)
x_topo[, c("id", "toid")]
```

Setting `return_dendritic = FALSE` preserves both downstream connections
by repeating `id == 1`. The result is no longer dendritic, so `hy()`
classifies it as `hy_flownetwork`:

```{r synth_fn, eval=TRUE}
x_fn <- add_toids(x_node, return_dendritic = FALSE)
class(x_fn)
nrow(x_fn)
x_fn[, c("id", "toid")]
```

Both downstream paths from feature 1 are preserved.
`hy_capabilities()` shows which functions are callable at each stage;
that workflow is demonstrated end to end in
`vignette("network_navigation")`.

## Bringing it all together

The example below shows how we can recreate the non-dendritic attributes and use them in practice.

We'll start with the small sample watershed that's included in `hydroloom` and select only the attributes required to recreate the non-dendritic network.

```{r}
x <- sf::read_sf(system.file("extdata/new_hope.gpkg",
  package = "hydroloom"))

# First we select only an id, a name, and a feature type.
flow_net <- x |>
  select(COMID, GNIS_ID, FTYPE) |>
  sf::st_transform(5070)

# Now we convert the geometric network to an attribute topology
# and convert that to a node topology and join our attributes back
flow_net <- flow_net |>
  make_attribute_topology(min_distance = 5) |>
  hydroloom::make_node_topology(add_div = TRUE) |>
  left_join(sf::st_drop_geometry(flow_net), by = "COMID")

# We only have one outlet so it doesn't matter if it is coastal
# or inland but we have to provide it.
outlets <- filter(flow_net, !tonode %in% fromnode)

# We have these feature types. A larger dataset might include
# things like canals which would not be considered  "major"
unique(flow_net$FTYPE)

# compare dendritic vs non-dendritic toid row counts to see how many
# secondary paths exist in the data
flow_net_with_div <- add_divergence(flow_net,
  coastal_outlet_ids = c(),
  inland_outlet_ids = outlets$COMID,
  name_attr = "GNIS_ID",
  type_attr = "FTYPE",
  major_types = unique(flow_net$FTYPE))

dend <- add_toids(flow_net_with_div, return_dendritic = TRUE)
nondend <- add_toids(flow_net_with_div, return_dendritic = FALSE)

nrow(dend)
nrow(nondend)
nrow(nondend) - nrow(dend)

# now we run the add_divergence, add_toids, and add_streamorder
flow_net <- add_divergence(flow_net,
  coastal_outlet_ids = c(),
  inland_outlet_ids = outlets$COMID,
  name_attr = "GNIS_ID",
  type_attr = "FTYPE",
  major_types = unique(flow_net$FTYPE)) |>
  add_toids() |>
  add_streamorder() |>
  add_return_divergence()

# Make sure we reproduce what came from our source NHDPlus data.
sum(flow_net$divergence == 2)
sum(x$Divergence == 2)
all(flow_net$divergence == x$Divergence)
sum(flow_net$return_divergence == x$RtnDiv)

names(flow_net)
```
With the above code, we removed all attributes other than an ID, a name and a feature type and recreated both a dendritic (toid) and non-dendritic (fromnode tonode) topology. We added `divergence` attribute, `stream_order`, `stream_calculator`, and `return_divergence` attributes.
