Adding Upstream / Downstream Links

Author:        Paul Jendrowski

Organization:  NOAA, National Weather Service, Blacksburg, VA

email:         paul.jendrowski@noaa.gov

• Overview

• Tips

• Procedure

Overview

1. Convert To and From node attributes in the original streamlink shapefile to upstream and downstream link attributes.
2. Identify and correct broken upstream/downstream links.
3. Convert multiple line segments within a basin into a single line.
4. Map streamlink segments to basins with a spatial join to assign the correct Pfafstetter code to the streamlinks.
5. Assign the upstream/downstream attributes from the streamlinks to the basins by joining on the Pfafstetter fields in the streamlink and basin shapefiles.

Tips

• If there are a large number of records selected which need to be inspected, it is recommended the selection be converted to a temporary theme. Select "Convert to Shapefile" from the theme menu. This will save only the selected records to a new shapefile (or all records if no selection). Then select all features in the temporary theme. Deselect individual features after inspecting; this will help keep track of what features still need to be checked.

• "Convert to Shapefile" is also a convenient way to copy a theme before editing. This feature will export only selected records if there is a selection. It also renames any fields with an Alias in the attribute table to the Alias value and only exports visible fields (a useful method for renaming and/or deleting fields).

• Save the project often when selecting or deselecting records in a table or features in theme; the selection is saved when saving the project. If you accidentally lose the selection, the project can be restarted to restore the selection at some intermediate point instead of having to start over.
• For the standard ArcView selection tool, remember to hold the shift key down to toggle on/off the selection without losing any previous selection. For the Basin Customization extension selection tools, hold down the shift key to add to the selection and hold down the control key to subtract from the selection.
• If you want to make a copy of a theme after you have made a selection but before actually making changes, you cannot use "Convert to Shapefile" because this will save only the selected features. Use the ArcView data manager:
1. From the File menu select Manage Data Sources (this only appears on the File menu with a View as the active window).
2. From the Shapefile Manager dialog box, you can copy, rename and delete shapefiles on disk.
• Use the "Zoom to Selected" button in the view to find where a selection is if it can not be readily seen.
  
• Keep the project cleaned up. Delete unneeded tables from the project and clean up temporary themes from disk using the Shapefile Manager. If you get a message from the Shapefile Manager that a data set is in use, make sure it is not in a view, its table has been deleted from the project and the project has been saved since deleting the theme from the project.
• Keep notes on all the edits you are performing. It is likely you may have to redo some steps so these notes will help speed up the process. Many of the Bsn-Tools functions write notes to the theme's comments (accessed from Theme Properties menu). These comments are only saved within the project. You can save these comments to a file by using the "Theme Comments" function on the Bsn-Tools menu. It is recommended you add descriptions of any manual edits to the theme's comments. Theme comments can be saved to disk using the "Theme Comments" function.
  

Procedure

• 1. Preprocess the original streamlink theme kxxx_str_gen.shp

• 2. Add upstream and downstream links to attribute table

• 3. Add end points to the attribute table

• 4. Check for mislabeled stream segments

• 5. Repair broken links

• 6. Delete Stream segments outside the basin polygons

• 7. Merge multiple line segments

• 8. Merge phantom streamlinks

• 9. Other Edits to Streamlink

• 10. Check final streamlink theme with Spatial Join

• 11. Join streamlink attributes to basin attribute table

• 12. Join the basin attributes to the streamlink

1. Preprocess the original streamlink theme kxxx_str_gen.shp

• Delete streamlink segments that fall outside the basin polygons. Manually search around the edge of the polygon theme and delete streamlink features that do not fall within a basin. (Note use the "Delete" function on the Edit menu, not on the Bsn-Tools menu).
• Check for streamlinks with no [Pfaf_orig] value.
1. Open the streamlink theme table
2. Check for records where [Pfaf_orig] is undefined (sort on the [Pfaf_orig] field, look at top and bottom of list).
3. Select individual records on the the sorted table and then use Zoom to Selected button on the View window to examine each feature to determine what action needs to be taken. In this case, some need to have the [Pfaf_orig] from the basin shapefile assigned to it and some may need to be deleted.
4. Make the table active and from the Table menu, select "Start Editing".
5. Manually enter any new values in the Attribute table. Use the identify tool on the basin shapefile to determine the [Pfaf_orig] value to enter.
6. On the view, select streamlinks that need to be deleted.
1. Open the theme table.
2. From the Edit menu, select "Delete Records".
7. After all edits are completed, from the Table menu, select "Stop Editing" and save the edits.
• Check for duplicate streamlinks by using the "Check for duplicates" function on the Table Bsn-Tools menu. There must not be any duplicate [Fnode_] values before proceeding to the adding upstream and downstream links step.
1. From View, make the original streamlink theme (kxxx_str_gen.shp) active.
2. Click on Open Theme Table button.
3. With the table active, select "Check for Duplicates" from the Bsn-Tools menu.
4. Select the "Fnode_" field from the list of fields.
5. If any records are selected, examine the attributes in the table and determine which of the duplicate [Fnode_] features need to be deleted.
1. On the view, Zoom in on one of the selected streamlinks.
2. Click on the Identify tool ("i" button). The selected features will flash. You can sometimes use this to see where the duplicate features overlap. Another way is to select individual records in the attribute table and while looking at the change in the view.
6. If there are a large number of duplicates, the [Streams_id] field will normally be unique. This field might be able to be used to easily select the duplicate features if the duplicates have [Streams_id] sufficiently different from the original records (i.e., [Streams_id] > 5000):
1. Open the Query dialog box (Table menu, Query or hammer button).
2. Query ([Streams_id] > 5000) replacing 5000 with the appropriate value and using "Select from Set" option.
3. Make sure there are not some valid streams that have a [Streams_id] > 5000 that are included in the selection and manually deselect the valid records in the table.
7. To apply the edits to a new shapefile:
1. Return to the View window and make the streamlink theme the only active theme.
2. From the Bsn-Tools menu, select "Switch Selection" to select the records to keep.
3. Select "Convert to Shapefile" from the theme menu and save the selected records to a new shapefile.
8. Or, to make the edits to the current shapefile:
1. Return to the View window and make the streamlink theme the only active theme.
2. Click on the Open Theme Table button.
3. From the Table menu, select "Start Editing".
4. From the Edit menu, select "Delete Records".
5. From the Table menu, select "Stop Editing" and save the edits.
   

2. Add upstream and downstream links to attribute table

1. Make the stream link theme active and run " Create Links " from the Bsn-Tools menu. Use slcreatelinks.shp for the new shapefile name.
2. Now the Select Upstream/Downstream Tools will work. Try the Select Upstream and Select Downstream tools on different stream segments.

• To facilitate working with the streamlink table, you may want to delete or turn off fields not needed. At the end of the conversion process, the streamlink theme will be assigned all the attributes of the basin polygons. The minimum set of fields required in this process are:
1. Shape
2. Basin_id
3. Parent_id
4. "Upstream" fields.
5. Pfaf_id
6. Flow_acc

3. Add end points to the attribute table

1. From the Bsn-Tools menu select " StreamLink Points ".
2. At the prompt for point to extract, select "First node" and then click the OK button. The coordinates will appear in the attribute table labeled "X_first" and "Y_first".
3. The option for "Mixed" is useful for obtaining a coordinate for creating a point theme for labeling. The mixed mode uses the center point unless the streamlink is a headwater; then the first point is returned.
   

4. Check for mislabeled stream segments

1. Open the streamlink theme's attribute table.
2. Check for records where Pfaf_id is undefined (sort on the [Pfaf_id] field).
3. Select individual records on the the sorted table and then use Zoom to Selected button on the View window to examine each feature to determine what action needs to be taken. In this case, some need to have the Pfaf_id from the basin shapefile assigned to it and some need to be deleted. Either write down the the Basin_id's of the segments that need to be redefined or use a graphic symbol to mark the segment on the View.
4. Make the table active and from the Table menu, select "Start Editing".
5. Manually enter any new Pfaf_ids in the Attribute table. Use the identify tool on the basin shapefile to determine the Pfaf_id to enter.
6. Stop editing the table when done and save the edits.
7. Select any streamlinks that need to be deleted.
8. From the Bsn-Tools menu, select " Delete Basins ".
   

5. Repair broken links

1. To identify possible broken links, perform the following query on the streamlink theme: "([Parent_id] = 0)" using "New Set".
2. Normally, all streams will flow out of the coverage area so look for selected segments in the interior (you will probably have to zoom and pan to see the selected segments). There should be only a few segments selected but coastal sites will have much larger number of valid segments with [Parent_id] = 0. Write down the [Basin_id] of the broken segments or mark their location with a graphic symbol.
3. After all broken links are identified, open the streamlink theme's attribute table and select "Start Editing" from the Table menu.
4. Return to the View containing the streamlink theme and for each broken link:
1. Using the normal selection tool, select the downstream stream link and the immediate upstream segments (only those that had ([Parent_id] = 0) to link together.
2. From the Bsn-Tools menu, select " Link Selected Basins ".
3. The edit can be checked by using the Select Next Downstream tool on the upstream segments that were linked. When using the tool, the upstream and downstream segments will be highlighted. The upstream links have not been corrected yet so the Select Upstream tools will not work.
5. After all links are repaired, open the attribute table and from the Table menu, select "Stop Editing".
6. From the Bsn-Tools menu, Select " Add Upstream Links ". This is required to recompute the upstream links and must always be performed anytime "Link Selected Basins" is used.

6. Delete Stream segments outside the basin polygons

1. Use the normal select tool to select individual stream segments or use the Select Upstream and Downstream tools.
1. For the standard ArcView selection tool, remember to hold the shift key down to toggle on/off the selection with out losing any previous selection. For the basin selection tools, hold down the shift key to add to the selection and hold down the control key to subtract from the selection.
2. Save the project frequently. Saving the project also saves the selection. Therefore, if you accidentally lose the selection, you can restart the project at some intermediate point in the selection process and not have to start over.
2. Once all the features are selected, from the Bsn-Tools menu, select "Delete Basins ". This performs the edits on the shapefile, hence the reason for making a copy of the shapefile before editing.

7. Merge multiple line segments

1. Make the streamlink theme active.
2. From the Bsn-Tools menu, select "Merge Multiple Stream Segments". This will save the edits to a new shapefile. Name this new shapefile slmrgsingle.shp.

8. Merge phantom streamlinks

1. Open the attribute table for the streamlink theme (i.e., slmrgsingle.shp).
2. Return to the view and from the View menu, select "Add Event Theme..."
1. For the table, select the attribute table for the streamlink theme.
2. For the X and Y fields, choose "X_first" and "Y_first", then click the OK button. A point theme will be added to the view.
3. Make the new point theme the active theme, and select "Properties" from the theme menu.
1. Rename the theme to sl-pts (this is to avoid confusion with the streamlink theme's attribute table).
4. Open the attribute table for the sl-pts theme.
5. From the Bsn-Tools menu, select "Check for Duplicates" and select the [Pfaf_id] field.
• If there are a large number of records selected that need to be inspected, it may be useful to export the selection to a new theme. Then select all records in the new theme and turn off individual records as they are inspected. This way it is easier to track which features have already been checked.
6. Inspect all selected records using the points to guide where to look but examining the spatial relationship between the line segments and the basin polygons. There may be some records where the [Pfaf_id] needs to be changed, the line segment merged or deleted. For this step, focus only on the segments that need to be merged (ignore other needed corrections for now). You will primarily be looking for situations as in the figure above, however there may be other situations where the line segments should be merged.
1. On the streamlink theme, select the line segments to be joined using the "Select Next Downstream" tool (inverted triangle symbol) or the normal select tool.
2. Beware of pairs of segments to be joined that are immediately downstream of each other. These may be merged into one segment instead of the desired sets. In this case, these basins will have to be merged in multiple iterations of this process.
   
3. Remember to save the project often in case you accidentally lose the previous selections.
   
7. Once all streamlinks to be merged have been selected, from the Bsn-Tools menu, select "Merge Basins " Name the new shapefile slmrgphantom.shp . Select "No" at the prompt to compute Area/Length (this will be done later).
8. Delete the sl-pts theme from the View and its attribute table from the project. If a temporary theme of the selection was created, delete the theme, its attribute table and clean up the files on disk.

9. Other Edits to Streamlink

1. Open the attribute table for the streamlink theme (i.e., slmrgphantom.shp) .
2. Return to the view and from the View menu, select "Add Event Theme..."
1. For the table, select the Attribute table for the streamlink theme.
2. For the X and Y fields, choose "X_first" and "Y_first", then click the OK button. A point theme will be added to the view.
3. Make the new point theme the active theme.
1. Select "Properties" from the theme menu.
2. Rename the theme to sl-pts (this is to avoid confusion with the stream link polyline theme's attribute table).
4. Open the attribute table for the sl-pts theme.
5. From the Bsn-Tools menu, select "Check for Duplicates" and select the [Pfaf_id] field.
• If there are a large number of records selected that need to be inspected, it may be useful to export the selection to a new theme. Then select all records in the new theme and turn off individual records as they are inspected. This way it is easier to track which features have already been checked.
6. Inspect all selected records using the points to guide where to look but examining the spatial relationship between the line segments and the basin polygons. There may be some records where the [Pfaf_id] needs to be changed, the line segment merged or deleted. Focus on one type of change (merge, delete, reset [Pfaf_id]) for each pass.
• Merging:
1. On the streamlink theme, select the line segments to be joined using the "Select Next Downstream" tool (inverted triangle symbol) or the normal select tool.
2. Beware of pairs of segments to be joined that are immediately downstream of each other. These may be merged into one segment instead of the desired sets. In this case, these basins will have to be merged in multiple iterations of this process.
3. Remember to save the project often in case you accidentally lose the previous selections.
4. Once all streamlinks to be merged have been selected, from the Bsn-Tools menu, select "Merge Basins ". Edits will be saved to a new shapefile. Select "No" at the prompt to compute Area/Length (this will be done later).
• Deleting:
1. It is recommended that the streamlink be saved to a new theme before using the Delete Basins function.
2. Normally only streamlinks around the edge are deleted or headwater segments.
3. Select all streamlink segments to be deleted.
4. From the Bsn-Tools menu, select " Delete Basins ". Warning: edits are made to the current theme.
• Resetting [Pfaf_id]:
1. Open the streamlink theme table.
2. Select "Start Editing" from the Table menu.
3. Return to the View and select the streamlink feature to be edited. Use the Identify tool on the basin shapefile to determine the new [Pfaf_id].
4. Return to the Table and use the Promote button to more easily see selected records in the table.
5. Manually change the [Pfaf_id] in the table.
6. After all changes are made, select "Stop Editing" from the Table menu and save changes.
   
7. Delete the sl-pts theme from the View and attribute tables from the project. If a temporary theme of the selection was created, delete the theme, its attribute table and clean up the files on disk.
8. Repeat these steps until there are no duplicate [Pfaf_id] values in the streamlink attribute table.
   

10. Check final streamlink theme with Spatial Join

1. Open the attribute table for the streamlink theme.
2. Return to the view and from the View menu, select "Add Event Theme..."
1. For the table, select the Attribute table for the streamlink theme.
2. For the X and Y fields, choose "X_first" and "Y_first", then click the OK button. A point theme will be added to the view.
3. Make the new point theme the active theme, and select "Properties" from the theme menu. Rename the theme to sl-pts (this is to avoid confusion with the streamlink polyline theme's attribute table).
4. Make the streamlink points theme and the basin polygon theme active.
5. Click on the Open Tables button to open the attribute tables for both the streamlink points theme and the basin polygon theme.
6. Select the [Shape] field in both tables.
7. Make the streamlink points table active and click on the "Join" button. Since the shape field is the active field in each table, this will perform a spatial join and link the attributes of the polygons to each point that falls within it.
8. Perform the following query on the streamlink points table: ([Pfaf_id] <> [Pfaf_orig]). This will select streamlink points where the Pfafstetter code currently assigned to the streamlink does not match the basin's Pfafstetter code.
• Most of the resulting selected points will not match because the point is actually not falling within the proper basin. This is usually occurs along shorelines where the streamlink line segment extends past the polygon boundary or with very short streamlinks. These points are okay if the Pfafstetter code of the point correctly matches the basin Pfafstetter code the streamlink truly should be associated with.
9. Correct any streamlinks than need correcting by redoing any of the previous processing steps (merge, delete, reset [Pfaf_id]).
10. Remove the streamlink points theme and attribute table.
11. Once all edits to the streamlink are completed, update/create length in miles:
1. Make the streamlink theme the active theme.
2. From the Bsn-Tools menu, select " Area/Length ".

11. Join streamlink attributes to basin attribute table

1. Open the streamlink table.
2. From the Table menu, select "Properties". Turn off the Visible check mark for the [Flow_acc] field and any other unneeded fields or fields that are duplicated in the basin attribute table. Do not turn off the following fields:
1. Shape
2. Basin_id
3. Parent_id
4. Any "Upstream" fields.
5. Pfaf_id
3. In the streamlink table, select the [Pfaf_id] field.
4. Open the attribute table for the basin theme and select the [Pfaf_orig] field.
5. Make sure both the streamlink and basin attribute tables are open and make the basin table the active table. Click on the "Join" button. The streamlink table will automatically close.
6. Return to the view with the basin theme and make the basin theme the active theme.
1. Clear any selection on the basin theme.
2. From the Theme menu, select "Convert to Shapefile..." and save to kxxx_ref_basins.shp. Answer yes to add the new theme to the view.
7. Return to the original basin table with the streamlinks joined to it.
1. In the View, make the basin theme the only active theme,
2. Click on the Open Theme Table button.
3. From the Table menu, select "Remove all Joins".
4. Remove the table from the project.
8. Open the attribute table for the new basin theme.
1. In the View, make the kxxx_ref_basins.shp theme the only active theme.
2. Click on the Open Theme Table button.
3. From the Bsn-Tools menu, Select " Init Basin_id ". This will insure the basins that did not have a corresponding streamlink will have valid [Basin_id], [Parent_id] and upstream values. The changed features will now be selected.
4. The selection now contains basins where there was no streamlink to set the [Basin_id] field. Examine these basins and determine if any of these basins should be deleted or needed to be linked to other basins. Perform any additional edits that are required.
• If you need to link any basins, remember to run "Add Upstream Links" after linking.
9. The upstream links are always sorted by flow accumulation. However, the flow accumulation values used by the streamlinks may not represent the basin areas so it is a good idea to recompute the flow accumulation based on the basin areas. First, the flow accumulation must be reset to zero. You may also want to identify basins that receive flow into them from outside the extent of the shapefile and set their flow accumulation to a flag or reasonable value. This will help identify the mainstem segments better.
1. Make the basin theme active.
2. Manually select basins that receive flow into them from outside the theme.
3. Open the attribute table and from the Table menu, select "Start Editing".
1. Select the [Flow_acc] field.
2. Click the Calculate button and enter a value (9999 for a major river like Mississippi or 999 for a major stream) to be applied to all the selected basins. Click the Ok button.
3. From the Edit menu, select "Switch Selection".
4. Click the Calculate button and enter 0 for the value. Click the Ok button.
5. From the Table menu, select "Stop Editing".
4. Return to the View and make the kxxx_ref_basins.shp theme active.
   
5. From the Bsn-Tools menu, select " Area/Length ".
6. From the Bsn-Tools menu, select " Compute Flow Accum ". This will take a while (> 30 minutes) depending on number of streamlinks, stream network topology and CPU speed.
7. From the Bsn-Tools menu, select " Add Upstream Links ".

12. Join the basin attributes to the streamlink

1. In the View, make the kxxx_ref_basins.shp theme the only active theme.
2. Click on the "Open Theme Table" button.
3. From the table menu, select "Properties..." and turn on the Visible check mark for all fields.
4. Make the [Basin_id] field active.
5. Open the streamlink table.
6. From the Table menu, select "Properties..."
7. Turn off the Visible check mark for fields that are duplicated in the basin theme including the [Pfaf_id], [Parent_id], [Flow_acc], and "Upstream" fields. The fields that must remain on are:
1. Shape
2. Basin_id
8. Make the [Basin_id] field active.
9. Make sure both the streamlink and basin attribute tables are open and make the streamlink table the active table. Click on the "Join" button. The basin table will automatically close.
10. Return to the View and make the streamlink theme the active theme.
11. Clear any selections on the theme.
    
12. From the Theme menu, select "Convert to Shapefile...". Save the new shapefile as kxxx_ref_sl.shp . Save this shapefile as the reference streamlink theme that corresponds to kxxx_ref_basins.shp.
13. Make the streamlink theme with the join the active theme.
1. Click on the "Open Theme Table" button.
2. From the Table menu, select "Remove All Joins".

Done!