veneer package

Submodules

veneer.bulk module

class veneer.bulk.VeneerRetriever(destination, port=9876, host='localhost', protocol='http', retrieve_daily=True, retreive_monthly=True, retrieve_annual=True, retrieve_slim_ts=True, retrieve_single_ts=True, retrieve_single_runs=True, retrieve_daily_for=[], retrieve_ts_json=True, retrieve_ts_csv=False, print_all=False, print_urls=True)

Bases: object

Retrieve all information from a Veneer web service and write it out to disk in the same path structure.

Typically used for creating/archiving static dashboards from an existing Veneer web application.

mkdirs(directory)
retrieve_across_runs(results_set)
retrieve_all(destination, **kwargs)
retrieve_csv(url)
retrieve_json(url, **kwargs)
retrieve_multi_ts(ts_results, run=None)
retrieve_resource(url, ext)
retrieve_runs()
retrieve_this_daily(ts_url)
retrieve_ts(ts_url)
retrieve_variables()
save_data(base_name, data, ext, mode='b')
translate_url(orig, run=None, loc=None, elem=None, var=None)
unique_results_across_runs(all_results)

veneer.extensions module

veneer.extensions.add_network_methods(target)

Attach extension methods to an object that represents a Veneer network. Note: The ‘network_‘ prefix will be removed from method names.

target: Veneer network object to attach extension methods to.

veneer.extensions.find_network_model(self, model_name)

Find information about a node type by its resource name

model_name: str, name of node type

Example:

v = Veneer() network = v.network() waterusers = network.find_network_model(‘WaterUserNodeModel’)

veneer.extensions.network_as_dataframe(self)

Find all the links in the network that are immediately downstream of a given node.

Parameters:

  • node - the node to search on. Expects the node feature object
veneer.extensions.network_models(self)

Return a list of models within the network.

Extracts this information through the icon resource attribute.

Example:

v = Veneer() network = v.network() node_models = network.models()

veneer.extensions.network_node_names(self)

Return a list of node names as found within the network.

Example:

v = Veneer() network = v.network() node_names = network.node_names()

veneer.extensions.network_outlet_nodes(self)

Find all the nodes in the network that are outlets - ie have no downstream nodes.

Note: Filters out Water User Nodes

Returns each node as a feature (ie with fields ‘geometry’ and ‘properties’)

Find all the links in the network that are immediately upstream of a given node.

Parameters:

  • node - the node to search on. Expects the node feature object

veneer.general module

class veneer.general.Veneer(port=9876, host='localhost', protocol='http', prefix='', live=True)

Bases: object

Acts as a high level client to the Veneer web service within eWater Source.

apply_input_set(name)

Have Source apply a given input set

configure_recording(enable=[], disable=[])

Enabled and disable time series recording in the Source model.

enable: List of time series selectors to enable,

disable: List of time series selectors to disable

Note: Each time series selector is a python dictionary object with up to three keys:
  • NetworkElement
  • RecordingElement
  • RecordingVariable

These are used to match time series available from the Source model. A given selector may match multiple time series. For example, a selector of {‘RecordingVariable’:’Downstream Flow Volume’} will match Downstream Flow Volume from all nodes and links.

Any empty dictionary {} will match ALL time series in the model.

So, for example, you could disable ALL recording in the model with

v = Veneer() v.configure_recording(disable=[{}])

Note, the time series selectors in enable and disable may both match the same time series in some cases. In this case, the ‘enable’ will take effect.

convert_dates(events)
data_source(name)

Return an individual data source, by name.

Note: Will include the each time series associated with the data source IN FULL

data_source_item(source, name=None, input_set='__all__')
data_sources()

Return a SearchableList of the data sources in the Source model

Note: Returns a summary (min,max,mean,etc) of individual time series - NOT the full record.

You can get the time series by retrieving individual data sources (data_source method)

drop_all_runs()

Tell Source to drop/delete ALL current run results from memory

drop_run(run='latest')

Tell Source to drop/delete a specific set of results from memory.

run: Run number to delete. Default =’latest’. Valid values are ‘latest’ and integers from 1

functions()

Return a SearchableList of the functions in the Source model.

input_sets()

Return a SearchableList of the input sets in the Source model

Each input set will be a Python dictionary representing the different information in the input set

network()

Retrieve the network from Veneer.

The result will be a Python dictionary in GeoJSON conventions.

The ‘features’ key of the returned dictionary will be a SearchableList, suitable for querying for different properties - eg to filter out just nodes, or links, or catchments.

Example: Find all the node names in the current Source model

v = Veneer() network = v.network() nodes = network[‘features’].find_by_feature_type(‘node’) node_names = nodes._unique_values(‘name’)

parse_veneer_date(txt)
post_json(url, data=None, async=False)
result_matches_criteria(result, criteria)
retrieve_csv(url)

Retrieve data from the Veneer service, at the given url path, in CSV format.

url: Path to required resource, relative to the root of the Veneer service.

NOTE: CSV responses are currently only available for time series results

retrieve_json(url)

Retrieve data from the Veneer service at the given url path.

url: Path to required resource, relative to the root of the Veneer service.

retrieve_multiple_time_series(run='latest', run_data=None, criteria={}, timestep='daily', name_fn=<function name_element_variable at 0x0000000007F068C8>)

Retrieve multiple time series from a run according to some criteria.

Return all time series in a single Pandas DataFrame with date time index.

you can an index of run results via run_data. If you don’t the method will first retrieve an index based on the value of the run parameter (default=’latest’)

criteria should be regexps for the fields in a Veneer time series record:
  • NetworkElement
  • RecordingElement
  • RecordingVariable
  • TimeSeriesName
  • TimeSeriesUrl

These criteria are used to identify which time series to retrieve.

timestep should be one of ‘daily’ (default), ‘monthly’, ‘annual’

All retrieved time series are returned in a single Data Frame.

You can specify a function for naming the columns of the Data Frame using name_fn. This function should take the results summary (from the index) and return a string. Example functions include:

  • veneer.name_time_series (uses the full name of the time series, as provided by Source)
  • veneer.name_element_variable (DEFAULT: users the name of the network element and the name of the variable)
  • veneer.name_for_location (just use the name of the network element)
  • veneer.name_for_variable (just use the name of the variable)
retrieve_run(run='latest')

Retrieve a results summary for a particular run.

This will include references to all of the time series results available for the run.

run: Run to retrieve. Either ‘latest’ (default) or an integer run number from 1

retrieve_runs()

Retrieve the list of available runs.

Individual runs can be used with retrieve_run to retrieve a summary of results

run_model(params={}, start=None, end=None, async=False, **kwargs)

Trigger a run of the Source model

params: Python dictionary of parameters to pass to Source. Should match the parameters expected
of the running configuration. (If you just want to set the start and end date of the simulation, use the start and end parameters

start, end: The start and end date of the simulation. Should be provided as Date objects or as text in the dd/mm/yyyy format

async: (default False). If True, the method will return immediately rather than waiting for the simulation to finish.
Useful for triggering parallel runs. Method will return a connection object that can then be queried to know when the run has finished.

kwargs: optional named parameters to be used to update the params dictionary

In the default behaviour (async=False), this method will return once the Source simulation has finished, and will return the URL of the results set in the Veneer service

run_server_side_script(script, async=False)

Run an IronPython script within Source.

Requires Veneer to be running with ‘Allow Scripts’ option.

script: the full text of an IronPython script to execute from within Source.

NOTE: In many cases, it is possible (and desirable) to call helper methods within Veneer.model, rather than write your own IronPython script.

send(url, method, payload=None, headers={}, async=False)
send_json(url, data, method, async=False)
update_function(fn, value)

Update a function within Source

fn: str, name of function to update.

update_input_set(name, input_set)

Modify the input set and send to Source.

name: str, name of input set input_set: A Python dictionary representing the updated input set. Should contain the same fields as the input set

returned from the input_sets method.
update_json(url, data, async=False)

Issue a PUT request to the Veneer service to update the data held at url

url: Path to required resource, relative to the root of the Veneer service.

data: Data to update.

NOTE: This method will typically be used internally, by other Veneer methods. Usually, you will want to call one of these other methods to update something specific. For example, configure_recording to enable and disable time series recorders in the model.

update_variable_piecewise(name, values)

Update piecewise linear function for a given variable.

name: str, variable name to update.

update_variable_time_series(name, timeseries)
variable(name)

Returns details of a particular variable

variable_piecewise(name)

Returns piecewise linear function for a particular variable

variable_time_series(name)

Returns time series for a particular variable

variables()

Return a SearchableList of the function variables in the Source model

veneer.general.log(text)
veneer.general.name_element_variable(result)
veneer.general.name_for_location(result)

Name the retrieved time series based on the network location only.

Useful when retrieving the same variable from multiple locations.

veneer.general.name_for_variable(result)

Name the retrieved time series based on the variable only.

Useful when retrieving multiple time series from one network location.

veneer.general.name_time_series(result)

Name the retrieved time series based on the full name of the time series (including variable and location)

veneer.general.read_sdt(fn)
veneer.general.to_source_date(the_date)

veneer.manage module

veneer.manage.configure_non_blocking_io(processes, stream)
veneer.manage.find_veneer_cmd_line_exe(project_fn=None, source_version=None)
veneer.manage.kill_all_on_exit(processes)
veneer.manage.start(project_fn, n_instances=1, ports=9876, debug=False, remote=True, script=True, veneer_exe=None)

Start one or more copies of the Veneer command line progeram with a given project file

Parameters:

  • project_fn - Path to a Source project file (.rsproj)

  • n_instances - Number of copies of the Veneer command line to start (default: 1)

  • ports - A single port number, indicating the port number of the first copy of the Veneer command line,

    OR a list of ports, in which case len(ports)==n_instances (default: 9876)

  • debug - Set to True to echo all output from Veneer Command Line during startup

  • remote - Allow remote connections (requires registration)

  • script - Allow IronPython scripts

  • veneer_exe - Optional (but often required) path to the Veneer Command Line. If not provided,

    veneer-py will attempt to identify the version of Veneer Command Line to invoke. If there is a source_version.txt file in the same directory as the project file, this text file will be consulted to identify the version of Source.

veneer.pest module

class veneer.pest.CalibrationObservations(veneer_prefix, delimiter)

Bases: veneer.pest.ConfigItemCollection

compare(ts_name, mod_ref, stat=<function nse at 0x0000000008286BF8>, target=None, aggregation=None, time_period=None, obsnme=None, mod_scale=1)
pif_line(obs)
pif_text()
script()
class veneer.pest.CalibrationParameters(delimiter='$', instruction_prefix='v.')

Bases: veneer.pest.DeferredActionCollection

declarations()
describe(parnme, parval1, parlbnd, parubnd, **kwargs)
referenced_parameters()
class veneer.pest.Case(name, optimiser='pest', model_servers=[9876], random_seed=1111)

Bases: object

outputs_fn()
pcf_fn()
pcf_text()
pif_fn()
pif_text()
prf_fn()
prf_text()
ptf_fn()
ptf_text()
run()
runner_fn()
slave_name(p)
stdio_params()
write_connection_file(wd, port)
class veneer.pest.ConfigItemCollection(template)

Bases: object

add(*pargs, **kwargs)
declarations()
validate()
class veneer.pest.DeferredActionCollection(delimiter, instruction_prefix)

Bases: object

script(transform=None)
script_line(instruction, transform=None)
class veneer.pest.DeferredCall(parameter, delimiter)

Bases: object

argstring()
is_cal_param(val)
to_arg(val)
class veneer.pest.ObservedData

Bases: veneer.pest.DeferredActionCollection

all_files()
copy_to(slave_dir)
class veneer.pest.PestParameter(parnme, **kwargs)

Bases: object

declaration()
veneer.pest.validate_dict(the_dict)

veneer.pest_runtime module

veneer.pest_runtime.find_port()
veneer.pest_runtime.write_outputs(outputs, fn)

veneer.server_side module

class veneer.server_side.VeneerCatchmentActions(ironpython)

Bases: veneer.server_side.VeneerFunctionalUnitActions

Helpers for querying/modifying the catchment model setup

Specific helpers exist under:

  • .runoff (rainfall runoff in functional units)
  • .generation (constituent generation in functional units)
  • .subcatchment (subcatchment level models)
get_areas(catchments=None)
get_functional_unit_areas(catchments=None, fus=None)

Return the area of each functional unit in each catchment:

Parameters:

catchments: Restrict to particular catchments by passing a list of catchment names

fus: Restrict to particular functional unit types by passing a list of FU names

get_functional_unit_types(catchments=None, fus=None)

Return a list of all functional unit types in the model

set_functional_unit_areas(values, catchments=None, fus=None)

Set the area of each functional unit in each catchment:

Parameters:

values: List of functional unit areas to apply.

catchments: Restrict to particular catchments by passing a list of catchment names

fus: Restrict to particular functional unit types by passing a list of FU names

class veneer.server_side.VeneerCatchmentGenerationActions(catchment)

Bases: veneer.server_side.VeneerFunctionalUnitActions

Helpers for querying/modifying the constituent generation model setup

Query options:

  • catchments - the name(s) of catchments to match when querying/configuring.
  • fus - the type(s) of functional units to match when querying/configuring
class veneer.server_side.VeneerFunctionalUnitActions(catchment)

Bases: veneer.server_side.VeneerNetworkElementActions

class veneer.server_side.VeneerIronPython(veneer)

Bases: object

Helper functions for manipulating the internals of the Source model itself.

These features rely on ‘Allow Scripting’ being enabled in Veneer in order to post custom IronPython scripts to Source.

Specific helpers for querying and modifying the catchment components and instream components exist under .catchment and .link, respectively.

eg

v = Veneer() v.model.catchment? v.model.link?

add_constituent(new_constituent)
add_to_list(theThing, theValue, namespace=None, literal=False, fromList=False, instantiate=False, allow_duplicates=False)
assign_time_series(theThing, theValue, column=0, from_list=False, literal=True, data_group=None, namespace=None)
call(theThing, namespace=None)
clean_script(script)
find_inputs(model_types)

Find the input names for a given model type or list of model types

Returns:
  • A list of inputs (if a single model type is provided)
  • A dictionary model type name -> list of inputs (if more than model type)
find_model_type(model_type, must_be_model=True)

Search for model types matching a given string pattern

eg

v.model.find_model_type(‘emc’)

find_parameters(model_types)

Find the parameter names for a given model type or list of model types

Returns:
  • A list of parameters (if a single model type is provided)
  • A dictionary model type name -> list of parameters (if more than model type)
find_properties(model_types)

Find all fields and properties for a given model type or list of model types

Returns:
  • A list of fields and properties (if a single model type is provided)
  • A dictionary model type name -> list of fields and properties (if more than model type)
get(theThing, namespace=None)

Retrieve a value, or list of values from Source using theThing as a query string.

Query should either start with scenario OR from a class imported using namespace

get_constituents()
get_data_sources(theThing, namespace=None)

Get references (Veneer URLs) to the

runScript(script, async=False)
run_script(script, async=False)
running_configuration(new_value=None, return_all=False)

Set or get the current running configuration for the Source model (eg Single Simulation), or get a list of all available running configurations

eg:

current = v.model.running_configuration()

v.model.running_configuration(‘Single Simulation’)

all_available = v.model.running_configuration(return_all=True)

save(fn=None)
set(theThing, theValue, namespace=None, literal=False, fromList=False, instantiate=False)
sourceHelp(theThing='scenario', namespace=None)

Get some help on what you can do with theThing, where theThing is something you can access from a scenario, or the scenario itself.

If theThing is a method, returns details on how to call the method If theThing is an object, returns a list of public methods and properties

eg v.model.sourceHelp() # returns help on the scenario object v.model.sourceHelp(‘scenario.CurrentConfiguration’)

sourceScenarioOptions(optionType, option=None, newVal=None)
source_scenario_options(optionType, option=None, newVal=None)
class veneer.server_side.VeneerLinkActions(ironpython)

Bases: object

create(from_node, to_node, name=None)
class veneer.server_side.VeneerLinkConstituentActions(link)

Bases: veneer.server_side.VeneerNetworkElementConstituentActions

class veneer.server_side.VeneerLinkRoutingActions(link)

Bases: veneer.server_side.VeneerNetworkElementActions

Queries and actions relating to streamflow routing models.

Query options:

  • links - the name(s) of links to match when querying/configuring.

For example:

v.model.links.routing.get_models(links=[‘Link #1’,’Link #2’])

set_models(models, fromList=False, **kwargs)
class veneer.server_side.VeneerNetworkElementActions(ironpython)

Bases: object

assign_time_series(parameter, values, data_group, column=0, literal=True, fromList=False, **kwargs)

Assign an input time series to a model input input

get_data_sources(parameter, by_name=False, **kwargs)

Return pointers (veneer URLs) to the data sources used as input to a particular parameter

get_models(by_name=False, **kwargs)

Return the models used in a particular context

get_param_values(parameter, by_name=False, **kwargs)

Return the values of a particular parameter used in a particular context

help(param=None, **kwargs)
names(**kwargs)

Return the names of the network elements

set_models(models, fromList=False, **kwargs)

Assign computation models.

set_param_values(parameter, values, literal=False, fromList=False, instantiate=False, **kwargs)

Set the values of a particular parameter used in a particular context

class veneer.server_side.VeneerNetworkElementConstituentActions(ironpy)

Bases: veneer.server_side.VeneerNetworkElementActions

assign_time_series(parameter, values, data_group, column=0, literal=True, fromList=False, aspect='played', **kwargs)

Assign an input time series to a rainfall runoff input

initialise_played_constituents(**kwargs)
class veneer.server_side.VeneerNodeActions(ironpython)

Bases: veneer.server_side.VeneerNetworkElementActions

Queries and actions relating to nodes (incuding node models).

Query options:

  • nodes - the name(s) of nodes to match when querying/configuring.
  • node_types - the type(s) of nodes to match when querying/configuring

For example:

v.model.nodes.get_models(nodes=’Fish River’)

create(name, node_type, location=None, schematic_location=None)
remove(name)
class veneer.server_side.VeneerNodeConstituentActions(node)

Bases: veneer.server_side.VeneerNetworkElementConstituentActions

Queries and actions relating to nodes (incuding node models).

Query options:

  • nodes - the name(s) of nodes to match when querying/configuring.
  • node_types - the type(s) of nodes to match when querying/configuring

For example:

v.model.nodes.constituents.get_models(nodes=’Fish River’)

class veneer.server_side.VeneerRunoffActions(catchment)

Bases: veneer.server_side.VeneerFunctionalUnitActions

Helpers for querying/modifying the rainfall runoff model setup

Query options:

  • catchments - the name(s) of catchments to match when querying/configuring.
  • fus - the type(s) of functional units to match when querying/configuring
assign_time_series(parameter, values, data_group, column=0, catchments=None, fus=None, literal=True, fromList=False)

Assign an input time series to a rainfall runoff input

class veneer.server_side.VeneerScriptGenerators(ironpython)

Bases: object

find_feature_by_name()
class veneer.server_side.VeneerSourceUIHelpers(ironpython)

Bases: object

open_editor(name_of_element)
class veneer.server_side.VeneerSubcatchmentActions(catchment)

Bases: veneer.server_side.VeneerNetworkElementActions

Helpers for querying/modifying the subcatchment-level models

Query options:

  • catchments - the name(s) of catchments to match when querying/configuring.
add_model(model_type, add_if_existing=False, catchments=None, allow_duplicates=False)
reset(namespace=None, **kwargs)

veneer.stats module

Statistics Module

Various bivariate statistics functions typically used as Goodness Of Fit measures for hydrologic models.

Functions modules contained here are designed to be called on pairs of DataFrames, (observed and predicted) with equivalent columns.

All functions should work in various scenarios, such as:

  • Dataframes with columns for different constituents,
  • Dataframes with columns for different sites
  • Daily, monthly, annual time steps
  • Other timeperiods, such data aggregated to water year

Broadly speaking, these statistics will work with data that

veneer.stats.PBIAS(obs, pred)
veneer.stats.intersect(obs, pred)

Return the input pair of dataframes (obs,pred) with a common index made up of the intersection of the input indexes

veneer.stats.ioad(obs, pred)

ioad - Index of Agreement

veneer.stats.nse(obs, pred)

Nash-Sutcliffe Efficiency

veneer.stats.ppmc(obs, pred)

Pearson Product Moment Correlation (PPMC)

veneer.stats.rae(obs, pred)

Relative Absolute Error (RAE)

veneer.stats.rsr(obs, pred)

veneer.utils module

class veneer.utils.GroupedDictionary(initial={})

Bases: collections.UserDict

count()
class veneer.utils.SearchableList(the_list, nested=[])

Bases: object

SearchableList of objects

Use

  • find_by_X(Y) to find all entries in the list where property X is equal to Y

  • group_by_X() to return a Python dictionary with keys being the unique values of property X and entries

    being a list of original list entries with matching X values

  • _unique_values(X) return a set of unique values of property X

  • _all_values(X) return a list of all values of property X (a simple select)

  • _select([‘X’,’Y’]) to select particular properties

For example:

v = Veneer() network = v.network() the_list = network[‘features’] # A searchable list of network features links = the_list.find_by_feature_type(‘link’) # All features have a ‘feature_type’ property

grouped = the_list.group_by_feature_type() grouped[‘link’] is a list of links grouped[‘node’] is a list of nodes grouped[‘catchment’] is a list of catchments

as_dataframe()
veneer.utils.objdict(orig)
veneer.utils.read_veneer_csv(text)

Module contents