Ctrl+K

Batches

Batches#

Bulk operations in AltDSS

Note: Although there are some runtime numbers in this document, required to show some of the motivation for the implementation, this is not intended as a benchmark. By default, it runs on DSS-Extensions modules only, and the times from the official Windows COM implementation should be higher – feel free to uncomment and rerun to check it yourself.

Notebook requirements

This is a Jupyter notebook. Running it as-is will try to download the required files.

You can open and then run this notebook on Google Colab for a quick overview if you don’t want to set up a local environment: Open in Colab.

# When running via Colab, install the package first
import os, subprocess
if os.getenv("COLAB_RELEASE_TAG"):
    print(subprocess.check_output('pip install altdss[all]', shell=True).decode())

# Download the sample circuits and test cases if they do not exist already
from dss.examples import download_repo_snapshot
BASE_PATH = download_repo_snapshot('.', repo_name='electricdss-tst', use_version=False)
CKT7_PATH = BASE_PATH / 'Version8/Distrib/EPRITestCircuits/ckt7/Master_ckt7.dss'
assert CKT7_PATH.exists()

If you are reading this notebook online and would like to run in your local Python installation, you can download both the AltDSS-Python repository (which contains this notebook) and the sample circuits with download_examples.

On Windows, run on a command prompt in your Python environment to install all optional dependencies:

pip install altdss[all]
python -c "from dss.examples import download_examples; download_examples(r'c:\temp', repo_name='AltDSS-Python')"
cd c:\temp\AltDSS-Python\docs\examples
jupyter lab

What are batches?#

Batches, in AltDSS, are Python objects that allow interacting with whole collections or a selection of filtered items of OpenDSS items from the AltDSS engine.

Uniform batches allow interacting with the OpenDSS properties and using most functions that the individual object would present.

Why use batches?#

The batch operations provide a simpler interface and usually better performance profile than directly interacting with each object, or interacting with the classic APIs through either the dedicated interfaces or the Active... interfaces.

Batches pass many operations to the engine level, hence the better performance in Python.

Since AltDSS can be used as a companion package with DSS-Python and/or OpenDSSDirect.py, users can adopt batches where they see fit, without requiring a full codebase rewrite.

Loading a circuit#

Before using the batches, let’s load a circuit and inspect some values.

from altdss import altdss
altdss(f'redirect "{CKT7_PATH}"')

While we don’t have a dedicated dataframe integration, let’s look into the loads using JSON and later compare with the batches:

import pandas as pd
import json
from dss import DSSJSONFlags
df_loads = pd.DataFrame(json.loads(altdss.Load.to_json(DSSJSONFlags.Full))) # include all properties for inspection, even those not set explicitly
df_loads.kV.value_counts()

kV
0.240    534
0.208    333
7.200     39
Name: count, dtype: int64

df_loads.Phases.value_counts()

Phases
1    573
3    333
Name: count, dtype: int64

df_loads.Conn.value_counts()

Conn
wye    906
Name: count, dtype: int64

import numpy as np
np.histogram(df_loads.kW)

(array([867,   0,   0,   0,   2,   9,   5,   6,   8,   9]),
 array([7.81118088e-01, 2.45423006e+02, 4.90064894e+02, 7.34706783e+02,
        9.79348671e+02, 1.22399056e+03, 1.46863245e+03, 1.71327434e+03,
        1.95791622e+03, 2.20255811e+03, 2.44720000e+03]))

Implicit collections as batches#

Implicit collections are the lists of objects of a certain types, as tracked by the OpenDSS engine.

AltDSS exposes collections for all DSS objects implemented in the AltDSS/DSS C-API engine. That is, in AltDSS, not only you can interact directly in Python with all DSS objects, you can use them in batches.

That is, the Load API itself is exposed as a dynamic batch. If new loads are added, this batch reflects that. Most user-created batches are static and are not updated automatically with new elements.

All the collections behave as batches and some can contain a few extra functions.

len(altdss.Load)

906

altdss.Load.Name[:10]

['25609_a',
 '25609_b',
 '25609_c',
 '25615_a',
 '25615_b',
 '25615_c',
 '25625_a',
 '25625_b',
 '25625_c',
 '25627_a']

altdss.Load.Bus1[:10]

['ckt7.1',
 'ckt7.2',
 'ckt7.3',
 'ckt7.1',
 'ckt7.2',
 'ckt7.3',
 'ckt7.1',
 'ckt7.2',
 'ckt7.3',
 'ckt7.1']

altdss.Vsource.Name

['source']

altdss.Spectrum.Name

['default',
 'defaultload',
 'defaultgen',
 'defaultvsource',
 'linear',
 'pwm6',
 'dc6']

altdss.Generator.Name

[]

Using the API methods#

Most DSS properties can be accessed, but also most of the classic API functions were generalized to operate on batches too!

Let’s compare how to get all powers from all loads in AltDSS and DSS-Python (which would be equivalent to the classic OpenDSS COM interface):

dss = altdss.to_dss_python() # Get a DSS-Python instance

If you wish to compare with EPRI’s COM implementation of OpenDSS instead (requires Windows), uncomment one of the blocks below (for win32com or comtypes):

# import win32com.client # Unfortunately win32com is still the most recommended option, but it's the slowest
# dss = win32com.client.gencache.EnsureDispatch('opendssengine.dss')
# dss.Text.Command = f'redirect "{CKT7_PATH}"'

# import comtypes.client # Prefer comtypes if you're using the official COM object
# dss = comtypes.client.CreateObject('opendssengine.dss')
# dss.Text.Command = f'redirect "{CKT7_PATH}"'

For classic COM-like access, without using the API extensions, the equivalent would be:

%%timeit
powers_classic = np.zeros(dss.ActiveCircuit.Loads.Count, dtype=complex)
idx = dss.ActiveCircuit.Loads.First
while idx != 0:
    powers_classic[idx - 1] = complex(*dss.ActiveCircuit.ActiveCktElement.TotalPowers)
    idx = dss.ActiveCircuit.Loads.Next

1.42 ms ± 20.9 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

In AltDSS:

%%timeit
powers_alt = altdss.Load.TotalPowers()

25.3 µs ± 94 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

The times are very different, but do the results match? Sure, the same numerical results:

powers_alt = altdss.Load.TotalPowers()
powers_classic = np.zeros(dss.ActiveCircuit.Loads.Count, dtype=complex)
idx = dss.ActiveCircuit.Loads.First
while idx != 0:
    powers_classic[idx - 1] = complex(*dss.ActiveCircuit.ActiveCktElement.TotalPowers)
    idx = dss.ActiveCircuit.Loads.Next

np.testing.assert_array_equal(powers_classic, powers_alt)

The main difference is the effort spent by the Python interpreter to create an array or list for each element in the loop, and calling the functions multiple times. It is common to get 10-100x speed-up in this kind of operation. There are minor optimizations that could be done with the classic API, but it’s not possible to achieve nearly the same performance.

Although not widely advertised, DSS-Python (and OpenDSSDirect.py) are typically faster than the official COM object for many of the operations illustrated above. That’s in part due to COM itself, part due to how win32com and comtypes are implemented, and part due to optimization done in DSS C-API.
That said, don’t be surprised it the time ratio between the COM impl. vs. the batch API from AltDSS reaches 1000x.

Using the DSS properties#

Using the DSS properties as seem in .DSS scripts can also be applied to batches.

%%timeit
power_spec_classic = np.zeros(dss.ActiveCircuit.Loads.Count, dtype=complex)
idx = dss.ActiveCircuit.Loads.First
while idx != 0:
    power_spec_classic[idx - 1] = complex(dss.ActiveCircuit.Loads.kW, dss.ActiveCircuit.Loads.kvar)
    idx = dss.ActiveCircuit.Loads.Next

432 µs ± 7.13 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

%%timeit
power_spec_alt = altdss.Load.kW + altdss.Load.kvar * 1j

12.7 µs ± 412 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)

Again, let’s check:

power_spec_classic = np.zeros(dss.ActiveCircuit.Loads.Count, dtype=complex)
idx = dss.ActiveCircuit.Loads.First
while idx != 0:
    power_spec_classic[idx - 1] = complex(dss.ActiveCircuit.Loads.kW, dss.ActiveCircuit.Loads.kvar)
    idx = dss.ActiveCircuit.Loads.Next

power_spec_alt = altdss.Load.kW + altdss.Load.kvar * 1j

np.testing.assert_array_equal(power_spec_classic, power_spec_alt)

In this case, there are no further allocations for arrays or buffers, we are just getting the internal float64 values for the “kW” and “kvar” DSS properties. Still, we can get around 30x speed-up on a simple test by using batches in AltDSS.

Technically, most DSS classes do not expose all properties in the API classes (COM or DDLL). For those, one would need to use either the Properties API, or the Text API. Both may result in some loss of precision in the values.

%%timeit
power_spec_classic_prop = np.zeros(dss.ActiveCircuit.Loads.Count, dtype=complex)
idx = dss.ActiveCircuit.Loads.First
while idx != 0:
    power_spec_classic_prop[idx - 1] = complex(
        float(dss.ActiveCircuit.ActiveDSSElement.Properties('kW').Val), 
        float(dss.ActiveCircuit.ActiveDSSElement.Properties('kvar').Val)
    )
    idx = dss.ActiveCircuit.Loads.Next

2.32 ms ± 9.82 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

%%timeit
power_spec_classic_prop = np.zeros(dss.ActiveCircuit.Loads.Count, dtype=complex)
dss.ActiveCircuit.SetActiveClass('Load')
idx = dss.ActiveCircuit.Loads.First
while idx != 0:
    dss.Text.Command = '? kW'
    kW =  dss.Text.Result
    dss.Text.Command = '? kvar'
    kvar = dss.Text.Result

    power_spec_classic_prop[idx - 1] = complex(
        float(kW), 
        float(kvar)
    )
    idx = dss.ActiveCircuit.Loads.Next

3.36 ms ± 8.52 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

We can see that these alternatives can take a lot longer, in the range 100-300x vs. the batch approach in AltDSS.

For comparison, here’s the time to run a simple snapshot solution:

%timeit altdss(f'solve mode=snap')

2.02 ms ± 3.61 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

As we can see, using the classic Text or Properties APIs to get the property values can be very expensive, reaching beyond the time required to actually solve the circuit.

Filtered batches#

Although performance benefits are nice, it can become cumbersome to get arrays and then have to filter them in a separate, sometimes complicated loop.

To avoid this, batches support filtering:

  • If you have a specific list of elements, you can create a batch for those (idx keyword argument)

  • You can use integer and float DSS properties to select objects

  • You can later filter an existing batch to create another

Batches from indices#

AltDSS uses 0-based indices for this.

some_loads = altdss.Load.batch(idx=[0, 10, 20, 30])
print(some_loads.Name)
print(abs(some_loads.TotalPowers()))

['25609_a', '25627_b', '25617_c', '25603_a']
[2474.93608703 2366.93848343 1517.96030515 1447.96260768]

This kind of batch is especially useful if you want to filter the loads using an external function. For example, let’s get the loads that result in more than 300 A each:

loads_300A = altdss.Load.batch(idx=np.where(altdss.Load.MaxCurrent(-1) > 300))
loads_300A.Name

['25609_a',
 '25609_b',
 '25609_c',
 '25615_a',
 '25615_b',
 '25615_c',
 '25627_a',
 '25627_b',
 '25627_c',
 '25629_b',
 '25629_c',
 '25601_a',
 '25601_b',
 '25601_c',
 '25605_a',
 '25605_c',
 '1001933-d1',
 '1001933-d2',
 '1001933-d3',
 '1001665-d1',
 '1001665-d2',
 '1001665-d3',
 '1001577-d1',
 '1001577-d2',
 '1001577-d3']

The batch can be converted to a list of Python objects using the to_list method or using the call operator. This way, each of this loads could be inspected individually, if the batch operations don’t fit well in the analysis.

loads_300A.to_list()

[<Load.25609_a>,
 <Load.25609_b>,
 <Load.25609_c>,
 <Load.25615_a>,
 <Load.25615_b>,
 <Load.25615_c>,
 <Load.25627_a>,
 <Load.25627_b>,
 <Load.25627_c>,
 <Load.25629_b>,
 <Load.25629_c>,
 <Load.25601_a>,
 <Load.25601_b>,
 <Load.25601_c>,
 <Load.25605_a>,
 <Load.25605_c>,
 <Load.1001933-d1>,
 <Load.1001933-d2>,
 <Load.1001933-d3>,
 <Load.1001665-d1>,
 <Load.1001665-d2>,
 <Load.1001665-d3>,
 <Load.1001577-d1>,
 <Load.1001577-d2>,
 <Load.1001577-d3>]

loads_300A()

[<Load.25609_a>,
 <Load.25609_b>,
 <Load.25609_c>,
 <Load.25615_a>,
 <Load.25615_b>,
 <Load.25615_c>,
 <Load.25627_a>,
 <Load.25627_b>,
 <Load.25627_c>,
 <Load.25629_b>,
 <Load.25629_c>,
 <Load.25601_a>,
 <Load.25601_b>,
 <Load.25601_c>,
 <Load.25605_a>,
 <Load.25605_c>,
 <Load.1001933-d1>,
 <Load.1001933-d2>,
 <Load.1001933-d3>,
 <Load.1001665-d1>,
 <Load.1001665-d2>,
 <Load.1001665-d3>,
 <Load.1001577-d1>,
 <Load.1001577-d2>,
 <Load.1001577-d3>]

Batches by filtering integer (DSS) properties#

Any integer property can be used. Let’s get all three-phase loads:

loads_3ph = altdss.Load.batch(Phases=3)
print(len(loads_3ph))

333

As expected from the dataframe in the “Loading the circuit” section, we get 333 three-phase loads.

Batches by filtering float (DSS) properties#

The recommended way to filter floating-point properties, especially nonzero values, is to use a range of values represented as a tuple. If you just have a lower (or higher) limit, use a very large (or small) value as the other limit:

loads_gt2MW = altdss.Load.batch(kW=(2000, 1e10))

len(loads_gt2MW.kW)

15

Filtering an existing batch#

Just apply the batch function again:

loads_3ph_100to200kW = loads_3ph.batch(kW=(100, 200))
loads_3ph_100to200kW.kW()

array([110.86689132, 110.86689132, 110.86689132, 110.86689132,
       110.86689132, 110.86689132])

Combining filters#

Multiple filters can be used at once. It can be useful to separate elements into multiple categories. Using properties:

loads_3ph_100to200kW = altdss.Load.batch(Phases=3, kW=(100, 200))

Using indices (from the main collection, always applied first) and properties combined:

batch_combined_filter = altdss.Load.batch(idx=np.where(altdss.Load.MaxCurrent(-1) > 300), Phases=3, kW=(100, 200))
batch_combined_filter.Name

['1001933-d1',
 '1001933-d2',
 '1001933-d3',
 '1001665-d1',
 '1001665-d2',
 '1001665-d3']

For fun, let’s compare how this last expression would be implemented in the classic API (assuming the same order of operations):

idx = dss.ActiveCircuit.Loads.First
CE = dss.ActiveCircuit.ActiveCktElement
Loads = dss.ActiveCircuit.Loads
names = []
while idx != 0:
    if max(CE.CurrentsMagAng[::2]) > 300 and Loads.Phases == 3 and 100 <= Loads.kW <= 200:
        names.append(Loads.Name)
        
    idx = dss.ActiveCircuit.Loads.Next

assert names == batch_combined_filter.Name

And the times:

%timeit altdss.Load.batch(idx=np.where(altdss.Load.MaxCurrent(-1) > 300), Phases=3, kW=(100, 200)).Name

29.2 µs ± 937 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

%%timeit
idx = dss.ActiveCircuit.Loads.First
CE = dss.ActiveCircuit.ActiveCktElement
Loads = dss.ActiveCircuit.Loads
names = []
while idx != 0:
    if max(CE.CurrentsMagAng[::2]) > 300 and CE.NumPhases == 3 and 100 <= Loads.kW <= 200:
        names.append(Loads.Name)
        
    idx = dss.ActiveCircuit.Loads.Next

1.49 ms ± 11.6 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

Other features and details#

  • All batch properties support broadcast from a single value, or a list/array of values. Objects can be set by their names, or a DSS object in Python.

batch_combined_filter.Daily = 'default'
batch_combined_filter.Daily

[<LoadShape.default>,
 <LoadShape.default>,
 <LoadShape.default>,
 <LoadShape.default>,
 <LoadShape.default>,
 <LoadShape.default>]

  • Array proxies are used when possible. This also pushes some operations to the engine:

batch_combined_filter.kW

<altdss.ArrayProxy.BatchFloat64ArrayProxy at 0x7f3c67d1e720>

batch_combined_filter.kW() # convert to array (load the data from the engine and create a NumPy array

array([110.86689132, 110.86689132, 110.86689132, 110.86689132,
       110.86689132, 110.86689132])

batch_combined_filter.kW + 1 # this is fine, converts to array and uses NumPy to do the sum

array([111.86689132, 111.86689132, 111.86689132, 111.86689132,
       111.86689132, 111.86689132])

batch_combined_filter.kW *= 1.2 # done in the engine!

batch_combined_filter.kW()

array([133.04026958, 133.04026958, 133.04026958, 133.04026958,
       133.04026958, 133.04026958])

  • There are some advanced features we need to document. Sometimes it’s not very apparent why some features exist. For example, the setter for each property allows for custom “setter flags”, which preserves the current Yprim for some circuit components. E.g. this always forces new Yprim and is equivalent to (but faster than) a DSS command edit load.LOAD_NAME kW=150 for each load in the batch:

batch_combined_filter.kW = 150

To avoid the full recalc and reproduce how the classic API kW property (DSS.ActiveCircuit.Loads.kW) works, we need to use a custom flag:

from dss import SetterFlags
batch_combined_filter._set_kW(150, SetterFlags.AvoidFullRecalc)

We are studying an alternative to still use batch_combined_filter.kW = 150 but pass the flags somehow.

Invalidation#

Like the objects and buses, batches are tracked and invalidated when the circuit is cleared, avoiding crashes and weird behavior. For example, if we clear and try to get the element names, we get an error mentioning “InvalidatedDSSObject”:

altdss.Clear()

batch_combined_filter.Name

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[46], line 1
----> 1 batch_combined_filter.Name

File /opt/python/lib/python3.12/site-packages/altdss/Batch.py:20, in BatchCommon.Name(self)
     16 @property
     17 def Name(self) -> List[str]:
     18     res = [
     19         self._ffi.string(self._lib.Obj_GetName(ptr)).decode(self._api_util.codec)
---> 20         for ptr in self._unpack()
     21     ]
     22     self._check_for_error()
     23     return res

File /opt/python/lib/python3.12/site-packages/altdss/Batch.py:58, in BatchCommon._unpack(self)
     55 if not cnt:
     56     return []
---> 58 return self._ffi.unpack(*self._get_ptr_cnt())

TypeError: unpack() argument 1 must be _cffi_backend._CDataBase, not InvalidatedDSSObject

What’s in the future?#

  • Investigate adding filters for the current state instead of just DSS properties.

  • Plotting integration of batches to allow easier interactive analysis.

  • Native Apache Arrow integration for cross-language dataframes.

  • General adjustments according to user feedback.

Have an idea or suggestion? Please feel free to reach out on GitHub Issues or Discussions.

Contents