org.das2.qds.DataSetUtil

Utilities for QDataSet, such as conversions from various forms to QDataSet, and doing a units conversion. TODO: DataSetUtil, DataSetOps, and org.virbo.dsops.Ops have become blurred over the years. These should either be combined or new mission statements need to be created.

DataSetUtil( )

PROPERTY_TYPE_STRING

PROPERTY_TYPE_NUMBER

PROPERTY_TYPE_BOOLEAN

PROPERTY_TYPE_MAP

PROPERTY_TYPE_QDATASET

PROPERTY_TYPE_CACHETAG

PROPERTY_TYPE_UNITS

addContext

adds the rank 0 dataset (a Datum) to the dataset's properties, after all the other CONTEXT_<i> properties.

Parameters

Returns:

addQube

add QUBE property to dataset, maybe verifying that it is a qube. This is intended to reduce code that builds datasets, not to verify that a dataset is a qube.

Parameters

Returns:

as2DArrayOfDoubles

convert the QDataSet to an array. Units and fill are ignored...

Parameters

Returns:

asArrayOfDoubles

convert the QDataSet to an array. Units and fill are ignored...

Parameters

Returns:

asDataSet

return the rank 1 dataset equivalent to the DatumVector

Parameters

Returns:

asDatum

Parameters

Returns:

asDatumRange

return the DatumRange equivalent of this 2-element, rank 1 bins dataset.

Parameters

Returns:

asDatumVector

return DatumVector, which is a 1-d array of Datums.

Parameters

Returns:

bestFormatter

return a DatumFormatter that can accurately format all of the datums in the dataset. The goal is to identify a formatter which is also efficient, and doesn't waste an excess of digits. This sort of code is often needed, and is also found in:

• QStream--where ASCII mode needs efficient representation

Parameters

Returns:

binarySearch

perform a binary search for key within ds, constraining the search to between low and high.

Parameters

Returns:

bundleNames

return the name for each column of the bundle. This simply calls SemanticOps.getComponentNames.

Parameters

Returns:

bundleWeightsDataSet

special weightsDataSet for when there is a bundle, and each component could have its own FILL_VALID and VALID_MAX. Each component gets its own weights dataset in a JoinDataSet.

Parameters

Returns:

canonizeFill

Iterate through the dataset, changing all points outside of validmin, validmax and with zero weight to fill=-1e31. VALID_MIN and VALID_MAX properties are cleared, and FILL_VALUE is set to -1e31. If the dataset is writable, then the dataset is modified.

Parameters

Returns:

checkListOfIndeces

verify that the indeces really are indeces, and a warning may be printed if the indeces don't appear to match the array by looking at the MAX_VALUE property of the indeces.

Parameters

Returns:

checkQube

check to see if a dataset really is a qube, even if there is a rank 2 dep1. Note this ignores BUNDLE_1 property if there is a DEPEND_1. This was motivated by the fftPower routine, which returned a rank 2 DEPEND_1, but is typically constant, and RBSP/ECT datasets that often have rank 2 DEPEND_1s that are constant. This will putProperty(QDataSet.QUBE,Boolean.TRUE) when the dataset really is a qube, or Boolean.FALSE if is clearly not a qube.

Parameters

Returns:

closest

return the index of the closest value in ds to d, using guess as a starting point. This implementation ignores guess, but wise clients will supply it as a parameter.

Parameters

Returns:

closestIndex

returns the index of the closest index in the data. This supports rank 1 datasets, and rank 2 bins datasets where the bin is min,max.

Parameters

Returns:

contextAsString

provide the context as a string, for example to label a plot. The dataset CONTEXT_i properties are inspected, each of which must be one of:

• rank 0 dataset
• rank 1 bins dataset
• rank 1 bundle

Parameters

Returns:

convertTo

convert the dataset to the given units.

Parameters

Returns:

copyDimensionProperties

Copy over all the dimension properties, including: UNITS, FORMAT, SCALE_TYPE, TYPICAL_MIN, TYPICAL_MAX, VALID_MIN, VALID_MAX, FILL_VALUE, NAME, LABEL, TITLE, USER_PROPERTIES These are dimension properties, as opposed to structural see dimensionProperties() for a list of dimension properties. TODO: This DOES NOT support join datasets yet.

Parameters

Returns:

correlativeProperties

properties that go along with the zeroth index. These are all QDataSets with dimensions compatible with the datasets. If you trim the dataset, then these must be trimmed as well.

Returns:

courserCadence

return the courser cadence of the two cadences. Das2's AverageTableRebinner needs to get the coursest of all the ytags.

Parameters

Returns:

dimensionProperties

return the list of properties that pertain to the dimension that dataset values exist. These are the properties that survive through most operations. For example, if you flattened the dataset, what properties would still exist? If you shuffled the data? These are not structural properties like DEPEND_0, BUNDLE_1, etc. Note that BUNDLE_1 will carry dimension properties as well.

Returns:

firstValidPoint

returns the first valid point found in a dataset, or null if no such point is found.

Parameters

Returns:

format

return a human-readable string representing the dataset

Parameters

Returns:

gcd

return the greatest common divisor, which is the unit for which all elements in the dataset are integer multiples of the result. This works on continuous data, however, so limit is used to determine the fuzz allowed. TODO: this needs review and is not for production use.

Parameters

Returns:

getCadenceWaveform

return the cadence between measurements of a waveform dataset. This is different than the cadence typically quoted, which is the cadence between waveform records.

Parameters

Returns:

getContext

collect the context for the dataset. This will be a rank 2 join of scalars or ranges, so CONTEXT_0 will be in the zeroth index of the result. Rank 0 CONTEXT values are joined to make them rank 1.

Parameters

Returns:

getDimensionProperties

return just the properties attached to the dataset, like UNITS and SCALE_TYPE, and not like DEPEND_x, etc.

Parameters

Returns:

See Also:

getNextIndex

returns the first column that is after the given datum. Note the if the datum identifies (==) an xtag, then the previous column is returned.

Parameters

Returns:

getNextIndexStrict

Returns the index of the value which is greater than the value less of the datum. Note for rank 2 bins, the first bin which has an beginning less than the datum.

Parameters

Returns:

getNextInterval

return the next interval (typically time) containing data, centered on data, with the roughly the same width.

Parameters

Returns:

getPreviousIndex

returns the first index that is before the given datum, or zero if no data is found before the datum. PARV! if the datum identifies (==) an xtag, then the previous column is returned.

Parameters

Returns:

getPreviousIndexStrict

Returns the index of the value which is less than the value less of the datum. Note for rank 2 bins, the first bin which has an end less than the datum.

Parameters

Returns:

getPreviousInterval

return the previous interval (typically time) containing data, centered on data, with the roughly the same width.

Parameters

Returns:

getProperties

return the properties listed, using the defaults if provided. See dimensionProperties(), globalProperties().

Parameters

Returns:

getProperty

Parameters

Returns:

getPropertyClass

return the class for the property, to support Jython.

Parameters

Returns:

See Also:

getPropertyType

return the type of the property, as a string to support use in Jython: String,Number,Boolean,Map,QDataSet,CacheTag,Units

Parameters

Returns:

See Also:

getStringValue

return the value, which gets units from index i. from rank 1 bundle dataset.

Parameters

Returns:

getUserProperty

return the "User" property, which allow for extensions of the data model that aren't used. This returns the property "name" under the name USER_PROPERTIES, which must either be null or a Map<String,Object>.

Parameters

Returns:

globalProperties

properties that describe the dataset itself, rather than those of a dimension or structural properties.

Returns:

guessCadence

return the cadence for the given x tags. The goal will be a rank 0 dataset that describes the intervals, but this will also return a rank 1 dataset when multiple cadences are found. The yds values for each xds value can also be set, specifying where the x values can be ignored because of fill. TODO: this needs review.

Parameters

Returns:

guessCadenceNew

returns a rank 0 dataset indicating the cadence of the dataset. Using a dataset as the result allows the result to indicate SCALE_TYPE and UNITS. History:

• 2011-02-21: keep track of repeat values, allowing zero to be considered either mono increasing or mono decreasing
• 2011-02-21: deal with interleaved fill values, keeping track of last valid value.

Parameters

Returns:

• SCALE_TYPE may be "log"
• UNITS will be a ratiometric unit when the SCALE_TYPE is log, and will be the offset unit for interval units like Units.t2000.

indexGenDataSet

creates a dataset of integers 0,1,2,...,n-1.

Parameters

Returns:

inferBins

This is the one code to infer bin boundaries when only the centers are available. This uses centers of adjacent data, and extrapolates to get the edge boundaries to create an acceptable limit. When the data is log-spaced, the centers are done in the ratiometric space. Small (<1000 element) datasets will be sorted if necessary.

Parameters

Returns:

inferBinsRank2

infer the bins for the rank 2 ytags. This was first used with Juno high-rate data, where the ytags follow the FCE implied by the magnetic field detector.

Parameters

Returns:

isConstant

return true if each record of DEPEND_0 is the same. Rank 0 datasets are trivially constant. TODO: ds.slice(i) can be slow because equivalent does so much with the metadata.

Parameters

Returns:

isDataAt

is data found at the time t, or might it be within a gap in the data?

Parameters

Returns:

isDimensionProperty

return true if the property name is a valid dimension property like "UNITS" or "FORMAT". See dimensionProperties().

Parameters

Returns:

isInheritedProperty

true if the property is one that is global and is relevant throughout the dataset, such as a title or the units. property( "TITLE",0,0 ) often returns property("TITLE"), but property( "DEPEND_0",0,0 ) should never return property("DEPEND_0"). This is false, for example, for DEPEND_1.

Parameters

Returns:

isLogSpacing

return true if the data appears to have log spacing. The data is assumed to be monotonically increasing or decreasing.

Parameters

Returns:

isMonotonic

returns true if the dataset is monotonically increasing. If the dataset has the MONOTONIC property set to Boolean.TRUE, believe it. The data can contain repeated values. An empty dataset is not monotonic. We now use a weights dataset to more thoroughly check for fill. The dataset may contain fill data, only the non-fill portions are considered.

Parameters

Returns:

See Also:

isMonotonicAndIncreasing

returns true if the dataset is monotonically increasing and does not contain repeat values. An empty dataset is not monotonic. The dataset may contain fill data, only the non-fill portions are considered.

Parameters

Returns:

See Also:

isMonotonicAndIncreasingQuick

quickly determine, with high accuracy, if data is monotonic. This should be a constant-time operation, and be extremely unlikely to fail.

Parameters

Returns:

See Also:

isMonotonicQuick

quickly determine, with high accuracy, if data is monotonic (repeated values allowed). This should be a constant-time operation, and be extremely unlikely to fail.

Parameters

Returns:

See Also:

isNotBundle

return true if the data is not a bundle and has uniform units throughout the dataset.

Parameters

Returns:

isQube

test to see that the dataset is a qube. TODO: this all needs review.

Parameters

Returns:

kmeansCadence

use K-means algorithm to find N means in a rank 1 dataset.

Parameters

Returns:

makeValid

throw out DEPEND and PLANE to make dataset valid.

Parameters

Returns:

maybeCopyRenderType

copy over the render type, if it is still appropriate. This nasty bit of code was introduced to support LANL data, where high-rank data is preferably plotted as a spectrogram, but can be plotted as a stack of lineplots.

Parameters

Returns:

nLargest

return the value of the nth biggest item. This keeps n values in memory.

Parameters

Returns:

oneUnit

Test for bundle scheme. Returns true if the data contains data with just one unit. For example ds[time=100,en=30] might have BUNDLE_1 to specify labels for each of the 30 channels, but each of its measurements have the same units. However, bundle(time,density) results in a dataset with BUNDLE_1 with the two datasets having different units. When two bundled dataset's units are convertible but not equal this will return false. This will return true for a zero-length dataset.

Parameters

Returns:

product

returns 1 for zero-length qube, the product otherwise.

Parameters

Returns:

propertyNames

Return the names of non-structural properties of the dataset, like the UNITS and CADENCE. These are the dimensionProperties, plus others specific to the dataset, such as CADENCE and DELTA_PLUS.

Returns:

putProperties

copy all properties into the dataset by iterating through the map. Properties that are equal to null are not copied, since null is equivalent to the property not found.

Parameters

Returns:

See Also:

qubeDims

provides a convenient way of indexing qubes, returning an int[] of length ds.rank() containing each dimension's length, or null if the dataset is not a qube.

Parameters

Returns:

qubeSubset

return a subset of the dataset which is a qube.

Parameters

Returns:

rangeOfMonotonic

returns the indeces of the min and max elements of the monotonic dataset. This uses DataSetUtil.isMonotonic() which would be slow if MONOTONIC is not set.

Parameters

Returns:

See Also:

repeatingSignal

return 0 or the number of values after which the signal repeats. For example, ds= dataset([1,2,3,4,1,2,3,4,1,2,3,4,1,2,3,4,1,2]) assert repeatingSignal(ds)==4 if the data contains fill, then it is also not repeating.

Parameters

Returns:

See Also:

replicateDataSet

creates a dataset with the given cadence, start and length.

Parameters

Returns:

samePopulation

true if the two datasets appear to be from the same population.

Parameters

Returns:

sliceProperties

return properties attached to the slice at index. Note the slice implementations use this, and this only returns properties from dimensionProperties(). http://autoplot.org//QDataSet#20150514 Note this does not look at BUNDLE_1 properties. TODO: consider this.

Parameters

Returns:

statsString

return a human readable statistical representation of the dataset. Currently this is the mean, stddev ad number of points.

Parameters

Returns:

sum

returns 0 for zero-length qube, the sum otherwise.

Parameters

Returns:

tagGenDataSet

creates a dataset with the given cadence, start and length. This is danger code, because the CADENCE must be reset if the UNITS are reset. use tagGenDataSet( int n, final double start, final double cadence, Units units ) if units are going to be specified.

Parameters

Returns:

toBundleDs

make a proper bundle ds from a simple bundle containing ordinal units This assumes that labels is a unique set of labels. See http://autoplot.org/QDataSet#DataSet_Properties under BUNDLE_1. See DataSetOps.unbundle

Parameters

Returns:

toSparkline

Make a unicode spark line http://www.ssec.wisc.edu/~tomw/java/unicode.html. This should be for human consumption, because future versions may include data reduction and doubling up characters. See commented code in MetadataPanel.histStr. (I knew I had done this before...)

Parameters

Returns:

toString

provide a string representation of the dataset. This is intended for human consumption, but does follow rules outlined in http://autoplot.org//developer.datasetToString

Parameters

Returns:

See Also:

totalLength

return the total number of values in the dataset. For qubes this is the product of the dimension lengths, for other datasets we create a dataset of lengths and total all the elements.

Parameters

Returns:

totalLengthAsLong

return the total number of values in the dataset, as a long. For qubes this is the product of the dimension lengths, for other datasets we create a dataset of lengths and total all the elements.

Parameters

Returns:

trimProperties

help out implementations of the QDataSet.trim() command. This does the dimension properties and geometric properties like DEPEND_0 and DELTA_PLUS. This also checks for indexed properties, which are NAME__i.

Parameters

Returns:

validPoints

return just the valid points of the dataset.

Parameters

Returns:

validate

returns true if the dataset is valid, false otherwise. If problems is non-null, then problems will be indicated here.

Parameters

Returns:

value

get the value of the rank 0 dataset in the specified units. For example, value( ds, Units.km )

Parameters

Returns:

weightsDataSet

Provide consistent valid logic to operators by providing a QDataSet with >0.0 where the data is valid, and 0.0 where the data is invalid. VALID_MIN, VALID_MAX and FILL_VALUE properties are used. Note, when FILL_VALUE is not specified, -1e31 is used. This is to support legacy logic. For convenience, the property SUGGEST_FILL is set to the fill value used.

Parameters

Returns:

xTagBinarySearch

returns the index of a tag, or the (-(insertion point) - 1). (See Arrays.binarySearch)

Parameters

Returns: