org.das2.fsm

Class FileStorageModel

java.lang.Object

org.das2.fsm.FileStorageModel

public class FileStorageModel
extends java.lang.Object

Represents a method for storing data sets in a set of files by time. The client provides a regex for the files and how each group of the regex is interpreted as a time digit. The model can then be used to provide the set of files that cover a time range, etc. This new implementation uses a TimeParser object to more quickly process file names.

Method Summary

Modifier and Type	Method and Description
void	cacheCleanup() remove files that have been identified as old versions.
boolean	containsFile(java.io.File file) return true if the file came (or could come) from this FileStorageModel.
boolean	containsName(java.lang.String name) return true if the name came (or could come) from this FileStorageModel.
static FileStorageModel	create(FileSystem root, java.lang.String template) creates a FileStorageModel for the given template, which uses:
static FileStorageModel	create(FileSystem root, java.lang.String template, java.lang.String fieldName, TimeParser.FieldHandler fieldHandler) creates a FileStorageModel for the given template, but with a custom FieldHandler and field.
java.lang.String[]	generateNamesFor(DatumRange range) generate the names of the files that would cover this range.
java.io.File[]	getBestFilesFor(DatumRange targetRange) return the best files found for the range, without progress feedback.
java.io.File[]	getBestFilesFor(DatumRange targetRange, ProgressMonitor monitor) Get the files for the range, using versioning info ($v,etc).
java.lang.String[]	getBestNamesFor(DatumRange targetRange) return the names in the range, minding version numbers, or all available names if the range is null.
java.lang.String[]	getBestNamesFor(DatumRange targetRange, ProgressMonitor monitor) return the names in the range, minding version numbers, or all available names if the range is null.
static CacheTag	getCacheTagFor(FileStorageModel fsm, DatumRange range, java.io.File[] files)
static CacheTag	getCacheTagFor(FileStorageModel fsm, DatumRange range, java.lang.String[] names)
static FileSystem	getChildFileSystem(FileSystem root, java.lang.String child, ProgressMonitor monitor) return a child filesystem, with special code for LocalFileSystems to support Windows.
java.lang.String	getField(java.lang.String field, java.lang.String name) return the field value for the given name.
java.io.File	getFileFor(java.lang.String name) download the file for the given name within the filesystem.
java.io.File	getFileFor(java.lang.String name, ProgressMonitor monitor) download the file for the given name within the filesystem.
java.lang.String	getFilenameFor(Datum start, Datum end) return a filename that would intersect the range.
java.io.File[]	getFilesFor(DatumRange targetRange)
java.io.File[]	getFilesFor(DatumRange targetRange, ProgressMonitor monitor) Download the files within the range.
java.io.File[]	getFilesFor(java.lang.String[] names, ProgressMonitor monitor) download the files for each of the given names within the filesystem.
FileSystem	getFileSystem() return the filesystem used to implement this.
java.lang.String	getNameFor(java.io.File file) Provides a way to recover the model name of a file.
java.lang.String[]	getNamesFor(DatumRange targetRange) return the names in the range, or all names if the range is null.
java.lang.String[]	getNamesFor(DatumRange targetRange, ProgressMonitor monitor) return the names in the range, or all available names if the range is null.
FileStorageModel	getParent() returns the parent or null if none exists.
DatumRange	getRangeFor(java.lang.String name) return the time range represented by this name.
static java.lang.String	getRepresentativeFile(FileStorageModel ths, ProgressMonitor monitor, java.lang.String childRegex, DatumRange range, int depth) Return a random file from the FSM, which can be used to represent a typical file.
java.lang.String	getRepresentativeFile(ProgressMonitor monitor) return a random file from the FSM, which can be used to represent a typical file.
java.lang.String	getRepresentativeFile(ProgressMonitor monitor, java.lang.String childRegex) return a random file from the FSM, which can be used to represent a typical file.
java.lang.String	getRepresentativeFile(ProgressMonitor monitor, java.lang.String childRegex, DatumRange range) Return a random file from the FSM, which can be used to represent a typical file.
java.lang.String	getRoot() return the root of the filesystem as a string.
boolean	hasField(java.lang.String field) returns true if the parser has the field.
protected static java.lang.String	makeCanonical(java.lang.String template) Autoplot introduced the dollar sign instead of the percent, because $ is more URI-friendly.
DatumRange	quantize(DatumRange timeRange) return the timerange that contains the given timerange and exactly contains a set of granules.
void	setContext(DatumRange trdr) set the datum range giving context to the files.
static int	splitIndex(java.lang.String surl) This returns the index splitting the static part of the filename from the templated part.
java.lang.String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Method Detail

getFileSystem

public FileSystem getFileSystem()

return the filesystem used to implement this.

Returns:

filesystem

getChildFileSystem

public static FileSystem getChildFileSystem(FileSystem root,
                                            java.lang.String child,
                                            ProgressMonitor monitor)
                                     throws FileSystem.FileSystemOfflineException,
                                            java.net.UnknownHostException,
                                            java.io.FileNotFoundException

return a child filesystem, with special code for LocalFileSystems to support Windows. TODO: look into .zip child.

Parameters:

root -

child -

monitor -

Returns:

the FileSystem

Throws:

FileSystem.FileSystemOfflineException

java.net.UnknownHostException

java.io.FileNotFoundException

See Also:

https://sourceforge.net/p/autoplot/bugs/2132

getRepresentativeFile

public java.lang.String getRepresentativeFile(ProgressMonitor monitor)
                                       throws java.io.IOException

return a random file from the FSM, which can be used to represent a typical file. For example, we need to look at metadata to see what is available.

Parameters:

monitor - progress monitor in case a file must be downloaded.

Returns:

a reference to the file within the FileSystem, or null if one is not found.

Throws:

java.io.IOException

getRepresentativeFile

public java.lang.String getRepresentativeFile(ProgressMonitor monitor,
                                              java.lang.String childRegex)
                                       throws java.io.IOException

return a random file from the FSM, which can be used to represent a typical file. For example, we need to look at metadata to see what is available.

Parameters:

monitor - progress monitor in case a file must be downloaded.

childRegex - the parent must contain a file/folder matching childRegex

Returns:

a reference to the file within the FileSystem, or null if one is not found.

Throws:

java.io.IOException

getRepresentativeFile

public java.lang.String getRepresentativeFile(ProgressMonitor monitor,
                                              java.lang.String childRegex,
                                              DatumRange range)
                                       throws java.io.IOException

Return a random file from the FSM, which can be used to represent a typical file. For example, we need to look at metadata to see what is available. This is introduced to support discovery, where we just need one file to get started. Before, there was code that would list all files, then use just the first one. This may return a skeleton file, but getFileFor() must return a result. This implementation does the same as getNames(), but stops after finding a file.

Parameters:

monitor - progress monitor in case a file must be downloaded.

childRegex - the parent must contain a file/folder matching childRegex, or null if there is no child.

range - hint at the range where we are looking, or null if there is not time range constraint.

Returns:

null if no file is found

Throws:

java.io.IOException - if the file cannot be downloaded.

getRepresentativeFile

public static java.lang.String getRepresentativeFile(FileStorageModel ths,
                                                     ProgressMonitor monitor,
                                                     java.lang.String childRegex,
                                                     DatumRange range,
                                                     int depth)
                                              throws java.io.IOException

Return a random file from the FSM, which can be used to represent a typical file. For example, we need to look at metadata of a given file to see what is available inside. This is introduced to support discovery, where we just need one file to get started. Before, there was code that would list all files, then use just the first one. This may return a skeleton file, but getFileFor() must return a result. This implementation does the same as getNames(), but stops after finding a file.

Parameters:

ths -

monitor - progress monitor in case a file must be downloaded.

childRegex - the parent must contain a file/folder matching childRegex, or null if there is no child.

range - hint at the range where we are looking.

depth - the recursion depth, useful for debugging.

Returns:

null if no file is found

Throws:

java.io.IOException - if the file cannot be downloaded.

setContext

public void setContext(DatumRange trdr)

set the datum range giving context to the files. For example, filenames are just $H$M$S.dat, and the context is "Jan 17th, 2015"

Parameters:

trdr - the context

getFilenameFor

public java.lang.String getFilenameFor(Datum start,
                                       Datum end)

return a filename that would intersect the range. Note this file may not actually exist. This may be used to quantize the range. The template may not contain versions.

Parameters:

start -

end -

Returns:

generateNamesFor

public java.lang.String[] generateNamesFor(DatumRange range)

generate the names of the files that would cover this range. This was taken from Autoplot's org.virbo.jythonsupport.Util. TODO: versioning, etc.

Parameters:

range - the time range to cover.

Returns:

the string names, each in the context of the filesystem.

quantize

public DatumRange quantize(DatumRange timeRange)

return the timerange that contains the given timerange and exactly contains a set of granules. This needs to be synchronized because the timeParser.

Parameters:

timeRange - arbitrary time range

Returns:

list of file timeranges covering file input timeRange.

getNamesFor

public java.lang.String[] getNamesFor(DatumRange targetRange)
                               throws java.io.IOException

return the names in the range, or all names if the range is null.

Parameters:

targetRange - range limit, or null.

Returns:

array of names within the system.

Throws:

java.io.IOException

getNamesFor

public java.lang.String[] getNamesFor(DatumRange targetRange,
                                      ProgressMonitor monitor)
                               throws java.io.IOException

return the names in the range, or all available names if the range is null. This will list directories.

Parameters:

targetRange - range limit, or null.

monitor -

Returns:

array of names within the system.

Throws:

java.io.IOException

getBestNamesFor

public java.lang.String[] getBestNamesFor(DatumRange targetRange,
                                          ProgressMonitor monitor)
                                   throws java.io.IOException

return the names in the range, minding version numbers, or all available names if the range is null. This will list directories.

Parameters:

targetRange - range limit, or null.

monitor -

Returns:

array of names within the system.

Throws:

java.io.IOException

getBestNamesFor

public java.lang.String[] getBestNamesFor(DatumRange targetRange)
                                   throws java.io.IOException

return the names in the range, minding version numbers, or all available names if the range is null. This will list directories.

Parameters:

targetRange - range limit, or null.

Returns:

array of names within the system.

Throws:

java.io.IOException

getCacheTagFor

public static CacheTag getCacheTagFor(FileStorageModel fsm,
                                      DatumRange range,
                                      java.lang.String[] names)

getCacheTagFor

public static CacheTag getCacheTagFor(FileStorageModel fsm,
                                      DatumRange range,
                                      java.io.File[] files)

getFilesFor

public java.io.File[] getFilesFor(DatumRange targetRange)
                           throws java.io.IOException

Throws:

java.io.IOException

getBestFilesFor

public java.io.File[] getBestFilesFor(DatumRange targetRange)
                               throws java.io.IOException

return the best files found for the range, without progress feedback.

Parameters:

targetRange -

Returns:

Throws:

java.io.IOException

cacheCleanup

public void cacheCleanup()

remove files that have been identified as old versions.

getRangeFor

public DatumRange getRangeFor(java.lang.String name)

return the time range represented by this name.

Parameters:

name - like 2013-10-31

Returns:

the timerange representing the day 2013-10-31

Throws:

java.lang.IllegalArgumentException - if the name is not part of the FileStorageModel.

hasField

public boolean hasField(java.lang.String field)

returns true if the parser has the field.

Parameters:

field - e.g. "x"

Returns:

true if the parser has the field.

getField

public java.lang.String getField(java.lang.String field,
                                 java.lang.String name)

return the field value for the given name. For example, if the spec is $Y/$m/$d/$Y$m$d_v$(v,sep).dat and the name matched is 2014/04/04/20140404_v2.3.dat then calling this for the field "v" would result in "2.3" This should not be used to retrieve fields that are components of the time range, such as $Y or $m.

Parameters:

field - field, for example "v"

name - name, for example 2014/04/04/20140404_v2.3.dat

Returns:

the field value, for example, "2.3" when the spec is $Y/$m/$d/$Y$m$d_v$v.dat

containsFile

public boolean containsFile(java.io.File file)

return true if the file came (or could come) from this FileStorageModel.

Parameters:

file - the file

Returns:

true if the file came (or could come) from this FileStorageModel.

containsName

public boolean containsName(java.lang.String name)

return true if the name came (or could come) from this FileStorageModel.

Parameters:

name - the name within the filesystem.

Returns:

true if the name came (or could come) from this FileStorageModel.

getNameFor

public java.lang.String getNameFor(java.io.File file)

Provides a way to recover the model name of a file. The returned File from getFilesFor can be anywhere, so it would be good to provide a way to get it back into a FSM name. For example, a filesystem might download the remote file to a cache directory, which is the File that is provided to the client, sometimes the client will need to recover the name of the corresponding FileObject, so this maps the File back to the name.

Parameters:

file - the file

Returns:

the canonical name of the file.

getRoot

public java.lang.String getRoot()

return the root of the filesystem as a string.

Returns:

the root of the filesystem as a string.

getFileFor

public java.io.File getFileFor(java.lang.String name)
                        throws java.io.IOException

download the file for the given name within the filesystem.

Parameters:

name - the name within the filesystem.

Returns:

null or a local file which can be opened.

Throws:

java.io.IOException

getFileFor

public java.io.File getFileFor(java.lang.String name,
                               ProgressMonitor monitor)
                        throws java.io.IOException

download the file for the given name within the filesystem.

Parameters:

name - the name within the filesystem.

monitor - monitor for the download.

Returns:

null or a local file which can be opened.

Throws:

java.io.IOException

getFilesFor

public java.io.File[] getFilesFor(java.lang.String[] names,
                                  ProgressMonitor monitor)
                           throws java.io.IOException

download the files for each of the given names within the filesystem.

Parameters:

names - array of names within the filesystem

monitor - monitor for the downloads (or null).

Returns:

local files that can be opened.

Throws:

java.io.IOException - during the transfer

getFilesFor

public java.io.File[] getFilesFor(DatumRange targetRange,
                                  ProgressMonitor monitor)
                           throws java.io.IOException

Download the files within the range. This might catch a bad link where getNamesFor does not.

Parameters:

targetRange - range limit, or null if no constraint used here.

monitor - the monitor

Returns:

a list of files that can be opened.

Throws:

java.io.IOException

getBestFilesFor

public java.io.File[] getBestFilesFor(DatumRange targetRange,
                                      ProgressMonitor monitor)
                               throws java.io.IOException

Get the files for the range, using versioning info ($v,etc).

Parameters:

targetRange - range limit, or null if no constraint used here.

monitor - the monitor

Returns:

a list of files that can be opened.

Throws:

java.io.IOException

getParent

public FileStorageModel getParent()

returns the parent or null if none exists. None will exist when there are no more wildcards. (/home/foo/$Y/$m-$d.dat has parent /home/foo/$Y which has parent null.)

Returns:

parent or null.

makeCanonical

protected static java.lang.String makeCanonical(java.lang.String template)

Autoplot introduced the dollar sign instead of the percent, because $ is more URI-friendly. Switch to this if it looks appropriate. NOTE: this does not correct commas in the qualifiers section. TODO: fix this!

Parameters:

template, - e.g "/%Y/%m%d.dat"

Returns:

"$Y/$m$d.dat"

splitIndex

public static int splitIndex(java.lang.String surl)

This returns the index splitting the static part of the filename from the templated part. For example, http://autoplot.org/data/C1_CP_EDI_EGD__$Y$m$d_V$v.cef is split into:

http://autoplot.org/data/
C1_CP_EDI_EGD__$Y$m$d_V$v.cef

and http://emfisis.physics.uiowa.edu/Flight/RBSP-A/Quick-Look/$Y/$m/$d/rbsp-a_magnetometer_4sec-gsm_emfisis-Quick-Look_$Y$m$d_v$(v,sep).cdf:

http://emfisis.physics.uiowa.edu/Flight/RBSP-A/Quick-Look/
$Y/$m/$d/rbsp-a_magnetometer_4sec-gsm_emfisis-Quick-Look_$Y$m$d_v$(v,sep).cdf

This new version uses regexs and is more complete than versions found in Autoplot, and they should eventually use this instead. Note the Autoplot one returns the index of the last /, whereas this returns that index plus one.

Taken from Autoplot's AggregatingDataSourceFactory, where Autoplot just has a URI and needs to get a file list. See also org/autoplot/pngwalk/WalkUtil.java splitIndex, which also allows wildcards like *.

Note, ages ago %Y was valid as well as $Y. This is no longer supported, since it is useful to have Python templates working alongside URI_Templates.

Parameters:

surl - a string like http://autoplot.org/data/C1_CP_EDI_EGD__$Y$m$d_V$v.cef

Returns:

an integer indicating the split index, so that surl.substring(0,i) includes the slash.

create

public static FileStorageModel create(FileSystem root,
                                      java.lang.String template)

creates a FileStorageModel for the given template, which uses:

%Y-%m-%dT%H:%M:%S.%{milli}Z";
    %Y  4-digit year
    %m  2-digit month
    %d  2-digit day of month
    %j  3-digit day of year
    %H  2-digit Hour
    %M  2-digit Minute
    %S  2-digit second
    %v  best version by number  Also %(v,sep) for 4.3.2  or %(v,alpha)
    %{milli}  3-digit milliseconds 

product_$(o,id=ftp://stevens.lanl.gov/pub/projects/rbsp/autoplot/orbits/rbspa_pp).png

Parameters:

root - FileSystem source of the files.

template - describes how filenames are constructed. This is converted to a regular expression and may contain regex elements without groups. The string may contain $ instead of percents as long as there are no percents in the string.

Returns:

a newly-created FileStorageModel.

create

public static FileStorageModel create(FileSystem root,
                                      java.lang.String template,
                                      java.lang.String fieldName,
                                      TimeParser.FieldHandler fieldHandler)

creates a FileStorageModel for the given template, but with a custom FieldHandler and field.

Parameters:

root - FileSystem source of the files.

template - describes how filenames are constructed. This is converted to a regular expression and may contain regex elements without groups. The string may contain $ instead of percents as long as there are no percents in the string.

fieldName - custom field name

fieldHandler - TimeParser.FieldHandler to call with the field contents.

Returns:

a newly-created FileStorageModel.

toString

public java.lang.String toString()

Overrides:

toString in class java.lang.Object