org.das2.util.filesystem

Class WebFileSystem

java.lang.Object

org.das2.util.filesystem.FileSystem

org.das2.util.filesystem.WebFileSystem

Direct Known Subclasses:

FTPBeanFileSystem, FTPFileSystem, HttpFileSystem, PDSPPIFileSystem, WGetFileSystem

public abstract class WebFileSystem
extends FileSystem

Base class for HTTP and FTP-based filesystems. A local cache is kept of the files.

Nested Class Summary

Nested classes/interfaces inherited from class org.das2.util.filesystem.FileSystem

FileSystem.DirectoryEntry, FileSystem.FileSystemOfflineException

Field Summary

Fields

Modifier and Type	Field and Description
static int	HTTP_CHECK_TIMESTAMP_LIMIT_MS timestamp checks will occur no more often than this.
protected static java.lang.String	id ID string for the process.
static int	LISTING_TIMEOUT_MS we keep a cached listing in on disk.
protected java.io.File	localRoot
protected static java.util.logging.Logger	logger
static int	MEMORY_LISTING_TIMEOUT_MS we keep a cached listing in memory for performance.
protected boolean	offline true means only local files are used from the cache.
protected java.lang.String	offlineMessage the response message explaining why the filesystem is offline.
protected int	offlineResponseCode
static java.lang.String	PROP_OFFLINE if true, then the remote filesystem is not accessible, but local cache copies may be accessed.
static java.lang.String	PROP_READ_ONLY_CACHE alternate location to check for file before downloading.
protected WebProtocol	protocol plug-in template for implementation.

Fields inherited from class org.das2.util.filesystem.FileSystem

loggerUrl, NULL, PROP_CASE_INSENSITIVE, properties

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	WebFileSystem(java.net.URI root, java.io.File localRoot) Creates a new instance of WebFileSystem

Method Summary

Modifier and Type	Method and Description
void	addPropertyChangeListener(java.beans.PropertyChangeListener listener)
protected java.util.Map<java.lang.String,FileSystem.DirectoryEntry>	addRoCacheEntries(java.lang.String directory, java.util.Map<java.lang.String,FileSystem.DirectoryEntry> remoteList) Allow the local RO Cache to contain files that are not yet in the remote filesystem, to support the case where a data provider tests locally available products before mirroring them out to the public website.
void	cacheListing(java.lang.String directory, FileSystem.DirectoryEntry[] listing)
static void	consumeStream(java.io.InputStream err) Deprecated. see HtmlUtil.consumeStream.
protected long	copyStream(java.io.InputStream is, java.io.OutputStream out, ProgressMonitor monitor) copies data from in to out, sending the number of bytesTransferred to the monitor.
protected abstract java.util.Map<java.lang.String,java.lang.String>	downloadFile(java.lang.String filename, java.io.File f, java.io.File partfile, ProgressMonitor monitor) Transfers the file from the remote store to a local copy f.
static java.io.File	getDownloadDirectory()
protected java.util.concurrent.locks.Lock	getDownloadLock(java.lang.String filename, java.io.File f, ProgressMonitor monitor) Request lock to download file.
FileObject	getFileObject(java.lang.String filename) Return the handle for this file.
protected long	getLastAccessed(java.lang.String filename) reduce the number of hits to a server by caching last access times for local files.
java.lang.String	getLocalName(java.io.File file) return the name of the File within the FileSystem, where File is a local file within the local copy of the filesystem.
java.lang.String	getLocalName(java.net.URL url)
java.io.File	getLocalRoot() return the folder that is a local copy of the filesystem.
java.lang.String	getLocalRootAbsPath() Deprecated. use getLocalRoot().getAbsolutePath()
java.lang.String	getOfflineMessage() return the reason (if any provided) why the filesystem is offline,
int	getOfflineResponseCode() if non-zero, the response code (e.g.
java.io.File	getPartFile(java.io.File localFile) return a name where the download is to be staged.
WebProtocol	getProtocol() return the protocol object, which allows access to the metadata.
java.io.File	getReadOnlyCache()
java.net.URL	getRootURL() return the root of the filesystem as a URL.
java.net.URI	getURI(java.lang.String filename) return the URI for the internal filename
java.net.URL	getURL(java.lang.String filename) return the URL for the internal filename, encoding the characters if necessary.
boolean	isAppletMode()
abstract boolean	isDirectory(java.lang.String filename) return true if the name refers to a directory, not a file.
static java.lang.String	isGitClone(java.io.File root, java.lang.String path) return the canonical path for the path, if it is part of a git clone.
boolean	isListingCached(java.lang.String directory) return true if the listing file (.listing) is available in the file system cache, and is still fresh.
boolean	isOffline()
abstract java.lang.String[]	listDirectory(java.lang.String directory) return the directory listing for the name.
java.lang.String[]	listDirectory(java.lang.String directory, java.lang.String regex) returns a list of the names of the files in a directory that match regex.
protected FileSystem.DirectoryEntry[]	listDirectoryFromMemory(java.lang.String directory) list the directory using the ram memory cache.
protected java.io.File	listingFile(java.lang.String directory) return the File for the cached listing, even if it does not exist.
static java.io.File	localRoot(java.net.URI root) return the local root for the URI.
FileSystem.DirectoryEntry	maybeUpdateDirectoryEntry(java.lang.String filename, boolean force) trigger an update of the in-memory listing, or check to see if it is in memory.
void	removePropertyChangeListener(java.beans.PropertyChangeListener listener)
void	resetListCache(java.lang.String directory) From FTPBeanFileSystem.
void	resetListingCache() reset the .listing files and the ram-memory caches.
void	setAppletMode(boolean applet)
void	setOffline(boolean offline)
void	setReadOnlyCache(java.io.File f)
java.lang.String	toString()

Methods inherited from class org.das2.util.filesystem.FileSystem

create, create, create, create, create, create, createFileSystem, getExceptionHandler, getListing, getListing, getProperty, getRootURI, listDirectory, listDirectory, listDirectoryDeep, peek, peekInstances, recreate, recreate, registerFileSystemFactory, reset, reset, setExceptionHandler, settings, splitUrl, toCanonicalFilename, toCanonicalFolderName, toCanonicalFolderName

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

logger

protected static final java.util.logging.Logger logger

LISTING_TIMEOUT_MS

public static final int LISTING_TIMEOUT_MS

we keep a cached listing in on disk. This is backed by the website.

See Also:

Constant Field Values

MEMORY_LISTING_TIMEOUT_MS

public static final int MEMORY_LISTING_TIMEOUT_MS

we keep a cached listing in memory for performance. This is backed by the .listing file.

See Also:

Constant Field Values

HTTP_CHECK_TIMESTAMP_LIMIT_MS

public static final int HTTP_CHECK_TIMESTAMP_LIMIT_MS

timestamp checks will occur no more often than this.

See Also:

Constant Field Values

localRoot

protected final java.io.File localRoot

protocol

protected WebProtocol protocol

plug-in template for implementation. if non-null, use this.

offline

protected boolean offline

true means only local files are used from the cache.

offlineMessage

protected java.lang.String offlineMessage

the response message explaining why the filesystem is offline.

PROP_OFFLINE

public static final java.lang.String PROP_OFFLINE

if true, then the remote filesystem is not accessible, but local cache copies may be accessed. See FileSystemSettings.allowOffline

See Also:

Constant Field Values

offlineResponseCode

protected int offlineResponseCode

PROP_READ_ONLY_CACHE

public static final java.lang.String PROP_READ_ONLY_CACHE

alternate location to check for file before downloading.

See Also:

Constant Field Values

id

protected static final java.lang.String id

ID string for the process.

Constructor Detail

WebFileSystem

protected WebFileSystem(java.net.URI root,
                        java.io.File localRoot)

Creates a new instance of WebFileSystem

Parameters:

root - the remote URI.

localRoot - local directory used to store local copies of the data.

Method Detail

getDownloadDirectory

public static java.io.File getDownloadDirectory()

isOffline

public boolean isOffline()

setOffline

public void setOffline(boolean offline)

Parameters:

offline -

getOfflineMessage

public java.lang.String getOfflineMessage()

return the reason (if any provided) why the filesystem is offline,

Returns:

the message for the response code

getOfflineResponseCode

public int getOfflineResponseCode()

if non-zero, the response code (e.g. 403) why the filesystem is offline.

Returns:

the response code.

setReadOnlyCache

public final void setReadOnlyCache(java.io.File f)

getReadOnlyCache

public final java.io.File getReadOnlyCache()

addPropertyChangeListener

public void addPropertyChangeListener(java.beans.PropertyChangeListener listener)

removePropertyChangeListener

public void removePropertyChangeListener(java.beans.PropertyChangeListener listener)

isGitClone

public static java.lang.String isGitClone(java.io.File root,
                                          java.lang.String path)

return the canonical path for the path, if it is part of a git clone.

Parameters:

root - a directory which is a clone of some path, possibly

path - the path within the path.

Returns:

the canonical path, or null if it is not a git clone.

addRoCacheEntries

protected java.util.Map<java.lang.String,FileSystem.DirectoryEntry> addRoCacheEntries(java.lang.String directory,
                                                                                      java.util.Map<java.lang.String,FileSystem.DirectoryEntry> remoteList)

Allow the local RO Cache to contain files that are not yet in the remote filesystem, to support the case where a data provider tests locally available products before mirroring them out to the public website.

Parameters:

directory -

remoteList -

Returns:

localRoot

public static java.io.File localRoot(java.net.URI root)

return the local root for the URI.

Parameters:

root - the URI such as http://das2.org/data/

Returns:

/home/jbf/autoplot_data/fscache/http/das2.org/data/

getPartFile

public final java.io.File getPartFile(java.io.File localFile)

return a name where the download is to be staged. This will be unique for any process. We make the following assumptions:

• multiple Java processes will be using and modifying the file database.
• java processes may be on different machines
• an ID conflict may occur should two processes have the same UID, hostname, and are started at the same clock time millisecond.

See https://sourceforge.net/p/autoplot/bugs/1301/

Parameters:

localFile -

Returns:

the temporary filename to use.

getDownloadLock

protected java.util.concurrent.locks.Lock getDownloadLock(java.lang.String filename,
                                                          java.io.File f,
                                                          ProgressMonitor monitor)
                                                   throws java.io.IOException

Request lock to download file. If this thread gets the lock, then it should download the file and call mutatorLock.unlock() when the download is complete. If another thread is downloading the file, this will block until the download is complete, and null will be returned to indicate that the file has already been downloaded. This must start the monitor when it gets the lock.

Parameters:

filename - the filename with in the filesystem.

f - the File which will be the local copy.

monitor - a monitor for the download. If a MutatorLock is returned, then the monitor is not touched, but other threads may use it to keep track of the download progress.

Returns:

Lock or null if another thread loaded the resource. The client should call lock.unlock() when the download is complete

Throws:

java.io.FileNotFoundException - if the file wasn't found after another thread loaded the file.

java.io.IOException

downloadFile

protected abstract java.util.Map<java.lang.String,java.lang.String> downloadFile(java.lang.String filename,
                                                                                 java.io.File f,
                                                                                 java.io.File partfile,
                                                                                 ProgressMonitor monitor)
                                                                          throws java.io.IOException

Transfers the file from the remote store to a local copy f. This should only be used within the class and subclasses, clients should use getFileObject( String ).getFile(). Subclasses implementing this should download data to partfile, then rename partfile to f after the download is complete. Note, this is non-trivial, since several threads and even several processes may be using the same area at once. See HttpFileSystem's implementation of this before attempting to implement the function.

Parameters:

filename - the name of the file, relative to the filesystem.

f - the file to where the file is downloaded.

partfile - the temporary file during download.

monitor - progress monitor

Returns:

metadata headers from the load, if any are available. See WebProtocol.

Throws:

java.io.IOException

See Also:

WebProtocol.getMetadata(org.das2.util.filesystem.WebFileObject)

getLocalRootAbsPath

public java.lang.String getLocalRootAbsPath()

Deprecated. use getLocalRoot().getAbsolutePath()

Get the root of the local file cache

Returns:

the root of the local file cache

getLocalRoot

public java.io.File getLocalRoot()

Description copied from class: FileSystem

return the folder that is a local copy of the filesystem. For LocalFilesystem, this is the same as the filesystem. For remote filesystems, this is a folder within their home directory. Note File.getAbsolutePath() returns the string representation of this root.

Specified by:

getLocalRoot in class FileSystem

Returns:

the folder that is a local copy of the filesystem.

getProtocol

public WebProtocol getProtocol()

return the protocol object, which allows access to the metadata. This may go away, so do not depend on this for production code.

Returns:

the protocol object.

resetListingCache

public void resetListingCache()

reset the .listing files and the ram-memory caches.

resetListCache

public void resetListCache(java.lang.String directory)

From FTPBeanFileSystem.

Parameters:

directory -

listingFile

protected java.io.File listingFile(java.lang.String directory)

return the File for the cached listing, even if it does not exist.

Parameters:

directory -

Returns:

isListingCached

public boolean isListingCached(java.lang.String directory)

return true if the listing file (.listing) is available in the file system cache, and is still fresh. LISTING_TIMEOUT_MS controls the freshness, where files older than LISTING_TIMEOUT_MS milliseconds will not be used. Note the timestamp on the file comes from the server providing the listing, so the age may be negative when clocks are not synchronized.

Parameters:

directory -

Returns:

true if the listing is cached.

cacheListing

public void cacheListing(java.lang.String directory,
                         FileSystem.DirectoryEntry[] listing)

listDirectoryFromMemory

protected FileSystem.DirectoryEntry[] listDirectoryFromMemory(java.lang.String directory)

list the directory using the ram memory cache. MEMORY_LISTING_TIMEOUT_MS=4s limits the lifespan of a cache entry, and this is really to avoid expensive listings of the same resource when searches are done.

Parameters:

directory - the directory name within the filesystem.

Returns:

null or the listing.

maybeUpdateDirectoryEntry

public FileSystem.DirectoryEntry maybeUpdateDirectoryEntry(java.lang.String filename,
                                                           boolean force)
                                                    throws java.io.IOException

trigger an update of the in-memory listing, or check to see if it is in memory.

Parameters:

filename - the particular file for which we need a listing.

force - if true, then list if it isn't available.

Returns:

null if the listing is not available, or if the element is not in the folder, the DirectoryEntry otherwise.

Throws:

java.io.IOException - when the directory is listed.

isDirectory

public abstract boolean isDirectory(java.lang.String filename)
                             throws java.io.IOException

return true if the name refers to a directory, not a file.

Specified by:

isDirectory in class FileSystem

Parameters:

filename -

Returns:

Throws:

java.io.IOException

listDirectory

public abstract java.lang.String[] listDirectory(java.lang.String directory)
                                          throws java.io.IOException

return the directory listing for the name.

Specified by:

listDirectory in class FileSystem

Parameters:

directory -

Returns:

Throws:

java.io.IOException

listDirectory

public java.lang.String[] listDirectory(java.lang.String directory,
                                        java.lang.String regex)
                                 throws java.io.IOException

Description copied from class: FileSystem

returns a list of the names of the files in a directory that match regex. Trailing slashes on directory names are not part of the name and need not be part of the regex. Note regex is a regular expression (.*\.dat), not a glob (*.dat)

Specified by:

listDirectory in class FileSystem

Parameters:

directory - the directory

regex - regular expression

Returns:

Throws:

java.io.IOException

getURL

public java.net.URL getURL(java.lang.String filename)

return the URL for the internal filename, encoding the characters if necessary.

Parameters:

filename - internal filename

Returns:

See Also:

DefaultHttpProtocol.urlEncodeSansSlash(java.lang.String)

getURI

public java.net.URI getURI(java.lang.String filename)

return the URI for the internal filename

Parameters:

filename - internal filename

Returns:

getRootURL

public java.net.URL getRootURL()

return the root of the filesystem as a URL. Since the root is stored as a URI and "ftp://jbf@mysite.com:@ftpproxy.net/temp/" is a legal address, check for this case.

Returns:

the root of the filesystem as a URL.

getLocalName

public java.lang.String getLocalName(java.io.File file)

return the name of the File within the FileSystem, where File is a local file within the local copy of the filesystem.

Parameters:

file -

Returns:

the name within the filesystem

getLocalName

public java.lang.String getLocalName(java.net.URL url)

getFileObject

public FileObject getFileObject(java.lang.String filename)

Return the handle for this file. This is not the file itself, and accessing this object does not necessarily download the resource. This will be a container for file metadata, in addition to providing access to the data file itself.

Specified by:

getFileObject in class FileSystem

Parameters:

filename - the name of the file within the filesystem.

Returns:

a FileObject for the file

getLastAccessed

protected long getLastAccessed(java.lang.String filename)

reduce the number of hits to a server by caching last access times for local files. Note subclasses of this must call markAccess to indicate the file is accessed. For example, we will not do a head request to check for an update more than once per minute.

Parameters:

filename - the filename within the filesystem.

Returns:

the last time the file was accessed via a HEAD request.

copyStream

protected long copyStream(java.io.InputStream is,
                          java.io.OutputStream out,
                          ProgressMonitor monitor)
                   throws java.io.IOException

copies data from in to out, sending the number of bytesTransferred to the monitor. NOTE: monitor.finished is not called, breaking monitor rules.

Parameters:

is - the input stream

out - the output stream

monitor - monitor for the task. Note this violates the monitor policy, and only calls setTaskProgress. Use with care!

Returns:

number of bytes transferred.

Throws:

java.io.IOException

consumeStream

public static void consumeStream(java.io.InputStream err)
                          throws java.io.IOException

Deprecated. see HtmlUtil.consumeStream.

nice clients consume both the stderr and stdout coming from websites. http://docs.oracle.com/javase/1.5.0/docs/guide/net/http-keepalive.html suggests that you "do not abandon connection"

Parameters:

err -

Throws:

java.io.IOException

See Also:

HtmlUtil.consumeStream(java.io.InputStream)

toString

public java.lang.String toString()

Overrides:

toString in class java.lang.Object

isAppletMode

public boolean isAppletMode()

setAppletMode

public void setAppletMode(boolean applet)