org.das2.util.filesystem

Class HttpFileSystem

java.lang.Object

org.das2.util.filesystem.FileSystem

org.das2.util.filesystem.WebFileSystem

org.das2.util.filesystem.HttpFileSystem

Direct Known Subclasses:

GitHubFileSystem

public class HttpFileSystem
extends WebFileSystem

Make a web folder accessible. This assumes listings are provided in html form by requesting links ending in a slash. For example, http://autoplot.org/data/pngwalk/. Links to resources "outside" of the filesystem are not considered part of the filesystem. Again, this assumes listings are HTML content, and I suspect this will be changing (xml+client-side-xslt)...

Nested Class Summary

Nested classes/interfaces inherited from class org.das2.util.filesystem.FileSystem

FileSystem.DirectoryEntry, FileSystem.FileSystemOfflineException

Field Summary

Fields

Modifier and Type	Field and Description
protected static java.util.logging.Logger	logger

Fields inherited from class org.das2.util.filesystem.WebFileSystem

HTTP_CHECK_TIMESTAMP_LIMIT_MS, id, LISTING_TIMEOUT_MS, localRoot, MEMORY_LISTING_TIMEOUT_MS, offline, offlineMessage, offlineResponseCode, PROP_OFFLINE, PROP_READ_ONLY_CACHE, protocol

Fields inherited from class org.das2.util.filesystem.FileSystem

loggerUrl, NULL, PROP_CASE_INSENSITIVE, properties

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	HttpFileSystem(java.net.URI root, java.io.File localRoot) Create a new HttpFileSystem mirroring the root, a URL pointing to "http" or "https", in the local folder.

Method Summary

Modifier and Type	Method and Description
static HttpFileSystem	createHttpFileSystem(java.net.URI rooturi) Create a filesystem from the URI.
protected java.util.Map<java.lang.String,java.lang.String>	downloadFile(java.lang.String filename, java.io.File targetFile, java.io.File partFile, ProgressMonitor monitor) Transfers the file from the remote store to a local copy f.
protected java.lang.String	getCookie() return the cookie needed.
protected java.util.Map<java.lang.String,java.lang.Object>	getHeadMeta(java.lang.String f) this is introduced to support checking if the symbol foo/bar is a folder by checking for a 303 redirect.
boolean	isDirectory(java.lang.String filename) dumb method looks for / in parent directory's listing.
static boolean	isRegexNoWild(java.lang.String regex) return true if the regular expression for the file is actually a single file with no wildcards.
java.lang.String[]	listDirectory(java.lang.String directory) list the directory, using the cached entry from listDirectoryFromMemory, or by HtmlUtil.getDirectoryListing.
java.lang.String[]	listDirectory(java.lang.String directory, java.lang.String regex) returns a list of the names of the files in a directory that match regex.
FileSystem.DirectoryEntry	maybeUpdateDirectoryEntry(java.lang.String filename, boolean force) HTTP listings are really done by querying the single file, so support this by issuing a head request
protected java.util.Map<java.lang.String,java.lang.String>	reduceMeta(java.net.URLConnection connect) pull out some of the headers to record ETag and Content_Type.

Methods inherited from class org.das2.util.filesystem.WebFileSystem

addPropertyChangeListener, addRoCacheEntries, cacheListing, consumeStream, copyStream, getDownloadDirectory, getDownloadLock, getFileObject, getLastAccessed, getLocalName, getLocalName, getLocalRoot, getLocalRootAbsPath, getOfflineMessage, getOfflineResponseCode, getPartFile, getProtocol, getReadOnlyCache, getRootURL, getURI, getURL, isAppletMode, isGitClone, isListingCached, isOffline, listDirectoryFromMemory, listingFile, localRoot, removePropertyChangeListener, resetListCache, resetListingCache, setAppletMode, setOffline, setReadOnlyCache, toString

Methods inherited from class org.das2.util.filesystem.FileSystem

create, create, create, create, create, create, createFileSystem, getExceptionHandler, getListing, getListing, getProperty, getRootURI, listDirectory, listDirectory, listDirectoryDeep, peek, peekInstances, recreate, recreate, registerFileSystemFactory, reset, reset, setExceptionHandler, settings, splitUrl, toCanonicalFilename, toCanonicalFolderName, toCanonicalFolderName

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

logger

protected static final java.util.logging.Logger logger

Constructor Detail

HttpFileSystem

protected HttpFileSystem(java.net.URI root,
                         java.io.File localRoot)

Create a new HttpFileSystem mirroring the root, a URL pointing to "http" or "https", in the local folder.

Parameters:

root - the root of the filesystem

localRoot - the local root where files are downloaded.

Method Detail

getCookie

protected java.lang.String getCookie()

return the cookie needed.

Returns:

createHttpFileSystem

public static HttpFileSystem createHttpFileSystem(java.net.URI rooturi)
                                           throws FileSystem.FileSystemOfflineException,
                                                  java.net.UnknownHostException,
                                                  java.io.FileNotFoundException

Create a filesystem from the URI. Note "user@" will be added to the URI if credentials are needed and added automatically.

Parameters:

rooturi -

Returns:

the filesystem.

Throws:

FileSystem.FileSystemOfflineException

java.net.UnknownHostException

java.io.FileNotFoundException

reduceMeta

protected java.util.Map<java.lang.String,java.lang.String> reduceMeta(java.net.URLConnection connect)

pull out some of the headers to record ETag and Content_Type.

Parameters:

connect -

Returns:

downloadFile

protected java.util.Map<java.lang.String,java.lang.String> downloadFile(java.lang.String filename,
                                                                        java.io.File targetFile,
                                                                        java.io.File partFile,
                                                                        ProgressMonitor monitor)
                                                                 throws java.io.IOException

Description copied from class: WebFileSystem

Transfers the file from the remote store to a local copy f. This should only be used within the class and subclasses, clients should use getFileObject( String ).getFile(). Subclasses implementing this should download data to partfile, then rename partfile to f after the download is complete. Note, this is non-trivial, since several threads and even several processes may be using the same area at once. See HttpFileSystem's implementation of this before attempting to implement the function.

Specified by:

downloadFile in class WebFileSystem

Parameters:

filename - filename within the filesystem.

targetFile - the target filename where the file is to be download.

partFile - use this file to stage the download

monitor - monitor the progress.

Returns:

metadata containing ETag if available.

Throws:

java.io.IOException

See Also:

WebProtocol.getMetadata(org.das2.util.filesystem.WebFileObject)

getHeadMeta

protected java.util.Map<java.lang.String,java.lang.Object> getHeadMeta(java.lang.String f)
                                                                throws java.io.IOException,
                                                                       CancelledOperationException

this is introduced to support checking if the symbol foo/bar is a folder by checking for a 303 redirect.

   EXIST->Boolean
   REAL_NAME->String

others are just HTTP header fields like (see wget --server-response https://raw.githubusercontent.com/autoplot/jyds/master/dd.jyds):

   Content-Length   the length in bytes of the resource.
   Cache-Control    max-age=300
   Date: Fri, 18 Jul 2014 12:07:18 GMT  time stamp.
   ETag: "750d4f66c58a0ac7fef2784253bf6954d4d38a85"
   Accept-Ranges    accepts requests for part of a file.

Parameters:

f - the name within the filesystem

Returns:

the metadata, such as the Date and ETag.

Throws:

java.io.IOException

CancelledOperationException

isDirectory

public boolean isDirectory(java.lang.String filename)
                    throws java.io.IOException

dumb method looks for / in parent directory's listing. Since we have to list the parent, then IOException can be thrown.

Specified by:

isDirectory in class WebFileSystem

Parameters:

filename - the name

Returns:

true if the name appears to be a directory (folder).

Throws:

java.io.IOException

listDirectory

public java.lang.String[] listDirectory(java.lang.String directory)
                                 throws java.io.IOException

list the directory, using the cached entry from listDirectoryFromMemory, or by HtmlUtil.getDirectoryListing. If there is a ro_cache, then add extra entries from here as well. Note the following extentions are hidden: .css, .php, .jnlp, .part.

Specified by:

listDirectory in class WebFileSystem

Parameters:

directory - name within the filesystem

Returns:

names within the directory

Throws:

java.io.IOException

See Also:

for listings constrained by regular expressions.

isRegexNoWild

public static boolean isRegexNoWild(java.lang.String regex)

return true if the regular expression for the file is actually a single file with no wildcards. This is tricky because for years a single period (".") has been left in the name, so that "." matches ".", and really screen.png would match screen_png. To contain this logic, this routine is introduced. Presently it just checks for "screen.png" so that a pngwalk of each screenshot (https://autoplot.org/jnlp/v$x/screen.png) can be made, which is useful when looking for old versions.

Parameters:

regex -

Returns:

listDirectory

public java.lang.String[] listDirectory(java.lang.String directory,
                                        java.lang.String regex)
                                 throws java.io.IOException

Description copied from class: FileSystem

returns a list of the names of the files in a directory that match regex. Trailing slashes on directory names are not part of the name and need not be part of the regex. Note regex is a regular expression (.*\.dat), not a glob (*.dat)

Overrides:

listDirectory in class WebFileSystem

Parameters:

directory - the directory

regex - regular expression

Returns:

Throws:

java.io.IOException

maybeUpdateDirectoryEntry

public FileSystem.DirectoryEntry maybeUpdateDirectoryEntry(java.lang.String filename,
                                                           boolean force)
                                                    throws java.io.IOException

HTTP listings are really done by querying the single file, so support this by issuing a head request

Overrides:

maybeUpdateDirectoryEntry in class WebFileSystem

Parameters:

filename - filename within the system

force - if true, then guarantee a listing and throw an IOException if it cannot be done.

Returns:

the DirectoryEntry showing size and date.

Throws:

java.io.IOException