org.das2.util.filesystem

Class GitHubFileSystem

java.lang.Object

org.das2.util.filesystem.FileSystem

org.das2.util.filesystem.WebFileSystem

org.das2.util.filesystem.HttpFileSystem

org.das2.util.filesystem.GitHubFileSystem

public class GitHubFileSystem
extends HttpFileSystem

GitHubFileSystem allows GitHub directories to be mounted directly, even though it is not a conventional filesystem with files residing in folders. For example, the file resource README.md found in https://github.com/autoplot/scripts/ is downloaded from https://github.com/autoplot/scripts/blob/master/README.md, with "blob/master/" added to the URL. Likewise directory "demos" is found under "tree/master/". GitHub also introduced a new problem, where dates cannot be used for evaluating file freshness. ETags are now supported in WebFileSystem to provide this functionality. Note, GitHub is slow to update the raw view, so five minutes may pass before a pushed change is visible. See https://sourceforge.net/p/autoplot/bugs/2203/ . This can be resolved by maintaining a local clone of the repo.

Nested Class Summary

Nested classes/interfaces inherited from class org.das2.util.filesystem.FileSystem

FileSystem.DirectoryEntry, FileSystem.FileSystemOfflineException

Field Summary

Fields inherited from class org.das2.util.filesystem.WebFileSystem

HTTP_CHECK_TIMESTAMP_LIMIT_MS, id, LISTING_TIMEOUT_MS, localRoot, MEMORY_LISTING_TIMEOUT_MS, offline, offlineMessage, offlineResponseCode, PROP_OFFLINE, PROP_READ_ONLY_CACHE, protocol

Fields inherited from class org.das2.util.filesystem.FileSystem

loggerUrl, NULL, PROP_CASE_INSENSITIVE, properties

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	GitHubFileSystem(java.net.URI root, java.io.File localRoot, java.lang.String branch, int baseOffset) Create a new GitHubFileSystem mirroring the root, a URL pointing to "http" or "https", in the local folder.

Method Summary

Modifier and Type	Method and Description
static GitHubFileSystem	createGitHubFileSystem(java.net.URI root) create GitLabs instance
static GitHubFileSystem	createGitHubFileSystem(java.net.URI root, int baseOffset)
protected java.util.Map<java.lang.String,java.lang.String>	downloadFile(java.lang.String filename, java.io.File targetFile, java.io.File partFile, ProgressMonitor monitor) Transfers the file from the remote store to a local copy f.
FileObject	getFileObject(java.lang.String filename) Return the handle for this file.
static java.io.File	getLocalRoot(java.net.URI root) return the location within the file cache of this GitHub filesystem.
java.net.URI	getURI(java.lang.String filename) return the URI for the internal filename
java.net.URL	getURL(java.lang.String filename) return the URL for the internal filename
java.net.URL	gitHubMapDir(java.net.URI root, java.lang.String filename) github puts directories for each project under "tree/master".
java.net.URL	gitHubMapFile(java.net.URI root, java.lang.String filename) Given the URI, convert this to the link which will download the file.
static java.lang.String	isGithubFileSystem(java.lang.String h, java.lang.String path) one place that lists the GitHub (GitLab) filesystems.
java.lang.String[]	listDirectory(java.lang.String directory) list the directory, using the cached entry from listDirectoryFromMemory, or by HtmlUtil.getDirectoryListing.
java.lang.String[]	listDirectoryGithub(java.lang.String directory) Use GitHub's API to list the directory.
java.lang.String[]	listDirectoryGitLab(java.lang.String directory) Use GitLab's API to list the directory.
java.lang.String[]	listDirectoryGitLabHowever(java.lang.String directory) At some point, Gitlab started returning the filename listing in a separate JSON response.
static java.lang.String	strjoin(java.lang.String[] c, java.lang.String delim, int start, int end) this will be replaced in Java 8.
java.lang.String	toString()

Methods inherited from class org.das2.util.filesystem.HttpFileSystem

createHttpFileSystem, getCookie, getHeadMeta, isDirectory, isRegexNoWild, listDirectory, maybeUpdateDirectoryEntry, reduceMeta

Methods inherited from class org.das2.util.filesystem.WebFileSystem

addPropertyChangeListener, addRoCacheEntries, cacheListing, consumeStream, copyStream, getDownloadDirectory, getDownloadLock, getLastAccessed, getLocalName, getLocalName, getLocalRoot, getLocalRootAbsPath, getOfflineMessage, getOfflineResponseCode, getPartFile, getProtocol, getReadOnlyCache, getRootURL, isAppletMode, isGitClone, isListingCached, isOffline, listDirectoryFromMemory, listingFile, localRoot, removePropertyChangeListener, resetListCache, resetListingCache, setAppletMode, setOffline, setReadOnlyCache

Methods inherited from class org.das2.util.filesystem.FileSystem

create, create, create, create, create, create, createFileSystem, getExceptionHandler, getListing, getListing, getProperty, getRootURI, listDirectory, listDirectory, listDirectoryDeep, peek, peekInstances, recreate, recreate, registerFileSystemFactory, reset, reset, setExceptionHandler, settings, splitUrl, toCanonicalFilename, toCanonicalFolderName, toCanonicalFolderName

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

GitHubFileSystem

protected GitHubFileSystem(java.net.URI root,
                           java.io.File localRoot,
                           java.lang.String branch,
                           int baseOffset)

Create a new GitHubFileSystem mirroring the root, a URL pointing to "http" or "https", in the local folder. The baseOffset is 0 for https://github.com/autoplot/dev, because github.com is the GitLabs instance. The baseOffset is 1 for https://jfaden.net/git.

Parameters:

root - the root of the filesystem

localRoot - the local root where files are downloaded.

branch - the branch, typically "master".

baseOffset - index of the first folder of the GitLabs server.

Method Detail

createGitHubFileSystem

public static GitHubFileSystem createGitHubFileSystem(java.net.URI root)

create GitLabs instance

Parameters:

root - the root

Returns:

the filesystem.

getLocalRoot

public static java.io.File getLocalRoot(java.net.URI root)

return the location within the file cache of this GitHub filesystem. TODO: There's something with branches that still needs work. Also the constructor does not use this.

Parameters:

root -

Returns:

the directory containing the resource.

isGithubFileSystem

public static java.lang.String isGithubFileSystem(java.lang.String h,
                                                  java.lang.String path)

one place that lists the GitHub (GitLab) filesystems.

Parameters:

h - the host

path - path to the top of the GitLabs instance.

Returns:

null if it is not a GitHub filesystem, or the initial path otherwise

createGitHubFileSystem

public static GitHubFileSystem createGitHubFileSystem(java.net.URI root,
                                                      int baseOffset)

Parameters:

root - the root

baseOffset - the number of folders after the host in the root, for this GitLabs instance.

Returns:

the filesystem.

listDirectoryGitLabHowever

public java.lang.String[] listDirectoryGitLabHowever(java.lang.String directory)
                                              throws java.io.IOException

At some point, Gitlab started returning the filename listing in a separate JSON response. TODO: It's impossible to distinguish files from folders. On jfaden.net/git, this API does not work.

Parameters:

directory -

Returns:

Throws:

java.io.IOException

listDirectoryGithub

public java.lang.String[] listDirectoryGithub(java.lang.String directory)
                                       throws java.io.IOException

Use GitHub's API to list the directory.

Parameters:

directory - within the filesystem

Returns:

the list of files, with / suffix for directories.

Throws:

java.io.IOException

listDirectoryGitLab

public java.lang.String[] listDirectoryGitLab(java.lang.String directory)
                                       throws java.io.IOException

Use GitLab's API to list the directory. For example, https://research-git.uiowa.edu/space-physics/rbsp/ap-script/-/tree/master/u/ivar/20210416/ should list the files in this directory, using calls like https://research-git.uiowa.edu/api/v4/projects/space-physics%2Frbsp%2Fap-script/repository/tree?path=u/ivar/20210416/&ref=master which will return a JSON listing.

Parameters:

directory - within the filesystem

Returns:

the list of files, with / suffix for directories.

Throws:

java.io.IOException

listDirectory

public java.lang.String[] listDirectory(java.lang.String directory)
                                 throws java.io.IOException

Description copied from class: HttpFileSystem

list the directory, using the cached entry from listDirectoryFromMemory, or by HtmlUtil.getDirectoryListing. If there is a ro_cache, then add extra entries from here as well. Note the following extentions are hidden: .css, .php, .jnlp, .part.

Overrides:

listDirectory in class HttpFileSystem

Parameters:

directory - name within the filesystem

Returns:

names within the directory

Throws:

java.io.IOException

See Also:

for listings constrained by regular expressions.

strjoin

public static java.lang.String strjoin(java.lang.String[] c,
                                       java.lang.String delim,
                                       int start,
                                       int end)

this will be replaced in Java 8.

Parameters:

c -

delim -

start - positive index, or negative from end.

end - positive index, or negative from end.

Returns:

gitHubMapFile

public java.net.URL gitHubMapFile(java.net.URI root,
                                  java.lang.String filename)
                           throws java.net.MalformedURLException

Given the URI, convert this to the link which will download the file. github puts directories for each project under "raw/master".

Parameters:

root -

filename -

Returns:

Translate:

%{code
 https://abbith.physics.uiowa.edu/jbf/myawesomepublicproject/blob/24dff04b9bcb275d8bfd85b38e0e8b039b21d655/sayAwesome.jy to 

 https://abbith.physics.uiowa.edu/jbf/myawesomepublicproject/raw/24dff04b9bcb275d8bfd85b38e0e8b039b21d655/sayAwesome.jy
 https://github.com/autoplot/app/raw/master/Autoplot/src/resources/badge_ok.png to
 https://github.com/autoplot/app/master/Autoplot/src/resources/badge_ok.png
 https://jfaden.net/git/jbfaden/public/blob/master/u/jeremy/2019/20191023/updates.jy

 https://research-git.uiowa.edu/space-physics/juno/ap-script/master/test/testap.jy to
 https://research-git.uiowa.edu/space-physics/juno/ap-script/-/raw/master/test/testap.jy 

 https://research-git.uiowa.edu/jbf/testproject/-/blob/master/script/testScript.jy to
 https://research-git.uiowa.edu/jbf/testproject/master/script/testScript.jy
 }
 

Throws:

java.net.MalformedURLException

gitHubMapDir

public java.net.URL gitHubMapDir(java.net.URI root,
                                 java.lang.String filename)
                          throws java.net.MalformedURLException

github puts directories for each project under "tree/master".

Parameters:

root -

filename -

Returns:

Throws:

java.net.MalformedURLException

getURI

public java.net.URI getURI(java.lang.String filename)

Description copied from class: WebFileSystem

return the URI for the internal filename

Overrides:

getURI in class WebFileSystem

Parameters:

filename - internal filename

Returns:

getURL

public java.net.URL getURL(java.lang.String filename)

return the URL for the internal filename

Overrides:

getURL in class WebFileSystem

Parameters:

filename - internal filename

Returns:

See Also:

DefaultHttpProtocol.urlEncodeSansSlash(java.lang.String)

downloadFile

protected java.util.Map<java.lang.String,java.lang.String> downloadFile(java.lang.String filename,
                                                                        java.io.File targetFile,
                                                                        java.io.File partFile,
                                                                        ProgressMonitor monitor)
                                                                 throws java.io.IOException

Description copied from class: WebFileSystem

Transfers the file from the remote store to a local copy f. This should only be used within the class and subclasses, clients should use getFileObject( String ).getFile(). Subclasses implementing this should download data to partfile, then rename partfile to f after the download is complete. Note, this is non-trivial, since several threads and even several processes may be using the same area at once. See HttpFileSystem's implementation of this before attempting to implement the function.

Overrides:

downloadFile in class HttpFileSystem

Parameters:

filename - filename within the filesystem.

targetFile - the target filename where the file is to be download.

partFile - use this file to stage the download

monitor - monitor the progress.

Returns:

metadata containing ETag if available.

Throws:

java.io.IOException

See Also:

WebProtocol.getMetadata(org.das2.util.filesystem.WebFileObject)

getFileObject

public FileObject getFileObject(java.lang.String filename)

Description copied from class: WebFileSystem

Return the handle for this file. This is not the file itself, and accessing this object does not necessarily download the resource. This will be a container for file metadata, in addition to providing access to the data file itself.

Overrides:

getFileObject in class WebFileSystem

Parameters:

filename - the name of the file within the filesystem.

Returns:

a FileObject for the file

toString

public java.lang.String toString()

Overrides:

toString in class WebFileSystem