org.das2.util.filesystem

Class GitHubFileSystem

java.lang.Object

org.das2.util.filesystem.FileSystem

org.das2.util.filesystem.WebFileSystem

org.das2.util.filesystem.HttpFileSystem

org.das2.util.filesystem.GitHubFileSystem

public class GitHubFileSystem
extends HttpFileSystem

GitHubFileSystem allows GitHub directories to be mounted directly, even though it is not a conventional filesystem with files residing in folders. For example, the file resource README.md found in https://github.com/autoplot/scripts/ is downloaded from https://github.com/autoplot/scripts/blob/master/README.md, with "blob/master/" added to the URL. Likewise directory "demos" is found under "tree/master/". GitHub also introduced a new problem, where dates cannot be used for evaluating file freshness. ETags are now supported in WebFileSystem to provide this functionality. Note, GitHub is slow to update the raw view, so five minutes may pass before a pushed change is visible. See https://sourceforge.net/p/autoplot/bugs/2203/ . This can be resolved by maintaining a local clone of the repo.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	GitHubFileSystem.Forge

Nested classes/interfaces inherited from class org.das2.util.filesystem.FileSystem

FileSystem.DirectoryEntry, FileSystem.FileSystemOfflineException

Field Summary

Fields inherited from class org.das2.util.filesystem.WebFileSystem

HTTP_CHECK_TIMESTAMP_LIMIT_MS, id, LISTING_TIMEOUT_MS, localRoot, MEMORY_LISTING_TIMEOUT_MS, offline, offlineMessage, offlineResponseCode, PROP_OFFLINE, PROP_READ_ONLY_CACHE, protocol

Fields inherited from class org.das2.util.filesystem.FileSystem

loggerUrl, NULL, PROP_CASE_INSENSITIVE, properties

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	GitHubFileSystem(java.net.URI root, java.io.File localRoot, java.lang.String branch, int baseOffset) Create a new GitHubFileSystem mirroring the root, a URL pointing to "http" or "https", in the local folder.

Method Summary

Modifier and Type	Method and Description
static GitHubFileSystem	createGitHubFileSystem(java.net.URI root) create GitLabs instance
static GitHubFileSystem	createGitHubFileSystem(java.net.URI root, int baseOffset)
static GitHubFileSystem.Forge	detectForge(java.net.URI root) detect if the URI is a GitHub or GitLab instance.
protected java.util.Map<java.lang.String,java.lang.String>	downloadFile(java.lang.String filename, java.io.File targetFile, java.io.File partFile, ProgressMonitor monitor) Transfers the file from the remote store to a local copy f.
static java.lang.String	getDefaultBranch(java.net.URI root, java.lang.String project) return the default branch (e.g.
static java.lang.String	getDefaultBranchGitHub(java.net.URI root, java.lang.String project) Return the default branch (e.g.
static java.lang.String	getDefaultBranchGitLab(java.net.URI root, java.lang.String project) Return the default branch (e.g.
FileObject	getFileObject(java.lang.String filename) Return the handle for this file.
static java.io.File	getLocalRoot(java.net.URI root) return the location within the file cache of this GitHub filesystem.
protected java.util.Map<java.lang.String,java.lang.String>	getRequestProperties() set the cookie if cookies are being used.
java.net.URI	getURI(java.lang.String filename) return the URI for the internal filename
java.net.URL	getURL(java.lang.String filename) return the URL for the internal filename
java.net.URL	gitHubMapDir(java.net.URI root, java.lang.String filename) github puts directories for each project under "tree/master".
java.net.URL	gitHubMapFile(java.net.URI root, java.lang.String filename) Given the URI, convert this to the link which will download the file.
static java.lang.String	isGithubFileSystem(java.lang.String h, java.lang.String path) one place that lists the GitHub (GitLab) filesystems.
java.lang.String[]	listDirectory(java.lang.String directory) list the directory, using the cached entry from listDirectoryFromMemory, or by HtmlUtil.getDirectoryListing.
java.lang.String[]	listDirectoryGithub(java.lang.String directory) Use GitHub's API to list the directory.
java.lang.String[]	listDirectoryGitLab(java.lang.String directory) Use GitLab's API to list the directory.
java.lang.String[]	listDirectoryGitLabHowever(java.lang.String directory) At some point, Gitlab started returning the filename listing in a separate JSON response.
static java.lang.String	strjoin(java.lang.String[] c, java.lang.String delim, int start, int end) this will be replaced in Java 8.
java.lang.String	toString()

Methods inherited from class org.das2.util.filesystem.HttpFileSystem

createHttpFileSystem, getCookie, getHeadMeta, isDirectory, isRegexNoWild, listDirectory, maybeUpdateDirectoryEntry, reduceMeta

Methods inherited from class org.das2.util.filesystem.WebFileSystem

addPropertyChangeListener, addRoCacheEntries, cacheListing, consumeStream, copyStream, getDownloadDirectory, getDownloadLock, getLastAccessed, getLocalName, getLocalName, getLocalRoot, getLocalRootAbsPath, getOfflineMessage, getOfflineResponseCode, getPartFile, getProtocol, getReadOnlyCache, getRootURL, isAppletMode, isGitClone, isListingCached, isOffline, listDirectoryFromMemory, listingFile, localRoot, removePropertyChangeListener, resetListCache, resetListingCache, setAppletMode, setOffline, setReadOnlyCache

Methods inherited from class org.das2.util.filesystem.FileSystem

create, create, create, create, create, create, createFileSystem, getExceptionHandler, getListing, getListing, getProperty, getRootURI, listDirectory, listDirectory, listDirectoryDeep, peek, peekInstances, recreate, recreate, registerFileSystemFactory, reset, reset, setExceptionHandler, settings, splitUrl, toCanonicalFilename, toCanonicalFolderName, toCanonicalFolderName, toCanonicalFolderNameString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

GitHubFileSystem

protected GitHubFileSystem(java.net.URI root,
                           java.io.File localRoot,
                           java.lang.String branch,
                           int baseOffset)

Create a new GitHubFileSystem mirroring the root, a URL pointing to "http" or "https", in the local folder. The baseOffset is 0 for https://github.com/autoplot/dev, because github.com is the GitLabs instance. The baseOffset is 1 for https://jfaden.net/git.

Parameters:

root - the root of the filesystem

localRoot - the local root where files are downloaded.

branch - the branch, typically "master", and "" if the default branch is not yet known.

baseOffset - index of the first folder of the GitLabs server.

Method Detail

getRequestProperties

protected java.util.Map<java.lang.String,java.lang.String> getRequestProperties()

Description copied from class: HttpFileSystem

set the cookie if cookies are being used.

Overrides:

getRequestProperties in class HttpFileSystem

Returns:

any properties which should be set with each transaction

createGitHubFileSystem

public static GitHubFileSystem createGitHubFileSystem(java.net.URI root)

create GitLabs instance

Parameters:

root - the root

Returns:

the filesystem.

detectForge

public static GitHubFileSystem.Forge detectForge(java.net.URI root)

detect if the URI is a GitHub or GitLab instance. This presently looks for github.com or gitlab.umn.edu.

Parameters:

root -

Returns:

Forge value.

getLocalRoot

public static java.io.File getLocalRoot(java.net.URI root)

return the location within the file cache of this GitHub filesystem. TODO: There's something with branches that still needs work. Also the constructor does not use this.

Parameters:

root -

Returns:

the directory containing the resource.

isGithubFileSystem

public static java.lang.String isGithubFileSystem(java.lang.String h,
                                                  java.lang.String path)

one place that lists the GitHub (GitLab) filesystems.

Parameters:

h - the host

path - path to the top of the GitLabs instance.

Returns:

null if it is not a GitHub filesystem, or the initial path otherwise

createGitHubFileSystem

public static GitHubFileSystem createGitHubFileSystem(java.net.URI root,
                                                      int baseOffset)

Parameters:

root - the root

baseOffset - the number of folders after the host in the root, for this GitLabs instance.

Returns:

the filesystem.

listDirectoryGitLabHowever

public java.lang.String[] listDirectoryGitLabHowever(java.lang.String directory)
                                              throws java.io.IOException

At some point, Gitlab started returning the filename listing in a separate JSON response. TODO: It's impossible to distinguish files from folders.

Parameters:

directory -

Returns:

Throws:

java.io.IOException

listDirectoryGithub

public java.lang.String[] listDirectoryGithub(java.lang.String directory)
                                       throws java.io.IOException

Use GitHub's API to list the directory.

Parameters:

directory - within the filesystem

Returns:

the list of files, with / suffix for directories.

Throws:

java.io.IOException

listDirectoryGitLab

public java.lang.String[] listDirectoryGitLab(java.lang.String directory)
                                       throws java.io.IOException

Use GitLab's API to list the directory. For example, https://research-git.uiowa.edu/space-physics/rbsp/ap-script/-/tree/master/u/ivar/20210416/ should list the files in this directory, using calls like https://research-git.uiowa.edu/api/v4/projects/space-physics%2Frbsp%2Fap-script/repository/tree?path=u/ivar/20210416/&ref=master which will return a JSON listing.

Parameters:

directory - within the filesystem

Returns:

the list of files, with / suffix for directories.

Throws:

java.io.IOException

getDefaultBranchGitLab

public static java.lang.String getDefaultBranchGitLab(java.net.URI root,
                                                      java.lang.String project)
                                               throws java.io.IOException

Return the default branch (e.g. main or master) for the project.

Parameters:

root - the GitLab root (e.g. https://research-git.uiowa.edu/)

project - the project (e.g. 'space-physics/rbsp/ap-script')

Returns:

the default branch (e.g. 'master')

Throws:

java.io.IOException

getDefaultBranchGitHub

public static java.lang.String getDefaultBranchGitHub(java.net.URI root,
                                                      java.lang.String project)
                                               throws java.io.IOException

Return the default branch (e.g. main or master) for the project.

Parameters:

root - the GitHub root (e.g. https://github.com/autoplot/)

project - the project (e.g. 'autoplot/dev')

Returns:

the default branch (e.g. 'master')

Throws:

java.io.IOException

getDefaultBranch

public static java.lang.String getDefaultBranch(java.net.URI root,
                                                java.lang.String project)
                                         throws java.io.IOException

return the default branch (e.g. master or main) for the repo

Parameters:

root -

project -

Returns:

Throws:

java.io.IOException

listDirectory

public java.lang.String[] listDirectory(java.lang.String directory)
                                 throws java.io.IOException

Description copied from class: HttpFileSystem

list the directory, using the cached entry from listDirectoryFromMemory, or by HtmlUtil.getDirectoryListing. If there is a ro_cache, then add extra entries from here as well. Note the following extentions are hidden: .css, .php, .jnlp, .part.

Overrides:

listDirectory in class HttpFileSystem

Parameters:

directory - name within the filesystem

Returns:

names within the directory

Throws:

java.io.IOException

See Also:

for listings constrained by regular expressions.

strjoin

public static java.lang.String strjoin(java.lang.String[] c,
                                       java.lang.String delim,
                                       int start,
                                       int end)

this will be replaced in Java 8.

Parameters:

c -

delim -

start - positive index, or negative from end.

end - positive index, or negative from end.

Returns:

gitHubMapFile

public java.net.URL gitHubMapFile(java.net.URI root,
                                  java.lang.String filename)
                           throws java.net.MalformedURLException

Given the URI, convert this to the link which will download the file. github puts directories for each project under "raw/master". TODO: verify this works (I don't think it will) when filename contains slash.

Parameters:

root - the Das2 FileSystem root, which can be a directory within a repo.

filename - the filename to download, not containing any slashes.

Returns:

the URL to download the file.

Throws:

java.net.MalformedURLException

See Also:

https://github.com/autoplot/dev/blob/master/bugs/ghdas2/demo173.jy

gitHubMapDir

public java.net.URL gitHubMapDir(java.net.URI root,
                                 java.lang.String filename)
                          throws java.net.MalformedURLException

github puts directories for each project under "tree/master".

Parameters:

root -

filename -

Returns:

Throws:

java.net.MalformedURLException

getURI

public java.net.URI getURI(java.lang.String filename)

Description copied from class: WebFileSystem

return the URI for the internal filename

Overrides:

getURI in class WebFileSystem

Parameters:

filename - internal filename

Returns:

getURL

public java.net.URL getURL(java.lang.String filename)

return the URL for the internal filename

Overrides:

getURL in class WebFileSystem

Parameters:

filename - internal filename

Returns:

See Also:

DefaultHttpProtocol.urlEncodeSansSlash(java.lang.String)

downloadFile

protected java.util.Map<java.lang.String,java.lang.String> downloadFile(java.lang.String filename,
                                                                        java.io.File targetFile,
                                                                        java.io.File partFile,
                                                                        ProgressMonitor monitor)
                                                                 throws java.io.IOException

Description copied from class: WebFileSystem

Transfers the file from the remote store to a local copy f. This should only be used within the class and subclasses, clients should use getFileObject( String ).getFile(). Subclasses implementing this should download data to partfile, then rename partfile to f after the download is complete. Note, this is non-trivial, since several threads and even several processes may be using the same area at once. See HttpFileSystem's implementation of this before attempting to implement the function.

Overrides:

downloadFile in class HttpFileSystem

Parameters:

filename - filename within the filesystem.

targetFile - the target filename where the file is to be download.

partFile - use this file to stage the download

monitor - monitor the progress.

Returns:

metadata containing ETag if available.

Throws:

java.io.IOException

See Also:

WebProtocol.getMetadata(org.das2.util.filesystem.WebFileObject)

getFileObject

public FileObject getFileObject(java.lang.String filename)

Description copied from class: WebFileSystem

Return the handle for this file. This is not the file itself, and accessing this object does not necessarily download the resource. This will be a container for file metadata, in addition to providing access to the data file itself.

Overrides:

getFileObject in class WebFileSystem

Parameters:

filename - the name of the file within the filesystem.

Returns:

a FileObject for the file

toString

public java.lang.String toString()

Overrides:

toString in class WebFileSystem