org.das2.util.filesystem

Class HtmlUtil

java.lang.Object

org.das2.util.filesystem.HtmlUtil

public class HtmlUtil
extends java.lang.Object

HTML utilities, such as getting a directory listing, where a "file" is a link below the directory we are listing, and read a URL into a String.

Field Summary

Fields

Modifier and Type	Field and Description
protected static java.util.logging.Logger	loggerUrl this logger is for opening connections to remote sites.

Constructor Summary

Constructors

Constructor and Description
HtmlUtil()

Method Summary

Modifier and Type	Method and Description
static java.net.URLConnection	checkRedirect(java.net.URLConnection urlConnection) Deprecated.
static void	consumeStream(java.io.InputStream err) Deprecated.
static java.net.URL[]	getDirectoryListing(java.net.URL url) Get the listing of the web directory, returning links that are "under" the given URL.
static java.net.URL[]	getDirectoryListing(java.net.URL url, java.io.InputStream urlStream) Get the listing of the web directory, returning links that are "under" the given URL.
static java.net.URL[]	getDirectoryListing(java.net.URL url, java.io.InputStream urlStream, boolean childCheck) Get the listing of the web directory, returning links that are "under" the given URL.
static java.io.InputStream	getInputStream(java.net.URL url) get the inputStream, following redirects if a 301 or 302 is encountered.
static java.util.List<java.net.URL>	getLinks(java.net.URL url, java.lang.String content) return the links found in the content, using url as the context.
static java.util.Map<java.lang.String,java.lang.String>	getMetadata(java.net.URL url, java.util.Map<java.lang.String,java.lang.String> props) Deprecated.
static boolean	isDirectory(java.net.URL url)
static java.lang.String	readToString(java.net.URL url) read the contents of the URL into a string, assuming UTF-8 encoding.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

loggerUrl

protected static final java.util.logging.Logger loggerUrl

this logger is for opening connections to remote sites.

Constructor Detail

HtmlUtil

public HtmlUtil()

Method Detail

isDirectory

public static boolean isDirectory(java.net.URL url)

consumeStream

@Deprecated
public static void consumeStream(java.io.InputStream err)
                                      throws java.io.IOException

Deprecated.

nice clients consume both the stderr and stdout coming from websites. This reads everything off of the stream and closes it. http://docs.oracle.com/javase/1.5.0/docs/guide/net/http-keepalive.html suggests that you "do not abandon connection"

Parameters:

err - the input stream

Throws:

java.io.IOException

See Also:

HttpUtil.consumeStream(java.io.InputStream)

getDirectoryListing

public static java.net.URL[] getDirectoryListing(java.net.URL url,
                                                 java.io.InputStream urlStream)
                                          throws java.io.IOException,
                                                 CancelledOperationException

Get the listing of the web directory, returning links that are "under" the given URL. Note this does not handle off-line modes where we need to log into a website first, as is often the case for a hotel. This was refactored to support caching of listings by simply writing the content to disk.

Parameters:

url - the address.

urlStream - stream containing the URL content, which must be UTF-8 (or US-ASCII)

Returns:

list of URIs referred to in the page.

Throws:

java.io.IOException

CancelledOperationException

getDirectoryListing

public static java.net.URL[] getDirectoryListing(java.net.URL url,
                                                 java.io.InputStream urlStream,
                                                 boolean childCheck)
                                          throws java.io.IOException,
                                                 CancelledOperationException

Get the listing of the web directory, returning links that are "under" the given URL. Note this does not handle off-line modes where we need to log into a website first, as is often the case for a hotel. This was refactored to support caching of listings by simply writing the content to disk.

Parameters:

url - the address.

urlStream - stream containing the URL content, which must be UTF-8 (or US-ASCII)

childCheck - only return links to URLs "under" the url.

Returns:

list of URIs referred to in the page.

Throws:

java.io.IOException

CancelledOperationException

getDirectoryListing

public static java.net.URL[] getDirectoryListing(java.net.URL url)
                                          throws java.io.IOException,
                                                 CancelledOperationException

Get the listing of the web directory, returning links that are "under" the given URL. Note this does not handle off-line modes where we need to log into a website first, as is often the case for a hotel.

Parameters:

url -

Returns:

list of URIs referred to in the page.

Throws:

java.io.IOException

CancelledOperationException

getInputStream

public static java.io.InputStream getInputStream(java.net.URL url)
                                          throws java.io.IOException,
                                                 CancelledOperationException

get the inputStream, following redirects if a 301 or 302 is encountered. The scientist may be prompted for a password, but only if "user@" is in the URL. Note this does not explicitly close the connections to the server, and Java may not know to release the resources. TODO: fix this by wrapping the input stream and closing the connection when the stream is closed. This was done in Autoplot's DataSetURI.downloadResourceAsTempFile

Parameters:

url -

Returns:

input stream

Throws:

java.io.IOException

CancelledOperationException

See Also:

DataSetURI.downloadResourceAsTempFile(java.net.URL, org.das2.util.monitor.ProgressMonitor)

readToString

public static java.lang.String readToString(java.net.URL url)
                                     throws java.io.IOException,
                                            CancelledOperationException

read the contents of the URL into a string, assuming UTF-8 encoding.

Parameters:

url -

Returns:

Throws:

java.io.IOException

CancelledOperationException

getMetadata

@Deprecated
public static java.util.Map<java.lang.String,java.lang.String> getMetadata(java.net.URL url,
                                                                                       java.util.Map<java.lang.String,java.lang.String> props)
                                                                                throws java.io.IOException

Deprecated.

return the metadata about a URL. This will support http, https, and ftp, and will check for redirects. This will allow caching of head requests.

Parameters:

url - ftp,https, or http URL

props, - if non-null, may be a map containing cookie.

Returns:

the metadata

Throws:

java.io.IOException - when HEAD requests are made.

See Also:

HttpUtil.getMetadata(java.net.URL, java.util.Map)

checkRedirect

@Deprecated
public static java.net.URLConnection checkRedirect(java.net.URLConnection urlConnection)
                                                        throws java.io.IOException

Deprecated.

check for 301, 302 or 303 redirects, and return a new connection in this case. This should be called immediately before the urlConnection.connect call, as this must connect to get the response code.

Parameters:

urlConnection - if an HttpUrlConnection, check for 301 or 302; return connection otherwise.

Returns:

a connection, typically the same one as passed in.

Throws:

java.io.IOException

See Also:

HttpUtil.checkRedirect(java.net.URLConnection)

getLinks

public static java.util.List<java.net.URL> getLinks(java.net.URL url,
                                                    java.lang.String content)

return the links found in the content, using url as the context.

Parameters:

url - null or the url for the context.

content - the html content.

Returns:

a list of URLs.