net.sf.jmatchparser.util
Class PrefetchDownloader

java.lang.Object
  extended by net.sf.jmatchparser.util.PrefetchDownloader

public class PrefetchDownloader
extends Object

Utility class that makes it easy to download files in the background as they are detected (over slow links) while still doing all of the parsing in a single thread. The main idea is, that whenever you encounter a file you have to parse later, you add it to the prefetcher by calling the download(String, String) method. All documents added to the prefetcher are loaded in a configurable number of background threads (and stored into the CachingDownloader's cache. When ready to parse the new document, you can request the result by calling waitFor(String), waiting for it if needed.


Constructor Summary
PrefetchDownloader(CachingDownloader cd, int threadcount, boolean cacheResults)
          Create a new prefetch downloader.
 
Method Summary
 void download(String url, String cacheName)
          Add a document to download
 void download(String url, String postData, String cacheName)
          Add a document to download, including POST data.
 void shutdown()
          Wait until all prefetching is finished and shut down the prefetch threads.
 InputStream waitFor(String cacheName)
          Wait until a given document has been prefetched.
 void waitPrefetch()
          Wait until all prefetching is finished.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

PrefetchDownloader

public PrefetchDownloader(CachingDownloader cd,
                          int threadcount,
                          boolean cacheResults)
Create a new prefetch downloader.

Parameters:
cd - Caching downloader to use
threadcount - Number of prefetch threads to start
cacheResults - Whether to cache the results in the prefetcher. If not, they are only written to the cache and can be retrieved from there
Method Detail

download

public void download(String url,
                     String cacheName)
Add a document to download

Parameters:
url - URL of the document
cacheName - Name of the document in the cache
See Also:
CachingDownloader.download(String, String)

download

public void download(String url,
                     String postData,
                     String cacheName)
Add a document to download, including POST data.

Parameters:
url - URL of the document
postData - postData to post
cacheName - Name of the document in the cache
See Also:
CachingDownloader.download(String, String, String)

waitPrefetch

public void waitPrefetch()
                  throws InterruptedException
Wait until all prefetching is finished. This is useful if the results do not have to be parsed, just downloaded, to make sure to wait until everything requested by now has been downloaded.

Throws:
InterruptedException

waitFor

public InputStream waitFor(String cacheName)
                    throws InterruptedException
Wait until a given document has been prefetched.

Parameters:
cacheName - Name of the document in the cache
Returns:
The document content as a stream
Throws:
InterruptedException

shutdown

public void shutdown()
              throws InterruptedException
Wait until all prefetching is finished and shut down the prefetch threads.

Throws:
InterruptedException


Copyright © 2011. All Rights Reserved.