Geoserver Image Pyramid with JPEG Compression - gis

Question

I have an Image Pyramid (created using gdal_retile) that I want to import into Geoserver (2.8.2, using the Image Pyramid plugin) and publish as WMS.
I have been able to do this successfully using GeoTIFF images, however as we are storing many terabytes of imagery, we would like to use JPEG compressed TIFF.
If I run gdal_retile using the compression argument, e.g:
#python %OSGEO4W_ROOT%\bin\gdal_retile.py -v -r bilinear -levels
%LEVELS% -tileIndex TileIndex.shp -ps %PIXELS% %PIXELS% -co
"TILED=YES" -co "COMPRESS=JPEG" -targetDir %INPUT%\PROCESSED\
--optfile %INPUT%\tilelist.txt
Then Geoserver errors when trying to publish the layer but doesn't seem to provide a complete error log (I've pasted what it supplied below).
There's some supporting info below - hopefully this is enough - thanks in advance!
I have also tried this on version 2.7.6, with the same results.
My folder structure after creating the Image Pyramid data store is as follows (NOTE: listed output/1 only as an example, all Level folders have the same structure):
../output
Level Folders (0,1,2,3,4,5)
output.prj
output.properties
../output/1
Contains images (JPEG Compressed TIFF)
Tile Index (1.shp,1.shx,1.prj,1.dbf)
Properties File (1.properties)
Fix File (1.fix) - not sure what this is.
I have run gdal_info on one of the outputted TIF files:
Driver: GTiff/GeoTIFF
Files: C:\geoserver\data\pyramids\constantia_comp\PROCESSED\output\1\constantia2
ransparent_mosaic_group1_4_3.tif
Size is 2048, 1757
Coordinate System is:
PROJCS["WGS 84 / UTM zone 34S",
GEOGCS["WGS 84",
DATUM["WGS_1984",
SPHEROID["WGS 84",6378137,298.257223563,
AUTHORITY["EPSG","7030"]],
AUTHORITY["EPSG","6326"]],
PRIMEM["Greenwich",0,
AUTHORITY["EPSG","8901"]],
UNIT["degree",0.0174532925199433,
AUTHORITY["EPSG","9122"]],
AUTHORITY["EPSG","4326"]],
PROJECTION["Transverse_Mercator"],
PARAMETER["latitude_of_origin",0],
PARAMETER["central_meridian",21],
PARAMETER["scale_factor",0.9996],
PARAMETER["false_easting",500000],
PARAMETER["false_northing",10000000],
UNIT["metre",1,
AUTHORITY["EPSG","9001"]],
AXIS["Easting",EAST],
AXIS["Northing",NORTH],
AUTHORITY["EPSG","32734"]]
Origin = (262297.873039999980000,6229349.579940000500000)
Pixel Size = (0.224620000000000,-0.224620000000000)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
COMPRESSION=JPEG
INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left ( 262297.873, 6229349.580) ( 18d25'29.43"E, 34d 2'58.65"S)
Lower Left ( 262297.873, 6228954.923) ( 18d25'29.04"E, 34d 3'11.45"S)
Upper Right ( 262757.895, 6229349.580) ( 18d25'47.35"E, 34d 2'59.02"S)
Lower Right ( 262757.895, 6228954.923) ( 18d25'46.96"E, 34d 3'11.83"S)
Center ( 262527.884, 6229152.251) ( 18d25'38.19"E, 34d 3' 5.24"S)
Band 1 Block=256x256 Type=Byte, ColorInterp=Red
Mask Flags: PER_DATASET ALPHA
Band 2 Block=256x256 Type=Byte, ColorInterp=Green
Mask Flags: PER_DATASET ALPHA
Band 3 Block=256x256 Type=Byte, ColorInterp=Blue
Mask Flags: PER_DATASET ALPHA
Band 4 Block=256x256 Type=Byte, ColorInterp=Alpha
Error Log:
java.lang.RuntimeException: Error occurred while building the resources for the configuration page
at org.geoserver.web.data.layer.NewLayerPage.buildLayerInfo(NewLayerPage.java:356)
at org.geoserver.web.data.layer.NewLayerPage$8.onClick(NewLayerPage.java:281)
at org.geoserver.web.wicket.SimpleAjaxLink$1.onClick(SimpleAjaxLink.java:46)
at org.apache.wicket.ajax.markup.html.AjaxLink$1.onEvent(AjaxLink.java:68)
at org.apache.wicket.ajax.AjaxEventBehavior.respond(AjaxEventBehavior.java:177)
at org.apache.wicket.ajax.AbstractDefaultAjaxBehavior.onRequest(AbstractDefaultAjaxBehavior.java:300)
at org.apache.wicket.request.target.component.listener.BehaviorRequestTarget.processEvents(BehaviorRequestTarget.java:119)
at org.apache.wicket.request.AbstractRequestCycleProcessor.processEvents(AbstractRequestCycleProcessor.java:92)
at org.apache.wicket.RequestCycle.processEventsAndRespond(RequestCycle.java:1250)
at org.apache.wicket.RequestCycle.step(RequestCycle.java:1329)
at org.apache.wicket.RequestCycle.steps(RequestCycle.java:1436)
at org.apache.wicket.RequestCycle.request(RequestCycle.java:545)
at org.apache.wicket.protocol.http.WicketFilter.doGet(WicketFilter.java:484)
at org.apache.wicket.protocol.http.WicketServlet.doGet(WicketServlet.java:138)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.springframework.web.servlet.mvc.ServletWrappingController.handleRequestInternal(ServletWrappingController.java:159)
at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:778)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1093)
at org.geoserver.filters.ThreadLocalsCleanupFilter.doFilter(ThreadLocalsCleanupFilter.java:28)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084)
at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:75)
at org.geoserver.wms.animate.AnimatorFilter.doFilter(AnimatorFilter.java:71)
at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:71)
at org.geoserver.filters.SpringDelegatingFilter.doFilter(SpringDelegatingFilter.java:46)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084)
at org.geoserver.platform.AdvancedDispatchFilter.doFilter(AdvancedDispatchFilter.java:50)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.geoserver.security.filter.GeoServerAnonymousAuthenticationFilter.doFilter(GeoServerAnonymousAuthenticationFilter.java:54)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
at org.geoserver.security.filter.GeoServerUserNamePasswordAuthenticationFilter.doFilter(GeoServerUserNamePasswordAuthenticationFilter.java:116)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:146)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at org.geoserver.security.filter.GeoServerSecurityContextPersistenceFilter$1.doFilter(GeoServerSecurityContextPersistenceFilter.java:53)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173)
at org.geoserver.security.GeoServerSecurityFilterChainProxy.doFilter(GeoServerSecurityFilterChainProxy.java:135)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084)
at org.geoserver.filters.LoggingFilter.doFilter(LoggingFilter.java:83)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084)
at org.geoserver.filters.GZIPFilter.doFilter(GZIPFilter.java:42)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084)
at org.geoserver.filters.SessionDebugFilter.doFilter(SessionDebugFilter.java:48)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084)
at org.geoserver.filters.FlushSafeFilter.doFilter(FlushSafeFilter.java:44)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084)
at org.vfny.geoserver.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:109)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:360)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:726)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:206)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:828)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:514)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:395)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:450) Caused by: org.geotools.data.DataSourceException: Unable to load properties from:C:\geoserver\data\pyramids\constantia_comp\PROCESSED\test\0\datastore.properties
at org.geotools.gce.imagemosaic.ImageMosaicReader.<init>(ImageMosaicReader.java:470)
at org.geotools.gce.imagepyramid.ImagePyramidReader.getImageMosaicReaderForLevel(ImagePyramidReader.java:655)
at org.geotools.gce.imagepyramid.ImagePyramidReader.loadRequestedTiles(ImagePyramidReader.java:484)
at org.geotools.gce.imagepyramid.ImagePyramidReader.loadTiles(ImagePyramidReader.java:431)
at org.geotools.gce.imagepyramid.ImagePyramidReader.read(ImagePyramidReader.java:372)
at org.geotools.coverage.grid.io.AbstractGridCoverage2DReader.read(AbstractGridCoverage2DReader.java:249)
at org.geoserver.catalog.CoverageDimensionCustomizerReader.read(CoverageDimensionCustomizerReader.java:227)
at org.geoserver.catalog.SingleGridCoverage2DReader.read(SingleGridCoverage2DReader.java:141)
at org.geoserver.catalog.CatalogBuilder.buildCoverage(CatalogBuilder.java:974)
at org.geoserver.catalog.CatalogBuilder.buildCoverage(CatalogBuilder.java:855)
at org.geoserver.web.data.layer.NewLayerPage.buildLayerInfo(NewLayerPage.java:346) ... 97 more Caused by: org.geotools.data.DataSourceException: Unable to load properties from:C:\geoserver\data\pyramids\constantia_comp\PROCESSED\test\0\datastore.properties
at org.geotools.gce.imagemosaic.ImageMosaicReader.initReaderFromURL(ImageMosaicReader.java:643)
at org.geotools.gce.imagemosaic.ImageMosaicReader.<init>(ImageMosaicReader.java:468) ... 107 more Caused by: java.io.IOException: Unable to load properties from:C:\geoserver\data\pyramids\constantia_comp\PROCESSED\test\0\datastore.properties
at org.geotools.gce.imagemosaic.CatalogManager.createGranuleCatalogProperties(CatalogManager.java:180)
at org.geotools.gce.imagemosaic.ImageMosaicReader.initReaderFromURL(ImageMosaicReader.java:576) ... 108 more