Page cache sizing

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Page cache sizing

Peter Ondruška-4
Dear all,

how does derby.storage.pageCacheSize parameter (https://db.apache.org/derby/docs/10.13/ref/rrefproper81359.html) work with database that has multiple page sizes--tables with default 4096 bytes and tables with long/blob of 32768 byte pages?

Thanks,

--
Peter Ondruška

kaibo, s.r.o., ID 28435036, registered with the commercial register administered by the Municipal Court in Prague, section C, insert 141269.
Registered office and postal address: kaibo, s.r.o., Kališnická 379/10, Prague 3, 130 00, Czech Republic.
https://www.kaibo.eu
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Page cache sizing

Bryan Pendleton-3


how does derby.storage.pageCacheSize parameter (https://db.apache.org/derby/docs/10.13/ref/rrefproper81359.html) work with database that has multiple page sizes--tables with default 4096 bytes and tables with long/blob of 32768 byte pages?


Hi Peter,

I'm not 100% sure how this works; I think you should run some experiments.

Here's what I *think* the behavior is:
1) The page cache is sized in 4K pages, so if you set pageCacheSize=1024, you get 4 meg of page cache memory
2) Tables with 4K pages simply use the cache as you expect. 
3) Tables with 32K pages chew up 8 cache "pages" at a time, each 32K chunk of page cache holding 1 32K page from that large-page table.

Some stuff I found while searching around, which might give you some ideas for experiments you could run:


and


Sorry I'm not of much more help here.

bryan

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Page cache sizing

Peter Ondruška-4
Thanks Bryan,

I have done different kind of test so far and this is my observation:

1. set network server JVM heap size to 512 M
2. set parameter derby.storage.pageCacheSize=16384
3. booted a database with mixed block sizes (default 4 K, and large 32 K)
4. after some time I get OOM when conneting network client, this OOM is reported back from network server, see * and ** (tried to backup as well)
5. only changed parameter derby.storage.pageCacheSize=4096
6. no more OOM

Conclusion seems to be that pageCacheSize may be based really on just the number of cached blocks (as documented) and if they are mixed you better size based on 32 K size. If this is confirmed by the developers I would suggest ammending documentation with such note because some objects may automatically use 32 K page (LOBs) and can cause OOM suddenly if pageCacheSize was set for 4 K block size. But ideally a separate pageCacheSize setting would be great for large/non-standard block size(s).

*
Sun Jul 09 00:24:20 CEST 2017 Thread[DRDAConnThread_3,5,main] (DATABASE = store), (DRDAID = {233}), Connection refused : java.lang.OutOfMemoryError
Sun Jul 09 01:02:48 CEST 2017 Thread[DRDAConnThread_7,5,main] (DATABASE = store), (DRDAID = {266}), Connection refused : java.lang.OutOfMemoryError
Sun Jul 09 01:24:11 CEST 2017 Thread[DRDAConnThread_6,5,main] (DATABASE = store), (DRDAID = {283}), Connection refused : java.lang.OutOfMemoryError
Sun Jul 09 02:24:12 CEST 2017 Thread[DRDAConnThread_7,5,main] (DATABASE = store), (DRDAID = {332}), Connection refused : java.lang.OutOfMemoryError
Sun Jul 09 03:24:11 CEST 2017 Thread[DRDAConnThread_4,5,main] (DATABASE = store), (DRDAID = {381}), Connection refused : java.lang.OutOfMemoryError
Sun Jul 09 04:24:10 CEST 2017 Thread[DRDAConnThread_10,5,main] (DATABASE = store), (DRDAID = {430}), Connection refused : java.lang.OutOfMemoryError
Sun Jul 09 04:30:30 CEST 2017 Thread[DRDAConnThread_9,5,main] (DATABASE = store), (DRDAID = {439}), Connection refused : java.lang.OutOfMemoryError
Sun Jul 09 04:30:31 CEST 2017 Thread[DRDAConnThread_9,5,main] (DATABASE = store), (DRDAID = {440}), Connection refused : java.lang.OutOfMemoryError
Sun Jul 09 05:24:11 CEST 2017 Thread[DRDAConnThread_3,5,main] (DATABASE = store), (DRDAID = {481}), Connection refused : java.lang.OutOfMemoryError
Sun Jul 09 06:24:12 CEST 2017 Thread[DRDAConnThread_2,5,main] (DATABASE = store), (DRDAID = {530}), Connection refused : java.lang.OutOfMemoryError
Sun Jul 09 07:24:12 CEST 2017 Thread[DRDAConnThread_10,5,main] (DATABASE = store), (DRDAID = {579}), Connection refused : java.lang.OutOfMemoryError
Sun Jul 09 07:24:45 CEST 2017 Thread[DRDAConnThread_3,5,main] (DATABASE = ejbtimer), (DRDAID = {582}), Connection refused : java.lang.OutOfMemoryError

**
 ERROR XJ001: Java exception: 'java.lang.OutOfMemoryError: Java heap space'.
        at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
        at org.apache.derby.iapi.error.StandardException.plainWrapException(Unknown Source)
        at org.apache.derby.impl.store.raw.RawStore.backupAndEnableLogArchiveMode(Unknown Source)
        at org.apache.derby.impl.store.access.RAMAccessManager.backupAndEnableLogArchiveMode(Unknown Source)
        at org.apache.derby.impl.db.BasicDatabase.backupAndEnableLogArchiveMode(Unknown Source)
        at org.apache.derby.catalog.SystemProcedures.SYSCS_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE(Unknown Source)
        at org.apache.derby.exe.ac2a2882b5x015dx2354xca08xffff89e47f210.g0(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
        at java.lang.reflect.Method.invoke(Method.java:508)
        at org.apache.derby.impl.services.reflect.ReflectMethod.invoke(Unknown Source)
        at org.apache.derby.impl.sql.execute.CallStatementResultSet.open(Unknown Source)
        at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown Source)
        at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedCallableStatement.executeStatement(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedPreparedStatement.execute(Unknown Source)
        at org.apache.derby.impl.drda.DRDAStatement.execute(Unknown Source)
        at org.apache.derby.impl.drda.DRDAConnThread.parseEXCSQLSTT(Unknown Source)
        at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
        at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)
Caused by: java.lang.OutOfMemoryError: Java heap space
        at org.apache.derby.impl.store.raw.data.CachedPage.setPageArray(Unknown Source)
        at org.apache.derby.impl.store.raw.data.CachedPage.readPage(Unknown Source)
        at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(Unknown Source)
        at org.apache.derby.impl.services.cache.ConcurrentCache.find(Unknown Source)
        at org.apache.derby.impl.store.raw.data.FileContainer.getLatchedPage(Unknown Source)
        at org.apache.derby.impl.store.raw.data.RAFContainer.backupContainer(Unknown Source)
        at org.apache.derby.impl.store.raw.data.BaseContainerHandle.backupContainer(Unknown Source)
        at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.backupDataFiles(Unknown Source)
        at org.apache.derby.impl.store.raw.RawStore.backup(Unknown Source)
        at org.apache.derby.impl.store.raw.RawStore.backup(Unknown Source)

Regards,
p.




On 7 July 2017 at 04:59, Bryan Pendleton <[hidden email]> wrote:


how does derby.storage.pageCacheSize parameter (https://db.apache.org/derby/docs/10.13/ref/rrefproper81359.html) work with database that has multiple page sizes--tables with default 4096 bytes and tables with long/blob of 32768 byte pages?


Hi Peter,

I'm not 100% sure how this works; I think you should run some experiments.

Here's what I *think* the behavior is:
1) The page cache is sized in 4K pages, so if you set pageCacheSize=1024, you get 4 meg of page cache memory
2) Tables with 4K pages simply use the cache as you expect. 
3) Tables with 32K pages chew up 8 cache "pages" at a time, each 32K chunk of page cache holding 1 32K page from that large-page table.

Some stuff I found while searching around, which might give you some ideas for experiments you could run:


and


Sorry I'm not of much more help here.

bryan




--
Peter Ondruška

kaibo, s.r.o., ID 28435036, registered with the commercial register administered by the Municipal Court in Prague, section C, insert 141269.
Registered office and postal address: kaibo, s.r.o., Kališnická 379/10, Prague 3, 130 00, Czech Republic.
https://www.kaibo.eu
Loading...