Very, very large .dat files

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Very, very large .dat files

Peter Hansson

We've seen .dat files  in seg0 directory grow to several hundreds of
gigabytes. While everything works ok such a file becomes unmanageable
from an OS point of view. Is there a way to control when Derby starts a
new conglomerate?  ... so that there are more .dat files but each of
them of less size ?

Thanks.

Reply | Threaded
Open this post in threaded view
|

Re: Very, very large .dat files

Peter Hansson

I guess the question is if Apache Derby supports table partitioning (the
term used in Oracle RDBMS).  I understand it doesn't.

This means that if I have an incredible large Derby table then that will
mean an incredible large single .dat file too. (a table is always stored
in a single .dat file, right?)

Any suggestions? Any props that may control this?


On 05-Dec-17 13:28, Peter Hansson wrote:
>
> We've seen .dat files  in seg0 directory grow to several hundreds of
> gigabytes. While everything works ok such a file becomes unmanageable
> from an OS point of view. Is there a way to control when Derby starts
> a new conglomerate?  ... so that there are more .dat files but each of
> them of less size ?
>
> Thanks.
>

Reply | Threaded
Open this post in threaded view
|

Re: Very, very large .dat files

Bryan Pendleton-3
You are correct, Derby does not provide table partitioning features
such as those provided by Oracle.

And you are correct, a single Derby table is a single .dat file.

But modern filesystems handle very large files without problems. Is
there a particular reason that you think a very large file will be a
problem for you? Or that you think a larger number of smaller .dat
files would have some benefit?

I believe there are a variety of library/framework approaches to
splitting up a logical table into multiple physical tables. People
often do this with data that is oriented around "time", and becomes of
less interest as it ages. I've seen applications which, e.g., create a
new table each day, named appropriately, and "rotate" these tables
over time so that the old data ages out and is dropped (by dropping
the older no-longer-wanted tables).

Careful use of CREATE VIEW can shield your application from most of
the impact of such techniques.

thanks,

bryan
Reply | Threaded
Open this post in threaded view
|

Re: Very, very large .dat files

Peter Hansson

Thank for taking time to answer. Appreciated.

You basically confirmed my understanding: I need to go for a DIU
approach if I want table partitioning.

On 05-Dec-17 17:30, Bryan Pendleton wrote:

> You are correct, Derby does not provide table partitioning features
> such as those provided by Oracle.
>
> And you are correct, a single Derby table is a single .dat file.
>
> But modern filesystems handle very large files without problems. Is
> there a particular reason that you think a very large file will be a
> problem for you? Or that you think a larger number of smaller .dat
> files would have some benefit?
>
> I believe there are a variety of library/framework approaches to
> splitting up a logical table into multiple physical tables. People
> often do this with data that is oriented around "time", and becomes of
> less interest as it ages. I've seen applications which, e.g., create a
> new table each day, named appropriately, and "rotate" these tables
> over time so that the old data ages out and is dropped (by dropping
> the older no-longer-wanted tables).
>
> Careful use of CREATE VIEW can shield your application from most of
> the impact of such techniques.
>
> thanks,
>
> bryan