ext5

[Anonymous Remailer (austria)]

will there be ext5? ext4 works just fine so far. but it could be even more faster. otherwise i have to jump to btrfs (when it's done).
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[t***@mit.edu]

Post by Anonymous Remailer (austria)
will there be ext5? ext4 works just fine so far. but it could be
even more faster. otherwise i have to jump to btrfs (when it's
done).

We currently don't have any plans for an "ext5". There might be some
new features that might gradually trickle into ext4; for example
there's someone who I may be mentoring who is interested in working on
an idea I've had to add read-only compression to ext4. (Actually, the
design I've sketched out makes 90% of the work be file system
independent, so it's something that could be retrofitted into other
filesystems: xfs, btrfs, etc.)

The benchmarks I've seen don't show that btrfs is that much faster;
for some workloads its faster, for others its slower. Of course,
there may be some file system tuning that still remains to be done,
both for btrfs and ext4, that may change the performance numbers
slightly.

The main reason why I suspect people will be interested in btrfs is to
because of some of its features (i.e., file system level snapshots,
data level checksums, etc.) that are unlikely to show up in ext4. I
also suspect that btrfs will take a while to mature, as all file
systems do. ZFS for example took good five years to development, and
perhaps another 3-4 before people started really trusting it for
critical production uses.

Regards,

- Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Manish Katiyar]

On Wed, Feb 10, 2010 at 12:40:05AM +0100, Anonymous Remailer (austria=

Post by Anonymous Remailer (austria)
will there be ext5? ext4 works just fine so far. but it could be
even more faster. otherwise i have to jump to btrfs (when it's
done).

We currently don't have any plans for an "ext5". =A0There might be so=

me

new features that might gradually trickle into ext4; for example
there's someone who I may be mentoring who is interested in working o=

n

an idea I've had to add read-only compression to ext4. =A0(Actually, =

the

design I've sketched out makes 90% of the work be file system
independent,

Hi Ted,

Is this design somewhere on net so that we can read/lookup it up ?

Thanks -
Manish

so it's something that could be retrofitted into other
filesystems: xfs, btrfs, etc.)
The benchmarks I've seen don't show that btrfs is that much faster;
for some workloads its faster, for others its slower. =A0Of course,
there may be some file system tuning that still remains to be done,
both for btrfs and ext4, that may change the performance numbers
slightly.
The main reason why I suspect people will be interested in btrfs is t=

o

because of some of its features (i.e., file system level snapshots,
data level checksums, etc.) that are unlikely to show up in ext4. =A0=

I

also suspect that btrfs will take a while to mature, as all file
systems do. =A0ZFS for example took good five years to development, a=

nd

perhaps another 3-4 before people started really trusting it for
critical production uses.
Regards,
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=

=A0 =A0 =A0 =A0 =A0 =A0- Ted

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4"=

in

More majordomo info at =A0http://vger.kernel.org/majordomo-info.html

--=20
Thanks -
Manish
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
[$\*.^ -- I miss being one of them
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" i=
n
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[t***@mit.edu]

Post by Manish Katiyar
Is this design somewhere on net so that we can read/lookup it up ?

Not yet, but the basic idea is to do the compression in userspace,
using libz with regular resync points every 64k or 128k of
uncompressed data. An array, indexed by each fixed-block of
uncompressed data, is located at the beginning of the file indicating
where each compressed block begins. The file is stored on-disk
written by the installer in a compressed format, and then the
installer flips an attribute bit which marks the file as containing
compressed data, and which also makes the file immutable. (Any
attempt to open the file read/write will result in an error.)

The advantage of this scheme is that it minimizes in-kernel
complexity, since the kernel doesn't have to deal with compressing
data, and it means we can use a simpler format which is more efficient
from a compression standpoint since we don't have to deal with random
access writes.

Another advantage of this scheme is that it's relatively easy to do
the bulk of the work in a file system-independent layer. Some minor
work would be needed to interface the attribute bit and the mmap
reading function into each client filesystem, but it should be
possible to isolate 95% of the work into a filesystem independent
layer, and then make this something which multiple file systems could
take advantage of --- not just ext3 and ext4, but also xfs, btrfs,
etc.

- Ted


--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Jan Kara]

Post by t***@mit.edu

Post by Manish Katiyar
Is this design somewhere on net so that we can read/lookup it up ?

Not yet, but the basic idea is to do the compression in userspace,
using libz with regular resync points every 64k or 128k of
uncompressed data. An array, indexed by each fixed-block of
uncompressed data, is located at the beginning of the file indicating
where each compressed block begins. The file is stored on-disk
written by the installer in a compressed format, and then the
installer flips an attribute bit which marks the file as containing
compressed data, and which also makes the file immutable. (Any
attempt to open the file read/write will result in an error.)

From the first reading, this sounds like what zisofs is doing.
The reading part is already implemented in kernel in fs/isofs/compress.c
so that might be lifted to a generic level and used for ext4 as well..

Honza

[t***@mit.edu]

Post by Jan Kara
From the first reading, this sounds like what zisofs is doing.
The reading part is already implemented in kernel in fs/isofs/compress.c
so that might be lifted to a generic level and used for ext4 as well..

I didn't know about zisofs; thanks for pointing that out! I'll
mention it to the student I'm helping out....

- Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Ron Johnson]

Post by t***@mit.edu

Post by Anonymous Remailer (austria)
will there be ext5? ext4 works just fine so far. but it could be
even more faster. otherwise i have to jump to btrfs (when it's
done).

We currently don't have any plans for an "ext5". There might be some
new features that might gradually trickle into ext4; for example
there's someone who I may be mentoring who is interested in working on
an idea I've had to add read-only compression to ext4. (Actually, the
design I've sketched out makes 90% of the work be file system
independent, so it's something that could be retrofitted into other
filesystems: xfs, btrfs, etc.)

I guess that means every file on the fs?

Windows-like per-file compression would be darned useful in certain
circumstances. Big mbox files, for example.

[t***@mit.edu]

Post by Ron Johnson

Post by t***@mit.edu
We currently don't have any plans for an "ext5". There might be some
new features that might gradually trickle into ext4; for example
there's someone who I may be mentoring who is interested in working on
an idea I've had to add read-only compression to ext4. (Actually, the
design I've sketched out makes 90% of the work be file system
independent, so it's something that could be retrofitted into other
filesystems: xfs, btrfs, etc.)

I guess that means every file on the fs?

No, I mean per-file compression, but a compressed file is immutable.
This is basically what Mac OS X has recently added, and while I
haven't looked at their implementation, Apple being one of those
closed source companies and all, I wouldn't be surprised if they did
things the same way.

Post by Ron Johnson
Windows-like per-file compression would be darned useful in certain
circumstances. Big mbox files, for example.

The problem with mbox files is that some mail readers try to smart
about how they modify them to avoid needing to rewrite the whole mbox
file; mutt will seak to the middle of the file, write to the end of
the file, and then trim off any excess space by using the truncate
system call. This is *hard* to support if the mbox file is
compressed; you can do it using a stacker-style compression technique,
but it's not as efficient, and it has a lot of complexity in the
kernel.

The idea with read-only compressed files is that they are useful for
large executables or large static files, where compressing them means
that it takes less time to read them off of an HDD.

- Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Ron Johnson]

Post by t***@mit.edu

Post by Ron Johnson

Post by t***@mit.edu
We currently don't have any plans for an "ext5". There might be some
new features that might gradually trickle into ext4; for example
there's someone who I may be mentoring who is interested in working on
an idea I've had to add read-only compression to ext4. (Actually, the
design I've sketched out makes 90% of the work be file system
independent, so it's something that could be retrofitted into other
filesystems: xfs, btrfs, etc.)

I guess that means every file on the fs?

No, I mean per-file compression, but a compressed file is immutable.
This is basically what Mac OS X has recently added, and while I
haven't looked at their implementation, Apple being one of those
closed source companies and all, I wouldn't be surprised if they did
things the same way.

Post by Ron Johnson
Windows-like per-file compression would be darned useful in certain
circumstances. Big mbox files, for example.

The problem with mbox files is that some mail readers try to smart
about how they modify them to avoid needing to rewrite the whole mbox
file; mutt will seak to the middle of the file, write to the end of
the file, and then trim off any excess space by using the truncate
system call. This is *hard* to support if the mbox file is
compressed; you can do it using a stacker-style compression technique,
but it's not as efficient, and it has a lot of complexity in the
kernel.

I guess that's how Windows does it?

Post by t***@mit.edu
The idea with read-only compressed files is that they are useful for
large executables or large static files, where compressing them means
that it takes less time to read them off of an HDD.

Sure. Anything is better than nothing!

[Goswin von Brederlow]

Post by t***@mit.edu

Post by Ron Johnson

Post by t***@mit.edu
We currently don't have any plans for an "ext5". There might be some
new features that might gradually trickle into ext4; for example
there's someone who I may be mentoring who is interested in working on
an idea I've had to add read-only compression to ext4. (Actually, the
design I've sketched out makes 90% of the work be file system
independent, so it's something that could be retrofitted into other
filesystems: xfs, btrfs, etc.)

I guess that means every file on the fs?

No, I mean per-file compression, but a compressed file is immutable.
This is basically what Mac OS X has recently added, and while I
haven't looked at their implementation, Apple being one of those
closed source companies and all, I wouldn't be surprised if they did
things the same way.

Do that with fuse or like ecryptfs. That is nothing that needs to be
inside any FS and is far better to have as layer on top of the fs of
your choice.

Post by t***@mit.edu

Post by Ron Johnson
Windows-like per-file compression would be darned useful in certain
circumstances. Big mbox files, for example.

The problem with mbox files is that some mail readers try to smart
about how they modify them to avoid needing to rewrite the whole mbox
file; mutt will seak to the middle of the file, write to the end of
the file, and then trim off any excess space by using the truncate
system call. This is *hard* to support if the mbox file is
compressed; you can do it using a stacker-style compression technique,
but it's not as efficient, and it has a lot of complexity in the
kernel.

You need a block based compression for that so that you can recompress
and replace blocks in any order. It also helps if you can have the
compressed blocks in random order so that you can put the block at the
end and zero fill the old location when it grows during recompression.

Otherwise you will be moving the compressed data around a lot to
accomodate changes in the middle of a file.

Post by t***@mit.edu
The idea with read-only compressed files is that they are useful for
large executables or large static files, where compressing them means
that it takes less time to read them off of an HDD.

Or when you only have so much flash.

MfG
Goswin
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Ron Johnson]

Post by Goswin von Brederlow

Post by t***@mit.edu
The idea with read-only compressed files is that they are useful for
large executables or large static files, where compressing them means
that it takes less time to read them off of an HDD.

Or when you only have so much flash.

Isn't that what squashfs is for?

[t***@mit.edu]

Post by Ron Johnson

Post by Goswin von Brederlow

Post by t***@mit.edu
The idea with read-only compressed files is that they are useful for
large executables or large static files, where compressing them means
that it takes less time to read them off of an HDD.

Or when you only have so much flash.

Isn't that what squashfs is for?

The problem with the squashfs, fuse, ecryptfs approaches for the use
case that I am envisioning is that it's an all-or-nothing sort of
thing. You may not want to encrypt all of the files in a file system.
Sure, you can play games with bind mounts, and/or accept the
performance hit of passing everything through fuse even for files that
aren't encrypted, but I think that's going to significantly inhibit
adoption of the technology.

Something which allows compressed and uncompressed files to co-exist
without any performance hits to the uncompressed files, and which
allows for a gradual transition after you upgrade rpm/dpkg and as
packages get upgraded is going to much easier time with adoption
rates.

- Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html