/tech/ - Technology★

Should kill myself.

> time convert wallpaper.png - >/dev/null
convert wallpaper.png - > /dev/null  0.34s user 0.03s system 99% cpu 0.373 total
> time (unxz -ck wallpaper.ppm.xz | convert - - >/dev/null)
( unxz -ck wallpaper.ppm.xz | convert - - > /dev/null; )  0.11s user 0.06s system 113% cpu 0.148 total
> time (unzstd -qck wallpaper.ppm.zst | convert - - >/dev/null)
( unzstd -qck wallpaper.ppm.zst | convert - - > /dev/null; )  0.08s user 0.04s system 108% cpu 0.108 total
> time (bunzip2 -ck wallpaper.ppm.bz2 | convert - - >/dev/null)
( bunzip2 -ck wallpaper.ppm.bz2 | convert - - > /dev/null; )  0.28s user 0.05s system 108% cpu 0.305 total
> time (gunzip -ck wallpaper.ppm.gz | convert - - >/dev/null)
( gunzip -ck wallpaper.ppm.gz | convert - - > /dev/null; )  0.10s user 0.04s system 110% cpu 0.128 total

Use farbfeld FAGGOT

>>980304

Why when netpbm is already suckless and well established?

I've had a few beers OP so maybe I screwed up somewhere but...

All *.ppm.* files are created with -9 using their respective utilities.

% file test*
test.png:     PNG image data, 5019 x 3346, 8-bit/color RGB, non-interlaced
test.ppm:     Netpbm image data, size = 5019 x 3346, rawbits, pixmap
test.ppm.bz2: bzip2 compressed data, block size = 900k
test.ppm.gz:  gzip compressed data, was "test.ppm", last modified: Sun Sep 30 21:42:10 2018, max compression, from Unix, original size 50380739
test.ppm.lz:  lzip compressed data, version: 1
test.ppm.xz:  XZ compressed data

A couple of timings. Similar results.

% time convert test.png - > /dev/null       
convert test.png - > /dev/null  4.36s user 0.07s system 99% cpu 4.432 total
% time (bunzip2 -ck test.ppm.bz2 | convert - - > /dev/null)
( bunzip2 -ck test.ppm.bz2 | convert - - > /dev/null; )  3.05s user 0.12s system 101% cpu 3.122 total
% time (gunzip -ck test.ppm.gz | convert - - > /dev/null)
( gunzip -ck test.ppm.gz | convert - - > /dev/null; )  0.82s user 0.08s system 103% cpu 0.878 total

Size comparison

13.4111%  28.9357M  test.png
14.1818%  30.5985M  test.ppm.bz2
15.2399%  32.8815M  test.ppm.lz
15.2696%  32.9454M  test.ppm.xz
18.5471%  40.0171M  test.ppm.gz
23.3505%  50.3807M  test.ppm
Total: 215.759M

Looks like we're trading off smaller file size for longer decompression time... which is exactly what compression is supposed to do. Maybe I'm missing something though.

>>980304

>farbfeld

>fixed 16bit 4chanel

One job need soap. Other job needs drain cleaner

Suckless solution: shitty, diluted drain cleaner.

But seriously, just use fucking netpbm

>>980308

>>980325

How does it work?

>In Farbfeld, pattern resolution is not done while converting, but while compressing the image. For example, farbfeld always stores the alpha-channel, even if the image doesn’t have alpha-variation. This may sound like a big waste at first, but as soon as you compress an image of this kind, the compression-algorithm (e.g. bz2) recognizes the pattern that every 48 bits the 16 bits store the same information. And the compression-algorithms get better and better at this.

>Same applies to the idea of having 16 bits per channel. It sounds excessive, but if you for instance only have a greyscale image, the R, G and B channels will store the same value, which is recognized by the compression algorithm easily.

>This effectively leads to filesizes you’d normally only reach with paletted images, and in some cases bz2 even beats png’s compression, for instance when you’re dealing with grayscale data, line drawings, decals and even photographs.

Why use 16-Bits-per-channel all the time? Isn’t this a total waste?

>Not when you take compression into account. To make this clearer, assume a paletted image with 5 colors and no transparency. So the data is only a set of regular chunks (color1, …, color5) in a certain order. Compression algorithms have been designed to recognize those chunks and can even look at how these chunks interact.

>Local tests have shown that farbfeld easily beats paletted PNG-images. Try for yourself and look at the bzipped results! There is no need for special grayscale, palette, RGB, 1-, 2-, 4-, 8-, 16-Bit subformats. Just use 16-Bit RGBA all the time and let compression take care of the rest.

Which compression should I use?

>bzip2 is recommended, which is widely available (anybody has it) and gives good results. As time will move forward and new algorithms hit the market, this recommendation might be rethought.

What about NetPBM?

>NetPBM is considered to be the most simple format around, however, there’s much room for improvement. In fact, it doesn’t help that the format is subdivided into Portable BitMaps, Portable GrayMaps and Portable PixMaps. It’s not helpful when a manpage can’t give a simple overview of a format in a few sentences.

>NetPBM’s big vice is that it has originally been developed to be hand-written and passed around as plain text. A binary format exists, but still handling optional comments in the header, base 10 ASCII width and height values, arbitrary whitespace inside the data and out-of-band image size and color depth is too painful for the sane user.

>Judging from the usage of the format considering how long it’s been around, it’s no surprise it never really took off. Additionally, functionality like alpha channels and 16-Bit color depth can only be achieved via extensions. Due to it being a textual format it also lacks the desired compression characteristics.

>The question you have to ask yourself is: Can I read in a format without consulting the manpages? If your answer is yes, then the format is simple enough. In this regard, NetPBM can be considered to be a failed format.

I think I'll try on a more diverse corpus to see how it works. I'll PAM because it supports alpha but TGA with -compress none could work too. Most of my tests still show a decrease in size.

For the fun of it, here's what real manly compression looks like (for manly men with a lot of free time, though):

> du -h test.*
1.1M    test.paq8px167
2.4M    test.png
4.5M    test.ppm
2.2M    test.ppm.bz2
3.2M    test.ppm.gz
2.4M    test.ppm.xz
2.9M    test.ppm.zst

>>980389

>>NetPBM is considered to be the most simple format around, however, there’s much room for improvement. In fact, it doesn’t help that the format is subdivided into Portable BitMaps, Portable GrayMaps and Portable PixMaps. It’s not helpful when a manpage can’t give a simple overview of a format in a few sentences.

But it can. Did you even try man pgm or pbm? The real "problem" with netpbm is the mess with PNM and PAM.

>NetPBM’s big vice is that it has originally been developed to be hand-written and passed around as plain text. A binary format exists, but still handling optional comments in the header, base 10 ASCII width and height values, arbitrary whitespace inside the data and out-of-band image size and color depth is too painful for the sane user.

Some subtle FUD in here. There are no whitespaces in the data for the binary formats.

>Judging from the usage of the format considering how long it’s been around, it’s no surprise it never really took off. Additionally, functionality like alpha channels and 16-Bit color depth can only be achieved via extensions.

PAM isn't an extension.

>Due to it being a textual format it also lacks the desired compression characteristics.

Only the header is textual in the binary format, and the header is a fraction of the total size.

>The question you have to ask yourself is: Can I read in a format without consulting the manpages? If your answer is yes, then the format is simple enough. In this regard, NetPBM can be considered to be a failed format.

The manpages are very clear.

This failure cares about size but makes alpha mandatory? What a fucking joke. And no bitmap/graymap format, just a replacement for PAM.

>>980567

Sounds like a lot of complexity to me anon. I'm going to stick with a nice simple format instead.

>>980291 (OP)

try FLIF.

https://flif.info/

>>980581

When the format to be known as JPEG was being standardized, Arithmetic Encoding was a hot thing. IBM owned it, though. The hooks for unofficial AE support was there for research purposes. JPEG2000 uses it, but no-one cares.

>>980575

Does flif and farbfeld do the same thing?

They're literally 1:1 when it comes to most files i've tested

Here's a bigger sample, most methods are still ahead. Zpaq is for the fun of it; it's too slow and the tool doesn't support outputing to stdout. As expected, probabilistic model methods work better than dictionary ones.

> du -h * | sort -h
289M    zpaq_4
331M    zpaq_3
333M    7z_ppmd_9
342M    bzip2_9
344M    7z_lzma_9
344M    xz_9
363M    png
451M    zstd_19
468M    gzip_9
1.6G    tiff

I used tiff because imagemagick and graphicsmagick really don't like TGA and it's lack of magic header.