Task #1904
open
replace xz with different compression tool to compress database dumps (xz is single-threaded, takes too long) - evaluate parallel compression
Added by Christian Lohmaier almost 10 years ago.
Updated almost 10 years ago.
Description
creating database dumps takes way too long with xz, better switch to different compression, or even to a tool that can use multiple cores..
We could use lbzip2 or pbzip2, they both use the number of cores available (or take a parameter to limit.
In order not to change the output files we could do:
tar cf - paths-to-archive | lbzip2 -9 -j 32 > archive.tar.gz
If the scripts are in SALT, I will be happy to change and test it.
Just saw "tar" has an option to do that
tar -c --use-compress-program=pigz -f tar.file dir_to_xz
- Subject changed from replace xz with different compression tool to compress database dumps (xz is single-threaded, takes too long) to replace xz with different compression tool to compress database dumps (xz is single-threaded, takes too long) - evaluate parallel compression
- Priority changed from Normal to Low
- % Done changed from 0 to 80
changed to use plain gzip in salt - leaving with low prio to maybe test out the parallel compression variants at a later time
Also available in: Atom
PDF
