Task #1904
open
replace xz with different compression tool to compress database dumps (xz is single-threaded, takes too long) - evaluate parallel compression
Added by Christian Lohmaier almost 8 years ago.
Updated almost 8 years ago.
Description
creating database dumps takes way too long with xz, better switch to different compression, or even to a tool that can use multiple cores..
We could use lbzip2 or pbzip2, they both use the number of cores available (or take a parameter to limit.
In order not to change the output files we could do:
tar cf - paths-to-archive | lbzip2 -9 -j 32 > archive.tar.gz
If the scripts are in SALT, I will be happy to change and test it.
Just saw "tar" has an option to do that
tar -c --use-compress-program=pigz -f tar.file dir_to_xz
- Subject changed from replace xz with different compression tool to compress database dumps (xz is single-threaded, takes too long) to replace xz with different compression tool to compress database dumps (xz is single-threaded, takes too long) - evaluate parallel compression
- Priority changed from Normal to Low
- % Done changed from 0 to 80
changed to use plain gzip in salt - leaving with low prio to maybe test out the parallel compression variants at a later time
Also available in: Atom
PDF