Project

General

Profile

Task #1904

replace xz with different compression tool to compress database dumps (xz is single-threaded, takes too long) - evaluate parallel compression

Added by Christian Lohmaier over 1 year ago. Updated over 1 year ago.

Status:
New
Priority:
Low
Assignee:
-
Category:
-
Target version:
-
Start date:
Due date:
% Done:

80%

Estimated time:
Tags:
URL:

Description

creating database dumps takes way too long with xz, better switch to different compression, or even to a tool that can use multiple cores..

History

#1 Updated by Jan Iversen over 1 year ago

We could use lbzip2 or pbzip2, they both use the number of cores available (or take a parameter to limit.

In order not to change the output files we could do:
tar cf - paths-to-archive | lbzip2 -9 -j 32 > archive.tar.gz

If the scripts are in SALT, I will be happy to change and test it.

Just saw "tar" has an option to do that
tar -c --use-compress-program=pigz -f tar.file dir_to_xz

#2 Updated by Christian Lohmaier over 1 year ago

  • Subject changed from replace xz with different compression tool to compress database dumps (xz is single-threaded, takes too long) to replace xz with different compression tool to compress database dumps (xz is single-threaded, takes too long) - evaluate parallel compression
  • Priority changed from Normal to Low
  • % Done changed from 0 to 80

changed to use plain gzip in salt - leaving with low prio to maybe test out the parallel compression variants at a later time

Also available in: Atom PDF