Project

General

Profile

Actions

Feature #606

closed

Feature #751: BugZilla migration

Stress Test Bug Tracker Instance

Added by Joel Madero over 9 years ago. Updated about 9 years ago.

Status:
Closed
Priority:
Normal
Category:
QA
Target version:
-
Start date:
2014-01-05
Due date:
2015-01-17
% Done:

0%

Tags:

Description

I think next step for migration is to stress test. Making this a sub issue.

@Robinson - please add other sub issues that kind of roadmap what will be done after we successfully stress test. Also - for hardware, can you and Cloph talk a bit about if our hardware can handle bugzilla instance, if not, we need to resolve that quickly (ie. board needs to investigate options and purchase equipment)

Actions #1

Updated by Bjoern Michaelsen over 9 years ago

  • Project changed from 49 to Infrastructure

moving to infrastructure project as per https://redmine.documentfoundation.org/issues/599#note-2

Actions #2

Updated by Bjoern Michaelsen over 9 years ago

  • Category set to QA
Actions #3

Updated by Bjoern Michaelsen over 9 years ago

  • Due date set to 2014-10-31

see 591 for rationale

Actions #4

Updated by Alexander Werner over 9 years ago

regarding hardware: The virtualization platform every service will be migrated to will be powerful enough to handle everything. Having the board purchase additional hardware would neglect its use.

Actions #5

Updated by Florian Effenberger over 9 years ago

  • Assignee set to Robinson Tryon
Actions #6

Updated by Florian Effenberger over 9 years ago

Wrt. hardware, with the new servers we have in place, I see no issues for the moment
How is the stress testing going? Any concrete results so far? How and what are you testing?

Actions #7

Updated by Florian Effenberger over 9 years ago

  • Priority changed from Normal to Urgent
Actions #8

Updated by Robinson Tryon over 9 years ago

With the people at the Seattle LibreFest, I think we can accomplish a pretty good stress-test of the instance w.r.t.
  • Account creation
  • Bug creation
  • Querying, and
  • Updates

My initial plan was to supplement the attendees with other people, but now that our attendance list has grown to over 25, I'm pretty confident that we can do all of the testing with the people right there. This will give us even better fine-grained control over what kind of a load we throw at the machine :-)

Actions #9

Updated by Robinson Tryon over 9 years ago

Robinson Tryon wrote:

With the people at the Seattle LibreFest, I think we can accomplish a pretty good stress-test of the instance w.r.t.
  • Account creation
  • Bug creation
  • Querying, and
  • Updates

My initial plan was to supplement the attendees with other people, but now that our attendance list has grown to over 25, I'm pretty confident that we can do all of the testing with the people right there. This will give us even better fine-grained control over what kind of a load we throw at the machine :-)

Turn-out at the LibreFest was excellent. I had some initial results of people using our test install, and attendees were most interested in how they could continue to contribute after the end of the event. After talking with Norbert about the best ways for us to schedule the migration and reloading the data he has acquired (the existing dumps of the FDO data), I'll be scheduling additional testing against our Bugzilla VM this week.

Actions #10

Updated by Florian Effenberger over 9 years ago

  • Parent task changed from #591 to #751
Actions #11

Updated by Robinson Tryon over 9 years ago

  • Status changed from New to In Progress
Next stress-test will take place once we've been given the Bugzilla 4.4.5 dump by Tollef. We'll test the same actions being performed in Bugzilla, including:
  • New account creation
  • Updating bugs
  • Bulk update
  • Heavy querying
Actions #12

Updated by Florian Effenberger over 9 years ago

  • Due date deleted (2014-10-31)
  • Priority changed from Urgent to Normal

There's a couple of other things that need doing first, so lowering priority on this one and removing deadline as concrete ETA will be determined when other tasks are dealt with

Actions #13

Updated by Robinson Tryon over 9 years ago

  • Start date set to 2014-01-05

Adding date a few weeks before the migration; we should have the updated db dump from Tollef by this point, so we can double-check that the import runs correctly and confirm that our VM allocations can handle a typical load

Actions #14

Updated by Florian Effenberger about 9 years ago

Keep in mind Tollef will be on vacation early January IIRC, so this should be dealt with soon

Actions #15

Updated by Robinson Tryon about 9 years ago

Florian Effenberger wrote:

Keep in mind Tollef will be on vacation early January IIRC, so this should be dealt with soon

Yes -- Tollef has been pinged again. I'll make sure to get the dump before he leaves :-)

Actions #16

Updated by Florian Effenberger about 9 years ago

  • Due date set to 2015-01-17

Should happen within the current week using the recent BZ VM dump, so we have time to react until migration day if needed

Actions #17

Updated by Florian Effenberger about 9 years ago

What's the status?

Actions #18

Updated by Robinson Tryon about 9 years ago

Florian Effenberger wrote:

What's the status?

Stress tests revealed some timeouts under heavy load, so I increased the memory for postgres and increased timeouts for nginx and FastCGI.

Did some attachment-upload tests as well today (4 people uploading 10 x 9MB files each, back to back). No timeouts, no particular speed issues with the site. Also re-ran query tests, and had no further issues with timeouts. Max load was 1.05 (with 16 cores allocated to the VM)

Actions #19

Updated by Robinson Tryon about 9 years ago

  • Status changed from In Progress to Closed

Done here

Actions

Also available in: Atom PDF