https://redmine.documentfoundation.org/https://redmine.documentfoundation.org/favicon.ico?16960560022017-01-24T01:40:22ZThe Document Foundation RedmineInfrastructure - Task #2145: Frequent 504 Gateway timeouts in Bugzilla upon bug changehttps://redmine.documentfoundation.org/issues/2145?journal_id=115112017-01-24T01:40:22ZAron Budeaaron.budea@gmail.com
<ul></ul><p>It's also coupled with extreme Bugzilla slowness :/.</p> Infrastructure - Task #2145: Frequent 504 Gateway timeouts in Bugzilla upon bug changehttps://redmine.documentfoundation.org/issues/2145?journal_id=115132017-01-24T10:16:15ZFlorian Effenbergerfloeff@documentfoundation.org
<ul><li><strong>Assignee</strong> set to <i>Guilhem Moulin</i></li><li><strong>Target version</strong> set to <i>Pool</i></li></ul><p>Guilhem, can you have a look?</p> Infrastructure - Task #2145: Frequent 504 Gateway timeouts in Bugzilla upon bug changehttps://redmine.documentfoundation.org/issues/2145?journal_id=115162017-01-24T10:39:51ZGuilhem Moulinguilhem@libreoffice.org
<ul><li><strong>Assignee</strong> deleted (<del><i>Guilhem Moulin</i></del>)</li><li><strong>Target version</strong> deleted (<del><i>Pool</i></del>)</li></ul><p>Yes I'm not it. I restarted PostgreSQL with some tweaks; it's usually ~instant, but now it seems insanely slow at starting up… bugs.tdf has been down for 20mins already :-( :-(</p> Infrastructure - Task #2145: Frequent 504 Gateway timeouts in Bugzilla upon bug changehttps://redmine.documentfoundation.org/issues/2145?journal_id=115182017-01-24T10:57:53ZGuilhem Moulinguilhem@libreoffice.org
<ul><li><strong>Assignee</strong> set to <i>Guilhem Moulin</i></li></ul><p>Sorry, I didn't see I removed myself from the Assignee, I guess it's because I had the page loaded before I got your message.</p>
<p>And of course I meant "I <strong>on</strong> it", sorry for the confusion</p> Infrastructure - Task #2145: Frequent 504 Gateway timeouts in Bugzilla upon bug changehttps://redmine.documentfoundation.org/issues/2145?journal_id=115192017-01-24T11:51:43ZGuilhem Moulinguilhem@libreoffice.org
<ul></ul><p>Restarting PostgreSQL and forcing VACUUM seem to have significant improvements on auto completion. I also tweaked the config (which was the reason of the restart to start with), which should improve write queries.</p>
<p>I leave the bug priority on High in the meantime as it's not a proper fix, though; I'll keep investigating.</p> Infrastructure - Task #2145: Frequent 504 Gateway timeouts in Bugzilla upon bug changehttps://redmine.documentfoundation.org/issues/2145?journal_id=117152017-02-20T13:12:03ZFlorian Effenbergerfloeff@documentfoundation.org
<ul><li><strong>Target version</strong> set to <i>Pool</i></li></ul><p>Any update? Have the problems been solved?</p> Infrastructure - Task #2145: Frequent 504 Gateway timeouts in Bugzilla upon bug changehttps://redmine.documentfoundation.org/issues/2145?journal_id=117452017-02-20T13:41:44ZBeluga Beluga
<ul></ul><p>Florian Effenberger wrote:</p>
<blockquote>
<p>Any update? Have the problems been solved?</p>
</blockquote>
<p>More investigation is needed as the problem has reappeared several times after this was filed. It should be noted that these have always appeared with our self-hosted BZ. Not sure, if the cause has always been the same.</p> Infrastructure - Task #2145: Frequent 504 Gateway timeouts in Bugzilla upon bug changehttps://redmine.documentfoundation.org/issues/2145?journal_id=117642017-02-21T13:53:58ZFlorian Effenbergerfloeff@documentfoundation.org
<ul></ul><blockquote>
<p>More investigation is needed as the problem has reappeared several times<br />after this was filed. It should be noted that these have always appeared<br />with our self-hosted BZ. Not sure, if the cause has always been the same.</p>
</blockquote>
<p>Do you have any timestamps, so we could look into the logs?</p> Infrastructure - Task #2145: Frequent 504 Gateway timeouts in Bugzilla upon bug changehttps://redmine.documentfoundation.org/issues/2145?journal_id=120652017-04-11T11:26:49ZFlorian Effenbergerfloeff@documentfoundation.org
<ul></ul><p>Any updates, or can we close this ticket?</p> Infrastructure - Task #2145: Frequent 504 Gateway timeouts in Bugzilla upon bug changehttps://redmine.documentfoundation.org/issues/2145?journal_id=121212017-04-18T11:28:36ZFlorian Effenbergerfloeff@documentfoundation.org
<ul></ul><p>Florian Effenberger wrote:</p>
<blockquote>
<p>Any updates, or can we close this ticket?</p>
</blockquote>
<p>Ping?</p> Infrastructure - Task #2145: Frequent 504 Gateway timeouts in Bugzilla upon bug changehttps://redmine.documentfoundation.org/issues/2145?journal_id=121252017-04-19T18:54:38ZGuilhem Moulinguilhem@libreoffice.org
<ul></ul><p>I'm still doing regular manual vacuums for now. I think it's best to keep the ticket open until we find a decent autovacuum configuration.</p> Infrastructure - Task #2145: Frequent 504 Gateway timeouts in Bugzilla upon bug changehttps://redmine.documentfoundation.org/issues/2145?journal_id=128732017-07-18T09:44:26ZFlorian Effenbergerfloeff@documentfoundation.org
<ul></ul><p>Guilhem Moulin wrote:</p>
<blockquote>
<p>I'm still doing regular manual vacuums for now. I think it's best to keep the ticket open until we find a decent autovacuum configuration.</p>
</blockquote>
<p>Any updates on the situation?</p> Infrastructure - Task #2145: Frequent 504 Gateway timeouts in Bugzilla upon bug changehttps://redmine.documentfoundation.org/issues/2145?journal_id=131602017-08-29T00:02:21ZGuilhem Moulinguilhem@libreoffice.org
<ul></ul><p>During the past 52 days we've had "only" 40 of these Gateway Time-out, for a total of just under 1.4M requests to the fastcgi server (incl. 64k requests to the REST API). So while we could probably tune PostgreSQL better, I'm now tempted to close this, or at least downgrade the severity.</p>
<p>Moreover 12 of these 40 failed requests came from our own infra (the wiki querying the REST API). Looking at the timestamp they mostly come in batch and I could correlate 2 batches with the following guster heals (dates are UTC):</p>
<pre><code>- 6x on 2017-07-14 from 10:10 to 10:15 [freeze+reboot of charly]<br /> - 9x on 2017-08-03 from 15:30 to 16:00 [corruption of delta volume]</code></pre> Infrastructure - Task #2145: Frequent 504 Gateway timeouts in Bugzilla upon bug changehttps://redmine.documentfoundation.org/issues/2145?journal_id=131692017-08-29T07:30:32ZFlorian Effenbergerfloeff@documentfoundation.org
<ul></ul><blockquote>
<p>During the past 52 days we've had "only" 40 of these Gateway Time-out, <br />for a total of just under 1.4M requests to the fastcgi server (incl. 64k <br />requests to the REST API). So while we could probably tune PostgreSQL <br />better, I'm now tempted to close this, or at least downgrade the severity.</p>
</blockquote>
<p>I heard no complaints either, so how about having a normal priority and <br />Qlater, so we can revisit later the year?</p> Infrastructure - Task #2145: Frequent 504 Gateway timeouts in Bugzilla upon bug changehttps://redmine.documentfoundation.org/issues/2145?journal_id=131722017-08-29T10:28:32ZGuilhem Moulinguilhem@libreoffice.org
<ul><li><strong>Priority</strong> changed from <i>High</i> to <i>Normal</i></li><li><strong>Target version</strong> changed from <i>Pool</i> to <i>Qlater</i></li></ul><p>Florian Effenberger wrote:</p>
<blockquote>
<p>I heard no complaints either, so how about having a normal priority and <br />Qlater, so we can revisit later the year?</p>
</blockquote>
<p>Sure, done.</p> Infrastructure - Task #2145: Frequent 504 Gateway timeouts in Bugzilla upon bug changehttps://redmine.documentfoundation.org/issues/2145?journal_id=131732017-08-29T10:28:44ZGuilhem Moulinguilhem@libreoffice.org
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> Infrastructure - Task #2145: Frequent 504 Gateway timeouts in Bugzilla upon bug changehttps://redmine.documentfoundation.org/issues/2145?journal_id=145742018-03-05T14:26:18ZFlorian Effenbergerfloeff@documentfoundation.org
<ul></ul><p>Any update? Can this be closed?</p> Infrastructure - Task #2145: Frequent 504 Gateway timeouts in Bugzilla upon bug changehttps://redmine.documentfoundation.org/issues/2145?journal_id=145872018-03-06T09:57:54ZGuilhem Moulinguilhem@libreoffice.org
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Closed</i></li></ul><p>Closing indeed. I still see a handful of timeouts in the logs, but was about .0005% of all CGI/REST requests issued during the past 2 months. And we haven't heard any further complaint.</p>