Project

General

Profile

Actions

Task #3518

closed

Windows crashreports at crash.libreoffice.org have useless backtraces (breakpad update needed?)

Added by Luboš Luňák about 3 years ago. Updated over 2 years ago.

Status:
Closed
Priority:
Low
Category:
-
Target version:
Team - Q3/2021
Start date:
Due date:
% Done:

70%

Tags:

Description

I have recently analyzed recent reports at crashreport.libreoffice.org and it appears that the vast majority of recent Windows crashreports have useless backtraces, with many frames not resolved to function names or the backtraces having only few frames.

I'm not sure what's caused this, but random observations:
- Linux backtraces seem to work fine (https://crashreport.libreoffice.org/stats/crash_details/c249c48f-5919-425b-bd9d-bb5dc1a34680)
- Backtraces involving Skia (built with Clang) are fine for frames in Skia (https://crashreport.libreoffice.org/stats/crash_details/e1b81d2a-1358-486f-9d81-4f8303e5be0c)
- 32bit (x86) backtraces appear to be fine (https://crashreport.libreoffice.org/stats/crash_details/473f460a-a782-413b-96e1-3e00bd7bfed0), although not all of them (https://crashreport.libreoffice.org/stats/crash_details/b2c17b07-8b32-469b-aedd-61bf13e095fc)
- Reports that are 1 year old appear to be broken (https://crashreport.libreoffice.org/stats/crash_details/6184857d-80dc-4964-bfb5-7e0cd84f40ad)
- Reports that are 2 years old appear to be fine (https://crashreport.libreoffice.org/stats/crash_details/7ad871cb-e187-44eb-bb5a-05a9cc792b0d)

Note that those URLs are just examples, I used psql queries to gather 10k+ crashreports from the most recent LO release from this year and practically all of them are useless.

Our snapshot of the breakpad library is from 2015, so one possible explanation is that this old snapshot cannot properly handle backtraces from recent MSVC 64bit builds. I have a Gerrit change with a recent breakpad snapshot ready, but I expect that the server side on crashreport.libreoffice.org needs to be updated first.

Can somebody please look into this?

Actions #1

Updated by Luboš Luňák about 3 years ago

Note that the pending Gerrit change is https://gerrit.libreoffice.org/c/core/+/113750 . The updated Breakpad snapshot for it is at https://dev-www.libreoffice.org/src/breakpad-b324760c7f53667af128a6b77b790323da04fcb9.tar.xz , if the server side needs an update as well, I suppose it'd be the best to use the same snapshot there as well.

Actions #2

Updated by Luboš Luňák almost 3 years ago

  • Assignee set to Guilhem Moulin
Actions #3

Updated by Guilhem Moulin almost 3 years ago

  • Assignee changed from Guilhem Moulin to Christian Lohmaier

Luboš Luňák wrote in #note-1:

The updated Breakpad snapshot for it is at https://dev-www.libreoffice.org/src/breakpad-b324760c7f53667af128a6b77b790323da04fcb9.tar.xz , if the server side needs an update as well, I suppose it'd be the best to use the same snapshot there as well.

I don't understand what that means so I'm probably not the most suited assignee for this ticket… Hopefull cloph can help, as seen in #3427 I'm not quite on top on the crashreport thing.

Actions #4

Updated by Luboš Luňák almost 3 years ago

Given that https://gerrit.libreoffice.org/c/core/+/113750 has been pushed, does that mean this is done and can be closed?

Actions #5

Updated by Christian Lohmaier almost 3 years ago

Yes and no, while breakpad for LO as well as the minidump_stackwalk tool on the crashreport server have been updated, there does indeed seem to be a problem with adding the symbols to the server.

$ ls -lh mergedlo.pdb/430BDD31990C40E99ED04F6D5A74FD701
total 92M
-rwxr-xr-x 1 cloph users 92M Dec 10 18:26 mergedlo.pd_
-rwxr-xr-x 1 cloph users 96 Dec 10 18:26 refs.ptr

the compressed symbols on dev-downloads vs
$ ls lh /srv/crashreport/symbols/mergedlo.pdb/430BDD31990C40E99ED04F6D5A74FD701
total 19M
-rw-r--r-
1 crashreport crashreport 19M Dec 15 11:21 mergedlo.sym

(that is one of the 7.0.4 rc2 ) - and while I don't expect the size to be the same, that looks a bit too good in terms of "compression" rate..)

Actions #6

Updated by Christian Lohmaier almost 3 years ago

(not all files are that small though, there are plenty others that are 100+MB, so need to dig deeper, and will probably have to use the symbol server's files to reextract them for breakpad)

Actions #7

Updated by Florian Effenberger almost 3 years ago

  • Status changed from New to In Progress
  • Target version set to Q2/2021

Cloph is investigating this, will report back in this ticket

Actions #8

Updated by Christian Lohmaier almost 3 years ago

  • % Done changed from 0 to 70

thankfully dump_syms uses diag tools to symbol location, that makes it possible to use regular symbolserver method for extracting/converting the debuginfo without having to manually downlaod & uncompress the files.
I already re-processed the final builds for 7.0 and 7.1 for both 64bit as well as 32bit. Current dump_syms doesn't crash when processing 32bit builds, but apparently the version used for those did, so there were also some incomplete symbols for 32bit windows.
I'll keep processing older builds in the next days..

# use master's dump_syms.exe for the reprocessing
cp workdir/UnpackedTarball/breakpad/src/tools/windows/dump_syms/Release/dump_syms.exe ~/symbolsreprocessing/dump_syms
# copy the symsrv.dll to enable using symbol server from Windows debugging tools
cp /cygdrive/c/Program\ Files\ \(x86\)/Windows\ Kits/8.1/Debuggers/x86/symsrv.dll ~/symbolsreprocessing/dump_syms
# set symbol server path. N:\symbols is local staging directory for uploading to dev-downloads,
# contains the compressed symbols for recent builds, so for the majority of the builds didn't go via the network
export _NT_SYMBOL_PATH="srv*c:\\cygwin64\\home\\buildslave\\symbolsreprocessing\\symbolcache*N:\\symbols;srv*c:\\cygwin64\\home\\buildslave\\symbolsreprocessing\\symbolcache*https://dev-downloads.libreoffice.org/symstore/symbols" 
# administrative/silent installation of the build to reprocess
msiexec.exe /a N:\\LibreOffice_7.0.6.2_Win_x86.msi TARGETDIR="c:\\cygwin64\\home\\buildslave\\symbolsreprocessing\\instdir" /q
# use the master version of the script to convert the symbols
time ~/source/libo-core/bin/symbolstore.py ~/symbolsreprocessing/dump_syms/dump_syms.exe ~/symbolsreprocessing/symbols ~/symbolsreprocessing/instdir/program/ >/dev/null
# pack as a zip to upload
cd symbols; zip -r ../symbols.zip * ; cd -
# sync to VM host where they are actually uploaded from
rsync -ahvP ~/symbolsreprocessing/symbols.zip cl@192.168.100.1:fixed_w32_7062.zip
# clear instdir and symbolscache and symbols.zip for next round…
rm -rf ~/symbolsreprocessing/instdir ~/symbolsreprocessing/symbol*
Actions #9

Updated by Florian Effenberger almost 3 years ago

  • Priority changed from Normal to Low
  • Target version changed from Q2/2021 to Q3/2021

Is work in progress, all 7.0 and up done
<7.0 (=EOL) in the works

Actions #10

Updated by Florian Effenberger over 2 years ago

  • Status changed from In Progress to Closed
Actions

Also available in: Atom PDF