Windows crashreports at crash.libreoffice.org have useless backtraces (breakpad update needed?)
I have recently analyzed recent reports at crashreport.libreoffice.org and it appears that the vast majority of recent Windows crashreports have useless backtraces, with many frames not resolved to function names or the backtraces having only few frames.
I'm not sure what's caused this, but random observations:
- Linux backtraces seem to work fine (https://crashreport.libreoffice.org/stats/crash_details/c249c48f-5919-425b-bd9d-bb5dc1a34680)
- Backtraces involving Skia (built with Clang) are fine for frames in Skia (https://crashreport.libreoffice.org/stats/crash_details/e1b81d2a-1358-486f-9d81-4f8303e5be0c)
- 32bit (x86) backtraces appear to be fine (https://crashreport.libreoffice.org/stats/crash_details/473f460a-a782-413b-96e1-3e00bd7bfed0), although not all of them (https://crashreport.libreoffice.org/stats/crash_details/b2c17b07-8b32-469b-aedd-61bf13e095fc)
- Reports that are 1 year old appear to be broken (https://crashreport.libreoffice.org/stats/crash_details/6184857d-80dc-4964-bfb5-7e0cd84f40ad)
- Reports that are 2 years old appear to be fine (https://crashreport.libreoffice.org/stats/crash_details/7ad871cb-e187-44eb-bb5a-05a9cc792b0d)
Note that those URLs are just examples, I used psql queries to gather 10k+ crashreports from the most recent LO release from this year and practically all of them are useless.
Our snapshot of the breakpad library is from 2015, so one possible explanation is that this old snapshot cannot properly handle backtraces from recent MSVC 64bit builds. I have a Gerrit change with a recent breakpad snapshot ready, but I expect that the server side on crashreport.libreoffice.org needs to be updated first.
Can somebody please look into this?
Updated by Luboš Luňák over 2 years ago
Note that the pending Gerrit change is https://gerrit.libreoffice.org/c/core/+/113750 . The updated Breakpad snapshot for it is at https://dev-www.libreoffice.org/src/breakpad-b324760c7f53667af128a6b77b790323da04fcb9.tar.xz , if the server side needs an update as well, I suppose it'd be the best to use the same snapshot there as well.
Updated by Guilhem Moulin over 2 years ago
- Assignee changed from Guilhem Moulin to Christian Lohmaier
Luboš Luňák wrote in #note-1:
The updated Breakpad snapshot for it is at https://dev-www.libreoffice.org/src/breakpad-b324760c7f53667af128a6b77b790323da04fcb9.tar.xz , if the server side needs an update as well, I suppose it'd be the best to use the same snapshot there as well.
I don't understand what that means so I'm probably not the most suited assignee for this ticket… Hopefull cloph can help, as seen in #3427 I'm not quite on top on the crashreport thing.
Updated by Christian Lohmaier over 2 years ago
Yes and no, while breakpad for LO as well as the minidump_stackwalk tool on the crashreport server have been updated, there does indeed seem to be a problem with adding the symbols to the server.
$ ls -lh mergedlo.pdb/430BDD31990C40E99ED04F6D5A74FD701
-rwxr-xr-x 1 cloph users 92M Dec 10 18:26 mergedlo.pd_
-rwxr-xr-x 1 cloph users 96 Dec 10 18:26 refs.ptr
the compressed symbols on dev-downloads vs
lh /srv/crashreport/symbols/mergedlo.pdb/430BDD31990C40E99ED04F6D5A74FD701 1 crashreport crashreport 19M Dec 15 11:21 mergedlo.sym
(that is one of the 7.0.4 rc2 ) - and while I don't expect the size to be the same, that looks a bit too good in terms of "compression" rate..)
Updated by Christian Lohmaier over 2 years ago
- % Done changed from 0 to 70
thankfully dump_syms uses diag tools to symbol location, that makes it possible to use regular symbolserver method for extracting/converting the debuginfo without having to manually downlaod & uncompress the files.
I already re-processed the final builds for 7.0 and 7.1 for both 64bit as well as 32bit. Current dump_syms doesn't crash when processing 32bit builds, but apparently the version used for those did, so there were also some incomplete symbols for 32bit windows.
I'll keep processing older builds in the next days..
# use master's dump_syms.exe for the reprocessing cp workdir/UnpackedTarball/breakpad/src/tools/windows/dump_syms/Release/dump_syms.exe ~/symbolsreprocessing/dump_syms # copy the symsrv.dll to enable using symbol server from Windows debugging tools cp /cygdrive/c/Program\ Files\ \(x86\)/Windows\ Kits/8.1/Debuggers/x86/symsrv.dll ~/symbolsreprocessing/dump_syms # set symbol server path. N:\symbols is local staging directory for uploading to dev-downloads, # contains the compressed symbols for recent builds, so for the majority of the builds didn't go via the network export _NT_SYMBOL_PATH="srv*c:\\cygwin64\\home\\buildslave\\symbolsreprocessing\\symbolcache*N:\\symbols;srv*c:\\cygwin64\\home\\buildslave\\symbolsreprocessing\\symbolcache*https://dev-downloads.libreoffice.org/symstore/symbols" # administrative/silent installation of the build to reprocess msiexec.exe /a N:\\LibreOffice_220.127.116.11_Win_x86.msi TARGETDIR="c:\\cygwin64\\home\\buildslave\\symbolsreprocessing\\instdir" /q # use the master version of the script to convert the symbols time ~/source/libo-core/bin/symbolstore.py ~/symbolsreprocessing/dump_syms/dump_syms.exe ~/symbolsreprocessing/symbols ~/symbolsreprocessing/instdir/program/ >/dev/null # pack as a zip to upload cd symbols; zip -r ../symbols.zip * ; cd - # sync to VM host where they are actually uploaded from rsync -ahvP ~/symbolsreprocessing/symbols.zip firstname.lastname@example.org:fixed_w32_7062.zip # clear instdir and symbolscache and symbols.zip for next round… rm -rf ~/symbolsreprocessing/instdir ~/symbolsreprocessing/symbol*