Created attachment 19617 [details] ThreadSanitizer output showing races in lld I experienced a hanging libruby24.so link job, using lld trunk r321545. I recompiled lld with -fsanitize=thread, output of a run is attached. LLD_REPRODUCE tarball (~2MiB): http://www.andric.com/freebsd/lld/ruby-race.tar.xz
Bisection shows this to be a regression from a previous fix in https://reviews.llvm.org/rL314711, introduced by https://reviews.llvm.org/rL316280. I tried to fix it with a local hack, but I didn't succeed. :(
Even if I partially revert r316280, by doing: Index: contrib/llvm/tools/lld/ELF/InputSection.h =================================================================== --- contrib/llvm/tools/lld/ELF/InputSection.h (revision 327379) +++ contrib/llvm/tools/lld/ELF/InputSection.h (working copy) @@ -200,13 +200,13 @@ class InputSectionBase : public SectionBase { // be found by looking at the next one). struct SectionPiece { SectionPiece(size_t Off, uint32_t Hash, bool Live) - : InputOff(Off), Hash(Hash), OutputOff(-1), - Live(Live || !Config->GcSections) {} + : InputOff(Off), Hash(Hash), Live(Live || !Config->GcSections), + OutputOff(-1) {} uint32_t InputOff; - uint32_t Hash; - int64_t OutputOff : 63; - uint64_t Live : 1; + uint32_t Hash : 31; + uint32_t Live : 1; + int64_t OutputOff; }; static_assert(sizeof(SectionPiece) == 16, "SectionPiece is too big"); I still get races between lld::elf::SectionBase::getOffset() and lld::elf::OutputSection::maybeCompress(): ================== WARNING: ThreadSanitizer: data race (pid=12076) Read of size 8 at 0x7b80000f4780 by thread T2: #0 lld::elf::SectionBase::getOffset(unsigned long) const /share/dim/src/freebsd/clang600-import/contrib/llvm/tools/lld/ELF/InputSection.cpp:171:38 (ld.lld+0x170b3dc) #1 getSymVA /share/dim/src/freebsd/clang600-import/contrib/llvm/tools/lld/ELF/Symbols.cpp:91:53 (ld.lld+0x17a14da) #2 lld::elf::Symbol::getVA(long) const /share/dim/src/freebsd/clang600-import/contrib/llvm/tools/lld/ELF/Symbols.cpp:120 (ld.lld+0x17a14da) #3 _ZN3lld3elf12InputSection16relocateNonAllocIN4llvm6object7ELFTypeILNS3_7support10endiannessE1ELb1EEENS4_12Elf_Rel_ImplIS8_Lb1EEEEEvPhNS3_8ArrayRefIT0_EE /share/dim/src/freebsd/clang600-import/contrib/llvm/tools/lld/ELF/InputSection.cpp:685:64 (ld.lld+0x171bb46) #4 _ZN3lld3elf16InputSectionBase8relocateIN4llvm6object7ELFTypeILNS3_7support10endiannessE1ELb1EEEEEvPhS9_ /share/dim/src/freebsd/clang600-import/contrib/llvm/tools/lld/ELF/InputSection.cpp:698:10 (ld.lld+0x17155f2) #5 _ZN3lld3elf12InputSection7writeToIN4llvm6object7ELFTypeILNS3_7support10endiannessE1ELb1EEEEEvPh /share/dim/src/freebsd/clang600-import/contrib/llvm/tools/lld/ELF/InputSection.cpp:780:3 (ld.lld+0x1714621) #6 operator() /share/dim/src/freebsd/clang600-import/contrib/llvm/tools/lld/ELF/OutputSections.cpp:255:9 (ld.lld+0x17509b0) #7 __invoke<(lambda at /share/dim/src/freebsd/clang600-import/contrib/llvm/tools/lld/ELF/OutputSections.cpp:253:40) &, unsigned long> /usr/include/c++/v1/type_traits:4342 (ld.lld+0x17509b0) #8 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN3lld3elf13OutputSection7writeToIN4llvm6object7ELFTypeILNS7_7support10endiannessE1ELb1EEEEEvPhEUlmE_mEEEvDpOT_ /usr/include/c++/v1/__functional_base:349 (ld.lld+0x17509b0) #9 _ZNSt3__110__function6__funcIZN3lld3elf13OutputSection7writeToIN4llvm6object7ELFTypeILNS6_7support10endiannessE1ELb1EEEEEvPhEUlmE_NS_9allocatorISD_EEFvmEEclEOm /usr/include/c++/v1/functional:1562:12 (ld.lld+0x1750914) #10 operator() /usr/include/c++/v1/functional:1916:12 (ld.lld+0x16baa34) #11 llvm::parallel::detail::parallel_for_each_n<unsigned long, std::__1::function<void()(unsigned long)> >(void, unsigned long, llvm::parallel::detail::parallel_for_each_n<unsigned long, std::__1::function<void()(unsigned long)> >, void()(unsigned long)) /share/dim/src/freebsd/clang600-import/contrib/llvm/include/llvm/Support/Parallel.h:183 (ld.lld+0x16baa34) #12 for_each_n<unsigned long, std::__1::function<void (unsigned long)> > /share/dim/src/freebsd/clang600-import/contrib/llvm/include/llvm/Support/Parallel.h:240:3 (ld.lld+0x16ba7a8) #13 lld::parallelForEachN(unsigned long, unsigned long, std::__1::function<void()(unsigned long)>) /share/dim/src/freebsd/clang600-import/contrib/llvm/tools/lld/include/lld/Common/Threads.h:79 (ld.lld+0x16ba7a8) #14 _ZN3lld3elf13OutputSection7writeToIN4llvm6object7ELFTypeILNS3_7support10endiannessE1ELb1EEEEEvPh /share/dim/src/freebsd/clang600-import/contrib/llvm/tools/lld/ELF/OutputSections.cpp:253:3 (ld.lld+0x1747b21) #15 _ZN3lld3elf13OutputSection13maybeCompressIN4llvm6object7ELFTypeILNS3_7support10endiannessE1ELb1EEEEEvv /share/dim/src/freebsd/clang600-import/contrib/llvm/tools/lld/ELF/OutputSections.cpp:204:3 (ld.lld+0x1749363) #16 operator() /share/dim/src/freebsd/clang600-import/contrib/llvm/tools/lld/ELF/Writer.cpp:434:49 (ld.lld+0x185fcff) #17 for_each<std::__1::__wrap_iter<lld::elf::OutputSection **>, (lambda at /share/dim/src/freebsd/clang600-import/contrib/llvm/tools/lld/ELF/Writer.cpp:434:19)> /usr/include/c++/v1/algorithm:965 (ld.lld+0x185fcff) #18 operator() /share/dim/src/freebsd/clang600-import/contrib/llvm/include/llvm/Support/Parallel.h:162 (ld.lld+0x185fcff) #19 __invoke<(lambda at /share/dim/src/freebsd/clang600-import/contrib/llvm/include/llvm/Support/Parallel.h:162:14) &> /usr/include/c++/v1/type_traits:4342 (ld.lld+0x185fcff) #20 __call<(lambda at /share/dim/src/freebsd/clang600-import/contrib/llvm/include/llvm/Support/Parallel.h:162:14) &> /usr/include/c++/v1/__functional_base:349 (ld.lld+0x185fcff) #21 _ZNSt3__110__function6__funcIZN4llvm8parallel6detail17parallel_for_eachINS_11__wrap_iterIPPN3lld3elf13OutputSectionEEEZN12_GLOBAL__N_16WriterINS2_6object7ELFTypeILNS2_7support10endiannessE1ELb1EEEE3runEvEUlSA_E_EEvT_SM_T0_EUlvE_NS_9allocatorISO_EEFvvEEclEv /usr/include/c++/v1/functional:1562 (ld.lld+0x185fcff) #22 operator() /usr/include/c++/v1/functional:1916:12 (ld.lld+0x1ae1b35) #23 operator() /share/dim/src/freebsd/clang600-import/contrib/llvm/lib/Support/Parallel.cpp:134 (ld.lld+0x1ae1b35) #24 __invoke<(lambda at /share/dim/src/freebsd/clang600-import/contrib/llvm/lib/Support/Parallel.cpp:133:39) &> /usr/include/c++/v1/type_traits:4342 (ld.lld+0x1ae1b35) #25 __call<(lambda at /share/dim/src/freebsd/clang600-import/contrib/llvm/lib/Support/Parallel.cpp:133:39) &> /usr/include/c++/v1/__functional_base:349 (ld.lld+0x1ae1b35) #26 std::__1::__function::__func<llvm::parallel::detail::TaskGroup::spawn(std::__1::function<void()(void)>)::$_0, std::__1::allocator<std::__1::function<void()(void)>>, std::__1::function>::operator()(void) /usr/include/c++/v1/functional:1562 (ld.lld+0x1ae1b35) #27 operator() /usr/include/c++/v1/functional:1916:12 (ld.lld+0x1ade88b) #28 (anonymous namespace)::ThreadPoolExecutor::work(void) /share/dim/src/freebsd/clang600-import/contrib/llvm/lib/Support/Parallel.cpp:111 (ld.lld+0x1ade88b) #29 operator() /share/dim/src/freebsd/clang600-import/contrib/llvm/lib/Support/Parallel.cpp:80:27 (ld.lld+0x1ade9dc) #30 __invoke<(lambda at /share/dim/src/freebsd/clang600-import/contrib/llvm/lib/Support/Parallel.cpp:80:21)> /usr/include/c++/v1/type_traits:4342 (ld.lld+0x1ade9dc) #31 __thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, (lambda at /share/dim/src/freebsd/clang600-import/contrib/llvm/lib/Support/Parallel.cpp:80:21)> /usr/include/c++/v1/thread:342 (ld.lld+0x1ade9dc) #32 _ZNSt3__114__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEZZN12_GLOBAL__N_118ThreadPoolExecutorC1EjENKUlvE_clEvEUlvE_EEEEEPvSC_ /usr/include/c++/v1/thread:352 (ld.lld+0x1ade9dc) Previous write of size 8 at 0x7b80000f4780 by thread T3: #0 _ZN3lld3elf13OutputSection13maybeCompressIN4llvm6object7ELFTypeILNS3_7support10endiannessE1ELb1EEEEEvv /share/dim/src/freebsd/clang600-import/contrib/llvm/tools/lld/ELF/OutputSections.cpp:191:23 (ld.lld+0x174913f) #1 operator() /share/dim/src/freebsd/clang600-import/contrib/llvm/tools/lld/ELF/Writer.cpp:434:49 (ld.lld+0x185fcff) #2 for_each<std::__1::__wrap_iter<lld::elf::OutputSection **>, (lambda at /share/dim/src/freebsd/clang600-import/contrib/llvm/tools/lld/ELF/Writer.cpp:434:19)> /usr/include/c++/v1/algorithm:965 (ld.lld+0x185fcff) #3 operator() /share/dim/src/freebsd/clang600-import/contrib/llvm/include/llvm/Support/Parallel.h:162 (ld.lld+0x185fcff) #4 __invoke<(lambda at /share/dim/src/freebsd/clang600-import/contrib/llvm/include/llvm/Support/Parallel.h:162:14) &> /usr/include/c++/v1/type_traits:4342 (ld.lld+0x185fcff) #5 __call<(lambda at /share/dim/src/freebsd/clang600-import/contrib/llvm/include/llvm/Support/Parallel.h:162:14) &> /usr/include/c++/v1/__functional_base:349 (ld.lld+0x185fcff) #6 _ZNSt3__110__function6__funcIZN4llvm8parallel6detail17parallel_for_eachINS_11__wrap_iterIPPN3lld3elf13OutputSectionEEEZN12_GLOBAL__N_16WriterINS2_6object7ELFTypeILNS2_7support10endiannessE1ELb1EEEE3runEvEUlSA_E_EEvT_SM_T0_EUlvE_NS_9allocatorISO_EEFvvEEclEv /usr/include/c++/v1/functional:1562 (ld.lld+0x185fcff) #7 operator() /usr/include/c++/v1/functional:1916:12 (ld.lld+0x1ae1b35) #8 operator() /share/dim/src/freebsd/clang600-import/contrib/llvm/lib/Support/Parallel.cpp:134 (ld.lld+0x1ae1b35) #9 __invoke<(lambda at /share/dim/src/freebsd/clang600-import/contrib/llvm/lib/Support/Parallel.cpp:133:39) &> /usr/include/c++/v1/type_traits:4342 (ld.lld+0x1ae1b35) #10 __call<(lambda at /share/dim/src/freebsd/clang600-import/contrib/llvm/lib/Support/Parallel.cpp:133:39) &> /usr/include/c++/v1/__functional_base:349 (ld.lld+0x1ae1b35) #11 std::__1::__function::__func<llvm::parallel::detail::TaskGroup::spawn(std::__1::function<void()(void)>)::$_0, std::__1::allocator<std::__1::function<void()(void)>>, std::__1::function>::operator()(void) /usr/include/c++/v1/functional:1562 (ld.lld+0x1ae1b35) #12 operator() /usr/include/c++/v1/functional:1916:12 (ld.lld+0x1ade88b) #13 (anonymous namespace)::ThreadPoolExecutor::work(void) /share/dim/src/freebsd/clang600-import/contrib/llvm/lib/Support/Parallel.cpp:111 (ld.lld+0x1ade88b) #14 operator() /share/dim/src/freebsd/clang600-import/contrib/llvm/lib/Support/Parallel.cpp:80:27 (ld.lld+0x1ade9dc) #15 __invoke<(lambda at /share/dim/src/freebsd/clang600-import/contrib/llvm/lib/Support/Parallel.cpp:80:21)> /usr/include/c++/v1/type_traits:4342 (ld.lld+0x1ade9dc) #16 __thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, (lambda at /share/dim/src/freebsd/clang600-import/contrib/llvm/lib/Support/Parallel.cpp:80:21)> /usr/include/c++/v1/thread:342 (ld.lld+0x1ade9dc) #17 _ZNSt3__114__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEZZN12_GLOBAL__N_118ThreadPoolExecutorC1EjENKUlvE_clEvEUlvE_EEEEEPvSC_ /usr/include/c++/v1/thread:352 (ld.lld+0x1ade9dc) Location is heap block of size 4096 at 0x7b80000f4000 allocated by main thread: #0 malloc /share/dim/src/freebsd/clang600-import/contrib/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:666:5 (ld.lld+0x162f132) #1 StartNewSlab /share/dim/src/freebsd/clang600-import/contrib/llvm/include/llvm/Support/Allocator.h:97:12 (ld.lld+0x167c348) #2 llvm::BumpPtrAllocatorImpl<llvm::MallocAllocator, 4096, 4096>::Allocate(unsigned long, unsigned long) /share/dim/src/freebsd/clang600-import/contrib/llvm/include/llvm/Support/Allocator.h:258 (ld.lld+0x167c348) #3 Allocate /share/dim/src/freebsd/clang600-import/contrib/llvm/include/llvm/Support/Allocator.h:57:43 (ld.lld+0x16ecbea) #4 Allocate<lld::elf::InputSection> /share/dim/src/freebsd/clang600-import/contrib/llvm/include/llvm/Support/Allocator.h:79 (ld.lld+0x16ecbea) #5 Allocate /share/dim/src/freebsd/clang600-import/contrib/llvm/include/llvm/Support/Allocator.h:434 (ld.lld+0x16ecbea) #6 _ZN3lld4makeINS_3elf12InputSectionEJRNS1_7ObjFileIN4llvm6object7ELFTypeILNS4_7support10endiannessE1ELb1EEEEERKNS5_13Elf_Shdr_ImplIS9_EERNS4_9StringRefEEEEPT_DpOT0_ /share/dim/src/freebsd/clang600-import/contrib/llvm/tools/lld/include/lld/Common/Memory.h:55 (ld.lld+0x16ecbea) #7 _ZN3lld3elf7ObjFileIN4llvm6object7ELFTypeILNS2_7support10endiannessE1ELb1EEEE18createInputSectionERKNS3_13Elf_Shdr_ImplIS7_EE /share/dim/src/freebsd/clang600-import/contrib/llvm/tools/lld/ELF/InputFiles.cpp (ld.lld+0x16eb2b5) #8 _ZN3lld3elf7ObjFileIN4llvm6object7ELFTypeILNS2_7support10endiannessE1ELb1EEEE18initializeSectionsERNS2_8DenseSetINS2_19CachedHashStringRefENS2_12DenseMapInfoISA_EEEE /share/dim/src/freebsd/clang600-import/contrib/llvm/tools/lld/ELF/InputFiles.cpp:425:27 (ld.lld+0x16e99e0) #9 _ZN3lld3elf7ObjFileIN4llvm6object7ELFTypeILNS2_7support10endiannessE1ELb1EEEE5parseERNS2_8DenseSetINS2_19CachedHashStringRefENS2_12DenseMapInfoISA_EEEE /share/dim/src/freebsd/clang600-import/contrib/llvm/tools/lld/ELF/InputFiles.cpp:287:3 (ld.lld+0x16e96d0) #10 _ZN3lld3elf11SymbolTable7addFileIN4llvm6object7ELFTypeILNS3_7support10endiannessE1ELb1EEEEEvPNS0_9InputFileE /share/dim/src/freebsd/clang600-import/contrib/llvm/tools/lld/ELF/SymbolTable.cpp:112:30 (ld.lld+0x179bb29) #11 _ZN3lld3elf12LinkerDriver4linkIN4llvm6object7ELFTypeILNS3_7support10endiannessE1ELb1EEEEEvRNS3_3opt12InputArgListE /share/dim/src/freebsd/clang600-import/contrib/llvm/tools/lld/ELF/Driver.cpp:1008:13 (ld.lld+0x16a1bf3) #12 bool lld::elf::LinkerDriver::main(llvm::ArrayRef<char const*>() /share/dim/src/freebsd/clang600-import/contrib/llvm/tools/lld/ELF/Driver.cpp:387:5 (ld.lld+0x16981fc) #13 bool lld::elf::link(llvm::ArrayRef<char const*>(lld::elf::link::raw_ostream&) /share/dim/src/freebsd/clang600-import/contrib/llvm/tools/lld/ELF/Driver.cpp:93:11 (ld.lld+0x169671c) #14 main /share/dim/src/freebsd/clang600-import/contrib/llvm/tools/lld/tools/lld/lld.cpp:115:11 (ld.lld+0x187be3d) Thread T2 (tid=135669427011328, running) created by thread T1 at: #0 pthread_create /share/dim/src/freebsd/clang600-import/contrib/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992:3 (ld.lld+0x1630791) #1 thread<(lambda at /share/dim/src/freebsd/clang600-import/contrib/llvm/lib/Support/Parallel.cpp:80:21), void> /usr/include/c++/v1/__threading_support:327:10 (ld.lld+0x1ade47f) #2 operator() /share/dim/src/freebsd/clang600-import/contrib/llvm/lib/Support/Parallel.cpp:80 (ld.lld+0x1ade47f) #3 __invoke<(lambda at /share/dim/src/freebsd/clang600-import/contrib/llvm/lib/Support/Parallel.cpp:78:17)> /usr/include/c++/v1/type_traits:4342 (ld.lld+0x1ade47f) #4 __thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, (lambda at /share/dim/src/freebsd/clang600-import/contrib/llvm/lib/Support/Parallel.cpp:78:17)> /usr/include/c++/v1/thread:342 (ld.lld+0x1ade47f) #5 _ZNSt3__114__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEZN12_GLOBAL__N_118ThreadPoolExecutorC1EjEUlvE_EEEEEPvSB_ /usr/include/c++/v1/thread:352 (ld.lld+0x1ade47f) Thread T3 (tid=135669427012608, running) created by thread T1 at: #0 pthread_create /share/dim/src/freebsd/clang600-import/contrib/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992:3 (ld.lld+0x1630791) #1 thread<(lambda at /share/dim/src/freebsd/clang600-import/contrib/llvm/lib/Support/Parallel.cpp:80:21), void> /usr/include/c++/v1/__threading_support:327:10 (ld.lld+0x1ade47f) #2 operator() /share/dim/src/freebsd/clang600-import/contrib/llvm/lib/Support/Parallel.cpp:80 (ld.lld+0x1ade47f) #3 __invoke<(lambda at /share/dim/src/freebsd/clang600-import/contrib/llvm/lib/Support/Parallel.cpp:78:17)> /usr/include/c++/v1/type_traits:4342 (ld.lld+0x1ade47f) #4 __thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, (lambda at /share/dim/src/freebsd/clang600-import/contrib/llvm/lib/Support/Parallel.cpp:78:17)> /usr/include/c++/v1/thread:342 (ld.lld+0x1ade47f) #5 _ZNSt3__114__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEZN12_GLOBAL__N_118ThreadPoolExecutorC1EjEUlvE_EEEEEPvSB_ /usr/include/c++/v1/thread:352 (ld.lld+0x1ade47f) SUMMARY: ThreadSanitizer: data race /share/dim/src/freebsd/clang600-import/contrib/llvm/tools/lld/ELF/InputSection.cpp:171:38 in lld::elf::SectionBase::getOffset(unsigned long) const ==================
Further bisection shows that https://reviews.llvm.org/rL320472 introduced the second race, between SectionBase::getOffset() and OutputSection::maybeCompress(). This last race seems to be the actual cause for the hang linking libruby24.so.
I'm taking a look at this now.
I have identified the issue. The problem came because I moved the calculation of OutSecOff for InputSections that are to be compressed into maybeCompress. maybeCompress is run in parallel for each OutputSection (although it does nothing for non-debug sections). Subsequent to the calculation of OutSecOff, it calls writeTo on the OutputSection, which in turn calls writeTo on each InputSection in the OutputSection. This in turn patches relocations. If a relocation refers to a symbol outside the OutputSection containing the corresponding InputSection, there is no guarantee that the value of that symbol has been determined yet, because the corresponding InputSection may belong to a different OutputSection that has not yet been processed. I am currently considering how to fix this.
Thinking about it, there's a fairly fundamental problem with maybeCompress and when it is called. Currently it is called before assignAddresses. This is necessary, because the compressed size of the debug sections is needed in order to lay them out. However, since the exact contents of the debug sections affects their compressed size, we cannot know this size until after we have patched all relocations in those sections. However, we do not know the value of symbols until after assignAddresses, resulting in a cyclical dependency. Prior to my previous change, compressed debug sections would still have contained incorrect contents, in the event that a linker script resulted in symbols referred to by debug data having a different value to their "natural" value. I can only see 2 ways of fixing this. Option 1 - forbid use of custom layout for any ELF via linker scripts, when using compressed debug sections - this would require reinstating the early Size and Offset setting done previously. Option 2 - firstly, completely ignore compressed debug sections during assingAddresses; secondly, calculate the InputSection OutSecOff and pre-compressed OutputSection Size; thirdly, run maybeCompress; finally, run a special layout pass to assign offsets to compressed debug sections. This will result in these sections always being at the end of the file. Option 1) feels like it would be easier, but more restrictive than necessary. Option 2) feels more like the "right" fix. @Rui - do you have a preferred approach?
Oh, wait, I misremembered the layout code slightly. I think it is possible to simply move maybeCompress to after assignAddresses, remove the skipping of SHF_COMPRESSED sections in assignOffsets, and remove the special loop I added in maybeCompress (thus also removing the race condition). This should give us support for custom layout of compressed debug sections as a side-benefit, with no cost, as far as I can tell. I will try to make the changes locally and then post a patch up for review.
I have created a patch that fixes the race condition and issues with relocation patching in compressed debug sections - see https://reviews.llvm.org/D41773.
Fixed in r321986.
(In reply to James Henderson from comment #9) > Fixed in r321986. It does indeed, fix the second race, but when I now attempt to fix the first race, it starts hanging. Shall I open a separate PR for the first race?
(In reply to Dimitry Andric from comment #10) > (In reply to James Henderson from comment #9) > > Fixed in r321986. > > It does indeed, fix the second race, but when I now attempt to fix the first > race, it starts hanging. Note that the hack I tried was this: --- contrib/llvm/tools/lld/ELF/InputSection.h (revision 327680) +++ contrib/llvm/tools/lld/ELF/InputSection.h (working copy) @@ -200,13 +200,13 @@ class InputSectionBase : public SectionBase { // be found by looking at the next one). struct SectionPiece { SectionPiece(size_t Off, uint32_t Hash, bool Live) - : InputOff(Off), Hash(Hash), OutputOff(-1), - Live(Live || !Config->GcSections) {} + : InputOff(Off), Hash(Hash), Live(Live || !Config->GcSections), + OutputOff(-1) {} uint32_t InputOff; - uint32_t Hash; - int64_t OutputOff : 63; - uint64_t Live : 1; + uint32_t Hash : 31; + uint32_t Live : 1; + int64_t OutputOff; }; static_assert(sizeof(SectionPiece) == 16, "SectionPiece is too big"); But I'm completely unsure whether shortening the hash to 31 bits will blow up things. Putting only this change in seemed to resolve at least the InputOff/OutputOff race.
What is the first race?
(In reply to Rui Ueyama from comment #12) > What is the first race? See comment #2, r316280 ("Assume that mergeable input sections are smaller than 4 GiB") caused a regression from r314711 ("Fix a data race found by tsan"). Basically, the SectionPiece::OutputOff and SectionPiece::Live members are bitfields, and cannot be updated from different threads atomically without locking. So I tried moving the Live bit somewhere between the InputOff and Hash members instead, but I'm unsure of the side effects.
For example, if I add this: Index: contrib/llvm/tools/lld/ELF/InputSection.h =================================================================== --- contrib/llvm/tools/lld/ELF/InputSection.h (revision 327680) +++ contrib/llvm/tools/lld/ELF/InputSection.h (working copy) @@ -205,11 +205,11 @@ struct SectionPiece { uint32_t InputOff; uint32_t Hash; - int64_t OutputOff : 63; - uint64_t Live : 1; + int64_t OutputOff; + uint8_t Live; }; -static_assert(sizeof(SectionPiece) == 16, "SectionPiece is too big"); +static_assert(sizeof(SectionPiece) == 24, "SectionPiece is too big"); // This corresponds to a SHF_MERGE section of an input file. class MergeInputSection : public InputSectionBase { Then the libruby24.so link job sometimes never finishes, and all threads are sitting in a cond_wait: (gdb) thread apply all bt Thread 6 (LWP 101253 of process 72867): #0 _umtx_op_err () at /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37 #1 0x0000000803c723e1 in _thr_umtx_timedwait_uint (mtx=0x8047ff068, id=<optimized out>, clockid=<optimized out>, abstime=<optimized out>, shared=<optimized out>) at /usr/src/lib/libthr/thread/thr_umtx.c:236 #2 0x0000000803c7c038 in cond_wait_user (abstime=0x0, cancel=1, cvp=<optimized out>, mp=<optimized out>) at /usr/src/lib/libthr/thread/thr_cond.c:306 #3 cond_wait_common (cond=<optimized out>, mutex=<optimized out>, abstime=0x0, cancel=1) at /usr/src/lib/libthr/thread/thr_cond.c:366 #4 0x000000000161e026 in __tsan::call_pthread_cancel_with_cleanup (fn=0x803c7c310 <__pthread_cond_wait>, c=0x8047ff068, m=0xf, abstime=0x0, cleanup=<optimized out>, arg=<optimized out>) at /usr/src/contrib/compiler-rt/lib/tsan/rtl/tsan_platform_linux.cc:344 #5 0x0000000001629184 in cond_wait (thr=0x80670b900, pc=23236459, si=<optimized out>, fn=0x803c7c310 <__pthread_cond_wait>, c=0x7fffdf7f9b08, m=0x7fffdf7f9b00, t=0x0) at /usr/src/contrib/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1150 #6 0x0000000001629062 in __interceptor_pthread_cond_wait (c=<optimized out>, m=0x7fffdf7f9b00) at /usr/src/contrib/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1161 #7 0x0000000803d40d62 in std::__1::__libcpp_condvar_wait (__cv=0x8047ff068, __m=0xf) at /usr/src/contrib/libc++/include/__threading_support:286 #8 std::__1::condition_variable::wait (this=0x8047ff068, lk=...) at /usr/src/contrib/libc++/src/condition_variable.cpp:44 #9 0x00000000016b2b3b in wait<(lambda at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:71:21)> () at /usr/include/c++/v1/__mutex_base:375 #10 sync () at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:71 #11 ~Latch () at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:56 #12 ~TaskGroup () at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:75 #13 parallel_for_each_n<unsigned long, std::__1::function<void (unsigned long)> >(void) () at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:184 #14 0x00000000016b2869 in for_each_n<unsigned long, std::__1::function<void (unsigned long)> > () at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:240 #15 parallelForEachN () at /usr/src/contrib/llvm/tools/lld/include/lld/Common/Threads.h:79 #16 0x000000000173fb12 in writeTo<llvm::object::ELFType<llvm::support::little, true> > () at /usr/src/contrib/llvm/tools/lld/ELF/OutputSections.cpp:244 #17 0x0000000001740eea in maybeCompress<llvm::object::ELFType<llvm::support::little, true> > () at /usr/src/contrib/llvm/tools/lld/ELF/OutputSections.cpp:195 #18 0x00000000018575e0 in operator() () at /usr/src/contrib/llvm/tools/lld/ELF/Writer.cpp:436 #19 for_each<std::__1::__wrap_iter<lld::elf::OutputSection **>, (lambda at /usr/src/contrib/llvm/tools/lld/ELF/Writer.cpp:436:19)> () at /usr/include/c++/v1/algorithm:965 #20 operator() () at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:162 #21 __invoke<(lambda at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:162:14) &> () at /usr/include/c++/v1/type_traits:4482 #22 __call<(lambda at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:162:14) &> () at /usr/include/c++/v1/__functional_base:349 #23 operator() () at /usr/include/c++/v1/functional:1562 #24 0x0000000001ad9786 in operator() () at /usr/include/c++/v1/functional:1916 #25 operator() () at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:134 #26 __invoke<(lambda at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:133:39) &> () at /usr/include/c++/v1/type_traits:4482 #27 __call<(lambda at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:133:39) &> () at /usr/include/c++/v1/__functional_base:349 #28 operator() () at /usr/include/c++/v1/functional:1562 #29 0x0000000001ad64dc in operator() () at /usr/include/c++/v1/functional:1916 #30 work () at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:111 #31 0x0000000001ad662d in operator() () at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:80 #32 __invoke<(lambda at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:80:21)> () at /usr/include/c++/v1/type_traits:4482 #33 __thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, (lambda at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:80:21)> () at /usr/include/c++/v1/thread:342 #34 __thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, (lambda at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:80:21)> >(void) () at /usr/include/c++/v1/thread:352 #35 0x0000000001628719 in __tsan_thread_start_func (arg=0x7fffdfdfceb8) at /usr/src/contrib/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:982 #36 0x0000000803c70426 in thread_start (curthread=0x7b6400010900) at /usr/src/lib/libthr/thread/thr_create.c:291 #37 0x0000000000000000 in ?? () Backtrace stopped: Cannot access memory at address 0x7fffdf7fa000 Thread 5 (LWP 101252 of process 72867): #0 _umtx_op_err () at /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37 #1 0x0000000803c723e1 in _thr_umtx_timedwait_uint (mtx=0x8047ff050, id=<optimized out>, clockid=<optimized out>, abstime=<optimized out>, shared=<optimized out>) at /usr/src/lib/libthr/thread/thr_umtx.c:236 #2 0x0000000803c7c038 in cond_wait_user (abstime=0x0, cancel=1, cvp=<optimized out>, mp=<optimized out>) at /usr/src/lib/libthr/thread/thr_cond.c:306 #3 cond_wait_common (cond=<optimized out>, mutex=<optimized out>, abstime=0x0, cancel=1) at /usr/src/lib/libthr/thread/thr_cond.c:366 #4 0x000000000161e026 in __tsan::call_pthread_cancel_with_cleanup (fn=0x803c7c310 <__pthread_cond_wait>, c=0x8047ff050, m=0xf, abstime=0x0, cleanup=<optimized out>, arg=<optimized out>) at /usr/src/contrib/compiler-rt/lib/tsan/rtl/tsan_platform_linux.cc:344 #5 0x0000000001629184 in cond_wait (thr=0x806689900, pc=23236459, si=<optimized out>, fn=0x803c7c310 <__pthread_cond_wait>, c=0x7fffdf9fab08, m=0x7fffdf9fab00, t=0x0) at /usr/src/contrib/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1150 #6 0x0000000001629062 in __interceptor_pthread_cond_wait (c=<optimized out>, m=0x7fffdf9fab00) at /usr/src/contrib/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1161 #7 0x0000000803d40d62 in std::__1::__libcpp_condvar_wait (__cv=0x8047ff050, __m=0xf) at /usr/src/contrib/libc++/include/__threading_support:286 #8 std::__1::condition_variable::wait (this=0x8047ff050, lk=...) at /usr/src/contrib/libc++/src/condition_variable.cpp:44 #9 0x00000000016b2b3b in wait<(lambda at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:71:21)> () at /usr/include/c++/v1/__mutex_base:375 #10 sync () at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:71 #11 ~Latch () at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:56 #12 ~TaskGroup () at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:75 #13 parallel_for_each_n<unsigned long, std::__1::function<void (unsigned long)> >(void) () at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:184 #14 0x00000000016b2869 in for_each_n<unsigned long, std::__1::function<void (unsigned long)> > () at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:240 #15 parallelForEachN () at /usr/src/contrib/llvm/tools/lld/include/lld/Common/Threads.h:79 #16 0x000000000173fb12 in writeTo<llvm::object::ELFType<llvm::support::little, true> > () at /usr/src/contrib/llvm/tools/lld/ELF/OutputSections.cpp:244 #17 0x0000000001740eea in maybeCompress<llvm::object::ELFType<llvm::support::little, true> > () at /usr/src/contrib/llvm/tools/lld/ELF/OutputSections.cpp:195 #18 0x00000000018575e0 in operator() () at /usr/src/contrib/llvm/tools/lld/ELF/Writer.cpp:436 #19 for_each<std::__1::__wrap_iter<lld::elf::OutputSection **>, (lambda at /usr/src/contrib/llvm/tools/lld/ELF/Writer.cpp:436:19)> () at /usr/include/c++/v1/algorithm:965 #20 operator() () at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:162 #21 __invoke<(lambda at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:162:14) &> () at /usr/include/c++/v1/type_traits:4482 #22 __call<(lambda at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:162:14) &> () at /usr/include/c++/v1/__functional_base:349 #23 operator() () at /usr/include/c++/v1/functional:1562 #24 0x0000000001ad9786 in operator() () at /usr/include/c++/v1/functional:1916 #25 operator() () at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:134 #26 __invoke<(lambda at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:133:39) &> () at /usr/include/c++/v1/type_traits:4482 #27 __call<(lambda at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:133:39) &> () at /usr/include/c++/v1/__functional_base:349 #28 operator() () at /usr/include/c++/v1/functional:1562 #29 0x0000000001ad64dc in operator() () at /usr/include/c++/v1/functional:1916 #30 work () at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:111 #31 0x0000000001ad662d in operator() () at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:80 #32 __invoke<(lambda at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:80:21)> () at /usr/include/c++/v1/type_traits:4482 #33 __thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, (lambda at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:80:21)> () at /usr/include/c++/v1/thread:342 #34 __thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, (lambda at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:80:21)> >(void) () at /usr/include/c++/v1/thread:352 #35 0x0000000001628719 in __tsan_thread_start_func (arg=0x7fffdfdfceb8) at /usr/src/contrib/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:982 #36 0x0000000803c70426 in thread_start (curthread=0x7b6400010400) at /usr/src/lib/libthr/thread/thr_create.c:291 #37 0x0000000000000000 in ?? () Backtrace stopped: Cannot access memory at address 0x7fffdf9fb000 Thread 4 (LWP 101251 of process 72867): #0 _umtx_op_err () at /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37 #1 0x0000000803c723e1 in _thr_umtx_timedwait_uint (mtx=0x8047ff038, id=<optimized out>, clockid=<optimized out>, abstime=<optimized out>, shared=<optimized out>) at /usr/src/lib/libthr/thread/thr_umtx.c:236 #2 0x0000000803c7c038 in cond_wait_user (abstime=0x0, cancel=1, cvp=<optimized out>, mp=<optimized out>) at /usr/src/lib/libthr/thread/thr_cond.c:306 #3 cond_wait_common (cond=<optimized out>, mutex=<optimized out>, abstime=0x0, cancel=1) at /usr/src/lib/libthr/thread/thr_cond.c:366 #4 0x000000000161e026 in __tsan::call_pthread_cancel_with_cleanup (fn=0x803c7c310 <__pthread_cond_wait>, c=0x8047ff038, m=0xf, abstime=0x0, cleanup=<optimized out>, arg=<optimized out>) at /usr/src/contrib/compiler-rt/lib/tsan/rtl/tsan_platform_linux.cc:344 #5 0x0000000001629184 in cond_wait (thr=0x8065d6900, pc=23236459, si=<optimized out>, fn=0x803c7c310 <__pthread_cond_wait>, c=0x7fffdfbfbb08, m=0x7fffdfbfbb00, t=0x0) at /usr/src/contrib/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1150 #6 0x0000000001629062 in __interceptor_pthread_cond_wait (c=<optimized out>, m=0x7fffdfbfbb00) at /usr/src/contrib/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1161 #7 0x0000000803d40d62 in std::__1::__libcpp_condvar_wait (__cv=0x8047ff038, __m=0xf) at /usr/src/contrib/libc++/include/__threading_support:286 #8 std::__1::condition_variable::wait (this=0x8047ff038, lk=...) at /usr/src/contrib/libc++/src/condition_variable.cpp:44 #9 0x00000000016b2b3b in wait<(lambda at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:71:21)> () at /usr/include/c++/v1/__mutex_base:375 #10 sync () at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:71 #11 ~Latch () at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:56 #12 ~TaskGroup () at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:75 #13 parallel_for_each_n<unsigned long, std::__1::function<void (unsigned long)> >(void) () at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:184 #14 0x00000000016b2869 in for_each_n<unsigned long, std::__1::function<void (unsigned long)> > () at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:240 #15 parallelForEachN () at /usr/src/contrib/llvm/tools/lld/include/lld/Common/Threads.h:79 #16 0x000000000173fb12 in writeTo<llvm::object::ELFType<llvm::support::little, true> > () at /usr/src/contrib/llvm/tools/lld/ELF/OutputSections.cpp:244 #17 0x0000000001740eea in maybeCompress<llvm::object::ELFType<llvm::support::little, true> > () at /usr/src/contrib/llvm/tools/lld/ELF/OutputSections.cpp:195 #18 0x00000000018575e0 in operator() () at /usr/src/contrib/llvm/tools/lld/ELF/Writer.cpp:436 #19 for_each<std::__1::__wrap_iter<lld::elf::OutputSection **>, (lambda at /usr/src/contrib/llvm/tools/lld/ELF/Writer.cpp:436:19)> () at /usr/include/c++/v1/algorithm:965 #20 operator() () at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:162 #21 __invoke<(lambda at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:162:14) &> () at /usr/include/c++/v1/type_traits:4482 #22 __call<(lambda at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:162:14) &> () at /usr/include/c++/v1/__functional_base:349 #23 operator() () at /usr/include/c++/v1/functional:1562 #24 0x0000000001ad9786 in operator() () at /usr/include/c++/v1/functional:1916 #25 operator() () at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:134 #26 __invoke<(lambda at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:133:39) &> () at /usr/include/c++/v1/type_traits:4482 #27 __call<(lambda at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:133:39) &> () at /usr/include/c++/v1/__functional_base:349 #28 operator() () at /usr/include/c++/v1/functional:1562 #29 0x0000000001ad64dc in operator() () at /usr/include/c++/v1/functional:1916 #30 work () at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:111 #31 0x0000000001ad662d in operator() () at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:80 #32 __invoke<(lambda at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:80:21)> () at /usr/include/c++/v1/type_traits:4482 #33 __thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, (lambda at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:80:21)> () at /usr/include/c++/v1/thread:342 #34 __thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, (lambda at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:80:21)> >(void) () at /usr/include/c++/v1/thread:352 #35 0x0000000001628719 in __tsan_thread_start_func (arg=0x7fffdfdfceb8) at /usr/src/contrib/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:982 #36 0x0000000803c70426 in thread_start (curthread=0x7b640000ff00) at /usr/src/lib/libthr/thread/thr_create.c:291 #37 0x0000000000000000 in ?? () Backtrace stopped: Cannot access memory at address 0x7fffdfbfc000 Thread 3 (LWP 101250 of process 72867): #0 _umtx_op_err () at /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37 #1 0x0000000803c723e1 in _thr_umtx_timedwait_uint (mtx=0x8047ff020, id=<optimized out>, clockid=<optimized out>, abstime=<optimized out>, shared=<optimized out>) at /usr/src/lib/libthr/thread/thr_umtx.c:236 #2 0x0000000803c7c038 in cond_wait_user (abstime=0x0, cancel=1, cvp=<optimized out>, mp=<optimized out>) at /usr/src/lib/libthr/thread/thr_cond.c:306 #3 cond_wait_common (cond=<optimized out>, mutex=<optimized out>, abstime=0x0, cancel=1) at /usr/src/lib/libthr/thread/thr_cond.c:366 #4 0x000000000161e026 in __tsan::call_pthread_cancel_with_cleanup (fn=0x803c7c310 <__pthread_cond_wait>, c=0x8047ff020, m=0xf, abstime=0x0, cleanup=<optimized out>, arg=<optimized out>) at /usr/src/contrib/compiler-rt/lib/tsan/rtl/tsan_platform_linux.cc:344 #5 0x0000000001629184 in cond_wait (thr=0x806554900, pc=23236459, si=<optimized out>, fn=0x803c7c310 <__pthread_cond_wait>, c=0x7fffdfdfcad8, m=0x7fffdfdfcad0, t=0x0) at /usr/src/contrib/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1150 #6 0x0000000001629062 in __interceptor_pthread_cond_wait (c=<optimized out>, m=0x7fffdfdfcad0) at /usr/src/contrib/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1161 #7 0x0000000803d40d62 in std::__1::__libcpp_condvar_wait (__cv=0x8047ff020, __m=0xf) at /usr/src/contrib/libc++/include/__threading_support:286 #8 std::__1::condition_variable::wait (this=0x8047ff020, lk=...) at /usr/src/contrib/libc++/src/condition_variable.cpp:44 #9 0x00000000016b2b3b in wait<(lambda at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:71:21)> () at /usr/include/c++/v1/__mutex_base:375 #10 sync () at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:71 #11 ~Latch () at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:56 #12 ~TaskGroup () at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:75 #13 parallel_for_each_n<unsigned long, std::__1::function<void (unsigned long)> >(void) () at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:184 #14 0x00000000016b2869 in for_each_n<unsigned long, std::__1::function<void (unsigned long)> > () at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:240 #15 parallelForEachN () at /usr/src/contrib/llvm/tools/lld/include/lld/Common/Threads.h:79 #16 0x000000000173fb12 in writeTo<llvm::object::ELFType<llvm::support::little, true> > () at /usr/src/contrib/llvm/tools/lld/ELF/OutputSections.cpp:244 #17 0x0000000001740eea in maybeCompress<llvm::object::ELFType<llvm::support::little, true> > () at /usr/src/contrib/llvm/tools/lld/ELF/OutputSections.cpp:195 #18 0x00000000018575e0 in operator() () at /usr/src/contrib/llvm/tools/lld/ELF/Writer.cpp:436 #19 for_each<std::__1::__wrap_iter<lld::elf::OutputSection **>, (lambda at /usr/src/contrib/llvm/tools/lld/ELF/Writer.cpp:436:19)> () at /usr/include/c++/v1/algorithm:965 #20 operator() () at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:162 #21 __invoke<(lambda at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:162:14) &> () at /usr/include/c++/v1/type_traits:4482 #22 __call<(lambda at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:162:14) &> () at /usr/include/c++/v1/__functional_base:349 #23 operator() () at /usr/include/c++/v1/functional:1562 #24 0x0000000001ad9786 in operator() () at /usr/include/c++/v1/functional:1916 #25 operator() () at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:134 #26 __invoke<(lambda at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:133:39) &> () at /usr/include/c++/v1/type_traits:4482 #27 __call<(lambda at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:133:39) &> () at /usr/include/c++/v1/__functional_base:349 #28 operator() () at /usr/include/c++/v1/functional:1562 #29 0x0000000001ad64dc in operator() () at /usr/include/c++/v1/functional:1916 #30 work () at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:111 #31 0x0000000001ad6106 in operator() () at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:82 #32 __invoke<(lambda at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:78:17)> () at /usr/include/c++/v1/type_traits:4482 #33 __thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, (lambda at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:78:17)> () at /usr/include/c++/v1/thread:342 #34 __thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, (lambda at /usr/src/contrib/llvm/lib/Support/Parallel.cpp:78:17)> >(void) () at /usr/include/c++/v1/thread:352 #35 0x0000000001628719 in __tsan_thread_start_func (arg=0x7fffffffd578) at /usr/src/contrib/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:982 #36 0x0000000803c70426 in thread_start (curthread=0x7b640000d200) at /usr/src/lib/libthr/thread/thr_create.c:291 #37 0x0000000000000000 in ?? () Backtrace stopped: Cannot access memory at address 0x7fffdfdfd000 Thread 2 (LWP 101247 of process 72867): #0 _nanosleep () at _nanosleep.S:3 #1 0x0000000803c72e2c in __thr_nanosleep (time_to_sleep=0x7fffdfffded0, time_remaining=0x0) at /usr/src/lib/libthr/thread/thr_syscalls.c:289 #2 0x0000000803f7c956 in __usleep (useconds=<optimized out>) at /usr/src/lib/libc/gen/usleep.c:54 #3 0x0000000001625fe1 in __interceptor_usleep (usec=100000) at /usr/src/contrib/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:348 #4 0x0000000001613f7a in __tsan::BackgroundThread (arg=<optimized out>) at /usr/src/contrib/compiler-rt/lib/tsan/rtl/tsan_rtl.cc:180 #5 0x0000000803c70426 in thread_start (curthread=0x7b640000cd00) at /usr/src/lib/libthr/thread/thr_create.c:291 #6 0x0000000000000000 in ?? () Backtrace stopped: Cannot access memory at address 0x7fffdfffe000 Thread 1 (LWP 100343 of process 72867): #0 _umtx_op_err () at /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37 #1 0x0000000803c723e1 in _thr_umtx_timedwait_uint (mtx=0x803c8bc10, id=<optimized out>, clockid=<optimized out>, abstime=<optimized out>, shared=<optimized out>) at /usr/src/lib/libthr/thread/thr_umtx.c:236 #2 0x0000000803c7c038 in cond_wait_user (abstime=0x0, cancel=1, cvp=<optimized out>, mp=<optimized out>) at /usr/src/lib/libthr/thread/thr_cond.c:306 #3 cond_wait_common (cond=<optimized out>, mutex=<optimized out>, abstime=0x0, cancel=1) at /usr/src/lib/libthr/thread/thr_cond.c:366 #4 0x000000000161e026 in __tsan::call_pthread_cancel_with_cleanup (fn=0x803c7c310 <__pthread_cond_wait>, c=0x803c8bc10, m=0xf, abstime=0x0, cleanup=<optimized out>, arg=<optimized out>) at /usr/src/contrib/compiler-rt/lib/tsan/rtl/tsan_platform_linux.cc:344 #5 0x0000000001629184 in cond_wait (thr=0x8042f8900, pc=23236459, si=<optimized out>, fn=0x803c7c310 <__pthread_cond_wait>, c=0x7fffffffd6b8, m=0x7fffffffd6b0, t=0x0) at /usr/src/contrib/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1150 #6 0x0000000001629062 in __interceptor_pthread_cond_wait (c=<optimized out>, m=0x7fffffffd6b0) at /usr/src/contrib/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1161 #7 0x0000000803d40d62 in std::__1::__libcpp_condvar_wait (__cv=0x803c8bc10, __m=0xf) at /usr/src/contrib/libc++/include/__threading_support:286 #8 std::__1::condition_variable::wait (this=0x803c8bc10, lk=...) at /usr/src/contrib/libc++/src/condition_variable.cpp:44 #9 0x0000000001822e5b in wait<(lambda at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:71:21)> () at /usr/include/c++/v1/__mutex_base:375 #10 sync () at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:71 #11 ~Latch () at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:56 #12 ~TaskGroup () at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:75 #13 parallel_for_each<std::__1::__wrap_iter<lld::elf::OutputSection **>, (lambda at /usr/src/contrib/llvm/tools/lld/ELF/Writer.cpp:436:19)> () at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:166 #14 for_each<std::__1::__wrap_iter<lld::elf::OutputSection **>, (lambda at /usr/src/contrib/llvm/tools/lld/ELF/Writer.cpp:436:19)> () at /usr/src/contrib/llvm/include/llvm/Support/Parallel.h:234 #15 parallelForEach<std::__1::vector<lld::elf::OutputSection *, std::__1::allocator<lld::elf::OutputSection *> > &, (lambda at /usr/src/contrib/llvm/tools/lld/ELF/Writer.cpp:436:19)> () at /usr/src/contrib/llvm/tools/lld/include/lld/Common/Threads.h:71 #16 run () at /usr/src/contrib/llvm/tools/lld/ELF/Writer.cpp:435 #17 writeResult<llvm::object::ELFType<llvm::support::little, true> > () at /usr/src/contrib/llvm/tools/lld/ELF/Writer.cpp:132 #18 0x000000000169aa21 in link<llvm::object::ELFType<llvm::support::little, true> > () at /usr/src/contrib/llvm/tools/lld/ELF/Driver.cpp:1126 #19 0x00000000016901fd in main () at /usr/src/contrib/llvm/tools/lld/ELF/Driver.cpp:387 #20 0x000000000168e71d in link () at /usr/src/contrib/llvm/tools/lld/ELF/Driver.cpp:93 #21 0x000000000187371e in main () at /usr/src/contrib/llvm/tools/lld/tools/lld/lld.cpp:115
If I remember correctly, LLVM's parallel-for functions have an issue with reentrancy. Since we call maybeCompress concurrently and that in turn call writeTo which concurrently writes all input sections, parallelFor is called recursively. A short term fix is to not call maybeCompress using paralellForEach. We could call that function with the regular for loop.
(In reply to Rui Ueyama from comment #15) > If I remember correctly, LLVM's parallel-for functions have an issue with > reentrancy. Since we call maybeCompress concurrently and that in turn call > writeTo which concurrently writes all input sections, parallelFor is called > recursively. > > A short term fix is to not call maybeCompress using paralellForEach. We > could call that function with the regular for loop. Yes please, I would very much like to have a high success-rate fix in 6.0.0, even if the performance might not be fully optimal. :)
Could you try this patch for me to see if this will fix your issue? diff --git a/lld/ELF/Writer.cpp b/lld/ELF/Writer.cpp index 1f5c038537a..5feff456ffa 100644 --- a/lld/ELF/Writer.cpp +++ b/lld/ELF/Writer.cpp @@ -432,8 +432,8 @@ template <class ELFT> void Writer<ELFT>::run() { // If -compressed-debug-sections is specified, we need to compress // .debug_* sections. Do it right now because it changes the size of // output sections. - parallelForEach(OutputSections, - [](OutputSection *Sec) { Sec->maybeCompress<ELFT>(); }); + for (OutputSection *Sec : OutputSections) + Sec->maybeCompress<ELFT>(); Script->allocateHeaders(Phdrs);
(In reply to Rui Ueyama from comment #17) > Could you try this patch for me to see if this will fix your issue? > > diff --git a/lld/ELF/Writer.cpp b/lld/ELF/Writer.cpp > index 1f5c038537a..5feff456ffa 100644 > --- a/lld/ELF/Writer.cpp > +++ b/lld/ELF/Writer.cpp > @@ -432,8 +432,8 @@ template <class ELFT> void Writer<ELFT>::run() { > // If -compressed-debug-sections is specified, we need to compress > // .debug_* sections. Do it right now because it changes the size of > // output sections. > - parallelForEach(OutputSections, > - [](OutputSection *Sec) { Sec->maybeCompress<ELFT>(); }); > + for (OutputSection *Sec : OutputSections) > + Sec->maybeCompress<ELFT>(); > > Script->allocateHeaders(Phdrs); Yes, if I put this on top of my diff in comment #14, I cannot get it to hang anymore. Thank you.
Done in r322041.
Ah, woops, I was focusing only on the second race condition, because it sounded like that's what you were concerned with!
(In reply to James Henderson from comment #20) > Ah, woops, I was focusing only on the second race condition, because it > sounded like that's what you were concerned with! No problem, with the first race condition (regarding the fields in SectionPiece) I am not sure what the best approach is. My local hack increased the size of SectionPiece from 16 to 24 bytes, by increasing OutputOffset to the full 64 bits, and adding Live as a separate non-bitfield member. But this makes SectionPiece a lot bigger. Another approach is to reduce the Hash size to 31 bits, and use the left-over bit for Live. But having less bits for the Hash might be bad for collisions? I saw that it is really a 64 bit hash originally, though, so we are already dropping a bunch of bits.
If we reduce the size of the hash from 32 to 31 bits, we need to update https://github.com/llvm-project/llvm-project-20170507/blob/e91029e1169e988177d8734371483e17ba78d293/lld/ELF/SyntheticSections.h#L707. Except that, I believe dropping one bit shouldn't be a problem.
(In reply to Rui Ueyama from comment #22) > If we reduce the size of the hash from 32 to 31 bits, we need to update > https://github.com/llvm-project/llvm-project-20170507/blob/ > e91029e1169e988177d8734371483e17ba78d293/lld/ELF/SyntheticSections.h#L707. > Except that, I believe dropping one bit shouldn't be a problem. Instead of changing the hash size, we used Rafael's idea of swapping the checks, in https://reviews.llvm.org/rL322264. I have created bug 35904 for merging the three changes (r321986, r322041 and r322264) into the 6.0 branch.