This is a meta-bug to track dependencies needed for LLD to be functional as FreeBSD's system linker.
Do you have a list of missing features for this bug?
There are the depends-on PRs: * We need __start_<section> and __stop_<section> as we use linker sets in a few programs * -r relocatable link I just closed PR23035 as the change was reapplied with a fixed test case, but have found a new issue relating to dependency handling (no PR yet): link main.o -l<static> -l<shared>, where libstatic.a depends on libshared.a. LLD reports an undefined symbol from libstatic.a. GNU ld accepts this. Arguably the link order ought to be changed around in our build, but other software will likely encounter this too. For the actual case in FreeBSD the static library was NTP - it builds a libntp.a that depends on libedit, and a binary linked with -ledit -lntp. I also observed one "error: unsupported emulation 'elf_i386_fbsd'." PRs for these two and other issues we find will be added as dependencies of this one.
We also need linker script support, although I don't have a list of the specific linker script features we rely on just yet.
Unknown arguments during world (userland) build: warning: ignoring unknown argument: --version-script=Version.map warning: ignoring unknown argument: -x warning: ignoring unknown argument: --fatal-warnings warning: ignoring unknown argument: --warn-shared-textrel From FreeBSD arm64 kernel link: --- kernel.debug --- linking kernel.debug warning: ignoring unknown argument: -Bdynamic warning: ignoring unknown argument: --no-warn-mismatch warning: ignoring unknown argument: --warn-common warning: ignoring unknown argument: --dynamic-linker warning: ignoring unknown argument: -X SymbolTable: error while merging __bss_start LLVM ERROR: duplicate symbol error
Note that this meta PR was created based on the first version of LLD ELF support, and needs to be revisited for elf2.
Trying with ELF2 turned up the issues below; we're not yet at the point in ELF2's development where it makes sense to track individual PRs for them. * PT_INTERP and ABI notes must be in the first page ELF2 currently rounds up to a page and stores the data for PT_INTERP and the NT_FREEBSD_ABI_TAG and NT_FREEBSD_NOINIT_TAG ELF notes there. These have to be in the first page on FreeBSD. Program Headers: Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align INTERP 0x00000000000010cc 0x00000000000110cc 0x00000000000110cc 0x0000000000000015 0x0000000000000015 R 0x1 [Requesting program interpreter: /libexec/ld-elf.so.1] * Undefined sym in shared library provided by regular object This is the FreeBSD __progname issue fixed in r218259 for the original ELF linker. * Options used in the FreeBSD base system These are options that need to be ignored or implemented (excluding those already ignored by ELF2) --enable-new-dtags --fatal-warnings and --no-fatal-warnings --warn-shared-textrel --output-filetype --version-script -Bsymbolic -r -znodelete --no-warn-mismatch --warn-common -T * Linker script support The FreeBSD base system linker scripts use (at least) the following set: ALIGN ENTRY GROUP KEEP OUTPUT_ARCH OUTPUT_FORMAT SEARCH_DIR SECTIONS PROVIDE (http://lists.llvm.org/pipermail/llvm-dev/2015-October/091072.html)
--enable/disable-new-dtags is r249428
What kind of expressions are used in SECTIONS linker script directive?
> What kind of expressions are used in SECTIONS linker script directive? Examples can be found here: https://svnweb.freebsd.org/base/head/sys/conf/ For amd64: https://svnweb.freebsd.org/base/head/sys/conf/ldscript.amd64?view=log Some examples: kernphys = CONSTANT (MAXPAGESIZE); . = kernbase + kernphys + SIZEOF_HEADERS; .interp : AT (kernphys + SIZEOF_HEADERS) { *(.interp) } .hash : { *(.hash) } .rel.text : { *(.rel.text .rel.text.* .rel.gnu.linkonce.t.*) } .init : { KEEP (*(.init)) } =0x90909090 PROVIDE (__etext = .); .eh_frame : ONLY_IF_RO { KEEP (*(.eh_frame)) } . = ALIGN (CONSTANT (MAXPAGESIZE)) - ((CONSTANT (MAXPAGESIZE) - .) & (CONSTANT (MAXPAGESIZE) - 1)); . = DATA_SEGMENT_ALIGN (CONSTANT (MAXPAGESIZE), CONSTANT (COMMONPAGESIZE)); .preinit_array : { PROVIDE_HIDDEN (__preinit_array_start = .); KEEP (*(.preinit_array)) PROVIDE_HIDDEN (__preinit_array_end = .); } . = ALIGN(. != 0 ? 64 / 8 : 1); .debug_pubtypes 0 : { *(.debug_pubtypes) }
With the change in http://reviews.llvm.org/D13637 LLD can now link a FreeBSD "hello world." The current userland world build fails due to unhandled R_X86_64_TLSGD: --- libc.so.7 --- unrecognized reloc 19 cc: error: linker command failed with exit code 1 (use -v to see invocation) The kernel build fails in the linker script parser: --- kernel.full --- linking kernel.full ) expected, but got ,
(In reply to comment #10) > With the change in http://reviews.llvm.org/D13637 LLD can now link a FreeBSD > "hello world." > > The current userland world build fails due to unhandled R_X86_64_TLSGD: > > --- libc.so.7 --- > > unrecognized reloc 19 > > cc: error: linker command failed with exit code 1 (use -v to see invocation) > Michael Spencer is working on it -- there's an initial patch under review so that should be unlocked soon. > > > The kernel build fails in the linker script parser: > > --- kernel.full --- > > linking kernel.full > > ) expected, but got , http://reviews.llvm.org/D13668
Update, With the WIP -r patch (http://reviews.llvm.org/D14382) linking static libpam produces a new error: $ ld -o openpam_static_modules.o -r --whole-archive openpam_static.o ../modules/pam_chroot/libpam_chroot.a ../modules/pam_deny/libpam_deny.a [...] FDE doesn't reference another section If I'm able to investigate and produce a small reproduction case I will submit a new PR. Also, the lack of GNU symbol version support is now becoming a blocking issue for making further progress on FreeBSD - our libc has some cases where it is not usable without a symver-supporting linker.
Do you keep track of the progress? I'm interested in the numbers of packages we can and can't link with lld (with test success).
> Do you keep track of the progress? I'm interested in the numbers of packages we > can and can't link with lld (with test success). We're not yet at the point of trying to link the ports tree with lld -- I'm working on the base system now. After recent commits and the WIP -r patch I've been able to reduce the set of hacks in my build tree to a small handful: - default library search paths - ld.bfd includes /lib as a built-in search path, for lld I need to specify it explicitly (in the libc.so DSO linker script) - skip libpam/static_modules due to "FDE doesn't reference another section" as mentioned in an earlier comment - skip the i386 boot loaders due to lack of -N support in lld (proposed implementation for the old elf linker in http://reviews.llvm.org/D968) - skip the gdb build - it failed because lld does not provide an _etext symbol - change -Tscript to -T script - remove the EFI boot loaders because they require linker script functionality not yet implemented - skip uathload as it relies on an unusual use of -r (converting a binary blob into an ELF object) For now I'm not trying to build the kernel with lld.
(In reply to comment #14) > > Do you keep track of the progress? I'm interested in the numbers of packages we > > can and can't link with lld (with test success). > > We're not yet at the point of trying to link the ports tree with lld -- I'm > working on the base system now. > > After recent commits and the WIP -r patch I've been able to reduce the set > of hacks in my build tree to a small handful: > > - default library search paths - ld.bfd includes /lib as a built-in search > path, for lld I need to specify it explicitly (in the libc.so DSO linker > script) > - skip libpam/static_modules due to "FDE doesn't reference another section" > as mentioned in an earlier comment > - skip the i386 boot loaders due to lack of -N support in lld (proposed > implementation for the old elf linker in http://reviews.llvm.org/D968) > - skip the gdb build - it failed because lld does not provide an _etext > symbol > - change -Tscript to -T script > - remove the EFI boot loaders because they require linker script > functionality not yet implemented > - skip uathload as it relies on an unusual use of -r (converting a binary > blob into an ELF object) If you can open individual bugs for each one that would be awesome :-)
> If you can open individual bugs for each one that would be awesome :-) Unlike the recent PRs I've opened up I'm not convinced all of these are bugs (in lld): > - default library search paths - ld.bfd includes /lib as a built-in search > path, for lld I need to specify it explicitly (in the libc.so DSO linker > script) The clang driver passes -L/usr/lib to ld and we expect to resolve -llib args from there; this issue only happens because /usr/lib/libc.so is a linker script that contains: GROUP ( libc.so.7 libc_nonshared.a libssp_nonshared.a ) I think it should probably be: GROUP ( /lib/libc.so.7 /usr/lib/libc_nonshared.a /usr/lib/libssp_nonshared.a ) > - skip libpam/static_modules due to "FDE doesn't reference another section" > as mentioned in an earlier comment This one indeed looks like an lld issue and I'll submit a PR once I have at least some detail. > - skip the i386 boot loaders due to lack of -N support in lld (proposed > implementation for the old elf linker in http://reviews.llvm.org/D968) -N should be easy to implement, but I'm not sure how useful or widely used it is, and it seems reasonable for us to rework our boot components to avoid it. > - skip the gdb build - it failed because lld does not provide an _etext > symbol PR 26729 > - change -Tscript to -T script PR 26730 We can easily make this change in FreeBSD, but as we start trying to compile large sets of packages with lld I'm sure this will come up again. > - remove the EFI boot loaders because they require linker script > functionality not yet implemented Will submit a PR referencing specific missing functionality > - skip uathload as it relies on an unusual use of -r (converting a binary > blob into an ELF object) While I believe -r support is important in lld I'm not sure I can argue for this use, and will probably change the FreeBSD build to use objcopy instead.
The set of workarounds I'm using to try building FreeBSD world with lld is in my github FreeBSD tree: https://github.com/emaste/freebsd/tree/lld-buildworld Other than those changes I'm testing with upstream FreeBSD and upstream lld. The major outstanding issue is lack of expression support in linker scripts - in my tree the kernel is being linked with ld.bfd for this reason. There are also a few other components that are disabled or linked with ld.bfd. I've built a VM image from this tree and large portions work; the system boots to a login prompt, shells work, clang works, etc. Anything linked against libxo (which includes ls and ps) segfaults - PR 26818. ps is used in many places in the startup scripts and thus results in a lot of console spam. Once PR 26818 is fixed I believe enough will be working that we can test lld in a FreeBSD ports experimental run, trying to build the ~25K third party software packages in the ports tree with lld.
Thank you very much for the updates. Linker script is indeed the next big one (and hopefully the last one), but before start working on it, I'm trying to support versioned symbols.
> I'm trying to support versioned symbols. Great. For my immediate testing I'm building a full system w/o symver and the lack of support isn't holding me up -- I might have a couple of workarounds that I'll need to apply, but it's not a significant problem for buildworld (userland). Using a ld.bfd-linked kernel fine for testing. Indeed, versioned symbol support will be very important for us if we're to provide a /usr/bin/ld.lld, in order to interoperate with system components -- particularly libc.so -- linked by GNU ld.
little note about: >- skip libpam/static_modules due to "FDE doesn't reference another section" >as mentioned in an earlier comment That should be fixed in r262590. That message was shown becauses eh_frame sections were proccessed during generating relocatable output and from that revision it is not happens anymore as well as this adds support for generating relocations against local symbols which were sections in that case.
In r262910, I add /usr and /usr/lib as default search paths. I think that should resolve one issue for FreeBSD.
Changes have been committed to FreeBSD to address some of these issues: * Library search paths, added to lld and then reverted. Now solved by leaving the full paths in the /usr/lib/libc.so linker script: https://svnweb.freebsd.org/changeset/base/296921 * ld -r to convert a binary object to ELF. Solved by rewriting the utility to avoid having a compiled-in binary object: https://svnweb.freebsd.org/changeset/base/296889 I'm now starting to test FreeBSD/arm64 (AArch64) buildworld with lld. It currently fails when building libc: /usr/local/aarch64-freebsd/bin/ld: getutxent.So(.debug_info+0x3c): R_AARCH64_ABS64 used with TLS symbol udb /usr/local/aarch64-freebsd/bin/ld: getutxent.So(.debug_info+0x59): R_AARCH64_ABS64 used with TLS symbol uf /usr/local/aarch64-freebsd/bin/ld: utxdb.So(.debug_info+0x5c): R_AARCH64_ABS64 used with TLS symbol futx_to_utx.ut /usr/local/aarch64-freebsd/bin/ld: jemalloc_tsd.So(.debug_info+0x3d): R_AARCH64_ABS64 used with TLS symbol __je_tsd_tls /usr/local/aarch64-freebsd/bin/ld: jemalloc_tsd.So(.debug_info+0x12dc): R_AARCH64_ABS64 used with TLS symbol __je_tsd_initialized /usr/local/aarch64-freebsd/bin/ld: xlocale.So(.debug_info+0x404): R_AARCH64_ABS64 used with TLS symbol __thread_locale /usr/local/aarch64-freebsd/bin/ld: setrunelocale.So(.debug_info+0x3d): R_AARCH64_ABS64 used with TLS symbol _ThreadRuneLocale Note that patches are open for arm64 TLS relocation work: http://reviews.llvm.org/D16201 http://reviews.llvm.org/D17980 http://reviews.llvm.org/D18026 http://reviews.llvm.org/D18031
(In reply to comment #16) > > - skip the i386 boot loaders due to lack of -N support in lld (proposed > > implementation for the old elf linker in http://reviews.llvm.org/D968) > > -N should be easy to implement, but I'm not sure how useful or widely used > it is, and it seems reasonable for us to rework our boot components to avoid > it. So, just in case. We are not going to support -N, right ? (otherwise I would take that).
Because -N option is for compatibility with old Unix systems that have died out in 80s or 90s. If you still want that behavior for some special case (such as bootloader or something), you should use a linker script instead.
(In reply to comment #24) > Because -N option is for compatibility with old Unix systems that have died > out in 80s or 90s. If you still want that behavior for some special case > (such as bootloader or something), you should use a linker script instead. Yes, I assumed that I will be changing the FreeBSD build to implement what's needed via a linker script. I believe lld's linker script support is not yet fully able to implement -N's functionality and I will continue to link these components with GNU ld until it can.
(In reply to comment #24) > Because -N option is for compatibility with old Unix systems that have died > out in 80s or 90s. If you still want that behavior for some special case > (such as bootloader or something), you should use a linker script instead. That's why I asked :)
Attempting to link FreeBSD/mips64 with lld reports: > warning: unknown argument: -EB I do not yet know why there's an explicit big/little endian argument passed in by Clang.
It looks like lib/Driver/Tools.cpp always produces -m elf{32,64}{bt,lt}smip flags, so maybe -EB/EL are just redundant?
> It looks like lib/Driver/Tools.cpp always produces -m elf{32,64}{bt,lt}smip flags, so maybe -EB/EL are just redundant? Yes, I'm having trouble imagining how -EB/-EL could be used, where endianness is not already determined by either -m or the object file(s). For testing I've just added -EB and -EL as ignored options.
Moving to "ELF" bugzilla component (instead of generic "All Bugs" for LLD).
Update on linking the FreeBSD kernel with -- following functionality used by the FreeBSD/amd64 kernel link script not yet in lld: - DATA_SEGMENT_RELRO_END https://reviews.llvm.org/D22676 - output section address assignment https://reviews.llvm.org/D22689 - AT (load address) https://reviews.llvm.org/D19272 - SIZEOF_HEADERS http://llvm.org/pr28688 - SORT - EXCLUDE_FILE - CONSTRUCTORS obj file specification - e.g. KEEP (*crtbegin.o(.dtors))
One more issue for the FreeBSD/amd64 kernel linker script: expressions with symbols on the right hand side of an assignment https://reviews.llvm.org/D22759
status update at lld r280348 on x86-64, based on the workarounds in my test tree: issues / missing functionality in lld: - [chars] wildcards in version scripts - ~ (bitwise not) in linker script expressions - -nostdlib, -f, -b binary options - MIPS mxgot or multigot - can't create dynamic relocation R_386_GOTOFF against readonly segment errors linking the 32-bit libraries FreeBSD issues / issues needing triage: - our system demangler produces non-standard output for some symbols, so extern "C++" in version scripts sometimes fails to match - build for boot loader components needs to be updated to use linker scripts instead of obsolete options - rescue (a multi-tool binary, like busybox) fails to build, investigation needed - kernel links wtih lld but I have not tested it yet
(In reply to comment #33) > status update at lld r280348 on x86-64, based on the workarounds in my test > tree: > > issues / missing functionality in lld: + Orphans handling I think. D23352
There are currently two issues remaining when linking all of FreeBSD head world + kernel with lld head (r288670), on amd64: 1) PR31277 - lld segfault when compiling the 32-bit compat version of ldd 2) EFI loaders, when linked with LLD, either hang or report "Invalid Parameter." The EFI loaders are built with a slightly convoluted set of steps: a) linked as ELF files with debug b) elfcopy/objcopy used to create a standalone debug file and stripped file % objcopy --strip-debug loader.sym.full loader.sym c) elfcopy/objcopy used to convert the stripped file to a PE/COFF EFI file % SOURCE_DATE_EPOCH=1451606400 objcopy \ -j .peheader -j .text -j .sdata -j .data \ -j .dynamic -j .dynsym -j .rel.dyn \ -j .rela.dyn -j .reloc -j .eh_frame \ --output-target=efi-app-x86_64 loader.sym.full loader.efi George R. reported errors from GNU objcopy at step (b), and success with a manual build when skipping (b) and running (c) on the non-separated-debug output from (a). I will submit a PR for the EFI loader issue once I have some more detail and am convinced it is something that ought to have an LLD change.
Created attachment 17721 [details] loader repro
(In reply to comment #35) > George R. reported errors from GNU objcopy at step (b), and success with a > manual build when skipping (b) and running (c) on the non-separated-debug > output from (a). Yes, I just retested loaders with different objcopy/elfcopy and -N flag. Script for loader has next calls: objcopy --strip-debug loader.sym.full loader.sym objcopy \ -j .peheader -j .text -j .sdata -j .data \ -j .dynamic -j .dynsym -j .rel.dyn \ -j .rela.dyn -j .reloc -j .eh_frame \ --output-target=efi-app-x86_64 loader.sym loader.efi My results using lld head (r288757): 1) Using GNU objcopy 2.17.50 [FreeBSD] 2007-07-03, objcopy errors out: BFD: loader.sym: section `.dynstr' can't be allocated in segment 5 objcopy: loader.sym: Bad value BFD: loader.sym: section `.dynstr' can't be allocated in segment 5 2) The same objcopy as above, but add -N (omagic). Loader works, no errors from objcopy. 3) Using GNU objcopy (GNU Binutils) 2.27 it works both with or without -N ! 4) Using elfcopy (elftoolchain HEAD FreeBSD svn:3500) instead of objcopy: Either with or without -N loader hangs on startup, without any errors from elfcopy side. So my conclusion is that looks elfcopy is just broken, since objcopy 2.27 works fine. Just in case - my reproduce files with script for testing are in the post above.
(In reply to comment #37) > > So my conclusion is that looks elfcopy is just broken, since objcopy 2.27 > works fine. > Just in case - my reproduce files with script for testing are in the post > above. Thanks for this, it turns out it's just a plain bug in ELF Tool Chain's elfcopy, now reported in https://sourceforge.net/p/elftoolchain/tickets/541/. As far as I can tell no LLD changes are necessary to make this work. (I'd still like to come back to the multiple .data sections and related differences vs. GNU ld later on).
(In reply to comment #38) > > Thanks for this, it turns out it's just a plain bug in ELF Tool Chain's > elfcopy, now reported in > https://sourceforge.net/p/elftoolchain/tickets/541/. That is now fixed and imported as FreeBSD r310634, and I can boot using a LLD-linked and elfcopy-converted loader.efi. An issue remains with an LLD-linked legacy 32-bit BIOS boot path. I'm currently bisecting building certain boot components with GNU ld vs LLD to determine what's failing.
(In reply to comment #39) > > An issue remains with an LLD-linked legacy 32-bit BIOS boot path. I'm > currently bisecting building certain boot components with GNU ld vs LLD to > determine what's failing. This was due to an old assumption in FreeBSD's btxldr (boot code protected mode wrapper) that ELF objects have two PT_LOAD segments. LLD uses an rodata segment by default and so produced three PT_LOAD segments, and btxldr stopped processing after the second. The fix is in https://reviews.freebsd.org/D8929 With that change applied I can link the entirety of the FreeBSD/amd64 base system (userland world and kernel) with LLD.
W00t! What a milestone! Thanks Ed and everyone at the lld community! How much of ports do we need to test before shipping an lld-freebsd alpha version for testing? It would be good to have a Kvm image somewhere to give it a go.
(In reply to comment #41) > W00t! What a milestone! Thanks Ed and everyone at the lld community! Thanks! I'm amazed at how far lld New-ELF has come in about a year and a half. > How much of ports do we need to test before shipping an lld-freebsd alpha > version for testing? Rafael's done an initial experimental Poudriere FreeBSD package build with lld head, and found almost 20K out of 26K ports built successfully. I'm now looking at getting CI running to test this on an ongoing basis. But, I think we're at the point where an experimental build makes sense. > It would be good to have a Kvm image somewhere to give it a go. We can quite easily produce VM images from FreeBSD-HEAD, which has Clang/LLVM/LLDB/LLD 3.9.1 (less useful). I'll see about hacking something together - either copying in an out-of-tree lld as with the Poudriere experiment, or perhaps dim@ is planning to import a new LLVM snapshot in the FreeBSD tree and I'll build from that.
> With that change applied I can link the entirety of the FreeBSD/amd64 base > system (userland world and kernel) with LLD. That is truly awesome! I am almost looking forward to the end of my vacations to get back to the poudriere run :-) What is the process for getting patches into ports? By now I assume most issues will be things like library order dependency which is better fixed in the package.
(In reply to comment #43) > > What is the process for getting patches into ports? By now I assume most > issues will be things like library order dependency which is better fixed in > the package. I suggest that during development we collect patches in a local git repo -- for example, I've started here for my Poudriere run https://github.com/emaste/freebsd-ports/commits/ports-lld The best case scenario is that we get any required changes committed upstream first, and I've done that with the pkg change here: https://github.com/freebsd/pkg/pull/1523 and bug report https://github.com/freebsd/pkg/issues/1522 (ld.bfd can link pkg despite this issue so I've submitted bug 31495, but it's really a bug in pkg.) Assuming that the changes are not accepted upstream (or move too slowly, as expected with libtool) then we can just submit a FreeBSD ports PR with the desired patch.
(In reply to comment #44) > (In reply to comment #43) > > > > What is the process for getting patches into ports? By now I assume most > > issues will be things like library order dependency which is better fixed in > > the package. > > I suggest that during development we collect patches in a local git repo -- > for example, I've started here for my Poudriere run > https://github.com/emaste/freebsd-ports/commits/ports-lld Interesting. I will send you pull requests and open upstream bugs. How do you get poudriere to use it? I thought it only had support for svn.
(In reply to comment #45) > Interesting. I will send you pull requests and open upstream bugs. How do > you get poudriere to use it? I thought it only had support for svn. Poudriere has support for fetching the ports tree from git (but not the base system src tree, for some reason), but I'm not using that. You can also configure Poudriere to build an from externally maintained ports tree, using a command like: % poudriere ports -c -F -f none -M /path/to/ports/tree -p lld This info is from https://github.com/freebsd/poudriere/wiki/use_system_ports_tree. It's not easy to find and not clear from the normal Poudriere documentation (e.g. the man pages). Using this approach you can then update the ports tree any way you see fit and then rerun. In my Poudriere run perl5 failed to build due to dtrace relocs /usr/bin/ld: error: dtrace_mini.o:(.SUNW_dof+0x180): unrecognized reloc 8 and that took out 19743 downstream ports. (I knew many ports would have a dependency chain passing through perl, but I'm surprised it's that extensive.)
> In my Poudriere run perl5 failed to build due to dtrace relocs This is the issue we've been discussing with markj; I believe dtrace support for perl was just recently enabled in the ports tree which would explain why Rafael didn't see it. I'll let my current run finish, and then restart when Mark's fix is available.
(In reply to comment #47) > > In my Poudriere run perl5 failed to build due to dtrace relocs > > This is the issue we've been discussing with markj; I believe dtrace support > for perl was just recently enabled in the ports tree which would explain why > Rafael didn't see it. I'll let my current run finish, and then restart when > Mark's fix is available. I actually hacked lld to ignore the broken dtrace created relocs. I posted the patch to the list.
(In reply to comment #48) > > I actually hacked lld to ignore the broken dtrace created relocs. I posted > the patch to the list. Yep, I saw that. I'm trying to look at this from the other side, so I want to avoid applying hacks to lld and will apply WIP patches / hacks / etc. to the FreeBSD ports tree for faults that appear to be FreeBSD (or other 3rd party) issues. It now looks like libtool is responsible for the majority of my failed / skipped ports. Unless we really think we'll add "not GNU" and other hacks to lld we're going to have to address libtool limitations upstream and in the FreeBSD tree. I did look into libtool a few weeks ago, but unfortunately haven't yet managed to produce a patch suitable for sending upstream.
I've just added 31716 and 31762 to Depends On. They are powerpc64 example issues. (I tend to explore and report things for powerpc64 and powerpc FreeBSD.) I had not known about 23214's META for LLD and had already listed those submittals in 25780's META for clang targeting powerpc64 and powerpc some time ago. (clang with the old system binutils in FreeBSD is insufficient to allow clang to be used as the system compiler targeting powerpc64 without an external binutils of some kind.) At least for 25780 (clang's FreeBSD META) Ed Maste has requested that I make additions to that Depends On list myself. I expect that would apply here as well.
A brief high-level status update: For the set of armv7l, arm64, i386, x86_64 LLD can now link the FreeBSD kernel and modules. On all but i386 the kernel and modules have been tested to boot and confirm basic functionality. i386 kernel has been booted. Kernel modules link but are untested. LLD 4.0.0 is in the FreeBSD base system source tree and is: 1) The default linker used to build the FreeBSD/arm64 base system (kernel+modules, userland) and installed as /usr/bin/ld, in HEAD and will be in the upcoming 11.1 release. 2) Usable for the FreeBSD/amd64 (x86-64) base system, with investigation on failures in the 27000 pieces of software in the ports tree ongoing. I am using LLD as /usr/bin/ld on my daily driver FreeBSD desktop and laptop. LLD HEAD has been imported into the FreeBSD clang500-import branch and is: 3) Usable as the linker for the FreeBSD/armv6 kernel+modules. Note that FreeBSD does not yet have a separate target for armv7; TARGET_ARCH=armv6 is what we use for the majority of our 32-bit arm devices. Linking libc fails due to PR 31159; testing has been done on an lld-linked kernel+modules with a ld.bfd-linked userland. 4) Usable as the linker for the FreeBSD/i386 kernel+modules. lld-linked kernel boots. Modules build but are untested. Linking the base system succeeds, but applications all segfault with assertion errors from jemalloc. No further investigation or testing has yet been done.
Two recent regressions affecting FreeBSD/arm64 are described in: https://reviews.llvm.org/rL307100 https://reviews.llvm.org/rL307364
FYI as of r327783 (https://reviews.freebsd.org/rS327783) FreeBSD is using lld as the bootstrap linker, to link the kernel and base system userland libraries and binaries on amd64. We're still using ld.bfd as /usr/bin/ld and all of the ports collection will thus be linked with bfd today, but I expect to change this in another couple of weeks. I expect we will then make the same series of changes for i386, and armv7 after. (FreeBSD/arm64 has been using lld as the bootstrap linker and /usr/bin/ld for some time already.) After that we'll move on to MIPS and PowerPC. Big thanks to everyone who's worked on lld, and in particular to those who have handled the bug reports that are listed as dependencies of this. (I expect I'll keep this PR open to track any final issues until we're using lld as the bootstrap linker and /usr/bin/ld on all Tier-1 architectures.)
(In reply to emaste from comment #53) > . . . > > After that we'll move on to MIPS and PowerPC. > > . . . Just an FYI about the status of clang for powerpc and powerpc64 for FreeBSD, taking specific example points (not everything). . . clang for powerpc and powerpc64 does not correctly implement __builtin_eh_return [llvm bugzilla 28644]. As a consequence if clang is used to build the code that uses __builtin_eh_return then throwing a C++ exception causes the program to crash, even for something as simple as: #include <exception> int main(void) { try { throw std::exception(); } catch (std::exception& e) {} return 0; } (As I understand __builtin_eh_return is basically unimplemented as far as its consequences go in what is generated.) Also, for powerpc64, the switch to clang 5.0.1 changed to use .rela.plt and R_PPC64_JMP_SLOT in kernel module builds but the kernel does not handle dynamically loading such, crashing the kernel during the load. (The older .rela.dyn with R_PPC64_RELATIVE and R_PPC64_ADDR64 usage was working.) I do not know enough to know which side should change for this issue and so have not submitted it to llvm so far.
(In reply to Mark Millard from comment #54) > > Also, for powerpc64, the switch to clang 5.0.1 changed > to use .rela.plt and R_PPC64_JMP_SLOT in kernel module > builds but the kernel does not handle dynamically > loading such, crashing the kernel during the load. > (The older .rela.dyn with R_PPC64_RELATIVE and > R_PPC64_ADDR64 usage was working.) I do not know > enough to know which side should change for this > issue and so have not submitted it to llvm so far. I wonder if it can be related with fact that PPC64 incorrectly works with PLT sections atm (D41613 addresses this).
(In reply to George Rimar from comment #55) > (In reply to Mark Millard from comment #54) > > > > Also, for powerpc64, the switch to clang 5.0.1 changed > > to use .rela.plt and R_PPC64_JMP_SLOT in kernel module > > builds but the kernel does not handle dynamically > > loading such, crashing the kernel during the load. > > (The older .rela.dyn with R_PPC64_RELATIVE and > > R_PPC64_ADDR64 usage was working.) I do not know > > enough to know which side should change for this > > issue and so have not submitted it to llvm so far. > > I wonder if it can be related with fact that PPC64 incorrectly > works with PLT sections atm (D41613 addresses this). I finally updated to a more modern FreeBSD vintage that is based on clang 6 as the system compiler: # clang --version FreeBSD clang version 6.0.0 (branches/release_60 325330) (based on LLVM 6.0.0) Target: x86_64-unknown-freebsd12.0 Thread model: posix InstalledDir: /usr/bin It still gets the .rela.plt and R_PPC64_JMP_SLOT use that 5.0.1 did. (Cross compile based buildworld buildkernel .) I've not checked if branches/release_60 325330 should have D41613 or not: I'm just reporting FreeBSD's status for the issue for now.
(In reply to George Rimar from comment #55) > (In reply to Mark Millard from comment #54) > > > > Also, for powerpc64, the switch to clang 5.0.1 changed > > to use .rela.plt and R_PPC64_JMP_SLOT in kernel module > > builds but the kernel does not handle dynamically > > loading such, crashing the kernel during the load. > > (The older .rela.dyn with R_PPC64_RELATIVE and > > R_PPC64_ADDR64 usage was working.) I do not know > > enough to know which side should change for this > > issue and so have not submitted it to llvm so far. > > I wonder if it can be related with fact that PPC64 incorrectly > works with PLT sections atm (D41613 addresses this). I looked at D41613 a little and it appears to just be for lld. The above type of problem happens even if devel/powerpc64-binutils (or equivalent) is used with clang 5.0.1 or clang 6. (So far, this is the normal way to experiment with clang targeting powerpc64, at least in my experiments.) I'm not clear on just were the responsibilities are for this, both for FreeBSD vs. llvm and for clang vs. lld vs. both on the llvm side of things. Primarily I'm just reporting the overall mismatch, not resolving how and where it should be handled.
I added a see-also back to the FreeBSD PR tracking outstanding issues in switching to LLD as the system linker -- https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229050 Very brief per-arch summary, for FreeBSD-current (i.e., SVN head): - arm64 and amd64 use lld for the bootstrap and system linker (i.e., done) - i386 was blocked on LLVM PR 37735 which is now fixed and cherry-picked into FreeBSD -- testing is underway before making the switch - armv7 is blocked on LLVM PR 36009 - armv6, arm (v4/v5) blocked on use of movt/movw, extended branch encoding, blx wh- mips successful proof of concept demonstrated - ppc/power userland linking fails with various relocation erros - sparc64 not investigated - riscv64 waiting on upstream support
Up-to-date FreeBSD-current status: - lld is the bootstrap and system linker for arm64, armv7, amd64 (aka x86_64), and i386 - armv6 has not yet been switched but lld 7 is now in the FreeBSD base system and we should be able to enable it there - arm (v4/v5) support is expected to be retired and will probably not switch to lld before then - mips is still at "proof of concept demonstrated"; we will likely be able to switch mips64 to Clang + lld soon - no progress on ppc/power, sparc64, riscv64
lld is (or is functional as, and soon will be) the bootstrap and system linker for all FreeBSD supported architectures except 32-bit powerpc and sparc64
(In reply to Mark Millard from comment #54) > (In reply to emaste from comment #53) > > > . . . > > > > After that we'll move on to MIPS and PowerPC. > > > > . . . > > Just an FYI about the status of clang for > powerpc and powerpc64 for FreeBSD, taking > specific example points (not everything). . . > > clang for powerpc and powerpc64 does not correctly > implement __builtin_eh_return [llvm bugzilla 28644]. > As a consequence if clang is used to build the code > that uses __builtin_eh_return then throwing a C++ > exception causes the program to crash, even for > something as simple as: > > #include <exception> > > int main(void) > { > try { throw std::exception(); } > catch (std::exception& e) {} > return 0; > } > > (As I understand __builtin_eh_return is basically > unimplemented as far as its consequences go in what > is generated.) > > Also, . . . FreeBSD has implicitly sidestepped the __builtin_eh_return issue by using llvm's libunwind, which does not depend on __builtin_eh_return causing the exception handling scratch registers r3-r6 to have cfi information. It is still the case that a libunwind library that is designed to require that cfi information can not be usefully built and used via clang: __builtin_eh_return does not lead to the cfi content required (Itanium ABI variant for powerpc64 or 32-bit powerpc). For FreeBSD, that should only matter if someone tried to build FreeBSD via WITHOUT_LLVM_LIBUNWIND= .
lld is now used as the system linker for all archs supported by FreeBSD One depends-on issue was not closed (PR30267). It has been worked around in FreeBSD and has been moved to the see-also list here. Thank you to everyone who helped make this possible!