Domains and packages with more deps than libc

[duff0097@…]

1. With a domain (even a cross-compile one) you don't need deps that are needed for building only. So autotools.eclass needs editing; otherwise everything that inherits that eclass wants to install autoconf, automake, or libtool. Thus, whatever the pkgcore equivalent of the following is required:

$ diff -Nur /usr/portage/eclass/autotools.eclass autotools.eclass 
--- /usr/portage/eclass/autotools.eclass        2007-07-13 01:06:10.000000000 -0500
+++ autotools.eclass    2007-07-13 01:06:04.000000000 -0500
@@ -35,9 +35,9 @@
        [[ ${WANT_AUTOCONF} == "latest" ]] && WANT_AUTOCONF="2.5"
        export WANT_AUTOCONF
 fi
-DEPEND="${_automake_atom}
-       ${_autoconf_atom}
-       sys-devel/libtool"
+#DEPEND="${_automake_atom}
+#      ${_autoconf_atom}
+#      sys-devel/libtool"
 RDEPEND=""
 unset _automake_atom _autoconf_atom

This isn't a probably solution, but all that is required is using the tools already installed on the system (which is done automatically anyway) - a version check on the 'livefs domain' vdb for those packages is all that is needed.

Fixing this problem allows the resolver to resolve the deps correctly and merge the correct packages (there are more exceptions I'm sure - like perl).

2. The next problem is headers and linking on the rootfs: both are required for compile, but currently the simple case of ROOT/usr/lib or ROOT/usr/include won't work. So time for another hacked script:

$ diff -Nur /usr/lib/python2.4/site-packages/pkgcore/bin/ebuild-env/ebuild-functions.sh ebuild-functions.sh 
--- /usr/lib/python2.4/site-packages/pkgcore/bin/ebuild-env/ebuild-functions.sh2007-07-13 01:02:26.000000000 -0500
+++ ebuild-functions.sh 2007-07-13 01:01:58.000000000 -0500
@@ -127,6 +127,9 @@
         EXTRA_ECONF="--libdir=${CONF_LIBDIR_RESULT} ${EXTRA_ECONF}"
     fi
     local EECONF_CACHE
+    export LDFLAGS="-L/mnt/root2/lib"
+    export CFLAGS="${CFLAGS} -I/mnt/root2/usr/include"
+    export ac_cv_func_malloc_0_nonnull=yes
     echo ${ECONF_SOURCE}/configure \
         --prefix=/usr \
         --host=${CHOST} \

This "fantastic" econf() hack allows the compilation of packages correctly. I've yet to test whether they actually run - but they should. This is necessary for any package that has a dependency on a package in the root of the domain.

3. The last issue is for trimming the install image. INSTALL_MASK only works at merge time. But that won't work. With an 'INSTALL_MASK="*.h"' it simply doesn't install header files at merge time. But if I have a package that depends on one with that INSTALL_MASK set, it won't find the heads, and it will fail. Either the mask simply diverts "*.h" files to an alternate directory to be included at build time, or have it stripped out of root once all packages have been installed. The latter has issues with adding packages to the image at a later time, where the former (an alternate root or temporary root) would still have the files and cause no issues with adding packages after an image has been created.

On top of this I've already seen examples of stripping directories that are unnecessary (e.g. /usr/share/man or /usr/share/doc).

With these changes in mind it will make cross-compile setup and execution very easy, imho.

My current domain setup and these changes has allowed me to rather easily create a usable image for my embedded computer. Thanks.

-duffolonious

[ferringb]

For issue 1, filtering out the deps on the fly isn't hard (MutatedPkg? basically). What will require a bit of tweaking is that the current resolver/pmerge implementation expects its dealing in a single domain- easy to tweak, but means an API breakage (meaning 0.4 is required). In the interim, a custom domain that filters those out (bailing if they can't be found in the parent domain), would that suffice? (barring that, may just force a 0.4 bump for the api).

For issue 2, merging CFLAGS on the fly there will be a bit special; better to have the domain set it itself (just set CFLAGS). Same for ac_cv_func_malloc_0_nonnull=yes; LDFLAGS *should* work also.

If the ebuild is mangling the settings somehow, *could* add a pre_econf hook that runs prior to the configure call (literally right before the echo from above); would allow last minute mangling. Thoughts on that one are welcome- would have to rely on a bashrc approach also.

For issue 3, whats the final image you're generating? Needs work/testing, but the thought there is to do a final 'export' of the image- basically, building the sucker you still need some stuff. When it's just binary data (iow: image is built on disk), could then export it to an iso/tar/fs-path, and have the INSTALL_MASK kick in there.

Thoughts? Would require a third domain (target) for export...

[ferringb]

resurrected duffs msg from the maws of akismet...

Solution to issue 1 is perfectly fine - custom domain that filters out those deps (specified in some set).

Issue 2, adding a pre_econf hook sounds great. I'm not sure if setting LDFLAGS works - I thought I tried it and it didn't work. I'll have to check that out. If that does work, it'll suffice for now. Is there a way to specify variables in CFLAGS (for example): CFLAGS="-O2 -I$root/usr/include" - where $root is the 'root="/path/to/root" in the domain configuration. Or is there a better way to do that? The 'pre_econf' hook will nullify the need for that at least, not sure if it will come up elsewhere.

Issue 3, for starters here is the command(s) to make the image:

//start commands
mkfs.jffs2 -l -U -e 0×20000 -d /mnt/root -D device_table.txt -o rootfs_arm_gumstix
sumtool -e 0×20000 -i rootfs_arm_gumstix -o rootfs_arm_gumstix.jffs2
//end commands

It's made from a directory (the 'root' dir '/mnt/root' in this example). So doing the INSTALL_MASK on creation of the image would require duplicating the root to some temp directory (where the INSTALL_MASK is invoked) - then creating the image from that temp directory. If is works best to create another domain, then so be it.

To make it more flexible I was planning on creating some "mkimage" script in bash or python that allows for specifying the file system type, extra bs, root dir, etc... I think mkfs.jffs2/sumtool is the most complex example. This is as opposed to an ebuild, but it will require access to some domain attributes - root at the very least (using portageq_emulation or equivalent).

With all these changes that should make this a breeze while still be sane to setup.

-duffolonious

[duff0097@…]

(just in case the "rescued" post above missed anything)

Issue 1, I'm not sure what you mean exactly. So you would have a list of packages to ignore in dep resolution. If that is it, I think that will work just fine for now. In my current testing libtool, autoconf, and automake are 90%+ of the problem. The tricky one is perl (where it generates c files for the given arch - it did this on old versions of openssl - perhaps going the way of the dodo).

Issue 2, seems to be resolved well enough as I can see. One "cool" thing would be a better way to specify root in CFLAGS and LDFLAGS, instead of spelling it out: CFLAGS="-O2 -march=blah -I${root}/usr/include" or some such thing. Is there a simple way to have bash expand these vars?

Issue 3, let me first just show some example (actual) image creation commands:

mkfs.jffs2 -l -U -e 0×20000 -d /mnt/root -D device_table.txt -o rootfs_arm_gumstix
sumtool -e 0×20000 -i rootfs_arm_gumstix -o rootfs_arm_gumstix.jffs2

Where the rootfs is in /mnt/root and the "device_table.txt" is a special file that designates devices to create (or regular files) in the rootfs upon creation. With that in mind, a temp dir would have to be created that has been stripped by INSTALL_MASK to create the image from. I think I'm paraphrasing what you have above, but just making sure.

I don't know about the third domain (target) idea - how would that be setup? I'm sure a temp dir in the existing domain would be fine (target_dir) where a script could run off of to generate the above commands correctly.

[jokey]

this is a task for you, ferringb, as you started with it already ;)