Ticket #235 (closed defect: fixed)

Opened 7 weeks ago

Last modified 4 weeks ago

CHOST is overwritten

Reported by: jkes Owned by:
Priority: normal Milestone:
Component: pkgcore Version: 0.5.9
Keywords: cross-compile Cc:

Description

Version 0.5.9 !!!

I have tried to cross-compile with pmerge.

The values CHOST, CBUILD or CTARGET, which are defined in a domain in a config file, are overwritten in ebd.py.

It works with following changes 

=== modified file 'pkgcore/ebuild/ebd.py'                        
--- pkgcore/ebuild/ebd.py       2009-12-14 06:16:44 +0000        
+++ pkgcore/ebuild/ebd.py       2010-01-20 12:35:29 +0000        
@@ -97,7 +97,7 @@                                                
         for x in ('chost', 'cbuild', 'ctarget'):                
             val = getattr(pkg, x)                               
             if val is not None:                                 
-                self.env[x.upper()] = val                       
+                self.env.setdefault(x.upper(),val)              
                                                                 
         if "PYTHONPATH" in os.environ:                          
             self.env["PYTHONPATH"] = os.environ["PYTHONPATH"]

Now CHOST, CBUILD or CTARGET are only overwritten, when there are not defined.

Change History

comment:1 Changed 7 weeks ago by ferringb

  • version changed from 0.5.8 to 0.5.9

I'm not convinced this is proper- how are you seeing this exactly?

Basically, the pkgs configuration wrapping (or in the case of binpkgs, hardcoded chost/cbuild/ctarget) ought to be valid- those settings work their way down from the domain after all.

Off the top of my head, I don't quite see how it's possible to get a situation where domain settings (where those env values come from) differ from the pkg- exempting binpkgs, which the values shouldn't change for.

Comments?

comment:2 Changed 6 weeks ago by jkes

The config file pkgcore_arm_1.conf contains all the definitions I am using.

Create some files and folders in a new and empty folder /test4 

mkdir -p /test4/packages
mkdir -p /test4/var/db/pkg
mkdir -p /test4/var/lib/portage
touch /test4/var/lib/portage/world

start the debugger with 

pmerge -dv wget

set a breakpoint in line 99 in pkgcore/ebuild/ebd.py 

        for x in ('chost', 'cbuild', 'ctarget'):
            val = getattr(pkg, x)    
            if val is not None:   #  <<< set breakpoint here     
                self.env.setdefault(x.upper(),val)

When the breakpoint is hit for the first time the values of CHOST, CBUILD and CTARGET are:
self.env.CHOST: i686-pc-linux-gnu
self.env.CBUILD: arm-unknown-linux-gnu
self.env.CTARGET: not defined

By stepping through the code with the debugger I get the values of variable val for
chost: arm-unknown-linux-gnu
cbuild: arm-unknown-linux-gnu
ctarget: arm-unknown-linux-gnu

When I change the name in 'livefs domain' to
CHOST 'dummy1'
the values of chost, cbuild and ctarget change also to 'dummy1'.
chost, cbuild and ctarget get their value only from CHOST in 'livefs domain' in pkgcore_arm_1.conf

My workaround for this is 

self.env.setdefault(x.upper(),val)

in line 100 in ebd.py.

I am using the following /etc/pkgcore.conf file 

[autoload-dhcpformat]
class=pkgcore.config.basics.parse_config_file
parser=pkgcore.config.dhcpformat.config_from_file
path=/etc/pkgcore_arm_1.conf

comment:3 Changed 6 weeks ago by jkes

The following config file pkgcore_arm_1.conf contains all the definitions I am using. 

# Definition for an arm cross-compiler
#
# root and config root are /test4
#
# I have created this file by using:
# export CONFIG_ROOT=/test4
# pconfig dump
# on an existing file /test4/etc/make.conf.
# I have been able to use /test4/etc/make.conf with emerge-wrapper.
# 
# The lines in the output of 'pconfig dump' which I had to change 
# afterwards to get it working are marked with ERROR 
#
# When I use this file I don't use ROOT or CONFIG_ROOT anymore
#

'/usr/portage' {           
    # typename of this section: repo
    class pkgcore.ebuild.repository.UnconfiguredTree;
    # type: refs:cache                               
    cache 'portdir cache';                           
    # type: list                                     
    default_mirrors 'ftp://ftp.gentoo.mesh-solutions.com/gentoo//distfiles';                                                                                                                                                                                   
    # type: ref:eclass_cache                                                                                                                                                                    
    # eclass_cache 'eclass stack';        # ERROR: alias doesn't work here
    eclass_cache '/usr/portage/eclass';
    # type: bool                                                                                                                                                                                
    ignore_paludis_versioning False;                                                                                                                                                            
    # type: str                                                                                                                                                                                 
    location '/usr/portage';                                                                                                                                                                    
    # type: lazy_ref:syncer                                                                                                                                                                     
    sync '/usr/portage syncer';                                                                                                                                                                 
}                                                                                                                                                                                               

'/usr/portage syncer' {
    # typename of this section: syncer
    class pkgcore.sync.rsync.rsync_timestamp_syncer;
    # type: str                                     
    basedir '/usr/portage';                         
    # type: str                                     
    retries '3';                                    
    # type: str                                     
    uri 'rsync://rsync.gentoo.org/gentoo-portage';  
}                                                   

'/usr/portage/eclass' {
    # typename of this section: eclass_cache
    class pkgcore.ebuild.eclass_cache.cache;
    # type: str                             
    path '/usr/portage/eclass';             
    # type: str                             
    portdir '/usr/portage';                 
}                                           

#'autoload-portage' {
#    # typename of this section: configsection
#    class pkgcore.ebuild.portage_conf.config_from_make_conf;
#}                                                           

'basic-formatter' {
    # typename of this section: pmerge_formatter
    class pkgcore.ebuild.formatter.factory;     
}                                               

'binary_debug_trigger' {
    # typename of this section: trigger
    class pkgcore.merge.triggers.BinaryDebug;
    # type: str                              
    mode 'strip';                            
}                                            

'binpkg' {
    # typename of this section: repo
    class pkgcore.binpkg.repository.tree;
    # type: bool                         
    ignore_paludis_versioning False;     
    # type: str                          
    location '/test4/packages';
}                                                  

'buildpkg_trigger' {
    # typename of this section: trigger
    class pkgcore.merge.triggers.SavePkg;
    # type: bool                         
    pristine False;                      
    # type: ref:repo                     
    target_repo 'binpkg';                
}                                        

#'eclass stack' {
#    # typename of this section: eclass_cache
#    class pkgcore.config.basics.section_alias;
#    typename 'eclass_cache'; # ERROR: line was missing
#    # type: ref:eclass_cache                  
#    target '/usr/portage/eclass';             
#}     

#'eclass stack' {
#    # typename of this section: eclass_cache
#    # class pkgcore.ebuild.eclass_cache
#    class pkgcore.config.basics.section_alias;
#    typename 'eclass_cache'; # ERROR: line was missing
#    # type: ref:eclass_cache                  
#    target '/usr/portage/eclass';             
#}                                             
                                        

'fetcher' {
    # typename of this section: fetcher
    class pkgcore.fetch.custom.fetcher;
    # type: str                        
    ACCEPT_KEYWORDS 'arm';             
    # type: str                        
    ARCH 'arm';                        
    # type: str                        
    AUTOCLEAN 'yes';                   
    # type: str                        
    CBUILD 'i686-pc-linux-gnu';        
    # type: str                        
    CFLAGS '-Os -pipe -mcpu=arm1136jf-s -fomit-frame-pointer'; # -I/usr/arm-unknown-linux-gnu/usr/include/ -I/usr/arm-unknown-linux-gnu/include/;
    # type: str                                                                                                                               
    CHOST 'arm-unknown-linux-gnu';                                                                                                            
    # type: str                                                                                                                               
    CLEAN_DELAY '5';                                                                                                                          
    # type: str                                                                                                                               
    COLLISION_IGNORE '/lib/modules';                                                                                                          
    # type: str                                                                                                                               
    CONFIG_PROTECT '/etc';                                                                                                                    
    # type: str                                                                                                                               
    CONFIG_PROTECT_MASK '/etc/env.d';                                                                                                         
    # type: str                                                                                                                               
    CXXFLAGS '-Os -pipe -mcpu=arm1136jf-s -fomit-frame-pointer'; # -I/usr/arm-unknown-linux-gnu/usr/include/ -I/usr/arm-unknown-linux-gnu/include/';
    # type: str                                                                                                                                 
    ELIBC 'glibc';                                                                                                                              
    # type: str                                                                                                                                 
    EMERGE_WARNING_DELAY '10';                                                                                                                  
    # type: str                                                                                                                                 
    E_MACHINE 'EM_ARM';                                                                                                                         
    # type: str                                                                                                                                 
    FEATURES '-collision-protect sandbox buildpkg noman noinfo nodoc';                                                                          
    # type: str                                                                                                                                 
    HOSTCC 'i686-pc-linux-gnu-gcc';                                                                                                             
    # type: str                                                                                                                                 
    INPUT_DEVICES 'evdev keyboard mouse touchscreen';                                                                                           
    # type: str                                                                                                                                 
    # LDFLAGS ''; #'-L/usr/arm-unknown-linux-gnu/lib -L/usr/arm-unknown-linux-gnu/usr/lib';                                                            
    # type: str                                                                                                                                 
    LIBDIR_amd64 'lib64';                                                                                                                       
    # type: str                                                                                                                                 
    LIBDIR_arm 'lib';                                                                                                                           
    # type: str                                                                                                                                 
    MAKEOPTS '-j1';                                                                                                                             
    # type: str                                                                                                                                 
    MARCH_TUNE '-mcpu=arm1136jf-s';                                                                                                             
    # type: str                                                                                                                                 
    PKG_CONFIG_PATH '/test4/usr/lib/pkgconfig/';                                                                            
    # type: str                                                                                                                                 
    PORTAGE_BINHOST_CHUNKSIZE '3000';                                                                                                           
    # type: str                                                                                                                                 
    PORTAGE_COMPRESS_EXCLUDE_SUFFIXES 'css gif htm[l]? jp[e]?g js pdf png';                                                                     
    # type: str                                                                                                                                 
    PORTAGE_ELOG_CLASSES 'log warn error';                                                                                                      
    # type: str                                                                                                                                 
    PORTAGE_ELOG_MAILFROM 'portage@localhost';                                                                                                  
    # type: str                                                                                                                                 
    PORTAGE_ELOG_MAILSUBJECT '[portage] ebuild log for ${PACKAGE} on ${HOST}';                                                                  
    # type: str                                                                                                                                 
    PORTAGE_ELOG_MAILURI 'root';                                                                                                                
    # type: str                                                                                                                                 
    PORTAGE_ELOG_SYSTEM 'save_summary echo';                                                                                                    
    # type: str                                                                                                                                 
    PORTAGE_FETCH_CHECKSUM_TRY_MIRRORS '5';                                                                                                     
    # type: str                                                                                                                                 
    PORTAGE_FETCH_RESUME_MIN_SIZE '350K';                                                                                                       
    # type: str                                                                                                                                 
    PORTAGE_INST_GID '1000';  # some user                                                                                                      
    # type: str                                                                                                                                 
    PORTAGE_INST_UID '1000';  # some user                                                                                                                     
    # type: str                                                                                                                                 
    PORTAGE_RSYNC_OPTS '--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages';                                                                                                                                                                                             
    # type: str                                                                                                                                                                                 
    PORTAGE_TMPDIR '/test4/tmp/';                                                                                                                                           
    # type: str                                                                                                                                                                                 
    PORTAGE_WORKDIR_MODE '0700';                                                                                                                                                                
    # type: str                                                                                                                                                                                 
    ROOT '/test4/';                                                                                                                                                         
    # type: str                                                                                                                                                                                 
    RPMDIR '/usr/portage/rpm';                                                                                                                                                                  
    # type: str                                                                                                                                                                                 
    UCLIBC_CPU_DEFAULT 'GENERIC_ARM';                                                                                                                                                           
    # type: str                                                                                                                                                                                 
    USE 'arm zlib bindist make-symlinks minimal         input_devices_keyboard input_devices_evdev         video_cards_fbdev video_cards_dummy';                                                
    # type: str                                                                                                                                                                                 
    USE_EXPAND 'video_cards input_devices';                                                                                                                                                     
    # type: str                                                                                                                                                                                 
    USE_ORDER 'env:pkg:conf:defaults:pkginternal:env.d';                                                                                                                                        
    # type: str                                                                                                                                                                                 
    VIDEO_CARDS 'fbdev dummy';                                                                                                                                                                  
    # type: str                                                                                                                                                                                 
    command '/usr/bin/wget -t 5 -T 60 --passive-ftp -O "${DISTDIR}/${FILE}" "${URI}"';                                                                                                          
    # type: str                                                                                                                                                                                 
    distdir '/usr/portage/distfiles';                                                                                                                                                           
    # type: str                                                                                                                                                                                 
    resume_command '/usr/bin/wget -c -t 5 -T 60 --passive-ftp -O "${DISTDIR}/${FILE}" "${URI}"';                                                                                                
}                                                                                                                                                                                               

'glsa' {
    # typename of this section: pkgset
    class pkgcore.config.basics.section_alias;
    typename 'pkgset';  # ERROR line was missing
    # type: ref:pkgset                        
    target 'vuln';                            
}                                             

'installed' {
    # typename of this section: pkgset
    class pkgcore.pkgsets.installed.Installed;
    # type: refs:repo                         
    vdb 'vdb';                                
}                                             

'livefs domain' {
    # typename of this section: domain
    class pkgcore.ebuild.domain.domain;
    default true;                      
    # type: list                       
    ACCEPT_KEYWORDS 'arm';             
    # type: str                        
    ARCH 'arm';                        
    # type: str                        
    AUTOCLEAN 'yes';                   
    # type: str                        
    CBUILD 'i686-pc-linux-gnu';        
    # type: str                        
    CFLAGS '-Os -pipe -mcpu=arm1136jf-s -fomit-frame-pointer'; #'-I/usr/arm-unknown-linux-gnu/usr/include/ -I/usr/arm-unknown-linux-gnu/include/';
    # type: str                                                                                                                               
    CHOST 'arm-unknown-linux-gnu';                                                                                                            
    # type: str                                                                                                                               
    CLEAN_DELAY '5';                                                                                                                          
    # type: str                                                                                                                               
    COLLISION_IGNORE '/lib/modules';                                                                                                          
    # type: list                                                                                                                              
    CONFIG_PROTECT '/etc';                                                                                                                    
    # type: list                                                                                                                              
    CONFIG_PROTECT_MASK '/etc/env.d';                                                                                                         
    # type: str                                                                                                                               
    CXXFLAGS '-Os -pipe -mcpu=arm1136jf-s -fomit-frame-pointer'; # -I/usr/arm-unknown-linux-gnu/usr/include/ -I/usr/arm-unknown-linux-gnu/include/';
    # type: str                                                                                                                                 
    ELIBC 'glibc';                                                                                                                              
    # type: str                                                                                                                                 
    EMERGE_WARNING_DELAY '10';                                                                                                                  
    # type: str                                                                                                                                 
    E_MACHINE 'EM_ARM';                                                                                                                         
    # type: list                                                                                                                                
    FEATURES '-collision-protect' 'sandbox' 'buildpkg' 'noman' 'noinfo' 'nodoc';                                                                
    # type: str                                                                                                                                 
    HOSTCC 'i686-pc-linux-gnu-gcc';                                                                                                             
    # type: str                                                                                                                                 
    INPUT_DEVICES 'evdev keyboard mouse touchscreen';                                                                                           
    # type: str                                                                                                                                 
    # LDFLAGS ''; #'-L/usr/arm-unknown-linux-gnu/lib -L/usr/arm-unknown-linux-gnu/usr/lib';                                                            
    # type: str                                                                                                                                 
    LIBDIR_amd64 'lib64';                                                                                                                       
    # type: str                                                                                                                                 
    LIBDIR_arm 'lib';                                                                                                                           
    # type: str                                                                                                                                 
    MAKEOPTS '-j1';                                                                                                                             
    # type: str                                                                                                                                 
    MARCH_TUNE '-mcpu=arm1136jf-s';                                                                                                             
    # type: str                                                                                                                                 
    PKG_CONFIG_PATH '/test4/usr/lib/pkgconfig/';                                                                            
    # type: str                                                                                                                                 
    PORTAGE_BINHOST_CHUNKSIZE '3000';                                                                                                           
    # type: str                                                                                                                                 
    PORTAGE_COMPRESS_EXCLUDE_SUFFIXES 'css gif htm[l]? jp[e]?g js pdf png';                                                                     
    # type: str                                                                                                                                 
    PORTAGE_ELOG_CLASSES 'log warn error';                                                                                                      
    # type: str                                                                                                                                 
    PORTAGE_ELOG_MAILFROM 'portage@localhost';                                                                                                  
    # type: str                                                                                                                                 
    PORTAGE_ELOG_MAILSUBJECT '[portage] ebuild log for ${PACKAGE} on ${HOST}';                                                                  
    # type: str                                                                                                                                 
    PORTAGE_ELOG_MAILURI 'root';                                                                                                                
    # type: str                                                                                                                                 
    PORTAGE_ELOG_SYSTEM 'save_summary echo';                                                                                                    
    # type: str                                                                                                                                 
    PORTAGE_FETCH_CHECKSUM_TRY_MIRRORS '5';                                                                                                     
    # type: str                                                                                                                                 
    PORTAGE_FETCH_RESUME_MIN_SIZE '350K';                                                                                                       
    # type: str                                                                                                                                 
    PORTAGE_INST_GID '1000';    # some user                                                                                                             
    # type: str                                                                                                                                 
    PORTAGE_INST_UID '1000';    # some user                                                                                                                   
    # type: str                                                                                                                                 
    PORTAGE_RSYNC_OPTS '--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages';                                                                                                                                                                                             
    # type: str                                                                                                                                                                                 
    PORTAGE_TMPDIR '/test4/tmp/';                                                                                                                                           
    # type: str                                                                                                                                                                                 
    PORTAGE_WORKDIR_MODE '0700';                                                                                                                                                                
    # type: str                                                                                                                                                                                 
    ROOT '/test4/';                                                                                                                                                         
    # type: str                                                                                                                                                                                 
    RPMDIR '/usr/portage/rpm';                                                                                                                                                                  
    # type: str                                                                                                                                                                                 
    UCLIBC_CPU_DEFAULT 'GENERIC_ARM';                                                                                                                                                           
    # type: list                                                                                                                                                                                
    USE 'arm' 'zlib' 'bindist' 'make-symlinks' 'minimal' 'input_devices_keyboard' 'input_devices_evdev' 'video_cards_fbdev' 'video_cards_dummy';                                                
    # type: str                                                                                                                                                                                 
    USE_EXPAND 'video_cards input_devices';                                                                                                                                                     
    # type: str                                                                                                                                                                                 
    USE_ORDER 'env:pkg:conf:defaults:pkginternal:env.d';                                                                                                                                        
    # type: str                                                                                                                                                                                 
    VIDEO_CARDS 'fbdev dummy';                                                                                                                                                                  
    # type: list                                                                                                                                                                                
    bashrc '/test4/etc/portage/bashrc';                                                                                                                                     
    # type: ref:fetcher                                                                                                                                                                         
    fetcher 'fetcher';                                                                                                                                                                          
    # type: str                                                                                                                                                                                 
    name 'livefs domain';                                                                                                                                                                       
    # type: ref:profile                                                                                                                                                                         
    profile 'profile';                                                                                                                                                                          
    # type: lazy_refs:repo                                                                                                                                                                      
    # repositories 'repo-stack' 'binpkg';  # ERROR: alias doesn't work here    
    repositories '/usr/portage' 'binpkg';   														      
    # type: str                                                                                                                                                                                 
    root '/test4/';                                                                                                                                                         
    # type: lazy_refs:trigger                                                                                                                                                                   
    triggers 'buildpkg_trigger' 'binary_debug_trigger';                                                                                                                                         
    # type: lazy_refs:repo                                                                                                                                                                      
    vdb 'vdb';                                                                                                                                                                                  
}                                                                                                                                                                                               

'paludis-formatter' {
    # typename of this section: pmerge_formatter
    class pkgcore.ebuild.formatter.factory;     
}                                               

'pkgcore-formatter' {
    # typename of this section: pmerge_formatter
    class pkgcore.ebuild.formatter.factory;     
}                                               

'portage-formatter' {
    # typename of this section: pmerge_formatter
    class pkgcore.ebuild.formatter.factory;     
    default true;                               
}                                               

'portage-verbose-formatter' {
    # typename of this section: pmerge_formatter
    class pkgcore.ebuild.formatter.portage_verbose_factory;
}                                                          

'portdir' {
    # typename of this section: repo
    class pkgcore.config.basics.section_alias;
    typename 'repo'; # ERROR: line was missing
    # type: ref:repo                          
    target '/usr/portage';                    
}                                             

'portdir cache' {
    # typename of this section: cache
    class pkgcore.cache.metadata.database;
    # type: ref:eclass_cache              
    eclasses '/usr/portage/eclass';       
    # type: str                           
    location '/usr/portage';              
    # type: bool                          
    readonly True;                        
}                                         

'profile' {
    # typename of this section: profile
    class pkgcore.ebuild.profiles.OnDiskProfile;
    # type: str                                 
    basepath '/usr/portage/profiles';
    # type: str                                     
    profile 'embedded';                             
}                                                   

'repo-stack' {
    # typename of this section: repo
    class pkgcore.config.basics.section_alias;
    typename 'repo'; # ERROR: line was missing
    # type: ref:repo                          
    target '/usr/portage';                    
}                                             

'system' {
    # typename of this section: pkgset
    class pkgcore.pkgsets.system.SystemSet;
    # type: ref:profile                    
    profile 'profile';                     
}                                          

'vdb' {
    # typename of this section: repo
    class pkgcore.vdb.ondisk.tree;  
    # type: str                     
    cache_location '/test4/var/cache/edb/dep/var/db/pkg';
    # type: str
    location '/test4/var/db/pkg';
}

'versioned-installed' {
    # typename of this section: pkgset
    class pkgcore.pkgsets.installed.VersionedInstalled;
    # type: refs:repo
    vdb 'vdb';
}

'vuln' {
    # typename of this section: pkgset
    class pkgcore.ebuild.portage_conf.SecurityUpgradesViaProfile;
    # type: ref:repo
    # ebuild_repo 'repo-stack'; # ERROR: alias doesn't work here
    ebuild_repo '/usr/portage';
    # type: ref:profile
    profile 'profile';
    # type: ref:repo
    vdb 'vdb';
}

'world' {
    # typename of this section: pkgset
    class pkgcore.pkgsets.filelist.WorldFile;
    # type: str
    location '/test4/var/lib/portage/world';
    #location '/var/lib/portage/world'; ## ERROR: config_root is missing here
}

comment:4 Changed 4 weeks ago by ferringb

  • status changed from new to closed
  • resolution set to fixed

Try 0.5.10 please; the problem I have w/ the fix you're using is that for binpkgs it would override the cbuild/ctarget/chost w/ the current domains settings- this isn't always correct (the pkg was built w/ those settings, that's what they are).

What *was* screwed up however was the fallback setting of CBUILD/CTARGET to CHOST- it was automatically stomping the values- specifically look for scope_update and cbuild/ctarget in pkgcore/ebuild/repository.py

The fix was basically- 

 scope_update = {'chost': chost}
-scope_update.update((x, domain_settings.get(x.upper, chost))
+scope_update.update((x, domain_settings.get(x.upper(), chost))
 for x in ('cbuild', 'ctarget'))

Since it was trying to look up the method rather then the method results (uppercase varname), it was just stomping cbuild/ctarget incorrectly.

This should restore CC support- if not, let me know, I'm expecting a 0.5.11 release in the next week or so.

Note: See TracTickets for help on using tickets.