2 # SPDX-License-Identifier: GPL-2.0+
4 # Author: Masahiro Yamada <yamada.masahiro@socionext.com>
8 Move config options from headers to defconfig files.
10 Since Kconfig was introduced to U-Boot, we have worked on moving
11 config options from headers to Kconfig (defconfig).
13 This tool intends to help this tremendous work.
19 First, you must edit the Kconfig to add the menu entries for the configs
22 And then run this tool giving CONFIG names you want to move.
23 For example, if you want to move CONFIG_CMD_USB and CONFIG_SYS_TEXT_BASE,
24 simply type as follows:
26 $ tools/moveconfig.py CONFIG_CMD_USB CONFIG_SYS_TEXT_BASE
28 The tool walks through all the defconfig files and move the given CONFIGs.
30 The log is also displayed on the terminal.
32 The log is printed for each defconfig as follows:
40 <defconfig_name> is the name of the defconfig.
42 <action*> shows what the tool did for that defconfig.
43 It looks like one of the following:
46 This config option was moved to the defconfig
48 - CONFIG_... is not defined in Kconfig. Do nothing.
49 The entry for this CONFIG was not found in Kconfig. The option is not
50 defined in the config header, either. So, this case can be just skipped.
52 - CONFIG_... is not defined in Kconfig (suspicious). Do nothing.
53 This option is defined in the config header, but its entry was not found
55 There are two common cases:
56 - You forgot to create an entry for the CONFIG before running
57 this tool, or made a typo in a CONFIG passed to this tool.
58 - The entry was hidden due to unmet 'depends on'.
59 The tool does not know if the result is reasonable, so please check it
62 - 'CONFIG_...' is the same as the define in Kconfig. Do nothing.
63 The define in the config header matched the one in Kconfig.
64 We do not need to touch it.
66 - Compiler is missing. Do nothing.
67 The compiler specified for this architecture was not found
68 in your PATH environment.
69 (If -e option is passed, the tool exits immediately.)
72 An error occurred during processing this defconfig. Skipped.
73 (If -e option is passed, the tool exits immediately on error.)
75 Finally, you will be asked, Clean up headers? [y/n]:
77 If you say 'y' here, the unnecessary config defines are removed
78 from the config headers (include/configs/*.h).
79 It just uses the regex method, so you should not rely on it.
80 Just in case, please do 'git diff' to see what happened.
86 This tool runs configuration and builds include/autoconf.mk for every
87 defconfig. The config options defined in Kconfig appear in the .config
88 file (unless they are hidden because of unmet dependency.)
89 On the other hand, the config options defined by board headers are seen
90 in include/autoconf.mk. The tool looks for the specified options in both
91 of them to decide the appropriate action for the options. If the given
92 config option is found in the .config, but its value does not match the
93 one from the board header, the config option in the .config is replaced
94 with the define in the board header. Then, the .config is synced by
95 "make savedefconfig" and the defconfig is updated with it.
97 For faster processing, this tool handles multi-threading. It creates
98 separate build directories where the out-of-tree build is run. The
99 temporary build directories are automatically created and deleted as
100 needed. The number of threads are chosen based on the number of the CPU
101 cores of your system although you can change it via -j (--jobs) option.
107 Appropriate toolchain are necessary to generate include/autoconf.mk
108 for all the architectures supported by U-Boot. Most of them are available
109 at the kernel.org site, some are not provided by kernel.org. This tool uses
110 the same tools as buildman, so see that tool for setup (e.g. --fetch-arch).
116 To sync only X86 defconfigs:
118 ./tools/moveconfig.py -s -d <(grep -l X86 configs/*)
122 grep -l X86 configs/* | ./tools/moveconfig.py -s -d -
124 To process CONFIG_CMD_FPGAD only for a subset of configs based on path match:
126 ls configs/{hrcon*,iocon*,strider*} | \
127 ./tools/moveconfig.py -Cy CONFIG_CMD_FPGAD -d -
130 Finding implied CONFIGs
131 -----------------------
133 Some CONFIG options can be implied by others and this can help to reduce
134 the size of the defconfig files. For example, CONFIG_X86 implies
135 CONFIG_CMD_IRQ, so we can put 'imply CMD_IRQ' under 'config X86' and
136 all x86 boards will have that option, avoiding adding CONFIG_CMD_IRQ to
137 each of the x86 defconfig files.
139 This tool can help find such configs. To use it, first build a database:
141 ./tools/moveconfig.py -b
143 Then try to query it:
145 ./tools/moveconfig.py -i CONFIG_CMD_IRQ
146 CONFIG_CMD_IRQ found in 311/2384 defconfigs
147 44 : CONFIG_SYS_FSL_ERRATUM_IFC_A002769
148 41 : CONFIG_SYS_FSL_ERRATUM_A007075
149 31 : CONFIG_SYS_FSL_DDR_VER_44
150 28 : CONFIG_ARCH_P1010
151 28 : CONFIG_SYS_FSL_ERRATUM_P1010_A003549
152 28 : CONFIG_SYS_FSL_ERRATUM_SEC_A003571
153 28 : CONFIG_SYS_FSL_ERRATUM_IFC_A003399
154 25 : CONFIG_SYS_FSL_ERRATUM_A008044
155 22 : CONFIG_ARCH_P1020
156 21 : CONFIG_SYS_FSL_DDR_VER_46
157 20 : CONFIG_MAX_PIRQ_LINKS
158 20 : CONFIG_HPET_ADDRESS
160 20 : CONFIG_PCIE_ECAM_SIZE
161 20 : CONFIG_IRQ_SLOT_COUNT
162 20 : CONFIG_I8259_PIC
163 20 : CONFIG_CPU_ADDR_BITS
165 20 : CONFIG_SYS_FSL_ERRATUM_A005871
166 20 : CONFIG_PCIE_ECAM_BASE
167 20 : CONFIG_X86_TSC_TIMER
168 20 : CONFIG_I8254_TIMER
169 20 : CONFIG_CMD_GETTIME
170 19 : CONFIG_SYS_FSL_ERRATUM_A005812
171 18 : CONFIG_X86_RUN_32BIT
172 17 : CONFIG_CMD_CHIP_CONFIG
175 This shows a list of config options which might imply CONFIG_CMD_EEPROM along
176 with how many defconfigs they cover. From this you can see that CONFIG_X86
177 implies CONFIG_CMD_EEPROM. Therefore, instead of adding CONFIG_CMD_EEPROM to
178 the defconfig of every x86 board, you could add a single imply line to the
182 bool "x86 architecture"
186 That will cover 20 defconfigs. Many of the options listed are not suitable as
187 they are not related. E.g. it would be odd for CONFIG_CMD_GETTIME to imply
190 Using this search you can reduce the size of moveconfig patches.
192 You can automatically add 'imply' statements in the Kconfig with the -a
195 ./tools/moveconfig.py -s -i CONFIG_SCSI \
196 -a CONFIG_ARCH_LS1021A,CONFIG_ARCH_LS1043A
198 This will add 'imply SCSI' to the two CONFIG options mentioned, assuming that
199 the database indicates that they do actually imply CONFIG_SCSI and do not
200 already have an 'imply SCSI'.
202 The output shows where the imply is added:
204 18 : CONFIG_ARCH_LS1021A arch/arm/cpu/armv7/ls102xa/Kconfig:1
205 13 : CONFIG_ARCH_LS1043A arch/arm/cpu/armv8/fsl-layerscape/Kconfig:11
206 12 : CONFIG_ARCH_LS1046A arch/arm/cpu/armv8/fsl-layerscape/Kconfig:31
208 The first number is the number of boards which can avoid having a special
209 CONFIG_SCSI option in their defconfig file if this 'imply' is added.
210 The location at the right is the Kconfig file and line number where the config
211 appears. For example, adding 'imply CONFIG_SCSI' to the 'config ARCH_LS1021A'
212 in arch/arm/cpu/armv7/ls102xa/Kconfig at line 1 will help 18 boards to reduce
213 the size of their defconfig files.
215 If you want to add an 'imply' to every imply config in the list, you can use
217 ./tools/moveconfig.py -s -i CONFIG_SCSI -a all
219 To control which ones are displayed, use -I <list> where list is a list of
220 options (use '-I help' to see possible options and their meaning).
222 To skip showing you options that already have an 'imply' attached, use -A.
224 When you have finished adding 'imply' options you can regenerate the
225 defconfig files for affected boards with something like:
227 git show --stat | ./tools/moveconfig.py -s -d -
229 This will regenerate only those defconfigs changed in the current commit.
230 If you start with (say) 100 defconfigs being changed in the commit, and add
231 a few 'imply' options as above, then regenerate, hopefully you can reduce the
232 number of defconfigs changed in the commit.
239 Surround each portion of the log with escape sequences to display it
240 in color on the terminal.
243 Create a git commit with the changes when the operation is complete. A
244 standard commit message is used which may need to be edited.
247 Specify a file containing a list of defconfigs to move. The defconfig
248 files can be given with shell-style wildcards. Use '-' to read from stdin.
251 Perform a trial run that does not make any changes. It is useful to
252 see what is going to happen before one actually runs it.
255 Exit immediately if Make exits with a non-zero status while processing
259 Do "make savedefconfig" forcibly for all the defconfig files.
260 If not specified, "make savedefconfig" only occurs for cases
261 where at least one CONFIG was moved.
264 Look for moved config options in spl/include/autoconf.mk instead of
265 include/autoconf.mk. This is useful for moving options for SPL build
266 because SPL related options (mostly prefixed with CONFIG_SPL_) are
267 sometimes blocked by CONFIG_SPL_BUILD ifdef conditionals.
270 Only cleanup the headers; skip the defconfig processing
273 Specify the number of threads to run simultaneously. If not specified,
274 the number of threads is the same as the number of CPU cores.
277 Specify the git ref to clone for building the autoconf.mk. If unspecified
278 use the CWD. This is useful for when changes to the Kconfig affect the
279 default values and you want to capture the state of the defconfig from
280 before that change was in effect. If in doubt, specify a ref pre-Kconfig
281 changes (use HEAD if Kconfig changes are not committed). Worst case it will
282 take a bit longer to run, but will always do the right thing.
285 Show any build errors as boards are built
288 Instead of prompting, automatically go ahead with all operations. This
289 includes cleaning up headers, CONFIG_SYS_EXTRA_OPTIONS, the config whitelist
292 To see the complete list of supported options, run
294 $ tools/moveconfig.py -h
305 import multiprocessing
317 sys.path.append(os.path.join(os.path.dirname(__file__), 'buildman'))
318 sys.path.append(os.path.join(os.path.dirname(__file__), 'patman'))
323 SHOW_GNU_MAKE = 'scripts/show-gnu-make'
329 STATE_SAVEDEFCONFIG = 3
333 ACTION_NO_ENTRY_WARN = 2
341 COLOR_PURPLE = '0;35'
343 COLOR_LIGHT_GRAY = '0;37'
344 COLOR_DARK_GRAY = '1;30'
345 COLOR_LIGHT_RED = '1;31'
346 COLOR_LIGHT_GREEN = '1;32'
347 COLOR_YELLOW = '1;33'
348 COLOR_LIGHT_BLUE = '1;34'
349 COLOR_LIGHT_PURPLE = '1;35'
350 COLOR_LIGHT_CYAN = '1;36'
353 AUTO_CONF_PATH = 'include/config/auto.conf'
354 CONFIG_DATABASE = 'moveconfig.db'
356 CONFIG_LEN = len('CONFIG_')
359 "SZ_1": 0x00000001, "SZ_2": 0x00000002,
360 "SZ_4": 0x00000004, "SZ_8": 0x00000008,
361 "SZ_16": 0x00000010, "SZ_32": 0x00000020,
362 "SZ_64": 0x00000040, "SZ_128": 0x00000080,
363 "SZ_256": 0x00000100, "SZ_512": 0x00000200,
364 "SZ_1K": 0x00000400, "SZ_2K": 0x00000800,
365 "SZ_4K": 0x00001000, "SZ_8K": 0x00002000,
366 "SZ_16K": 0x00004000, "SZ_32K": 0x00008000,
367 "SZ_64K": 0x00010000, "SZ_128K": 0x00020000,
368 "SZ_256K": 0x00040000, "SZ_512K": 0x00080000,
369 "SZ_1M": 0x00100000, "SZ_2M": 0x00200000,
370 "SZ_4M": 0x00400000, "SZ_8M": 0x00800000,
371 "SZ_16M": 0x01000000, "SZ_32M": 0x02000000,
372 "SZ_64M": 0x04000000, "SZ_128M": 0x08000000,
373 "SZ_256M": 0x10000000, "SZ_512M": 0x20000000,
374 "SZ_1G": 0x40000000, "SZ_2G": 0x80000000,
378 ### helper functions ###
380 """Get the file object of '/dev/null' device."""
382 devnull = subprocess.DEVNULL # py3k
383 except AttributeError:
384 devnull = open(os.devnull, 'wb')
387 def check_top_directory():
388 """Exit if we are not at the top of source directory."""
389 for f in ('README', 'Licenses'):
390 if not os.path.exists(f):
391 sys.exit('Please run at the top of source directory.')
393 def check_clean_directory():
394 """Exit if the source tree is not clean."""
395 for f in ('.config', 'include/config'):
396 if os.path.exists(f):
397 sys.exit("source tree is not clean, please run 'make mrproper'")
400 """Get the command name of GNU Make.
402 U-Boot needs GNU Make for building, but the command name is not
403 necessarily "make". (for example, "gmake" on FreeBSD).
404 Returns the most appropriate command name on your system.
406 process = subprocess.Popen([SHOW_GNU_MAKE], stdout=subprocess.PIPE)
407 ret = process.communicate()
408 if process.returncode:
409 sys.exit('GNU Make not found')
410 return ret[0].rstrip()
412 def get_matched_defconfig(line):
413 """Get the defconfig files that match a pattern
416 line: Path or filename to match, e.g. 'configs/snow_defconfig' or
417 'k2*_defconfig'. If no directory is provided, 'configs/' is
421 a list of matching defconfig files
423 dirname = os.path.dirname(line)
427 pattern = os.path.join('configs', line)
428 return glob.glob(pattern) + glob.glob(pattern + '_defconfig')
430 def get_matched_defconfigs(defconfigs_file):
431 """Get all the defconfig files that match the patterns in a file.
434 defconfigs_file: File containing a list of defconfigs to process, or
435 '-' to read the list from stdin
438 A list of paths to defconfig files, with no duplicates
441 if defconfigs_file == '-':
443 defconfigs_file = 'stdin'
445 fd = open(defconfigs_file)
446 for i, line in enumerate(fd):
449 continue # skip blank lines silently
451 line = line.split(' ')[0] # handle 'git log' input
452 matched = get_matched_defconfig(line)
454 print("warning: %s:%d: no defconfig matched '%s'" % \
455 (defconfigs_file, i + 1, line), file=sys.stderr)
457 defconfigs += matched
459 # use set() to drop multiple matching
460 return [ defconfig[len('configs') + 1:] for defconfig in set(defconfigs) ]
462 def get_all_defconfigs():
463 """Get all the defconfig files under the configs/ directory."""
465 for (dirpath, dirnames, filenames) in os.walk('configs'):
466 dirpath = dirpath[len('configs') + 1:]
467 for filename in fnmatch.filter(filenames, '*_defconfig'):
468 defconfigs.append(os.path.join(dirpath, filename))
472 def color_text(color_enabled, color, string):
473 """Return colored string."""
475 # LF should not be surrounded by the escape sequence.
476 # Otherwise, additional whitespace or line-feed might be printed.
477 return '\n'.join([ '\033[' + color + 'm' + s + '\033[0m' if s else ''
478 for s in string.split('\n') ])
482 def show_diff(a, b, file_path, color_enabled):
483 """Show unidified diff.
486 a: A list of lines (before)
487 b: A list of lines (after)
488 file_path: Path to the file
489 color_enabled: Display the diff in color
492 diff = difflib.unified_diff(a, b,
493 fromfile=os.path.join('a', file_path),
494 tofile=os.path.join('b', file_path))
497 if line[0] == '-' and line[1] != '-':
498 print(color_text(color_enabled, COLOR_RED, line), end=' ')
499 elif line[0] == '+' and line[1] != '+':
500 print(color_text(color_enabled, COLOR_GREEN, line), end=' ')
504 def extend_matched_lines(lines, matched, pre_patterns, post_patterns, extend_pre,
506 """Extend matched lines if desired patterns are found before/after already
510 lines: A list of lines handled.
511 matched: A list of line numbers that have been already matched.
512 (will be updated by this function)
513 pre_patterns: A list of regular expression that should be matched as
515 post_patterns: A list of regular expression that should be matched as
517 extend_pre: Add the line number of matched preamble to the matched list.
518 extend_post: Add the line number of matched postamble to the matched list.
520 extended_matched = []
533 for p in pre_patterns:
534 if p.search(lines[i - 1]):
540 for p in post_patterns:
541 if p.search(lines[j]):
548 extended_matched.append(i - 1)
550 extended_matched.append(j)
552 matched += extended_matched
555 def confirm(options, prompt):
558 choice = input('{} [y/n]: '.format(prompt))
559 choice = choice.lower()
561 if choice == 'y' or choice == 'n':
569 def cleanup_empty_blocks(header_path, options):
570 """Clean up empty conditional blocks
573 header_path: path to the cleaned file.
574 options: option flags.
576 pattern = re.compile(r'^\s*#\s*if.*$\n^\s*#\s*endif.*$\n*', flags=re.M)
577 with open(header_path) as f:
580 new_data = pattern.sub('\n', data)
582 show_diff(data.splitlines(True), new_data.splitlines(True), header_path,
588 with open(header_path, 'w') as f:
591 def cleanup_one_header(header_path, patterns, options):
592 """Clean regex-matched lines away from a file.
595 header_path: path to the cleaned file.
596 patterns: list of regex patterns. Any lines matching to these
597 patterns are deleted.
598 options: option flags.
600 with open(header_path) as f:
601 lines = f.readlines()
604 for i, line in enumerate(lines):
605 if i - 1 in matched and lines[i - 1][-2:] == '\\\n':
608 for pattern in patterns:
609 if pattern.search(line):
616 # remove empty #ifdef ... #endif, successive blank lines
617 pattern_if = re.compile(r'#\s*if(def|ndef)?\W') # #if, #ifdef, #ifndef
618 pattern_elif = re.compile(r'#\s*el(if|se)\W') # #elif, #else
619 pattern_endif = re.compile(r'#\s*endif\W') # #endif
620 pattern_blank = re.compile(r'^\s*$') # empty line
623 old_matched = copy.copy(matched)
624 extend_matched_lines(lines, matched, [pattern_if],
625 [pattern_endif], True, True)
626 extend_matched_lines(lines, matched, [pattern_elif],
627 [pattern_elif, pattern_endif], True, False)
628 extend_matched_lines(lines, matched, [pattern_if, pattern_elif],
629 [pattern_blank], False, True)
630 extend_matched_lines(lines, matched, [pattern_blank],
631 [pattern_elif, pattern_endif], True, False)
632 extend_matched_lines(lines, matched, [pattern_blank],
633 [pattern_blank], True, False)
634 if matched == old_matched:
637 tolines = copy.copy(lines)
639 for i in reversed(matched):
642 show_diff(lines, tolines, header_path, options.color)
647 with open(header_path, 'w') as f:
651 def cleanup_headers(configs, options):
652 """Delete config defines from board headers.
655 configs: A list of CONFIGs to remove.
656 options: option flags.
658 if not confirm(options, 'Clean up headers?'):
662 for config in configs:
663 patterns.append(re.compile(r'#\s*define\s+%s\W' % config))
664 patterns.append(re.compile(r'#\s*undef\s+%s\W' % config))
666 for dir in 'include', 'arch', 'board':
667 for (dirpath, dirnames, filenames) in os.walk(dir):
668 if dirpath == os.path.join('include', 'generated'):
670 for filename in filenames:
671 if not filename.endswith(('~', '.dts', '.dtsi')):
672 header_path = os.path.join(dirpath, filename)
673 # This file contains UTF-16 data and no CONFIG symbols
674 if header_path == 'include/video_font_data.h':
676 cleanup_one_header(header_path, patterns, options)
677 cleanup_empty_blocks(header_path, options)
679 def cleanup_one_extra_option(defconfig_path, configs, options):
680 """Delete config defines in CONFIG_SYS_EXTRA_OPTIONS in one defconfig file.
683 defconfig_path: path to the cleaned defconfig file.
684 configs: A list of CONFIGs to remove.
685 options: option flags.
688 start = 'CONFIG_SYS_EXTRA_OPTIONS="'
691 with open(defconfig_path) as f:
692 lines = f.readlines()
694 for i, line in enumerate(lines):
695 if line.startswith(start) and line.endswith(end):
698 # CONFIG_SYS_EXTRA_OPTIONS was not found in this defconfig
701 old_tokens = line[len(start):-len(end)].split(',')
704 for token in old_tokens:
705 pos = token.find('=')
706 if not (token[:pos] if pos >= 0 else token) in configs:
707 new_tokens.append(token)
709 if new_tokens == old_tokens:
712 tolines = copy.copy(lines)
715 tolines[i] = start + ','.join(new_tokens) + end
719 show_diff(lines, tolines, defconfig_path, options.color)
724 with open(defconfig_path, 'w') as f:
728 def cleanup_extra_options(configs, options):
729 """Delete config defines in CONFIG_SYS_EXTRA_OPTIONS in defconfig files.
732 configs: A list of CONFIGs to remove.
733 options: option flags.
735 if not confirm(options, 'Clean up CONFIG_SYS_EXTRA_OPTIONS?'):
738 configs = [ config[len('CONFIG_'):] for config in configs ]
740 defconfigs = get_all_defconfigs()
742 for defconfig in defconfigs:
743 cleanup_one_extra_option(os.path.join('configs', defconfig), configs,
746 def cleanup_whitelist(configs, options):
747 """Delete config whitelist entries
750 configs: A list of CONFIGs to remove.
751 options: option flags.
753 if not confirm(options, 'Clean up whitelist entries?'):
756 with open(os.path.join('scripts', 'config_whitelist.txt')) as f:
757 lines = f.readlines()
759 lines = [x for x in lines if x.strip() not in configs]
761 with open(os.path.join('scripts', 'config_whitelist.txt'), 'w') as f:
762 f.write(''.join(lines))
764 def find_matching(patterns, line):
770 def cleanup_readme(configs, options):
771 """Delete config description in README
774 configs: A list of CONFIGs to remove.
775 options: option flags.
777 if not confirm(options, 'Clean up README?'):
781 for config in configs:
782 patterns.append(re.compile(r'^\s+%s' % config))
784 with open('README') as f:
785 lines = f.readlines()
791 found = find_matching(patterns, line)
795 if found and re.search(r'^\s+CONFIG', line):
799 newlines.append(line)
801 with open('README', 'w') as f:
802 f.write(''.join(newlines))
804 def try_expand(line):
805 """If value looks like an expression, try expanding it
806 Otherwise just return the existing value
808 if line.find('=') == -1:
812 aeval = asteval.Interpreter( usersyms=SIZES, minimal=True )
813 cfg, val = re.split("=", line)
815 if re.search("[*+-/]|<<|SZ_+|\(([^\)]+)\)", val):
816 newval = hex(aeval(val))
817 print("\tExpanded expression %s to %s" % (val, newval))
818 return cfg+'='+newval
820 print("\tFailed to expand expression in %s" % line)
828 """Progress Indicator"""
830 def __init__(self, total):
831 """Create a new progress indicator.
834 total: A number of defconfig files to process.
840 """Increment the number of processed defconfig files."""
845 """Display the progress."""
846 print(' %d defconfigs out of %d\r' % (self.current, self.total), end=' ')
850 class KconfigScanner:
851 """Kconfig scanner."""
854 """Scan all the Kconfig files and create a Config object."""
855 # Define environment variables referenced from Kconfig
856 os.environ['srctree'] = os.getcwd()
857 os.environ['UBOOTVERSION'] = 'dummy'
858 os.environ['KCONFIG_OBJDIR'] = ''
859 self.conf = kconfiglib.Kconfig()
864 """A parser of .config and include/autoconf.mk."""
866 re_arch = re.compile(r'CONFIG_SYS_ARCH="(.*)"')
867 re_cpu = re.compile(r'CONFIG_SYS_CPU="(.*)"')
869 def __init__(self, configs, options, build_dir):
870 """Create a new parser.
873 configs: A list of CONFIGs to move.
874 options: option flags.
875 build_dir: Build directory.
877 self.configs = configs
878 self.options = options
879 self.dotconfig = os.path.join(build_dir, '.config')
880 self.autoconf = os.path.join(build_dir, 'include', 'autoconf.mk')
881 self.spl_autoconf = os.path.join(build_dir, 'spl', 'include',
883 self.config_autoconf = os.path.join(build_dir, AUTO_CONF_PATH)
884 self.defconfig = os.path.join(build_dir, 'defconfig')
887 """Parse .config file and return the architecture.
890 Architecture name (e.g. 'arm').
894 for line in open(self.dotconfig):
895 m = self.re_arch.match(line)
899 m = self.re_cpu.match(line)
907 if arch == 'arm' and cpu == 'armv8':
912 def parse_one_config(self, config, dotconfig_lines, autoconf_lines):
913 """Parse .config, defconfig, include/autoconf.mk for one config.
915 This function looks for the config options in the lines from
916 defconfig, .config, and include/autoconf.mk in order to decide
917 which action should be taken for this defconfig.
920 config: CONFIG name to parse.
921 dotconfig_lines: lines from the .config file.
922 autoconf_lines: lines from the include/autoconf.mk file.
925 A tupple of the action for this defconfig and the line
926 matched for the config.
928 not_set = '# %s is not set' % config
930 for line in autoconf_lines:
932 if line.startswith(config + '='):
938 new_val = try_expand(new_val)
940 for line in dotconfig_lines:
942 if line.startswith(config + '=') or line == not_set:
946 if new_val == not_set:
947 return (ACTION_NO_ENTRY, config)
949 return (ACTION_NO_ENTRY_WARN, config)
951 # If this CONFIG is neither bool nor trisate
952 if old_val[-2:] != '=y' and old_val[-2:] != '=m' and old_val != not_set:
953 # tools/scripts/define2mk.sed changes '1' to 'y'.
954 # This is a problem if the CONFIG is int type.
955 # Check the type in Kconfig and handle it correctly.
956 if new_val[-2:] == '=y':
957 new_val = new_val[:-1] + '1'
959 return (ACTION_NO_CHANGE if old_val == new_val else ACTION_MOVE,
962 def update_dotconfig(self):
963 """Parse files for the config options and update the .config.
965 This function parses the generated .config and include/autoconf.mk
966 searching the target options.
967 Move the config option(s) to the .config as needed.
970 defconfig: defconfig name.
973 Return a tuple of (updated flag, log string).
974 The "updated flag" is True if the .config was updated, False
975 otherwise. The "log string" shows what happend to the .config.
981 rm_files = [self.config_autoconf, self.autoconf]
984 if os.path.exists(self.spl_autoconf):
985 autoconf_path = self.spl_autoconf
986 rm_files.append(self.spl_autoconf)
990 return (updated, suspicious,
991 color_text(self.options.color, COLOR_BROWN,
992 "SPL is not enabled. Skipped.") + '\n')
994 autoconf_path = self.autoconf
996 with open(self.dotconfig) as f:
997 dotconfig_lines = f.readlines()
999 with open(autoconf_path) as f:
1000 autoconf_lines = f.readlines()
1002 for config in self.configs:
1003 result = self.parse_one_config(config, dotconfig_lines,
1005 results.append(result)
1009 for (action, value) in results:
1010 if action == ACTION_MOVE:
1011 actlog = "Move '%s'" % value
1012 log_color = COLOR_LIGHT_GREEN
1013 elif action == ACTION_NO_ENTRY:
1014 actlog = "%s is not defined in Kconfig. Do nothing." % value
1015 log_color = COLOR_LIGHT_BLUE
1016 elif action == ACTION_NO_ENTRY_WARN:
1017 actlog = "%s is not defined in Kconfig (suspicious). Do nothing." % value
1018 log_color = COLOR_YELLOW
1020 elif action == ACTION_NO_CHANGE:
1021 actlog = "'%s' is the same as the define in Kconfig. Do nothing." \
1023 log_color = COLOR_LIGHT_PURPLE
1024 elif action == ACTION_SPL_NOT_EXIST:
1025 actlog = "SPL is not enabled for this defconfig. Skip."
1026 log_color = COLOR_PURPLE
1028 sys.exit("Internal Error. This should not happen.")
1030 log += color_text(self.options.color, log_color, actlog) + '\n'
1032 with open(self.dotconfig, 'a') as f:
1033 for (action, value) in results:
1034 if action == ACTION_MOVE:
1035 f.write(value + '\n')
1038 self.results = results
1042 return (updated, suspicious, log)
1044 def check_defconfig(self):
1045 """Check the defconfig after savedefconfig
1048 Return additional log if moved CONFIGs were removed again by
1049 'make savedefconfig'.
1054 with open(self.defconfig) as f:
1055 defconfig_lines = f.readlines()
1057 for (action, value) in self.results:
1058 if action != ACTION_MOVE:
1060 if not value + '\n' in defconfig_lines:
1061 log += color_text(self.options.color, COLOR_YELLOW,
1062 "'%s' was removed by savedefconfig.\n" %
1068 class DatabaseThread(threading.Thread):
1069 """This thread processes results from Slot threads.
1071 It collects the data in the master config directary. There is only one
1072 result thread, and this helps to serialise the build output.
1074 def __init__(self, config_db, db_queue):
1075 """Set up a new result thread
1078 builder: Builder which will be sent each result
1080 threading.Thread.__init__(self)
1081 self.config_db = config_db
1082 self.db_queue= db_queue
1085 """Called to start up the result thread.
1087 We collect the next result job and pass it on to the build.
1090 defconfig, configs = self.db_queue.get()
1091 self.config_db[defconfig] = configs
1092 self.db_queue.task_done()
1097 """A slot to store a subprocess.
1099 Each instance of this class handles one subprocess.
1100 This class is useful to control multiple threads
1101 for faster processing.
1104 def __init__(self, toolchains, configs, options, progress, devnull,
1105 make_cmd, reference_src_dir, db_queue):
1106 """Create a new process slot.
1109 toolchains: Toolchains object containing toolchains.
1110 configs: A list of CONFIGs to move.
1111 options: option flags.
1112 progress: A progress indicator.
1113 devnull: A file object of '/dev/null'.
1114 make_cmd: command name of GNU Make.
1115 reference_src_dir: Determine the true starting config state from this
1117 db_queue: output queue to write config info for the database
1119 self.toolchains = toolchains
1120 self.options = options
1121 self.progress = progress
1122 self.build_dir = tempfile.mkdtemp()
1123 self.devnull = devnull
1124 self.make_cmd = (make_cmd, 'O=' + self.build_dir)
1125 self.reference_src_dir = reference_src_dir
1126 self.db_queue = db_queue
1127 self.parser = KconfigParser(configs, options, self.build_dir)
1128 self.state = STATE_IDLE
1129 self.failed_boards = set()
1130 self.suspicious_boards = set()
1133 """Delete the working directory
1135 This function makes sure the temporary directory is cleaned away
1136 even if Python suddenly dies due to error. It should be done in here
1137 because it is guaranteed the destructor is always invoked when the
1138 instance of the class gets unreferenced.
1140 If the subprocess is still running, wait until it finishes.
1142 if self.state != STATE_IDLE:
1143 while self.ps.poll() == None:
1145 shutil.rmtree(self.build_dir)
1147 def add(self, defconfig):
1148 """Assign a new subprocess for defconfig and add it to the slot.
1150 If the slot is vacant, create a new subprocess for processing the
1151 given defconfig and add it to the slot. Just returns False if
1152 the slot is occupied (i.e. the current subprocess is still running).
1155 defconfig: defconfig name.
1158 Return True on success or False on failure
1160 if self.state != STATE_IDLE:
1163 self.defconfig = defconfig
1165 self.current_src_dir = self.reference_src_dir
1170 """Check the status of the subprocess and handle it as needed.
1172 Returns True if the slot is vacant (i.e. in idle state).
1173 If the configuration is successfully finished, assign a new
1174 subprocess to build include/autoconf.mk.
1175 If include/autoconf.mk is generated, invoke the parser to
1176 parse the .config and the include/autoconf.mk, moving
1177 config options to the .config as needed.
1178 If the .config was updated, run "make savedefconfig" to sync
1179 it, update the original defconfig, and then set the slot back
1183 Return True if the subprocess is terminated, False otherwise
1185 if self.state == STATE_IDLE:
1188 if self.ps.poll() == None:
1191 if self.ps.poll() != 0:
1193 elif self.state == STATE_DEFCONFIG:
1194 if self.reference_src_dir and not self.current_src_dir:
1195 self.do_savedefconfig()
1198 elif self.state == STATE_AUTOCONF:
1199 if self.current_src_dir:
1200 self.current_src_dir = None
1202 elif self.options.build_db:
1205 self.do_savedefconfig()
1206 elif self.state == STATE_SAVEDEFCONFIG:
1207 self.update_defconfig()
1209 sys.exit("Internal Error. This should not happen.")
1211 return True if self.state == STATE_IDLE else False
1213 def handle_error(self):
1214 """Handle error cases."""
1216 self.log += color_text(self.options.color, COLOR_LIGHT_RED,
1217 "Failed to process.\n")
1218 if self.options.verbose:
1219 self.log += color_text(self.options.color, COLOR_LIGHT_CYAN,
1220 self.ps.stderr.read().decode())
1223 def do_defconfig(self):
1224 """Run 'make <board>_defconfig' to create the .config file."""
1226 cmd = list(self.make_cmd)
1227 cmd.append(self.defconfig)
1228 self.ps = subprocess.Popen(cmd, stdout=self.devnull,
1229 stderr=subprocess.PIPE,
1230 cwd=self.current_src_dir)
1231 self.state = STATE_DEFCONFIG
1233 def do_autoconf(self):
1234 """Run 'make AUTO_CONF_PATH'."""
1236 arch = self.parser.get_arch()
1238 toolchain = self.toolchains.Select(arch)
1240 self.log += color_text(self.options.color, COLOR_YELLOW,
1241 "Tool chain for '%s' is missing. Do nothing.\n" % arch)
1244 env = toolchain.MakeEnvironment(False)
1246 cmd = list(self.make_cmd)
1247 cmd.append('KCONFIG_IGNORE_DUPLICATES=1')
1248 cmd.append(AUTO_CONF_PATH)
1249 self.ps = subprocess.Popen(cmd, stdout=self.devnull, env=env,
1250 stderr=subprocess.PIPE,
1251 cwd=self.current_src_dir)
1252 self.state = STATE_AUTOCONF
1254 def do_build_db(self):
1255 """Add the board to the database"""
1257 with open(os.path.join(self.build_dir, AUTO_CONF_PATH)) as fd:
1258 for line in fd.readlines():
1259 if line.startswith('CONFIG'):
1260 config, value = line.split('=', 1)
1261 configs[config] = value.rstrip()
1262 self.db_queue.put([self.defconfig, configs])
1265 def do_savedefconfig(self):
1266 """Update the .config and run 'make savedefconfig'."""
1268 (updated, suspicious, log) = self.parser.update_dotconfig()
1270 self.suspicious_boards.add(self.defconfig)
1273 if not self.options.force_sync and not updated:
1277 self.log += color_text(self.options.color, COLOR_LIGHT_GREEN,
1278 "Syncing by savedefconfig...\n")
1280 self.log += "Syncing by savedefconfig (forced by option)...\n"
1282 cmd = list(self.make_cmd)
1283 cmd.append('savedefconfig')
1284 self.ps = subprocess.Popen(cmd, stdout=self.devnull,
1285 stderr=subprocess.PIPE)
1286 self.state = STATE_SAVEDEFCONFIG
1288 def update_defconfig(self):
1289 """Update the input defconfig and go back to the idle state."""
1291 log = self.parser.check_defconfig()
1293 self.suspicious_boards.add(self.defconfig)
1295 orig_defconfig = os.path.join('configs', self.defconfig)
1296 new_defconfig = os.path.join(self.build_dir, 'defconfig')
1297 updated = not filecmp.cmp(orig_defconfig, new_defconfig)
1300 self.log += color_text(self.options.color, COLOR_LIGHT_BLUE,
1301 "defconfig was updated.\n")
1303 if not self.options.dry_run and updated:
1304 shutil.move(new_defconfig, orig_defconfig)
1307 def finish(self, success):
1308 """Display log along with progress and go to the idle state.
1311 success: Should be True when the defconfig was processed
1312 successfully, or False when it fails.
1314 # output at least 30 characters to hide the "* defconfigs out of *".
1315 log = self.defconfig.ljust(30) + '\n'
1317 log += '\n'.join([ ' ' + s for s in self.log.split('\n') ])
1318 # Some threads are running in parallel.
1319 # Print log atomically to not mix up logs from different threads.
1320 print(log, file=(sys.stdout if success else sys.stderr))
1323 if self.options.exit_on_error:
1324 sys.exit("Exit on error.")
1325 # If --exit-on-error flag is not set, skip this board and continue.
1326 # Record the failed board.
1327 self.failed_boards.add(self.defconfig)
1330 self.progress.show()
1331 self.state = STATE_IDLE
1333 def get_failed_boards(self):
1334 """Returns a set of failed boards (defconfigs) in this slot.
1336 return self.failed_boards
1338 def get_suspicious_boards(self):
1339 """Returns a set of boards (defconfigs) with possible misconversion.
1341 return self.suspicious_boards - self.failed_boards
1345 """Controller of the array of subprocess slots."""
1347 def __init__(self, toolchains, configs, options, progress,
1348 reference_src_dir, db_queue):
1349 """Create a new slots controller.
1352 toolchains: Toolchains object containing toolchains.
1353 configs: A list of CONFIGs to move.
1354 options: option flags.
1355 progress: A progress indicator.
1356 reference_src_dir: Determine the true starting config state from this
1358 db_queue: output queue to write config info for the database
1360 self.options = options
1362 devnull = get_devnull()
1363 make_cmd = get_make_cmd()
1364 for i in range(options.jobs):
1365 self.slots.append(Slot(toolchains, configs, options, progress,
1366 devnull, make_cmd, reference_src_dir,
1369 def add(self, defconfig):
1370 """Add a new subprocess if a vacant slot is found.
1373 defconfig: defconfig name to be put into.
1376 Return True on success or False on failure
1378 for slot in self.slots:
1379 if slot.add(defconfig):
1383 def available(self):
1384 """Check if there is a vacant slot.
1387 Return True if at lease one vacant slot is found, False otherwise.
1389 for slot in self.slots:
1395 """Check if all slots are vacant.
1398 Return True if all the slots are vacant, False otherwise.
1401 for slot in self.slots:
1406 def show_failed_boards(self):
1407 """Display all of the failed boards (defconfigs)."""
1409 output_file = 'moveconfig.failed'
1411 for slot in self.slots:
1412 boards |= slot.get_failed_boards()
1415 boards = '\n'.join(boards) + '\n'
1416 msg = "The following boards were not processed due to error:\n"
1418 msg += "(the list has been saved in %s)\n" % output_file
1419 print(color_text(self.options.color, COLOR_LIGHT_RED,
1420 msg), file=sys.stderr)
1422 with open(output_file, 'w') as f:
1425 def show_suspicious_boards(self):
1426 """Display all boards (defconfigs) with possible misconversion."""
1428 output_file = 'moveconfig.suspicious'
1430 for slot in self.slots:
1431 boards |= slot.get_suspicious_boards()
1434 boards = '\n'.join(boards) + '\n'
1435 msg = "The following boards might have been converted incorrectly.\n"
1436 msg += "It is highly recommended to check them manually:\n"
1438 msg += "(the list has been saved in %s)\n" % output_file
1439 print(color_text(self.options.color, COLOR_YELLOW,
1440 msg), file=sys.stderr)
1442 with open(output_file, 'w') as f:
1445 class ReferenceSource:
1447 """Reference source against which original configs should be parsed."""
1449 def __init__(self, commit):
1450 """Create a reference source directory based on a specified commit.
1453 commit: commit to git-clone
1455 self.src_dir = tempfile.mkdtemp()
1456 print("Cloning git repo to a separate work directory...")
1457 subprocess.check_output(['git', 'clone', os.getcwd(), '.'],
1459 print("Checkout '%s' to build the original autoconf.mk." % \
1460 subprocess.check_output(['git', 'rev-parse', '--short', commit]).strip())
1461 subprocess.check_output(['git', 'checkout', commit],
1462 stderr=subprocess.STDOUT, cwd=self.src_dir)
1465 """Delete the reference source directory
1467 This function makes sure the temporary directory is cleaned away
1468 even if Python suddenly dies due to error. It should be done in here
1469 because it is guaranteed the destructor is always invoked when the
1470 instance of the class gets unreferenced.
1472 shutil.rmtree(self.src_dir)
1475 """Return the absolute path to the reference source directory."""
1479 def move_config(toolchains, configs, options, db_queue):
1480 """Move config options to defconfig files.
1483 configs: A list of CONFIGs to move.
1484 options: option flags
1486 if len(configs) == 0:
1487 if options.force_sync:
1488 print('No CONFIG is specified. You are probably syncing defconfigs.', end=' ')
1489 elif options.build_db:
1490 print('Building %s database' % CONFIG_DATABASE)
1492 print('Neither CONFIG nor --force-sync is specified. Nothing will happen.', end=' ')
1494 print('Move ' + ', '.join(configs), end=' ')
1495 print('(jobs: %d)\n' % options.jobs)
1498 reference_src = ReferenceSource(options.git_ref)
1499 reference_src_dir = reference_src.get_dir()
1501 reference_src_dir = None
1503 if options.defconfigs:
1504 defconfigs = get_matched_defconfigs(options.defconfigs)
1506 defconfigs = get_all_defconfigs()
1508 progress = Progress(len(defconfigs))
1509 slots = Slots(toolchains, configs, options, progress, reference_src_dir,
1512 # Main loop to process defconfig files:
1513 # Add a new subprocess into a vacant slot.
1514 # Sleep if there is no available slot.
1515 for defconfig in defconfigs:
1516 while not slots.add(defconfig):
1517 while not slots.available():
1518 # No available slot: sleep for a while
1519 time.sleep(SLEEP_TIME)
1521 # wait until all the subprocesses finish
1522 while not slots.empty():
1523 time.sleep(SLEEP_TIME)
1526 slots.show_failed_boards()
1527 slots.show_suspicious_boards()
1529 def find_kconfig_rules(kconf, config, imply_config):
1530 """Check whether a config has a 'select' or 'imply' keyword
1533 kconf: Kconfiglib.Kconfig object
1534 config: Name of config to check (without CONFIG_ prefix)
1535 imply_config: Implying config (without CONFIG_ prefix) which may or
1536 may not have an 'imply' for 'config')
1539 Symbol object for 'config' if found, else None
1541 sym = kconf.syms.get(imply_config)
1543 for sel in sym.get_selected_symbols() | sym.get_implied_symbols():
1544 if sel.get_name() == config:
1548 def check_imply_rule(kconf, config, imply_config):
1549 """Check if we can add an 'imply' option
1551 This finds imply_config in the Kconfig and looks to see if it is possible
1552 to add an 'imply' for 'config' to that part of the Kconfig.
1555 kconf: Kconfiglib.Kconfig object
1556 config: Name of config to check (without CONFIG_ prefix)
1557 imply_config: Implying config (without CONFIG_ prefix) which may or
1558 may not have an 'imply' for 'config')
1562 filename of Kconfig file containing imply_config, or None if none
1563 line number within the Kconfig file, or 0 if none
1564 message indicating the result
1566 sym = kconf.syms.get(imply_config)
1568 return 'cannot find sym'
1569 locs = sym.get_def_locations()
1571 return '%d locations' % len(locs)
1572 fname, linenum = locs[0]
1574 if cwd and fname.startswith(cwd):
1575 fname = fname[len(cwd) + 1:]
1576 file_line = ' at %s:%d' % (fname, linenum)
1577 with open(fname) as fd:
1578 data = fd.read().splitlines()
1579 if data[linenum - 1] != 'config %s' % imply_config:
1580 return None, 0, 'bad sym format %s%s' % (data[linenum], file_line)
1581 return fname, linenum, 'adding%s' % file_line
1583 def add_imply_rule(config, fname, linenum):
1584 """Add a new 'imply' option to a Kconfig
1587 config: config option to add an imply for (without CONFIG_ prefix)
1588 fname: Kconfig filename to update
1589 linenum: Line number to place the 'imply' before
1592 Message indicating the result
1594 file_line = ' at %s:%d' % (fname, linenum)
1595 data = open(fname).read().splitlines()
1598 for offset, line in enumerate(data[linenum:]):
1599 if line.strip().startswith('help') or not line:
1600 data.insert(linenum + offset, '\timply %s' % config)
1601 with open(fname, 'w') as fd:
1602 fd.write('\n'.join(data) + '\n')
1603 return 'added%s' % file_line
1605 return 'could not insert%s'
1607 (IMPLY_MIN_2, IMPLY_TARGET, IMPLY_CMD, IMPLY_NON_ARCH_BOARD) = (
1611 'min2': [IMPLY_MIN_2, 'Show options which imply >2 boards (normally >5)'],
1612 'target': [IMPLY_TARGET, 'Allow CONFIG_TARGET_... options to imply'],
1613 'cmd': [IMPLY_CMD, 'Allow CONFIG_CMD_... to imply'],
1615 IMPLY_NON_ARCH_BOARD,
1616 'Allow Kconfig options outside arch/ and /board/ to imply'],
1619 def do_imply_config(config_list, add_imply, imply_flags, skip_added,
1620 check_kconfig=True, find_superset=False):
1621 """Find CONFIG options which imply those in the list
1623 Some CONFIG options can be implied by others and this can help to reduce
1624 the size of the defconfig files. For example, CONFIG_X86 implies
1625 CONFIG_CMD_IRQ, so we can put 'imply CMD_IRQ' under 'config X86' and
1626 all x86 boards will have that option, avoiding adding CONFIG_CMD_IRQ to
1627 each of the x86 defconfig files.
1629 This function uses the moveconfig database to find such options. It
1630 displays a list of things that could possibly imply those in the list.
1631 The algorithm ignores any that start with CONFIG_TARGET since these
1632 typically refer to only a few defconfigs (often one). It also does not
1633 display a config with less than 5 defconfigs.
1635 The algorithm works using sets. For each target config in config_list:
1636 - Get the set 'defconfigs' which use that target config
1637 - For each config (from a list of all configs):
1638 - Get the set 'imply_defconfig' of defconfigs which use that config
1640 - If imply_defconfigs contains anything not in defconfigs then
1641 this config does not imply the target config
1644 config_list: List of CONFIG options to check (each a string)
1645 add_imply: Automatically add an 'imply' for each config.
1646 imply_flags: Flags which control which implying configs are allowed
1648 skip_added: Don't show options which already have an imply added.
1649 check_kconfig: Check if implied symbols already have an 'imply' or
1650 'select' for the target config, and show this information if so.
1651 find_superset: True to look for configs which are a superset of those
1652 already found. So for example if CONFIG_EXYNOS5 implies an option,
1653 but CONFIG_EXYNOS covers a larger set of defconfigs and also
1654 implies that option, this will drop the former in favour of the
1655 latter. In practice this option has not proved very used.
1657 Note the terminoloy:
1658 config - a CONFIG_XXX options (a string, e.g. 'CONFIG_CMD_EEPROM')
1659 defconfig - a defconfig file (a string, e.g. 'configs/snow_defconfig')
1661 kconf = KconfigScanner().conf if check_kconfig else None
1662 if add_imply and add_imply != 'all':
1663 add_imply = add_imply.split()
1665 # key is defconfig name, value is dict of (CONFIG_xxx, value)
1668 # Holds a dict containing the set of defconfigs that contain each config
1669 # key is config, value is set of defconfigs using that config
1670 defconfig_db = collections.defaultdict(set)
1672 # Set of all config options we have seen
1675 # Set of all defconfigs we have seen
1676 all_defconfigs = set()
1678 # Read in the database
1680 with open(CONFIG_DATABASE) as fd:
1681 for line in fd.readlines():
1682 line = line.rstrip()
1683 if not line: # Separator between defconfigs
1684 config_db[defconfig] = configs
1685 all_defconfigs.add(defconfig)
1687 elif line[0] == ' ': # CONFIG line
1688 config, value = line.strip().split('=', 1)
1689 configs[config] = value
1690 defconfig_db[config].add(defconfig)
1691 all_configs.add(config)
1692 else: # New defconfig
1695 # Work through each target config option in tern, independently
1696 for config in config_list:
1697 defconfigs = defconfig_db.get(config)
1699 print('%s not found in any defconfig' % config)
1702 # Get the set of defconfigs without this one (since a config cannot
1704 non_defconfigs = all_defconfigs - defconfigs
1705 num_defconfigs = len(defconfigs)
1706 print('%s found in %d/%d defconfigs' % (config, num_defconfigs,
1709 # This will hold the results: key=config, value=defconfigs containing it
1711 rest_configs = all_configs - set([config])
1713 # Look at every possible config, except the target one
1714 for imply_config in rest_configs:
1715 if 'ERRATUM' in imply_config:
1717 if not (imply_flags & IMPLY_CMD):
1718 if 'CONFIG_CMD' in imply_config:
1720 if not (imply_flags & IMPLY_TARGET):
1721 if 'CONFIG_TARGET' in imply_config:
1724 # Find set of defconfigs that have this config
1725 imply_defconfig = defconfig_db[imply_config]
1727 # Get the intersection of this with defconfigs containing the
1729 common_defconfigs = imply_defconfig & defconfigs
1731 # Get the set of defconfigs containing this config which DO NOT
1732 # also contain the taret config. If this set is non-empty it means
1733 # that this config affects other defconfigs as well as (possibly)
1734 # the ones affected by the target config. This means it implies
1735 # things we don't want to imply.
1736 not_common_defconfigs = imply_defconfig & non_defconfigs
1737 if not_common_defconfigs:
1740 # If there are common defconfigs, imply_config may be useful
1741 if common_defconfigs:
1744 for prev in list(imply_configs.keys()):
1745 prev_count = len(imply_configs[prev])
1746 count = len(common_defconfigs)
1747 if (prev_count > count and
1748 (imply_configs[prev] & common_defconfigs ==
1749 common_defconfigs)):
1750 # skip imply_config because prev is a superset
1753 elif count > prev_count:
1754 # delete prev because imply_config is a superset
1755 del imply_configs[prev]
1757 imply_configs[imply_config] = common_defconfigs
1759 # Now we have a dict imply_configs of configs which imply each config
1760 # The value of each dict item is the set of defconfigs containing that
1761 # config. Rank them so that we print the configs that imply the largest
1762 # number of defconfigs first.
1763 ranked_iconfigs = sorted(imply_configs,
1764 key=lambda k: len(imply_configs[k]), reverse=True)
1767 add_list = collections.defaultdict(list)
1768 for iconfig in ranked_iconfigs:
1769 num_common = len(imply_configs[iconfig])
1771 # Don't bother if there are less than 5 defconfigs affected.
1772 if num_common < (2 if imply_flags & IMPLY_MIN_2 else 5):
1774 missing = defconfigs - imply_configs[iconfig]
1775 missing_str = ', '.join(missing) if missing else 'all'
1779 sym = find_kconfig_rules(kconf, config[CONFIG_LEN:],
1780 iconfig[CONFIG_LEN:])
1783 locs = sym.get_def_locations()
1785 fname, linenum = locs[0]
1786 if cwd and fname.startswith(cwd):
1787 fname = fname[len(cwd) + 1:]
1788 kconfig_info = '%s:%d' % (fname, linenum)
1792 sym = kconf.syms.get(iconfig[CONFIG_LEN:])
1795 locs = sym.get_def_locations()
1797 fname, linenum = locs[0]
1798 if cwd and fname.startswith(cwd):
1799 fname = fname[len(cwd) + 1:]
1800 in_arch_board = not sym or (fname.startswith('arch') or
1801 fname.startswith('board'))
1802 if (not in_arch_board and
1803 not (imply_flags & IMPLY_NON_ARCH_BOARD)):
1806 if add_imply and (add_imply == 'all' or
1807 iconfig in add_imply):
1808 fname, linenum, kconfig_info = (check_imply_rule(kconf,
1809 config[CONFIG_LEN:], iconfig[CONFIG_LEN:]))
1811 add_list[fname].append(linenum)
1813 if show and kconfig_info != 'skip':
1814 print('%5d : %-30s%-25s %s' % (num_common, iconfig.ljust(30),
1815 kconfig_info, missing_str))
1817 # Having collected a list of things to add, now we add them. We process
1818 # each file from the largest line number to the smallest so that
1819 # earlier additions do not affect our line numbers. E.g. if we added an
1820 # imply at line 20 it would change the position of each line after
1822 for fname, linenums in add_list.items():
1823 for linenum in sorted(linenums, reverse=True):
1824 add_imply_rule(config[CONFIG_LEN:], fname, linenum)
1829 cpu_count = multiprocessing.cpu_count()
1830 except NotImplementedError:
1833 parser = optparse.OptionParser()
1835 parser.add_option('-a', '--add-imply', type='string', default='',
1836 help='comma-separated list of CONFIG options to add '
1837 "an 'imply' statement to for the CONFIG in -i")
1838 parser.add_option('-A', '--skip-added', action='store_true', default=False,
1839 help="don't show options which are already marked as "
1841 parser.add_option('-b', '--build-db', action='store_true', default=False,
1842 help='build a CONFIG database')
1843 parser.add_option('-c', '--color', action='store_true', default=False,
1844 help='display the log in color')
1845 parser.add_option('-C', '--commit', action='store_true', default=False,
1846 help='Create a git commit for the operation')
1847 parser.add_option('-d', '--defconfigs', type='string',
1848 help='a file containing a list of defconfigs to move, '
1849 "one per line (for example 'snow_defconfig') "
1850 "or '-' to read from stdin")
1851 parser.add_option('-i', '--imply', action='store_true', default=False,
1852 help='find options which imply others')
1853 parser.add_option('-I', '--imply-flags', type='string', default='',
1854 help="control the -i option ('help' for help")
1855 parser.add_option('-n', '--dry-run', action='store_true', default=False,
1856 help='perform a trial run (show log with no changes)')
1857 parser.add_option('-e', '--exit-on-error', action='store_true',
1859 help='exit immediately on any error')
1860 parser.add_option('-s', '--force-sync', action='store_true', default=False,
1861 help='force sync by savedefconfig')
1862 parser.add_option('-S', '--spl', action='store_true', default=False,
1863 help='parse config options defined for SPL build')
1864 parser.add_option('-H', '--headers-only', dest='cleanup_headers_only',
1865 action='store_true', default=False,
1866 help='only cleanup the headers')
1867 parser.add_option('-j', '--jobs', type='int', default=cpu_count,
1868 help='the number of jobs to run simultaneously')
1869 parser.add_option('-r', '--git-ref', type='string',
1870 help='the git ref to clone for building the autoconf.mk')
1871 parser.add_option('-y', '--yes', action='store_true', default=False,
1872 help="respond 'yes' to any prompts")
1873 parser.add_option('-v', '--verbose', action='store_true', default=False,
1874 help='show any build errors as boards are built')
1875 parser.usage += ' CONFIG ...'
1877 (options, configs) = parser.parse_args()
1879 if len(configs) == 0 and not any((options.force_sync, options.build_db,
1881 parser.print_usage()
1884 # prefix the option name with CONFIG_ if missing
1885 configs = [ config if config.startswith('CONFIG_') else 'CONFIG_' + config
1886 for config in configs ]
1888 check_top_directory()
1892 if options.imply_flags == 'all':
1895 elif options.imply_flags:
1896 for flag in options.imply_flags.split(','):
1897 bad = flag not in IMPLY_FLAGS
1899 print("Invalid flag '%s'" % flag)
1900 if flag == 'help' or bad:
1901 print("Imply flags: (separate with ',')")
1902 for name, info in IMPLY_FLAGS.items():
1903 print(' %-15s: %s' % (name, info[1]))
1904 parser.print_usage()
1906 imply_flags |= IMPLY_FLAGS[flag][0]
1908 do_imply_config(configs, options.add_imply, imply_flags,
1913 db_queue = queue.Queue()
1914 t = DatabaseThread(config_db, db_queue)
1918 if not options.cleanup_headers_only:
1919 check_clean_directory()
1921 toolchains = toolchain.Toolchains()
1922 toolchains.GetSettings()
1923 toolchains.Scan(verbose=False)
1924 move_config(toolchains, configs, options, db_queue)
1928 cleanup_headers(configs, options)
1929 cleanup_extra_options(configs, options)
1930 cleanup_whitelist(configs, options)
1931 cleanup_readme(configs, options)
1934 subprocess.call(['git', 'add', '-u'])
1936 msg = 'Convert %s %sto Kconfig' % (configs[0],
1937 'et al ' if len(configs) > 1 else '')
1938 msg += ('\n\nThis converts the following to Kconfig:\n %s\n' %
1939 '\n '.join(configs))
1941 msg = 'configs: Resync with savedefconfig'
1942 msg += '\n\nRsync all defconfig files using moveconfig.py'
1943 subprocess.call(['git', 'commit', '-s', '-m', msg])
1945 if options.build_db:
1946 with open(CONFIG_DATABASE, 'w') as fd:
1947 for defconfig, configs in config_db.items():
1948 fd.write('%s\n' % defconfig)
1949 for config in sorted(configs.keys()):
1950 fd.write(' %s=%s\n' % (config, configs[config]))
1953 if __name__ == '__main__':