[bitbake-devel] [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache
Martin Jansa
martin.jansa at gmail.com
Sat Dec 28 13:24:13 UTC 2013
On Wed, Dec 18, 2013 at 04:21:27PM +0000, Richard Purdie wrote:
> Its useful to understand where the delta starts against an existing sstate cache
> for a given target. Adding this to the output of the -S option seems like a
> natural fit.
>
> We use the hashvalidate function to figure this out and assume it can find siginfo
> files for more than just the setscene tasks.
I haven't tried to find smaller reproducer or exact part of code where
it happends, but in bigger builds I have it fails with
I'll try to reproduce and narrow it a bit, but if you have tip where
to look first..
IndexError: list index out of range:
NOTE: Reparsing files to collect dependency data
> self.write_diffscenetasks(invalidtasks)
self.state = runQueueComplete
File "/OE/webos/owub/build-starfish-master/bitbake/lib/bb/runqueue.py", line 1227, in RunQueue.write_diffscenetasks(invalidtasks=set([16694, 11264, 5803, 4100, 2054, 16391, 2060, 16, 14354, 4115, 22, 14360, 6148, 8218, 2075, 10268, 13658, 8224, 4130, 18467, 16421, 38, 2087, 10283, 6188, 18687, 4145, 6194, 12686, 2105, 10298, 12347, 62, 14399, 4160, 12353, 16452, 14406, 7231, 6218, 13312, 77, 6224, 2130, 83, 14421, 4184, 15756, 4190, 6239, 2145, 98, 10939, 14436, 10598, 2151, 8296, 4202, 8302, 11665, 6257, 6163, 16763, 14949, 123, 4220, 14461, 2175, 10368, 129, 14468, 2181, 8326, 6849, 12507, 16748, 4235, 6286, 10383, 14480, 13336, 146, 8341, 10390, 10391, 4250, 6302, 161, 14498, 2211, 8357, 11735, 16552, 4265, 8363, 6316, 8562, 12462, 10415, 14513, 2226, 10421, 16567, 8378, 4283, 4286, 8565, 5152, 10436, 14710, 12486, 4295, 200, 8393, 2250, 16588, 12322, 206, 14543, 10452, 6357, 4310, 8408, 6363, 221, 14558, 2272, 7888, 10619, 6375, 17623, 12522, 14375, 10476, 4334, 2287, 8432, 16784, 16627, 18677, 8438, 16633, 4349, 2302, 17409, 10497, 6402, 4355, 7851, 13697, 6408, 12554, 268, 2317, 8462, 13357, 13818, 8468, 16376, 14650, 16663, 6424, 283, 2332, 15823, 10528, 8483, 16648, 6439, 2347, 16192, 4401, 8498, 14644, 6454, 10552, 313, 14023, 7903, 9285, 8513, 2371, 6469, 13718, 14741, 2377, 16446, 14668, 10573, 4431, 337, 6484, 12629, 343, 11741, 16733, 4446, 8544, 14689, 6499, 2404, 358, 1425, 10604, 12656, 6514, 2419, 8254, 374, 6889, 12692, 14665, 5183, 8574, 12671, 6529, 2434, 16107, 389, 4487, 2440, 4493, 8590, 14735, 6331, 10644, 405, 406, 2455, 6553, 4508, 8605, 3141, 420, 12709, 2470, 10667, 17938, 8621, 6574, 12153, 4532, 10313, 4379, 16825, 4538, 6559, 444, 6589, 13381, 7584, 450, 14787, 2500, 8645, 17814, 10697, 8651, 6604, 8269, 4562, 2515, 16856, 10713, 474, 6620, 477, 13733, 12291, 14817, 4578, 12710, 4584, 2539, 12370, 12382, 14832, 2545, 498, 12371, 501, 4599, 8696, 15444, 8814, 6653, 510, 14847, 6656, 2561, 12033, 6665, 8714, 8717, 2575, 10328, 4629, 535, 8728, 7599, 18567, 541, 2590, 12831, 6689, 9307, 4645, 8742, 6695, 14888, 7943, 556, 2605, 17584, 7262, 6710, 12855, 12724, 571, 10813, 14910, 15991, 10821, 2630, 16823, 2636, 4685, 17273, 15459, 596, 8789, 10838, 10839, 15460, 12891, 4700, 10682, 16300, 13111, 14955, 613, 11068, 2664, 619, 6764, 16146, 2670, 6770, 8820, 5566, 14970, 9663, 2685, 13251, 3861, 4736, 6786, 643, 12932, 10006, 14985, 7523, 10893, 4751, 658, 12947, 10900, 14038, 15000, 12740, 8859, 2716, 6815, 8865, 674, 10915, 12093, 10353, 14108, 13426, 689, 12978, 2740, 15030, 695, 8889, 2746, 4795, 8895, 4801, 710, 2761, 10954, 13004, 13005, 8910, 8529, 4816, 13971, 15060, 6873, 13019, 735, 10976, 2785, 16507, 10822, 741, 8934, 2791, 4841, 16528, 13035, 15084, 8943, 8946, 12755, 9777, 5588, 4858, 6271, 8956, 15146, 11007, 2816, 12416, 15106, 772, 2822, 16513, 4873, 6922, 9687, 11022, 4879, 6928, 15121, 787, 2837, 14620, 13080, 17348, 8666, 4894, 12779, 802, 2852, 11047, 6952, 7644, 9002, 9003, 15090, 11053, 6621, 13104, 817, 2868, 4918, 6967, 13112, 5940, 17690, 4924, 14772, 832, 9026, 15499, 9029, 15176, 17483, 2891, 10262, 15182, 847, 13136, 6997, 15161, 4954, 12431, 15197, 862, 2912, 13153, 9062, 4969, 877, 10767, 10728, 7029, 11128, 4985, 7038, 2943, 13184, 7041, 6635, 9093, 902, 13191, 5000, 15243, 908, 11153, 17300, 11159, 5016, 5017, 15258, 9117, 13215, 2976, 7075, 933, 5031, 15273, 9132, 8690, 2991, 7090, 15435, 948, 11189, 16603, 5047, 7096, 3003, 957, 16467, 4939, 11204, 5062, 7111, 9162, 971, 3021, 15312, 12785, 5077, 7126, 17368, 9177, 986, 3036, 7674, 9183, 11234, 5092, 7141, 1001, 16799, 3051, 9199, 11249, 5107, 7157, 1016, 3066, 12117, 9216, 10752, 7171, 15364, 17765, 3081, 5131, 9228, 12446, 1040, 5137, 7186, 11696, 15380, 3096, 11289, 9243, 2906, 13342, 1056, 15392, 1062, 3111, 7210, 9261, 11311, 7216, 15410, 13402, 1078, 17615, 5177, 14799, 16709, 11326, 3135, 16483, 9056, 11098, 1093, 9291, 7246, 10083, 13396, 1109, 11350, 5208, 9991, 4623, 11356, 3165, 5214, 16229, 3171, 1124, 16406, 15474, 9322, 8039, 5229, 13501, 3186, 1139, 8835, 16006, 13432, 9337, 15551, 5244, 10773, 1156, 1157, 7302, 13447, 3210, 5259, 11478, 1171, 7317, 13462, 9367, 7320, 3225, 5274, 15515, 3231, 15521, 11426, 16582, 9383, 5289, 7340, 7341, 13486, 6041, 7973, 1202, 9398, 3255, 5304, 15545, 7355, 3261, 18557, 15221, 1218, 12815, 13510, 11463, 13513, 7372, 7373, 12316, 9423, 11596, 3285, 5334, 2937, 1242, 3291, 11083, 1248, 13538, 9445, 5350, 17639, 7402, 13863, 15596, 3309, 11502, 14704, 9460, 6064, 1273, 13562, 15611, 5374, 12501, 11520, 13569, 9475, 9430, 11526, 5383, 3336, 5386, 15575, 1294, 10797, 5395, 11541, 7447, 3352, 3354, 9499, 1309, 13598, 15237, 5410, 7387, 11556, 7477, 7462, 7729, 3368, 9514, 16022, 11571, 1333, 13622, 15672, 9529, 1339, 13628, 11296, 9535, 3392, 5441, 7493, 3398, 15687, 13643, 8774, 9550, 8024, 11602, 1363, 7508, 3413, 15702, 13881, 5466, 5472, 11617, 7820, 1379, 3428, 15590, 9575, 13673, 17770, 13276, 5487, 1394, 3443, 17784, 14247, 11643, 7548, 9597, 3648, 1410, 10208, 13703, 17644, 11658, 3467, 9612, 5518, 3473, 15762, 11667, 14871, 14574, 12800, 1431, 9627, 3488, 16283, 7750, 1447, 11690, 15787, 5550, 3503, 5552, 15793, 9651, 9660, 13757, 3518, 1471, 15808, 13763, 7623, 9672, 12876, 17868, 7629, 1486, 14575, 14017, 5585, 13778, 1492, 11711, 3542, 7417, 3548, 5597, 13221, 17888, 13793, 10960, 9702, 15848, 7659, 5612, 1517, 253, 11371, 3572, 9717, 11766, 6740, 3578, 5627, 11772, 1533, 15870, 13824, 1539, 9732, 3593, 5642, 11787, 15885, 13839, 1554, 9747, 5654, 15961, 3608, 7705, 11802, 11183, 17948, 1569, 9762, 17958, 3623, 5672, 11441, 14599, 15916, 9138, 7554, 1288, 11829, 7735, 13236, 15931, 15285, 1600, 5697, 11844, 14262, 3654, 13896, 12215, 13902, 9807, 5712, 11859, 7765, 5718, 1624, 11865, 17679, 16177, 13917, 1630, 10853, 955, 5733, 9831, 16316, 13932, 7789, 1646, 11889, 15635, 5748, 3701, 9846, 11895, 13947, 7804, 4715, 3716, 1670, 11911, 9864, 18028, 3722, 1676, 18061, 9870, 12568, 7826, 10558, 4718, 11926, 15741, 13977, 13050, 5787, 9885, 16034, 3747, 1701, 3753, 1707, 9901, 11950, 5809, 14002, 14605, 16052, 3765, 12916, 1722, 9916, 17354, 15717, 7873, 15947, 11972, 18038, 3783, 1737, 9931, 16076, 15650, 5840, 3798, 9946, 16091, 11996, 14925, 5855, 13592, 1761, 12002, 18644, 3814, 1767, 9961, 12583, 5870, 10877, 15656, 12018, 3828, 14069, 7928, 16122, 1792, 9985, 3843, 14084, 5895, 16140, 7949, 1807, 12048, 7698, 5763, 1813, 5910, 3864, 17028, 5916, 12063, 3873, 16162, 1829, 11228, 7979, 12078, 3888, 14129, 1844, 5944, 15327, 5949, 10046, 3903, 14144, 7277, 8003, 1860, 12598, 14756, 5961, 12269, 13282, 3918, 14159, 17720, 8018, 10068, 1877, 5976, 16217, 3933, 14174, 15333, 16226, 5435, 12132, 1893, 5991, 13065, 16362, 10098, 12147, 1908, 3957, 8054, 14199, 10160, 3963, 16253, 6015, 1923, 8069, 14216, 12169, 10123, 16268, 14222, 6031, 1938, 3987, 8084, 6039, 12184, 3993, 10139, 17591, 12613, 1953, 9792, 13297, 4008, 6812, 10154, 8108, 1968, 12209, 8114, 6070, 4425, 14268, 16317, 12861, 1983, 4033, 8130, 6085, 12230, 4039, 8727, 10185, 14283, 1998, 10193, 10194, 12963, 16340, 12245, 4055, 16346, 6112, 8163, 14308, 4069, 6118, 2023, 12547, 14314, 15015, 18071, 2029, 10223, 7569, 8178, 6133, 14327, 4088, 4091, 16844, 12285, 10238])):
matches = {k : v for k, v in matches.iteritems() if h not in k}
> latestmatch = sorted(matches.keys(), key=lambda f: matches[f])[-1]
prevh = __find_md5__.search(latestmatch).group(0)
IndexError: list index out of range
>
> Signed-off-by: Richard Purdie <richard.purdie at linuxfoundation.org>
> ---
> diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
> index 428cff1..b7a602b 100644
> --- a/bitbake/lib/bb/runqueue.py
> +++ b/bitbake/lib/bb/runqueue.py
> @@ -1039,6 +1039,7 @@ class RunQueue:
>
> if self.state is runQueueSceneInit:
> if self.cooker.configuration.dump_signatures:
> + self.print_diffscenetasks()
> self.dump_signatures()
> else:
> self.start_worker()
> @@ -1123,6 +1124,82 @@ class RunQueue:
>
> return
>
> + def print_diffscenetasks(self):
> +
> + valid = []
> + sq_hash = []
> + sq_hashfn = []
> + sq_fn = []
> + sq_taskname = []
> + sq_task = []
> + noexec = []
> + stamppresent = []
> + valid_new = set()
> +
> + for task in xrange(len(self.rqdata.runq_fnid)):
> + fn = self.rqdata.taskData.fn_index[self.rqdata.runq_fnid[task]]
> + taskname = self.rqdata.runq_task[task]
> + taskdep = self.rqdata.dataCache.task_deps[fn]
> +
> + if 'noexec' in taskdep and taskname in taskdep['noexec']:
> + noexec.append(task)
> + continue
> +
> + sq_fn.append(fn)
> + sq_hashfn.append(self.rqdata.dataCache.hashfn[fn])
> + sq_hash.append(self.rqdata.runq_hash[task])
> + sq_taskname.append(taskname)
> + sq_task.append(task)
> + call = self.hashvalidate + "(sq_fn, sq_task, sq_hash, sq_hashfn, d)"
> + locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname, "sq_hash" : sq_hash, "sq_hashfn" : sq_hashfn, "d" : self.cooker.data }
> + valid = bb.utils.better_eval(call, locs)
> + for v in valid:
> + valid_new.add(sq_task[v])
> +
> + # Tasks which are both setscene and noexec never care about dependencies
> + # We therefore find tasks which are setscene and noexec and mark their
> + # unique dependencies as valid.
> + for task in noexec:
> + if task not in self.rqdata.runq_setscene:
> + continue
> + for dep in self.rqdata.runq_depends[task]:
> + hasnoexecparents = True
> + for dep2 in self.rqdata.runq_revdeps[dep]:
> + if dep2 in self.rqdata.runq_setscene and dep2 in noexec:
> + continue
> + hasnoexecparents = False
> + break
> + if hasnoexecparents:
> + valid_new.add(dep)
> +
> + invalidtasks = set()
> + for task in xrange(len(self.rqdata.runq_fnid)):
> + if task not in valid_new and task not in noexec:
> + invalidtasks.add(task)
> +
> + found = set()
> + processed = set()
> + for task in invalidtasks:
> + toprocess = set([task])
> + while toprocess:
> + next = set()
> + for t in toprocess:
> + for dep in self.rqdata.runq_depends[t]:
> + if dep in invalidtasks:
> + found.add(task)
> + if dep not in processed:
> + processed.add(dep)
> + next.add(dep)
> + toprocess = next
> + if task in found:
> + toprocess = set()
> +
> + tasklist = []
> + for task in invalidtasks.difference(found):
> + tasklist.append(self.rqdata.get_user_idstring(task))
> +
> + if tasklist:
> + bb.plain("The differences between the current build and any cached tasks start at the following tasks:\n" + "\n".join(tasklist))
>
> class RunQueueExecute:
>
>
>
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel at lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
--
Martin 'JaMa' Jansa jabber: Martin.Jansa at gmail.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://lists.openembedded.org/pipermail/bitbake-devel/attachments/20131228/22731510/attachment-0002.sig>
More information about the bitbake-devel
mailing list