问题描述
我很好奇使用迭代器的最快方式是什么,也是最 Pythonic 的方式.
I am curious what the fastest way to consume an iterator would be, and the most Pythonic way.
例如,假设我想创建一个带有 map 内置函数的迭代器,它会累积一些东西作为副作用.我实际上并不关心 map 的结果,只关心副作用,所以我想以尽可能少的开销或样板文件来完成迭代.比如:
For example, say that I want to create an iterator with the map builtin that accumulates something as a side-effect. I don't actually care about the result of the map, just the side effect, so I want to blow through the iteration with as little overhead or boilerplate as possible. Something like:
my_set = set() my_map = map(lambda x, y: my_set.add((x, y)), my_x, my_y)
在这个例子中,我只是想通过迭代器来累积 my_set 中的东西,而 my_set 只是一个空集,直到我真正运行 我的地图.比如:
In this example, I just want to blow through the iterator to accumulate things in my_set, and my_set is just an empty set until I actually run through my_map. Something like:
for _ in my_map: pass
或赤身裸体
[_ for _ in my_map]
有效,但他们都觉得笨重.有没有更 Pythonic 的方法来确保迭代器快速迭代,以便您从一些副作用中受益?
works, but they both feel clunky. Is there a more Pythonic way to make sure an iterator iterates quickly so that you can benefit from some side-effect?
我在以下方面测试了上述两种方法:
I tested the two methods above on the following:
my_x = np.random.randint(100, size=int(1e6)) my_y = np.random.randint(100, size=int(1e6))
与上面定义的 my_set 和 my_map 一起使用.我用 timeit 得到了以下结果:
with my_set and my_map as defined above. I got the following results with timeit:
for _ in my_map: pass 468 ms ± 20.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) [_ for _ in my_map] 476 ms ± 12.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
两者之间没有真正的区别,而且都感觉很笨重.
No real difference between the two, and they both feel clunky.
注意,我使用 list(my_map) 获得了类似的性能,这是评论中的建议.
Note, I got similar performance with list(my_map), which was a suggestion in the comments.
推荐答案
虽然您不应该仅仅为了副作用而创建地图对象,但实际上在 itertools 文档:
While you shouldn't be creating a map object just for side effects, there is in fact a standard recipe for consuming iterators in the itertools docs:
def consume(iterator, n=None): "Advance the iterator n-steps ahead. If n is None, consume entirely." # Use functions that consume iterators at C speed. if n is None: # feed the entire iterator into a zero-length deque collections.deque(iterator, maxlen=0) else: # advance to the empty slice starting at position n next(islice(iterator, n, n), None)
对于完全消费"的情况,这可以简化为
For just the "consume entirely" case, this can be simplified to
def consume(iterator): collections.deque(iterator, maxlen=0)
以这种方式使用 collections.deque 可以避免存储所有元素(因为 maxlen=0)并以 C 速度迭代,没有字节码解释开销.双端队列中甚至还有一个专用快速路径使用 maxlen=0 双端队列来使用迭代器的实现.
Using collections.deque this way avoids storing all the elements (because maxlen=0) and iterates at C speed, without bytecode interpretation overhead. There's even a dedicated fast path in the deque implementation for using a maxlen=0 deque to consume an iterator.
时间:
In [1]: import collections In [2]: x = range(1000) In [3]: %%timeit ...: i = iter(x) ...: for _ in i: ...: pass ...: 16.5 μs ± 829 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each) In [4]: %%timeit ...: i = iter(x) ...: collections.deque(i, maxlen=0) ...: 12 μs ± 566 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
当然,这都是基于 CPython 的.解释器开销的整个性质在其他 Python 实现中非常不同,并且 maxlen=0 快速路径特定于 CPython.有关其他 Python 实现,请参阅 abarnert 的回答.
Of course, this is all based on CPython. The entire nature of interpreter overhead is very different on other Python implementations, and the maxlen=0 fast path is specific to CPython. See abarnert's answer for other Python implementations.