chainer.iterators.MultiprocessIterator

class chainer.iterators.MultiprocessIterator(dataset, batch_size, repeat=True, shuffle=True, n_processes=None, n_prefetch=1, shared_mem=None)

Dataset iterator that loads examples in parallel.

This is an implementation of Iterator that loads examples with worker processes. It uses the standard multiprocessing module to parallelize the loading. The dataset is sent to the worker processes in the standard way using pickle.

Note that this iterator effectively prefetches the examples for the next batch asynchronously after the current batch is returned.

This iterator saves -1 instead of None in snapshots since some serializers do not support None.

Parameters:

• dataset (Dataset) – Dataset to iterate.
• batch_size (int) – Number of examples within each batch.
• repeat (bool) – If True, it infinitely loops over the dataset. Otherwise, it stops iteration at the end of the first epoch.
• shuffle (bool) – If True, the order of examples is shuffled at the beginning of each epoch. Otherwise, examples are extracted in the order of indexes.
• n_processes (int) – Number of worker processes. The number of CPUs is used by default.
• n_prefetch (int) – Number of prefetch batches.
• shared_mem (int) – The size of using shared memory per data. If None, size is adjusted automatically.

Methods

__next__()

Returns the next batch.

This is a part of the iterator protocol of Python. It may raise the StopIteration exception when it stops the iteration.

__iter__()

Returns self.

__copy__()

finalize()

next()

Returns the next batch.

This is a part of the iterator protocol of Python. It may raise the StopIteration exception when it stops the iteration.

reset()

serialize(serializer)

Serializes the internal state of the iterator.

This is a method to support serializer protocol of Chainer.

Note

It should only serialize the internal state that changes over the iteration. It should not serializes what is set manually by users such as the batch size.

Attributes

epoch_detail

previous_epoch_detail