Java BlockingQueueの例
Javaでは、BlockingQueueを使用して、プロデューサーとコンシューマーの両方で共有されるキューを作成できます。
-
プロデューサー–データを生成してキューに入れます。
-
コンシューマ–キューからデータを削除します。
Note
この記事を読んで、producer and consumerとは何かを理解してください。
BlockingQueueの実装はスレッドセーフであり、複数のプロデューサーと複数のコンシューマーで安全に使用できます。
1. BlockingQueue
単純なBlockingQueueの例では、プロデューサーがデータを生成してキューに入れ、同時にコンシューマーが同じキューからデータを取得します。
1.1 Producer – A Runnable object to put 20 integers into a queue.
ExecutorExample1.java
package com.example.concurrency.queue.simple.raw;
import java.util.concurrent.BlockingQueue;
public class Producer implements Runnable {
private final BlockingQueue queue;
@Override
public void run() {
try {
process();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
private void process() throws InterruptedException {
// Put 20 ints into Queue
for (int i = 0; i < 20; i++) {
System.out.println("[Producer] Put : " + i);
queue.put(i);
System.out.println("[Producer] Queue remainingCapacity : " + queue.remainingCapacity());
Thread.sleep(100);
}
}
public Producer(BlockingQueue queue) {
this.queue = queue;
}
}
1.2 Consumer – A Runnable object to take items from a queue.
Consumer.java
package com.example.concurrency.queue.simple.raw;
import java.util.concurrent.BlockingQueue;
public class Consumer implements Runnable {
private final BlockingQueue queue;
@Override
public void run() {
try {
while (true) {
Integer take = queue.take();
process(take);
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
private void process(Integer take) throws InterruptedException {
System.out.println("[Consumer] Take : " + take);
Thread.sleep(500);
}
public Consumer(BlockingQueue queue) {
this.queue = queue;
}
}
1.3 Run it. 1つのプロデューサーと1つのコンシューマーを開始し、サイズが10のキューを作成します。
Main.java
package com.example.concurrency.queue.simple;
import com.example.concurrency.queue.simple.raw.Consumer;
import com.example.concurrency.queue.simple.raw.Producer;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
public class Main {
public static void main(String[] args) {
BlockingQueue queue = new LinkedBlockingQueue<>(10);
new Thread(new Producer(queue)).start();
new Thread(new Consumer(queue)).start();
}
}
出力
プロデューサーは、データがいっぱいの場合、それ以上データをキューに入れようとしません。
[Producer] Put : 0 [Producer] Queue remainingCapacity : 9 [Consumer] Take : 0 [Producer] Put : 1 [Producer] Queue remainingCapacity : 9 [Producer] Put : 2 [Producer] Queue remainingCapacity : 8 [Producer] Put : 3 [Producer] Queue remainingCapacity : 7 [Producer] Put : 4 [Producer] Queue remainingCapacity : 6 [Producer] Put : 5 [Producer] Queue remainingCapacity : 5 [Consumer] Take : 1 [Producer] Put : 6 [Producer] Queue remainingCapacity : 5 [Producer] Put : 7 [Producer] Queue remainingCapacity : 4 [Producer] Put : 8 [Producer] Queue remainingCapacity : 3 [Producer] Put : 9 [Producer] Queue remainingCapacity : 2 [Producer] Put : 10 [Producer] Queue remainingCapacity : 1 [Consumer] Take : 2 [Producer] Put : 11 [Producer] Queue remainingCapacity : 1 [Producer] Put : 12 [Producer] Queue remainingCapacity : 0 [Producer] Put : 13 [Consumer] Take : 3 [Producer] Queue remainingCapacity : 0 [Producer] Put : 14 [Consumer] Take : 4 [Producer] Queue remainingCapacity : 0 [Producer] Put : 15 [Consumer] Take : 5 [Producer] Queue remainingCapacity : 0 [Producer] Put : 16 [Consumer] Take : 6 [Producer] Queue remainingCapacity : 0 [Producer] Put : 17 [Consumer] Take : 7 [Producer] Queue remainingCapacity : 0 [Producer] Put : 18 [Consumer] Take : 8 [Producer] Queue remainingCapacity : 0 [Producer] Put : 19 [Consumer] Take : 9 [Producer] Queue remainingCapacity : 0 [Consumer] Take : 10 [Consumer] Take : 11 [Consumer] Take : 12 [Consumer] Take : 13 [Consumer] Take : 14 [Consumer] Take : 15 [Consumer] Take : 16 [Consumer] Take : 17 [Consumer] Take : 18 [Consumer] Take : 19
プログラムは停止または終了せず、BlockingQueueからデータを出し入れするために、そこで実行を続けます。
2. BlockingQueue +毒薬
「毒薬」は、生産者スレッドと消費者スレッドの両方を停止または中断する一般的なソリューションです。 アイデアは、生産者が「毒薬」をキューに入れて終了し、「消費者」が「毒薬」を見ると停止して終了するというものです。
2.1 A producer with poison pill solution.
ProducerPoison.java
package com.example.concurrency.queue.simple.poison;
import java.util.concurrent.BlockingQueue;
public class ProducerPoison implements Runnable {
private final BlockingQueue queue;
private final Integer POISON;
@Override
public void run() {
try {
process();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
} finally {
while (true) {
try {
queue.put(POISON);
break;
} catch (InterruptedException e) {
//...
}
}
}
}
private void process() throws InterruptedException {
// Put 20 elements into Queue
for (int i = 0; i < 20; i++) {
System.out.println("[Producer] Put : " + i);
queue.put(i);
System.out.println("[Producer] Queue remainingCapacity : " + queue.remainingCapacity());
Thread.sleep(100);
}
}
public ProducerPoison(BlockingQueue queue, Integer POISON) {
this.queue = queue;
this.POISON = POISON;
}
}
2.2 A consumer with poison pill solution.
ConsumerPoison.java
package com.example.concurrency.queue.simple.poison;
import java.util.concurrent.BlockingQueue;
public class ConsumerPoison implements Runnable {
private final BlockingQueue queue;
private final Integer POISON;
@Override
public void run() {
try {
while (true) {
Integer take = queue.take();
process(take);
// if this is a poison pill, break, exit
if (take == POISON) {
break;
}
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
private void process(Integer take) throws InterruptedException {
System.out.println("[Consumer] Take : " + take);
Thread.sleep(500);
}
public ConsumerPoison(BlockingQueue queue, Integer POISON) {
this.queue = queue;
this.POISON = POISON;
}
}
2.3 Start 2 producers and 2 consumers.
Main.java
package com.example.concurrency.queue.simple;
import com.example.concurrency.queue.simple.poison.ConsumerPoison;
import com.example.concurrency.queue.simple.poison.ProducerPoison;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
public class Main {
public static void main(String[] args) {
BlockingQueue queue = new LinkedBlockingQueue<>(10);
//new Thread(new Producer(queue)).start();
//new Thread(new Consumer(queue)).start();
Integer poison = -1;
new Thread(new ProducerPoison(queue, poison)).start();
new Thread(new ProducerPoison(queue, poison)).start();
new Thread(new ConsumerPoison(queue, poison)).start();
new Thread(new ConsumerPoison(queue, poison)).start();
}
}
出力
[Producer] Put : 0 [Producer] Put : 0 [//... [Consumer] Take : 18 [Consumer] Take : 18 [Consumer] Take : 19 [Consumer] Take : 19 [Consumer] Take : -1 [Consumer] Take : -1 Process finished with exit code 0
3. BlockingQueue +ファイルのインデックス作成
単純なファイルインデックスエンジンを作成するためのBlockingQueueの例。 プロデューサーはディレクトリをクロールし、ファイル名をキューに入れます。同時に、コンシューマーは同じキューからファイル名を取得してインデックスを作成します。
3.1 Producer.
FileCrawlerProducer.java
package com.example.concurrency.queue.crawler;
import java.io.File;
import java.io.FileFilter;
import java.util.concurrent.BlockingQueue;
// Producer
// Crawl file system and put the filename in BlockingQueue.
public class FileCrawlerProducer implements Runnable {
private final BlockingQueue fileQueue;
private final FileFilter fileFilter;
private final File file;
private final File POISON;
private final int N_POISON_PILL_PER_PRODUCER;
@Override
public void run() {
try {
crawl(file);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
} finally {
while (true) {
try {
System.out.println(Thread.currentThread().getName()
+ " - FileCrawlerProducer is done, try poison all the consumers!");
// poison all threads
for (int i = 0; i < N_POISON_PILL_PER_PRODUCER; i++) {
System.out.println(Thread.currentThread().getName() + " - puts poison pill!");
fileQueue.put(POISON);
}
break;
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
public FileCrawlerProducer(BlockingQueue fileQueue,
FileFilter fileFilter, File file,
File POISON, int n_POISON_PILL_PER_PRODUCER) {
this.fileQueue = fileQueue;
this.fileFilter = fileFilter;
this.file = file;
this.POISON = POISON;
N_POISON_PILL_PER_PRODUCER = n_POISON_PILL_PER_PRODUCER;
}
private void crawl(File root) throws InterruptedException {
File[] entries = root.listFiles(fileFilter);
if (entries != null) {
for (File entry : entries) {
if (entry.isDirectory()) {
crawl(entry);
} else if (!isIndexed(entry)) {
System.out.println("[FileCrawlerProducer] - Found..."
+ entry.getAbsoluteFile());
fileQueue.put(entry);
}
}
}
}
private boolean isIndexed(File f) {
return false;
}
}
3.2 Consumer.
IndexerConsumer.java
package com.example.concurrency.queue.crawler;
import java.io.File;
import java.util.concurrent.BlockingQueue;
// Consumer
public class IndexerConsumer implements Runnable {
private final BlockingQueue fileQueue;
private final File POISON;
@Override
public void run() {
try {
while (true) {
File take = fileQueue.take();
if (take == POISON) {
System.out.println(Thread.currentThread().getName() + " die");
break;
}
indexFile(take);
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
public void indexFile(File file) {
if (file.isFile()) {
System.out.println(Thread.currentThread().getName()
+ " [IndexerConsumer] - Indexing..." + file.getAbsoluteFile());
}
}
public IndexerConsumer(BlockingQueue fileQueue, File POISON) {
this.fileQueue = fileQueue;
this.POISON = POISON;
}
}
3.3 Start 1 producer and 2 consumers.
Main.java
package com.example.concurrency.queue.crawler;
import java.io.File;
import java.io.FileFilter;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
public class Main {
private static final File POISON = new File("This is a POISON PILL");
public static void main(String[] args) {
int N_PRODUCERS = 1;
int N_CONSUMERS = 2;//Runtime.getRuntime().availableProcessors();
int N_POISON_PILL_PER_PRODUCER = N_CONSUMERS / N_PRODUCERS;
int N_POISON_PILL_REMAIN = N_CONSUMERS % N_PRODUCERS;
System.out.println("N_PRODUCERS : " + N_PRODUCERS);
System.out.println("N_CONSUMERS : " + N_CONSUMERS);
System.out.println("N_POISON_PILL_PER_PRODUCER : " + N_POISON_PILL_PER_PRODUCER);
System.out.println("N_POISON_PILL_REMAIN : " + N_POISON_PILL_REMAIN);
//unbound queue, no limit
BlockingQueue queue = new LinkedBlockingQueue<>();
FileFilter filter = new FileFilter() {
public boolean accept(File file) {
return true;
}
};
File root = new File("C:\\users");
for (int i = 0; i < N_PRODUCERS - 1; i++) {
new Thread(new FileCrawlerProducer(queue, filter, root,
POISON, N_POISON_PILL_PER_PRODUCER)).start();
}
new Thread(new FileCrawlerProducer(queue, filter, root, POISON,
N_POISON_PILL_PER_PRODUCER + N_POISON_PILL_REMAIN)).start();
for (int i = 0; i < N_CONSUMERS; i++) {
new Thread(new IndexerConsumer(queue, POISON)).start();
}
}
}
出力
//... [FileCrawlerProducer] - Found...C:\users\Public\Videos\desktop.ini Thread-2 [IndexerConsumer] - Indexing...C:\users\Public\Videos\desktop.ini Thread-0 - FileCrawlerProducer is done, try poison all the consumers! Thread-0 - puts poison pill! Thread-0 - puts poison pill! Thread-1 die Thread-2 die Process finished with exit code 0