edu.iu.dsc.tws.data.memory

Class BufferedMemoryManager

java.lang.Object

edu.iu.dsc.tws.data.memory.AbstractMemoryManager

edu.iu.dsc.tws.data.memory.BufferedMemoryManager

All Implemented Interfaces:

MemoryManager

public class BufferedMemoryManager
extends AbstractMemoryManager

Inserts into the memory store in batches. Only one instance per executor.

Field Summary

Fields inherited from class edu.iu.dsc.tws.data.memory.AbstractMemoryManager

operationMap

Constructor Summary

Constructors

Constructor and Description
BufferedMemoryManager(Path dataPath)

Method Summary

Modifier and Type	Method and Description
OperationMemoryManager	addOperation(int opID, DataMessageType type) Add an memory manager instance for the given operation
OperationMemoryManager	addOperation(int opID, DataMessageType type, DataMessageType keyType)
boolean	append(int opID, java.nio.ByteBuffer key, java.nio.ByteBuffer value) Appends the given byte buffer to the current value in the store.
boolean	append(int opID, java.lang.String key, java.nio.ByteBuffer value)
boolean	appendBulk(int opID, java.lang.String key, java.nio.ByteBuffer value) Buffers the inputs before submitting to the store.
boolean	close(int opID, java.nio.ByteBuffer key) flush and close the given key.
boolean	close(int opID, java.lang.String key) Closing the key will make the BufferedMemoryManager to flush the current data into the store and delete all the key information.
boolean	containsKey(int opID, java.nio.ByteBuffer key) checks if the given key is in the memory manager
boolean	containsKey(int opID, java.lang.String key) checks if the given key is in the memory manager
boolean	delete(int opID, java.nio.ByteBuffer key) delete the given key from the store
boolean	delete(int opID, java.lang.String key) delete the given key from the store
void	deleteFromBMM(int opID, java.lang.String key)
boolean	flush(int opID)
boolean	flush(int opID, java.nio.ByteBuffer key) Flush all the data into the store.
boolean	flush(int opID, java.lang.String key) Makes sure all the data that is held in the BufferedMemoryManager is pushed into the memory store
boolean	flush(int opID, java.lang.String key, java.nio.ByteBuffer last) Slight variation of flush for so that the last ByteBuffer does not need to be copied into the map
boolean	flushAll(int opID) Flush all the keys that are stored for the given operation id.
java.nio.ByteBuffer	get(int opID, java.nio.ByteBuffer key) Get the corresponding value as ByteBuffer for the given key from the store
java.nio.ByteBuffer	get(int opID, java.lang.String key) Get the corresponding value as ByteBuffer for the given key from the store
java.util.Iterator<java.lang.Object>	getIterator(int opID, DataMessageType keyType, DataMessageType valueType, KryoSerializer deSerializer, java.nio.ByteOrder order)
java.util.Iterator<java.lang.Object>	getIterator(int opID, DataMessageType valueType, KryoSerializer deSerializer, java.nio.ByteOrder order)
boolean	init() Initializes the Memory manager
boolean	put(int opID, byte[] key, byte[] data)
boolean	put(int opID, java.nio.ByteBuffer key, java.nio.ByteBuffer value) Stores the give key value pair in the memory manager
boolean	put(int opID, java.lang.String key, java.nio.ByteBuffer value) Stores the give key value pair in the memory manager
boolean	putBulk(int opID, java.lang.String key, java.nio.ByteBuffer value) Buffers the inputs before submitting it to the store.
boolean	registerKey(int opID, java.lang.String key)
boolean	registerKey(int opID, java.lang.String key, int step) Register the key
boolean	removeOperation(int opID) Remove an operation from the memory manager.

Methods inherited from class edu.iu.dsc.tws.data.memory.AbstractMemoryManager

getDeserialized, getDeserialized, getDeserialized

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BufferedMemoryManager

public BufferedMemoryManager(Path dataPath)

Method Detail

init

public boolean init()

Description copied from interface: MemoryManager

Initializes the Memory manager

append

public boolean append(int opID,
                      java.nio.ByteBuffer key,
                      java.nio.ByteBuffer value)

Description copied from interface: MemoryManager

Appends the given byte buffer to the current value in the store. If the store does not contain the given key append works as a put operation

Parameters:

opID - id value of the operation

key - key to append to

value - value to be appended

Returns:

true if append was performed, false otherwise

append

public boolean append(int opID,
                      java.lang.String key,
                      java.nio.ByteBuffer value)

put

public boolean put(int opID,
                   java.nio.ByteBuffer key,
                   java.nio.ByteBuffer value)

Description copied from interface: MemoryManager

Stores the give key value pair in the memory manager

put

public boolean put(int opID,
                   byte[] key,
                   byte[] data)

put

public boolean put(int opID,
                   java.lang.String key,
                   java.nio.ByteBuffer value)

Description copied from interface: MemoryManager

Stores the give key value pair in the memory manager

get

public java.nio.ByteBuffer get(int opID,
                               java.nio.ByteBuffer key)

Description copied from interface: MemoryManager

Get the corresponding value as ByteBuffer for the given key from the store

get

public java.nio.ByteBuffer get(int opID,
                               java.lang.String key)

Description copied from interface: MemoryManager

Get the corresponding value as ByteBuffer for the given key from the store

containsKey

public boolean containsKey(int opID,
                           java.nio.ByteBuffer key)

Description copied from interface: MemoryManager

checks if the given key is in the memory manager

key - key to be checked

Returns:

true if the key is present, false otherwise

containsKey

public boolean containsKey(int opID,
                           java.lang.String key)

Description copied from interface: MemoryManager

checks if the given key is in the memory manager

key - key to be checked

Returns:

true if the key is present, false otherwise

delete

public boolean delete(int opID,
                      java.nio.ByteBuffer key)

Description copied from interface: MemoryManager

delete the given key from the store

delete

public boolean delete(int opID,
                      java.lang.String key)

Description copied from interface: MemoryManager

delete the given key from the store

deleteFromBMM

public void deleteFromBMM(int opID,
                          java.lang.String key)

addOperation

public OperationMemoryManager addOperation(int opID,
                                           DataMessageType type)

Description copied from interface: MemoryManager

Add an memory manager instance for the given operation

Returns:

returns the memory manager or null (Null is returned at the base layer)

addOperation

public OperationMemoryManager addOperation(int opID,
                                           DataMessageType type,
                                           DataMessageType keyType)

removeOperation

public boolean removeOperation(int opID)

Description copied from interface: MemoryManager

Remove an operation from the memory manager. This will result in loss of all the data related to the operation in the memory manager

Parameters:

opID - operation id to be removed

Returns:

true if the operation was removed, false otherwise

registerKey

public boolean registerKey(int opID,
                           java.lang.String key,
                           int step)

Register the key

Parameters:

key - key value to be registered

step - the step size. The Memory manager will write to the store once this value is reached

Returns:

true if the key was registered and false if the key is already present

registerKey

public boolean registerKey(int opID,
                           java.lang.String key)

putBulk

public boolean putBulk(int opID,
                       java.lang.String key,
                       java.nio.ByteBuffer value)

Buffers the inputs before submitting it to the store. If the value is already present it will be replaced

appendBulk

public boolean appendBulk(int opID,
                          java.lang.String key,
                          java.nio.ByteBuffer value)

Buffers the inputs before submitting to the store. The new values will be appended to the end

flush

public boolean flush(int opID,
                     java.nio.ByteBuffer key)

Description copied from interface: MemoryManager

Flush all the data into the store. This method is needed only if buffering is done at the Memory Manager

Parameters:

opID - id of the operation

key - key to be flushed

flush

public boolean flush(int opID,
                     java.lang.String key)

Makes sure all the data that is held in the BufferedMemoryManager is pushed into the memory store

flush

public boolean flush(int opID)

flush

public boolean flush(int opID,
                     java.lang.String key,
                     java.nio.ByteBuffer last)

Slight variation of flush for so that the last ByteBuffer does not need to be copied into the map

Parameters:

key - key to flush

last - the last value that needs to be appended to the ByteBuffers that correspond to the given key

flushAll

public boolean flushAll(int opID)

Flush all the keys that are stored for the given operation id. this must be called before a range of keys will be read as in the iterator methods

close

public boolean close(int opID,
                     java.nio.ByteBuffer key)

Description copied from interface: MemoryManager

flush and close the given key. After close is performed the key data will only be available in the store. This method is needed only if buffering is done at the Memory Manager

Parameters:

opID - id of the operation

key - key to be closed

close

public boolean close(int opID,
                     java.lang.String key)

Closing the key will make the BufferedMemoryManager to flush the current data into the store and delete all the key information. This is done once we know that no more values will be sent for this key

getIterator

public java.util.Iterator<java.lang.Object> getIterator(int opID,
                                                        DataMessageType keyType,
                                                        DataMessageType valueType,
                                                        KryoSerializer deSerializer,
                                                        java.nio.ByteOrder order)

getIterator

public java.util.Iterator<java.lang.Object> getIterator(int opID,
                                                        DataMessageType valueType,
                                                        KryoSerializer deSerializer,
                                                        java.nio.ByteOrder order)