org.apache.iotdb.tsfile.encoding.encoder

Class RleEncoder<T extends Comparable<T>>

java.lang.Object

org.apache.iotdb.tsfile.encoding.encoder.Encoder

org.apache.iotdb.tsfile.encoding.encoder.RleEncoder<T>

Type Parameters:

T - data type T for RLE

Direct Known Subclasses:

IntRleEncoder, LongRleEncoder

public abstract class RleEncoder<T extends Comparable<T>>
extends Encoder

Encodes values using a combination of run length encoding and bit packing, according to the following grammar:

 rle-bit-packing-hybrid: <length> <bitwidth> <encoded-data>
 length := length of the <bitwidth> <encoded-data> in bytes stored as 4 bytes little endian
 bitwidth := bitwidth for all encoded data in <encoded-data>
 encoded-data := <run>*
 run := <bit-packed-run> | <rle-run>
 bit-packed-run := <bit-packed-header> <lastBitPackedNum> <bit-packed-values>
 bit-packed-header := varint-encode(<bit-pack-count> << 1 | 1)
 lastBitPackedNum := the number of useful value in last bit-pack may be less than 8, so
 lastBitPackedNum indicates how many values are useful
 bit-packed-values :=  bit packed
 rle-run := <rle-header> <repeated-value>
 rle-header := varint-encode( (number of times repeated) << 1)
 repeated-value := value that is repeated, using a fixed-width of round-up-to-next-byte(bit-width)
 

.

Field Summary

Fields

Modifier and Type	Field and Description
protected int	bitPackedGroupCount the number of group which using bit packing, it is saved in header.
protected int	bitWidth the bit width used for bit-packing and rle.
protected T[]	bufferedValues array to buffer values temporarily.
protected ByteArrayOutputStream	byteCache output stream to buffer <bitwidth> <encoded-data>.
protected List<byte[]>	bytesBuffer we will write all bytes using bit-packing to OutputStream once.
protected TSFileConfig	config
protected boolean	isBitPackRun flag which indicate encoding mode false -- rle true -- bit-packing.
protected boolean	isBitWidthSaved
protected int	numBufferedValues the number of buffered value in array.
protected T	preValue previous value written, used to detect repeated values.
protected int	repeatCount for a given value now buffered, how many times it occurs.
protected List<T>	values we save all value in a list and calculate its bitwidth.

Fields inherited from class org.apache.iotdb.tsfile.encoding.encoder.Encoder

MAX_POINT_NUMBER, MAX_STRING_LENGTH

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	RleEncoder() constructor.

Method Summary

Modifier and Type	Method and Description
protected abstract void	clearBuffer() clean all useless value in bufferedValues and set 0.
protected abstract void	convertBuffer()
void	encode(BigDecimal value, ByteArrayOutputStream out)
void	encode(Binary value, ByteArrayOutputStream out)
void	encode(boolean value, ByteArrayOutputStream out)
void	encode(double value, ByteArrayOutputStream out)
void	encode(float value, ByteArrayOutputStream out)
void	encode(int value, ByteArrayOutputStream out)
void	encode(long value, ByteArrayOutputStream out)
void	encode(short value, ByteArrayOutputStream out)
protected void	encodeValue(T value) Encode T value using rle or bit-packing.
protected void	endPreviousBitPackedRun(int lastBitPackedNum) End a bit-packing run write all bit-packing group to OutputStream bit-packing format: [header][lastBitPackedNum][bit-packing group]+ [bit-packing group]+ are saved in List<byte[]> bytesBuffer .
void	flush(ByteArrayOutputStream out) Write all values buffered in cache to OutputStream.
protected void	reset()
void	writeOrAppendBitPackedRun() Start a bit-packing run transform values to bytes and buffer them in cache.
protected abstract void	writeRleRun() Write bytes to OutputStream using rle.

Methods inherited from class org.apache.iotdb.tsfile.encoding.encoder.Encoder

getMaxByteSize, getOneItemMaxSize, getType, setType

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

values

protected List<T extends Comparable<T>> values

we save all value in a list and calculate its bitwidth.

bitWidth

protected int bitWidth

the bit width used for bit-packing and rle.

repeatCount

protected int repeatCount

for a given value now buffered, how many times it occurs.

bitPackedGroupCount

protected int bitPackedGroupCount

the number of group which using bit packing, it is saved in header.

numBufferedValues

protected int numBufferedValues

the number of buffered value in array.

bytesBuffer

protected List<byte[]> bytesBuffer

we will write all bytes using bit-packing to OutputStream once. Before that, all bytes are saved in list.

isBitPackRun

protected boolean isBitPackRun

flag which indicate encoding mode false -- rle true -- bit-packing.

preValue

protected T extends Comparable<T> preValue

previous value written, used to detect repeated values.

bufferedValues

protected T extends Comparable<T>[] bufferedValues

array to buffer values temporarily.

isBitWidthSaved

protected boolean isBitWidthSaved

byteCache

protected ByteArrayOutputStream byteCache

output stream to buffer <bitwidth> <encoded-data>.

config

protected TSFileConfig config

Constructor Detail

RleEncoder

protected RleEncoder()

constructor.

Method Detail

reset

protected void reset()

flush

public void flush(ByteArrayOutputStream out)
           throws IOException

Write all values buffered in cache to OutputStream.

Specified by:

flush in class Encoder

Parameters:

out - - byteArrayOutputStream

Throws:

IOException - cannot flush to OutputStream

writeRleRun

protected abstract void writeRleRun()
                             throws IOException

Write bytes to OutputStream using rle. rle format: [header][value] header: (repeated value) << 1

Throws:

IOException - cannot write RLE run

writeOrAppendBitPackedRun

public void writeOrAppendBitPackedRun()

Start a bit-packing run transform values to bytes and buffer them in cache.

endPreviousBitPackedRun

protected void endPreviousBitPackedRun(int lastBitPackedNum)

End a bit-packing run write all bit-packing group to OutputStream bit-packing format: [header][lastBitPackedNum][bit-packing group]+ [bit-packing group]+ are saved in List<byte[]> bytesBuffer .

Parameters:

lastBitPackedNum - - in last bit-packing group, it may have useful values less than 8. This param indicates how many values are useful

encodeValue

protected void encodeValue(T value)

Encode T value using rle or bit-packing. It may not write to OutputStream immediately

Parameters:

value - - value to encode

clearBuffer

protected abstract void clearBuffer()

clean all useless value in bufferedValues and set 0.

convertBuffer

protected abstract void convertBuffer()

encode

public void encode(boolean value,
                   ByteArrayOutputStream out)

Overrides:

encode in class Encoder

encode

public void encode(short value,
                   ByteArrayOutputStream out)

Overrides:

encode in class Encoder

encode

public void encode(int value,
                   ByteArrayOutputStream out)

Overrides:

encode in class Encoder

encode

public void encode(long value,
                   ByteArrayOutputStream out)

Overrides:

encode in class Encoder

encode

public void encode(float value,
                   ByteArrayOutputStream out)

Overrides:

encode in class Encoder

encode

public void encode(double value,
                   ByteArrayOutputStream out)

Overrides:

encode in class Encoder

encode

public void encode(Binary value,
                   ByteArrayOutputStream out)

Overrides:

encode in class Encoder

encode

public void encode(BigDecimal value,
                   ByteArrayOutputStream out)

Overrides:

encode in class Encoder