org.apache.iotdb.tsfile.read

Class TsFileSequenceReader

java.lang.Object

org.apache.iotdb.tsfile.read.TsFileSequenceReader

All Implemented Interfaces:

AutoCloseable

Direct Known Subclasses:

TsFileRestorableReader, TsFileSequenceReaderForV2, UnClosedTsFileReader

public class TsFileSequenceReader
extends Object
implements AutoCloseable

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TsFileSequenceReader.LocateStatus The location of a chunkGroupMetaData with respect to a space partition constraint.

Field Summary

Fields

Modifier and Type	Field and Description
protected static TSFileConfig	config
protected String	file
protected long	fileMetadataPos
protected int	fileMetadataSize
protected TsFileInput	tsFileInput
protected TsFileMetadata	tsFileMetaData

Constructor Summary

Constructors

Constructor and Description
TsFileSequenceReader(String file) Create a file reader of the given file.
TsFileSequenceReader(String file, boolean loadMetadataSize) construct function for TsFileSequenceReader.
TsFileSequenceReader(String file, boolean loadMetadata, boolean cacheDeviceMetadata)
TsFileSequenceReader(TsFileInput input) Create a file reader of the given file.
TsFileSequenceReader(TsFileInput input, boolean loadMetadataSize) construct function for TsFileSequenceReader.
TsFileSequenceReader(TsFileInput input, long fileMetadataPos, int fileMetadataSize) construct function for TsFileSequenceReader.

Method Summary

Modifier and Type	Method and Description
protected int	binarySearchInTimeseriesMetadataList(List<TimeseriesMetadata> timeseriesMetadataList, String key)
long	checkChunkAndPagesStatistics(IChunkMetadata chunkMetadata)
void	close()
long	fileSize()
List<AlignedChunkMetadata>	getAlignedChunkMetadata(String device) Get AlignedChunkMetadata of sensors under one device
List<String>	getAllDevices()
TsFileDeviceIterator	getAllDevicesIteratorWithIsAligned()
Map<String,TSDataType>	getAllMeasurements() get all measurements in this file
List<Path>	getAllPaths() this function return all timeseries names
Map<String,List<TimeseriesMetadata>>	getAllTimeseriesMetadata(boolean needChunkMetadata)
List<ChunkMetadata>	getChunkMetadataList(Path path)
List<ChunkMetadata>	getChunkMetadataList(Path path, boolean ignoreNotExists) get ChunkMetaDatas of given path, and throw exception if path not exists
Map<String,List<String>>	getDeviceMeasurementsMap()
List<String>	getDeviceNameInRange(long start, long end) get device names which has valid chunks in [start, end)
long	getFileMetadataPos()
long	getFileMetadataSize() Return the whole meta data size of this tsfile, including ChunkMetadata, TimeseriesMetadata and etc.
String	getFileName()
List<IChunkMetadata>	getIChunkMetadataList(Path path)
long	getMaxPlanIndex()
Iterator<Map<String,List<ChunkMetadata>>>	getMeasurementChunkMetadataListMapIterator(String device)
protected Pair<MetadataIndexEntry,Long>	getMetadataAndEndOffset(MetadataIndexNode metadataIndex, String name, boolean isDeviceLevel, boolean exactSearch) Get target MetadataIndexEntry and its end offset
MetadataIndexNode	getMetadataIndexNode(long startOffset, long endOffset) get metadata index node
long	getMinPlanIndex()
Iterator<List<Path>>	getPathsIterator()
int	getTsFileMetadataSize()
int	hashCode()
boolean	isComplete() whether the file is a complete TsFile: only if the head magic and tail magic string exists.
void	loadMetadataSize()
long	position()
void	position(long offset)
BloomFilter	readBloomFilter() this function does not modify the position of the file reader.
ChunkGroupHeader	readChunkGroupHeader() read data from current position of the input, and deserialize it to a CHUNK_GROUP_FOOTER.
ChunkGroupHeader	readChunkGroupHeader(long position, boolean markerRead) read data from current position of the input, and deserialize it to a CHUNK_GROUP_FOOTER.
ChunkHeader	readChunkHeader(byte chunkType) read data from current position of the input, and deserialize it to a CHUNK_HEADER.
Map<String,List<ChunkMetadata>>	readChunkMetadataInDevice(String device) read all ChunkMetaDatas of given device
List<ChunkMetadata>	readChunkMetaDataList(TimeseriesMetadata timeseriesMetaData) get ChunkMetaDatas in given TimeseriesMetaData
ByteBuffer	readCompressedPage(PageHeader header)
protected ByteBuffer	readData(long position, int totalSize) read data from tsFileInput, from the current position (if position = -1), or the given position.
protected ByteBuffer	readData(long start, long end) read data from tsFileInput, from the current position (if position = -1), or the given position.
Map<String,TimeseriesMetadata>	readDeviceMetadata(String device) this function reads measurements and TimeseriesMetaDatas in given device Thread Safe
TsFileMetadata	readFileMetadata() this function does not modify the position of the file reader.
String	readHeadMagic() this function does not modify the position of the file reader.
List<IChunkMetadata>	readIChunkMetaDataList(ITimeSeriesMetadata timeseriesMetaData)
ITimeSeriesMetadata	readITimeseriesMetadata(Path path, boolean ignoreNotExists)
List<ITimeSeriesMetadata>	readITimeseriesMetadata(String device, Set<String> measurements)
byte	readMarker() read one byte from the input.
Chunk	readMemChunk(CachedChunkLoaderImpl.ChunkCacheKey chunkCacheKey) read memory chunk.
Chunk	readMemChunk(ChunkMetadata metaData) read memory chunk.
ByteBuffer	readPage(PageHeader header, CompressionType type)
PageHeader	readPageHeader(TSDataType type, boolean hasStatistic) not thread safe.
void	readPlanIndex()
int	readRaw(long position, int length, ByteBuffer target) notice, the target bytebuffer are not flipped.
String	readTailMagic() this function does not modify the position of the file reader.
TimeseriesMetadata	readTimeseriesMetadata(Path path, boolean ignoreNotExists)
List<TimeseriesMetadata>	readTimeseriesMetadata(Path path, Set<String> allSensors)
byte	readVersionNumber() this function reads version number and checks compatibility of TsFile.
long	selfCheck(Map<Path,IMeasurementSchema> newSchema, List<ChunkGroupMetadata> chunkGroupMetadataList, boolean fastFinish) Self Check the file and return the position before where the data is safe.
long	selfCheckWithInfo(String filename, boolean fastFinish, Map<Long,Pair<Path,TimeseriesMetadata>> timeseriesMetadataMap) Self Check the file and return whether the file is safe.
void	skipPageData(PageHeader header)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Field Detail

config

protected static final TSFileConfig config

file

protected String file

tsFileInput

protected TsFileInput tsFileInput

fileMetadataPos

protected long fileMetadataPos

fileMetadataSize

protected int fileMetadataSize

tsFileMetaData

protected TsFileMetadata tsFileMetaData

Constructor Detail

TsFileSequenceReader

public TsFileSequenceReader(String file)
                     throws IOException

Create a file reader of the given file. The reader will read the tail of the file to get the file metadata size.Then the reader will skip the first TSFileConfig.MAGIC_STRING.getBytes().length + TSFileConfig.NUMBER_VERSION.getBytes().length bytes of the file for preparing reading real data.

Parameters:

file - the data file

Throws:

IOException - If some I/O error occurs

TsFileSequenceReader

public TsFileSequenceReader(String file,
                            boolean loadMetadataSize)
                     throws IOException

construct function for TsFileSequenceReader.

Parameters:

file - -given file name

loadMetadataSize - -whether load meta data size

Throws:

IOException

TsFileSequenceReader

public TsFileSequenceReader(String file,
                            boolean loadMetadata,
                            boolean cacheDeviceMetadata)
                     throws IOException

Throws:

IOException

TsFileSequenceReader

public TsFileSequenceReader(TsFileInput input)
                     throws IOException

Create a file reader of the given file. The reader will read the tail of the file to get the file metadata size.Then the reader will skip the first TSFileConfig.MAGIC_STRING.getBytes().length + TSFileConfig.NUMBER_VERSION.getBytes().length bytes of the file for preparing reading real data.

Parameters:

input - given input

Throws:

IOException

TsFileSequenceReader

public TsFileSequenceReader(TsFileInput input,
                            boolean loadMetadataSize)
                     throws IOException

construct function for TsFileSequenceReader.

Parameters:

input - -given input

loadMetadataSize - -load meta data size

Throws:

IOException

TsFileSequenceReader

public TsFileSequenceReader(TsFileInput input,
                            long fileMetadataPos,
                            int fileMetadataSize)

construct function for TsFileSequenceReader.

Parameters:

input - the input of a tsfile. The current position should be a marker and then a chunk Header, rather than the magic number

fileMetadataPos - the position of the file metadata in the TsFileInput from the beginning of the input to the current position

fileMetadataSize - the byte size of the file metadata in the input

Method Detail

loadMetadataSize

public void loadMetadataSize()
                      throws IOException

Throws:

IOException

getFileMetadataPos

public long getFileMetadataPos()

getTsFileMetadataSize

public int getTsFileMetadataSize()

getFileMetadataSize

public long getFileMetadataSize()
                         throws IOException

Return the whole meta data size of this tsfile, including ChunkMetadata, TimeseriesMetadata and etc.

Throws:

IOException

readTailMagic

public String readTailMagic()
                     throws IOException

this function does not modify the position of the file reader.

Throws:

IOException

isComplete

public boolean isComplete()
                   throws IOException

whether the file is a complete TsFile: only if the head magic and tail magic string exists.

Throws:

IOException

readHeadMagic

public String readHeadMagic()
                     throws IOException

this function does not modify the position of the file reader.

Throws:

IOException

readVersionNumber

public byte readVersionNumber()
                       throws IOException

this function reads version number and checks compatibility of TsFile.

Throws:

IOException

readFileMetadata

public TsFileMetadata readFileMetadata()
                                throws IOException

this function does not modify the position of the file reader.

Throws:

IOException - io error

readBloomFilter

public BloomFilter readBloomFilter()
                            throws IOException

this function does not modify the position of the file reader.

Throws:

IOException - io error

readDeviceMetadata

public Map<String,TimeseriesMetadata> readDeviceMetadata(String device)
                                                  throws IOException

this function reads measurements and TimeseriesMetaDatas in given device Thread Safe

Parameters:

device - name

Returns:

the map measurementId -> TimeseriesMetaData in one device

Throws:

IOException - io error

readTimeseriesMetadata

public TimeseriesMetadata readTimeseriesMetadata(Path path,
                                                 boolean ignoreNotExists)
                                          throws IOException

Throws:

IOException

readITimeseriesMetadata

public ITimeSeriesMetadata readITimeseriesMetadata(Path path,
                                                   boolean ignoreNotExists)
                                            throws IOException

Throws:

IOException

readTimeseriesMetadata

public List<TimeseriesMetadata> readTimeseriesMetadata(Path path,
                                                       Set<String> allSensors)
                                                throws IOException

Throws:

IOException

readITimeseriesMetadata

public List<ITimeSeriesMetadata> readITimeseriesMetadata(String device,
                                                         Set<String> measurements)
                                                  throws IOException

Throws:

IOException

binarySearchInTimeseriesMetadataList

protected int binarySearchInTimeseriesMetadataList(List<TimeseriesMetadata> timeseriesMetadataList,
                                                   String key)

getAllDevices

public List<String> getAllDevices()
                           throws IOException

Throws:

IOException

getAllDevicesIteratorWithIsAligned

public TsFileDeviceIterator getAllDevicesIteratorWithIsAligned()
                                                        throws IOException

Returns:

an iterator of "device, isAligned" list, in which names of devices are ordered in dictionary order, and isAligned represents whether the device is aligned

Throws:

IOException

readChunkMetadataInDevice

public Map<String,List<ChunkMetadata>> readChunkMetadataInDevice(String device)
                                                          throws IOException

read all ChunkMetaDatas of given device

Parameters:

device - name

Returns:

measurement -> ChunkMetadata list

Throws:

IOException - io error

getAllPaths

public List<Path> getAllPaths()
                       throws IOException

this function return all timeseries names

Returns:

list of Paths

Throws:

IOException - io error

getPathsIterator

public Iterator<List<Path>> getPathsIterator()
                                      throws IOException

Returns:

an iterator of timeseries list, in which names of timeseries are ordered in dictionary order

Throws:

IOException - io error

getAllTimeseriesMetadata

public Map<String,List<TimeseriesMetadata>> getAllTimeseriesMetadata(boolean needChunkMetadata)
                                                              throws IOException

Throws:

IOException

getMetadataAndEndOffset

protected Pair<MetadataIndexEntry,Long> getMetadataAndEndOffset(MetadataIndexNode metadataIndex,
                                                                String name,
                                                                boolean isDeviceLevel,
                                                                boolean exactSearch)
                                                         throws IOException

Get target MetadataIndexEntry and its end offset

Parameters:

metadataIndex - given MetadataIndexNode

name - target device / measurement name

isDeviceLevel - whether target MetadataIndexNode is device level

exactSearch - whether is in exact search mode, return null when there is no entry with name; or else return the nearest MetadataIndexEntry before it (for deeper search)

Returns:

target MetadataIndexEntry, endOffset pair

Throws:

IOException

readChunkGroupHeader

public ChunkGroupHeader readChunkGroupHeader()
                                      throws IOException

read data from current position of the input, and deserialize it to a CHUNK_GROUP_FOOTER.
This method is not threadsafe.

Returns:

a CHUNK_GROUP_FOOTER

Throws:

IOException - io error

readChunkGroupHeader

public ChunkGroupHeader readChunkGroupHeader(long position,
                                             boolean markerRead)
                                      throws IOException

read data from current position of the input, and deserialize it to a CHUNK_GROUP_FOOTER.

Parameters:

position - the offset of the chunk group footer in the file

markerRead - true if the offset does not contains the marker , otherwise false

Returns:

a CHUNK_GROUP_FOOTER

Throws:

IOException - io error

readPlanIndex

public void readPlanIndex()
                   throws IOException

Throws:

IOException

readChunkHeader

public ChunkHeader readChunkHeader(byte chunkType)
                            throws IOException

read data from current position of the input, and deserialize it to a CHUNK_HEADER.
This method is not threadsafe.

Returns:

a CHUNK_HEADER

Throws:

IOException - io error

readMemChunk

public Chunk readMemChunk(ChunkMetadata metaData)
                   throws IOException

read memory chunk.

Parameters:

metaData - -given chunk meta data

Returns:

-chunk

Throws:

IOException

readMemChunk

public Chunk readMemChunk(CachedChunkLoaderImpl.ChunkCacheKey chunkCacheKey)
                   throws IOException

read memory chunk.

Parameters:

chunkCacheKey - given key of chunk LRUCache

Returns:

chunk

Throws:

IOException

readPageHeader

public PageHeader readPageHeader(TSDataType type,
                                 boolean hasStatistic)
                          throws IOException

not thread safe.

Parameters:

type - given tsfile data type

Throws:

IOException

position

public long position()
              throws IOException

Throws:

IOException

position

public void position(long offset)
              throws IOException

Throws:

IOException

skipPageData

public void skipPageData(PageHeader header)
                  throws IOException

Throws:

IOException

readCompressedPage

public ByteBuffer readCompressedPage(PageHeader header)
                              throws IOException

Throws:

IOException

readPage

public ByteBuffer readPage(PageHeader header,
                           CompressionType type)
                    throws IOException

Throws:

IOException

readMarker

public byte readMarker()
                throws IOException

read one byte from the input.
this method is not thread safe

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Throws:

IOException

getFileName

public String getFileName()

fileSize

public long fileSize()
              throws IOException

Throws:

IOException

readData

protected ByteBuffer readData(long position,
                              int totalSize)
                       throws IOException

read data from tsFileInput, from the current position (if position = -1), or the given position.
if position = -1, the tsFileInput's position will be changed to the current position + real data size that been read. Other wise, the tsFileInput's position is not changed.

Parameters:

position - the start position of data in the tsFileInput, or the current position if position = -1

totalSize - the size of data that want to read

Returns:

data that been read.

Throws:

IOException

readData

protected ByteBuffer readData(long start,
                              long end)
                       throws IOException

read data from tsFileInput, from the current position (if position = -1), or the given position.

Parameters:

start - the start position of data in the tsFileInput, or the current position if position = -1

end - the end position of data that want to read

Returns:

data that been read.

Throws:

IOException

readRaw

public int readRaw(long position,
                   int length,
                   ByteBuffer target)
            throws IOException

notice, the target bytebuffer are not flipped.

Throws:

IOException

selfCheck

public long selfCheck(Map<Path,IMeasurementSchema> newSchema,
                      List<ChunkGroupMetadata> chunkGroupMetadataList,
                      boolean fastFinish)
               throws IOException

Self Check the file and return the position before where the data is safe.

Parameters:

newSchema - the schema on each time series in the file

chunkGroupMetadataList - ChunkGroupMetadata List

fastFinish - if true and the file is complete, then newSchema and chunkGroupMetadataList parameter will be not modified.

Returns:

the position of the file that is fine. All data after the position in the file should be truncated.

Throws:

IOException

selfCheckWithInfo

public long selfCheckWithInfo(String filename,
                              boolean fastFinish,
                              Map<Long,Pair<Path,TimeseriesMetadata>> timeseriesMetadataMap)
                       throws IOException,
                              TsFileStatisticsMistakesException

Self Check the file and return whether the file is safe.

Parameters:

filename - the path of file

fastFinish - if true, the method will only check the format of head (Magic String TsFile, Version Number) and tail (Magic String TsFile) of TsFile.

Returns:

the status of TsFile

Throws:

IOException

TsFileStatisticsMistakesException

checkChunkAndPagesStatistics

public long checkChunkAndPagesStatistics(IChunkMetadata chunkMetadata)
                                  throws IOException

Throws:

IOException

getChunkMetadataList

public List<ChunkMetadata> getChunkMetadataList(Path path,
                                                boolean ignoreNotExists)
                                         throws IOException

get ChunkMetaDatas of given path, and throw exception if path not exists

Parameters:

path - timeseries path

Returns:

List of ChunkMetaData

Throws:

IOException

getIChunkMetadataList

public List<IChunkMetadata> getIChunkMetadataList(Path path)
                                           throws IOException

Throws:

IOException

getChunkMetadataList

public List<ChunkMetadata> getChunkMetadataList(Path path)
                                         throws IOException

Throws:

IOException

getAlignedChunkMetadata

public List<AlignedChunkMetadata> getAlignedChunkMetadata(String device)
                                                   throws IOException

Get AlignedChunkMetadata of sensors under one device

Parameters:

device - device name

Throws:

IOException

readChunkMetaDataList

public List<ChunkMetadata> readChunkMetaDataList(TimeseriesMetadata timeseriesMetaData)
                                          throws IOException

get ChunkMetaDatas in given TimeseriesMetaData

Returns:

List of ChunkMetaData

Throws:

IOException

readIChunkMetaDataList

public List<IChunkMetadata> readIChunkMetaDataList(ITimeSeriesMetadata timeseriesMetaData)

getAllMeasurements

public Map<String,TSDataType> getAllMeasurements()
                                          throws IOException

get all measurements in this file

Returns:

measurement -> datatype

Throws:

IOException

getDeviceMeasurementsMap

public Map<String,List<String>> getDeviceMeasurementsMap()
                                                  throws IOException

Throws:

IOException

getDeviceNameInRange

public List<String> getDeviceNameInRange(long start,
                                         long end)
                                  throws IOException

get device names which has valid chunks in [start, end)

Parameters:

start - start of the partition

end - end of the partition

Returns:

device names in range

Throws:

IOException

getMetadataIndexNode

public MetadataIndexNode getMetadataIndexNode(long startOffset,
                                              long endOffset)
                                       throws IOException

get metadata index node

Parameters:

startOffset - start read offset

endOffset - end read offset

Returns:

MetadataIndexNode

Throws:

IOException

getMinPlanIndex

public long getMinPlanIndex()

getMaxPlanIndex

public long getMaxPlanIndex()

getMeasurementChunkMetadataListMapIterator

public Iterator<Map<String,List<ChunkMetadata>>> getMeasurementChunkMetadataListMapIterator(String device)
                                                                                     throws IOException

Returns:

An iterator of linked hashmaps ( measurement -> chunk metadata list ). When traversing the linked hashmap, you will get chunk metadata lists according to the lexicographic order of the measurements. The first measurement of the linked hashmap of each iteration is always larger than the last measurement of the linked hashmap of the previous iteration in lexicographic order.

Throws:

IOException

hashCode

public int hashCode()

Overrides:

hashCode in class Object