consultingoreo.blogg.se

08:15:46,055 DEBUG .: Removing:hdfs://hbase/.archive/table/4e48ffc1ec089082c66e6d1b5f018fb5/M/729e8bc1430540cb9b2c147c90039cdc from archiveĪnd my solution is very simple: When hfiles and hlogs are archived, we set the modify time of files after rename.

08:15:46,055 DEBUG .: Life:40033567, ttl:300000, current:1367972146054, from: 1367932112487 apache / hbase Public master hbase/bin/hbase-cleanup.sh Go to file Cannot retrieve contributors at this time executable file 155 lines (139 sloc) 4.73 KB Raw Blame /usr/bin/env bash / Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. of hfile retention which relies on zookeeper communicating the // change back to the hfile cleaner. 08:15:46,055 DEBUG .ProtobufRpcEngine: Call: getFileInfo took 1ms hbase/.archive Contains data that would otherwise. So the hfile may be deleted immediatly by HFileCleaner after it is moved to archives. The rename op will not change the modify time of the hfile. But, the modify time of the hdfs file is time when its writer is closed. In this case, when we run compaction, the regions will go offline and data load fails. But, flume agents loads data to staging tables contineously. Parameters: file - full Path of the file to be checked.

If it is valid, then the file will go through the cleaner delegates, but otherwise the file is just deleted. And timeToLiveHFileCleaner uses the modify time of the hfile to determine if it should be deleted. Hi Harsh, The TTL option works well on most of the tables/cases. Validate the file to see if it even belongs in the directory. When a HFileLink is created a back reference file is created in: /hbase/archive. HFiles can be deleted only if there're no links to them. This proposal is another improvement working together with HBASE-18083 to resolve our online issue (archive dir consumed more than 1. HFileLink cleaner that determines if a hfile should be deleted. And when data abnormally accumulated to some huge volume in archive directory, the cleaning speed might not be enough. TimeToLiveHFileCleaner is configed to '' in hbase-default.xml. InterfaceAudience.LimitedPrivate (value'Configuration') public class HFileLinkCleaner extends BaseHFileCleanerDelegate.