gist JS

Friday, August 16, 2013

hbase scan: batch vs cache

Here's today's contribution to the Internet: tl;dr When it comes to HBase scanner settings, you want caching, not batchsize. Maybe this is totally clear to everyone else. But for those of us who are 'newer to hbase' I can never quite remember what I'm doing.
Say you've got this code:
Scan s = new Scan(startKey);s.setCaching(foo);s.setBatch(bar);ResultScanner scanner = new ResultScanner(s);for (final Result r : scanner) {  //stuff}
But you're clever and you don't want to do RPC calls to HBase for every row. You might even say you'd like to 'batch' the results from your scanner. 

So you read http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Scan.html public void setBatch(int batch)
Set the maximum number of values to return for each call to next()
public void setCaching(int caching)
Set the number of rows for caching that will be passed to scanners. If not set, the default setting from HTable.getScannerCaching() will apply. Higher caching values will enable faster scanners but will use more memory.
Annnd.... not sure. I mean, I only want one Result every time I call next()in my iterator, right? What would  a number >1 even mean? 

And I'm sure I shouldn't set 'caching' that sounds like it will 'cache' something. I want to read the real stuff.

But you do want caching. Caching is how many things come back in a batch from your scanner. 

Ok. Fine. Caching got named poorly.  What is batch?

Batch is in case you have super wide rows. Say you have 250 columns. Batch of 100 would give your iterator:
  • Iteration 1: Result id 0. Columns 0-99
  • Iteration 2: Result id 0. Columns 100-199
  • Iteration 3: Result id 0. Columns 200-249
  • Iteration 4: Result id 1. Columns 0-99
  • Iteration 5: Result id 1. Columns 100-199
Or at least that's what http://twitter.com/monkeyatlarge told me.


Anonymous said...

Does this apply for HBase REST as well?

The caching seems to not work using REST or probably me doing it wrong.

Jeff Dwyer said...

Haven't used the REST stuff myself. I presume you've seen https://issues.apache.org/jira/browse/HBASE-7803 ? Seems as though it is supposed to work.

Dan Häberlein said...

Good stuff!

angela michael said...

commission express reviews I really like and appreciate your blog post.Much thanks again. Want more.

Govt Jobs said...

I am one of those many candidates who completely rely on this website to know about government jobs. Thanks a lot providing exams syllabus along with good material to easily prepare.

Brigade Buena Vista said...

Nice Blog. Useful Information.Thanks for sharing with us.
Brigade Buena Vista
Lodha bellezza Hyderabad
Godrej Avenues Bangalore
Goyal Orchid Whitefield

Mark Smith said...

Rolex Watches Authentic Mens & Ladies Rolex Datejust, President Watches for Sale at JavyEstrella.com.

Amlon Terence said...

Florida Accident Lawyer Sinclair Law - Leading Florida Law Firm Attorney Brad Sinclair Represents Personal Injury, Car Accident, Motorcycle Accident Victims in Melbourne.

Saradha Devi said...

interesting article to read
Selenium Training | Selenium Training Institute in Chennai | Best Selenium Training Institutes in Chennai | Software Testing Training in Chennai

jessie karthik said...

JNTUK 2-1 Results Check

Manasa Garugu said...

Sarkari Recruitment is one of the biggest Indian Job Site so here you will getPPSC Jobs 2017 so

Creative Template said...

nice post.
Free Wordpress Themes

proudstories said...

thanks for sharing information.
quotes by famous people

Srinu said...

Nice blog and thanks for sharing the post. Check out some latest projects.

Goyal Orchid Whitefield
Prestige Lakeside Habitat
Rainbow Vistas Rock Garden
Prestige Ivy League
Lodha Meridian
Lodha Bellezza Sky Villas
Prestige Kew Gardens

Rajapriya R said...

great blog to read.. after reading this blog i learned more useful information from this blog

hadoop training institute in velachery | big data training institute in velachery | hadoop training in chennai velachery | big data training in chennai velachery

Ananthi S said...

this blog really helpful to develop my career in Hadoop.. thank you for sharing

hadoop training institute in adyar | big data training institute in adyar | hadoop training in chennai adyar | big data training in chennai adyar

Sujit kUmar said...

nice post for sharing Selenium article. its really helpful for me. keep sharing tutorials????????????????

Rajesh Kumar K said...

I just came across from this article and I read it about some tutorial on selenium related. It is too good and worth reading thanks for sharing

kingrani said...

Click here to Get UPSC IFS Admit Card
Apply Online for DPMU Jamshedpur Recruitment

Mirnalini Sathya said...

The way you have explained the Big Data concept is really superb! I have never come across such a informative blog in my career. Nowadays, Hadoop course is in high demand by most of the professional to enhance their career.


Big Data Training |
Big Data Course in Chennai

Suba said...

Nice blog on Selenium Testing!!! I agree with your points, Testing will help whole organization to handle the projects in efficient manner.

Selenium Training |
selenium training in T Nagar 

sophia sophia said...

Thanks for this wonderful post. The information in this article is very helpful to me. Thanks a lot for sharing. Keep blogging.
Interest Waypoint
Best Truth or Dare Questions for Kids

aashutosh webtrackker said...

Aws online training in india
Salesforce online training in india

SAS Online Training in india

Salesforce admin online training in india

Linux Online training in India

React.JS online training in india

aashutosh webtrackker said...

Python online Training in India

Oracle DBA online training in India

Java online Training in India

SAP online Training In india

Digital Marketing Online Training in India

aashutosh webtrackker said...

Aws online training in india
Salesforce online training in india

SAS Online Training in india

Salesforce admin online training in india

Linux Online training in India

React.JS online training in india

Srinu said...

Nice blog and thanks for sharing the post. Check out some latest projects.

Provident Kenworth
SMR Vinay Iconia
Provident The Pearl
Prestige Ivy League
Western Exotica
NCC Urban Gardenia
Quiescent Heights
NCC Urban One
NorthStar Homes District1
Provident Kenworth Hyderabad

ASO Services said...

There are so many useful information in your post, I like it all the time. And I want to share this boost app ranking with you, you can take a look. This is very useful.

Richard Swayar said...

Australia Best Tutor is one of the best Online Assignment Help providers at an affordable price. Here All Learners or Students are getting best quality assignment help with reference and styles formatting.

Visit us for more Information

Australia Best Tutor
Sydney, NSW, Australia
Call @ +61-730-407-305
Live Chat @ https://www.australiabesttutor.com

Our Services

Online assignment help
my assignment help Student
Assignment help Student
help with assignment Student
Students instant assignment help
Students Assignment help Services

priyanka sria said...

create android app without coding

Srinu said...

Nice blog and thanks for sharing the post.

Salarpuria Sattva Magnus
Sumadhura Acropolis
SMR Vinay Iconia
Prestige Ivy League
Jayabheri The Peak
Muppas Indraprastha
Patel Smondo
Vertex Panache
Western Exotica
NCC Urban Gardenia

priyanka sria said...

fax through gmail

Cricket Fan said...

Ramadan fasting greetings 2018

Mahesh said...

Really very informative and creative contents. This concept is a good way to enhance the knowledge.thanks for sharing.
please keep it up.
selenium training in gurgaon

tansitanu said...

Thanks a lot very much for the high quality and results-oriented help. I won’t think twice to endorse your blog post to anybody who wants and needs support about this area.
Best Hadoop Training Institute In chennai

Cricket Fan said...

Top 5 Best after effects plugins list

priyanka sria said...

Animated Movies

Cricket Fan said...

Create Your Own App for Free - Stunning Features & Design
3 Step Drag & Drop DIY App Maker - No Coding Needed - Create Your App Now!

Cricket Fan said...

How Webmail send and receive mail online

Cricket Fan said...

How to draw and learn drawing