1. 서론
hbase 에 java VM agent를 추가하여 region 서버 프로세스의 메모리 address를 lock할 수 있도록 기능을 추가할 예정이다. 이를 mlock_all이라 불렀다.
차기버전인 0.95 (아님 0.98부터??) 부터는 mlockall을 사용할 예정인데, 잠깐 훝어본다.
2. mlockall 의미
3. 관련 내용
주어진 물리 메모리보다 많은 job들을 하다보면 Swapping이 일어나는데, 이 때문에 Region Server 이 문제가 되거나 Zookeeper의 connection을 잃어버리기도 한다. swapping없이 메모리를 RAM에서만 쓰도록 하게 할 수 있도록 mlockall을 쓰게 한다.
만약 root권한이 아니면 다음과 같은 로그가 발생된다.
JNI에서 JVMTI를 이용했다.
소스.
http://svn.apache.org/viewvc/hbase/trunk/hbase-server/src/main/native/src/mlockall_agent/mlockall_agent.c?view=markup&pathrev=1353289
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
/**
* mlockall_agent is a simple VM Agent that allows to lock the address space of
* the process. This avoids the process' memory eviction under pressure.
*
* One example is when on the same machine you run the Region Server and
* some map-reduce tasks, some unused data in the region server may get swapped
* and this affects the region server performance.
*
* You can load the agent by adding it as a jvm option:
* export HBASE_REGIONSERVER_OPTS="-agentpath:./libmlockall_agent.so=user=hbase"
*/
#include <libgen.h>
#include <grp.h>
#include <pwd.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/resource.h>
#include <sys/time.h>
#include <sys/types.h>
#include <unistd.h>
#include "jvmti.h"
typedef struct opts {
char *setuid_user;
} opts_t;
#define PREFIX "mlockall_agent: "
#define LOG(fmt, ...) { fprintf(stderr, PREFIX fmt, #__VA_ARGS__); }
static int parse_options (const char *options, opts_t *parsed) {
char *optr, *opts_dup;
char *save2 = NULL;
char *save = NULL;
char *key, *val;
int ret = 0;
char *tok;
memset(parsed, 0, sizeof(opts_t));
if ((opts_dup = strdup(options)) == NULL)
return(-1);
optr = opts_dup;
while ((tok = strtok_r(optr, ",", &save)) != NULL) {
optr = NULL;
save2 = NULL;
key = strtok_r(tok, "=", &save2);
val = strtok_r(NULL, "=", &save2);
if (!strcmp(key, "user")) {
parsed->setuid_user = strdup(val);
} else {
LOG("Unknown agent parameter '%s'\n", key);
ret = 1;
}
}
free(opts_dup);
return(ret);
}
static void warn_unless_root() {
if (geteuid() != 0) {
LOG("(this may be because java was not run as root!)\n");
}
}
JNIEXPORT jint JNICALL Agent_OnLoad(JavaVM *vm, char *init_str, void *reserved) {
struct passwd *pwd = NULL;
opts_t opts;
if (parse_options(init_str, &opts)) {
return(1);
}
// Check that the target user for setuid is specified if current user is root
if (opts.setuid_user == NULL) {
LOG("Unable to setuid: specify a target username as the agent option user=<username>\n");
return(1);
}
// Check that this user exists
if ((pwd = getpwnam(opts.setuid_user)) == NULL) {
LOG("Unable to setuid: could not find user '%s'\n", opts.setuid_user);
return(1);
}
// Boost the mlock limit up to infinity
struct rlimit lim;
lim.rlim_max = RLIM_INFINITY;
lim.rlim_cur = lim.rlim_max;
if (setrlimit(RLIMIT_MEMLOCK, &lim)) {
perror(PREFIX "Unable to boost memlock resource limit");
warn_unless_root();
return(1);
}
// Actually lock our memory, including future allocations.
if (mlockall(MCL_CURRENT | MCL_FUTURE)) {
perror(PREFIX "Unable to lock memory.");
warn_unless_root();
return(1);
}
// Drop down to the user's supplemental group list
if (initgroups(opts.setuid_user, pwd->pw_gid)) {
perror(PREFIX "Unable to initgroups");
warn_unless_root();
return(1);
}
// And primary group ID
if (setgid(pwd->pw_gid)) {
perror(PREFIX "Unable to setgid");
warn_unless_root();
return(1);
}
// And user ID
if (setuid(pwd->pw_uid)) {
perror(PREFIX "Unable to setuid");
warn_unless_root();
return(1);
}
LOG("Successfully locked memory and setuid to %s\n", opts.setuid_user);
return(0);
}
4. 참고
elastic search에도 비슷한 내용이 들어가 있다.
Memory Settings
There is an option to use mlockall to try and lock the process address space so it won’t be swapped. For this to work, the bootstrap.mlockall should be set to true and it is recommended to set both the min and max memory allocation to be the same.
In order to see if this works or not, set the common.jna logging to DEBUG level. A solution to “Unknown mlockall error 0” can be to set ulimit -l unlimited.
Note, this is experimental feature, and might cause the JVM or shell session to exit if failing to allocate the memory (because not enough memory is available on the machine).
Redis맛보기.pdf
