Path /runtime-mnt/alluxio/default/hbase3/alluxio-12 is not mounted Starting AlluxioFuse process: mounting alluxio path "/" to local mount point "/runtime-mnt/alluxio/default/hbase3/alluxio-12" 2024-09-24 19:45:46,760 INFO AlluxioFuse - Alluxio version: 2.8.1-SNAPSHOT-5a002cedeb7b188aa83eca39c5ab9615929547b3 2024-09-24 19:45:47,041 INFO LibFuse - JNIFUSE_SHAREDLIB_DIR / temp dir: null 2024-09-24 19:45:47,043 INFO NativeLibraryLoader - sharedLibraryName: jnifuse3, jniLibraryName: jnifuse3jni-linux-aarch64, sharedLibraryFileName: libjnifuse3.so, jniLibraryFileName: libjnifuse3jni-linux-aarch64.so. 2024-09-24 19:45:47,046 INFO NativeLibraryLoader - loadLibraryFromJarToTemp params: libjnifuse3.so,libjnifuse3jni-linux-aarch64.so,null 2024-09-24 19:45:47,048 INFO NativeLibraryLoader - temp file: /tmp/libjnifuse8025725468782352104.so 2024-09-24 19:45:47,052 INFO NativeLibraryLoader - libPath /tmp/libjnifuse8025725468782352104.so INFO ../../src/main/native/libjnifuse/jnifuse_onload.cc:25 Loaded libjnifuse 2024-09-24 19:45:47,052 INFO NativeLibraryLoader - Loaded lib by jar from path /tmp/libjnifuse8025725468782352104.so. 2024-09-24 19:45:47,052 INFO NativeLibraryLoader - Loaded libjnifuse with libfuse version 3. 2024-09-24 19:45:47,133 INFO MetricsSystem - Starting sinks with config: {}. 2024-09-24 19:45:47,138 INFO MetricsHeartbeatContext - Created metrics heartbeat with ID app-3246309585540611550. This ID will be used for identifying info from the client. It can be set manually through the alluxio.user.app.id property 2024-09-24 19:45:47,283 INFO NettyUtils - EPOLL_MODE is available 2024-09-24 19:45:48,059 INFO TieredIdentityFactory - Initialized tiered identity TieredIdentity(node=, rack=null) 2024-09-24 19:45:48,319 INFO Reflections - Reflections took 156 ms to scan 1 urls, producing 58 keys and 199 values 2024-09-24 19:45:48,356 INFO AlluxioFuse - Mounting AlluxioJniFuseFileSystem: mount point="/runtime-mnt/alluxio/default/hbase3/alluxio-12", OPTIONS="[-okernel_cache, -oro, -oattr_timeout=7200, -oentry_timeout=7200, -oallow_other, -omax_readahead=0, -omax_write=131072]" 2024-09-24 19:45:48,356 INFO AbstractFuseFileSystem - Mounting /runtime-mnt/alluxio/default/hbase3/alluxio-12: blocking=true, debug=true, fuseOpts="[-okernel_cache, -oro, -oattr_timeout=7200, -oentry_timeout=7200, -oallow_other, -omax_readahead=0, -omax_write=131072]" INFO ../../src/main/native/libjnifuse/jnifuse_helper.cc:33 Start initializing JNIFuse ERROR ../../src/main/native/libjnifuse/jnifuse_helper.cc:34 Validate standard errors can be logged as expected FUSE library version: 3.9.2 nullpath_ok: 0 unique: 1, opcode: INIT (26), nodeid: 0, insize: 56, pid: 0
另起终端执行stat命令,报错如下:
1 2
[root@clusterd-test ~]# stat /runtime-mnt/alluxio/default/hbase3/alluxio-12 stat: cannot statx '/runtime-mnt/alluxio/default/hbase3/alluxio-12': 输入/输出错误
kubectl get pod hbase-fuse-sh7jm -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES hbase-fuse-sh7jm 1/1 Running 0 15m 172.17.0.5 clusterd-test <none> <none>
1 2 3
kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME clusterd-test Ready control-plane,master 20d v1.23.0 172.17.0.5 <none> CTyunOS 22.06 4.19.90-2102.2.0.0068.3.ctl2.aarch64 docker://20.10.12
根因定位
1 2
trace-cmd list -f | grep -i fuse | grep stat fuse_statfs [fuse]
1 2 3
trace-cmd record -p function_graph -g fuse_statfs
trace-cmd report
1 2 3 4 5
strace -f -e trace=file -T -o ls.log -tt ls /runtime-mnt/alluxio/default/hbase3/alluxio-12
trace-cmd record -p function_graph -g vfs_statx ls /runtime-mnt/alluxio/default/hbase3/alluxio-12 trace-cmd report > trace-cmd-alluxio-fuse-ls.log vim trace-cmd-alluxio-fuse-ls.log
fuse hello
1 2 3
trace-cmd record -p function_graph -g vfs_statx ls /runtime-mnt/alluxio/default/hbase3/alluxio-13 trace-cmd report > trace-cmd-hello-ls.log vim trace-cmd-hello-ls.log
[root@clusterd-test ~]# /opt/alluxio-2.8.0-0617/bin/alluxio-start.sh local Assuming NoMount by default. Successfully Killed 1 process(es) successfully on clusterd-test Successfully Killed 1 process(es) successfully on clusterd-test Successfully Killed 1 process(es) successfully on clusterd-test Successfully Killed 1 process(es) successfully on clusterd-test Successfully Killed 1 process(es) successfully on clusterd-test Successfully Killed 1 process(es) successfully on clusterd-test Starting master @ clusterd-test. Logging to /opt/alluxio-2.8.0-0617/logs Starting secondary master @ clusterd-test. Logging to /opt/alluxio-2.8.0-0617/logs Starting job master @ clusterd-test. Logging to /opt/alluxio-2.8.0-0617/logs Starting worker @ clusterd-test. Logging to /opt/alluxio-2.8.0-0617/logs Starting job worker @ clusterd-test. Logging to /opt/alluxio-2.8.0-0617/logs Starting proxy @ clusterd-test. Logging to /opt/alluxio-2.8.0-0617/logs ----------------------------------------- Starting to monitor all local services. ----------------------------------------- --- [ OK ] The master service @ clusterd-test is in a healthy state. --- [ OK ] The job_master service @ clusterd-test is in a healthy state. --- [ OK ] The worker service @ clusterd-test is in a healthy state. --- [ OK ] The job_worker service @ clusterd-test is in a healthy state. --- [ OK ] The proxy service @ clusterd-test is in a healthy state.
1 2
[root@clusterd-test user]# /opt/alluxio-2.8.0-0617/bin/alluxio fs touch /hello.txt /hello.txt has been created
1 2
[root@clusterd-test user]# /opt/alluxio-2.8.0-0617/bin/alluxio fs mkdir /test Successfully created directory /test
1 2 3
[root@clusterd-test logs]# /opt/alluxio-2.8.0-0617/bin/alluxio fs ls / 0 NOT_PERSISTED 09-26-2024 16:01:41:082 DIR /test 0 TO_BE_PERSISTED 09-26-2024 16:00:23:446 100% /hello.txt
1 2
[root@clusterd-test ~]# ls /mnt/alluxio-fuse ls: 无法访问 '/mnt/alluxio-fuse': 输入/输出错误
480 2024-09-26 16:34:34,483 WARN DefaultFileSystemMaster - Failed to delete UFS file /underFSStorage/.alluxio_ufs_persistence/hello.txt.alluxio.1727338650414.5b3a84d3-10ef-4a9d-8f60-af0922e06a8b.tmp. 481 2024-09-26 16:34:36,512 WARN DefaultFileSystemMaster - The persist job (id=1727337417428) for file /hello.txt (id=33554431) failed: Task execution failed: Failed to create /underFSStorage/.alluxio_ufs_persistence/hello.txt.alluxio.1727339674483.010f3621-ca80-4a6e-99c9-2a778bfcd6b3.tmp with permission readType: NO_CACHE 482 updateLastAccessTime: false 483 because its ancestor /underFSStorage/.alluxio_ufs_persistence is not a directory