fs: btrfs: fix btrfs methods return values on failure