Point Cloud Encoders
Flemme supports various point cloud encoders using MLP (1-D convolution), transformer and SSM as backbones. Flemme accommodates both non-hierarchical and hierarchical architectural designs for point cloud encoders and decoders.
Non-Hierarchical
Non-hierarchical encoders and decoders follow a pipeline illustrated in following figure (similar to PointNet and DGCNN):
Note that only one of the components enclosed within red dashed box should be employed.
PointNet
PointNetEncoder use sharedMLP as backbones and has a similar architecture to PointNet. In addition, we also integrate dynamic local feature extraction (from DGCNN) into PointNetEncoder. To initialize a PointNet encoder, you need to specify the following parameters:
1PointNetEncoder.__init__(self, point_dim=3,
2 projection_channel = 64,
3 time_channel = 0,
4 ## parameter of KNN for local feature extraction.
5 ## if num_neighbors_k is set as 0, local feature extraction will not be performed.
6 num_neighbors_k=0,
7 local_feature_channels = [64, 64, 128, 256],
8 num_blocks = 2,
9 dense_channels = [256, 256],
10 building_block = 'dense',
11 normalization = 'group',
12 num_norm_groups = 8,
13 activation = 'lrelu',
14 dropout = 0.,
15 z_count = 1,
16 ## If vector_embedding is true, the latent embedding is a vector;
17 ## otherwise, the latent embedding is point-wise features.
18 vector_embedding = True,
19 last_activation = True,
20 **kwargs)
In the above, if num_neighbors_k is specified, we will use edge convolution to extraction dynamic local features through KNN algorithm. Otherwise, local features will not be computed.
If vector_embedding is set as True, the encoder will return a latent vector for each inputs. Otherwise, the encoder will return point-wise features.
PointNetDecoder is used to decode points from latent space through MLPs, which can used for reconstruction, segmentation, and other tasks. Typically, we dirrectly recover points from latent embeddings. In addition, we also integrate Folding operation (from FoldingNet) into PointNetDecoder when you specify folding_times. We support grid2d, grid3d, cylinder, uvsphere and icosphere as base shapes for folding operation.
Note that PointNetDecoder can be used for all point cloud encoders.
1PointNetDecoder.__init__(self, point_dim=3,
2 point_num = 2048,
3 ## out_channel of encoder is the in channel of decoder
4 in_channel = 256,
5 dense_channels = [256],
6 time_channel = 0,
7 normalization = 'group',
8 num_norm_groups = 8,
9 activation = 'lrelu',
10 dropout = 0.,
11 ## if folding_times > 0, we will perform folding operations.
12 folding_times = 0,
13 ## channel of hidden layers in folding operations.
14 folding_hidden_channels = [512, 512],
15 base_shape_config = {},
16 num_blocks = 2,
17 ## If folding_times = 0, we process results through a final MLPs.
18 final_channels = [512, 512],
19 ## Note that vector_embedding need to be True if you want to perform folding operations.
20 vector_embedding = True,
21 **kwargs)
The aboved parameters can be defined in the config file, in which the in_channel and out_channel refer to the point_dim of encoder and decoder respectively., decoder_fc_channel refers to fc_channel of decoder.
As we discussed before, you don’t need to define all parameters for encoder and decoder in the configuration file.
1encoder:
2 name: PointNet
3 in_channel: 3
4 out_channel: 46
5 point_num: 2048
6 building_block: dense
7 # num_neighbors_k: 20
8 local_feature_channels: [64, 64, 128, 256, 512]
9 dense_channels: [1024, 512, 256]
10 final_channels: [512, 512]
11 activation: lrelu
12 normalization: group
Supported building_block for PointNet encoder and decoder: [dense, res_dense, double_dense].
PointTrans
PointTrans indicates the encoders have a similar architecture as PointNet encoder but using transformer as backbones. To initialize a PointTransEncoder, you need to specify the following parameters:
1PointTransEncoder.__init__(self, point_dim=3,
2 projection_channel = 64,
3 time_channel = 0,
4 num_neighbors_k=0,
5 local_feature_channels = [64, 64, 128, 256],
6 num_blocks = 2,
7 dense_channels = [256, 256],
8 building_block = 'pct_sa',
9 normalization = 'group', num_norm_groups = 8,
10 activation = 'lrelu', dropout = 0.,
11 ### transformer parameters
12 num_heads = 4, d_k = None,
13 qkv_bias = True,
14 qk_scale = None,
15 atten_dropout = None,
16 residual_attention = False,
17 skip_connection = True,
18 z_count = 1,
19 vector_embedding = True,
20 last_activation = True,
21 **kwargs)
Note that PointTransDecoder is just an alias of PointNetDecoder. Supported building_block for PointTrans encoder: [pct_sa, pct_oa], with pct_sa denoting self-attention and pct_oa denoting offset-attention from PCT, respectively.
PointMamba
PointMamba indicates the encoders using state space model (mamba) as backbones. To initialize a PointMambaEncoder,
you need to specify the following parameters:
1PointMambaEncoder.__init__(self, point_dim=3,
2 projection_channel = 64,
3 time_channel = 0,
4 num_neighbors_k=0,
5 local_feature_channels = [64, 64, 128, 256],
6 num_blocks = 2,
7 dense_channels = [256, 256],
8 building_block = 'pmamba',
9 normalization = 'group', num_norm_groups = 8,
10 activation = 'lrelu', dropout = 0.,
11 state_channel = 64,
12 conv_kernel_size = 4, inner_factor = 2.0,
13 head_channel = 64,
14 conv_bias=True, bias=False,
15 learnable_init_states = True, chunk_size=256,
16 dt_min=0.001, A_init_range=(1, 16),
17 dt_max=0.1, dt_init_floor=1e-4,
18 dt_rank = None, dt_scale = 1.0,
19 z_count = 1, vector_embedding = True,
20 last_activation = True,
21 skip_connection = True,
22 **kwargs)
Note that PointMambaDecoder is just an alias of PointNetDecoder. Supported building_block for PointMamba encoder: [pmamba, pmamba2].
Hierarchical
Hierarchical point cloud encoders and decoders follow a pipeline illustrated in following figure (similar to PointNet++):
In hierarchical encoders, we down-sample point cloud into different levels through the farthest-point sampling (FPS) and aggregate points through multi-scale neighbor queries.
We will elaborate supported hierarchical encoders in the remainder of this article.
PointNet2
PointNet2 has a hierarchical architecture with sharedMLP backbones. To define a PointNet2Encoder, you need to specify the following parameters:
1PointNet2Encoder.__init__(self, point_dim = 3,
2 projection_channel = 64,
3 time_channel = 0,
4 ## number of fps points for each level
5 num_fps_points = [1024, 512, 256, 64],
6 ## number of neighbor query for each level
7 num_neighbors_k = 32,
8 ## max radius of radius query for each level
9 neighbor_radius = [0.1, 0.2, 0.4, 0.8],
10 ## number of feature channels for each level
11 fps_feature_channels = [128, 256, 512, 1024],
12 num_blocks = 2,
13 ## number of scales
14 ## the radius and number of feature channels at each scale will be assigned based on the neighbor_radius and fps_feature_channels
15 num_scales = 2,
16 ## concat point position embedding in feature extraction
17 use_xyz = True,
18 ## sort the radius query results by distance (knn query returns a sorted result by default).
19 sorted_query = False,
20 ## use knn query or radius query.
21 ## if knn_query is not false, it should be one of ['xyz', 'xyz_embed', 'feature'], which determine the knn query space.
22 knn_query = False,
23 dense_channels = [1024],
24 building_block = 'dense',
25 normalization = 'group', num_norm_groups = 8,
26 activation = 'lrelu', dropout = 0.,
27 vector_embedding = True,
28 ## if the decoder is a pointnet2-like decoder, the feature list will also be returned.
29 is_point2decoder = False,
30 z_count = 1,
31 return_xyz = False,
32 last_activation = True,
33 ## final concatenation of sample features at different levels
34 final_concat = False,
35 ## enable positional embedding
36 pos_embedding = False,
37 **kwargs)
PointTrans2
PointTrans2 has a hierarchical architecture with transformer backbones. To define a PointTrans2Encoder, you need to specify the following parameters:
1PointTrans2Encoder.__init__(self, point_dim = 3,
2 projection_channel = 64,
3 time_channel = 0,
4 num_fps_points = [1024, 512, 256, 64],
5 num_neighbors_k = 32,
6 neighbor_radius = [0.1, 0.2, 0.4, 0.8],
7 fps_feature_channels = [128, 256, 512, 1024],
8 num_blocks = 2,
9 num_scales = 2,
10 use_xyz = True,
11 sorted_query = False,
12 knn_query = False,
13 dense_channels = [1024],
14 building_block = 'dense',
15 normalization = 'group', num_norm_groups = 8,
16 activation = 'lrelu', dropout = 0.,
17 num_heads = 4, d_k = None,
18 qkv_bias = True, qk_scale = None, atten_dropout = None,
19 residual_attention = False, skip_connection = True,
20 vector_embedding = True,
21 is_point2decoder = False,
22 ## Perform long range modeling on a global scale.
23 ## Long range modeling is only suitable for sequence-modeling backbones, e.g., transformer and ssm.
24 long_range_modeling = False,
25 z_count = 1,
26 return_xyz = False,
27 last_activation = True,
28 final_concat = False,
29 pos_embedding = False,
30 **kwargs)
PointMamba2
PointMamba2 has a hierarchical architecture with state space model (mamba) backbones.
A more detailed illustration of PointMamba2 Encoder with xyz¢er scanning and long-range modeling is presented in:
To define a PointMamba2Encoder, you need to specify the following parameters:
1PointMamba2Encoder.__init__(self, point_dim = 3,
2 projection_channel = 64,
3 time_channel = 0,
4 num_fps_points = [1024, 512, 256, 64],
5 num_neighbors_k = 32,
6 neighbor_radius = [0.1, 0.2, 0.4, 0.8],
7 fps_feature_channels = [128, 256, 512, 1024],
8 num_blocks = 2,
9 num_scales = 2,
10 use_xyz = True,
11 sorted_query = False,
12 knn_query = False,
13 dense_channels = [1024],
14 building_block = 'dense',
15 flip_scan = False,
16 normalization = 'group', num_norm_groups = 8,
17 activation = 'lrelu', dropout = 0.,
18 vector_embedding = True,
19 state_channel = 64,
20 conv_kernel_size = 4, inner_factor = 2.0,
21 head_channel = 64,
22 conv_bias=True, bias=False,
23 learnable_init_states = True, chunk_size=256,
24 dt_min=0.001, A_init_range=(1, 16),
25 dt_max=0.1, dt_init_floor=1e-4,
26 dt_rank = None, dt_scale = 1.0,
27 skip_connection = True,
28 is_point2decoder = False,
29 long_range_modeling = False,
30 ## point serialization
31 ## scan strategies should be a list whose elements are from ['x_order', 'y_order', 'z_order', 'center_dist', 'nonsort']
32 scan_strategies = None,
33 z_count = 1,
34 return_xyz = False,
35 last_activation = True,
36 final_concat = False,
37 pos_embedding = False,
38 **kwargs)
To summarize, we support the following point cloud encoders:
Encoder |
Backbones |
Building Blocks |
Long-range Modeling |
Scanning |
|---|---|---|---|---|
PointNet |
MLP |
dense, double_dense, res_dense |
× |
× |
PointNet2 |
MLP |
dense, double_dense, res_dense |
× |
× |
PointTrans |
transformer |
pct_sa, pct_oa |
× |
× |
PointTrans2 |
transformer |
pct_sa, pct_oa |
√ |
× |
PointMamba |
ssm (mamba) |
pmamba, pmamba2 |
× |
× |
PointMamba2 |
ssm (mamba) |
pmamba, pmamba2 |
√ |
√ |