NLP - BERT

A Bidirectional Encoder Representations from Transformers (aka BERT) encodes natural language using adjacent words to enhance contextual understanding. The provided implementation allows building on top of an existing BERT model:

  • Model: distilbert-base-uncased
  • Model is trained in a federated learning structure. Local models are trained per client, then models are averaged.
  • Tokenizer: DistilBertTokenizerFast

Operation

  • When using add_agreement() to forge an agreement on the trained model, use Operation.EXECUTE for the operation parameter.
  • When using add_agreement() to allow a counterparty to use your dataset for model training, or using create_job() to train your model, use Operation.BERT_SEQ_CLF_TRAIN.

Parameters

  • optimizer_meta: OptimizerParams = OptimizerParams()
  • epochs: int = 1
  • batchsize: int = 32
  • model_path: str = ""
  • dataset_path: str = ""
  • send_model: bool = False
  • test_size: float = 0.0
  • model_output: str = "binary" # Union["binary", "multiclass", "regression"]
  • binary_metric: str = "accuracy" # 🔗f1_score, 🔗roc_auc_score, 🔗precision_recall_curve
  • n_classes: int = None # currently used for object detection. This should not include background class.
  • num_clients: int = 0
  • client_index: int = 0
  • federated_rounds: int = 1
  • num_labels: int = 2
  • data_column: str = ""
  • target_column: str = ""

Limitations

  • BERT is not supported in SMPC.