Fine-Tuning Pre-trained Models
Last updated July 1, 2024
Introduction: Fine-tuning a pre-trained model allows you to adapt it to your specific task using your custom dataset. This process can significantly enhance the model's performance on specialized tasks.
Steps:
- Preparing Your Dataset
- Format Your Data: Ensure your dataset is in a suitable format, such as CSV or JSON, with clearly labeled inputs and outputs.
- Load the Dataset:
from datasets import load_dataset dataset = load_dataset('path/to/your/dataset')
- Loading a Pre-trained Model and Tokenizer Select a Suitable Model: from transformers import AutoTokenizer, AutoModelForSequenceClassification model_name = "bert-base-uncased" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForSequenceClassification.from_pretrained(model_name)
- Tokenizing the Dataset Tokenize Inputs: deftokenize_function(examples): return tokenizer(examples['text'], padding="max_length", truncation=True) tokenized_datasets = dataset.map(tokenize_function, batched=True)
- Setting Up Training Parameters Training Arguments: from transformers import TrainingArguments training_args = TrainingArguments( output_dir='./results', evaluation_strategy="epoch", learning_rate=2e-5, per_device_train_batch_size=16, per_device_eval_batch_size=16, num_train_epochs=3, weight_decay=0.01, )
- Training the Model Trainer Setup and Training: from transformers import Trainer trainer = Trainer( model=model, args=training_args, train_dataset=tokenized_datasets['train'], eval_dataset=tokenized_datasets['validation'], ) trainer.train()
- Evaluating the Model
Evaluate on Validation Set: eval_results = trainer.evaluate() print(f"Evaluation results: {eval_results}")
Was this article helpful?