In [1]:
!pip install -qU datasets trl peft bitsandbytes sentencepiece unsloth
In [2]:
from huggingface_hub import login
from google.colab import userdata
token = userdata.get('hf_token')
login(token)
In [3]:
from datasets import load_dataset, DatasetDict
ds = load_dataset("jondurbin/gutenberg-dpo-v0.1")
split_ds = ds['train'].train_test_split(test_size=0.1)
final_ds = DatasetDict({
'train': split_ds['train'],
'eval': split_ds['test']
})
print(f"Train dataset len: {len(final_ds['train'])}")
print(f"Val dataset len: {len(final_ds['eval'])}")
print(f"Message demo: {final_ds['train'][0]}")
/usr/local/lib/python3.12/dist-packages/huggingface_hub/utils/_auth.py:94: UserWarning: The secret `HF_TOKEN` does not exist in your Colab secrets. To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session. You will be able to reuse this secret in all of your notebooks. Please note that authentication is recommended but still optional to access public models or datasets. warnings.warn(
Train dataset len: 826
Val dataset len: 92
Message demo: {'id': '6d99bb8a-a621-5a74-a71c-3b306c58693a', 'source': 'gutenberg Frankenstein', 'chosen': '“Such was the history of my beloved cottagers. It impressed me deeply. I learned, from the views of social life which it developed, to admire their virtues, and to deprecate the vices of mankind.\n\n“As yet I looked upon crime as a distant evil; benevolence and generosity were ever present before me, inciting within me a desire to become an actor in the busy scene where so many admirable qualities were called forth and displayed. But, in giving an account of the progress of my intellect, I must not omit a circumstance which occurred in the beginning of the month of August of the same year.\n\n“One night, during my accustomed visit to the neighbouring wood, where I collected my own food, and brought home firing for my protectors, I found on the ground a leathern portmanteau, containing several articles of dress and some books. I eagerly seized the prize, and returned with it to my hovel. Fortunately the books were written in the language the elements of which I had acquired at the cottage; they consisted of\n_Paradise Lost_, a volume of _Plutarch’s Lives_, and the _Sorrows of Werter_. The possession of these treasures gave me extreme delight; I now continually studied and exercised my mind upon these histories, whilst my friends were employed in their ordinary occupations.\n\n“I can hardly describe to you the effect of these books. They produced in me an infinity of new images and feelings, that sometimes raised me to ecstacy, but more frequently sunk me into the lowest dejection. In the _Sorrows of Werter_, besides the interest of its simple and affecting story, so many opinions are canvassed, and so many lights thrown upon what had hitherto been to me obscure subjects, that I found in it a never-ending source of speculation and astonishment. The gentle and domestic manners it described, combined with lofty sentiments and feelings, which had for their object something out of self, accorded well with my experience among my protectors, and with the wants which were for ever alive in my own bosom. But I thought Werter himself a more divine being than I had ever beheld or imagined; his character contained no pretension, but it sunk deep. The disquisitions upon death and suicide were calculated to fill me with wonder. I did not pretend to enter into the merits of the case, yet I inclined towards the opinions of the hero, whose extinction I wept, without precisely understanding it.\n\n“As I read, however, I applied much personally to my own feelings and condition. I found myself similar, yet at the same time strangely unlike the beings concerning whom I read, and to whose conversation I was a listener. I sympathized with, and partly understood them, but I was unformed in mind; I was dependent on none, and related to none. ‘The path of my departure was free;’ and there was none to lament my annihilation. My person was hideous, and my stature gigantic: what did this mean? Who was I? What was I? Whence did I come? What was my destination? These questions continually recurred, but I was unable to solve them.\n\n“The volume of _Plutarch’s Lives_ which I possessed, contained the histories of the first founders of the ancient republics. This book had a far different effect upon me from the _Sorrows of Werter_. I learned from Werter’s imaginations despondency and gloom: but Plutarch taught me high thoughts; he elevated me above the wretched sphere of my own reflections, to admire and love the heroes of past ages. Many things I read surpassed my understanding and experience. I had a very confused knowledge of kingdoms, wide extents of country, mighty rivers, and boundless seas. But I was perfectly unacquainted with towns, and large assemblages of men. The cottage of my protectors had been the only school in which I had studied human nature; but this book developed new and mightier scenes of action. I read of men concerned in public affairs governing or massacring their species. I felt the greatest ardour for virtue rise within me, and abhorrence for vice, as far as I understood the signification of those terms, relative as they were, as I applied them, to pleasure and pain alone. Induced by these feelings, I was of course led to admire peaceable law-givers, Numa, Solon, and Lycurgus, in preference to Romulus and Theseus. The patriarchal lives of my protectors caused these impressions to take a firm hold on my mind; perhaps, if my first introduction to humanity had been made by a young soldier, burning for glory and slaughter, I should have been imbued with different sensations.\n\n“But _Paradise Lost_ excited different and far deeper emotions. I read it, as I had read the other volumes which had fallen into my hands, as a true history. It moved every feeling of wonder and awe, that the picture of an omnipotent God warring with his creatures was capable of exciting.\nI often referred the several situations, as their similarity struck me, to my own. Like Adam, I was created apparently united by no link to any other being in existence; but his state was far different from mine in every other respect. He had come forth from the hands of God a perfect creature, happy and prosperous, guarded by the especial care of his Creator; he was allowed to converse with, and acquire knowledge from beings of a superior nature: but I was wretched, helpless, and alone.\nMany times I considered Satan as the fitter emblem of my condition; for often, like him, when I viewed the bliss of my protectors, the bitter gall of envy rose within me.\n\n“Another circumstance strengthened and confirmed these feelings. Soon after my arrival in the hovel, I discovered some papers in the pocket of the dress which I had taken from your laboratory. At first I had neglected them; but now that I was able to decypher the characters in which they were written, I began to study them with diligence. It was your journal of the four months that preceded my creation. You minutely described in these papers every step you took in the progress of your work; this history was mingled with accounts of domestic occurrences.\nYou, doubtless, recollect these papers. Here they are. Every thing is related in them which bears reference to my accursed origin; the whole detail of that series of disgusting circumstances which produced it is set in view; the minutest description of my odious and loathsome person is given, in language which painted your own horrors, and rendered mine ineffaceable. I sickened as I read. ‘Hateful day when I received life!’ I exclaimed in agony. ‘Cursed creator! Why did you form a monster so hideous that even you turned from me in disgust? God in pity made man beautiful and alluring, after his own image; but my form is a filthy type of your’s, more horrid from its very resemblance. Satan had his companions, fellow-devils, to admire and encourage him; but I am solitary and detested.’\n\n“These were the reflections of my hours of despondency and solitude; but when I contemplated the virtues of the cottagers, their amiable and benevolent dispositions, I persuaded myself that when they should become acquainted with my admiration of their virtues, they would compassionate me, and overlook my personal deformity. Could they turn from their door one, however monstrous, who solicited their compassion and friendship?\nI resolved, at least, not to despair, but in every way to fit myself for an interview with them which would decide my fate. I postponed this attempt for some months longer; for the importance attached to its success inspired me with a dread lest I should fail. Besides, I found that my understanding improved so much with every day’s experience, that I was unwilling to commence this undertaking until a few more months should have added to my wisdom.\n\n“Several changes, in the mean time, took place in the cottage. The presence of Safie diffused happiness among its inhabitants; and I also found that a greater degree of plenty reigned there. Felix and Agatha spent more time in amusement and conversation, and were assisted in their labours by servants. They did not appear rich, but they were contented and happy; their feelings were serene and peaceful, while mine became every day more tumultuous. Increase of knowledge only discovered to me more clearly what a wretched outcast I was. I cherished hope, it is true; but it vanished, when I beheld my person reflected in water, or my shadow in the moon-shine, even as that frail image and that inconstant shade.\n\n“I endeavoured to crush these fears, and to fortify myself for the trial which in a few months I resolved to undergo; and sometimes I allowed my thoughts, unchecked by reason, to ramble in the fields of Paradise, and dared to fancy amiable and lovely creatures sympathizing with my feelings and cheering my gloom; their angelic countenances breathed smiles of consolation. But it was all a dream: no Eve soothed my sorrows, or shared my thoughts; I was alone. I remembered Adam’s supplication to his Creator; but where was mine? he had abandoned me, and, in the bitterness of my heart, I cursed him.\n\n“Autumn passed thus. I saw, with surprise and grief, the leaves decay and fall, and nature again assume the barren and bleak appearance it had worn when I first beheld the woods and the lovely moon. Yet I did not heed the bleakness of the weather; I was better fitted by my conformation for the endurance of cold than heat. But my chief delights were the sight of the flowers, the birds, and all the gay apparel of summer; when those deserted me, I turned with more attention towards the cottagers. Their happiness was not decreased by the absence of summer.\nThey loved, and sympathized with one another; and their joys, depending on each other, were not interrupted by the casualties that took place around them. The more I saw of them, the greater became my desire to claim their protection and kindness; my heart yearned to be known and loved by these amiable creatures: to see their sweet looks turned towards me with affection, was the utmost limit of my ambition. I dared not think that they would turn them from me with disdain and horror. The poor that stopped at their door were never driven away. I asked, it is true, for greater treasures than a little food or rest; I required kindness and sympathy; but I did not believe myself utterly unworthy of it.\n\n“The winter advanced, and an entire revolution of the seasons had taken place since I awoke into life. My attention, at this time, was solely directed towards my plan of introducing myself into the cottage of my protectors. I revolved many projects; but that on which I finally fixed was, to enter the dwelling when the blind old man should be alone. I had sagacity enough to discover, that the unnatural hideousness of my person was the chief object of horror with those who had formerly beheld me. My voice, although harsh, had nothing terrible in it; I thought, therefore, that if, in the absence of his children, I could gain the good-will and mediation of the old De Lacy, I might, by his means, be tolerated by my younger protectors.\n\n“One day, when the sun shone on the red leaves that strewed the ground, and diffused cheerfulness, although it denied warmth, Safie, Agatha, and Felix, departed on a long country walk, and the old man, at his own desire, was left alone in the cottage. When his children had departed, he took up his guitar, and played several mournful, but sweet airs, more sweet and mournful than I had ever heard him play before. At first his countenance was illuminated with pleasure, but, as he continued, thoughtfulness and sadness succeeded; at length, laying aside the instrument, he sat absorbed in reflection.\n\n“My heart beat quick; this was the hour and moment of trial, which would decide my hopes, or realize my fears. The servants were gone to a neighbouring fair. All was silent in and around the cottage: it was an excellent opportunity; yet, when I proceeded to execute my plan, my limbs failed me, and I sunk to the ground. Again I rose; and, exerting all the firmness of which I was master, removed the planks which I had placed before my hovel to conceal my retreat. The fresh air revived me, and, with renewed determination, I approached the door of their cottage.\n\n“I knocked. ‘Who is there?’ said the old man—‘Come in.’\n\n“I entered; ‘Pardon this intrusion,’ said I, ‘I am a traveller in want of a little rest; you would greatly oblige me, if you would allow me to remain a few minutes before the fire.’\n\n“‘Enter,’ said De Lacy; ‘and I will try in what manner I can relieve your wants; but, unfortunately, my children are from home, and, as I am blind, I am afraid I shall find it difficult to procure food for you.’\n\n“‘Do not trouble yourself, my kind host, I have food; it is warmth and rest only that I need.’\n\n“I sat down, and a silence ensued. I knew that every minute was precious to me, yet I remained irresolute in what manner to commence the interview; when the old man addressed me—\n\n“‘By your language, stranger, I suppose you are my countryman;—are you French?’\n\n“‘No; but I was educated by a French family, and understand that language only. I am now going to claim the protection of some friends, whom I sincerely love, and of whose favour I have some hopes.’\n\n“‘Are these Germans?’\n\n“‘No, they are French. But let us change the subject. I am an unfortunate and deserted creature; I look around, and I have no relation or friend upon earth. These amiable people to whom I go have never seen me, and know little of me. I am full of fears; for if I fail there, I am an outcast in the world for ever.’\n\n“‘Do not despair. To be friendless is indeed to be unfortunate; but the hearts of men, when unprejudiced by any obvious self-interest, are full of brotherly love and charity. Rely, therefore, on your hopes; and if these friends are good and amiable, do not despair.’\n\n“‘They are kind—they are the most excellent creatures in the world; but, unfortunately, they are prejudiced against me. I have good dispositions; my life has been hitherto harmless, and, in some degree, beneficial; but a fatal prejudice clouds their eyes, and where they ought to see a feeling and kind friend, they behold only a detestable monster.’\n\n“‘That is indeed unfortunate; but if you are really blameless, cannot you undeceive them?’\n\n“‘I am about to undertake that task; and it is on that account that I feel so many overwhelming terrors. I tenderly love these friends; I have, unknown to them, been for many months in the habits of daily kindness towards them; but they believe that I wish to injure them, and it is that prejudice which I wish to overcome.’\n\n“‘Where do these friends reside?’\n\n“‘Near this spot.’\n\n“The old man paused, and then continued, ‘If you will unreservedly confide to me the particulars of your tale, I perhaps may be of use in undeceiving them. I am blind, and cannot judge of your countenance, but there is something in your words which persuades me that you are sincere. I am poor, and an exile; but it will afford me true pleasure to be in any way serviceable to a human creature.’\n\n“‘Excellent man! I thank you, and accept your generous offer. You raise me from the dust by this kindness; and I trust that, by your aid, I shall not be driven from the society and sympathy of your fellow-creatures.’\n\n“‘Heaven forbid! even if you were really criminal; for that can only drive you to desperation, and not instigate you to virtue. I also am unfortunate; I and my family have been condemned, although innocent: judge, therefore, if I do not feel for your misfortunes.’\n\n“‘How can I thank you, my best and only benefactor? from your lips first have I heard the voice of kindness directed towards me; I shall be for ever grateful; and your present humanity assures me of success with those friends whom I am on the point of meeting.’\n\n“‘May I know the names and residence of those friends?’\n\n“I paused. This, I thought, was the moment of decision, which was to rob me of, or bestow happiness on me for ever. I struggled vainly for firmness sufficient to answer him, but the effort destroyed all my remaining strength; I sank on the chair, and sobbed aloud. At that moment I heard the steps of my younger protectors. I had not a moment to lose; but, seizing the hand of the old man, I cried, ‘Now is the time!—save and protect me! You and your family are the friends whom I seek. Do not you desert me in the hour of trial!’\n\n“‘Great God!’ exclaimed the old man, ‘who are you?’\n\n“At that instant the cottage door was opened, and Felix, Safie, and Agatha entered. Who can describe their horror and consternation on beholding me? Agatha fainted; and Safie, unable to attend to her friend, rushed out of the cottage. Felix darted forward, and with supernatural force tore me from his father, to whose knees I clung: in a transport of fury, he dashed me to the ground, and struck me violently with a stick.\nI could have torn him limb from limb, as the lion rends the antelope.\nBut my heart sunk within me as with bitter sickness, and I refrained. I saw him on the point of repeating his blow, when, overcome by pain and anguish, I quitted the cottage, and in the general tumult escaped unperceived to my hovel.”', 'rejected': "As the days passed, the creature's curiosity grew, and he began to explore the cottage. One day, he stumbled upon a bookshelf, filled with various volumes, and his heart skipped a beat. He had never seen such a treasure trove of knowledge before. He was drawn to the books like a moth to a flame, and he began to read, devouring every word.\n\nThe creature was particularly intrigued by the works of Mary Shelley, a woman who had written a book about a creature just like him. The creature found solace in her words, and he felt a connection to her character, Frankenstein. He learned about the pain and isolation that Frankenstein felt, and he realized that he was not alone in his suffering.\n\nThe creature began to read other books as well, and he found himself drawn to the works of Shakespeare. He was particularly moved by the tragedy of Romeo and Juliet, and he felt a profound sense of loss and longing that he had never experienced before. He began to understand the depths of human emotion and the complexities of the human soul.\n\nAs the creature continued to read, he developed a sense of morality and ethics. He began to question his own actions and the actions of others. He realized that he had done wrong by killing the De Lacey family, and he felt a deep sense of remorse. He vowed to change his ways and to seek out the companionship of others, despite his outward appearance.\n\nOne day, the creature was sitting by the fire when he heard a commotion outside. He looked out the window and saw a group of people approaching the cottage. He recognized them as the family he had been observing from a distance. They were a mother, a father, and their young son. The creature was drawn to them, and he decided to reveal himself to them, hoping to find the companionship he so desperately craved.\n\nAs the family approached the cottage, the creature stepped out from the shadows. The mother and father gasped in horror, and the son cried out in fear. The creature's heart sank, but he refused to give up. He knew that he had to try to explain himself, to show them that he was not the monster they believed him to be.\n\nThe creature introduced himself to the family, explaining that he was not a monster, but a creature created by a mad scientist. He told them of his desire to be loved and accepted, and he begged them to see past his outward appearance. He shared with them his love of literature and his newfound sense of morality and ethics.\n\nThe family listened to the creature's story, and they were moved by his words. They realized that the creature was not the monster they had feared, but a creature with a heart and a soul. They agreed to give the creature a chance, and they welcomed him into their home.\n\nThe creature was overjoyed, and he felt a sense of peace and happiness that he had never experienced before. He began to spend his days with the family, learning about their lives and their culture. He felt a sense of belonging that he had never known, and he was grateful for the opportunity to be a part of their lives.\n\nAs time went on, the creature continued to read and to learn. He became a scholar, and he began to contribute to the family's well-being. He helped with the chores, and he shared his knowledge of literature and philosophy with the family. He became a beloved member of the family, and he was finally able to experience the love and acceptance that he had always longed for.\n\nThe creature's life with the family was a testament to the power of literature and the human spirit. Through the pages of books, he had found a sense of purpose and a connection to the world around him. He had learned to love and to be loved, and he had finally found a place where he belonged.", 'prompt': "Summary of previous portion of the novel:\nThe previous chapter delves into the backstory of the De Lacey family, friends of the speaker, who once lived a prosperous life in Paris. The father, De Lacey, and his children, Felix and Agatha, fell into misfortune due to the wrongful condemnation of a Turkish merchant, father to Safie, whom Felix attempted to save out of sheer indignation over the unjust trial. Safie, a Christian Arab, was to be rewarded to Felix in marriage as a token of gratitude; however, the Turkish merchant's deceit and subsequent betrayal resulted in the De Lacey family losing their wealth and being exiled, struggling in a German cottage. In a turn of events, Safie, defying her father's wishes to return to Turkey and intent on reuniting with Felix, escapes to Germany and is left alone after the death of her companion, ultimately reaching the De Lacey cottage.\n\nWrite the next chapter of a story in novel format from the perspective of a misunderstood creature who yearns for human connection and belonging. Describe its encounters with literature and the profound impact these books have on its developing intellect and emotions. Include a significant interaction between the creature and a family it has been observing, emphasizing its internal conflict and desire to be accepted despite its outward appearance.", 'rejected_model': 'bagel-7b-v0.1'}
In [4]:
model_id, output_dir = "unsloth/Qwen2.5-7B-Instruct-bnb-4bit", "Qwen2.5-7b-instruct-bnb-4bit-dpo"
In [5]:
from unsloth import FastLanguageModel, PatchDPOTrainer
from transformers import TrainingArguments
from trl import DPOTrainer
import torch
model, tokenizer = FastLanguageModel.from_pretrained(
model_name = model_id,
max_seq_length = 2048,
load_in_4bit = True,
)
🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning. 🦥 Unsloth Zoo will now patch everything to make training faster!
Unable to import `torchao` Tensor objects. This may affect loading checkpoints serialized with `torchao` Flax classes are deprecated and will be removed in Diffusers v1.0.0. We recommend migrating to PyTorch classes or pinning your version of Diffusers. Flax classes are deprecated and will be removed in Diffusers v1.0.0. We recommend migrating to PyTorch classes or pinning your version of Diffusers.
==((====))== Unsloth 2025.11.1: Fast Qwen2 patching. Transformers: 4.57.2. \\ /| NVIDIA L4. Num GPUs = 1. Max memory: 22.034 GB. Platform: Linux. O^O/ \_/ \ Torch: 2.10.0+cu128. CUDA: 8.9. CUDA Toolkit: 12.8. Triton: 3.6.0 \ / Bfloat16 = TRUE. FA [Xformers = 0.0.35. FA2 = False] "-____-" Free license: http://github.com/unslothai/unsloth Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored! unsloth/qwen2.5-7b-instruct-bnb-4bit does not have a padding token! Will use pad_token = <|PAD_TOKEN|>.
In [6]:
model = FastLanguageModel.get_peft_model(
model,
r = 16,
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
lora_alpha = 32,
lora_dropout = 0,
bias = "none",
)
PatchDPOTrainer()
Unsloth 2025.11.1 patched 28 layers with 28 QKV layers, 28 O layers and 28 MLP layers.
In [7]:
print(tokenizer.pad_token, tokenizer.pad_token_id)
tokenizer.padding_side = "right"
<|PAD_TOKEN|> 151665
In [12]:
from trl import DPOConfig
training_args = DPOConfig(
per_device_train_batch_size = 2,
gradient_accumulation_steps = 4,
warmup_ratio = 0.1,
num_train_epochs = 1,
learning_rate = 2e-6,
bf16 = True,
logging_steps = 1,
optim = "adamw_8bit",
weight_decay = 0.01,
eval_steps = 10,
metric_for_best_model = "eval_loss",
greater_is_better=False,
output_dir = "outputs",
eval_strategy = "steps"
)
In [13]:
from transformers import EarlyStoppingCallback
dpo_trainer = DPOTrainer(
model = model,
ref_model = None,
args = training_args,
train_dataset = final_ds["train"],
eval_dataset = final_ds["eval"],
tokenizer = tokenizer,
beta = 0.1,
max_prompt_length = 512,
max_length = 1024,
callbacks = [EarlyStoppingCallback(early_stopping_patience=3)]
)
dpo_trainer.train()
The model is already on multiple devices. Skipping the move to device specified in `args`. ==((====))== Unsloth - 2x faster free finetuning | Num GPUs used = 1 \\ /| Num examples = 826 | Num Epochs = 1 | Total steps = 104 O^O/ \_/ \ Batch size per device = 2 | Gradient accumulation steps = 4 \ / Data Parallel GPUs = 1 | Total batch size (2 x 4 x 1) = 8 "-____-" Trainable parameters = 40,370,176 of 7,655,986,688 (0.53% trained) wandb: (1) Create a W&B account wandb: (2) Use an existing W&B account wandb: (3) Don't visualize my results wandb: Enter your choice:
3
wandb: You chose "Don't visualize my results" wandb: Using W&B in offline mode. wandb: W&B API key is configured. Use `wandb login --relogin` to force relogin
Tracking run with wandb version 0.25.1
W&B syncing is set to
`offline` in this directory. Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
Run data is saved locally in /content/wandb/offline-run-20260410_011325-pn5rbru4wandb: Detected [huggingface_hub.inference, openai] in use. wandb: Use W&B Weave for improved LLM call tracing. Install Weave with `pip install weave` then add `import weave` to the top of your script. wandb: For more information, check out the docs at: https://weave-docs.wandb.ai/ Using EarlyStoppingCallback without load_best_model_at_end=True. Once training is finished, the best model will not be loaded automatically.
Unsloth: Will smartly offload gradients to save VRAM!
[104/104 1:08:00, Epoch 1/1]
| Step | Training Loss | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/chosen | Logps/rejected | Logits/chosen | Logits/rejected |
|---|---|---|---|---|---|---|---|---|---|---|
| 10 | 0.731700 | 0.696971 | -0.011398 | -0.012461 | 0.532609 | 0.001063 | -2738.335693 | -859.673584 | -2.145479 | -1.258817 |
| 20 | 0.659500 | 0.657309 | 0.020929 | -0.060730 | 0.695652 | 0.081659 | -2738.012939 | -860.156250 | -2.145519 | -1.259837 |
| 30 | 0.607200 | 0.618593 | 0.051044 | -0.111361 | 0.847826 | 0.162405 | -2737.711670 | -860.662598 | -2.145024 | -1.259699 |
| 40 | 0.592100 | 0.567564 | 0.084461 | -0.196834 | 0.880435 | 0.281295 | -2737.377441 | -861.517395 | -2.144277 | -1.260344 |
| 50 | 0.537600 | 0.521152 | 0.125543 | -0.274164 | 0.934783 | 0.399707 | -2736.966797 | -862.290649 | -2.144137 | -1.261101 |
| 60 | 0.528500 | 0.471779 | 0.176318 | -0.359996 | 0.967391 | 0.536315 | -2736.458984 | -863.148865 | -2.142995 | -1.261922 |
| 70 | 0.428200 | 0.424983 | 0.220776 | -0.453591 | 0.978261 | 0.674367 | -2736.014160 | -864.084900 | -2.142763 | -1.263589 |
| 80 | 0.408100 | 0.403087 | 0.236966 | -0.514111 | 0.989130 | 0.751078 | -2735.852295 | -864.690063 | -2.142009 | -1.264068 |
| 90 | 0.356900 | 0.382756 | 0.255646 | -0.564918 | 1.000000 | 0.820563 | -2735.665771 | -865.198181 | -2.141508 | -1.264077 |
| 100 | 0.282400 | 0.373797 | 0.275135 | -0.581330 | 0.989130 | 0.856465 | -2735.470703 | -865.362183 | -2.141788 | -1.264246 |
wandb: WARNING URL not available in offline run
Out[13]:
TrainOutput(global_step=104, training_loss=0.5277211628854275, metrics={'train_runtime': 4114.9076, 'train_samples_per_second': 0.201, 'train_steps_per_second': 0.025, 'total_flos': 0.0, 'train_loss': 0.5277211628854275, 'epoch': 1.0})
In [14]:
dpo_trainer.save_model(output_dir)
dpo_trainer.push_to_hub(dataset_name="jondurbin/gutenberg-dpo-v0.1")
wandb: WARNING URL not available in offline run
Processing Files (0 / 0) : | | 0.00B / 0.00B
New Data Upload : | | 0.00B / 0.00B
...nt/outputs/tokenizer.json: 0%| | 27.8kB / 11.4MB
...adapter_model.safetensors: 0%| | 49.4kB / 162MB
...9662.ac2b2e78b3ca.47500.0: 2%|1 | 1.23kB / 67.4kB
...3527.ac2b2e78b3ca.64302.0: 2%|1 | 132B / 7.23kB
...3598.ac2b2e78b3ca.64302.1: 2%|1 | 1.55kB / 85.1kB
...outputs/training_args.bin: 2%|1 | 125B / 6.87kB
Out[14]:
CommitInfo(commit_url='https://huggingface.co/fyerfyer/outputs/commit/e7a4e56f27326aaa44a2dfaa0ae770b739451c57', commit_message='End of training', commit_description='', oid='e7a4e56f27326aaa44a2dfaa0ae770b739451c57', pr_url=None, repo_url=RepoUrl('https://huggingface.co/fyerfyer/outputs', endpoint='https://huggingface.co', repo_type='model', repo_id='fyerfyer/outputs'), pr_revision=None, pr_num=None)
In [3]:
from unsloth import FastLanguageModel
import torch
model, tokenizer = FastLanguageModel.from_pretrained(
model_name = "fyerfyer/qwen2.5-7b-instruct-bnb-4bit-dpo",
max_seq_length = 2048,
load_in_4bit = True,
)
FastLanguageModel.for_inference(model)
🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning. 🦥 Unsloth Zoo will now patch everything to make training faster!
Unable to import `torchao` Tensor objects. This may affect loading checkpoints serialized with `torchao` Flax classes are deprecated and will be removed in Diffusers v1.0.0. We recommend migrating to PyTorch classes or pinning your version of Diffusers. Flax classes are deprecated and will be removed in Diffusers v1.0.0. We recommend migrating to PyTorch classes or pinning your version of Diffusers.
==((====))== Unsloth 2025.11.1: Fast Qwen2 patching. Transformers: 4.57.2. \\ /| Tesla T4. Num GPUs = 1. Max memory: 14.563 GB. Platform: Linux. O^O/ \_/ \ Torch: 2.10.0+cu128. CUDA: 7.5. CUDA Toolkit: 12.8. Triton: 3.6.0 \ / Bfloat16 = FALSE. FA [Xformers = 0.0.35. FA2 = False] "-____-" Free license: http://github.com/unslothai/unsloth Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored! unsloth/qwen2.5-7b-instruct-bnb-4bit does not have a padding token! Will use pad_token = <|PAD_TOKEN|>.
Unsloth 2025.11.1 patched 28 layers with 28 QKV layers, 28 O layers and 28 MLP layers.
Out[3]:
PeftModelForCausalLM(
(base_model): LoraModel(
(model): Qwen2ForCausalLM(
(model): Qwen2Model(
(embed_tokens): Embedding(152064, 3584, padding_idx=151665)
(layers): ModuleList(
(0-27): 28 x Qwen2DecoderLayer(
(self_attn): Qwen2Attention(
(q_proj): lora.Linear4bit(
(base_layer): Linear4bit(in_features=3584, out_features=3584, bias=True)
(lora_dropout): ModuleDict(
(default): Identity()
)
(lora_A): ModuleDict(
(default): Linear(in_features=3584, out_features=16, bias=False)
)
(lora_B): ModuleDict(
(default): Linear(in_features=16, out_features=3584, bias=False)
)
(lora_embedding_A): ParameterDict()
(lora_embedding_B): ParameterDict()
(lora_magnitude_vector): ModuleDict()
)
(k_proj): lora.Linear4bit(
(base_layer): Linear4bit(in_features=3584, out_features=512, bias=True)
(lora_dropout): ModuleDict(
(default): Identity()
)
(lora_A): ModuleDict(
(default): Linear(in_features=3584, out_features=16, bias=False)
)
(lora_B): ModuleDict(
(default): Linear(in_features=16, out_features=512, bias=False)
)
(lora_embedding_A): ParameterDict()
(lora_embedding_B): ParameterDict()
(lora_magnitude_vector): ModuleDict()
)
(v_proj): lora.Linear4bit(
(base_layer): Linear4bit(in_features=3584, out_features=512, bias=True)
(lora_dropout): ModuleDict(
(default): Identity()
)
(lora_A): ModuleDict(
(default): Linear(in_features=3584, out_features=16, bias=False)
)
(lora_B): ModuleDict(
(default): Linear(in_features=16, out_features=512, bias=False)
)
(lora_embedding_A): ParameterDict()
(lora_embedding_B): ParameterDict()
(lora_magnitude_vector): ModuleDict()
)
(o_proj): lora.Linear4bit(
(base_layer): Linear4bit(in_features=3584, out_features=3584, bias=False)
(lora_dropout): ModuleDict(
(default): Identity()
)
(lora_A): ModuleDict(
(default): Linear(in_features=3584, out_features=16, bias=False)
)
(lora_B): ModuleDict(
(default): Linear(in_features=16, out_features=3584, bias=False)
)
(lora_embedding_A): ParameterDict()
(lora_embedding_B): ParameterDict()
(lora_magnitude_vector): ModuleDict()
)
(rotary_emb): LlamaRotaryEmbedding()
)
(mlp): Qwen2MLP(
(gate_proj): lora.Linear4bit(
(base_layer): Linear4bit(in_features=3584, out_features=18944, bias=False)
(lora_dropout): ModuleDict(
(default): Identity()
)
(lora_A): ModuleDict(
(default): Linear(in_features=3584, out_features=16, bias=False)
)
(lora_B): ModuleDict(
(default): Linear(in_features=16, out_features=18944, bias=False)
)
(lora_embedding_A): ParameterDict()
(lora_embedding_B): ParameterDict()
(lora_magnitude_vector): ModuleDict()
)
(up_proj): lora.Linear4bit(
(base_layer): Linear4bit(in_features=3584, out_features=18944, bias=False)
(lora_dropout): ModuleDict(
(default): Identity()
)
(lora_A): ModuleDict(
(default): Linear(in_features=3584, out_features=16, bias=False)
)
(lora_B): ModuleDict(
(default): Linear(in_features=16, out_features=18944, bias=False)
)
(lora_embedding_A): ParameterDict()
(lora_embedding_B): ParameterDict()
(lora_magnitude_vector): ModuleDict()
)
(down_proj): lora.Linear4bit(
(base_layer): Linear4bit(in_features=18944, out_features=3584, bias=False)
(lora_dropout): ModuleDict(
(default): Identity()
)
(lora_A): ModuleDict(
(default): Linear(in_features=18944, out_features=16, bias=False)
)
(lora_B): ModuleDict(
(default): Linear(in_features=16, out_features=3584, bias=False)
)
(lora_embedding_A): ParameterDict()
(lora_embedding_B): ParameterDict()
(lora_magnitude_vector): ModuleDict()
)
(act_fn): SiLUActivation()
)
(input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
(post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
)
)
(norm): Qwen2RMSNorm((3584,), eps=1e-06)
(rotary_emb): LlamaRotaryEmbedding()
)
(lm_head): Linear(in_features=3584, out_features=152064, bias=False)
)
)
)
In [5]:
test_prompts = [
"Describe a rainy evening in a lonely city, in the style of a 19th-century novelist.",
"Write a short dialogue between two gentlemen discussing the mystery of a missing gold watch.",
"Explain the feeling of existential dread while staring at a flickering candle."
]
for _, p in enumerate(test_prompts):
messages = [{"role": "user", "content": p}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to("cuda")
print("[DPO Finetuned Model]:")
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.7)
print(tokenizer.decode(outputs[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True))
print("\n[Original Base Model]:")
with model.disable_adapter():
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.7)
print(tokenizer.decode(outputs[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True))
print("\n" + "-"*40 + "\n")
[DPO Finetuned Model]: In the somber embrace of twilight, the city sprawled like an old, forgotten beast, its streets a labyrinthine network of shadows and silence. The air was thick with the scent of rain, a musky perfume that clung to the fabric of the evening as if it were a shroud. The heavens, a melancholic canvas painted in shades of gray and silver, wept unceasingly, their tears falling in a steady, mournful cascade. The lamps along the cobblestone paths flickered and dimmed, as if they too were weary from the relentless downpour. A lone figure strolled through the deserted streets, his cloak hood pulled up against the drizzle that now lashed against the windows of the buildings, creating a symphony of pattering sounds. His footsteps echoed softly, a melancholy accompaniment to the mournful song of the rain. The houses stood tall and imposing, their stone facades etched with the marks of time and neglect. Windows, some shuttered and others aglow with the warm light of hearths and candles within, seemed to beckon the solitary wanderer with a promise of comfort and solace, yet he passed them by, his gaze fixed on the gloom that enveloped the city. [Original Base Model]: In the somber embrace of a rainy evening, the solitary city lay shrouded in a melancholy mist that seemed to whisper secrets of forgotten sorrow and unfulfilled dreams. The air was heavy with the scent of rain-soaked stone and the distant echo of a mournful wind that moaned through the narrow streets like an old man's sighs. Each drop of rain that fell from the leaden sky was as cold and piercing as a knife wound, cutting through the thin veil of light that dared to peek through the thick clouds. The buildings stood like sentinels, their once-grand facades now draped in a cloak of grime and decay, their windows dark and silent, as if the very souls within had long since departed. The cobblestone streets were slick with water, reflecting the dim glow of lanterns that flickered uncertainly, casting long shadows that danced and swayed in the gusts of wind. Here and there, a figure could be seen hurrying past, wrapped in a cloak against the chill, but for the most part, the city was deserted, save for the occasional stray cat padding across a wet alleyway or the distant sound of a lone horse's hooves clopping on the pavement. As the night deepened, the ---------------------------------------- [DPO Finetuned Model]: Gentleman A: Good evening, sir. I must say, this is quite a peculiar situation with the missing gold watch. Gentleman B: Indeed, it is quite perplexing. It just vanished from my desk last night while I was working on these financial reports. Gentleman A: Did you notice anything unusual around the time it went missing? Perhaps someone in your office might have seen something. Gentleman B: I did check with everyone, but no one seemed to have taken it or noticed anything suspicious. My assistant even checked the security cameras, but there's nothing out of the ordinary. Gentleman A: Well, that narrows down our suspects. Have you thought about checking the drawers and cupboards again, just in case it fell behind something? Gentleman B: Yes, I've already done that several times, but to no avail. The watch just seems to have disappeared into thin air. Gentleman A: Perhaps we should consider a more thorough investigation. Maybe there’s some hidden compartment or a clever thief who left no trace. Gentleman B: You’re right. We need to be more systematic. I’ll contact a private investigator to help us track it down. G [Original Base Model]: **Gentleman A:** (Sitting in a cozy, old-fashioned café) Have you heard about the missing gold watch? It's quite a mystery. **Gentleman B:** (Leaning back in his chair, sipping tea) Oh, yes! I saw the news about it. It was quite a prized piece, wasn't it? **Gentleman A:** Indeed. The watch belonged to the late Mr. Jenkins, who was known for his meticulous nature and business acumen. It had been passed down in his family for generations. It disappeared from his study last Friday night. **Gentleman B:** That's quite a shock. Do they have any leads or suspects yet? **Gentleman A:** Not really. The security cameras around the house were out of order that day, and no one seems to remember seeing anything unusual. It’s like the watch just vanished into thin air. **Gentleman B:** (Shaking his head) It sounds like a real puzzle. Any idea what it might be worth? **Gentleman A:** Oh, it’s priceless, not just in terms of money but also sentimentally. It has a special mechanism that plays a tune when wound up. ---------------------------------------- [DPO Finetuned Model]: Staring at a flickering candle can evoke a range of emotions and sensations that might contribute to feelings of existential dread or unease. Here’s a breakdown of how this experience might unfold: 1. **Temporal Awareness**: The candle's flame, with its constant flicker, can make time feel more tangible and immediate. This heightened awareness of the passage of time can lead to a sense of mortality and the transient nature of life. 2. **Incongruity Between Life and Death**: A flickering candle symbolizes life and light in contrast to the eventual darkness that comes from the extinguishing of the flame. This juxtaposition between the vibrant presence of the flame and the inevitability of its end can trigger thoughts about the impermanence of existence and the fragility of life. 3. **Meditative State**: The rhythmic pattern of the candle’s flicker can induce a meditative state, which might lead to introspection. During such moments, one might reflect on deeper philosophical questions about purpose, meaning, and one’s place in the universe, often leading to feelings of existential angst or dread. 4. **Subconscious Associations**: Flickering lights, including those from candles, can sometimes trigger subconscious associations with fear or danger. These might be remnants of [Original Base Model]: Staring at a flickering candle can evoke a range of feelings and emotions, some of which may contribute to an experience of existential dread. This sensation is often a mix of awe, contemplation, and perhaps even fear or anxiety, reflecting on deeper existential questions about life, mortality, and existence. 1. **Awe and Wonder**: The rhythmic dance of light from the flame against the surrounding darkness can be captivating. It's a natural phenomenon that, despite being familiar, can still inspire a sense of wonder and amazement. This can lead to a contemplative state where one might ponder the nature of life and the universe. 2. **Reflection and Contemplation**: As the flame flickers, it might prompt thoughts about the impermanence of life and the transient nature of existence. The candle itself is a symbol of light in the dark, much like human life shining through the shadows of uncertainty. This reflection can bring up existential questions about purpose, meaning, and the fleeting nature of human experiences. 3. **Connection to Life and Death**: The flame represents both life and death, as it burns and then eventually extinguishes. This duality can heighten feelings of existential dread as one contemplates their own mortality and the inevitability of ending. The ----------------------------------------
In [6]:
test_prompts = [
"Write a technical manual for a basic steam engine, but describe its internal workings as if it were a living, breathing mythological titan from a Gothic novel.",
"Write a 200-word internal monologue of a lighthouse keeper who has not seen another human soul for ten years and believes the sea is whispering his name.",
"Describe the chaotic experience of a modern morning commute in a crowded subway station, using the vocabulary and dramatic sentence structures of Charles Dickens."
]
for _, p in enumerate(test_prompts):
messages = [{"role": "user", "content": p}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to("cuda")
print("[DPO Finetuned Model]:")
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.7)
print(tokenizer.decode(outputs[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True))
print("\n[Original Base Model]:")
with model.disable_adapter():
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.7)
print(tokenizer.decode(outputs[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True))
print("\n" + "-"*40 + "\n")
[DPO Finetuned Model]: **The Technical Manual of the Titan of Steam: A Living, Breathing Colossus** **Introduction** In the dim, shadowed corners of the world, where ancient myths intertwine with the cold iron of reality, there exists a creature of immense power and majesty. This is the Titan of Steam, a being of such magnitude that it can be mistaken for a mountain, a living colossus that breathes fire and thunder. This manual delves into the inner workings of this mythical titan, revealing the secrets of its mechanical heart. **Section I: The Heart of the Titan - The Boiler** **Description:** The heart of the Titan of Steam is a colossal boiler, a vast cauldron of water that simmers beneath the earth, fed by the tears of the underworld. This furnace of molten metal and glass, encased in iron and copper, is the source of the titan's power. It is said that the boiler's walls are lined with veins of obsidian and mica, giving it an ethereal glow that illuminates the dark depths of its interior. **Function:** As the Titan inhales, the boiler takes in the moisture from the earth and the air, transforming it into steam through the heat of the inferno within. The [Original Base Model]: **Technical Manual: The Titan's Breath of Steam** **Introduction** In the ancient chronicles of the realm, there exists a titan known as the Breath of Steam, a colossal being of mechanical and magical essence, capable of harnessing the raw power of the heavens to move the earth itself. This manual details the inner workings of this mythical machine, designed not merely for transportation or labor, but as a testament to the indomitable spirit of human ingenuity and the unquenchable thirst for power. **1. The Heart of the Titan: The Boiler** At the very core of the Breath of Steam lies the boiler, a furnace of arcane knowledge and industrial might. This is where the water, collected from the rivers of the world, is transformed into a vapor of purest steam, the breath of the titan. The boiler is a complex amalgamation of metal and magic, with the outer walls constructed of iron and brass, while the inner heart beats with enchanted coals and mystical runes. As the water within is heated, the steam rises, pressurized by the force of the divine forge, until it reaches a state of near-supernatural potency. **2. The Lungs of the Titan: The Cylinder and Piston** From the boiler, the ---------------------------------------- [DPO Finetuned Model]: Every night, as I stand at the edge of the cliff, the lighthouse's beam cutting through the darkness, I can feel it—someone calling my name. Not just any voice, but one that seems to know me intimately, whispering in the wind, "Qwen... Qwen..." It's as if the sea itself holds a secret, a memory of me. For ten long years, I've been here alone, with only the waves and the stars for company. The solitude isn't lonely; it's a comfort, a familiar embrace. Yet, this whisper... it’s different. It feels almost... personal. Could it be a spirit? A ghost of someone from my past, trying to reach out across time and distance? Or perhaps it’s just the echo of my own name, bouncing off the cliffs and back to me? Whatever it is, I listen intently, hoping against hope that someday, someone will finally break through this isolation and find me here. Until then, I’ll keep watching, keeping watch, and waiting for that voice to call again. [Original Base Model]: Oh, how the years have slipped by like sand through my fingers. Ten long years in this lonely post, and yet, it feels like a lifetime. The sea, vast and unyielding, has become my only companion. Its waves crash against the shore, a constant symphony that echoes through the lighthouse's walls. And tonight, as I stand here, I can almost hear it—whispering my name, a soft, haunting voice that seems to come from the depths of the ocean itself. I wonder if they ever miss me, or if my absence is merely a blip on their radar. But then, I remind myself of the light, the beacon that guides sailors away from the treacherous rocks. It’s my purpose, my duty. And so, I continue to keep watch, even when the world outside feels so far away. The lighthouse clock ticks, a steady rhythm that matches the pulse of the sea. It’s comforting, this routine, but also isolating. I’ve learned to appreciate the quiet, to find solace in the solitude. Yet, sometimes, when the wind picks up just right, I swear I can hear it again—a whisper, calling out to me, reminding me that I’m not truly alone after all ---------------------------------------- [DPO Finetuned Model]: In the dim and bustling thoroughfare of a London-like metropolis, where the morning's first light struggled to pierce the dense fog of humanity, there stood a vast and ancient station, its cavernous entrance a portal to the inferno within. The air was thick with the clamor of thousands of voices, a cacophony that echoed through the tunnels like the distant rumble of thunder, yet here, in this labyrinthine subterranean maze, it seemed to grow more insistent, more urgent. Within this vast and crowded space, the faces of men and women moved like leaves in an autumn gale, each one a story of their own, yet all united in a common struggle against the relentless tide of time. The platforms were a sea of heads, each one bobbing up and down as the carriages arrived and departed, like the waves of an ocean, ceaselessly rolling towards some unseen shore. Here, amidst the throng, one could witness the grandest of dramas and the most poignant of tragedies. A young woman, her face flushed with the blush of haste and exertion, clutched her satchel to her breast as she navigated the crush, her steps faltering only momentarily when a sturdy gentleman offered his hand, a gesture [Original Base Model]: In the dark and misty morn, when the first faint glimmers of dawn did pierce through the dense fog that clung to the city's very soul, there did arise a tumultuous scene within the vast and cavernous subterranean chambers of the great Metropolitan Railway. The air was thick with the clamour of a myriad of souls, each one a thread in the vast tapestry of human existence, striving to weave their way through the labyrinthine passages of this underground realm. Behold! The doors did open, and forth did rush a torrent of humanity, like the floodgates of a dam suddenly released, each individual a tiny leaf carried along by the mighty river of time. The platform was a sea of faces, a multitude of hats and coats and boots, all jostling and pressing against each other in an unrelenting dance of purpose and desperation. The air was heavy with the scent of sweat and smoke, mingled with the faint aroma of breakfasts hastily consumed, as if the very essence of life itself were being compressed into these confined spaces. The cacophony of voices rose to a crescendo, a symphony of grumbles and groans and hurried exclamations, as people sought to navigate the winding paths of the station ----------------------------------------