How to save model output/predictions
Problem Description:
I have trained a model. Now I want to export it’s output which is type (str). How I can I save it’s output results in a dataframe or any other form that I can use for future purpose.
gf = df['findings'].astype(str)
preprocess_text = gf.str.strip().replace("n","")
t5_prepared_Text = "summarize: "+preprocess_text print ("original text preprocessed: n", preprocess_text)
tokenized_text = tokenizer.encode(str(t5_prepared_Text, return_tensors="pt").to(device)
# summmarize
summary_ids = model.generate(tokenized_text, num_beams=4, no_repeat_ngram_size=2, min_length=30, max_length=100, early_stopping=True)
output = tokenizer.decode(summary_ids[0], skip_special_tokens=True) print ("nnSummarized text: n"
Output of the model
0 summarize: There is XXXX increased opacity wit...
1 summarize: There is XXXX increased opacity wit...
2 summarize: There is XXXX increased opacity wit...
3 summarize: Interstitial markings are diffusely...
4 summarize: Interstitial markings are diffusely...
5 summarize: nan
6 summarize: nan
Name: findings, dtype: object:
So far I have tried like this
prediction = pd.DataFrame([text]).to_csv('prediction.csv')
But it saves all these rows in just one cell of the csv (first cell) and all in half form like below.
0 summarize: There is XXXX increased opacity wit...
1 summarize: There is XXXX increased opacity wit...
2 summarize: There is XXXX increased opacity wit...
3 summarize: Interstitial markings are diffusely...
4 summarize: Interstitial markings are diffusely...
5 summarize: nan
6 summarize: nan
Name: findings, dtype: object:
Solution – 1
Just replace this
prediction = pd.DataFrame([text]).to_csv('prediction.csv')
With this
prediction = pd.DataFrame([text]).to_csv('prediction.csv', sep=";")