Unable to remove tasks from timesheets

I have timesheets not tasks.
I have a lot of rows when i open the timesheets.
They are tasks.
I want to delete certain tasks.
I am able to delete only one or two tasks.
I am not able to delete other tasks.
Both the types of the tasks have remaining work.
Please suggest.
The issue is Remove Tasks Button goes grey.I have tried to recall the timesheet and then delete but it still does not work.