Add support for deleting datasets in the internal scheduler

Answered

Follow

anders.borud

• May 11, 2020 06:31
• Edited

In the old sesam test client (https://github.com/sesam-io/sesam) we use the community scheduler to run the configuration. During a run with that scheduler, the default config is to delete the datasets on the node.

For more details on the old client see:
https://github.com/sesam-io/sesam/blob/master/main.go#L239
https://github.com/sesam-community/scheduler/blob/master/service/scheduler.py#L265

The internal scheduler (the "/pipes/run-all-pipes" endpoint) doesn't seem to have that option. We would like to benefit from the faster runs with the internal scheduler when doing CI, but since this feature is missing, we have to rely on doing a full wipe before every new test in order to get consistent behaviour. This is not ideal, since the wipe is a quite costly operation and adds a lot of unnecessary load on the node.

5

• Official comment

  

  Geir Ove Grønmo

  • September 29, 2020 13:16

  We have recently pushed a change that causes the sink dataset to be deleted automatically when the pipe is removed (together will all other pipe state). Any orphaned datasets are also being removed when the node is started. Is this enough to solve your issues?

  Comment actions Permalink
• 

  anders.borud

  • May 15, 2020 10:55

  This is also relevant in regards to completely replacing the old client with the new one. As the default behaviour is not the same, some teams are reluctant to do the switch.

  1

  Comment actions Permalink

Please sign in to leave a comment.