Add support for deleting datasets in the internal schedulerAnswered
In the old sesam test client (https://github.com/sesam-io/sesam) we use the community scheduler to run the configuration. During a run with that scheduler, the default config is to delete the datasets on the node.
For more details on the old client see:
The internal scheduler (the "/pipes/run-all-pipes" endpoint) doesn't seem to have that option. We would like to benefit from the faster runs with the internal scheduler when doing CI, but since this feature is missing, we have to rely on doing a full wipe before every new test in order to get consistent behaviour. This is not ideal, since the wipe is a quite costly operation and adds a lot of unnecessary load on the node.
We have recently pushed a change that causes the sink dataset to be deleted automatically when the pipe is removed (together will all other pipe state). Any orphaned datasets are also being removed when the node is started. Is this enough to solve your issues?Comment actions
Please sign in to leave a comment.