Update last seen

Answered

Follow

erik.leven

• January 13, 2020 10:54

It would be nice with a way to update last seen for a pipe with multiple sources without having to specify every re-start point for every incoming pipe, i.e. update last seen for only one of the incoming pipes

0

• Official comment

  

  Geir Ove Grønmo

  • October 13, 2020 11:07

  You don't mention which source type you have in mind. The merge source is technically a black box and cannot be rewound. The merge_datasets and union_datasets sources uses a dot notation in their _updated values. You can edit those manually if you want.

  
  What is the use-case for rewinding a single source?

  Comment actions Permalink
• 

  erik.leven

  • October 13, 2020 11:39

  Yeah, this was regarding the merge source. Well, when I do set the "update last seen"-value through dot separation, starting with the first dataset in the datasets poperty of the source, it very often gives me some weird results. I'll try to show an example.

  In this case I'm merging 4 datasets. The datasets property of the source is the following:

  "datasets": ["cvpartner-customer cvc", "currenttime-company cc", "currenttime-company-extra ccx", "ad-customer ad"]

  Starting with cv-partner-customer and ending with ad-customer, their most recent _update-values are:

  629.5105.6044.44407

  If I want to re-send only entities from currenttime-company, from _updated value 5000 I would expect to write 629.5000.6044.44407 in the "update last seen" option.

  This is the result:

  

   

  In other words, nothing happens after I hit refresh. I often find it hard to correctly do this for merge sources, and quite often I either get no result or the source resend entities I did not specify. That's why I was wondering if there could be a way to specify one dataset individually, instead of marking down all of the, This can also be quite tedious when there are 20 datasets feeding into the source.

  0

  Comment actions Permalink
• 

  Geir Ove Grønmo

  • October 16, 2020 11:48

  Do you mean "merge_datasets"? The "merge" source does not have _updated values like 629.5105.6044.44407.

  0

  Comment actions Permalink
• 

  Geir Ove Grønmo

  • June 09, 2021 07:54

  The "merge" source does not support rewinding; well technically it does, but it makes no attempt to change the merging part itself. It just means that the *already merge* entities will come out of the source once more and run through the transforms and into the sink. You cannot use rewinding to reapply equality rules. Only resetting the pipe will do that.
  
  The "merge_datasets" source does support rewinding like the one you showed (where you rewind one of the offsets in the composite since value).

  0

  Comment actions Permalink

Please sign in to leave a comment.