[Toaster] Errors with composite/multiple date filters (6040)

[Reyna, David]

Hi Alex,

I have looked at the date filtering problem up close, and here is my results and recommendations. This issue is blocking 6040.

1) The "__lte" does not do the equal, it just does "__lt". That is why for "yesterday" you have to do "started_on__gte;Bstarted_on__lt" of "<day>;<day+1>".

RECOMMENDATION: I can just use "__lt" for now and adjust the end date accordingly.

2) The multiple filter function of "_get_filtering_query" does not work correctly, whereas if you do the filters as separate operations you get the expected results.

Here is a an example of what I found. I ran a set of date range URLs with the existing mutiple filters and then with a routine that applies the filters separately.

My "Started on" dates are these: 4/4/14, 6/4/14, 7/4/14

Here is what happens, where the invalid results have an 'x':

  filter=started_on__gte%3Bstarted_on__lt%3A2014-04-04%3B2014-04-05
  Filter Together: 4/4/14, x6/4/14, x7/4/14
  Filter Separate: 4/4/14

  filter=started_on__gte%3Bstarted_on__lt%3A2014-04-05%3B2014-04-06
  Filter Together: x6/4/14, x7/4/14
  Filter Separate: <none>

  filter=started_on__gte%3Bstarted_on__lt%3A2014-04-06%3B2014-04-07
  Filter Together: 6/4/14, x7/4/14
  Filter Separate: 6/4/14

  filter=started_on__gte%3Bstarted_on__lt%3A2014-04-07%3B2014-04-08
  Filter Together: 7/4/14
  Filter Separate: 7/4/14

  filter=started_on__gte%3Bstarted_on__lt%3A2014-04-04%3B2014-04-07
  Filter Together: 4/4/14, 6/4/14, x7/4/14
  Filter Separate: 4/4/14, 6/4/14

  filter=started_on__gte%3Bstarted_on__lt%3A2014-04-04%3B2014-04-08
  Filter Together: 4/4/14, 6/4/14, 7/4/14
  Filter Separate: 4/4/14, 6/4/14, 7/4/14

In all cases the separate filter application gave the expected answer, where the multiple filter often did not.

RECOMMENDATION: Until we can debug and fix "_get_filtering_query", change it for now to execute the filter queries separately, and I can rebase on that. Currently only the data range filters use the multiple filter syntax, so the impact is limited. If you do not want to change this routine, then I have my work around listed below.

Here is my sample routine for executing the separate filtering:

def date_filtering_query(queryset, filter_string):
    search_terms = filter_string.split(FIELD_SEPARATOR)
    keys = search_terms[0].split(VALUE_SEPARATOR)
    values = search_terms[1].split(VALUE_SEPARATOR)
    for i in range(len(keys)):
                filter_query = _get_filtering_query(keys[i]+":"+values[i])
                queryset = queryset.filter(filter_query)
                _log("FILTER_COUNT="+str(queryset.count()))
    return queryset

I called it this way in my 'build' view class.

    ...
    if True and (filter_string.startswith('started_on') or filter_string.startswith('completed_on')):
        queryset = _get_queryset(Build, queryset_all, '', search_term, ordering_string, '-completed_on')
        queryset = date_filtering_query(queryset,filter_string)
    else:
        queryset = _get_queryset(Build, queryset_all, filter_string, search_term, ordering_string, '-completed_on')
    ...

- David


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/toaster/attachments/20140408/09db47ab/attachment.html>