[Toaster] [review-request] 6040 and 7249: daterange filtering for builds page
Reyna, David
david.reyna at windriver.com
Thu Mar 19 15:43:27 PDT 2015
Hi Michael,
Thank you for your review.
I will of course take most of your suggestions. Some I recommend differing for the reasons I give inline.
- David
> -----Original Message-----
> From: Michael Wood [mailto:michael.g.wood at intel.com]
> Sent: Thursday, March 19, 2015 11:31 AM
> To: Reyna, David; BARROS PENA, BELEN; DAMIAN, ALEXANDRU
> Cc: toaster at yoctoproject.org
> Subject: Re: [Toaster] [review-request] 6040 and 7249: daterange filtering
> for builds page
>
> Review in-line
>
> On 19/03/15 14:48, Michael Wood wrote:
> > From: David Reyna <David.Reyna at windriver.com>
> >
> > Enable date range selections for build start and build complete in
> > all builds page, managed and interactive.
> >
> > [YOCTO #6040]
> > [YOCTO #7249]
> >
> > Signed-off-by: David Reyna <David.Reyna at windriver.com>
> > ---
> > .../lib/toaster/toastergui/templates/build.html | 18 +++
> > .../toastergui/templates/filtersnippet.html | 69 +++++++-
> > .../toastergui/templates/managed_builds.html | 18 +++
> > .../toaster/toastergui/templatetags/projecttags.py | 4 +
> > bitbake/lib/toaster/toastergui/views.py | 173
> ++++++++++++++++++---
> > 5 files changed, 253 insertions(+), 29 deletions(-)
> >
> > diff --git a/bitbake/lib/toaster/toastergui/templates/build.html
> b/bitbake/lib/toaster/toastergui/templates/build.html
> > index 684ec65..ee0272f 100644
> > --- a/bitbake/lib/toaster/toastergui/templates/build.html
> > +++ b/bitbake/lib/toaster/toastergui/templates/build.html
> > @@ -4,7 +4,25 @@
> > {% load projecttags %}
> > {% load humanize %}
> >
> > +{% block extraheadcontent %}
> > +<link rel="stylesheet" href="/static/css/jquery-ui.min.css"
> type='text/css'>
> > +<link rel="stylesheet" href="/static/css/jquery-ui.structure.min.css"
> type='text/css'>
> > +<link rel="stylesheet" href="/static/css/jquery-ui.theme.min.css"
> type='text/css'>
> > +<script src="/static/js/jquery-ui.min.js"></script>
> > +{% endblock %}
> > +
> > {% block pagecontent %}
> > +
> > +<script>
> > + // intiialize the date range controls
> > + $(document).ready(function () {
> > + date_init_started_on();
> > + date_init_completed_on();
> > + date_range_init_started_on({%if daterange_filter
> %}'enable'{%else%}'disable'{%endif%});
> > + date_range_init_completed_on({%if daterange_filter
> %}'enable'{%else%}'disable'{%endif%});
> > + });
> > +</script>
> > +
>
> We can put initialisation of the datepicker in libtoaster with a special
> class as if we have a date field we will want this date picker across
> toaster.
[DLR] Yes we can. The reasons I did it this way for this pass are:
a) I wanted to be explicit what objects were being initialized, and insure that it happened at the top of the affected page(s)
b) The range names are different for the managed versus the non-managed page, plus those names are overloaded with the database filter element name, so we are limited in how we set up a generic class.
c) This complexity (e.g. ("{%if daterange_filter%}") is in part related to how the date input field and the radio buttons are overloaded in "filtersnippet.html".
I recommend that this optimization over all datetimes we defer to 1.9.
> > <div class="row-fluid">
> >
> > {% include "mrb_section.html" %}
> > diff --git a/bitbake/lib/toaster/toastergui/templates/filtersnippet.html
> b/bitbake/lib/toaster/toastergui/templates/filtersnippet.html
> > index fe70e71..703be88 100644
> > --- a/bitbake/lib/toaster/toastergui/templates/filtersnippet.html
> > +++ b/bitbake/lib/toaster/toastergui/templates/filtersnippet.html
> > @@ -16,16 +16,72 @@
> > <input type="radio" name="filter" {%if
> request.GET.filter%}{{f.options|check_filter_status:request.GET.filter}}
> {%else%} checked {%endif%} value=""> All {%if
> filter_search_display%}{{filter_search_display|title}}{%else%}{{objectname|t
> itle}}{%endif%} ({{total_count}})
> > </label>
> > {% for option in f.options %}
>
> > - {% if option.2 %}
> > - <label class="radio">
> > - <input type="radio" name="filter" {%if
> request.GET.filter == option.1 %}checked{%endif%} value="{{option.1}}">
> {{option.0}} (<span id="{{option.1}}_count">{{option.2}}</span>)
> > + {% if option.1 == 'daterange' %}
> > + <div class="form-inline">
> > + <label class="radio">
> > + <input type="radio" name="filter" {%if
> daterange_filter %}checked{%endif%} value="{{option.1}}"> {{option.0}}
> > {% else %}
> > - <label class="radio muted">
> > - <input type="radio" name="filter" disabled {%if
> request.GET.filter == option.1 %}checked{%endif%} value="{{option.1}}">
> {{option.0}} (<span id="{{option.1}}_count">{{option.2}}</span>)
> > + {% if option.2 %}
> > + <label class="radio">
> > + <input type="radio" name="filter" {%if
> request.GET.filter == option.1 %}checked{%endif%} value="{{option.1}}">
> {{option.0}} ({{option.2}})
> > + {% else %}
> > + <label class="radio muted">
> > + <input type="radio" name="filter" disabled {%if
> request.GET.filter == option.1 %}checked{%endif%} value="{{option.1}}">
> {{option.0}} ({{option.2}})
> > + {% endif %}
> > {% endif %}
> > {% if option.3 %}<i class="icon-question-sign get-help"
> data-placement="right" title="{{option.3}}"></i>{% endif %}
> > - </label>
> > + </label>
> > + {% if option.1 == 'daterange' %}
> > + <input type="text" id="date_from_{{option.4}}"
> name="filter" disabled class="input-small" /><label class="help-
> inline">to</label>
> > + <input type="text" id="date_to_{{option.4}}"
> name="date_to_{{option.4}}" disabled class="input-small" />
> > + <input type="hidden" name="date_id"
> value="{{option.4}}"/>
> > + <label class="help-inline get-help" >(dd/mm/yy)</label>
> > + </div>
> > +
> > + <script>
> This JavaScript function is inside the for loop which means it will be
> copied N times, and put into the global scope, we don't want to have
> javascript snippets across toaster as they're very difficult to debug,
> even more difficult if they're generated by template logic. A new
> javascript file is needed.
[DLR] Yes I can certainly do that for the bodies of the calls.
It was a bear to develop and debug, so it was very handy to have the code in one place. Now that it is working I can spend the day to factor it out to a separate JS file and retest.
>
>
> > + function date_range_init_{{option.4}} (action) {
> > + if (action == 'enable'){
> > +
> $("#date_from_{{option.4}},#date_to_{{option.4}}").removeAttr("disabled");
> > +
> > +
> //alert("date_range_init=ENABLE_{{option.4}}!");
> > +
> $("#date_from_{{option.4}},#date_to_{{option.4}}").datepicker();
> > +
> $("#date_from_{{option.4}},#date_to_{{option.4}}").datepicker( "option",
> "dateFormat", "dd/mm/y" );
> > +
> > + $("#date_from_{{option.4}}").datepicker(
> "setDate", "{{option.5}}" );
> > + $("#date_to_{{option.4}}" ).datepicker(
> "setDate", "{{option.6}}" );
> > + $("#date_from_{{option.4}}").datepicker(
> "option", "minDate", "{{option.7}}" );
> > + $("#date_to_{{option.4}}" ).datepicker(
> "option", "minDate", "{{option.7}}" );
> > + $("#date_from_{{option.4}}").datepicker(
> "option", "maxDate", "{{option.8}}" );
> > + $("#date_to_{{option.4}}" ).datepicker(
> "option", "maxDate", "{{option.8}}" );
> > +
> > + } else {
> > +
> $("#date_from_{{option.4}},#date_to_{{option.4}}").attr("disabled" ,
> "disabled");
> > + }
> > + }
> > +
> > + function date_init_{{option.4}} () {
> > + //enable date range input fields in Completed on
> filter
> > +
> $('input:radio[name="filter"]').change(function(){
> > + if($(this).val() == 'daterange'){
> > + date_range_init_{{option.4}}('enable');
> > + } else {
> > + date_range_init_{{option.4}}('disable');
> > + }
> > + });
> > +
> > + // use new 'from' date as minDate for 'to' dates
> > + $("#date_from_{{option.4}}").change(function(){
> > + from_date =
> $("#date_from_{{option.4}}").val();
> > + $("#date_to_{{option.4}}").datepicker(
> "option", "minDate", from_date );
> > + });
> > + };
> > + </script>
> > +
> > + {% endif %}
> > {% endfor %}
> > + <!-- daterange persistence -->
> > + <input type="hidden" name="last_date_from"
> value="{{last_date_from}}"/>
> > + <input type="hidden" name="last_date_to"
> value="{{last_date_to}}"/>
> > </div>
> > <div class="modal-footer">
> > <button type="submit" class="btn btn-primary">Apply</button>
> > @@ -36,4 +92,3 @@
> > {% endif %}
> > </div>
> > </form>
> > -
> > diff --git a/bitbake/lib/toaster/toastergui/templates/managed_builds.html
> b/bitbake/lib/toaster/toastergui/templates/managed_builds.html
> > index e23b832..ec5bbe1 100644
> > --- a/bitbake/lib/toaster/toastergui/templates/managed_builds.html
> > +++ b/bitbake/lib/toaster/toastergui/templates/managed_builds.html
> > @@ -4,7 +4,25 @@
> > {% load projecttags %}
> > {% load humanize %}
> >
> > +{% block extraheadcontent %}
> > +<link rel="stylesheet" href="/static/css/jquery-ui.min.css"
> type='text/css'>
> > +<link rel="stylesheet" href="/static/css/jquery-ui.structure.min.css"
> type='text/css'>
> > +<link rel="stylesheet" href="/static/css/jquery-ui.theme.min.css"
> type='text/css'>
> > +<script src="/static/js/jquery-ui.min.js"></script>
> > +{% endblock %}
> > +
> > {% block pagecontent %}
> > +
> > +<script>
> > + // intiialize the date range controls
> > + $(document).ready(function () {
> > + date_init_created();
> > + date_init_updated();
> > + date_range_init_created({%if daterange_filter
> %}'enable'{%else%}'disable'{%endif%});
> > + date_range_init_updated({%if daterange_filter
> %}'enable'{%else%}'disable'{%endif%});
> > + });
> > +</script>
> > +
> > <div class="row-fluid">
> >
> > {% include "managed_mrb_section.html" %}
> > diff --git a/bitbake/lib/toaster/toastergui/templatetags/projecttags.py
> b/bitbake/lib/toaster/toastergui/templatetags/projecttags.py
> > index e66910c..3a4a5df 100644
> > --- a/bitbake/lib/toaster/toastergui/templatetags/projecttags.py
> > +++ b/bitbake/lib/toaster/toastergui/templatetags/projecttags.py
> > @@ -125,6 +125,8 @@ def filtered_icon(options, filter):
> > for option in options:
> > if filter == option[1]:
> > return "btn-primary"
> > + if ('daterange' == option[1]) and filter.startswith(option[4]):
> > + return "btn-primary"
> > return ""
> >
> > @register.filter
> > @@ -134,6 +136,8 @@ def filtered_tooltip(options, filter):
> > for option in options:
> > if filter == option[1]:
> > return "Showing only %s"%option[0]
> > + if ('daterange' == option[1]) and filter.startswith(option[4]):
> > + return "Showing only %s"%option[0]
> > return ""
> >
> > @register.filter
> > diff --git a/bitbake/lib/toaster/toastergui/views.py
> b/bitbake/lib/toaster/toastergui/views.py
> > index 8034cfc..088162d 100755
> > --- a/bitbake/lib/toaster/toastergui/views.py
> > +++ b/bitbake/lib/toaster/toastergui/views.py
> > @@ -35,7 +35,7 @@ from django.core.paginator import Paginator, EmptyPage,
> PageNotAnInteger
> > from django.http import HttpResponseBadRequest, HttpResponseNotFound
> > from django.utils import timezone
> > from django.utils.html import escape
> > -from datetime import timedelta
> > +from datetime import timedelta, datetime
> > from django.utils import formats
> > from toastergui.templatetags.projecttags import json as jsonfilter
> > import json
> > @@ -276,6 +276,56 @@ def _save_parameters_cookies(response, pagesize,
> orderby, request):
> > response.set_cookie(key='orderby',
> value=html_parser.unescape(orderby), path=request.path)
> > return response
> >
> > +# date range: normalize GUI's dd/mm/yy to date object
> > +def _normalize_input_date(request,field,default):
>
> Not sure what the purpose of this is, for display you can send date
> objects and use the built in filter:
> https://docs.djangoproject.com/en/1.7/ref/templates/builtins/#date
[DLR] Yes you could, and Django recommends it, having the data in UTC+0 and using filters like this to translate to the local timezone in the templates.
However, I found that the dates stored in the Toaster database are already in the local time and not in UTC+0, so this filter does not apply. Additionally, those filters are for templates, but not python or JS.
Here are the additional problems that this routine overcomes.
1) Belén wants the edit date fields in "dd/mm/yy", so they do have to be converted at some point to and from the database format.
2) While the Datepicker will always return valid dates, the user can enter false and/or badly formed dates, so this code is used to sanitize those inputs.
> If I understand the django documentation the date objects are
> automatically localised to your timezone because we have USE_TZ enabled.
[DLR] As I discovered in actual usage here in California (UTC+7), they are and they are not.
1) Obviously, the date fields coming back in a form input are not date objects, so they need to be converted and explicitly localized.
2) More importantly, as I emailed previously about the observed symptoms and as you observe below, the "activate()" function was not yet implemented in Toaster.
> > + date_str=request.GET.get(field,'')
> > + date_str=re.sub('/', '-', date_str)
> > + # accept dd-mm-yy to d-m-yyyy
> > + match=re.match(r'(\d+)-(\d+)-([0-9]{4}|[0-9]{2})',date_str)
>
> You do not need to parse a date in this way, there is a function that
> does this, datetime.datetime.strptime
> e.g. datetime.datetime.strptime("2012-2-3", "%Y-%m-%d")
[DLR] I will look at this to insure that it also protects against invalid user-entered dates, which was my goal in this code.
> > + if match:
> > + try:
> > + year_str=match.group(3)
> > + if len(year_str) == 2:
> > + year_str = default.strftime("%Y")[0:2] + year_str
> > + # set timezone so object not 'naive'
> > + return
> datetime(int(year_str),int(match.group(2)),int(match.group(1)),0,0,0,0,defau
> lt.tzinfo)
> > + except ValueError:
> > + return default
> > + return default
> > +
> > +# extract/apply dynamic date range filter and return updated request.GET
> > +def get_date_range_request(request):
> > + if '' == request.GET.get('date_id',''):
> > + return None
> > + date_id=request.GET.get('date_id')
> > + # was the date range radio button selected?
> > + if '' == request.GET.get('date_to_' +date_id,''):
> > + return None
> > +
> > + # normalize GUI dates to database format
> > + today = timezone.localtime(timezone.now())
> > + date_from = _normalize_input_date(request,'filter'
> ,today)
> > + date_to =
> _normalize_input_date(request,'date_to_'+date_id,today)
> > + last_date_from = request.GET.get('filter')
> > + last_date_to = request.GET.get('date_to_' +date_id,'')
> > + # swap dates if manually set dates are out of order
> > + if date_to < date_from:
> > + tmp=date_to; date_to=date_from; date_from=tmp
> > + tmp=last_date_to; last_date_to=last_date_from; last_date_from=tmp
> Semi colons creeping in there again (should be on new lines)
>
> You can do a swap easily without the need for a tmp variable by doing
> a,b = b,a
> https://docs.python.org/2/reference/expressions.html#evaluation-order
Yes, I know, but some days I just do not feel like being clever.
> > + # convert to strings, adjusting 'date_to' for '__lt'
> > + date_from_str = date_from.strftime("%Y-%m-%d")
> > + date_to_str = (date_to+timedelta(days=1)).strftime("%Y-%m-%d")
> > +
> > + request.GET = request.GET.copy()
> > +
> filter_string=date_id+'__gte!'+date_id+'__lt:'+date_from_str+'!'+date_to_str
> > + request.GET['last_date_from' ]=last_date_from
> > + request.GET['last_date_to' ]=last_date_to
> > + request.GET['filter' ]=filter_string
> > + request.GET['daterange_filter']='1'
>
> Please use the code style that is used in the rest of Toaster which is
> along the lines of pep 8, this means not putting whitespace in array
> index keys like "request.GET['filter' ]" and "date_to = _norm"
>
> https://www.python.org/dev/peps/pep-0008/
Style wars :-/
> > + # block recursion
> > + del request.GET['date_id']
> > + # return the new filter page request
> > + return request.GET
> >
> If you use the date from the browser (i.e in the JavaScript) you won't
> need to fudge the request, intercepting and creating a new version for
> the request is not something we want to start doing as we could end up
> with race conditions on the request object, if it's being modified
> outside it's normal life cycle.
>
> see http://www.w3schools.com/jsref/jsref_obj_date.asp for date api
> reference.
[DLR] What you are really asking me for is to unravel the selection model of "filtersnippet.html", which forces an unholy mix of input boxes and radio buttons and the "filter" filter. That was the sole reason for this fix-up and reformatting.
I will see about implementing this requested optimization hopefully without redoing "filtersnippet.html" altogether. What that probably means is catching the send button and re-valuing the "filter" object (i.e. validating the user inputs and merging this and the end date into the complex composite filter string) before it is sent.
> > ##
> > # build dashboard for a single build, coming in as argument
> > @@ -1807,6 +1857,14 @@ if toastermain.settings.MANAGED:
> > # be able to display something. 'count' and 'page' are
> mandatory for all views
> > # that use paginators.
> >
> > + # extract/apply dynamic date range filter, if any
> > + date_range_request_GET = get_date_range_request(request)
> > + if None != date_range_request_GET:
> > + # assert the new filter page request
> > + (pagesize, orderby) = _get_parameters_values(request, 10,
> 'created:-')
> > + mandatory_parameters = { 'count': pagesize, 'page' : 1,
> 'orderby' : orderby }
> > + return _redirect_parameters( 'all-builds',
> date_range_request_GET, mandatory_parameters)
> > +
> > buildrequests = BuildRequest.objects.exclude(state__lte =
> BuildRequest.REQ_INPROGRESS).exclude(state=BuildRequest.REQ_DELETED)
> >
> > try:
> > @@ -1853,14 +1911,13 @@ if toastermain.settings.MANAGED:
> > request.GET['orderby'] = ":".join(ordering_params)
> >
> > # boilerplate code that takes a request for an object type and
> returns a queryset
> > - # for that object type. copypasta for all needed table searches
> > + # for that object type. copypaste for all needed table searches
> > (filter_string, search_term, ordering_string) =
> _search_tuple(request, BuildRequest)
> > # we don't display in-progress or deleted builds
> > queryset_all = buildrequests.exclude(state =
> BuildRequest.REQ_DELETED)
> > queryset_all = queryset_all.select_related("build",
> "build__project").annotate(Count('brerror'))
> > queryset_with_search = _get_queryset(BuildRequest, queryset_all,
> filter_string, search_term, ordering_string, '-updated')
> >
> > -
> > # retrieve the objects that will be displayed in the table;
> builds a paginator and gets a page range to display
> > build_info = _build_page_range(Paginator(queryset_with_search,
> pagesize), request.GET.get('page', 1))
> >
> > @@ -1894,7 +1951,6 @@ if toastermain.settings.MANAGED:
> > comma = ", "
> > fstypes_map[build_request.build.id]=extensions
> >
> > -
> > # send the data to the template
> > context = {
> > # specific info for
> > @@ -1905,7 +1961,7 @@ if toastermain.settings.MANAGED:
> > 'default_orderby' : 'updated:-',
> > 'fstypes' : fstypes_map,
> > 'search_term' : search_term,
> > - 'total_count' : queryset_with_search.count(),
> > + 'total_count' : queryset_all.count(),
> > # Specifies the display of columns for the table,
> appearance in "Edit columns" box, toggling default show/hide, and specifying
> filters for columns
> > 'tablecols' : [
> > {'name': 'Outcome',
> # column with a single filter
> > @@ -1944,6 +2000,24 @@ if toastermain.settings.MANAGED:
> > }
> > )
> >
> > + # calculate the exact begining of local today and yesterday
> > + today_begin = timezone.localtime(timezone.now());
> > + todays_seconds =
> int(today_begin.strftime("%H"))*60*60+int(today_begin.strftime("%M"))*60+int
> (today_begin.strftime("%S"));
> > + today_begin = today_begin-timedelta(seconds=todays_seconds);
> > + yesterday_begin = today_begin-timedelta(days=1);
>
> The beginning of the day is easier to work out by just creating a new
> date object only initialised with the d/m/y e.g.
[DLR] Ok.
> today = timezone.now()
> today_start = datetime.date(today.year, today.month, today.day)
>
> To use the timezone.localtime api we need to activate() a timezone, this
> has to come from the client otherwise it will assume (UTC / GMT) once
> activated django should "do the right thing" in terms of timezones
> everywhere, this needs a bit of experimenting as it's not very intuitive...
[DLR] Since that initialization needs to happen in one place, and that place is outside of the scope of this commit (and may indeed require experimenting as you), I would like the existing code to remain as it is for now since (a) it is working, and (b) it will continue to work once the activate() is in place, at which time this can be optimized.
> Note that you're muscle memory is adding semi colons on the end up there!
[DLR] Ok.
> > + # add daterange persistent
> > + context['last_date_from'] =
> request.GET.get('last_date_from',timezone.localtime(timezone.now()).strftime
> ("%d/%m/%y"))
> > + context['last_date_to' ] = request.GET.get('last_date_to'
> ,context['last_date_from'])
> > + # calculate the date ranges, avoid second sort for 'created'
> > + queryset_created=queryset_all.order_by('created')
> > +
> dateMinCreated=timezone.localtime(queryset_created[0].created).strftime("%d/
> %m/%y")
> > +
> dateMaxCreated=timezone.localtime(queryset_created[queryset_created.count()-
> 1].created).strftime("%d/%m/%y")
> > + dateMinUpdated=dateMinCreated
> > +
> dateMaxUpdated=timezone.localtime(queryset_created[queryset_created.count()-
> 1].created+timedelta(days=1)).strftime("%d/%m/%y")
>
> Use QuerySet first() and last() methods to get the valid selectable date
> range as it's more descriptive
[DLR] Ok.
, I'd also convert the date format in the
> template using the |date filter , objects should be kept in their data
> type right up until the moment you want to display it so that they can
> be manipulated later on using the object's api (datetime in this case)
[DLR] No, the dates are already coming out of the database in local timezone, so this would double the zone offset. You will need to fix the database first.
> > + # if dataranging, check the datagange radio button
> > + if '' != request.GET.get('daterange_filter',''):
> > + context['daterange_filter']='1'
> > +
> > context['tablecols'].append(
> > {'name': 'Started on', 'clclass': 'started_on',
> 'hidden' : 1, # this is an unchecked box, which hides the column
> > 'qhelp': "The date and time you started the build",
> > @@ -1952,9 +2026,16 @@ if toastermain.settings.MANAGED:
> > 'filter' : {'class' : 'created',
> > 'label': 'Show:',
> > 'options' : [
> > - ("Today's builds" ,
> 'created__gte:'+timezone.now().strftime("%Y-%m-%d"),
> queryset_all.filter(created__gte=timezone.now()).count()),
> > - ("Yesterday's builds",
> 'created__gte:'+(timezone.now()-timedelta(hours=24)).strftime("%Y-%m-%d"),
> queryset_all.filter(created__gte=(timezone.now()-
> timedelta(hours=24))).count()),
> > - ("This week's builds",
> 'created__gte:'+(timezone.now()-timedelta(days=7)).strftime("%Y-%m-%d"),
> queryset_all.filter(created__gte=(timezone.now()-
> timedelta(days=7))).count()),
> > + ("Today's builds" ,
> 'created__gte:'+today_begin.strftime("%Y-%m-%d"),
> queryset_all.filter(created__gte=today_begin).count()),
> > + ("Yesterday's builds",
> > +
> 'created__gte!created__lt:'
> > +
> +yesterday_begin.strftime("%Y-%m-%d")+'!'
> > +
> +today_begin.strftime("%Y-%m-%d"),
> > + queryset_all.filter(
> > +
> created__gte=yesterday_begin,
> > +
> created__lt=today_begin
> > + ).count()),
> > + ("Builds from
> "+context['last_date_from']+" to "+context['last_date_to'], 'daterange', 1,
> '', 'created', context['last_date_from'], context['last_date_to'],
> dateMinCreated, dateMaxCreated, ),
>
> I understand that the tuple/list is how it's done else where but
> unfortunately this is not correct it's not a good idea to rely on the
> fact that the date option is the 4th option, anyone changing that will
> not realise it and it will all break.
[DLR] I am just following the model that Belén set up.
We can redo this using named pairs and add extra parsing code on the client side, but I recommend that we wait until YP-1.9 since I do not want to possibly compromise all filters at this late date.
> > ]
> > }
> > }
> > @@ -1968,9 +2049,16 @@ if toastermain.settings.MANAGED:
> > 'filter' : {'class' : 'updated',
> > 'label': 'Show:',
> > 'options' : [
> > - ("Today's builds",
> 'updated__gte:'+timezone.now().strftime("%Y-%m-%d"),
> queryset_all.filter(updated__gte=timezone.now()).count()),
> > - ("Yesterday's builds",
> 'updated__gte:'+(timezone.now()-timedelta(hours=24)).strftime("%Y-%m-%d"),
> queryset_all.filter(updated__gte=(timezone.now()-
> timedelta(hours=24))).count()),
> > - ("This week's builds",
> 'updated__gte:'+(timezone.now()-timedelta(days=7)).strftime("%Y-%m-%d"),
> queryset_all.filter(updated__gte=(timezone.now()-
> timedelta(days=7))).count()),
> > + ("Today's builds" ,
> 'updated__gte:'+today_begin.strftime("%Y-%m-%d"),
> queryset_all.filter(updated__gte=today_begin).count()),
> > + ("Yesterday's builds",
> > +
> 'updated__gte!updated__lt:'
> > +
> +yesterday_begin.strftime("%Y-%m-%d")+'!'
> > +
> +today_begin.strftime("%Y-%m-%d"),
> > + queryset_all.filter(
> > +
> updated__gte=yesterday_begin,
> > +
> updated__lt=today_begin
> > + ).count()),
> > + ("Builds from
> "+context['last_date_from']+" to "+context['last_date_to'], 'daterange', 1,
> '', 'updated', context['last_date_from'], context['last_date_to'],
> dateMinUpdated, dateMaxUpdated, ),
> > ]
> > }
> > }
> > @@ -2625,7 +2713,7 @@ if toastermain.settings.MANAGED:
> > return _redirect_parameters( 'layers', request.GET,
> mandatory_parameters)
> >
> > # boilerplate code that takes a request for an object type and
> returns a queryset
> > - # for that object type. copypasta for all needed table searches
> > + # for that object type. copypaste for all needed table searches
> > (filter_string, search_term, ordering_string) =
> _search_tuple(request, Layer_Version)
> >
> > prj = Project.objects.get(pk = request.session['project_id'])
> > @@ -2838,7 +2926,7 @@ if toastermain.settings.MANAGED:
> > return _redirect_parameters( 'machines', request.GET,
> mandatory_parameters)
> >
> > # boilerplate code that takes a request for an object type and
> returns a queryset
> > - # for that object type. copypasta for all needed table searches
> > + # for that object type. copypaste for all needed table searches
> > (filter_string, search_term, ordering_string) =
> _search_tuple(request, Machine)
> >
> > queryset_all = Machine.objects.all()
> > @@ -3115,7 +3203,7 @@ if toastermain.settings.MANAGED:
> > queryset_all = Project.objects.all()
> >
> > # boilerplate code that takes a request for an object type and
> returns a queryset
> > - # for that object type. copypasta for all needed table searches
> > + # for that object type. copypaste for all needed table searches
> > (filter_string, search_term, ordering_string) =
> _search_tuple(request, Project)
> > queryset_with_search = _get_queryset(Project, queryset_all,
> None, search_term, ordering_string, '-updated')
> > queryset = _get_queryset(Project, queryset_all, filter_string,
> search_term, ordering_string, '-updated')
> > @@ -3233,8 +3321,14 @@ else:
> > if retval:
> > return _redirect_parameters( 'all-builds', request.GET,
> mandatory_parameters)
> >
> > + # extract/apply dynamic date range filter, if any
> > + date_range_request_GET = get_date_range_request(request)
> > + if None != date_range_request_GET:
> > + # assert the new filter page request
> > + return _redirect_parameters( 'all-builds',
> date_range_request_GET, mandatory_parameters)
> > +
> > # boilerplate code that takes a request for an object type and
> returns a queryset
> > - # for that object type. copypasta for all needed table searches
> > + # for that object type. copypaste for all needed table searches
> > (filter_string, search_term, ordering_string) =
> _search_tuple(request, Build)
> > queryset_all = Build.objects.exclude(outcome =
> Build.IN_PROGRESS)
> > queryset_with_search = _get_queryset(Build, queryset_all, None,
> search_term, ordering_string, '-completed_on')
> > @@ -3265,6 +3359,20 @@ else:
> > comma = ", "
> > fstypes_map[build.id]=extensions
> >
> > + # calculate the exact begining of local today and yesterday
> > + today_begin = timezone.localtime(timezone.now());
> > + todays_seconds =
> int(today_begin.strftime("%H"))*60*60+int(today_begin.strftime("%M"))*60+int
> (today_begin.strftime("%S"));
> > + today_begin = today_begin-timedelta(seconds=todays_seconds);
> > + yesterday_begin = today_begin-timedelta(days=1);
> > + last_date_from =
> request.GET.get('last_date_from',timezone.localtime(timezone.now()).strftime
> ("%d/%m/%y"))
> > + last_date_to = request.GET.get('last_date_to'
> ,last_date_from)
> > + # calculate the date ranges, avoid second sort for 'completed_on'
> > + queryset_started_on=queryset_all.order_by('started_on')
> > +
> dateMinStarted=timezone.localtime(queryset_started_on[0].started_on).strftim
> e("%d/%m/%y")
> > +
> dateMaxStarted=timezone.localtime(queryset_started_on[queryset_started_on.co
> unt()-1].started_on).strftime("%d/%m/%y")
> > + dateMinCompleted=dateMinStarted
> > +
> dateMaxCompleted=timezone.localtime(queryset_started_on[queryset_started_on.
> count()-1].started_on+timedelta(days=1)).strftime("%d/%m/%y")
> > +
> > # send the data to the template
> > context = {
> > # specific info for
> > @@ -3275,7 +3383,7 @@ else:
> > 'default_orderby' : 'completed_on:-',
> > 'fstypes' : fstypes_map,
> > 'search_term' : search_term,
> > - 'total_count' : queryset_with_search.count(),
> > + 'total_count' : queryset_all.count(),
> > # Specifies the display of columns for the table,
> appearance in "Edit columns" box, toggling default show/hide, and specifying
> filters for columns
> > 'tablecols' : [
> > {'name': 'Outcome',
> # column with a single filter
> > @@ -3312,9 +3420,16 @@ else:
> > 'filter' : {'class' : 'started_on',
> > 'label': 'Show:',
> > 'options' : [
> > - ("Today's builds" ,
> 'started_on__gte:'+timezone.now().strftime("%Y-%m-%d"),
> queryset_with_search.filter(started_on__gte=timezone.now()).count()),
> > - ("Yesterday's builds",
> 'started_on__gte:'+(timezone.now()-timedelta(hours=24)).strftime("%Y-%m-
> %d"), queryset_with_search.filter(started_on__gte=(timezone.now()-
> timedelta(hours=24))).count()),
> > - ("This week's builds",
> 'started_on__gte:'+(timezone.now()-timedelta(days=7)).strftime("%Y-%m-%d"),
> queryset_with_search.filter(started_on__gte=(timezone.now()-
> timedelta(days=7))).count()),
> > + ("Today's builds" ,
> 'started_on__gte:'+today_begin.strftime("%Y-%m-%d"),
> queryset_all.filter(started_on__gte=today_begin).count()),
> > + ("Yesterday's builds",
> > +
> 'started_on__gte!started_on__lt:'
> > +
> +yesterday_begin.strftime("%Y-%m-%d")+'!'
> > +
> +today_begin.strftime("%Y-%m-%d"),
> > + queryset_all.filter(
> > +
> started_on__gte=yesterday_begin,
> > +
> started_on__lt=today_begin
> > + ).count()),
> > + ("Builds from
> "+last_date_from+" to "+last_date_to, 'daterange', 1, '', 'started_on',
> last_date_from, last_date_to, dateMinStarted, dateMaxStarted, ),
> > ]
> > }
> > },
> > @@ -3326,9 +3441,16 @@ else:
> > 'filter' : {'class' : 'completed_on',
> > 'label': 'Show:',
> > 'options' : [
> > - ("Today's builds",
> 'completed_on__gte:'+timezone.now().strftime("%Y-%m-%d"),
> queryset_with_search.filter(completed_on__gte=timezone.now()).count()),
> > - ("Yesterday's builds",
> 'completed_on__gte:'+(timezone.now()-timedelta(hours=24)).strftime("%Y-%m-
> %d"), queryset_with_search.filter(completed_on__gte=(timezone.now()-
> timedelta(hours=24))).count()),
> > - ("This week's builds",
> 'completed_on__gte:'+(timezone.now()-timedelta(days=7)).strftime("%Y-%m-
> %d"), queryset_with_search.filter(completed_on__gte=(timezone.now()-
> timedelta(days=7))).count()),
> > + ("Today's builds" ,
> 'completed_on__gte:'+today_begin.strftime("%Y-%m-%d"),
> queryset_all.filter(completed_on__gte=today_begin).count()),
> > + ("Yesterday's builds",
> > +
> 'completed_on__gte!completed_on__lt:'
> > +
> +yesterday_begin.strftime("%Y-%m-%d")+'!'
> > +
> +today_begin.strftime("%Y-%m-%d"),
> > + queryset_all.filter(
> > +
> completed_on__gte=yesterday_begin,
> > +
> completed_on__lt=today_begin
> > + ).count()),
> > + ("Builds from
> "+last_date_from+" to "+last_date_to, 'daterange', 1, '', 'completed_on',
> last_date_from, last_date_to, dateMinCompleted, dateMaxCompleted, ),
> > ]
> > }
> > },
> > @@ -3389,6 +3511,13 @@ else:
> > ]
> > }
> >
> > + # add daterange persistent
> > + context['last_date_from'] = last_date_from
> > + context['last_date_to' ] = last_date_to
> > + # if dataranging, check the datagange radio button
> > + if '' != request.GET.get('daterange_filter',''):
> > + context['daterange_filter']='1'
> > +
> > response = render(request, template, context)
> > _save_parameters_cookies(response, pagesize, orderby, request)
> > return response
>
> All above is duplicated code, we really need to put this in it's own
> function or even better a new utility object.
>
>
> Michael
>
>
> On 17/03/15 23:25, Reyna, David wrote:
> > Hi Belén,
> > I have pushed the patch for the date ranging filtering in two commits
> > to make it easier to read, the first with just the static files from
> > jQuery-ui and the second with the actual view-model page changes. They
> > are not order dependent, but both are of course required.
> > dreyna/build_datepicker_static_6040
> > dreyna/build_datepicker_6040
> > Implementation notes:
> > 1) PENDING ISSUE: I observed that 'total_count' always uses
> > "queryset_with_search.count()", but that is wrong since that query
> > also include any filtering. The observed error is that the "all" count
> > in the filter popup will show the current limited filtered/searched
> > count and not the full count, and that just looks wrong.
> > I fixed it in the two all builds tables, but I have not touched the
> > other view classes yet. Here is my fix:
> > 'total_count' : queryset_all.count(),
> > 2) I observed that 'today' and 'yesterday' were broken for all
> > timezones outside of London, because it turns out that
> > "timezone.now()" always returns UTC+0. I added
> > "timezone.localtime(timezone.now())" to fix that, plus I added
> > explicit code for the beginning of 'today' and 'yesterday' to insure
> > everything is in sync. The database date values are of course always
> > in the local timezone, which saves filtering in the views.
> > This is not something you can test yourself, unless you have set up a
> > test host using a different time zone.
> > 3) I went ahead and fixed the typo "copypasta" with "copypaste", since
> > as yummy as it sounded it looked odd :-)
> > 4) I did a lot of testing and hair pulling (not that I had much to
> > begin with), and I have hopefully captured all of the edge cases.
> > * I did test for both managed mode and interactive mode.
> > * All illegal dates (entered by hand) are caught and internally
> > mapped to the today's date. The datepicker is of course very clean
> > with valid dates.
> > * Swapped dates (entered by hand) are indeed swapped back. If you
> > use the datepicker, the start date is automatically set as the end
> > date's minimum date, so no date swapping possible there.
> > * I did allow for single digit days and months (again entered by
> > hand), plus both 2 and 4 digit years, because it was easy to code and
> > it allows for fewer surprises for the users.
> > * I did use the jQuery UI download builder and selected only for
> > datepicker, as per my previous email. That turned out to be easy and fun!
> > * I do have page persistence for the date ranges. The
> > created/updates and starter/completed pairs use the same persistence
> > values for code economy.
> > - David
> >
> >
More information about the toaster
mailing list