User Acceptance Testing (UAT) Status
- Chris Luken
- Tim Garton
- Farman Siddiqi (Unlicensed)
- Scott Fiaschetti
This page provides an externally-facing summary of the current status of the Geopath API v2.2 (using the 2023 Forecast data) in the STAGING and PRODUCTION environments. The audience of this page is the GEOPATH INSIGHTS COMMITTEE and other Geopath members who are participating in User Acceptance Testing (UAT) activities.
UAT group: Please send all ‘new’ bugs to geekout@geopath.org, along with expected results, actual results, steps to reproduce, environment/URLs used, and additional business context.
PLEASE NOTE:
All market plans have a 1500 spot ‘cap’; Geopath cannot support market plans that exceed 1,500 spots
All inventory plans have a 15,000 spot ‘cap’; Geopath cannot support inventory plans over 15,000 spots
All CSV exports from the Geopath Insights Suite are limited to 12,000 spots
Summaries of changes in each API release can be seen on the Version 2.2 Release Notes page
Announcements (environment-wide updates):
API release (2.2.4) deployed to PRODUCTION at 10pm ET on Friday, 7/14/23
Monthly media status updates released to PRODUCTION at 9pm ET on Sunday, 6/25/23
Latest UI enhancements deployed to PRODUCTION at 9am ET on Thursday, 6/29/23
Resolved issues in GEOPATH API v2.2 STAGING & PRODUCTION ENVIRONMENTS:
Summary of Issue Observed | Status and Root Cause Summary | Next Step for UAT group (in STAGING) |
|---|---|---|
Market Plan is returning high reach % results for certain fleet and roadside packages | Root cause confirmed; update applied to QA on 12/22 and tested OK (pending promotion to staging environment) | READY FOR RETEST (passes Geopath testing) |
API 2.2 performance is slower than expected | Attempting additional performance improvements | READY FOR RETEST (passes Geopath testing) |
Market Plan is returning low reach % results for certain fleet packages | Root cause confirmed (configuration changes and API implementation issues); additional data-level issues are addressing LinkNYC packages; supplemental processing ran 12/20 & 12/21 and tested OK in QA & STAGING on 1/4/23 | READY FOR RETEST (passes Geopath testing) [Note: Test cases using market plan cannot exceed package size of 1500 spots] |
Certain spots are showing reach that is too high (at unit level) | Data patch is currently being processed during the evening of 12/19; supplemental processing ran 12/20 & 12/21 and tested OK in QA & STAGING on 1/4/23 | READY FOR RETEST (passes Geopath testing) |
Certain packages are showing reach that is too high (when certain spots with reach that’s too high are included) | Data patch is currently being processed during the evening of 12/19; supplemental processing ran 12/20 & 12/21 and is now undergoing testing in the QA environment before promotion (additional variations are being addressed) | READY FOR RETEST (passes Geopath testing) |
Certain place-based inventory is showing low reach & impressions counts | Data patch is currently being processed during the evening of 12/19; supplemental processing ran 12/20 & 12/21 and tested OK in QA & STAGING on 1/4/23 | READY FOR RETEST in all environments (passes Geopath testing) |
Multiple P0/P5/P18/P21+ audiences are shown in the Geopath Insights Suite (in staging) | Geopath UI Team implemented multiple types of P0+ audiences for testing/validation purposes and did not remove them from the staging build | READY FOR RETEST (passes Geopath testing); CONFIRMED RESOLVED by UAT group |
Target audiences beyond P0/P5/P18/P21+ are shown in the Geopath Insights Suite (in staging) but are not functional | UI Team implemented all expected audiences shown in Target Audiences Available for 2023 & 2024 Forecasts prior to all supporting data available | READY FOR RETEST (passes Geopath testing) |
Direct API request to /inventory/search endpoint for organization-wide queries returns a ‘417’ error | API environment configuration-related issue (developers are debugging) | READY FOR RETEST (passes Geopath testing); CONFIRMED RESOLVED by UAT group |
API /inventory/search/summary (and potentially inventory plans with reach goals) that use multiple summarization criteria will show aggregate (summed) results rather than the units that meet all summarization criteria | API bugfix applied in QA & STAGING environment and tested OK | READY FOR RETEST (passes Geopath testing) |
‘representations’ array in /inventory/measures/summary is returning NULL values | API bugfix applied in QA & STAGING environment and tested OK | READY FOR RETEST (passes Geopath testing) |
/inventory/search/summary against ‘all US population' scenarios shows an in-market population 44% greater than the USA (476M vs 333M) | API investigation complete + fix applied/validated in STAGING | READY FOR RETEST (passes Geopath testing) |
Total population in market plan UI appears to be larger than the DMA supports | UI vs API investigation complete; data fix applied | READY FOR RETEST (passes Geopath testing) |
Fleet reach/impression metrics in NYC show variances between 5 boroughs (as counties) vs NYC DMA [elevated to high severity on 2/8 as issue may impact all media types when using multi-geography plans] | UI vs API investigation underway (examining market plan vs inventory plan; Fix applied to QA environment, supplemental update applied on 2/8 and retesting of both fixes now underway | READY FOR RETEST in STAGING (passes Geopath testing) |
Package containing a single spot will render different result in inventory plan vs a single spot in search/summary [low severity; fleet only] | Search/summary API design calls for returning a maximum amount of ‘frames’ (ad units across ALL spots) vs a single frame as the default output; Fix applied to QA environment and retest complete | READY FOR RETEST (passes Geopath testing) |
‘Total population’ column is mislabeled in market plan UI (rather than ‘target market population’) [low severity] | UI fix underway (and expedited), will investigate as a candidate for release next week (given low priority) | READY FOR RETEST (passes Geopath testing) |
Operator-specific spot-specific, construction-specific, face-specific and frame-specific IDs are not consistently available in staging data | Operator-specific IDs were partially loaded into API v2.2 inventory data schema (e.g., plant_spot_id becomes spot_id_alias_primary, etc) | READY FOR RETEST (passes Geopath testing) |
/inventory/summary/measures returns an overall reach_net value that matches the largest batch of spot_ids (rather than exceeds the value of the largest batch of spot_ids as expected) | API investigation underway to determine if data-level or API-level issue persists; API patch is applied to QA and is being tested | READY FOR RETEST (passes Geopath testing); clarifying point: in scenarios where one ‘batch’ of spot_ids has a greater reach than another ‘batch’ and audience duplication likelihood is large, the largest reach net & % is returned |
Approximately 10K spots (out of nearly 1.5M) show net_reach = 0 | Data-level fix underway to address conditions where unit-level reach is null or 0 for inventory in ‘published-measured’ status | READY FOR RETEST (passes Geopath testing); now, only 30 spots with <7 people/week reach show a reach_net value rounded down to 0 |
Named searches fail to generate if more than 500 spots match the search criteria | API-level & environment-level investigation underway; database issue identified and addressed, API-level caching issue identified and addressed, caching & authentication issues under investigation | READY FOR RETEST (passes Geopath testing) |
inventory/search results may return 500 errors when page_size values >=1000 are used and/or if 1500+ frame_ids/spot_ids are used as input parameters | API-level & environment-level investigation complete; database issue & configuration issue identified and remediation applied across environments | READY FOR RETEST (passes Geopath testing) |
market plans endpoint (/inventory/plans) no longer accepts ‘calc_period_days’ as a parameter | Issue confirmed in staging environment; fix applied to QA environment | READY FOR RETEST in all environments (passes Geopath testing) |
market plan queries submitted with specific inventory parameters will yield different results each time the market plan is run | Issue confirmed in staging & QA environments; confirmed root cause and onshore data team working to resolve | READY FOR RETEST in all environments (passes Geopath testing) |
reach and frequency may fluctuate at different package size levels for roadside media | Roadside reach model is an estimate, which is undergoing continuous testing for improved consistency between package sizes. In addition, USA-wide reach estimates require supplemental data updates for accuracy. | READY FOR RETEST in all environments (passes Geopath testing) |
certain target audiences run against single spots in inventory plan (via Insights Suite UI) will yield 0 & N/A results (workaround is to use Explore module and filter on single spot for target audience) | Code fixes have been tested OK in lower environments and has been promoted to STAGING & PRODUCTION | READY FOR RETEST in all environments (passes Geopath testing) |
Hispanic audiences 18+ / 21+ are only available as more granular audience segments, no aggregate audience (e.g., ‘Hispanic persons 18+’) exists for 2023 data (as context: 200+ additional target audiences will be available by EOM March) | Works as designed; not a defect, but an requirement not previously addressed in the target audience list for 2023. Incremental 200 audiences have been promoted to STAGING & PRODUCTION APIs, with UI updates to occur on 3/27 | READY FOR RETEST in all environments (passes Geopath testing in API; Production UI updates to include new audiences scheduled for 3/27) |
market plans show lower reach % in 5 boroughs of NYC compared to DMA (the reach % in 5 counties of NYC should be higher than the DMA) | Code fix has been tested OK in lower environments and has been promoted to STAGING & PRODUCTION | READY FOR RETEST in all environments (passes Geopath testing) |
certain large market plans reach market-level maximums (e.g., 95% of DMA) | Code fix has been tested OK in lower environments and has been promoted to STAGING & PRODUCTION | READY FOR RETEST in all environments (passes Geopath testing) |
when unpublished spots or spots with no reach are included in a batch requested from inventory/measures/summary, a 500 error is returned | Code fix has been tested OK in lower environments and has been promoted to STAGING & PRODUCTION | READY FOR RETEST in all environments (passes Geopath testing) |
when using the UI, some market plans that include ‘operator’ parameter won’t show results despite rendering a nonzero count of ‘total spots’ for the selected criteria | Root cause of issue identified (UI is passing incorrect parameters to API; have deployed a hotfix into production & staging UIs) | READY FOR RETEST in all environments (passes Geopath testing) |
/inventory/measures/summary times-out when too many batches of spots are submitted for aggregation | Infrastructure and API performance enhancements have been applied to extend performance on /inventory/measures/summary endpoint (should increase the maximum number of spots per batch supported as well as the number of batches allowable per API call) | READY FOR RETEST in all environments (passes Geopath testing) [technical note: a workaround exists, as API users can submit multiple batches via multiple API calls in the event of timeouts] |
inventory statuses of some units are incorrectly shown in the insights suite (e.g., “audit level 1”) | relational data update with locations and statuses was not universally applied in all circumstances and user interface ‘map tiles’ were not completely updated | READY FOR RETEST in all environments (passes Geopath testing) |
for place-based and fleet market plans with a large count of units, reach %s approach market-level maximums (e.g., 85% of DMA) | place based & transit reach boundaries are undergoing continuous testing for improved consistency between package sizes; updates applied in QA & STAGING | READY FOR RETEST in all environments (passes Geopath testing) |
retrieving place-specific audience measures via inventory/search/summary by summarizing by place (rather than using place_name as an input) yields reach and population values that represent all the DMAs returned by the search query rather than the DMA containing the specific place | Code has been investigated and enhancement request has been applied in both QA and STAGING environments | READY FOR RETEST in all environments (passes Geopath testing) |
when computing roadside R/F for some types of roadside panels, reported reach % may be excessively high or excessively low | Root cause of issue identified and resolved (API code-level issue) | READY FOR RETEST in all environments (passes Geopath testing) |
certain large market plans in scheduled fleet scenarios report excessively high reach | Root cause of issue identified and resolved (API code-level issue) | READY FOR RETEST in all environments (passes Geopath testing) |
start and end dates for a 1-day time period requested via /inventory/measures/summary, when spanning multiple unique calendar dates, will return audience measures that represent multi-day periods | API code & related logic has been updated and a fix has been applied in Geopath’s QA environment (will be promoted to staging during the first week of May) | READY FOR RETEST in all environments (passes Geopath testing) |
local_dma produces different R/F outputs than using equivalent DMA on inventory/search/summary | Root cause of issue identified and resolved (API code-level issue) | READY FOR RETEST in all environments (passes Geopath testing) |
inventory plans, market plans, and inventory/search/summary for a single inventory unit should return the same measures as inventory/spot/{spot_id} | Root cause of issue identified and resolved (API code-level issue) | READY FOR RETEST in all environments (passes Geopath testing) |
inventory/measures/summary returns 500 error when running two spots from the same frame | Root cause of issue identified and resolved (API code-level issue) | READY FOR RETEST in all environments (passes Geopath testing) |
market plans for some scheduled fleet networks will not return audience measures | Code-level review has completed and a fix has been applied in all environments | READY FOR RETEST in all environments (passes Geopath testing) |
market plans for some scheduled fleet networks will return negative numbers for audience measures | Code-level review has completed and a fix has been applied in all environments | READY FOR RETEST in all environments (passes Geopath testing) |
market plans for certain roadside unit types (digital urban panels, digital video panels, urban panels) will not return audience measures | Code-level review has completed and a fix has been applied in all environments | READY FOR RETEST in all environments (passes Geopath testing) |
market plan using 3+ separate media class aggregation breakouts may time-out | Code-level review has completed and a fix has been applied in all environments | READY FOR RETEST in all environments (passes Geopath testing) |
inventory plans for some CBSAs will not return audience measures | Code-level review has completed and a fix has been applied in all environments | READY FOR RETEST in all environments (passes Geopath testing) |
Known issues in GEOPATH API v2.2 STAGING & PRODUCTION ENVIRONMENTS:
Summary of Issue Observed | Status and Root Cause Summary | Next Step for UAT group (in STAGING) |
|---|---|---|
some R&F metrics generated by the inventory/summary/measures (inventory plan) do not match inventory/plans (market plan), inventory/search/summary (explore), and/or inventory/spot/{spot_id} (spot-level details) | API-level & data-level investigations completed and fix applied; remaining mismatches occur in <2.5% of cases, likely due to API-level logic; investigation into edge cases continues | PARTIALLY RESOLVED IN ALL ENVIRONMENTS (GEOPATH CONTINUES INVESTIGATING THE REMAINING SCENARIOS WHERE MISMATCHES OCCUR; THIS SMALL MISMATCHING % SHOULDN’T BLOCK INTEGRATIONS) |
retrieving aggregate audience measures where a specific market (or collection of markets) is unspecified, and the implication is 'aggregate all inventory across the USA’, reach numbers may appear too low | Code has been investigated and enhancement request is in progress | NO ACTION REQUIRED YET (fix is in progress and awaits deployment/testing in STAGING environment) |
named searches (i.e. downloadable results in JSON format) for very large spot counts (e.g., all roadside spots) do not always consistently deliver downloadable outputs | Infrastructure and API code-level optimizations are under investigation; multiple environment configuration updates applied to be more performant | NO ACTION REQUIRED YET (reasonable scaling has been applied to production environment but users cannot export ALL inventory in one request; guidance is to break-down named searches by DMA for best results) |